To answer Geary's last question first. There apparently are at least two "modified chilled water" measurements. The one which can be seen from the STAR Online website (DB monitoring->ONLINE PLOTS->Environment->Platform) is actually for the WAH temperature chillers (i.e. those big noise fans which regulate the temperature for us humans). I am working on getting the historical values for this.

The MCW which is cooling the LV supplies and tray on-board electronics is not in the ChannelArchiver, hence no historical information available (there appears to be no correlation between the two MCW's). There is a readout of the input and output temperature by the magnet control monitor. Currently it is reading 62.8F input and 65.6F output. From re-reading Bill C. reply to the January 25th email it refers to this MCW:

 

There is a problem with the MCW cooling today. It's running up around 65 
F instead of the desired 62 F. The problem causing this increased 
temperature goes back about 6 weeks, when a problem developed with the 
MCW chiller. We've been running with the Tower water, which works fine 
as long as it's cold outside. Today a warm wet storm is coming over the 
island, with Temps in the 50's and a lot of rain. It is expected that 
the temperature outside will drop again tonight, and stay low enough 
that we should be able to maintain the desired 62 F through the week.
From a meeting I attended this morning, it sounds like after 6 weeks 
this problem with the MCW chiller is finally getting some attention.
If we can't run with the current temperatures during the day today just 
let us know, and we'll have to shut down.

 

According to the shift log both MCW's were fixed by January 30th (see 14:23 entry)

 

 

In fact this Tuesday (2/2) when I started my shift someone indeed came to do the followup work – no idea what he did though as he never came back to the control room.

 

Back to the PL512 temperatures. So after looking at the temperatures of Run 9 and 10 for the channels which hit the 42 celsius HIGH threshold (and two randomly chosen ones) here are my observations.

 

For the two randomly chosen ones the temperatures are in blue and terminal voltage (to see when the channel was on) in pink

 

 

The large temperature jump between the two Runs is due to the East side not being fully used as most of the trays weren't install during last years run. This means the supplies beneath East 1 weren't powering trays and hence there wasn't warm air flowing upwards. One other interesting thing to note is brief peak in temperatures just before the 26th of January, i.e. on the 25 the MCW for the platforms went up a few degrees.

 

As for the west

 

There isn't a dramatic jump in temperatures between the two runs given the West was almost fully TOF'd up (except for the trays missing behind the TPC support arm). No idea yet why there is this sudden in temperature drop towards the end of Run 9. As with the East graph from above there is a peak in temperature just before the 26 of January. As for the sudden jump of 1 degree between the 11th and 26th, read on :)

 

Here is the Run9/10 temperature history for one of the two channels in the bottom most PL512 for the West which hit the 42 degree alarm threshold

 

Here one sees a dramatic rise in the temperature just after the 11th of January and then the 42 C peak on the 25th. Similarly for the other channel on this supply which had a temperature alarmed

 

 

If one zooms in on just Run 10 temperatures

 

 

The red is the terminal voltage for this channel, so the dip in temperature in late December is due to TOF just not being turned on. As for the sudden 6 degree spike in temperature just after the 13 of January, this coincides with the supply actually being swapped out. I was throwing ideas around with Daniel and since he is going through the QA plots he reminded me that was the first access day, where we swapped lvps3 (one channel had become faulty causing us to temporarily loose tray 58) with lvps10. To a certain extent it makes sense, a different supply will have a different temperature profile – although I am also shocked by how much the temperature jumped.

 

Since this is the bottom most supply, how did it affect the ones above it? Well, lvps9 which is the middle supply in 1B4 had five channels reach the 42 alarm threshold. Looking at two of them

 

 

In both cases one sees the temperature be pretty much the same with Run 9 until shortly after January 11th (i.e. January 13th when the supply got swapped). After the supply swap the "nominal" operating temperature increase by a couple of degrees. Combined then with the increase in temperature of the MCW on the 25th of January leads to those channels reaching the alarm.

 

When reporting this increase in temperature, by chance the January 24th values I compared the 42C temperatures with were just in a "cooler" period – hence the more dramatic effect of 4-7 degree temperature jump. The increase in temperature of the chilled water on the 24th caused a 2 degree jump, while the swapping out of the bottom most supply in 1B4 on the 13th caused a 2-4 degree jump.

 

As for the comment of