I was taking shifts as “shift leader”, which means that for 8 hours I am coordinating the data-taking operations. 7 am, start of shift, and for the past hour the detector has been in a state that can be best summarized as “challenging”.
The night shifter informs me that while there was stable beam in the accelerator, and the experiment was happily recording data from proton-proton collisions, we had a power cut. The status screen is full: more than 8000 single alarm lines are present. “Bon courage,” he says, as the recovery operations will occur during my shift.
Normally, the control room is staffed with eight people 24/7, but there are already six more at 7 am today; the system administrator is restarting the servers. For about two hours, experts could not remotely access the computers which control the detector, though the network was working fine here at Point 1.
Usually, a power cut requires no less than six hours to recover. But this time it was quite clear there was something more unusual occurring: the experiment magnets were heating up. These magnets are superconducting and are kept at 4.8 K at all times. If they warm up completely, it can take weeks to cool down again. The big magnet (“toroid”) was already at 22 K. I was looking at this when Alex, the ATLAS run coordinator, calls me (hands free) from his car: he’s driving to the daily meeting, which takes place at the other site of CERN, and has to report about the status of ATLAS. The magnets were driven to a “slow dump” of the current, which is good, but then one was driven by some other event to a “fast dump”, which heats it up, while the current circulating in it decreases quickly.
During that meeting, we learned about the real cause of the power cut: a marten dug under the fence protecting a high voltage transformer and caused a short circuit by walking or jumping near the 66 kV line. The cables were severely damaged by the spark. The poor animal would not learn a lesson from this excursion, as he did not survive.
CERN is located at the edge of the Jura regional park, a place where hunting is forbidden: foxes, deer, boars are normal sights, especially early in the morning or late at night. I even met a badger once. Interaction of wildlife with particle accelerators is not new. The accelerator near Chicago – the Tevatron at Fermilab - was affected by a similar problem when a raccoon managed to enter an underground area and touched a high voltage cable. But the problem that time was quite minor.
By 9.30, all of the experts have arrived at the control room for the daily meeting, when colleagues in charge of every part of the detector report about data-taking and possible problems, and the plan of the day is sketched out. The power had been missing for a few seconds and, in some of the racks, the circuit breaker went off. By noon most of the racks could be repowered and all sub systems are being checked.
As the morning went on, all our experts were working to turn on their part of detector once again. Most of my job was to “open the door”, metaphorically, to every expert who connects remotely. The sound alert for the access request is really a doorbell. The control room is now crowded, with about 30 people, but we are looking at the screen, where an increasing temperature of a LHC sector is a sign that there will be no collision for several days. This delay will be named “the days of the marten”, no doubt.
By the time my shift came to an end, calibration of most of the sub-detectors had begun, with the forecast that the next days we’d mostly look at the naturally provided high energy particles: cosmic rays. A good end to a difficult day, thanks to the quick response of countless ATLAS teams.