Period Coordinator: Ashik Ikbal Sheikh

Period Coordinator: Xiaoxuan Chu

Tuesday, July 8 

RHIC Schedule:

56mHz yesterday, then AuAu200 GeV, leveling at 100kHz for HiLumi, vertical and longitudinal cooling on. 

Today:

Possibility of getting the tetro replaced at 1004, which may take about 8 hours. 

Then physics store

9AM CAD daily meetings: MWF  
STAR operation meeting is still daily

 

STAR Status:

Magnet temp is high, no magnet trip alarm but Prashanth called and he received the notice from CAS that one sector of the magnet has tripped, magnet is not working correctly. Detectors were put in magnet ramping down mode.

MCR just called us, they will switch to low lumi, and it will last 2hr and then they will dump the beam, they need 2hr access.

Bring magnet to half field; If we keep half field, require 13kHz leveling from the beginning of next fill

 

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice

HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Detector Issues:

• Anode sector 4:5 tripped several times: lowered the voltage (1390 V to 1345 V) and raised the current (2 uA to 4 uA),
• iTPC RDOs 10:2 and 16:3 were masked out
• No trip alarm when sTGC HV Cable 28 tripped: David will help
• TOF RDOs failed for too many trays.
  "tof01 CAUTION tofMain tof.C:#470 TOF: too many trays (103>5) require power cycle -- consider TOF/MTD CANBUS reset! "
  "tof01 CRITICAL tofMain tof.C:#495 Too many recoveries -- POWERCYCLE TOF LV ".

Power cycle TOF LV didn't work, then reset TOF Canbus, maybe not successful, not work:

"mtdMain Too many recoveries -- POWERCYCLE MTD LV" tofMain Too many recoveries -- POWERCYCLE TOF LV"

canbus restart and MTD LV power cycle, fixed, follow the procedure from the instruction

 

Shifter leader: write the shift log timely and accurately

Perform calibration run today

 

Plan:

Vernier scan this week: will discuss with CAD, Jeff will help to put back the vernier scan trigger config

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

Sunday, July 7

RHIC Schedule:

56 MHz: 900-1700, APEX now. Then AuAu200 GeV, leveling at 100kHz for HiLumi, vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF  
STAR operation meeting is still daily

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice

HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Issues:

• iTPC RDO i22:4 masked out
• sTGC ROB 15: power cycle
• TOF LV too many recoveries: power cycle
• DAQ EVB error: Tonko helped to flush the files to HPSS, fixed
• Accidently turned off TPC FEES: Tonko and Alexei helped, power cycle VME 58

Shifter leader: write the shift log timely and accurately

Perform calibration run today

 

Plan:

Vernier scan next week

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

 

 

Sunday, July 6

RHIC Schedule:

AuAu200 GeV for the long weekend. Leveling at 100kHz for HiLumi; vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF  
STAR operation meeting is still daily

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice

HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Issues:

• Many BSMD auto recoveries; require restarting RDO2; power cycle VME 39 for RDO2; then we got red alarms on low currents from the associated crates: Oleg and David helped, we turned on the FEEs related to those crates from the BEMC Main Control GUI, fixed; 3 runs without BSMD, 2 runs partially affected
• ITPC: RDO S16:2, S20:2, S12:1and S03:4 masked out
• "FCS: DEP02:7 failed”: fixed by starting a new run
• Too many TOF tray recoveries: reset Canbus, fixed
• EQ4 not working: power cycle VME 84 EQ4, fixed
• GMT gas was switched
• Check laser in the trigger when changing to low-lumi

Perform calibration run today

Shift leader: write the shift log timely and confirm QA plots are good with online crew for each run

 

Discussion:

Figure out the cause of the platform power dip (Wed, July 3)

Vernier scan next week

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

Saturday, July 5

RHIC Schedule:

AuAu200 GeV for the long weekend. Leveling at 100kHz for HiLumi; vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF  
STAR operation meeting is still daily

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice

HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Issues:

• DAQ monitoring dead and L2 machine off: Jeff helped bring them back
• "l2new l2new.c:#2181 More than 2000 timeouts...” twice: https://www.star.bnl.gov/public/trg/trouble/L2_stop_run_recovery.txtand reboot trigger, fixed
• VPD error during pedestal_rhicclock_clean: Tonko helped, fixed; should be no problem we can go ahead with the pedestal_rhicclock_clean
• iTPC RDOs auto recovery failed: manually power cycle, fixed; iTPC S5-3 masked out
• sTGC HV: 3:6 tripped: cleared
• TOF gas switched; TOF tray error 23 and 32: fixed
• EPD East<ADC> has a light yellow bands, back to normal since next run
• sTGC FOB04 error: fixed

 

Discussion:

Figure out the cause of the platform power dip (Wed, July 3)

Vernier scan next week

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

Friday, July 4

 

RHIC Schedule:

AuAu200 GeV for the long weekend. Leveling at 100kHz for HiLumi; vertical and longitudinal cooling on. 

9AM CAD daily meetings: no mgt today  
STAR operation meeting is still daily

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice: HiLumi: DAQ rate should be ~2kHz, deadtime ~<25%; LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Issues:

• A smell like burning plastic/wiring was reported at 1430, Prashanth called CAS, we were informed to monitor the smell and not call the fire department if no smoke is seen / no fire alarm is tripped. The smell decayed on its own over the next 30 minutes or so.

 

• Power dip:
  
  • Power dip and magnet trip at 7:30pm, shifters were following the checklist for power dip recovery. At that momentum, subsystems were fine.
  • But at 9:10pm, CAS people came and worked in the dap room; we got gas system and water system alarms, CAS set the system into bypass mode; we lost power from the platform inside; people from fire department came and nothing obvious found.
  • Experts came to STAR or joined online to get back the subsystems.
  • At 10:10pm, CAS brought back the power for the platform.

 

• Detectors:
  
  • We didn’t have time to perform daily calibration yesterday: need calibration runs today
  • MTD RDOs auto recoveries failed/too many recoveries, and requiring TOF trays power cycle: TOF/MTD CANBUS reset
  • ETOW Crate ID 1: power cycle; EEMC GUI froze: power cycle corresponding VME and “reload master”
  • If TPC anode trips serval times: call Alexei first and follow the expert’s advice

 

Discussion:

Need to figure out the cause of the platform power dip

Vernier scan next week

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

Thursday, July 3

 

RHIC Schedule:

AuAu200 GeV Continued after maintenance. Starting from next leveling at 100kHz for HiLumi; vertical and longitudinal cooling on. 9AM CAD daily meetings: no mgt this Friday  
STAR operation meeting is still daily
Physics for long weekend

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice; leveling to 100kHz from next fill
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Issues:

• TPC computer froze, fixed by experts: shifters should be aware of alternative solution by connecting SC5, in case it froze again
• TPC Anode S4 C5 trips: reduce voltage by 45V
• iTPC S18-4 was masked out for “db-check” error message (means PS info is not consistent): we should start the new run with this, keep it in mind, now it’s unmasked; S18-1 stays masked
• AC water leaking: will be fixed today

  

Discussion:

• Need to discuss about vernier scan next week

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

02 July 2025

(Period Coordinator: Xiaoxuan Chu)

Maintenance day Wednesday (July 2, today 800-1600), AuAu200 GeV Continues afterwards.

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice; Will request leveling to 100kHz from next fill
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Issues:

• ZDC TCIM wrong threshold after the power dip, ZDC rate was not reading correctly at the beginning of the fill (noon time yesterday), fixed by reloading the threshold values. Reload ZDC TCIM threshold values after the power dip – add this instruction in the checklist
• EPD: was off since the power dip, didn’t take day for two fills, GUI doesn’t show it’s off, but the online plots tell: more attention from the shifters;

SLs should confirm with the crew that QA are good for each run;

Shifters follow the checklist for the power dip;

Calibration runs should be performed properly.

• Laser PC was frozen, fixed
• ETOW crate ID error: toke out for one run and included it in the next run, fixed
• iTPC RDOs auto-recoveries

 

Today’s schedule:

Magnet is off now

BTOW: Oleg and Tim

EPD: Tim

TPC electronics: Tonko

FST: Prithwish

eTOF: Alexei

EIC related walkthrough

 

Discussion:

We will request leveling to 100kHz from the next fill (inform CAD)

Jeff will implement dimuon and BHT1 trigger adjustments: 

• Low-lumi: Remove prescale 1 from and save them to st-physics 
• Hi-lumi: will cut rate to forward, and upc above 85 khz

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

 

01 July 2025

(Period Coordinator: Xiaoxuan Chu)

RHIC Schedule:

AuAu200 GeV Continued. Intensity up to 1.95x10^11 at the level of the best stores. turn-around between fills is about 40 minutes; 100-140KHz ZDC rates, request leveling at 85kHz for HiLumi; vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF only  
STAR operation meeting is still daily. 
Mainly physics, maintenance day Wednesday (July 2)

 

STAR Status:

Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice; 85kHz leveling
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%

 

Everything has been brought back after the power dip last night; cosmic data-taking now

• TOF: Tray 47 masked out
• FST: Failure Codes were not solved by power cycle, access requested for Emergency Restart procedure, fixed
• BTOW: RDOs and configuration failed, fixed
• EEMC: communication lost, fixed

 

Plans:

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle 85kHz leveling, first 4 hours

low-lumi: 1mrad cross angle and 13kHz leveling 

 

Discussion:

Test for no leveling fill (if before 12:00 EDT): Tonko will call in and help

Prescale BHT1 into low-lumi, for high-lumi we stay with 10: Jeff

 

Plan for tomorrow (Wed 800-1600, maintenance):

BTOW: Oleg, we will ask Tim for help, with magnet down

TPC electronics: Tonko

FST: Prithwish

eTOF: Alexei

EIC related walkthrough

 

Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth (if no response, down the list). Inform shift leaders about this procedure. 

https://drupal.star.bnl.gov/STAR/public/operations

30 June 2025

(Period Coordinator: Zhangbu Xu)
(incoming period coordinator: Xiaoxuan Chu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 1.95x10^11 at the level of the best stores. turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 85kHz for HiLumi; 
vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF only  
STAR operation meeting is still daily. 
Mainly physics, maintenance day Wednesday (July 2)

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 06/20; 85kHz leveling; 
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%
If these numbers are off by a lot, something is wrong, restart run if you cannot find the root cause. 

·       zdc_mb 1.2B; BHT3 1/nb; (exclude zdc_mb in HiLumi and halfField)

·       West poletip trip at 9:30PM. CAS people replaced the broken regulator PS.

·       BTOW canbus connection down. Access at 1:30AM to power cycle (Oleg and David)
SCSERV power cycled => communication restored
BEMC back 
(radstone board needed to be rebooted)

·       AC water leaking: fixed?

·       How can we achieve the luminosity goal?
vxWork machine login user/password, BEMCCAN

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle 85kHz leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

29 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 1.95x10^11 at the level of the best stores. turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 85kHz for HiLumi; 
vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF only  
STAR operation meeting is still daily. 
Mainly physics, maintenance day next week (July 2)

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 06/20; 85kHz leveling; 
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%
If these numbers are off by a lot, something is wrong, restart run if you cannot find the root cause. 

·       zdc_mb 1.16B; BHT3 830/ub; (exclude zdc_mb in HiLumi and halfField)

·       AC water leaking: fixed?

·       “Master reload” fixes the issue. ETOW instruction 

05:36:15

1

etow

CAUTION

etowMain

emc.C:#846

ETOW: configuration failed -- watch ETOW triggers or restart run.

 

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle 85kHz leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

28 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 1.95x10^11 at the level of the best stores. turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 85kHz for HiLumi; 
vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF only  
STAR operation meeting is still daily. 
Mainly physics, maintenance day next week (July 2)

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 06/20; 85kHz leveling; 
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%
If these numbers are off by a lot, something is wrong, restart run if you can not find the root cause. 

·       zdc_mb 1.1B; BHT3 700/ub; (exclude zdc_mb in HiLumi and halfField)

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Seven RDOs masked out. test if the critical message works (issue resolved). 
Unmasked all, but need to fix some issues. Remasked 3. 

 

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle 85kHz leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

27 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 1.95x10^11 at the level of the best stores. turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 85kHz for HiLumi; 
vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF only  
STAR operation meeting is still daily. 
Mainly physics, maintenance day next week (July 2)

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 06/20; 85kHz leveling; 
HiLumi: DAQ rate should be ~2kHz, deadtime ~<25% 
LoLumi: DAQ rate should be ~4kHz, deadtime~<50%
If these numbers are off by a lot, something is wrong, restart run if you can not find the root cause. 

·       zdc_mb 107M; BHT3 600/ub; (exclude zdc_mb in HiLumi and halffield)

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. test if the critical message works. 
Unmasked all, but need to fix some issues. Remasked 4. 

·       BTOW configuration issues this morning.

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle 85kHz leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

26 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 2.2x10^11 at the level of the best stores. turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 85kHz for HiLumi; 
vertical and longitudinal cooling on. 

9AM CAD daily meetings: MWF only  
Mainly physics, maintenance day next week (July 2)

 

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 06/20; 85kHz leveling next fill; 

·       zdc_mb 950M; BHT3 500/ub; (exclude zdc_mb in HiLumi and halffield)

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. Send Tonko critical support (still masked out). 
test if the critical message works. 

·       TPC trip alarm handler did not go off? Single channel trip no alarm handler goes off. 
Alexei: will talk to Tonko and David; FIXED 

·       L4 halfField tracking FIXED.

·       Jamie requested two more scaler rate triggers; luminosity monitoring. 

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle 85kHz leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

25 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 2.2x10^11 at the level of the best stores. potentially 8 hour fill, turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 100kHz (did not happen last fill) for HiLumi; 
vertical and longitudinal cooling on. 
APEX Wednesday from 9AM—midnight (STAR magnet OFF)

 

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 06/20
potentially 8 hour fill, we do 5+3

·       zdc_mb 1B; BHT3 630/ub; (exclude zdc_mb in HiLumi and halffield)

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. Send Tonko critical support (still masked out). 

·       TPC trip alarm handler did not go off? Single channel trip no alarm handler goes off. 
Alexei: will talk to Tonko and David

·       Unusual spots in TPC? (Flemming/Gene)
more discussion offline, and get back to see if we need to do anything ONLINE 
about cluster algorithm, Gene investigated cluster and observed that TPC tracks lose hits at hiLumi. 

·       halfField L4 reconstruction (B field is incorrect in the reconstruction? Diyu and Jeff) 
B=0.5 or 0T, Jeff fixed this. hardcoded 0.5 in various places, not used in TPC tracking.
strange tails due to calibration server not updated for the first run. 

·       Slow control database not updating since June 9, David restarted it. 

 

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle not leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

24 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity up to 2.2x10^11 at the level of the best stores. potentially 8 hour fill, turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, request leveling at 100kHz (did not happen last fill) for HiLumi; 
vertical and longitudinal cooling on. 
APEX Wednesday from 8AM—midnight (STAR magnet OFF)

 

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon
potentially 8 hour fill, should we do 5+3

·       zdc_mb 1B; BHT3 600/ub; (exclude zdc_mb in HiLumi and halffield?)

·       Hot weather yesterday and today, run magnet at half-field yesterday afternoon, and today 8AM-7PM. Configuration production_halffield_AuAu_200 with UPC_mb and zdc_mb as main triggers (1mrad crossing angle and 13kHz leveling)

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. Send Tonko critical support (still masked out). 

·       TPC trip alarm handler did not go off? Single channel trip no alarm handler goes off. 
Alexei: will talk to Tonko and David

·       Unusual spots in TPC? (Flemming/Gene)
more discussion offline, and get back to see if we need to do anything ONLINE 
about cluster algorithm, Gene investigated cluster and observed that TPC tracks lose hits at hiLumi. 

·       halfField L4 reconstruction (B field is incorrect in the reconstruction?) 

·       HiLumi vpt100->vpd50 reduces event rate and deadtime 

·       EPD and BBC rates are higher than fullField 

 

Plans

STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle not leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
When change magnet to HalfField or from halffield back to fullField, call Prashanth. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

23 June 2025

 

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity at 1.9x10^11 for now. potentially 8 hour fill, turn-around between fills is about 40 minutes; 100—140KHz ZDC rates, leveling at 100K or 80KHz; 
vertical and longitudinal cooling on. 

 

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon
potentially 8 hour fill, should we do 5+3

·       Enough AbortGapTest, do not need to take it anymore. 
STAR: 1Hz is better; currently we are doing 1Hz abort-gap cleaning 

·       zdc_mb 1B; BHT3 580/ub; 

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. Send Tonko critical support (still masked out). 

·       TPC trip alarm handler did not go off? Single channel trip no alarm handler goes off. 
Alexei: will talk to Tonko and David

·       BSMD during pedestal run should be OFF; no change to instruction 

·       VPD vertex cut range: max tac diff 4404; min tac diff  3596 (Geary to Jeff); done

·       TOF auto-powercycle frequently?

·       Vertex scaler to CAD low statistics during hiLumi run, Jeff removed zdc-mb prescale to L4 and increased data rate by 10. OK

·       Vertex position +-5cm, should call MCR to correct for the vertex position if  |meanZ| >5cm during low-Lumi data-taking 

·       Unusual spots in TPC? (Flemming/Gene)
more discussion offline, and get back to see if we need to do anything ONLINE 
about cluster algorithm

 

 

Plans

Physics stores the rest of the week until Monday 
STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle not leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

22 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity at 1.9x10^11 for now. 
vertical and longitudinal cooling on. 

 

STAR status

 

·       Run with high-lumi and low-lumi (4+2) dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon

·       Abort-gap cleaning with ¼ Hz on since yesterday afternoon, 
took a couple of abortGapTest for the last three fills. 
Both STAR and sPHENIX requested this morning to switch back to 1Hz

·       zdc_mb 940M BHT3 410/ub; 17 hours data-taking yesterday 

·       AC water leaking: fixed?

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. Send Tonko critical support (still masked out). 

·       TPC trip alarm handler did not go off? Single channel trip no alarm handler goes off. 
Alexei: will talk to Tonko and David

·       VPD vertex cut range: max tac diff 4404; min tac diff  3596 (Geary to Jeff)

·       Vertex scaler to CAD low statistics during hiLumi run, Jeff removed zdc-mb prescale to L4 and increased data rate by 10. 

·       Vertex position +-5cm, should call MCR to correct for the vertex position if  |meanZ| >5cm during low-Lumi data-taking 

·       EPD IV scan takes about 16 minutes, note in detector list. 

·       fcs pre-shower LV issue overnight. Akio turned it on this morning, stays ON. 

 

 

Plans

Physics stores the rest of the week until Monday 
STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle not leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
EIC prototype safety walkthrough, if there is downtime >1/2 hour 
maintenance day is July 2. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

21 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued. Intensity at 1.7—1.8x10^11 for now. 
vertical and longitudinal cooling on. 

 

STAR status

 

·       Run with high-lumi and low-lumi dual mode. This mode for the remaining operation until further notice. Implemented Friday afternoon 

·       Abort-gap cleaning check, preliminary findings: no visible effects in DCA from cleaning which started at the beginning of the fill; Visible effect when cleaning was done only at the end of the fill. Fills were with 1 Hz cleaning, will try ¼ Hz?
take a couple of abortGapTest

·       4 BTOW PMT boxes tripped several times, called at 1AM and 2AM. 
Oleg fixed them in the morning. 

·       zdc_mb 891M BHT3 210/ub

·       sPHENIX observed increase background when STAR collision rate is high (no crossing angle) 

·       AC water leaking: maintenance for air conditioning came to the control room.

·       TPC anode trips (lower HV by 50V), raised back to full HV, setting tripping limit to 2uA. Three RDOs masked out. Send Tonko critical support. 

 

 

Plans

Physics stores the rest of the week until Monday 
STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle not leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
EIC prototype safety walkthrough, if there is downtime >1/2 hour 
maintenance day is July 2. 

20 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued, beam since last evening. Ground fault at main booster magnet was fixed. Intensity at 1.7—1.8x10^11 for now. 

 

STAR status

 

·       1mrad crossing angle, leveling at 13kHz ZDC rate (with 1us killerbit) current fill 

·       continuous abort-gap cleaning (~1Hz) would be an issue with background, re-assemble DAQ files chronically. Gene and Jeff work on a scheme today. Exercise abort-gap cleaning at the end of this fill, Jeff set up a special configuration for this (configuration available). 

·       Next fill will be high-lumi and low-lumi dual mode. This mode for the remaining operation until further notice. 

·       zdc_mb 841M BHT3 120/ub

·       Magnet cooling tower cleaning by CAD DONE.

·       AC water leaking

 

Plans

Physics stores the rest of the week until Monday 
STAR requests to change to high-lumi and low-lumi operation mode Friday (6 hours fill): 
high-lumi: no cross angle not leveling, first 4 hours. 

low-lumi: 1mrad cross angle and 13kHz leveling 
EIC prototype safety walkthrough, if there is downtime >1/2 hour 
maintenance day is July 2. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth 
             (if no response, down the list).  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

19 June 2025

(Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued, beam last night. maintenance this morning, probably into afternoon 

Update:

There has been an intermittent ground fault for the Booster main magnet that came to a head this afternoon.  Power supply group wants to fix it; work planning was done today and tomorrow riggers are coming in early in the morning to access the trench where the cable tray is (this has been identified as the fault location).  We were able to get a store up this evening that we will hold for 8 hours.  MCR will attempt to put up another one in the wee hours that we will hold as long as possible while the work is going on.  Our plan is to continue normal physics running when we get the booster back. — Travis (CAD)

 

STAR status

 

·       1mrad crossing angle, leveling at 13kHz ZDC rate (with 1us killerbit)

·       One of the TOF LVs is yellow (5.9).  But data seem to be OK.

·       sTGC gas alarms this morning, should write in the runlog exactly which alarm. 

·       zdc_mb 712M BHT3 80/ub

·       continuous abort-gap cleaning would be an issue with background, re-assemble DAQ files chronically. Gene and Jeff work on a scheme today. There is not a timeline when CAD starts this abort-gap cleaning. 

 

Plans

Fix ground fault at AGS main booster today 
Physics stores the rest of the week until Monday  
No 9AM meeting Thursday 
STAR requests to change to high-lumi and low-lumi operation mode Friday: 
high-lumi: no cross angle not leveling, first 4 hours. During the day, each store will be 4 hours. 
low-lumi: 1mrad cross angle and 13kHz leveling 
Magnet cooling tower cleaning by CAD, opportunity before July 2. 
EIC prototype safety walkthrough, if there is downtime >1/2 hour 
maintenance day is July 2. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth.  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

18 June 2025

 (Period Coordinator: Zhangbu Xu)

 

RHIC Schedule

 

RHIC status: 

 AuAu200 GeV Continued

vertical plane SC cooling started yesterday morning. 

 

STAR status

 

·       1mrad crossing angle, leveling at 13kHz ZDC rate (with 1us killerbit)

·       laser very few entries, Alexei will check.

·       One of the TOF LVs is yellow (5.9).  But data seem to be OK.

·       zdc_mb 580M BHT3 80/ub

·       bifurcation of the rates limits DAQ rate to be 3-4kHz. no backgrounds leaking to this bifurcation.  a lot of gaps on the order of 1-2 seconds, and these lead to smaller rates in the less detailed plots.  The 1-2 second deadtimes are TPX/TPC deadtimes. This is coming from the RDOs, but I don't know if this is fast autocorrections, or whether they are due to blocking when front end buffers are filled.   The issue went away after TPC reboot yesterday.  Jeff’s blog:https://drupal.star.bnl.gov/STAR/blog/jml/Data-Size-time-0

·       continuous abort-gap cleaning would be an issue with background, re-assemble DAQ files chronically. Gene and Jeff work on a scheme today. There is not a timeline when CAD starts this abort-gap cleaning. 

 

Plans

 

Beam development today starting 9:30AM 6-8 hours
Physics stores the rest of the week until Monday  
No 9AM meeting Thursday 
STAR requests to change to high-lumi and low-lumi operation mode Friday: 
high-lumi: no cross angle not leveling, first 4 hours. During the day, each store will be 4 hours. 
low-lumi: 1mrad cross angle and 13kHz leveling 
Magnet cooling tower cleaning by CAD, opportunity before July 2. 
EIC prototype safety walkthrough, if there is downtime >1/2 hour 
maintenance day is July 2. 

Discussion

            Thunderstorm/powerdip preventive procedure (APEX mode) when CAD informs. 
            If power dip occurs, call experts. If global interlock at DAQ room alarms, call Prashanth.  
            inform shiftleaders about this procedure. 
            https://drupal.star.bnl.gov/STAR/public/operations

17 June 2025

 

Summary

·       AuAu200 GeV Continued

·       Beam from around 10am to around 3:30pm

o   Short gas room alarm

o   Little later TPC gas PT2 yellow warning - gas pressure dipped to 2.2 then went back to 3.1 on its own. The alarm then stopped.

o   EVBx6 got problematic again and was removed from the runs

§  Keep it in until it again misbehaves

o   MCR informed us they have a solution for their problem and would like to dump a beam and have an access

·       Here is the log of the access timeline – this was a 1 hour access according to MCR

o   16:51 General MCR dumped yellow beam. Called MCR to request ~1 hr access. Oleg went in. Alexei is on his way (will be here in 10 min) and would like to access too.

o   17:20 – Oleg: Run 26167034 - 1. Started gas flow for BSMD. 2. Put HV card#2 (Tim received from CAEN) into BSMD Sy1527, verified operation is OK.

o   17:41 – Alexei: I had an access to WAH, checked both lasers- everything is OK. Still a mystery, why east laser failed last night

o   From this morning RHIC status meeting: we were asked to not hold a beam, this was in reference to yesterday’s access. I assume they were done with their 1 hour access before we requested hours. This should be better coordinated.

o   PC should be contacted for every access so they coordinate the process and clearly communicate with MCR and have a clear understanding of the process if the questions arise later.

·       More beam development after, starting with 6x6 beam.

·       New beam around 21:10

·       Unfortunately, DAQ rate was around 2kHz until 6 am this morning

o   Around 3am shift crew started noticing the high rates

§  Some Sca Hz are red:
BHT2-vpd100 ~ 110

§  TPC hot RDO was causing this. It was fixed around  6 am this morning.

o   Shift crew should be explicitly instructed to monitor the DAQ rate to be around 4 kHz

·       This morning beam dumped around 9am

·       We are awaiting beam now

·       For the moment about 303 M events collected

 

·       Tomorrow beam development whole day tomorrow.

 

·       Zhangbu will take over as a PC starting today. Have a great rest of the Run25.

Trigger/DAQ

TPC/iTPC

TOF/MTD

EEMC

BEMC

EPD

eTOF

GMT

FCS

sTGC

FST

Slow control

In progress / to do

Discussion

·       Tonko: TPC has more charge this year than last year that affects the data-taking rate this year. (4 kHz vs. 5-6 kHz last year). pp vs. AuAu is possible natural answer.

·       Jeff: there are other signs in trigger rates (e.g. UPC event size). Perhaps a background. Needs to be understood.

·       Oleg: when should BSMD be turned on?

o   J.H.: BSMD starting Friday as discussed at the trigger board meeting.

o   Oleg: will start working with the shift crew about BSMD on Thursday.

 16 June 2025

 

Summary

·       AuAu200 GeV Continued

·       Uninterrupted data-taking from 11am to 3:30 pm

·       EPD calibrations were finalized before the start of the next beam

·       Around 5pm all triggers elevated to PHYSICS

·       EVB4 and EVB6 were unmasked as instructed by Jeff

·       Around 6pm new fill (35605) and run (26166052) with all triggers in physics running

·       Around 1:20 am laser camera stopped working. Shift crew tried to restore it but unsuccessfully. Alexei needs a short access to fix the problem.

·       There were two more fills overnight, both dumped unexpectedly

·       There has been no beam from about 2:20 am (issues with the machine), but they were working with the beam, so we just waited

·       About 81M more events collected with current tally standing at about 184 M zdc_mb events collected

 

·       No maintenance day tomorrow (neither on Wednesday)

 

Trigger/DAQ

TPC/iTPC

TOF/MTD

EEMC

·       Shift crew had a hard time bringing tripped crates 3 and 4 back

·       They were instructed by expert that corresponding VME crate needs to be restarted during this procedure

·       Please have a look at the instructions manual if this needs to be added for clarity

·       Some runs were taken without ETOW in while DOs worked on bringing it back

BEMC

EPD

eTOF

GMT

FCS

sTGC

·       Was not included in the first 111x111 data-taking. Prashanth then instructed to include it as normal in the following runs.

FST

Slow control

In progress / to do

Discussion

·       Alexei: east laser may be overheated. It should be OK to run unless it overheats.

·       Should we request a short access? Neither access for STAR are urgent, so we wait for the next possibility, unless east laser fully fails.

·       There is a possibility to have separate configurations for the beginning of the beam and the lower luminosity part. To be discussed at the trigger board discussion.
 

15 June 2025

Summary

Trigger/DAQ

TPC/iTPC

TOF/MTD

EEMC

BEMC

EPD

eTOF

GMT

FCS

sTGC

FST

Slow control

In progress / to do

Discussion

14 June 2025

Summary

Trigger/DAQ

TPC/iTPC

TOF/MTD

EEMC

BEMC

EPD

eTOF

GMT

FCS

sTGC

FST

Slow control

In progress / to do

Discussion

 

 

9 June 2025

AuAu200 collisions

 

 

 

Summary

 

·       Around 22:00 yesterday machine setup started with 6x6 bunches (problems with yellow beam before that), then 12x12, then 28x28 (RHIC Fill # 35577)

·       Around 1:30 am this morning the first AuAu200 collisions with 28 bunches, followed by the:

o   BBC/VPD/ZDC setup

§  V124 setting same as in Run24 AuAu.

§  1st Yellow ch8 fine delay = 116. The only change from last year is RevTick fine delay moved from 46 to 26 upon request from Eleanor to align new RevTicks

§  Akio put in the shift log that new delays waits checks from Eleanor

o   EPD setup/calibration

o   Around 3:20am sPHENIX requested beam dump

o   Next fill requested with 6x6 bunches

o   EPD setup/calibration continued

§  Finished near the morning

o   production_AuAu2025 runs taken

 

 

·       Plan for STAR

o   Elevate triggers and declare physics

·       QA plots are down, Jeff and Wayne will be looking into this

o   Some data sending problems… work in progress…

 

 

 

Trigger/DAQ

TPC/iTPC

 

·       TPX: RDO S18 – could not be auto-recovered, was masked

 

TOF/MTD

EEMC

 

·       Note from Will Jacobs: EMC timing needs to be set. A timing run with cal_scan, BTOW+ETOW+TRG+DAQ is needed. Is there a config in the trig setup selection for that? ETOW is up can configured with HV on.

·        

 

BEMC

EPD

 

·       Are more calibrations pending?

·       Data is being analyzed

 

eTOF

 

·       What is the plan?

·       To be updated…

 

GMT

FCS

sTGC

 

·       What is the plan?

·       Prashanth will update.

 

FST

 

·       What is the plan?

·       Expert will be looking at it late this afternoon

 

Slow control

In progress / to do

Discussion

 

            Pedestal runs with BTOW+ETOW before the cal_scan

            Prashanth: condenser fan was disconnected to address the ground fault issue.

            TPC air blower was fixed by Alexei (last Friday)

            Gene: working on automating fast offline, for now contact Gene for manual reconstruction.

            Geary: vpd looks good based on the last run.

Hank: RevTicks are lined up correctly

SL reported: TPC laser run has problems – drift velocity plot seems to not represent the triggers collected. Alexei will have a look.

 

Sunday, June 8, 2025

Status & Issues:

• Gas system alarm flashed 3 times, interlock alarm row 4 went off and on, checked DAQ GIS, everything seems ok. - Keep monitoring

• Global interlock alarm for ground fault ~ 19:00. CAS watch sent experts to investigate it. - Cleared 

• GMT HV alarm, u3 trip. - Fixed; both bottles of GMT gas seem low from the GUI

• sTGC: PT3 around threshold (yellow alarm)

• TPC cosmic+ped test: start immediately after turning on the FEEs. Lots of RDO errors (iS18:1, S3:3, S4:1), masked iS18:1

• Laser: did not get any laser events in 5 min. Bringing down the laser because MCR is injecting soon. -  will test later

• EPD IV Scan take really long (~25mins, still running) -Maria will look at it

• FST: don’t turn it on tonight, will wait for the stable beam for operation  

• IR temperature has a sharp rise this morning (84F) - CAS watch sent someone to STAR, still under investigation  

Schedule & Plans:

• Much troubleshooting of the large yellow transition loss … after rewiring, full transmission to the flattop of both rings was achieved. Fixing up yellow RF is happening now, then collimator setup, then rebucketing. We expect to see collisions today (earliest ~ 17:00, likely 20:00).

• Experts wanted to be contacted if there is a beam: Akio, J.H., Maria

 

Saturday, June 7, 2025

Status & Issues:

• The magnet chiller is on; The communication error is fixed. The magnet has been full since yesterday at noon

• Alexei had an access and checked the TPC air blower flow error; the clog was cleared, and now everything is OK.

• TPC: Try to study the cosmic+ped, but have multiple TPC RDO autorecovery/bad errors (iS11:2, iS18:1, iS09:3). It seems more tests are needed. Had a new instruction from Flemming. Will do it later if there is a chance.

• A couple of EPD tests requested by Maria - EPD is ready to go

• ETOW/EEMC is under STAR crew control. Will turned the HV on

Schedule & Plans:

• Some issues with the yellow ramp - will continue to troubleshoot today 

• Rebucketing schedule (nominally) for noon - will keep us posted (depending on the troughshooting of the yellow ramp), still have a chance to have collision tonight

• Experts wanted to be contacted if there is a beam: Akio, J.H., Maria

 

Friday, June 6, 2025 

Didn’t take any cosmic run, had 2 magnet trips during the day; under APEX during the night.

We expect to see collisions for physics tomorrow night (6/7).

Status & Issues:

• Magnet tripped 3 times (8:48, 12:04; then 1:01): 

  • 1st and 2nd trip: temperature-related issue. 200 T chiller needs some prep work to be turned on (Today or Monday). Turn on the magnet when T < 80°F and run for the next three days.

  • 3rd trip: a SC related communication problem from CAD to STAR. CAS watch confirmed everything looked good from their side when we saw the alarm in the control room. (outside T is ~70°F ) - David is following this up

• Global interlock alarm at 4:35, "flammable gas detector fault". Called Prashanth and CAS watch. No problem was found, CAS watch reset the detector.

Schedule & Plans:

• Focus on power supply work and 56MHz work during the day, ramp with both beams tonight, and tomorrow morning will do rebucketing. 

• We expect to see collisions for physics tomorrow night (6/7).

• We will turn on the backup dehumidifier 

 

Thursday, Jun 5, 2025

Status & Issues:

• TPC:

  • Tonko checked a few RDOs:

    • iS02-4 had 2 bad FEEs which I masked -- RDO unmasked

    • iS03-3 can't find anything wrong -- unmasked

    • iS05-3 can't find anything wrong -- unmasked

    • iS11-2 can't find anything wrong -- unmasked

    • iS24-4 remains masked, a lot of odd issues with all FEEs

  • Owl shift did the cosmic+ped test ~ 2hs 

  • Masked TPX S18 RDO5 after it was causing many problems - owl shift

• Completed all the planned work yesterday 

  • ETOW: Replaced control card PROM: crate now displays readback and configures digitizer cards.

  • BOTW: Replaced fuse holder in tower power distribution box (East 0xa and 0xB).

  • Magnet water work.

• One magnet trip ~ 8:48 today. CAS came and diagnosed it is a cooling issue. The temperature was peaking around the time when the magnet tripped. CAS recommended we wait until they confirm the status of the pump and they bring up the power supply otherwise the magnet might trip again.

Schedule & Plans:

No changes on the RHIC schedule (follow yesterday’s note): 

Thur. (6/5): Day: PS work. Evening: beam in blue

Wednesday, June 4th, 2025

Yesterday 56MHz work during the day; 20:30, C07 booster injection bump supply tripped (CAD will address it further today). We took cosmic run since mid-night 

Injector intensity up to 1.6x10^9/bunch without recurrent instability

Status & Issues:

• TPC

  • iTPC Sectors 3 and 4 - Tonko will take a look today

  • TPX S18:5 stopped DAQ once

• BTOW: Oleg replaced fuses on BTOW crate 9. bTOW is now included in the runs. If it will be error with crate 9, just ignore it (leave a log) and continue running.

• Trigger: 1st Yellow V124 ch6 fine delay was changed from 46 to 26 (0.5nsec unit) requested by Eleanor to align new RevTicks to the old ones - Akio

Schedule & Plans:

• Work for today (please notify the Shift leader before accessing):

  • ETOW crate 2 - Tim

  • Akio - EIC prototype related work

• RHIC Schedule:

  • Wed. (6/4): 

    • Day: maintenance. RHIC sweeps at 15:00 

    • Evening: beam in blue (chrome adjustments, preliminary store mechanics setup, raise total intensity)

  • Thur. (6/5): Day: PS work. Evening: beam in blue

  • Fri. (6/6): Evening: yellow beam injection

  • Sat. (6/7): Ramp development

  • Sun. (6/8): Day: rebucketing and store setup; Evening: 56x56 store

  • Mon. (6/9): Investigation of MVTX background - expect to see some low lumi, signal beam fills

 

Tuesday, June 3rd, 2025

Status & Issues:

• AC (in the IR) maintenance work - Done

• TPC:

  • ITPC Sectors 3 and 4 have some noise in adc vs timebin - known issue (due to the fluctuation of the pedestal level), Tonko will take a look tomorrow

  • Masked S11:2

  • Unmasked S18:5, iS18:1

Schedule & Plans:

• Latest report from CAD: 

  • Blue quad PS problem was a configuration issue, - solved now; Yesterday achieved 111@1.1/bunch, 1mrad

  • Cryo ~4k started, yellow beam work this weekend

  • 56 MHz works today, no beam until later today

• STAR work plans:

  • Tomorrow is maintenance day, starting from 8:00 to 15:00 (could be 17:00)

  • Bring the detector to the magnet ramping state tomorrow morning, inform MCR, then ramp down the magnet (7:30)

  • BTOW work - see if we can access today by Oleg

  • ETOW work - Tomorrow, during maintenance day by Tim 

 

Monday, June 2nd, 2025

Status & Issues:

A miscommunication problem yesterday: MCR called us said we should expect beam in at least 20-30 mins. Shift leader told them to call us again before the injection, but did not received the call. When we called again to ask about injection ETA, they had already taken one shot. - Luckly no damage 

We informed MCR this morning in the meeting to call us before the beam. To shifters: for the future similar situation, bring detectors to APEX 

• TPC:

  • Masked iTPC S11:2;

  • iTPC S7:4, S9:3 showed spikes in the adc vs timebin histograms

  • TPX Sector 18:5 hung the DAQ many times, masked for laser run

• TOF:

  • Tray 77 hot - masked out of trigger by Geary

  • TOF Gas alarm, change from line A to B

Schedule & Plans:

• Latest report from CAD:

  • PS works during the day, blue beam expected in the night

  • Expect some delay for yellow, so back to the original schedule 6/8

• STAR work plans

• AC work in IR - ongoing

• BTOW and ETOW work - Tim and Oleg: Start from noon (~4h); After the meeting, call MCR for restricted access, bring down the STAR magnet before noon

Sunday, June 1st, 2025

STAR Swept ~ 15:00; MCR called us ~19:12; Some beam activities 21:00~2:00; MCR done of tests ~2:30, then we start to take cosmic 

Status & Issues:

• TPC: 

  • #26151029: a spike in iTPC adc vs time sector #3; power-cycled the whole sector

  • Power-cycled: S3:4, S18:5 

  • #26151036: hot channel in 2D adc plot, did a new pedestal run. After that, the spike in iTPC adc vs time sector #3 cleared. Single hot channel persists in 2D ADC plot.

  • When starting the 1st cosmic run in the morning, TPC hung the DAQ; power-cycling RDOs solved the problem.

Schedule & Plans:

• Latest report from CAD:

  • Everything from their list for last night was done; PS  work continues today, the goal is ramping blue overnight

  • Cryo reports that the yellow 4k wave should begin tomorrow. - one day earlier than the previous schedule. store setups on 6/6

• STAR work plans for tomorrow (Monday):

  • AC in IR (T looks ok so far)

  • Work on BTOW (change fuses)

  • Work on ETOW crate #2 - will check with CAD how long do we have in tomorrow’s 9:00 meeting

 

Saturday, May 31, 2025
Status & Issues:

• AC in the IR: The repair company will come on Monday. - To shift crews: please monitor the STAR hall temperature (needs to be lower than 90) - Contact Prashanth if it goes above

• MTD: SF6 Gas - installed a needle valve - works ok now

• BTOW: Errors in Crate IDs: 9 - will replace the fuses on Monday; - To shift crews: BOTW stay off over the weekends

• ETOW: Tim has time on Monday (6/2) afternoon - Will check with CAD schedule, see if we can manage an access

• TPC:

  • TPC Cathode Off alarm, along with int_row4_alm6 interlock alarm; 

  • Could not bring back TPC Cathode HV using normal procedures; contacted Prashanth and Alexei - 

  • Reset the interlock on the AB panel, now recovered

  • S18:5 and S04:1 error once

Schedule & Plans:

• Sweep ~ 3pm today, blue beam in the evening - Call MCR after sweep, remind them to call us back before injection. Once receive their call, bring detectors to APEX mode

 

Friday, May 30, 2025

Status & Issues:

• ETOW: All LV on, HV off, and crates 1,3,4,5,6 configured, with crate #2 "uninstalled" - To fix the problem for crate #2, we will need help from Tim, and need to bring down the STAR magnet. We will try to manage the access next week or during the maintenance day - Next Monday/Tuesday?

• TPC:

  • TPX: RDO S18:5 -- auto-recovery failed. Powercycled this RDO manually. - Tonko will take a look

  • TPX[33] [0xBA21] died/rebooted - reboot from the run control, doesn't fix the problem; powercycled the machine in the DAQ room

• sTGC: crate:2 rob:10 channel:1 running at alarm threshold (0.94~0.95 A). Disabled alarm for this channel. -David will confirm with Tim later

• AC in STAR Hall - Can Prashanth follow this up?

Schedule & Plans:

• Latest schedule from CAD: 

  • Beam in blue during the weekends

  • Potential work to fix the PS next Monday/Tuesday - access opportunities

  • potentially one day earlier compared to the previous schedule - store setups on 6/7

• STAR will likely sweep this afternoon - not sure the time 

• We will keep running Cosmic when there is no beam

• Call MCR tomorrow early morning to remind them to give us a call before playing beam, so that we can bring detectors to a safe state (APEX)

• Cafeteria will open until 7:00 pm starting from next Monday

• Subsystem:

  • BEMC: 

    • “network problem” - fixed

    • PMT 39 is out for now

  • MTD: SF6 gas 

  •  

 

Thursday, May 29, 2025

• FST and BTOW are also included in the run

Status & Issues:

• BEMC: Network issues? - Need to hear back from Wayne/Oleg

• EEMC: Prashanth checked the rack power and air blower - start the EEMC? - Need to hear back from Will/Scott

• Laser: Runs well now

• EPD: A few test runs (ped, pedasphys) requested by experts

• Other minor issues: 

  • Restarted the frozen run control

  • Restarted the alarm handler to unmute the alarm

  • TPX[28] - reboot TPX

  • TPX:RDO S18:5 - power-cycled

  • TPC hot channel - To shifters: please also check the 2D map for the hot channel, for details see the “TPC reference plots and issue problem solving” document in the operation Drupal page

Schedule & Plans:

• Latest schedule from CAD:

  • Blue 4k wave started yesterday; Expect to have the first night beam activity Saturday evening (blue) - Any work plans in STAR Hall for today & tomorrow?

  • Yellow 4k wave will start 6/3, store setups on 6/8

• Subsystem

  • TPC: ready

  • TOF: ready

  • BEMC: ready (FEE on, HV off) - update on the network issues (?)

  • EEMC: (?)

  • FCS: ready

  • sTGC: ready

  • EPD: ready 

  • FST: ready

  • GMT: ready

  • eTOF: HV on (1 sector), won’t be on for cosmic run

  • MTD: ready -  SF6 gas

 

Wednesday, May 28, 2025

MTD is newly included in the cosmic run

A further delay on beam is expected (see schedule & plans)

Status & Issues:

• TPC: 

  • Spike in S19 and S12; power cyced the GG; Tim found GGDII outer sector 12 twinax cable and inner sector 19 twinax were not fully seated - now fixed (spikes are gone) 

  • Alexei raised trip level from 2 uA to 4 uA for sector 4 channel 5, and increased voltage from 400 V to 1380 V.

  • Many times “TPX: RDO S18:5 auto-recovery failed.” - power-cycled manually

• Laser: double peaks in #26147055, worked well in the next run - will do another laser run for checking ~ 11:00

• FST: refilled the NOVEC coolant, started the cooling

• GMT U3 channel high voltage tripped. - cleared

• EEMC: Scott wants to check 1) turn on tower air blower; 2) turn on breakers for LV; 3) check/turn on HVSys controller AC switches; for the EEMC can be turned on - Prashanth is helping with this

• Maintenance going on for AC in IR

Schedule & Plans:

• According to CAD report this morning: 

  • Blue cooldown is still on target  (5/31) 

  • Yellow coolddown to 45k continues to be an issue; 4k wave can’t start until 6/3. Cold enough for beam activity on 6/5

  • Put rebucketing on 6/7, and experimental setup follows

• Likely sweep STAR on Saturday

• Subsystem

  • TPC: ready

  • TOF: ready

  • BEMC: today - Wayne is looking into the network issues

  • EEMC: experts will turn it on once checked rack power and cooling

  • FCS: ready

  • sTGC: ready

  • EPD: ready 

  • FST: cooling is running, will test the HV shortly

  • GMT: ready

  • eTOF: won’t be on for cosmic run

  • MTD: ready -  SF6 gas

Tuesday, May 27, 2025

Status & Issues:

• MTD: follow up on yesterday’s gas alarm - Flow master Hastings closed all flowmeters at about 1AM. Aleix rebooted Hastings and restored Freon and I-butan, but failed with SF-6, will fix it this week.

• Trigger: The BCE and bce_dsm2 nodes were put back into the trigger

• Masked out EVB[8] this moring with the instruction from Jeff. EVB[8] will stay out until further notice

• Light problem outside the control room - Prashanth is following

 

 

Schedule & Plans:

• TPC: Power-cycle TPC Gating Grid when David is around, in case of GUI mess-up. Some instruction is needed: crate 50

• Subsystems:

  • TPC: ready

  • TOF: ready

  • BEMC: tomorrow

  • EEMC: (?)

  • FCS: ready

  • sTGC: ready

  • EPD: ready 

  • FST: today

  • GMT: ready

  • eTOF: won’t be on for cosmic run

  • MTD: will not be ready until next week

 

Monday, May 26, 2025

Status & Issues:

Trigger detectors are ON (BBC, ZDC, and VPD) now

GMT, sTGC are newly included in the cosmic run

• Trigger: Power cycled BBC crate (VME-78) and rebooted all trigger nodes due to L2 error.

• TOF: a few times high error in TOF of DAQ, and DOs reset TOF CANBUS and LV powercycled - To shifters: depends on the error messages, maybe just try to restart a new run first; If it doesn’t work, then try to power cycle the LV / CANBUS based on the instruction

• TPC: 

  • TPC_CAEN was frozen, click “wake me up” - fixed

  • TPC Sector 4 Channel 5 tripped when brought from ped to phy (the channel which was set to 400V) - cleared the trip (Tonko found S11:8 is also “weak”)

  • Timebins problem from QA plots: outer TPC S12:18~20; inner TPC iS19, timebines 39, 40;   - require power-cycle TPC GG, contact David first

  • Noisy FEE in outer  S10 (details see shift log) - Tonko is looking into it

  • 3 random pads of iS11, iS12, iS18 are high and flatlined across the tb, Tonko handled them

• MTD: 

  • gas alarm, switched bottle yesterday at noon; 

  • Another gas alarm (low) ~ midnight, noticed in the shift log, but no action was taken (should call expert immediately). Noticed by the day shift in the morning, found both flow meters 1 and 2 are 0 (Isobutane fraction 0), the day shift called the expert. Alexi is on his way to the lab to investigate it 

• GMT: HV (U3) trip, cleared manually

Schedule & Plans:

• Running cosmic for the rest of the day, will resume bringing up the rest of the detectors on Tuesday

 

Sunday, May 25, 2025

• No major issues yesterday, running cosmic

• L4 is included in the run

Status & Issues:

• TPC: 100% dead after 5~15 mins -  power-cycled all the FEEs, rebooted “all” in the run control -  fixed the problem

• TOF: error message on “too many recoveries”, powercycled LV, also did the CANBUS reset -  fixed the problem

• sTGC: David was working on ramping the HV (Standby)

• Trigger: 

#26144078: Trigger was 100% dead intermittently for about 2/3 of the run, a large number of TOF errors (~100,000), but no corresponding messages on the daq monitoring page. There are also a large number of events in the event builders. - Reboot trigger seems to have fixed the problem

• Light outside the control room - Prashanth is following this up 

 

 

Schedule & Plans:

• Running cosmic for the rest of the day, will resume bringing up the rest of the detectors on Tuesday

Saturday, May 24, 2025

• ~ 11:00, we were informed by MCR that the maintenance team wanted to work on cleaning the magnet cooling (13:00~15:00). 

• ~ 13:30, we received some temperature alarms from the VME crates and TOF, and we realized the MCW was getting warmer, and then realized they were actually working on cleaning the tower of the MCW, not the magnet cooling. 

• ~14:00, we turned off all the VME crates, TOF, MTD LV, and BEMC.

• Prashanth followed up with CAD/F&O to avoid confusion in the future.

• ~ 16:15, STAR magnet back to full

Status & Issues:

Trigger:

• Slightly change the trigger, bound the BCE + bce_dsm2 trigger

• Jeff is working on testing the L4 triggers

TOF:

• started TOF gas circulation mode since yesterday 12:14

• Powercycled LV

TPC:

• TPC anode sector 4, Channel 5 tripped - not cleared after trying to clear the trip; put to a safe voltage 400V. Experts will look at this later

• TPX & ITPC were 100% dead after some time since last night. In the morning, checked the iTPC LED status, and sector 4-1 is yellow and blinking. Masked itpc sector 4, 1st rb - still under test

• A brief PT-2 gas alarm - cleared by itself

• Power cycled FEEs: 

  • S11:2, S17:4; 

  • S16, S17, S18

Schedule & Plans:

• TPC: ready

• TOF: ready

• BEMC: ready, stay off over the weekends

• EEMC: (?)

• FCS: ready

• sTGC: flowing flammable gas

• EPD: ready 

• FST: Experts will try to turn on the chiller and HV trip testing today

• GMT: For pedestal run, include GMT, which is at Standby status. And Sunday Afternoon, we can include GMT with Physics status for Cosmic runs.

• eTOF: won’t be on for cosmic run

• MTD: will not be ready until next week

 

Friday,  May 23, 2025

Taking cosmic since yesterday 17:23. Laser runs every 4 hours, no major issues

Status & Issues:

• Tigger: CRC errors in L1 (pedAsPhys). Power cycled VME-62 for L0/L1 to fix CRC errors. The trigger BCE crate has a DFM problem. -  Eleanor is investigating it

• Shiftleader PC crashed ~23:10, rebooted manually (good now)

• TPC: TPX RDO S18:5 power-recycled.

  • Make sure the pedestal runs for the TPC are taken with the electronics (FEEs) have been turned on for at least 1/2 hours.

• TOF: ready since yesterday afternoon, 

  • slightly low current in tray 65; 

  • LV tripped twice, everything else looks ok

• Light outside the control room: very dark at night

 

 

Schedule & Plans:

• Slid 1 day due to cryo complications with the cooldown, 05/29 -> 05/30 is the start up

STAR Subsystem states for cosmics:

• TPC: ready, have daily ped run & laser run with local clock.

• TOF: ready

• BEMC: BEMC Front End is on and configured. HV will stay off until next week. BEMC FEEs under shift control.

• EEMC: (?)

• FCS: ready

• sTGC: flowing flammable gas, start to raise HV today - David

• EPD: ready - take ped, ivscan

• FST: Experts will try to turn on the chiller and HV trip testing today

• GMT: Nikolai will come today to turn on GMT

• eTOF: won’t be on for cosmic run

• MTD: will not be ready until next week

Trigger detector states (?) ZDC, VPD, BBC needed. (David will fix the VPD GUI problem.)Thursday, May 22, 2025

Status & Issues:

• TPC: 

  • Alexei started circulation mode for the STAR TPC gas system at 12:30. Needs to keep an eye on the gas system (see details introduction from the run log).

  • TPC laser tested and looked OK at 14:00, laser power needs to be tuned.

  • TPC Anode GUI restored by David

  • A few times yellow alarm overnight, cleared by itself

• Magnet:

  • STAR Magnet reached its set point at 11:40.

  • CAS confirmed the completion of magnet work at 16:08, and it is stable.

• Trigger:

  • VME-62 came online by rebooting (Alexei).

  • VME-84 came Green from Yellow (David). Showed how it is done to DOs. Noted that it is on manual. Watch the fan speed.

• Alarm Handler:

  • Switched the computer to run the alarm handler

• sTGC:

  • Flowing gas since 14:00 (David and Prashanth)

• EPD: turned EPD ON.

Schedule & Plans:

Subsystem states for cosmics:

• TPC: ready, have daily ped run & laser run with local clock.

• TOF: Geary is working on it, shift crews can bring up the GUI, and turn on the LVs

• BEMC:

• EEMC:

• FCS: 

• sTGC: flowing flammable gas, start to raising HV tomorrow - David

• EPD: ready - take ped, ivscan

• FST: Experts will try to turn on the chiller and HV trip testing later this week

• GMT: Nikolai will come this Friday to turn on GMT

• eTOF: won’t be on for cosmic run

• MTD

• Tigger detector: BBC, ZDC, VPD

Wednesday, May 21, 2025

Status & Issues:

• TPC: gas  (P10) started ~ 12:50; was checked every 2 hrs, all good so far.

• Magnet: Ramped up ~ 14:00; Tripped ~ 19:00; Again tripped ~ 20:40; stay off over the night.  -  under investigation now 

• VME-76 (BCW): Wayne rebooted manually - will try to fix the remote control (power-cycle) in the next week (David will look into this)

• sTGC interlock alarm -  low pressure, Prashanth and David came and switched the bottles, fixed now.

 

Schedule & Plans:

 

• Please take a look at the expert call list, let me know if you need any changes

• Yu will prepare a new “detector states sheet” and “data taking guide” for shift

• Current schedule: 4K on the 29th, 1st collision on Jun 3rd. We will bring up all the subsystems for Cosmic once the magnet is up. Please let shift crews know if you want to turn on the detectors by yourself or by DOs, and if it is ready for cosmic.

  • TPC:  might be ready this afternoon, Alexei will let the shifters know if it’s ready

  • sTGC: David&Prashanth will start flowing flammable gas today

  • FST: Experts will try to turn on the chiller and HV trip testing later this week

  • GMT: Nikolai will come this Friday to turn on GMT

  • FCS: Oleg will turn it on this afternoon

  • TOF: Geary will let shifters know when the system is ready for cosmic

  • eTOF: won’t be on for cosmic run

STAR daily operation meeting 06/17/2025