SNS Project Status

ICS – Software Engineering Group

1

SNS Project Status

Ernest L. Williams Jr.

27 April 2005

EPICS Collaboration Meeting @ SLAC


2

Outline

Installation Status» Global Systems

– Network– Timing– MPS– CCR

» Accelerator Systems» Cryogenic Controls » PPS» Target Controls

Operational Improvements Issues Summary


3

The Spallation Neutron Source

The SNS construction project will conclude in 2006

At 1.4 MW it will be ~8x ISIS, the world’s leading pulsed spallation source

An upgrade to 3MW has been approved for CD0 planning.


4

Communication Infrastructure is nearly complete

CD4 Fiber Backbone is complete

Tubes installed for future addition of “blown” fiber

“Global” cables nearing completion

» Ethernet» Timing» MPS

All accelerator cable SROs entered

CISCO Switch firmware bug found (and fixed)

Fiber Backbone

Cable Infrastructure


5

Timing System Installation is Keeping Up

All IOCs (including NADS)

Multiple Fixed Rep Rates for RF Systems

CLO Test Timing Network» Infrastructure for Diagnostics,

LLRF, XFD, and Power supply groups

Line Synch monitor» Delivered from BNL» Setting up test network for XFD

Chopper testing

“General Time” added to EPICS» Provides best time from multiple

sources» Assures monotonicity» Accepted into “EPICS Core”

SNS Event Link Monitor» Event statistics» Actual Events vs.

Scheduled Events


6

Number of MPS Inputs has nearly Doubled(170 → 313)

Linac Only

MPS Inputs MEBT_BS CCL_BS L_Dmp MEBT_BS CCL_BS L_Dmp

IBCM 1

BLM / ND 66 129 66 129

Beam Stop / Dump 2 4 6 2 4 6Intercepting diagnostics 11 27 6 11 27 6

Power Supplies 11 15 52 11 61 90

Vacuum 3 8 10 3 8 10

RF Systems 11 26 104 11 26 104

MPS inter connects 9 4 6 9 4 6DBCM 6

Totals 47 150 313 48 202 351

Commissioning - no correctors After CD4 - correctors added

All linac MPS hardware installed – includes 27 new chassis

Copper phased out, using fiber everywhere

After CD4, corrector PS status will be added

170


7

Operations have moved to the CCR(Central Control Room in CLO)

Move took place in February as scheduled

Two console arcs are complete (ahead of schedule)

PPS and ODH systems moved (That was the worrisome part)

Software Development Area in use All work planned for FY05 is

complete Servers are moved to “CER”

(Central Equipment Room) TPS also now in the CCR Prototype Console moved to

Target Control Room Upgraded all multi-head OPIs

» Faster processor» More RAM» Faster graphics adapter (Matrox

G450) X-windows and Java Apps have a

much better performance

PPS Equipment


8

Linac Controls Installation isKeeping Up with the Project Schedule

We have kept up (barely!!) with RF testing, RF Conditioning, PS testing, cryomodule cool-down, etc.

Medium-Beta SCL controls mostly complete» Cryomodule MB1 controls testing remains» Some “warm section” vacuum controls testing remains» Some MPS hardware installation remains

High-Beta SCL controls approaching completion» RF, Cryo, & Vacuum controls checkout are well along» Magnet power supply interface checkout has been

integrated with overall power supply checkout

HEBT controls installation in progress Automated procedures have helped keep schedule


9

Linac LLRF Installation is Keeping Up(This is the biggest task in terms of numbers and complexity)

Number of installed LLRF IOCs: 46 (up from 12) for 80 systems (out of a total of 96 systems)

Configuration automated via scripts, avoiding inconsistencies between SCL systems

From last ASAC: "Add Adaptive feed forward to firmware if required"

» Was required for higher beam current» Not foreseen in current FCM, so successfully added to IOC software

– recently refined to eliminate parameters that were hard to tune

Automation of RF ramp-up, warmup, loop closure» Allowed hands-off recovery from trips on the NC Linac» Needs adaptation for SCL

Waveform updates synchronized to timing system» Allows physics apps to compare data from several systems for the

same beam pulse Refining overview screens, giving better notice of trip reasons,

active overrides, etc.


10

Cryogenic Control System nearly complete!!

All CHL Controls equipment installed and

All IOC and PLC programming complete» CHL Gas Management, Purifier and

Warm Compressor» CHL 2.1K and 4.5K Cold Boxes» Medium and High Beta Cryomodules» HEBT Service Building IOC and PLC

2K Cold Box Turn-on was automated Cryomodules will complete in April

Cool DownFirst Liquid Full RF Power

Warm Up

tested


11

The Cryogenic Control System layout

11 IOCs

17 PLCs

2 Servers

4 OPIs


12

Ring Controls is ahead of schedule:BNL effort completed in March

Handover Plan - Complete and in use (cooling water and ramping software scope moved to ORNL)

FSDs - Complete, in use and being updated as needed to keep them as-built

Design - Essentially complete. All drawings and specs needed are in use for fabrication and installation and being updated as needed to keep them as-built

Hardware Fabrication - WBS 1.9 work complete except for installing 1 PLC for cooling water skid. WBS 1.5 work complete except for 2 motion control racks, 2 PLCs for RF, and Vacuum racks

Procurement - All hardware procured except 1 PLC for water skid and some miscellaneous parts

Cabling - Schedule for Ring and RTBT cables depends on budget allocations» HEBT Communications cables pulled, being terminated

» HEBT Process control cabling in process

Software - Vac, BLM, Mag PSC/PSI, HPRF, Beam Dump TCs ready for handover. All software ready for field testing by Jun 05

Test Plans - Drafted for all systems except RF - Will be updated as needed during installation


13

PPS System Installation is keeping up(in spite of many configuration changes)

75% of the PPS controlled devices in the Linac have been connected

The temporary Linac shielding has been removed and the Linac has been returned to one segment for phase 1.3

PPS remote I/O racks for the HEBT are installed

Conduit & cable for PPS devices installed in HEBT tunnel

Installation of conduit & cable for PPS devices in Ring tunnel is 100% complete

The PPS control room racks have been relocated to the CLO CCR as a part of phase 1.3 (SCL commissioning)


14

The Target & Instrument PPS are alsothe responsibility of the Controls Team

Cable & conduit design for target building complete, installation underway using AIMSI crafts (beamline 2 and transfer service bay) and fixed price subcontractor (high bay)

Target PPS equipment design is 85% complete

Target PPS equipment fabrication is 70% complete

Equipment installation ramping up as electricians become available

Instrument PPS design for beamline 2 underway. Prototype user panel being fabricated


15

223 IOCs currently on Accelerator Network:» 109 VME / VXI --- vxWorks-based » 2 Linux-based “Soft IOC” Servers

– Running 24 Soft IOCs (18 + 6)

» 90 Intel-Pentium --- Win32-based IOCs (the “NADS”)– Wirescanners, Faraday Cups, BPMs, BCMs

Upgraded control system to EPICS R3.14.7» Improved reliability in channel access.

– Results in fewer white screens in the control room.

Upgraded OS for VME/VXI-based IOCs» vxWorks 5.5.1 (SNS CVS Release SNS07)

– Better network driver and network stack configuration– Auto-retry on boot-up if boot server is unavailable.– Better memory management

The number of IOCs has risen dramatically


16

Power Supply Group is Routinely UsingAutomated Testing and Calibration

Method uses EPICS and so assures “end-to-end” testing including control system

Resulted in large schedule reduction – one test instead of two

PS Calibration, current/field mapping and cycling profiles come from Oracle RDB

9 SCL IOCs installed, 8 operational

1 HEBT IOC installed and operational


17

Turn-on and Turn-off of HPRF Systemshas been automated

State Diagram monitors the processfor each HVCM

Overall summary is displayed


18

The New Archive Engine is stable and amany-featured Java Archive Viewer is now in use

Archives now well organized with central configuration file and scripts for» generation of directory structure and skeleton files» start/stop/restart/monitoring of sampling engines» update of status web page» email notification of engine status and disk usage

Currently 11 distinct archives each running one or more sampling engines (total: 25).

» (Conventional Facilities, Magnets, RF, Cryo, ...) Data rate kept to ~2GB/day with occasional peaks at 4GB/day. Limiting performance factor is currently neither CPU nor network but disk I/O.

New Java Viewer is stable and many features are being exploited » Uses XAL Framework» Web interface available» Plots actual data in various formats» Offers absolute and relative request times» Performs computations, exports data to spreadsheets» Investigating new uses via command line and web» Scroll, zoom, legends, titles, multiple plot styles, arithmetic and boolean operations


19

The new Archive Viewer (Chevtsov)has many useful features

•Relative start/end times "-5 d(ays)" →

"now"

•Multiple value axes

•Tool-tip info for individual data points

•XAL Framework

Support for calculations based on archived data

Export to spreadsheets, progress bar

Green squares in plot launch tool-tips for meta data, in this case daily restart of sampling engine.


20

Database Collaboration with APS has grown(SNS, APS, SLAC, TRIUMF, BESSY, FNAL, Diamond, PSI)

• IRMIS: Integrated Relational Model for Installed Systems (v1.0 release in April)

• Agreed “Core Schema” describes control system with three hierarchies:

• Signal• Housing• Power

• Uses the SNS Physics XAL “Framework”• Pelaia et al

• RDB synchronized using “crawlers”

• 106 SNS IOCs successfully “crawled”• EPICS versions 3.13.9, 3.14.6, 3.14.7• 228,270 PVs found

– SCL 112141– DTL 33607 – CCL 27443 – MEBT 10918 – HEBT 8074

• “Crawlers” in development for all EPICS clients, IOCs, edm, Archiver, Alarms, etc.


21

Issues

Increased installation pace.

RF Processing and installation checkout means Control Systems Group must provide operational support.

Hardware/Software Configuration Management is highly correlated with reliability/availability. Work is in progress using a RDB, CVS, and good release control scheme.


22

Issues: How to manage increasing number of IOCS?

223 IOCs currently on Accelerator Network:» 109 VME / VXI --- vxWorks-based » 2 Linux-based “Soft IOC” Servers

– Running 24 Soft IOCs (18 + 6)

» 90 Intel-Pentium --- Win32-based IOCs (the “NADS”)– Wirescanners, Faraday Cups, BPMs, BCMs

Upgraded control system to EPICS R3.14.7» Improved reliability in channel access.

– Results in fewer white screens in the control room.

Upgraded OS for VME/VXI-based IOCs» vxWorks 5.5.1 (SNS CVS Release SNS07)

– Better network driver and network stack configuration– Auto-retry on boot-up if boot server is unavailable.– Better memory management


23



24



25

Summary

Installation is keeping pace» Required use of more automated configuration and

testing, database use» PPS keeps pace notwithstanding changes» Cryo controls nearing completion» Will be ready for SCL commissioning» Ready to pick up on Ring controls

Operational Improvements as well» Automated HPRF turn-on» Automated 2K Cold Box turn-on» Improved Archive viewing tools

SNS Project Status

Documents

Transcript of SNS Project Status