EPICS Control System Upgrade for PANDA
Introduction
PANDA is a large-scale, thermal-hydraulics test facility at the Paul Scherrer Institute that is used for investigating containment system behavior and related phenomena for different designs of advanced light water reactors (ALWR), and also for large-scale separate effect tests [1]. Investigations carried out at PANDA have been included in a variety of international projects, most of which fall under the auspices of the EU and The Organization for Economic Co-operation and Development (OECD), and with the support of a large number of organizations worldwide, e.g. regulatory bodies, technical support organizations, national laboratories, electric utilities, industries [2].
The PANDA control system allows all the main operations needed for preconditioning the facility and for performing a test from a control room. The original control system was based on FactoryLink SCADA software (SIEMENS) and while this control system had been operational since 1993, and it was still running, SIEMENS stopped development and support of FactoryLink in May 2007 such that the maintenance of FactoryLink became more and more critical. It also became clear that there was some functionality, e.g. updating configuration for new hardware with the operator interfaces or anyone being able to start data acquisition, that was needed but which the old system didn’t have. So, PSI contracted Cosylab to perform an EPICS integration of PANDA.
PANDA Control and Data Acquisition System Architecture
The architecture of the original PANDA control and data acquisition system is shown in Figure 1. The controls aspect was managed by FactoryLink SCADA software running on the FactoryLink PC. The data acquisition was controlled by the HP 1000 A990. Communication between the HP 1000 A990 and FactoryLink was handled with code written by PSI, in 1993, based on Berkeley sockets. The last change to the control system was in 2012, when the communication code was converted from the original C to C++.
Basically, all sensor signals are converted to a voltage and sent to the data acquisition system. Part of the actuator signals are left unchanged and are sent to the Selectron or Siemens PLCs. Control over the vessels is carried out by Selectron and Siemens PLCs.
Figure 1: PANDA Control System Architecture
The data acquisition system is an integrated system which measures signals and converts them to engineering units. It consists of completely integrated components to enable the user to perform all the significant actions related to data acquisition and data conversion.
The back end of the data acquisition system consisted of an HP 1000 A990 Real Time computer system. This back-end of the data acquisition system was running with software developed by PSI and was controlled by a dedicated computer. The software was written in the programming language Fortran. Starting and running the data acquisition system software produced a “data acquisition system protocol”, which contained all relevant information about the task versions used (e.g. conversion task) and selected subtasks.
The front end of the data acquisition system comprises of a HP 3852A Mainframe plus four HP-3853 extenders. The Mainframe and each extender contain a HP-44704 16-bit high-speed voltmeter and several HP-44713 24-channel multiplexers. There is a total of 30 multiplexers (i.e. 720 measurement channels) spread over the five different levels of the PANDA facility. This part of the system was left unchanged after the upgrade.
The front end of the data acquisition system software takes measurements at a rate of 5 Hz maximum. The back end of the data acquisition system reads all the measurements carried out by the front end. The data is transferred to a “conversion” task at a rate of 0.5 Hz which was the maximum rate for data storage and data visualization. The converted data was then sent to the FactoryLink PC where the main data storage took place continuously. The engineering interface of FactoryLink allowed users to configure and maintain the system.
Figure 2: HP 1000 A990
Control System Upgrade
Siemens stopped development of FactoryLink in May 2007 and so the PANDA control system started to become unreliable. Since there were additional control system and visualization requirements, it was decided to replace FactoryLink.
The new control system uses EPICS (Experimental Physics and Industrial Control System) [3] and runs on Scientific Linux 7. EPICS is a popular Control System framework used worldwide to create distributed soft real-time control systems for scientific instruments such as a particle accelerators, telescopes and other large scientific experiments. We used caQtDM [4] as a Display manager. The caQtDM package uses the standard Qt GUI for designing synoptic displays and offers a collection of special widgets in order to display and manage control system data in various ways. The new archiving system is based on the EPICS Archiver Appliance software [5]. EPICS Archiver Appliance, written in Java, already provides a complete archiving solution designed to scale up to millions of PVs. The environment set up is very similar to that for the latest large research facility at PSI, SwissFEL [9].
An EPICS IOC is a machine running one or more EPICS IOC Applications. Communication between the EPICS IOC and the Selectron PLC is based on a Modbus RTU link layer by using the modbus [6] and asyn [7] EPICS drivers. The interface between the EPICS IOC and Siemens PLCs is based on the Siemens send/receive protocol and was implemented through the s7plc EPICS driver.
Figure 3: HP 3852A Mainframe [6]
An E5810B GPIB/LAN gateway, to replace the back-end of the data acquisition system (HP 1000 A990), was connected to existing hardware (the back-end of the data acquisition system, the HP 3852A) as shown in Figure‑4. The PANDA data acquisition system was integrated into the EPICS control system using a “byte stream” based communication interface. The communication with the HP 3852A is described in a StreamDevice protocol file [8] which contains protocols for HP 3852A functions and variables which affect how the commands in a protocol work.
The control system upgrade made the system maintenance easier.The EPICS application that was written covers the read-backs and set-points of all relevant control points. All additional functionalities such as the Reactor Pressure Vessel heating control, communication logic with the PLCs, etc. were implemented with standard EPICS record types. Adding a new sensor, editing physical units or making general changes to the control system, for example, now just requires editing an appropriate ASCII file. The graphical user interfaces can run on all PANDA workstations. PANDA operators now have the possibility to archive all signals, even intermediate ones (raw binary data, signals before / after correction and conversion, derived values), which were not archived in the old control system. Finally, the maximum rate for data storage and data visualization Data Acquisition channels is no longer limited to 0.5 Hz but can be configured up to 5 Hz.
Figure 4: Updated PANDA Control System. Very little in the overall system (Figure 1) changed.
Conclusion
Upgrading the PANDA Control System software proved to be a wise decision taken at the right time. The new system is more reliable, has functionality to address requirements that the old system was not able to handle, for example there is now one button to start data acquisition instead of the 4 manual steps that were required in the old system and also UIs for new hardware can be easily created as needed. In addition, the performance was improved. What is extremely important is that now all devices can be integrated into the control system in the same way by using standard technologies. Even more convenient is that the robust, scalable, reliable and secure control system is easily manageable by operators.
- For any questions about the PANDA Control System upgrade, or a related topic, feel free to contact the author uros.zezula@cosylab.com
About the authors
Ralf Kapulla holds a Diploma in Theoretical Mechanical Engineering from the University of Karlsruhe (1993, Germany) and a Doctoral degree in Technical Science from the Swiss Federal Institut of Technology (ETHZ, 2000). He has 25 years experiences in optical measurement techniques like Laser-Doppler Anemometry (LDA), Phase-Doppler Anemometry (PDA), Particle Image Velocimetry (PIV), Laser Induced Fluorescence (LIF) Schlieren technique and Shadowgraphy applied to all kinds of turbulent flows. He was involved in large international projects, e.g. ARTIST (2002–2008, aerosol behavior in a ruptured steam generator), SETH-2 (2008–2011, with the aim of resolving key computational issues for the simulation of thermal-hydraulic conditions in reactor containments), ERCOSAM (2011–2013, containment thermal-hydraulics of current and future LWRs for severe accident management) and HYMERES (2012–2016) and HYMERES-2 (2017-ongoing, to improve the understanding of the hydrogen risk phenomena in containment). He serves as a supervisor of PhDs dealing with mixing in un-stratified and stratified flows and teaches “Optical measurement techniques”.
Uroš Zezula joined Cosylab as a member of the PSI on-site team in 2015. He has worked on various EPICS integration projects for the SwissFEL accelerator, where he is still responsible for providing services in the field of control system integration (implementation of support modules, databases, device support and user interfaces). He was also responsible for the control system upgrade of PANDA, a large-scale, thermal-hydraulics test facility. In his free time, he enjoys spending time with his family, and outdoor sports like hiking and cycling.