How Cosylab and SINAP Collaborated on Producing a Fast Machine Interlock System
What is interlocking?
An interlock in engineering terms is a system feature that makes the states of machine mechanisms or functions mutually dependent in such a way to prevent damage to the machine and harm to its users. A simple interlock, for example, is the electromechanical module in a clothes-washing machine that during the high-speed water-extraction prevents user-access to the spinning drum and stops the drum’s rotation if the door is somehow forced open. In this way, a Machine Interlock System (MIS) blocks an element from changing to a critical state if a problem is detected in another element.
Optimal performance is of even greater importance for the newest and next-generation Big Physics facilities, particularly those containing charged particle accelerators with powerful beams, such as such the High-Luminosity LHC and the European Spallation Source.
Machine availability is a key yardstick for the performance of such particle accelerators, and a highly reliable MIS should minimize system downtime due to possible damage-induced states, while at the same time keeping “false positive” interlocks to a bare minimum.
Our Fast MIS physical unit
The choice of the MIS architecture and its electronics components can have a large influence on the achievable level of dependability of the whole Big Physics facility. There is a kind of yin and yang, a duality of opposing and complementing principles regarding interlock system architecture – a MIS can safeguard availability by preventing damage to accelerator equipment, but on the flip side it can diminish system availability due to internal errors and failures.
Why did we choose our specific design?
As modern accelerator systems are, by their very nature, highly complex, it is quite a challenge for control system architects to determine the best balance between machine availability and the safety of equipment and personnel. Both Cosylab and the Shanghai Institute of Applied Physics (SINAP) were quite aware of this challenge, having dealt with a variety of MIS designs and implementations in the past. It therefore made sense to join forces in creating a reliable software and hardware product that is an out-of-the-box solution for the Machine Protection System (MPS) engineer.
Our MIS unit and its main components
Big Physics machines, such as accelerators, vary widely in size, cost and complexity, which means that in some instances the MIS may be just a small part of the MPS, while in others it may constitute almost the entire protection system. Based on discussions with several accelerator facilities around the world, we determined that the requirements of the greater part of the community could not be met with commercial, off-the-shelf equipment. Therefore, we designed a reliable distributed system that is flexible and fast.
The Design Solution
The fast MIS that we have developed has a proven hardware platform and utilises powerful and radiation-tolerant FPGAs based on non-volatile flash technology. It also enables redundancies for the power supply, hardware components and logic, and is fully configurable from the industry standard and open-source software infrastructure EPICS.
Our design also meets other requirements for a modern MIS, such as fast response times, signal path determinism, IO capability, scalability, and excellent integration with the control system and the timing system. Other functionality includes post-mortem logging and configuration verification.
There was one user requirement that was of special concern to us: guaranteeing a response time (RT) ≤ 5 µs for failures in the crucial parts of the accelerator. We achieved a short and deterministic interlock local RT of less than 400 ns, repeatable at 20 ns, and an interlock global RT of less than 700 ns (with two MIS units interconnected with a 0.5 m optical cable).
The core unit of the MIS has backplane hardware which is largely based on the CompactPCI design, which we adapted for specific functionality of our MIS platform. The latter’s main components are the Input card, the Output card, the Monitor card and the Interlock card. As risks of failure and complexity of risk analysis often increases nonlinearly with increasing system complexity, we isolated the safety critical part, which handles interlock responses, from the safety non-critical part, which operates the configuration, management and monitoring parts of the system. Our MIS design can support various IO standards for Input and Output cards, such as TTL and HFBR IO, analogue-threshold and relay cards.
Safety critical Functionality is delivered by the low-level FPGA logic in the Interlock cards which enable deterministic and sub-microsecond local response times. For dealing with failures of the MIS hardware itself, the system enables redundancy of the interlock logic by duplicating its functionality in two separate Interlock cards while also featuring redundant power supplies.
Multiple MIS crates distribute interlock logic and can be connected in a fully redundant optical network with a ring topology, which is highly scalable. The global interlock propagation time is halved by propagating redundant interlock signals in opposite directions to the redundant Interlock cards. As the interlock logic is distributed, our MIS is smoothly expandable by adding additional MIS crates to the protected machine.
Redundant optical network with a ring topology
Safety non-critical Functionality is provided by the Monitor card which controls all other cards in the system, and performs post-mortem logging, timestamping via integration with the timing system, interfacing with the control system, and more.
Benefits of Our Fast MIS
Currently, our MIS is slated for use in the patient-treatment interlock system of APTRON, the Advanced Proton Therapy Facility in Shanghai, China. To be useful in all of these scenarios, the fast MIS platform designed by Cosylab and manufactured by SINAP is an autonomous and significantly scalable and adaptable system. Our solution presents an industrial-grade high-tech solution which covers all common MIS functionality with the speed, flexibility, reliability, availability, determinism and consistent response which modern Big Physics facilities need for a robust Machine Protection System.
The work was carried out in the framework of the GOSTOP programme, which is partially financed by the Republic of Slovenia – Ministry of Education, Science and Sport, and the European Union – European Regional Development Fund.
