Edmund Blomley

The commissioning and operation of future large-scale scientific experiments will challenge current tuning and control methods. Reinforcement learning (RL) algorithms are a promising solution due to their ability to dynamically adapt to changing environments and consider delayed consequences. In many real-world applications, RL policies must produce actions in real time, often within microseconds to milliseconds, imposing significant constraints on system latency and computational overhead that conventional machine learning (ML) libraries are not designed to handle. To control phenomena in real time at these timescales, RL needs to be deployed on-the-edge, namely on dedicated hardware located near the system it controls, without relying on a host CPU or cloud-based inference. In this work we present the design and deployment of an experience accumulator system in a particle accelerator. In this system, deep-RL algorithms run using hardware acceleration and act within a few microseconds, enabling the use of RL for control of phenomena like beam instabilities. The training uses the collected data offline to reduce the number of operations carried out on the acceleration hardware. The proposed architecture was tested in real experimental conditions at the Karlsruhe research accelerator (KARA), a synchrotron light source, where the system was used to control artificially induced horizontal betatron oscillations in real-time, with a control loop period of just 2.7 µs. The results showed a performance comparable to the commercial feedback system available at the accelerator, demonstrating the viability and potential of this approach. Due to the self-learning and reconfiguration capability of this implementation, a seamless application to other control problems is possible. Applications range from particle accelerators to large-scale research and industrial facilities.

In electron storage rings, it is possible to increase the bunch length by applying phase modulation to the radio-frequency accelerating field by choosing appropriate parameters for the modulation. Such a bunch lengthening effect improves beam parameters such as beam lifetime, which can help us achieve better beam stability. It is well known that the modulation frequency around the double synchrotron frequency is effective in lengthening the bunch. The dependence of bunch lengthening on modulation frequency, the so-called detuning property, has a peak around the double synchrotron frequency with a frequency width that depends on the modulation amplitude. Nonlinear effects due to phase modulation lead to a peak frequency shift in a negative direction from the double synchrotron frequency, accompanied by an asymmetric peak shape. Beam current also affects the properties of the detuning condition of such bunch lengthening. We investigated the detuning property using a theoretical model and systematic measurements at the electron storage ring Karlsruhe Research Accelerator, and we verified the qualitative agreement between the experiments and the theoretical model. The macroparticle simulations have confirmed the agreement.

In electron storage rings, it is possible to increase the electron bunch length by applying a phase modulation on the radio frequency accelerating field by choosing appropriate parameters for the modulation. Such a bunch lengthening effect improves beam parameters such as the beam lifetime, which can help us to get better beam stability. The dependence of the bunch lengthening on the modulation frequency, the so-called detuning property, tends to have a peak with asymmetric slopes around it. The modulation amplitude and the beam current also affect the properties of the detuning condition of such bunch lengthening. We have investigated the detuning property with systematic measurements at the electron storage ring KARA. The experimental results agree with the theoretical model and the simulation results.

Coherent synchrotron radiation (CSR) is generated when the electron bunch length is in the order of the magnitude of the wavelength of the emitted radiation. The self-interaction of short electron bunches with their own electromagnetic fields changes the longitudinal beam dynamics significantly. Above a certain current threshold the micro-bunching instability develops, characterized by the appearance of distinguishable substructures in the longitudinal phase space of the bunch. To stabilize the CSR emission, a real-time feedback control loop based on Reinforcement Learning (RL) is proposed. Informed by the available THz diagnostics, the feedback is designed to act on the RF system of the storage ring in order to mitigate the micro-bunching dynamics. To satisfy low-latency requirements given by the longitudinal beam dynamics, the RL controller has been implemented on hardware (FPGA). In this paper, a real-time feedback loop architecture and its performance is presented and compared with a software implementation using Keras-RL on CPU/GPU. The results obtained with the CSR simulation Inovesa demonstrate that the functionality of both platforms is equivalent. The training performance of the hardware implementation is similar to software solution, while it outperforms the Keras-RL implementation by an order of magnitude. The presented RL hardware controller is considered as an essential platform for the development of intelligent CSR control systems.