Annual Review 2024-2025

D I A M O N D L I G H T S O U R C E L I M I T E D 41 S C I E N T I F I C S O F T WA R E , C O N T R O L S A N D C OM P U TAT I O N • high performance sample stages; • detector readout, data compression, and reduction; • modernisation of data acquisition software framework; • science-specific data analysis software developments; • data archiving; • post-visit data analysis services; • user administration and information management. This project, one of the five pillars of the Diamond-II programme, is described in terms of six work streams: Hardware Infrastructure, Software Infrastructure, Data to Information, Real-time Data, Experiment Management, and Information Management. These are divided into over thirty detailed work packages. A phased delivery plan has been developed to prepare for the improved brightness of the Diamond-II machine and enable new flagship capabilities. This project will provide incremental benefits to Diamond, reducing technical debt, addressing obsolescence, and deploying new capabilities with greater flexibility and extensibility. Early successes include the rollout of a new web-based engineering user interface to the accelerator control system and the development of a new experiment orchestration platform. For the latter initial core services are being developed and deployed onto a simulated beamline. In addition a range of “opportunity projects” are being used to thoroughly test new capabilities examples being live data streaming for ptychography, and live analysis of X-ray diffraction data for experiment feedback, proving the viability of GPUs to reduce image processing times. Preparing for new opportunities coming from Artificial Intelligence Artificial Intelligence and Machine Learning (AI & ML) techniques are now foundational enabling technologies in data science. Recent advances in areas such as Large Language Models (LLMs) are also having a science impact and have the potential to unlock new capabilities for the science programme, operation of the accelerators, and business functionality. Looking forward to Diamond-II three key AI & ML themes have been identified as driving improvements in the science programme. These are Making Experiments Easier; Making Experiments Better; and Understanding Experiments Better. Each of these themes are aimed at opening new capabilities to Diamond’s users and so enabling new science to be conducted as well as enhancing our existing science programme. We hope that the early adoption and integration of these technologies not only helps with the ‘data deluge’ but also assist in making experiments conducted at Diamond more dynamic and easier for the facility’s users. An example of improving operation of the accelerators is our machine stability project. This is a multi-year collaboration with STFC’s SciML Group, that has the potential to benefits to all the beamlines across Diamond through improved electron beam characteristics. In addition to science-focused AI and ML initiatives, the application of Generative AI tool using Retrieval Augmented Generation (RAG) is being tested across Diamond to support business functions. This enables secure access not only to business data within LLMs, but also to Diamond’s corporate information. Delivering challenging IT projects IAM as an example A recent IT transformation project involved replacing Diamond’s Identity and Access Management system (IAM) from a historical database, the Corporate Data Repository (CDR), with a commercial cloud service. Successfully delivering this project was a significant milestone for our team, given that the project faced challenges such as complex scope definition, tight deadlines, and managing a complex rollout. Risks about readiness were mitigated by extensive testing, and so the final transition was seamless. Careful planning around key resources was essential to the success of the project. Diamond’s core team collaborated closely with development partners whose expertise was crucial for aligning the