Diamond Annual Review 2023/24

68 D I A M O N D L I G H T S O U R C E A N N U A L R E V I E W 2 0 2 3 / 2 4 Scientific Software, Controls and Computation S cientific Software, Controls and Computation (SSCC) department manages all software, computing and control systems to facilitate and support the science programme of Diamond. The department functions as nine groups: Scientific Computing, Data Analysis, Data Acquisition, Beamline Controls, Accelerator Controls, Electronic Systems, Scientific Information Management Systems, Diamond-II Integrated Software and Cyber Security. The overall structure and function of these areas recognises the importance of, and is optimised to provide, the best possible delivery and support for software, computing, and control systems. During the past year the Diamond-II project was funded. One of the five pillars of the Diamond-II project, Pillar 1.3, will deliver and uplift the underlying capabilities of software and computing. It was recognised early on in the planning of Diamond-II that there needed to be significant developments in underlying software and computing capabilities to prepare for the substantial increase in data rates that will come with Diamond-II. This has been addressed through the design of a new software architecture for photon beamlines and the definition of an extensive core software and computing programme for Diamond-II to deliver new enabling capabilities. In addition, SSCC will deliver new software, control systems and computing as part of the machine upgrade and beamline developments for Diamond- II. The following section, Update on Diamond-II software and computing, provides an overview of developments in Pillar 1.3 and recognises key deliverables over the last year. As experiments conducted at Diamond produce increasingly large and complex data sets, it becomes more difficult for users to transport their data back to their home institute for processing. To address this Diamond is increasingly providing users with data processing services to enable information to be extracted from their data. To facilitate this Diamond develops and maintains a suite of data analysis applications to support the photon science and electron microscope programmes. These same tools can also provide near real-time feedback to the user as their experiment progresses. However, some tools are computationally demanding and there is a programme to accelerate the processing using faster computing technologies, such as GPUs, to provide near real-time feedback during experiments. While the end objective of a new architecture is to enable new science capabilities for Diamond-II, the development process includes a series of intermediate deployments on Diamond’s existing beamlines. These will not only deliver new capabilities early for Diamond, but will provide for a mechanism to debug the new software and ensure it is fit for purpose before major deployment as part of Diamond-II. The year saw the first user experiment conducted with the new software architecture. Diamond has systematically maintained an archive of all data collected since operations began. To date that data has been “owned” by the scientist that have conducted the experiment. It is recognised that there are added opportunities from the experiments if the data is made open and so accessible to others. To achieve this Diamond will move to making its data repository open after an embargo period whereby not only the scientists who conducted the experiment have access to the data. For this to work the data will need to be FAIR (findable, accessible, interoperable and reusable) and additional metadata will need to be recorded about the experiment. While this has been very successfully delivered for some science techniques it was not possible across all techniques due to structural limitations in the available information management system. To address this, a new Universal Laboratory Information Management Systems is being developed to provide sufficient flexibility to capture metadata across all physical science beamlines. This is increasingly important as such metadata is a key part of using machine learning techniques to mine data for new information. Diamond produced more than 10PB (a PB of data is equivalent to 213,000 DVDs) of data last year from photon beamlines and electron microscopes. To support the capture and processing of the data, Diamond operates an extensive computing online resource. This is designed to manage the high throughput and processing of data, as experiments are conducted. It is recognised that provisioning all computing services within the Diamond infrastructure is not a sustainable model, so work is ongoing to decouple applications and services from the existing computing infrastructure by using containerisation technologies. The containers can be deployed on both private and public cloud infrastructure. Section2a: The Diamond-II Core Software, Controls and Computing project as a bound waterfall project exploiting agile loops to deliver incremental benefits to Diamond. Figure 2b: Responsibilities in the new integrated beamline software architecture. X-ray technologies