Data Acquisition Group

Expectations of software have increased dramatically since Diamond started building beamlines. As we continue building our last phase of beamlines our attention is on more advanced automation. Automation software, through abstracting away and speeding up many experimental processes, is the key to increasing Diamond’s users’ productivity. Automation software can increase throughput by automating collections and sample handling and by aiding non-expert users, and in some cases running entirely unattended experiments. Software can help users run experiments that combine multiple techniques and to make sense of ever larger volumes of data. It can also streamline the collection process by integrating it with the processes that occur before and after data collection. Without an advanced level of automation many of the beamlines and end stations we are currently working on would not be worth building.

Jacob Filik, Mark Basham, Keith Ralphs and James Mudd in the I18 Control Cabin working on the mapping project.

Diamond’s Data Acquisition group provides the software used by almost all of our users to drive their experiments. This software, and the people responsible for it, pull together and integrate the wealth of systems required to make every user’s visit successful. It has been a busy year for us and the groups we work with, during which we collected over 2 PB of data. We support 32 beamlines, nine of which were in either construction, commissioning or planning phases. Across these 32 beamlines we support around 44 branches and end stations, 15 of which are in either construction, commissioning or planning phases. Diamond’s completed beamlines continue to be subject to hardware upgrades and enhancements so that they remain at the forefront of scientific research. Half of these beamlines are receiving end station upgrades or enhancements, improvements to detectors, or better integration with downstream processing.

Automation encompasses anything from a small script which performs an often repeated process to systems of software which work together to execute unattended data collections including, experiment planning or oversight, reduction, analysis and reporting. As more control is handed to software we move toward asking some of our beamlines questions rather than telling them what to do. Together with Diamond’s Scientific Software team we are collaborating with researchers to help develop and integrate strategy software capable of planning experiments to iteratively answer these questions. This software will help users make the most of their samples, beamline’s capabilities, analysis software and their time at Diamond. An interesting side effect is that we will be capturing and archiving the intent of an experiment along with the data and results.

In some cases the benefits of quite simple strategies can be dramatic. The High Throughput Small Angle X-ray Scattering (SAXS) beamline (B21), for example, have improved their collection and analysis pipeline to stop collecting data from a sample once a certain signal-to-noise ratio is met.

The Versatile MX in situ beamline (VMXi) and the X-ray Pair Distribution Function beamline (I15-1) have been designed to run entirely unattended data collections from day one and beamlines such as B21 and the Long Duration Experiments branch of I11 (I11-LDE) are moving in this direction too. Users will ship their samples to Diamond and indicate the data collections and analysis they would like carried out. Diamond’s systems, with the help of a combination of staff and robots, will get the samples to the right place, collect data and then trigger analysis pipelines provided by Diamond’s Scientific Software group. This will all be accomplished without the users ever needing to know that this is going on.

Davide Pincini (left), Charles Mita (centre) and Giles Knap (right) on I16.

Most beamlines built recently collect data from multiple detectors and reduce it in different ways to produce multi-modal data. One of the goals of Diamond’s mapping project was to standardise and simplify this process. We are currently commissioning and planning two beamlines, Versatile MX micro (VMXm) and Dual Imaging and Diffraction (DIAD), which will take this much further. VMXm will use an electron microscope to identify small crystals likely to diffract before then collecting data, while DIAD will go so far as to have two independent beams with different optics. The first beam will perform tomography experiments in order to track interesting parts of a sample as it undergoes some process and the second will carry out diffraction experiments. A number of other beamlines are also beginning the work to add tomography during their experiments.

The challenge then for Diamond’s software and computing groups is to fully encapsulate what used to be considered the end goal of an entire beamline into a small part that can be used seamlessly during bigger experiments.

Fajin Yuan story mapping.

Experiments at almost all of Diamond’s beamlines are run using our Generic Data Acquisition (GDA) software. This provides a graphical user interface customised for particular scientific techniques or operations and a server that is configured to work with the hardware, analysis pipelines and databases on each beamline.

Dr Moritz Hoesch, former Principal Beamline Scientist for the Angle-Resolved PhotoEmission Spectroscopy (ARPES) beamline (I05), enjoyed the benfits granted by GDA: ”The GDA interface for ARPES on I05 proves to be of just the right level of complexity that the users don't get lost. However the intrinsic flexibility of GDA's Python scripting allows for the quick implementation of any interesting scan mode or sequence of data acquisitions. The data are saved into a newly defined NeXuS file format. The users find it easy to locate their data in the file and the compression leads to files of moderate size that are transferred easily.”

James Mudd working on code for detector implementation in GDA.

Diamond’s mapping project is providing us with the solid foundations we need to handle the challenges we now face. We have an architecture in which the software and hardware provided by five or six groups at Diamond fits together well. As important, staff from these groups are learning to work together better than ever; something which is vital as we all work together to deliver a large number of complex and very capable beamlines.