Data Acquisition

Rob Walton, Current Acting Group Leader - Data Acquisition Software

Diamond’s data acquisition staff provide the software used by most of Diamond’s users to drive their experiments. It has been a busy year for data acquisition, during which nearly two petabytes of data have been collected. The group supported 31 beamlines, eight of which are in either construction or commissioning phases. Across these 31 beamlines, around 43 branches and end stations are supported, 15 of which are in either construction or commissioning phases. Many of the completed beamlines were also subject to continual upgrades and enhancements so that they remain in the forefront of scientific research.

The challenges for data acquisition include scan optimisation, support for detectors of ever increasing performance, and increasingly sophisticated experimental processes. Diamond’s users frequently need to apply more than one scientific technique to a sample (Fig. 1). The data acquisition staff have been striving to increase the throughput of Diamond’s beamlines and to this end they are working to make the beamlines more automated, to integrate more live and post processing, to track samples and, in some cases, to suggest experiment strategies. To meet these challenges they are working ever more closely with Diamond's Scientific Software, Scientific Computing and Controls groups.

Figure 1: Transmission and fluorescence maps of a particle from Arctic volcano sediments taken on Diamond’s newly commissioned beamline I08. Courtesy L. Benning, U. Leeds and B. Kaulich, T. Araki, M. Abyane.

Data acquisition at Diamond is performed on almost all beamlines using the Generic Data Acquisition (GDA) software that provides a graphical user interface customised for particular scientific techniques (Fig. 2) and a server that is configured to work with the hardware on each beamline. Moritz Hoesch, Principal Beamline Scientist for I05, comments:

Figure 2: A newly built GUI for configuring and visualising time resolved EXAFS experiments on Diamond’s Energy Dispersive EXAFS brachline (I20-EDE).

Figure 3: Tomography reconstruction performed using Diamond’s tomography pipeline running on a cluster of GPUs. The GDA software triggers a reconstruction by handing the pipeline a NeXus file that conforms to an application definition newly accepted by the NeXus community.

Figure 4: 3D dynamic imaging of injected supercritical CO2 in natural rock. Taken on I13 using fast flyscan tomography. Courtesy Matthew Andrew, Department of Earth Sciences and Engineering, Imperial College, London.

“The GDA interface for ARPES on beamline I05 proves to be of just the right level of complexity that the users don’t get lost. However the intrinsic flexibility of GDA’s Python scripting allows for the quick implementation of any interesting scan mode or sequence of data acquisitions. The data are saved into a newly defined NeXuS file format. The users find it easy to locate their data in the file and the compression leads to files of moderate size that are transferred easily.”

The data acquisition staff have been working to ensure that different beamlines in related scientific areas such as spectroscopy or mapping have as consistent and coherent interfaces as possible. This simplifies the user experience and reduces development and support costs.

Diamond is adopting the NeXus file format as its preferred way of recording data and the data acquisition staff have been continuing its roll out. NeXus files are self-describing binary files with a tree structure agreed by the NeXus community. These files include everything needed to understand the data collection they describe including, for example: the name and structure of the sample, the geometry of the beamline including the sample stages and the orientation and description of detectors, the exposure time, the images and other information gathered during the beamline’s calibration, a description of the scan performed and, of course, the data itself. Fundamentally this level of completeness will preserve the significance of data collected at Diamond and will mean that Diamond data can be easily interpreted by the researchers who took it and by others. More immediately it provides a tidy interface through which to trigger Diamond’s Scientific Software group’s data reduction pipelines or to analyse and visualise the data after an experiment using Diamond’s DAWN software or with third party tools.

Figure 5: A DAWN image viewer component shows live images from a Pilatus detector during a scan. The same DAWN tools used for offline analysis and visualisation can be used on these live images: here a horizontal line profile is shown. Two rectangular regions have been configured. The sum from each is shown on the scan plot view on the left and saved along with the images.

This year the NeXus community approved a new tomography application definition which describes what data must be included in the file, where to put it and what it must be called in order to be tagged as including tomography data. This allows the group to trigger an automated tomography reconstruction without having to provide any context in addition to the data files (Fig. 3 and Fig. 4). To ensure the files produced meet the NeXus standard the group has started a programme to test them automatically at the start of each run.

Figure 6: The ARPES perspective used for alignment on I05. Not shown here, this user interface software also includes a perspective for configuring and running basic collections and one for scripting more complex ones.

The GDA software’s user interfaces are built using the Eclipse RCP framework. This user interface toolkit is also used by Diamond’s Scientific Software group and will be used for the next generation of beamline synoptics developed by Diamond’s Controls group. Using this common framework allows the data acquisition staff to integrate components from across other groups to create unified user interfaces. They are yet to see where the closer ties with Diamond’s controls software will take them but it is clear that the ability to include components written by Diamond’s Scientific Software group is very powerful. For example, many beamlines include a component for controlling detectors and viewing their data; at the core of this are the same tools used in Diamond’s DAWN software for investigating images offline (Fig. 5). Data acquisition staff also build far more specific tools from these components such as in I05’s ARPES user interface (Fig. 6). Sometimes a specific tool developed for DAWN just slots right into a beamline’s GDA user interface. An example is the ISPyB data collection perspective in Diamond’s MX software, which shows a history of data collections and the results obtained during post processing and also provides ways to request that data be reprocessed on Diamond’s own cluster (Fig. 7).

Figure 7: The ISPyB data collection perspective from Diamond’s standard MX GDA user interface. This is dropped straight in from the DAWN software and shows a history of data collections and the results obtained during post processing. It can also be used for example to reprocess data using Diamond’s MX pipeline running on clusters supported by Diamond’s Scientific Computing group.