The Data Analysis and Scientific Software Group is embedded within Diamond’s science, software and computing groups along with the wider scientific and industrial communities to help maximise the impact and information derived from the challenging experiments carried out at the facility. The trend of experiments towards intuitive, automated, high data volume or computationally intensive analysis has led the group and our collaborators to develop a suite of open source software tools that are also finding uses in the wider scientific community.
DAWN is continuing to be a popular asset for visualisation and data processing, being used heavily on many beamlines along with hundreds of downloads and thousands of YouTube help video views over the last year.
One of our main successes this year has been to make DAWN’s processing tools run automatically at the end of the data collection, or occur live as data is being collected. This work was done in collaboration with the beamlines and data acquisition team. An example of this improved feature is the use of the 2D powder diffraction/scattering project tools being run automatically on data collected at the Small Angle Scattering and Diffraction beamline (I22) and the High Throughput Small Angle X-ray Scattering (SAXS) beamline (B21), which is the subject of an article recently published in J Appl Cryst (Fig. 1). Live processing and visualisation has also been running successfully on the Microfocus Spectroscopy beamline (I18) and is in testing on both the Scanning X-ray Microscopy beamline (I08) and X-ray Pair Distribution Function (XDPF) beamline (I15-1).
Figure 1: Figure 1: SAXS data from a large grid scan across a bone sample processed in DAWN: (a) four intensity vs. azimuthal angle curves from different positions in the sample, each showing different orientation, overlaid on the raw SAXS image from an area of orientated mineral. (b) a map showing the degree of orientation of the scanned area of the sample. Thanks to S. R. Inamdar, Queen Mary University, for supplying the bone data obtained at I22 (proposal SM10311) as part of her collaboration with R. Serra, University of Alabama.
PhD student, Imanol Luengo Muntion and PDRA, Michele Darrow using SuRVoS on the B24 beamline.
Zocalo is a data processing infrastructure for automated single crystal diffraction experiments, while xia2 is a system working to develop automated processes for in situ experiments by carrying out X-ray diffraction data processing. The existing infrastructure for automated data processing has served Diamond well over the past decade but it is no longer able to handle the throughput of experiments typical of Macromolecular Crystallography (MX) beamlines. The Scientific Software team are now replacing the existing infrastructure with a queue based system which will allow much more rapid initiation of processing tasks as well as richer feedback to GDA (Generic Data Acquisition, Diamond’s open source core data acquisition framework).
Unlike most MX beamlines at Diamond, VMXi is optimised for measuring partial data sets from multiple crystals still in crystallisation plates. In some cases the crystals will have preferred orientations, making it harder to get uniform coverage of reciprocal space. However, this is only known after the data have been analysed (Fig. 2) making rapid feedback from analysis to the experiment even more critical. Once good coverage has been achieved the data may be merged, though care must also be taken to ensure that the data being merged are all from the same type of crystal (i.e. are isomorphous). Automated analysis tools are also available (Fig. 3) to cluster the data sets to find outliers, ensuring that the final data set is representative of the correct crystal type. This is paramount when merging data from dozens or hundreds of samples. Many groups at Diamond including DIALS, the xia2 development team in Scientific Software and BLEND have contributed to improving the automated data processing infrastructure.
Silvia da Graca Ramos from Diamond Scientific Computing Group inspecting components of Diamonds archive catalogue which now has over 7 PB of catalogued data.
Savu is a python-based messagepassing interfacing (MPI) framework for the simultaneous processing of multiple N-dimensional large tomography datasets. A steady increase in the popularity of tomographic imaging, due to improvements in data acquisition and computer technology, has led to a broadening of the range of tomographic experiments, and their complexity, across multiple fields. With time-resolved imaging and multi-modal data collection now commonplace at Diamond, the Data Analysis team have developed a new tomography processing tool, named ‘Savu’. The aim of this new tool is to address the ensuing additional software requirements (Fig. 4).
Figure 4: This rendering of data reconstructed using Savu shows a 500 μm diameter aluminum (green) magnesium (blue) alloy pin, with a droplet of sea water (red) on the top of the pin. The study was a time resolved experiment looking at the corrosion on the surface of the pin and the corresponding hydrogen bubbles in the sea water (visible at the top of the droplet).
Harry Powell (left) explaining the analysis software to user Mike Probert (right)
from Newcastle University on the I19 beamline.
Segmentation of biological volumes is a crucial step needed to fully analyse their scientific content. With conventional methods this can take weeks or even months to obtain a useful segmentation. SuRVoS is the result of collaboration between the Imaging Team at Diamond and the Computer Vision Group at the University of Nottingham. The software has accelerated the process of segmenting biological volumes such as those collected on the Cryo- Transmission X-ray Microscope beamline (B24) by using a combination of super-region and machine learning methodologies (Fig. 5).
Figure 5: Overlays of many of the functionalities of SuRVoS Workbench on a cryo soft X-ray dataset of a biological sample collected on the B24 beamline.
We wish to thank all our collaborators, Diamond beamline staff and users for their continued support and input.
Diamond Light Source is the UK's national synchrotron science facility, located at the Harwell Science and Innovation Campus in Oxfordshire.
Copyright © 2017 Diamond Light Source
Diamond Light Source Ltd
Harwell Science & Innovation Campus