Diffraction software gets global refinement

Users given enhanced processing for multiple data sets, with optimised diffraction geometry

All users of X-ray diffraction beamlines at Diamond Light Source are now benefiting from a behind-the-scenes boost to their diffraction processing capabilities. The new Diffraction Integration for Advanced Light Sources (DIALS) software provides a global layer of optimisation during data processing and has the power to model multiple experiments simultaneously.

The freely-available, open-source software was released in early 2016 and looks set to help all crystallographers at Diamond and beyond, particularly those running complex experiments requiring multicrystal merging. The team behind the upgraded optimisation software have been running tests at Diamond for a year prior to its launch to check that it compares favourably with rival packages. The details of a core component of DIALS have been outlined in a recent publication in Acta Crystallographica Section D: Structural Biology.

Modular open-source software
Previous versions of diffraction processing software were rarely open-source and not extensible. As new X-ray detectors emerged, it became more obvious the software was out-dated and not keeping pace with hardware innovations.
A team of scientists led by Dr Gwyndaf Evans, Principal Beamline Scientist for VMXm (Diamond’s micro/nanofocus MX beamline, currently under construction), set out to write an entirely new X-ray diffraction processing package from scratch. Now four years later, their DIALS software has been welcomed by macromolecular crystallographers worldwide.
According to Dr Evans, the main drivers for DIALS were: "to be responsive to new developments in the field and to capture the breadth of emerging data collection methodologies. The aim is also to train a younger generation of crystallographic methods developers through workshops and access to open-source code."
Gwyndaf Evans (left) and David Waterman.
Dr David Waterman, Scientific Programmer at the STFC Rutherford Appleton Laboratory and lead scientist for a core component of the software known as ‘DIALS refinement’, explained the key motivation for writing the new software: “We wanted to make the software modern, modular, and extensible, and we also didn’t want to make it closed or commercial; it is open and freely available to everyone.”

International collaboration bears fruit

The software was the result of a fruitful collaboration between CCP4 (Collaborative Computational Project Number 4), The Lawrence Berkeley National Laboratory, and Diamond. The work was funded in the UK by the EU BioStruct-X grant, STFC, and Diamond, and in the USA by the National Institutes of Health (NIH) and the National Institute of General Medical Sciences (NIGMS). The collaboration allowed the teams to work independently on different aspects of the software package, with the UK arm working on synchrotron applications, and the US arm focusing on X-ray free electron laser (XFEL) applications.

DIALS refinement, programmed by Dr Waterman, models diffraction experiments by representing the positions of the beam, crystal, and detector in space. Essentially, it creates an extremely accurate physical model of what happens during imaging so that the user can put more trust in the downstream processing steps. The most obvious difference between DIALS and similar software is the global modelling. As Dr Waterman explained: “older packages process data as a series of individual jobs to map the experiment as it proceeds; whereas DIALS fits all the data to one smoothly evolving model that reflects any changes during the experiment. That means the model has no discontinuities in it and you can do the integration very rapidly as you already have your model in place.”

Multicrystal merging
Users with low-quality crystals are some of the main beneficiaries of DIALS refinement. Instead of analysing just a single crystal that might have taken months to prepare in the lab, multiple small crystals of comparatively poorer quality can be assessed simultaneously with the aid of DIALS before merging. “Advanced users who are trying to do this type of multicrystal merging can definitely take advantage of the global multi-experiment refinement that the program offers as a preparatory step before subsequent processing. You can more accurately determine the differences between the samples by treating them all together in one job. In this way, you remove systematic biases that may obscure the real differences between the crystals. By using DIALS you don’t corrupt the clusters, so it is a very good preparatory step prior to analysis.” assures Dr Waterman.

Multicrystal merging, or serial crystallography, is a hot topic for crystallographers. The XFEL offers an extreme form of serial crystallography where a liquid jet of crystals is hit by a pulsed electron laser. The US counterparts of the DIALS team made sure that the software package could process data from an XFEL, so they have future-proofed the software for this emerging technology.

Geometry correction
Another huge advantage of DIALS is its capacity to correct the geometry of complex detectors. Dr Waterman stated: “There is a barrel shaped detector at beamline I23, and most of the other packages fail straight away because they never expected a detector to be barrel shaped, but our software can handle it. We are also able to correct for the tiny differences in the geometry and position of different panels of detectors.”  

As well as continuing to improve the functionality of DIALS, the team hope to improve the user interface of the software, to make it more user-friendly. Moreover, they want to increase the number of users, so that more crystallographers can benefit from this next-generation processing package.
To find out more about DIALS and to access downloads, documentation, tutorials, and related journal articles, go to dials.diamond.ac.uk.

Related publication: (Open Access)

Waterman D et al. Diffraction-geometry refinement in the DIALS framework. Acta Cryst 72, 558–575 (2016). DOI:10.1107/S2059798316002187


A schematic of the diffraction geometry showing the beam, crystal, goniometer and detector. Some of the parameters optimised by DIALS refinement are indicated by directions and labelled. These parameters are refined globally for a complete data set, or indeed multiple data sets. This ensures the procedure is robust and reduces the correlation of effects between the parameters. For multi-experiment refinement this is critical in determining real differences between sets of data that are candidates for merging together.


For a tetragonal thaumatin data set, smoothly-varying unit-cell parameters produced by DIALS refinement are shown in blue, with their values refined within blocks of 5° by XDS shown in red. The globally-determined smooth curves from DIALS closely match the trends observed in the locally-determined values from XDS. Either model can be used for accurate spot prediction in integration, however because the DIALS model is determined in advance of any integration, this can proceed using parallel processing, with no loss of information.


A representation of multiple experiments that share some components. A single experiment is one path from top to bottom (such as the one highlighted in red) passing through a beam (B), a goniometer (G), a crystal (C), a scan (S), and to the detector (D). Waterman et al. 2016. Reproduced with permission of the International Union of Crystallography.