Opportunities at Diamond

Learn more about career and student opportunities at Diamond:

Subscribe to our Corporate and Science Update mailing list

Keep up to date with the latest research and developments from Diamond. Sign up for news on our scientific output, facility updates and plans for the future.

Subscribe here

Scientific Software, Controls and Computation

Mark Heron, Head of Scientific Software Controls and Computation

The past year brought a major organisational change within Diamond, with all Software and Computing groups brought together under one new management structure, to create the department for Scientific Software, Controls and Computation. This new department is structured as seven groups: Scientific Computing, Data Analysis, Data Acquisition, Beamline Controls, Accelerator Controls, Web Applications and Electronic Systems. This change recognises the importance of software, computing and control systems to the scientific output from Diamond; the up-and-coming challenges Diamond faces from Big Data as a consequence of the new detectors; and the need to develop the organisation to fully exploit the scientific opportunities of Big Data.

: Scientific Software, Controls and Computation (SSCC) Management Team, from left to right: Joe Handford, Simon Lay, Austen Rose, Mark Heron, Alun Ashton, Andrew Richards, and Ulrik Pedersen.

The new structure further positions Diamond well for the developments needed for software and computing in preparation for Diamond-II, which will produce data rates and data volumes far in excess of Diamond. Some initial considerations of the detectors in the Diamond-II era indicate raw data rates in Petabytes (PB) per hour as being credible. These data rates will make acquisition and analysis very challenging and require new solutions such as Machine Learning to be able to effectively analyse the data. At these data rates, the growth of data volumes will substantially exceed the increases in computing due to technological progress (Moore’s law giving a doubling on computing power every two years), and so will require developments in the storage and compute resource available on-site and off-site.

To ensure that these challenges are met, a medium term strategy for Scientific Software Controls and Computation is now being developed. Following an external review in the summer of 2019, it will be presented to the Science Advisory Committee in the autumn.

The following are examples of some of the exciting developments across the broad range of science software and computing which are taking place within the Scientific Software, Controls and Computation department.

Implementing the next generation of High Performance Computing (HPC) at Diamond
Odin High Performance Detector readout
PandABox: Flexible triggering for continuous scanning
Adding mapping to generic data acquisition application
Machine learning to accelerate our understanding of viruses

: The new Hamilton cluster provided by Lenovo for data acquisition and data processing.

The increasing demands for data acquisition and subsequent data processing are driving Diamond to not only invest in new, additional, computational capacity on-premise, but to also explore the use of the cloud.

On 1^st March 2019, the newly named Hamilton cluster, after the NASA scientist Margaret Hamilton, entered its production life and is now collecting data for the first time. A year long process of procurement and installation brings into service a new data centric HPC computing platform for the acquisition and analysis of scientific data. Doubling existing storage and compute capabilities on-premise to a total of approximately 16 PB of storage, 5,000 computing cores and 200 GPUs of compute power, this new system will enable researchers to keep up with the ever increasing data rates from new detectors and the increasing volumes of data for post-processing.

Although the new system is in place, the time taken to implement and deploy new systems, and the increasing demands for post-processing capability when users have returned home to their institutes, mean that Diamond is also beginning the move to using cloud based computational resources. Feasibility studies are already underway with major commercial cloud providers and locally as part of the the Science and Technology Facilities Council (STFC) funded IRIS e-Infrastructure project. Using OpenStack technology, IRIS provides distributed compute and storage capabilities at STFC Rutherford Appleton Laboratories, and the universities of Cambridge and Manchester. In determining how best to provide post-processing services for users of Diamond, the Scientific Computing Team are developing proof of concept services in 2019 to leverage the growing opportunity that cloud computing presents and looking at how best to present to end users such cloud based data analysis services. The use of cloud techniques is introducing concepts into the on-premise infrastructure already, such as the use of containerisation (both Docker and Singularity) and looking at how future on-premise infrastructure may become more virtualised or abstracted from the end user. This will facilitate more transparent high performance computer hardware upgrades in the future and ensure Diamond can leverage new technologies emerging from various companies in the most efficient way, to sustain the increasing data rates from detectors across Diamond.

: Beamline I13 detector, with Odin readout.

Odin is a software framework for Data Acquisition (DAQ) and Control of high-performance detector systems. It is the result of a close collaboration between STFC Technology Department and Diamond’s Beamline Controls Group over several years. The software framework provides a highly scalable DAQ software pipeline to capture data from the latest and fastest new detectors, and is designed to cater for further developments in fast detectors. Easy integration and roll-out to beamlines is supported through the EPICS area Detector interface that most imaging detectors at Diamond use.

The Odin software has been deployed on beamlines I13 and I14 to control and capture data from the Excalibur 3M detectors to support Ptychography imaging experiments. This improved the maximum possible framerate from around 50 frames per second (fps) to over 100fps while also improving the overall reliability. On Macromolecular Crystallography beamlines, the Odin framework now supports readouts from three new Eiger detectors. The first two being Eiger 4M’s on beamline B21 and VMXi and the third an Eiger2 16M on beamline I04.

The Odin software framework has been designed from the start to scale and grow as new detectors produce ever-increasing data rates. This is achieved by a parallel architecture which allows more nodes to be added to process the data streams in parallel. The next generation of detectors developed at Diamond all use Odin with development and testing phases already inplace. These are Tristan a new large-angle time resolved detector based on the Timepix sensor; and Percival, a collaborative development of a Complementary Metal Oxide Semiconductor (CMOS) sensor, for low energy applications. In both cases, the Odin framework delivers continuous readout in multi-gigabytes of data per second.

: PandABox: Flexible triggering for continuous scanning.

PandABox is a collaborative development project between Synchrotron SOLEIL and Diamond; with SOLEIL responsible for the hardware and Diamond for the firmware and software. It provides a highly flexible solution to enable new hardware synchronisation techniques in sample scanning. PandABox was a key component in speeding up scanning of samples delivered by the Mapping Project, improving scanning rates from 5 Hz up to 300 Hz on some beamlines. Some recent developments will enable 10kHz scanning to be delivered in areas like the Ptychography project.

PandA uses the latest system-on-chip Field Programmable Gate Array (FPGA) technology where a powerful FPGA fabric, Central Processing Unit (CPU) and peripherals are all functionally on the same chip. This is connected to a range of digital and analogue input/output (I/O), allowing connections to detectors, motion stages, diodes and Diamond’s timing system. To provide ease of use it is configured via a web page that allows logic blocks to be rewired at run time to produce new applications without having to rebuild the firmware in the FPGA.

Diamond has worked with Quantum Detectors to commercialise PandA and so other institutes can benefit from the platform. To further aid this, the hardware, firmware and software are open source allowing external contributors to help make PandABox even better in the future.

: Ptychography User interface in GDA utilising mapping for Realtime
Visualisation.

Diamond’s Generic Data Acquisition (GDA) system, used on the majority of beamlines, is a Java based Client Server Application that provides beamline experiment orchestration, data collection and real-time data visualisation managed through a customisable Graphical User Interface and Python syntax scripting environment.

In 2018 the Data Acquisition Group continued to develop a consolidated and streamlined scripting syntax to shield users from the differences between software and hardware controlled scanning. Simple software controlled scans and more complex hardware scans can now both be specified using an intuitive syntax. This enables users to seamlessly move from slower software scans to faster hardware scans, thereby enabling greater experiment capabilities through recording higher resolution data in the same or shorter periods of time. The range of available scan maps was further extended to include those appropriate to support new science areas such as Ptychography. Initial tests of the Ptychography technique using the new spiral path yielded an order of magnitude improvement in resolution.

The consolidated scan syntax development work represents a first step in achieving one of the goals of the strategic roadmap for GDA, this being to insulate clients from the Server implementation providing a well-defined but extensible interface. In 2019 the team will be building on this initial work by adding a secure REST API and publish/subscribe message bus to the GDA server. Isolating clients from the GDA server implementation will make GDA a product that can be more easily adopted outside of Diamond and also facilitate the move to a web-based GDA Client. The message bus will make it easy to dynamically change the available auto-processing applications, and have per scan auto-processing queued with the scan request.

: Mark Basham, Senior Software Scientist, during the launch of 'Science Scribbler: Virus Factory'.

In order to study the life stage of viruses, scientists working in our state-of-the-art electron Bio-Imaging Centre (eBIC) have collected a three-dimensional snapshot of rheovirus particles replicating inside mammalian cells 12 hours post-infection. This data was acquired using Cryo-Electron Tomography, a technique in which a thin slice of frozen material is tilted through a range of angles in the microscope and a sequence of images are recorded. These images are then reconstructed into the 3D volume using computational techniques.

To extract useful information from this data, we have turned to a crowdsourcing model and called on people of all ages around the world to help speed up the analysis. This methodology not only allows us to find where the viruses are present in the cell and what life stage they are at, but it also engages the public with the science carried out at Diamond. In addition, these data annotations will prove valuable for training machine learning models. Once these models are trained, further data analysis and the resulting findings will be much quicker to achieve.

As part of this, we have teamed up with The Zooniverse (www.zooniverse.org), a platform that encourages and enables curious minds to contribute to scientific research projects. Our project entitled 'Science Scribbler: Virus Factory' does not require any specialised training or expertise and is open to all ages. In the first task, Virus Picker, a randomly selected window of the data is presented to the citizen scientist and they simply have to click on any virus particles that they see in this image. The data is recorded in the Zooniverse database for us to download and analyse at a later date. To ensure the data is robust, each data window is shown to five annotators before being retired from the workflow. Thanks to the hard work of all the contributors, this stage is now complete.

The second task, Virus Classifier, takes the individual virus particles selected from the first workflow and presents them to the volunteer. The task of the citizen scientist in this scenario is to classify the virus into the correct life- stage category according to information provided to them in a key that shows representative examples.

This is just the first of a number of projects that have been planned for hosting on The Zooniverse over the next three years and has been made possible through a collaboration with the Department of Structural Biology at the University of Oxford and by funding from the Wellcome Trust. Already, incredibly useful information on the location and life stage of every single virus particle in a host cell dataset is streaming in from all four corners of the globe. This will pave the way to more powerful data analysis pipelines and result in important scientific insights that will help us understand the infection and replication mechanisms of these pathogens.

Diamond Light Source is the UK's national synchrotron science facility, located at the Harwell Science and Innovation Campus in Oxfordshire.

Diamond Light Source Ltd
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE

See on Google Maps

Diamond Light Source^® and the Diamond logo are registered trademarks of Diamond Light Source Ltd

Registered in England and Wales at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom. Company number: 4375679. VAT number: 287 461 957. Economic Operators Registration and Identification (EORI) number: GB287461957003.

feedback

On this website

About Diamond

For Users

Industry

Science

Instruments

Careers

Public

Procurement

Software & Tools

Sub-navigation

In This Section

Sub Navigation

Opportunities at Diamond

Subscribe to our Corporate and Science Update mailing list

Scientific Software, Controls and Computation

Mark Heron, Head of Scientific Software Controls and Computation