Analysis Pipelines

A range of different software packages are used to automatically process data collected on the MX Beamlines. These are generally separated into Auto Processing and Downstream Processing pipelines. Auto processing pipelines aim to provide merged intensity data (MTZ), which the downstream pipelines can use to attempt phasing and model building, providing initial PDB files.

Note that some pipelines require additional information to be uploaded to ISPyB while Registering a protein.

For each auto processing and downstream processing pipeline the files generated are accessible via ISPyB, under the "Logs & Files" button.

Clicking this button opens a menu showing the possible logs and files that can be downloaded or viewed for that pipeline.

Items with a view button can be viewed in a modal window on the webpage. HTML files may contain graphs and logs. JSON files are typically graphs and .txt or .log files are plain text.

This contains only a subset of the most used files generated from the pipeline. If this is insufficient, then you will need to find the path to the data on disk. This will be related to the path found for the raw data which can be seen on ISPyB by clicking the triangle arrow on a dataset to expand this field.

The end of the path will need to be changed, however. Autoprocessed loges/files are found in the `processed` folder found in your visit. From there, follow the directories for your specific protein and sample name until you reach a folder containing data from the different processing pipelines. Note that for some beamlines, you will need image numbers / well numbers. This information can be found back on the dataset in ISPyB. If the data have been archived, the Connecting to Diamond page has instructions for access. There is also an archive button on ISPyB which will download a zipped directory of files related to that processing.

If you are missing access to files you expect to find, please contact [email protected] in the first instance who can redirect you appropriately.

Any files of the type "Result" will have a button to upload them to CCP4 Cloud. If you do not have a CCP4 Cloud account, you can set one up here.

You will then need your CCP4 Cloud username and your CloudRun ID as credentials to put into Synchweb. You can find your CloudRun ID on the "My Account" page of CCP4 Cloud.

Uploaded files will be stored in your "My Cloud Storage", and you can then access them using the "Import from cloud storage" task.

There is more help available here.