Workpackage 2 - Real-time Data Processing

It is common among many PNI experiments that they deliver series of consecutive single measurements (frames) with high frame rates and large amounts of detector and auxiliary data in each frame. Usually, evaluation starts on a frame-by-frame basis, then data obtained from all collected frames is evaluated further, and finally the results have to be visualized and archived. An online data evaluation system that fulfils the requirements of real-time data analysis and visualization must be fast and must provide sufficient flexibility to cover the very different analysis demands of a great variety of experiments.

The development of the different applications of such a system should be driven by the needs of the most data demanding experiments like Small Angle Scattering (time resolved in-situ SAXS, SANS and raster scanning micro/nano-SAXS), neutron time-of-flight experiments, Micro-Tomography, Coherent Diffraction Imaging, X-ray Photon Correlation Spectroscopy, X-ray Ion Coincidence Spectroscopy, microfluorescence tomography, and macromolecular crystallography.

The architecture of the data processing system for real-time data assessment with multi-dimensional pixel detectors is outlined in Figure 1. The processing is split in two domains defined by the data rate that can be handled by today’s PC technology. We propose the development of a high-performance ‘computing system’ based on single PC computer architecture. The internal system bus (PCIe) offers an excellent bandwidth and available high-end graphic processors units (GPUs) can provide the required amount of parallel computing power. The computing cell is fed by digital data from pixel detectors (including analog-to-digital conversion). A significant class of experiments would hugely benefit from processing steps in hardware, in order to shrink data volume before starting the evaluation in the PC. On the other end of the data processing chain the data will be stored for later analysis. Realtime data assessment will help to reduce the amount of stored data and at the same time to evaluate their quality.

hdri-1-en.png

Simplified scheme of data flow at PNI facilities.

This workpackage focuses on three key aspects for real-time quality checks:

  • Essential data processing with dedicated hardware
  • Real-time data assessment with parallel computing on PCs
  • Analysis methods / Applications