Gaia data processing

DPAC membership map

In a previous blog entry you have learned about the rehearsals taking place in preparation for the Gaia data processing. Unlike a mission such as the Hubble Space Telescope, Gaia does not produce data that is immediately scientifically useful. The raw telemetry must first be processed before we can obtain the sought after distances, motions, and properties of the stars observed by Gaia. This immense task will be undertaken by a pan-European collaboration, the Gaia Data Processing and Analysis Consortium (DPAC).

The DPAC consists of about 450 persons, spread over academic institutes and space agencies throughout Europe and beyond, who are actively contributing to writing the millions of lines of code needed for the data processing and to subsequently operate the software systems and validate the resulting output.

Schematic overview of data flow during processing

To organise the large amount of tasks to be carried out the DPAC has been subdivided into nine so-called Coordination Units (CUs). Each CU takes the responsibility for the development of a specific part of the Gaia data processing: system architecture, simulations, astrometry, photometry, spectroscopy, object processing, variability processing, astrophysical parameters, and catalogue publication.

The astronomers in the CUs conceive the scientific algorithms for the data processing and also carry out a large fraction of the software development. The software is then run at one of the six data processing centres (DPCs). The personnel at the data processing centres also provide the much needed software engineering expertise. Such a large software system cannot be developed and operated by astronomers alone!

The figures show a map of the countries where DPAC members are located (with red dots indicating the locations of the DPCs) and a schematic overview of the data flow within DPAC during the data processing. The schematic shows how each CU is supported by a specific DPC (indicated in red). The data exchange within DPAC will take place through the so-called Main Data Base (MDB) housed at ESAC. After the completion of a processing cycle data is then extracted from the MDB and prepared for release.

Note that the Gaia project is unique in that the scientific data produced by DPAC are not subject to a proprietary period. On completion of a processing cycle the results are immediately available to the scientific community and also to the general public.

Watch this space for more detailed blog entries on DPAC’s activities.

This entry was submitted by Anthony G.A. Brown, chair of the Data Processing and Analysis Consortium.