Work Package 10 (WP10): Access to Science Data Centres. Space missions

Access to the most demanded European Science Data Centres, providing data gathered by the solar satellites HINODE, IRIS, and the Solar Dynamics Observatory (SDO), will be offered as well as ground-based data from GREGOR, IBIS, and SST (not previously offered through EC funding). A novelty in this project is the addition of access to numerical simulations, including synthetic observables, to enable close collaboration between observations and theory.  This access to data for solar research will expand significantly the content of the Programmes for the high-resolution solar physics community supported in the past.

Access to the database repositories is freely provided through the internet connection.ich

Hinode SDC

Hinode Science Data Centre Europe is run by the Institute of Theoretical Astrophysics at the University of Oslo on behalf of the Norwegian Space Centre (NSC). It contains all data from the Japanese solar satellite Hinode and the NASA Small Explorer Satellite Interface Region Imaging Spectrograph (IRIS). Data are available very shortly after the downlink to Earth (typically 24 hours), through advanced and very efficient search pages, primarily to users in Europe. The emphasis on user needs and very efficient searches through the usage of caching and custom-made code has made the centre the most popular data centre for Hinode data with a wide usage also outside Europe. Thanks to the FP7 SOLARNET project, the data centre was expanded to also include all data from IRIS and a few 3D numerical radiation MagnetoHydroDynamical simulations were also added. As part of the new services that will be available for users during this SOLARNET project, the centre will be expanded to contain a wide variety of numerical simulations, including synthetic observables derived from those, and also gradually make available fully calibrated datasets from the Swedish 1-m Solar Telescope (SST).
Services currently offered by the infrastructure: All data are made available through advanced and very efficient search pages, primarily to users in Europe but also to other countries. Auxilliary data, like thumbnails, wavelength calibrated data, active region identifications, etc are calculated and provided. All web searches are also given in the equivalent IDL source code to enable efficient searches from scripts.

BE-WISSDOM

Name of the infrastructure: Belgian Web Incessant Screening for SDO Mission
Location of the infrastructure: Brussels, Belgium
Web site address: http://wissdom.oma.be
The WISSDOM data center is hosted by the Royal Observatory of Belgium (Observatoire Royal de Belgique, ORB). It receives funding from the Belgian Science Policy and from ORB’s endowment and has benefitted from funding under the FP7 SOLARNET project. It provides access to various datasets produced by space as well as ground-based telescopes. It hosts two services. First, a SDO data center (http://sdo.oma.be/) is dedicated to the acquisition of SDO-AIA and SDO-HMI data from the Center for Astrophysics located in Harvard, and the subsequent provision of the data throughout Europe. Second, a prototype of a Solar Virtual Observatory (http://SOLARNET.oma.be/) offers a single point of access to a variety of datasets stemming from both ground-based telescopes as well as space-based missions.
In 2015 the SDO data center was completely refurbished. Brand new database, storage, and web servers were bought, bringing its hardware up to date with current technology. The center was equipped with 160TB of usable storage, out of which 100TB is reserved for the SDO mission. This change in hardware was coupled with a change in the software for data acquisition, redistribution, and user access, which was completely rewritten and now runs independently from the netDRMS software provided by the JSOC in Stanford. The SDO mission gathering a huge archive of useful data (about 1 petabyte every two years), we decided to shape our service on the efficient provision of data on demand rather than on the latest available data. The new software better satisfies the actual needs of the users and offers a more user-friendly web interface to search and access the data.
Thanks to the FP7 SOLARNET project, a prototype of a Solar Virtual Observatory was developed. It allows searching metadata and downloading the corresponding data from the data provider site. As the software for downloading data from the SDO data center and the SVO software were developed in parallel, an efficient use of hardware resources could be implemented. As part of this SOLARNET project, the SVO prototype will be turned into a fully operational software. This will allow for a user-friendly level response time of the server and facilitate the ingestion of new data from existing datasets. The SDO data center and the SVO support research in all aspects of solar physics as well as operational services that gather information for space weather purposes. The SVO serves both high resolution ground-based data, often recorded during campaigns, as well as synoptic datasets. As such, it increases awareness in high resolution datasets within the solar physics community at large. Reversely, researchers primarily interested in high resolution observations may easily find context images complementing their high-resolution observations.

Services currently offered by the infrastructure:
ORB’s infrastructure provides a public remote access to the SDO data using a number of tools:
1. The web application SDO wizard http://sdo.oma.be/wizard/ provides a user friendly way to search and access the data,
2. ORB is a data provider for the US Virtual Solar Observatory (VSO), which is readily accessible from within IDL/SWW and within Python/SunPy,
3. Full size SDO/AIA images at a one hour cadence and reduced size AIA images at a 2 minutes cadence are directly available via ftp,
4. SDO/AIA movies of the latest 24h are produced and made available and the latest quick look data can be fetched via http.
In 2017, the sdo.oma.be website received 16532 visits and served 12TB of data to external users from many countries around the world, including the United Kingdom, the United States of America, Greece, Belgium, the Czech Republic, Germany, France and Italy. The prototype SVO allows searching and downloading data either from a web application http://SOLARNET.oma.be or from within IDL (resp. Python) thanks to a custom made IDL procedure (resp. a Python library). Cross dataset search by date of observation, wavelength, tags and telescope; search by solar events date and type; co-observation searches e.g. with data overlap; thumbnail quick look, and FITS header are all available within the SVO. In addition, all metadata and data locations are available through a RESTful API. It is thus possible for external tools to be interfaced with the SVO’s underlying metadata database via the RESTful protocol.

Stockholm University Department of Physics

Stockholm SST Archive

The Swedish 1-m Solar Telescope is operated by the Institute for Solar Physics of Stockholm University. The telescope is situated on La Palma, Canary Islands. Data is transferred to the Institute premises in AlbaNova University Centre in Stockholm (picture) and processed there. The data centre in Stockholm is currently in a phase of expansion with investments in new hardware and new recruitment. This will make it possible to store and reduce data for all observers, not only Swedish ones. One of the components will be a data archive of reduced data sets that are accessible from the outside. We expect that the main way of accessing data will be via the user-friendly Solar Virtual Observatory hosted by partner ORB.
Services currently offered by the infrastructure: The SST and its instrumentation remain at the forefront in high-spatial-resolution solar physics and its data can therefore be expected to be in great demand the coming years.

The Stockholm SST archive is expected to come online in 2020 and will gradually be built up with data released by the various PIs. Direct access will be possible through a simple interface, but the main purpose of the archive is to allow access via the Solar Virtual Observatory. To that end, we will provide the SVO with the relevant metadata that will be ingested in its database to allow searches. Once the user has selected data, the Stockholm SST archive will be accessed.

IBIS-A

Name of the infrastructure: IBIS data Archive
Location of the infrastructure: Rome, Italy
Web site address: http://ibis.oa-roma.inaf.it/IBISA/

The IBIS data Archive (IBIS-A) contains data acquired with IBIS (Interferometric BIdimensional Spectropolarimeter), an imaging spectro-polarimeter based on a dual Fabry-Perot interferometric system. IBIS allows spectropolarimetric observations of the solar photosphere and chromosphere at high spatial (pixel scale of 0.09 arcsec), spectral (> 200000), and temporal resolution (8-15 fps). A typical IBIS data set consists of measurements taken in sequence over multiple spectral lines (e.g. Fe I at 6302 Å and 6173 Å, and Ca II at 8542 Å); each line is sampled at several spectral positions (more often between 10 and 30), and each position at six polarimetric states (I+Q, I-Q, I+V, I-V, I+U and I-U). Therefore, a single scientific observation consists of many data (e.g. hundreds of image files, 1000 x 1000 pixel each, with variable cadence). IBIS-A has been designed to realize the storage, the management, and the retrieval of the IBIS data. Currently, IBIS-A includes 20.0 TB of data taken during 14 observing campaigns carried out from 2012 to 2017 on 92 days. The IBIS-A has been realized in the framework of the FP7 SOLARNET High-resolution Solar Physics Network, which aimed to integrate the major European infrastructures in the field of high-resolution solar physics, as a step towards the realization of the 4m EST European Solar Telescope. As part of the new services that will be available for users during this new SOLARNET project, IBIS-A will be expanded to contain more observations derived from past and new observing campaigns. Besides, an effort will be made to gradually make available fully calibrated datasets from all available observations and to develop new services that are described in the following.

Services currently offered by the infrastructure:
IBIS-A provides a public remote access to the IBIS data derived from both PI- and service-mode telescope operations during several observing campaigns performed from 2012 to present. The raw data in FITS format acquired during observing campaigns are transferred and stored on a RS24S3 SuperNAS server (96 TB, 64 GB Ram, 2 CPU Haswell 4C E5-2623V3 3G), following the data definition and file arrangement given at the telescope. IBIS-A allows data users to search and access the data via an HTML/Javascript based web page. The IBIS-A interface allows users to search data according to various criteria: solar target, disc position, observational mode, data range; criteria can be combined in order to refine searches. The products of the data search include information on available data, movies and plots of the atmospheric seeing during observations, as basic information to allow users to assess the data quality. Auxiliary data, e.g. FITS header contents and fully calibrated observations are also provided for some datasets, as well as the IDL code to process the raw data for instrumental calibrations. IBIS observing days have been in great demand in the past and we therefore expect that a more general access to IBIS data will be a very attractive addition to the virtual access programme.

GRIS Data Archive

Name of the infrastructure: GRIS Data Archive
Location of the infrastructure: Freiburg im Breisgau, Germany
Web site address: http://archive.leibniz-kis.de/pub/gris/

The GREGOR Infrared Spectrograph (GRIS) online archive was initially developed to facilitating access to data for all member institutes of the GREGOR consortium. KIS developed an archiving concept to keep track of all observations and to minimize the obstacles for scientific exploitation of the data. After the early science phase of the GREGOR telescope, data in the GRIS archive became publicly available to the entire scientific community in 2017. Currently it contains 9TB of observational data and runs on a dedicated KIS server. Users can access the archive via a web interface or using an FTP connection. An improvement that is foreseen during the SOLARNET project, is the full automatization of calibration and injection of data into the archive as well as the computation of higher level data products to provide the scientist with science ready data as quickly as possible. We plan for an improved query system that allows the user to search for previously defined data tags (e.g. sunspot or faculae). Additionally, data from second generation GREGOR instruments, such as the GRIS Integrated Field Unit, shall be processed and archived in the near future.

Services currently offered by the infrastructure: The service includes data calibration, archiving, curation and computation of context data. Currently the GRIS archive contains 1000 unique data-sets with unprecedented spatial and spectral resolution in the 1.0 μm and 1.5 μm regime. The archive consists of a Web interface with links to webpages that contain overview maps of all the scans and context data. From there the scientist can choose and download calibrated science ready data. Available are raster scans across solar targets, time sequences and sit-and-stare observations. Most of the data is spectropolarimetric, but there are some spectroscopic measurements too. Additionally, KIS provides an IDL GUI (including a stand-alone version) to visualize the GRIS data and to perform simple data analysis. We also offer higher level data products (e.g. Doppler velocity and configuration of the magnetic field) from spectral line analysis.