Notes for Delivering Data

This document details requirements and procedures for delivering scientific data to the Legacy Archive for Microwave Background Data Analysis (LAMBDA). This information is also available as a printable PDF.

Data Delivery Format

LAMBDA accepts data in a wide variety of formats. By and large, full sky maps are most commonly provided in binary FITS (Flexible Image Transport System) tables using the HEALPix pixelization scheme, with a table that may contain Stokes I, Q, U and Nobs. Some measure of uncertainty per pixel in these files is strongly encouraged. Bandpower/power spectra are typically provided as ASCII tables. And likelihood software is generally provided in compressed tar files. However, virtually any useful file format may be provided.

FITS files should be readable by standard Astronomical research tools. The LAMBDA team typically tests incoming files using ftverify and astropy to ensure that files can be read and will work with a science team to tweak FITS files as appropriate and desired.

In any case, detailed documentation should be provided that describes how incoming data is formatted and how it should be read and used. This documentation should describe how the data is or should be calibrated. Data should be accompanied by a description of the experiment that provides details such as bandwidth, frequency, resolution, dates operational, etc. Pointers to published papers are sufficient to satisfy these requirements.

In response to a question from the user community, the LAMBDA team has begun evaluating the Hierarchical Progressive Survey (HiPS) system for presenting skymaps to the user community. Further information describing HiPS can be found at http://www.ivoa.net/documents/

The LAMBDA team currently has no plans to translate their data collection to support this system though this is still under evaluation. However, they will certainly entertain data delivered in this format and will work with a science team to evaluate its feasibility and implementation if ultimately this form is used.

Transferring Data to LAMBDA

The LAMBDA team has set up an upload-capable anonymous FTP server to facilitate the delivery of data. For security reasons, this server will only be started when a data delivery is expected; close coordination between the LAMBDA and science teams is essential.

Per NASA requirements, this server only supports FTPS (an encrypted protocol) and uses a certificate from DigiCert to enable that encryption and associated server validation. Only the TLS 1.2 protocol is supported at this time. And finally, only passive FTPS is supported. An exhaustive list of tools capable of interacting with this service isn't feasible but the LAMBDA team will update this document with tools and associated pitfalls as they come to light.

No data is delivered to the community by LAMBDA through this service; it exists only to facilitate data uploads. The top of the FTP directory tree is therefore very spartan, consisting of a welcome message and an uploads directory. The user should navigate into the uploads directory and upload their data there. For security reasons, the permission on this directory are very limited, allowing the user to write files into it but denying the user the ability to retrieve or even to view those files. In effect, files uploaded here are uploaded into a black hole. Upon delivery the user should let their contact in the LAMBDA team know that the files are there; LAMBDA will take it from there.

All data delivered to LAMBDA will be vetted by staff prior to being made publicly available through the web site.

The user of this site will not be able to create subdirectories within the upload area. Therefore, it is recommended that each directory tree be consolidated into a compressed tarball and those tarballs are then uploaded. For example, a delivery consists of three directory trees: maps, spectra and docs. At the top level, three tarballs should be created: maps.tar.bz2, spectra.tar.bz2 and docs.tar.bz2. The commands could be:

cd /delivery/path
tar cjf maps.tar.bz2 maps
tar cjf spectra.tar.bz2 spectra
tar cjf docs.tar.bz2 docs
Alternatively, a single compressed tarball could be created and pushed. This retains the directory structure envisioned by the science team.

Tools

The standard Linux/Unix ftp tool does not support FTPS so it is necessary to find other tools to use to interact with this service. Once such tool available in the Linux environment is lftp; a discussion of this tool follows. Other tools are available for other environments, such as Filezilla, etc.

lftp

This service has been tested with the lftp tool. Due to the passive nature of the service, some configuration is necessary to enable this to work. At the minimum the port must be specified and ssl forced on; the following command (in a Linux environment) will work:

lftp -u anonymous -e 'set ftp:ssl-force true' -p21 ftps://gs66-lambdaftp.ndc.nasa.gov

Alternatively options may be set in an lftp configuration file (~/.lftprc); the following has been used successfully:

set ftp:ssl-auth TLS
set ftp:ssl-force true
set ftp:ssl-protect-list yes
set ftp:ssl-protect-data yes
set ssl:verify-certificate no
With this in place, the lftp command may become:
lftp -u anonymous -p21 ftps://gs66-lambdaftp.ndc.nasa.gov

Upon connection, normal FTP-style commands can be used. For example,

lftp> cd uploads
lftp> put data.file.fits
lftp> mput *.fits
lftp> bye

Data Distribution

The LAMBDA team will work in collaboration with each science team to develop an interface into the data to be distributed. This typically consists of a front page describing the experiment or, in the case of a small experiment, a line in an Experiments Table providing basic experiment information. Either option links to one or more pages describing each data product in tabular format, which in turn provides access to data download pages. Each product will also be accompanied by a detailed data description page.

If the standard site approach is deemed inappropriate or insufficient, LAMBDA is open to other approaches. To that end, science teams are welcome to provide draft versions of the web pages or user interface that will present their data to the public. However, for reasons derived from NASA web policies, LAMBDA must retain the final authority over the final design.

Project Web Sites

As an experiment or project nears or passes its fiscal existence, any public facing infrastructure such as a descriptive web site frequently disappear without a trace. LAMBDA is willing to maintain an archival copy of an experiment’s public web site and make it available once a project needs to take their site down.

A service of the HEASARC and of the Astrophysics Science Division at NASA/GSFC
Goddard Space Flight Center, National Aeronautics and Space Administration
HEASARC Director: Dr. Andrew F. Ptak
LAMBDA Director: Dr. Thomas M. Essinger-Hileman
NASA Official: Dr. Thomas M. Essinger-Hileman
Web Curator: Mr. Michael R. Greason