How to Submit Data


To submit data you must be a member of the CALIPSO Validation Team. To become a member of the Validation Team please visit the Getting Involved Page. After you provide the necessary information and we add you to the database we will assign you an account with a username and password.

Submitting data

1 Overview

The content of this page describes the procedure for the CALIPSO validation team to submit data to the data catalog for use in CALIPSO data validation. The data catalog provides the CALIPSO science team with one location to identify all data available from the CALIPSO validation team for validatin the CALIPSO data products. Use your username and password to login to the Data Catalog on the CALIPSO validation site.

All validation data can exist either at the Hampton University data storage or at an offsite network archive. Regardless of the location of the data the validation team should follow the specific procedure outlined within this document before placing data in an archive to ensure that the users of the data are capable of reading the data. All data files should include well-defined variables so they can be readily compared with other data sets. The source of the data (i.e. instrument), and the time and location the data was observed should be clearly indicated. Use of all data will follow the Data Exchange Protocol that both science team and validation team members must agree to every time they access the data catalog. If you have any questions, concerns, or problems with any aspect of the data exchange process please email to: casvalidation@hamptonu.edu.

1.1 HU data storage

Hampton University provides an on-line location for the validation team to place files with preliminary data for use by the CALIPSO science team and their designees. Once logged into the data catalog you will be taken to your validation team member website. You need to click on the link of the site for which you want to upload data. On the site information page you will see a form for submitting data. You can add one file at a time using the browse button. Once all files are added you click 'Upload' to upload the data to the Hampton University server.

1.2 Offsite data archive

If your site is part of a large network of sites with an existing data archive it is not necessary to store your data at the Hampton University data storage. However, you will need to create a directory specifically for coincident data with CALIPSO overpasses. Coincident data is defined in the CALIPSO QPQ Validation Plan and Data Exchange Protocol. Specific instructions for accessing this data should be provided to the QPQ validation coordinator so that it may be presented to a potential user prior to attempting to access the data. These directories will be searched for new files once per day and the file names and times of upload will be stored in a database used by the data catalog search engine.

2 File Details

2.1 File names

File names should follow a format that describes the data, time, instrument, and data format. The file naming convention can be set by the validation team member, but all files from those sites should follow the same convention and that convention should be described specifically in a provided Meta file or Readme file. If no file naming convention is available to you we suggest you use the following:

  • IIYYMMDD.EEE
    where II is instument name abbreviation
      YY is 2 digits year
      MM is 2 digits month
      DD is 2 digits day
      EEE is file format name abbreviation
     
  • Instument name abbreviations(II):
      BL - Backscatter Lidar
      BL - Backscattersonde
      CM - Ceilometer
      CR - Cloud Radar
      HL - HSRL Lidar
      MF - MFRSR
      RL - Raman Lidar
      SU - Sun Photometer
     
  • File format name abbreviations(EEE)
      doc - Microsoft Word document
      hdf - hierarchical data format
      nc - netcdf
      pdf - Adobe portable document format
      ps - postscript
      txt - text


2.2 Data format

The CALIPSO science team suggests that all files are of the HDF (format used for CALIPSO data) or the netCDF file format. However, we recognize that no funds are supplied for this data exchange and therefore no time may be available to conform to these data formats. However, ASCII files take up a lot of space and if the files in your directory in the Hampton University data storage take up too much space it will be moved to tape and any user will need to request the data. Also, data that cannot be understood is useless. So, if you cannot follow one of these file formats, some guidelines must at least be adhered to. All data files should have a header that clearly defines the originator of the file, the source of the data (instrument and location), the units used for each variable including any scale factors, and a missing data number or flag. Each column of data should be clearly labeled or this information should appear in the header. Each observation should be time-stamped in Universal Time Coordinated (UTC) or Greenwich Mean Time (GMT). For vertically resolved data, each data point should be altitude or pressure stamped.

2.3 Meta-data files and Readme files

The validation team member should supply a metadata file or Readme file in a text file, portable document format (PDF), Microsoft Word document (DOC), or Postscript file (PS) that describes all data supplied by each site. The information should include the file naming convention, contact name, description of the data in each type of files to include, for example, units, techniques, assumptions, calibration, etc.

2.4 Read files

The user could supply a read file or files to more easily facilitate use of the data. These files should be discussed in the README file (i.e. a description of which read files pertain to which types of files). The read files should be in one or more of the following formats: IDL, Matlab, C, C++, Fortran, or Object Pascal (Delphi) .


Home | News | Project Documents
Data Exchange Protocol | Instrumented Network Links
HDF File Format | NetCDF File Format | How To Submit Data
How To Receive Data | Getting Involved | Data Catalog | Overpass Times