Previous: Madrigal data model   Up: Madrigal admin guide   Next: Creating Madrigal data files

How Madrigal data is organized

In the Madrigal database, the data are organized by experiment. An experiment consists of data from a single instrument, and, as a rule, is meant to address a particular scientific goal. Most often an experiment will correspond to a particular set of operating modes run for a contiguous interval of time. For example, the data collected by an incoherent scatter radar during a world day would constitute an experiment. More complicated situations exist, and judgment may have to be exercised in determining what constitutes an experiment. For example, an experiment might be interrupted temporarily in order to use a different set of operating modes in support of a satellite overpass. In this case there could be two experiments which overlap in time.

To each experiment there corresponds a directory. These directories are of the form madroot/experiments[0-9]*/<year>/<instrument>/<directory>, where year is the four-digit year. The next level is the experiments[0-9]*, where the directory must begin "experiments" and then may be followed by any number of digits. This is to allow additional hard drives to be mounted to expand the directory space. The next level, instrument, is the 3-letter mnemonic for the instrument (set in the instTab.txt file), and <directory> is an arbitrary directory name. An example might be /opt/madrigal/experiments/1997/son/06jan1997_001, which would contain a 1997 Sondrestrom (son) experiment. The use of the starting date in the experiment directory name is not required.

Prior to Madrigal 2.5, the final directory name was required to be in the form DDmmmYY*, where the date was the start date of the experiment and * is an optional character to distinguish different experiments with the same start date. For example, /opt/madrigal/experiments/1997/son/06jan97 contained Sondrestrom data for an experiment beginning on 6 January, 1997, where year =1997, instrument =son, start_date =06jan97, and there was no optional character because there was only one Sondrestrom experiment starting on that date. This convention has been dropped as a requirement, although it is still commonly used as a convention.

Previously, experiments that did not follow the above naming convention were effectively "hidden". With Madrigal 2.5, the security field in the metadata file expTab.txt can be used to hide (or restrict access to) an experiment. (See the change experiment status tool).

Each experiment directory must contain:

Each experiment directory may contain:

An essential role for a Madrigal administrator is to run the script madroot/bin/updateMaster on a regular basis. This script performs a number of functions:

It is recommended that this script be set as a cron job to run once a day.

For Madrigal sites with limited bandwidth, updateMaster can occasionally be run with the -s flag. This will skip any possible download of the latest geophysical files, and so maybe run faster. However, it is still important to run updateMaster without the -s flag on a regular basis.

A complete description of the various ways to add and modify experiments in a Madrigal database is given in the next section. The updateMaster script must be run after each change described in that section for the changes to take effect.

Note that an experiment need not contain any Madrigal files.

Previous: Madrigal data model   Up: Madrigal admin guide   Next: Creating Madrigal data files