Previous: Madrigal web services   Up: Remote access programming tutorial toc   Next: Remote access using python

Matlab Madrigal remote data access tutorial

The easiest way to use the Madrigal Matlab remote data access API is to simply let the web interface write the script you need for you. Just choose the Access data pull-down menu and choose Create a command to download multiple exps. Then follow the instructions, and you will have the command you need to download whatever you want from Madrigal. Be sure to select Matlab as the language you want to create the command with. You can choose to download files as they are in Madrigal in either column-delimited ascii, Hdf5, or netCDF4 formats, or you can choose the parameters yourself (including derived parameters), and optionally include filters on the data you get back.

The rest of this tutorial is for those who want to go beyond the automatically generated commands and write more advanced Matlab applications that access Madrigal data.

The following are the Matlab methods for accessing Madrigal data remotely:

A good way to learn how to use this Matlab API is to run this example.

   ***********************************************************
function cgiUrl = getMadrigalCgiUrl(url)
  getMadrigalCgiUrl  	parse the main madrigal page to get the cgi url
 
  With Madrigal 3, this method simply returns the original url.

  input: url to Madrigal

  output: cgi url for that Madrigal Site

  Note: parses the homepage for the accessData link
   ***********************************************************
function version = getVersion(cgiurl)
  getVersion  	returns a a string representing the Madrigal version

  inputs:  cgiurl (string) to Madrigal site cgi directory 
      (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
      Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 

  output:
    version - a atring representing the Madrigal version.

 Returns 2.5 if Madrigal does not contain the getVersion service
 
  Example:
  version = getVersion('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')

 Written by Bill Rideout (brideout@haystack.mit.edu)
  $Id: README 5381 2015-10-05 15:26:27Z brideout $
   ***********************************************************
function instArray = getInstrumentsWeb(cgiurl)
  getInstrumentsWeb  	returns an array of instrument structs of instruments found on remote Madrigal server.

  inputs:  cgiurl (string) to Madrigal site cgi directory 
      (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
      Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 

  output:
     instArray - array of instrument structs found

       instrument struct has the fields:

     instrument.name (string) Example: 'Millstone Hill Incoherent Scatter Radar'
     instrument.code (int) Example: 30
     instrument.mnemonic (3 char string) Example: 'mlh'
     instrument.latitude (double) Example: 45.0
     instrument.longitude (double) Example: 110.0
     instrument.altitude (double)  Example: 0.015 (km) 
     instrument.category (string) Instrument category description.  May be
       if Madrigal 2.6 or earlier.

  Raises error if unable to return instrument array
 
  Example:
  getInstrumentsWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
   ***********************************************************
function expArray = getExperimentsWeb(cgiurl, instCodeArray, starttime, endtime, localFlag)
  getExperimentsWeb  	returns an array of experiment structs given input filter arguments from a remote Madrigal server.

  Inputs:

      1. cgiurl (string) to Madrigal site cgi directory 
        (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 
 
      2. instCodeArray - a 1 X N array of ints containing selected instrument codes.  Special value of 0 selects all instruments.
 
      3. starttime - Matlab datenum double (must be UTC)
 
      4. endtime - Matlab datenum double (must be UTC)

      5. localFlag - 1 if local experiments only, 0 if all experiments 

   Returns a startime sorted array of Experiment struct (May be empty):
   
   experiment.id (int) Example: 10000111
   experiment.url (string) Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97'
      Deprecated url used only in metadata. To see real url, use realUrl
      field described below
   experiment.name (string) Example: 'Wide Latitude Substorm Study'
   experiment.siteid (int) Example: 1
   experiment.sitename (string) Example: 'Millstone Hill Observatory'
   experiment.instcode (int) Code of instrument. Example: 30
   experiment.instname (string) Instrument name. Example: 'Millstone Hill Incoherent Scatter Radar'
   experiment.starttime (double) Matlab datenum of experiment start
   experiment.endtime (double) Matlab datenum of experiment end
   experiment.isLocal (int) 1 if local, 0 if not 
   experiment.madrigalUrl (string) - home url of Madrigal site with this
       experiment. Example 'http://madrigal.haystack.mit.edu/madrigal'
   experiment.PI - experiment principal investigator.  May be unknown for
       Madrigal 2.5 and earlier sites.
   experiment.PIEmail - PI email. May be unknown for Madrigal 2.6 or
       earlier.
   realUrl - real url to experiment valid for web browser

  Raises error if unable to return experiment array
 
  Example: expArray = getExperimentsWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
                                         30, datenum('01/01/1998'), datenum('12/31/1998'), 1);

   Note that if the returned
   experiment is not local, the experiment.id will be -1.  This means that you
   will need to call getExperimentsWeb a second time with the cgiurl of the 
   non-local experiment (getCgiurlForExperiment(experiment.madrigalUrl)).  This is because 
   while Madrigal sites share metadata about experiments, the real experiment ids are only
   known by the individual Madrigal sites.  See testMadmatlab.m
   for an example of this.
   ***********************************************************
function cgiurl = getCgiurlForExperiment(experiment)
  getCgiurlForExperiment  	returns cgiurl of experiment.url as returned by getExperimentsWeb.

  inputs:  experiment struct as returned by getExperimentsWeb or getExperiments.


  output:
     cgiurl of experiment

  Simply truncates experiment.url to remove /madtoc///
 
  Example: If expArray is the value returned in the getExperimentsWeb example, and
           expArray(1).url = 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madtoc/1998/mlh/07jan98', then

     getCgiurlForExperiment(expArray(1))

     returns: 

          'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/'
   ***********************************************************
function expFileArray = getExperimentFilesWeb(cgiurl, experimentId)
  getExperimentFilesWeb  	returns an array of experiment file structs given experiment id from a remote Madrigal server.

  Note that it is assumed that experiment is local to cgiurl.  If not,
  empty list will be returned. Use getCgiurlForExperiment to get the correct
  cgiurl for any given experiment struct.

  Inputs:

      1. cgiurl (string) to Madrigal site cgi directory that has that
      experiment.
        (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 
 
      2. experiment id (int) as returned by getExperiments or
         getExperimentsWeb

   Return array of Experiment File struct (May be empty):
   
   file.name (string) Example '/opt/mdarigal/blah/mlh980120g.001'
   file.kindat (int) Kindat code.  Example: 3001
   file.kindatdesc (string) Kindat description: Example 'Basic Derived Parameters'
   file.category (int) (1=default, 2=variant, 3=history, 4=real-time)
   file.status (string)('preliminary', 'final', or any other description)
   file.permission (int)  0 for public, 1 for private 

  Raises error if unable to return experiment file array
 
  Example: expFileArray =
  getExperimentFilesWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', 10001686);
   ***********************************************************
function parmArray = getParametersWeb(cgiurl, filename)
  getParametesWeb  	returns an array of parameter structs given filename from a remote Madrigal server.

  Note that it is assumed that filename is local to cgiurl.  If not,
  empty list will be returned.

  Inputs:

      1. cgiurl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 
 
      2. filename (string) as returned by getExperimentFiles or
         getExperimentFilesWeb

   Return array of Parameter struct:
   
       parameter.mnemonic (string) Example 'dti'
       parameter.description (string) Example:
          "F10.7 Multiday average observed (Ott) - Units: W/m2/Hz"
       parameter.isError (int) 1 if error parameter, 0 if not
       parameter.units (string) Example "W/m2/Hz"
       parameter.isMeasured (int) 1 if measured, 0 if derivable
       parameter.category (string) Example: "Time Related Parameter" 
       parameter.isSure (int) 1 if can be found for all records, 0 if only
           for some records (implies not all records have same measured
           parameters)

  Raises error if unable to return parameter array
 
  Example: parmArray = getParametersWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
                                        '/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001')
   ***********************************************************
function records = isprintWeb(cgiUrl, file, parms, user_fullname, user_email, user_affiliation, filters, outputFile, missing, assumed, knownbad)
  isprintWeb  	Create an isprint-like 3D array of doubles via a command similar to the isprint command-line application, but access data via the web
  
  The calling syntax is:
  
  		[records] = isprintWeb(cgiUrl, file, parms, user_fullname, user_email, user_affiliation, [filters, [outputFile, [missing, [assumed, [knownbad] ] ] ] ])
  
   where 

     cgiUrl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiUrl.
  
     file is path to file
         (example = '/home/brideout/data/mlh980120g.001')
  
     parms is the desired parameters in the form of a comma-delimited
         string of Madrigal mnemonics (example = 'gdlat,ti,dti')

     user_fullname - is user name (string)

     user_email - is user email address (string)

     user_affiliation - is user affiliation (string) 
  
     filters is the optional filters requested in exactly the form given in isprint
         command line (example = 'time1=15:00:00 date1=01/20/1998 
                       time2=15:30:00 date2=01/20/1998 filter=ti,500,1000')
         See:  Documentation -> Administrator's Guide -> Using isprint for file quick looks for details

     outputFile - save the output to an output file.  If externsion is in
       .h5, .hdf, or .hdf5, the output format will be Madrigal Hdf5.  If
       extension is .nc, it will be netCDF4.  Otherwise ascii.

     missing is an optional double to represent missing values.  Defaults to NaN

     assumed is an optional double to represent assumed values.  Defaults to NaN

     knownbad is an optional double to represent knownbad values.  Defaults to NaN

     The returned records is a three dimensional array of double with the dimensions:

         [Number of rows, number of parameters requested, number of records]

     If error or no data returned, will return error explanation string instead.

   Example: data = isprintWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
                                '/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001', ...
                                'gdlat,ti,dti', ...
                                'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');

   For now avoids limits with urlread by only asking for maxRecs records at once.  Repeatedly
   calls isprintUnguarded, which is a method without any additional filtering.

    $Id: README 5381 2015-10-05 15:26:27Z brideout $
   ***********************************************************
function  result  = madDownloadFile(cgiurl,  fullFilename, outputFile, user_fullname, user_email, user_affiliation, format )
madDownloadFile downloads a Madrigal file to local computer in various
formats

    The calling syntax is:
       result  = madDownloadFile(cgiurl,  fullFilename, outputFile, [ format ] )
   
 Inputs: 

      1. cgiurl (string) to Madrigal site cgi directory 
        (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into
         cgiurl. 

      2. fullFilename - file to download as returned by getExperimentFilesWeb.m

      3. outputFile - name to save new file to

      4. user_fullname - is user name (string)

      5. user_email - is user email address (string)

      6. user_affiliation - is user affiliation (string) 

      7. format - one of the following strings:
          'simple', 'hdf5', 'netCDF4'
              'simple' is the default if not specified.  

         'netCDF4' option will only work with Madrigal 3.0 or greater

    $Id: README 5381 2015-10-05 15:26:27Z brideout $

result = -1;
   ***********************************************************
function record = madCalculatorWeb(cgiUrl, ...
                                   time, ...
                                   startLat, ...
                                   endLat, ...
                                   stepLat, ...
                                   startLong, ...
                                   endLong, ...
                                   stepLong, ... 
                                   startAlt, ...
                                   endAlt, ...
                                   stepAlt, ...
                                   parms)
  madCalculatorWeb  	Create a matrix of doubles via a the Madrigal derivation engine for a time and range of lat, long, and alt
  
  The calling syntax is:
  
  		[record] = madCalculatorWeb(cgiurl, time, startLat, endLat, stepLat, startLong, endLong, stepLong, 
                                   startAlt, endAlt, stepAlt, parms)
  
   where 

     cgiurl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
  
     time - Matlab datenum double (must be UTC)	7. startLat - Starting geodetic latitude, -90 to 90 (required)
     
     endLat - Ending geodetic latitude, -90 to 90 (required)
     
     stepLat - Latitude step (0.1 to 90) (required)
     
     startLong - Starting geodetic longitude, -180 to 180 (required)
     
     endLong - Ending geodetic longitude, -180 to 180 (required)
     
     stepLong - Longitude step (0.1 to 180) (required)
     
     startAlt - Starting geodetic altitude, >= 0 (required)
     
     endAlt - Ending geodetic altitude, > 0 (required)
     
     stepAlt - Altitude step (>= 0.1) (required)
  
     parms is the desired parameters in the form of a comma-delimited
         string of Madrigal mnemonics (example = 'gdlat,ti,dti')
  

     The returned record is a matrix of doubles with the dimensions:

         [(num lat steps * num long steps * num alt steps), 3 + num of parms]

     The first three columns will always be lat, long, and alt, so there are three
     additional columns to the number of parameters requested via the parms argument.

     If error or no data returned, will return error explanation string instead.

     Example: result = madCalculatorWeb('http:/grail.haystack.mit.edu/cgi-bin/madrigal', ...
                                         now,45,55,5,45,55,5,200,300,50,'bmag,bn');
   ***********************************************************
function [] = globalDownload(url, ...
    outputDir, ...
    user_fullname, ...
    user_email, ...
    user_affiliation, ...
    format, ...
    startTime, ...
    endTime, ...
    inst, ...
    kindats, ...
    expName, ...
    fileDesc)
 globalDownload is a script to search through the entire Madrigal database
 for appropriate files in ascii, hdf5, or netCDF4 format to store locally

    Inputs:

        url - url to homepage of site to be searched (Example: 
              'http://madrigal.haystack.mit.edu/madrigal/'

        outputDir - the local directory to store the downloaded files.
                 (Example: '/tmp/isprint.txt')

        user_fullname - the full user name (Example: 'Bill Rideout')

        user email -  Example: 'brideout@haystack.mit.edu'

        user_affiliation - Example: 'MIT'

        format - either "ascii" or "hdf5" or "netCDF4" (if Madrigal 3 site).  Ascii is simple column
        delimited acsii

        startTime - a Matlab time to begin search at. Example:
                    datenum('20-Jan-1998 00:00:00') Time in UT

        endTime - a Matlab time to end search at. Example:
                  datenum('21-Jan-1998 23:59:59') Time in UT

        inst - instrument code (integer).  See 
            http://cedar.openmadrigal.org/instMetadata/
            "Instrument Table" for this list. Examples: 30 for Millstone
            Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
            Scatter Radar

    Optional inputs

        kindats - is an optional array of kindat (kinds of data) codes to accept.
           The default is an empty array, which will accept all kindats.
  
        expName - a case insensitive regular expression that matches the experiment
           name.  Default is zero-length string, which matches all experiment names.

        fileDesc - a case insensitive regular expression that matches the file description.
           Default is zero-length string, which matches all file descriptions.

    Returns: Nothing.

    Affects: Writes downloaded files to outputDir

        

  Example: globalDownload('http://madrigal.haystack.mit.edu/madrigal/', ...
                         '/tmp/download', ...
                         'Bill Rideout', ...
                         'brideout@haystack.mit.edu', ...
                         'MIT', ...
                         'ascii', ...
                         datenum('20-Jan-1998 00:00:00'), ...
                         datenum('21-Jan-1998 23:59:59'), ...
                         30);

  $Id: README 5381 2015-10-05 15:26:27Z brideout $

   ***********************************************************
function [] = globalIsprint(url, ...
    parms, ...
    output, ...
    user_fullname, ...
    user_email, ...
    user_affiliation, ...
    startTime, ...
    endTime, ...
    inst, ...
    format, ...
    filters, ...
    kindats, ...
    expName, ...
    fileDesc)
 globalIsprint is a script to search through the entire Madrigal database
 for appropriate data to print in ascii to a file

    Inputs:

        url - url to homepage of site to be searched (Example: 
              'http://madrigal.haystack.mit.edu/madrigal/'

        parms - a comma delimited string listing the desired Madrigal 
            parameters in mnemonic form.  
            (Example: 'year,month,day,hour,min,sec,gdalt,dte,te').  
            Ascii space-separated data will be returned in the same  
            order as given in this string. See 
            http://cedar.openmadrigal.org/parameterMetadata/
            "Parameter code table" for all possible parameters.

        output - the local file name to store the resulting ascii data, or
                 local directory to store files (one for each Madrigal
                 file) if format set. If a file name, then all data is
                 stored in column-delimited ascii in one file.  If a
                 directory, then one file created per Madrigal file read,
                 and format of output files must be set via format arg.

        user_fullname - the full user name (Example: 'Bill Rideout')

        user email -  Example: 'brideout@haystack.mit.edu'

        user_affiliation - Example: 'MIT'

        startTime - a Matlab time to begin search at. Example:
                    datenum('20-Jan-1998 00:00:00') Time in UT

        endTime - a Matlab time to end search at. Example:
                  datenum('21-Jan-1998 23:59:59') Time in UT

        inst - instrument code (integer).  See 
            http://cedar.openmadrigal.org/instMetadata/
            "Instrument Table" for this list. Examples: 30 for Millstone
            Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
            Scatter Radar

    Optional inputs
        
        format - either empty string (the default) is saving to a single file, or
           'Hdf5', 'netCDF4', or 'ascii' if saving individual files to a
           directory.

        filters - is the optional filters requested in exactly the form given in isprint
         command line (example = 'filter=gdalt,,500 filter=ti,500,1000')
         See:  Documentation -> Administrator's Guide -> Using isprint for file quick looks for details

        kindats - is an optional array of kindat (kinds of data) codes to accept.
           The default is an empty array, which will accept all kindats.
  
        expName - a case insensitive regular expression that matches the experiment
           name.  Default is zero-length string, which matches all experiment names.

        fileDesc - a case insensitive regular expression that matches the file description.
           Default is zero-length string, which matches all file descriptions.

    Returns: Nothing.

    Affects: Writes results to output file

        

  Example: globalIsprint('http://madrigal.haystack.mit.edu/madrigal/', ...
                         'year,month,day,hour,min,sec,gdalt,dte,te', ...
                         '/tmp/isprint.txt', ...
                         'Bill Rideout', ...
                         'brideout@haystack.mit.edu', ...
                         'MIT', ...
                         datenum('20-Jan-1998 00:00:00'), ...
                         datenum('21-Jan-1998 23:59:59'), ...
                         30);

  $Id: README 5381 2015-10-05 15:26:27Z brideout $

Example Matlab code use this API

  % demo program of madmatlab running on a pc or linux

madurl = 'http://madrigal.haystack.mit.edu/madrigal';

cgiurl = getMadrigalCgiUrl(madurl)

'List all instruments, and their latitudes and longitudes:'
instArray = getInstrumentsWeb(cgiurl);
for i = 1:length(instArray)
    [s,errmsg] = sprintf('Instrument: %s, at lat %f and long %f', ...
            instArray(i).name, ...
            instArray(i).latitude, ...
            instArray(i).longitude);
     s
end
% now list all experiments from local Madrigal site with mlh (code 30) in
% 1998 - should be data if default files installed...
startdate = datenum('01/01/1998');
enddate = datenum('12/31/1998');
expArray = getExperimentsWeb(cgiurl, 30, startdate, enddate, 1);
for i = 1:length(expArray)
    [s,errmsg] = sprintf('Experiment name: %s, at url %s and id %i', ...
            expArray(i).name, ...
            expArray(i).url, ...
            expArray(i).id);
     s
end

% now list all files in the first experiment
expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
for i = 1:length(expFileArray)
    [s,errmsg] = sprintf('File name: %s, with kindat %i', ...
            expFileArray(i).name, ...
            expFileArray(i).kindat);
     s
end
% now first 2 parameters in the last file
parmArray = getParametersWeb(cgiurl, expFileArray(end).name)
for i = 1:10
    [s,errmsg] = sprintf('Parameter mnemonic: %s, description "%s" -- isMeasured = %i', ...
            parmArray(i).mnemonic, ...
            parmArray(i).description, ...
            parmArray(i).isMeasured);
     s
end
%  run isprintWeb for that file for first two parameters
parmStr = sprintf('%s,%s', parmArray(1).mnemonic, parmArray(2).mnemonic);
data = isprintWeb(cgiurl, expFileArray(1).name, parmStr, 'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');
% print first 10 records
data(:,:,1:10)

% download that data file in simple format
result = madDownloadFile(cgiurl, expFileArray(1).name, '/tmp/junk.txt', 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT');
'downloaded file to /tmp/junk.txt'

% run globalIsprint, which can gather data from multiple files at once
globalIsprint(madurl, ...
              'year,month,day,hour,min,sec,gdalt,dte,te', ...
              '/tmp/isprint.txt', ...
              'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', ...
              datenum('20-Jan-1998 00:00:00'), datenum('21-Jan-1998 23:59:59'), 30);
'globalIsprint output saved to /tmp/isprint.txt'

% run globalDownload, which can download multiple files at once
globalDownload(madurl, ...
                         '/tmp', ...
                         'Bill Rideout', ...
                         'brideout@haystack.mit.edu', ...
                         'MIT', ...
                         'ascii', ...
                         datenum('20-Jan-1998 00:00:00'), ...
                         datenum('21-Jan-1998 23:59:59'), ...
                         30);
'globalDownload saved files to /tmp'

% madCalculatorWeb runs the Madrigal derivation engine for any point
data = madCalculatorWeb(cgiurl, datenum(1999,2,15,12,30,0), 45,55,5,-170,-150,10,200,200,0,'sdwht,kp');
'madCalculator output'
% print data
data

'The following is an example of searching for non-local experiments'
% 61 is the instrument id of Poker Flat ISR - so this will return an
% experiment not from the Millstone Madrigal site
startdate = datenum('04/01/2008');
enddate = datenum('04/30/2008');
expArray = getExperimentsWeb(cgiurl, 61, startdate, enddate, 0);

% calling this now would fail: expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
% Instead, get the cgiurl of the non-local experiment
if (expArray(1).isLocal == 0)
    cgiurl = getMadrigalCgiUrl(expArray(1).madrigalUrl)
    expArray = getExperimentsWeb(cgiurl, 61, startdate, enddate, 0);
end

% now you can get the files
expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
for i = 1:length(expFileArray)
    [s,errmsg] = sprintf('File name: %s, with kindat %i', ...
            expFileArray(i).name, ...
            expFileArray(i).kindat);
     s
end
Previous: Madrigal web services   Up: Remote access programming tutorial toc   Next: Remote access using python