Getting Started with the Command-Line API

Introduction

The graphical user interface (GUI) is a good way to get started with the GCE Data Toolbox, but learning to use command-line equivalents of GUI programs and lower-level application programmer interface (API) core functions is key to taking full advantage of the toolbox for automating repetitive data management tasks. Some of the most useful toolbox functions are described below, and the full list is available online for reference. Complete syntax and usage notes are available from the MATLAB command window by typing 'help ' before the function name (e.g. help exp_ascii).

Local Data Import

  • imp_ascii(): imports data from a simple delimited text file (minimal header, no missing value codes)
  • imp_filter(): imports data from a text file with missing value codes, large headers
  • imp_matlab(): imports data from MATLAB file or a workspace numeric, character or cell array variable
  • imp_struct(): imports data from a generic MATLAB structure in a file or workspace variable
  • imp_campbell_toa5(), parse_seabird(), imp_csi_array(),...: logger-specific import filters
  • batch_import(): imports all or selected files in a directory using a specified import filter

Network Data Import

  • fetch_usgs(): retrieves instantaneous or daily data for a USGS station
  • fetch_ncdc_ghcnd(): retrieves climate data from NOAA for GHCN stations
  • fetch_noaa_hads(): retrieves data from the NOAA Hydro-meteorological Automated Data System
  • fetch_climdb_data(): retrieves data from ClimDB
  • fetch_eml_data(): retrieves EML-described data tables from the LTER Data Portal or other repository
  • sql2gceds(): retrieves data and attribute metadata from an SQL database via JDBC (MATLAB Database Toolbox required)

Data Export

  • extract(): extracts data arrays from a structure into the base MATLAB workspace
  • exp_ascii(): exports data and metadata to a delimited text file
  • exp_matlab(): exports data and metadata to a MATLAB file as standard arrays or matrices
  • gceds2struct(): generates a scalar or multi-dimensional structure from a GCE Data Structure with fields for each variable and associated flags
  • gceds2eml(): generates a delimited text data file and accompanying Ecological Metadata Language metadata document for archiving
  • gceds2kml(): generates a Google Earth Keyhole Markup Language file from a geo-referenced GCE Data Structure, with record values in popup balloons
  • gceds2html(): generates an HTML table from columns in a GCE Data Structure for web display

Creating and Updating Data Structures

  • newstruct(): creates an empty data structure
  • addcol(): adds a data array to a structure as a new column
  • add_calcexpr(): adds a derived column based on a mathematical expression
  • copycols/copyrows(): copies or re--]orders columns or rows in a data structure, resp.
  • insertrows(): inserts rows into specified columns of a data structure

Manipulating Data Structures

  • update_data(): update values in a data column with a new array, logging value changes
  • deletecols(): deletes one or more columns in a data structure
  • deleterows(): deletes one or more rows in a data structure
  • querydata(): filters data based on a criteria expression (e.g. Salinity<30 & Temp_Air>10)
  • sortdata(): sorts rows based on values in one or more data columns
  • unit_convert(): converts column units based on pre--]defined conversions or an equation
  • pad_date_gaps(): fills in missing date/time records to generate a uniform time--]series
  • shift_datetime(): adjusts dates and times by a specified data part and Interval
  • convert_date_format(): converts date values between various numeric and text formats

Structure Information and Metadata Functions

  • listcols(): lists all columns, including index number, units and data type
  • listhist(): displays the processing history (lineage) as formatted text
  • num_records(): lists the number of data rows in a data structure
  • listmeta(): generates formatted metadata in various styles (text and XML)
  • lookupmeta(): retrieves a metadata entry for a specified section and field name
  • addmeta(): adds or updates metadata based on an array of sections, fields and values

QA/QC Functions

  • dataflag(): updates Q/C flags based on rules in metadata (generally automatic)
  • copyflags(): copies flags from one or more columns to specified dependent columns
  • flags2cols(): instantiates Q/C flags as text columns (e.g. for text file export)
  • cols2flags(): creates Q/C flags from text columns (e.g. importing predefined flags)
  • nullflags(): converts flagged values to NaN/empty (any or selected)
  • cullflags(): deletes rows with flagged values (any or selected)
  • flag_locks(): locks or unlocks flag rules to prevent or restore automatic flagging, resp.

Statistical Summary and Re-sampling Functions

  • aggr_stats(): aggregates data and calculate stats for specified columns
  • aggr_datetime(): aggregates data by date intervals and calculates stats
  • aggr_bindata(): aggregates data and bins by intervals of a specified column
  • viewstats(): displays a statistical summary in a scrollable text box

Plotting Functions

  • plotdata(): generates line/symbol plots
  • plotgroups(): generates line/symbol plots for 1 variable split into groups by value
  • dateplot2html, dateplot2template, dateplot2xml(): generates date interval plots for the web with index pages of captioned thumbnails
  • monthticks(), yearticks(): sets the X-axis date range to even months/years and generates precise monthly or yearly tick marks for time series plots

Data Integration Functions

  • datamerge(): merges (concatenates) two data structures, automatically aligning columns
  • merge_by_date(): merges two data structures by date, trimming overlapping dates
  • multimerge(): merges multiple data structure files
  • joindata(): joins two data structures based on common key values