GCE Data Toolbox Version History

2.96 to 2.97: Bug fixes and enhancements, including:

fixed serious bug in 'ui_qccriteria' that caused edits to QA/QC criteria rules to not be saved in some circumstances
fixed bug in 'inlist' that prevented the use of single match criteria
fixed bugs in 'convert_datatype', 'split_dataseries' and 'ui_editor' caused by recent changes to 'inlist' syntax
added new function 'running_mean' for calculating a running mean along a user-specified window for smoothing data
added new parser for SeaBird Electronics CTD data ('parse_seabird') and custom flag function ('flag_ctd_soak_period') and templates for automating CTD data processing
enhanced 'ui_clearflags' to support removing flag assignments (calling 'clearflags') in addition to removing values or records assigned specified flags (calling 'nullflags' or 'cullflags', resp.)
added new 'Q/C Flag Functions' menu option to data editor for removing flag assignments

2.95 to 2.96: Bug fixes and updates, including:

replaced corrupted version of 'add_anomalies' in 2.95 that interfered with the operation of several import filters and prevented proper documentation of anomalies in data set metadata
updated 'ui_qccriteria' and 'ui_template' to support editing QA/QC criteria in metadata templates using a GUI dialog (in addition to the data grid view)
Note that beginning with version 2.96 the project version of the toolbox is no longer pre-compiled into .p files. However, the code is still copyrighted and permission must be obtained to distribute GCE software outside of the project

2.93 to 2.95: Various minor enhancements, including

added new function 'aggr_movingdatewindow' and 'ui_aggrmovingdate' for generating a smoothed time series data set by statistically summarizing values along a sliding date window
added support for defining common text Q/C rules in 'ui_qccriteria' using uicontrols ('IS', 'IS NOT', 'IN' and 'NOT IN' rules for Q/C checks against string values or lists)
expanded support for filtering on text criteria to 'ui_querybuilder' ('IS', 'IS NOT', 'IN' and 'NOT IN', for parity with 'ui_qccriteria')
'startup.m', 'contents.m', and documentation in 'gce_datatools.mat' now auto-generated during relase builds for formatting consistency and accuracy
standardized help text for all functions for improved parsing during release builds

2.92 to 2.93: Added new function 'normalize_cols' and corresponding GUI dialog 'ui_normalizecols' for normalizing a data set by merging multiple columns to form combined parameter name and parameter value columns with records in other specified columns repeated for each group of parameters (i.e. converts a dataset with repeating groups to a normalized database table). Note that this function complements 'split_dataseries' (and 'ui_splitseries'), which generates individual columns from normalized parameter name, parameter value columns.

2.91 to 2.92: Bug fixes and updates, including:

bug fixes and enhancements to 'nullflags' to improve performance with large data sets and prevent potential removal of flags from incorrect columns when discontinuous column ranges are specified
significant performance improvements in 'dataflag' (2-3x speedup with large data sets)
added support for selective flag removal plus one additional QA/QC rule option to statistical aggregation tool GUIs ('ui_aggrstats', 'ui_aggrdatetime', 'ui_aggrbindata')
added new command-line function for retrieving syntax info for toolbox functions ('parse_gce_syntax'), and used this function to develop a new GUI dialog for building QA/QC criteria based on function calls ('ui_flagfunctions')
updated QA/QC reference documentation

2.90 to 2.91: Bug fixes and updates, including:

bug fix in 'cols2flags' to prevent truncation of pre-existing flags when codes in a text flag column (e.g. data provider-specified qualifiers) are converted to Q/C flags under certain conditions
updates to ClimDB/HydroDB XSLT sheets to accommodate commas in station descriptions that were causing parsing failures
updates to USGS NWIS data parsing function ('parse_usgs') to support retention of individual 'P' (provisional value) flags
added function 'clearflags' to allow specific assigned flags to be cleared (e.g. to suppress display of 'P' flags on plots)
added function 'multi_meta' to support applying multiple, date-dependent metadata templates to data sets to accommodate changes in parameter metadata over time (also supports automatic unit conversions to harmonize attributes)
additional

2.83 to 2.90: Major enhancements to statistical functions and dialogs to support more flexible assignment of Q/C rules for derived data columns ('aggr_stats', 'ui_aggrstats', 'aggr_datetime', 'ui_aggrdatetime', 'aggr_bindata', 'ui_bindata'). Multiple rule sets can now be specified to flag derived values based on number or percentage of flagged and/or missing values in the source data; therefore, multiple thresholds can be defined for flagging statistics derived from incomplete or flagged source data (e.g. flag as Q if >5% missing values, flag as I if >25% missing values, ...).

2.82 to 2.83: Performance enhancements to 'dataflag' and minor modification to 'aggr_data' to prepend 'Total_' instead of 'Sum_' to column names for aggregate totals for compatibility with attribute name conventions used in ClimDB and elsewhere (also made corresponding name mapping changes to several metadata templates and the 'exp_climdb.mat' conversion mappings)

2.81 to 2.82: Minor bugfix in 'imp_matlab' and minor refinements to several GUI forms

2.80 to 2.81: Bug fix in 'imp_datastruct' to correct path handling errors introduced in version 2.80 that prevented merges and joins to data structure files on disk

2.70 to 2.80: Major usability enhancements to the data editor ('ui_editor'), including:

relabeled commands in the File menu to improve consistency with the Search Engine menus and commands used by other common applications
added a 'Window' menu and new commands for arranging multiple editor windows on the console (cascade, tile and stack windows), as well as selecting an open window to view from a menu listing the titles of all open data sets
added new commands to the 'Edit' menu for transforming data columns between data types, casting existing values into the new type using appropriate conversion functions and filters (logging the transform function used in the metadata)
added support for choosing open data sets for merge and join operations, allowing data to be integrated on the fly without first saving files to disk or transferring results to the search engine
added an automated date/time join option, which identifies compatible date/time columns in two time-series data sets to automatically create join conditions, simplifying the process

Note: beginning with version 2.80, MATLAB 6.5 (release 13) or higher will be required due to increased dependence on networking and XML features that are not available in prior versions

2.63c to 2.70: Bug fix in 'ui_harvestclimdb' to properly enable the 'Proceed' button after all fields are filled in when the dialog is run for the first time without a saved settings file. Significant enhancements and new features, including:

added new commands to the Editor window 'Q/C Functions' menu for locking/unlocking automatic recalculation of Q/C flags after data set changes ('ui_editor')
reconfigured options on the Editor window 'Tools' menu to simplify finding dialogs for various data transformation and analysis tasks ('ui_editor')
added a new function and Editor window command for splitting a compound data series on values in a specified column, and then joining the split columns on other specified data columns to create a normalized data table (e.g. splitting records from a merged data set on column 'Site', creating separate 'Salinity' and 'Temperature' columns for each value in 'Site', and joining split 'Salinity' and 'Temperature' columns on date/time columns 'Year','Month','Day','Hour','Minute' to create a tabular data set with separate columns for observations from each site) ('split_dataseries')
added new command-line data mining functions for retrieving National Estuarine Research Reserve data from the NOAA/NWS Hydrometeorological Data System ('fetch_nerr_data', 'parse_nerr_data')
numerous usability improvements, including automatically recalling the most recent settings used on plotting dialogs and automatically prompting users to download a pre-compiled search index for GCE web-based data sets when starting the 'Search Engine' application for the first time

2.63b to 2.63c: Minor bug fixes and refinements to 'imp_ascii' to prevent problems parsing ClimDB information files; changes to Q/C rules for generated date component columns in 'add_datepartcols' to improve flagging logic; enhancements to 'ui_visual_qc' to support closing all plotting dialogs after manually-updating QA/QC flags (i.e. closing dialogs containing cached copies of the outdated structure); updates to USGS metadata templates to remove duplicate attribute metadata references; bug fixes in polygon editing functions in mapping tools 'edit_poly' and 'poly_mgr'

2.63a to 2.63b: Modifications to USGS data parsing function ('parse_usgs') and metadata templates to accommodate additional USGS NWIS data format revisions

2.63 to 2.63a: bug fix in 'dataflag' to correct an issue that prevented clearing residual flags after flagged values were changed or removed under certain circumstances

2.62 to 2.63: Modifications to USGS data processing functions ('parse_usgs', 'fetch_usgs', 'ui_harvestusgs') and metadata templates to accommodate revised format of daily data provided by the USGS NWIS web applications (i.e. flagging of all data values as accepted (_A) or provisional (_P) in addition to other flags such as _e for estimated value, etc). Note that accepted flags are automatically removed from processed data for compatibility with the GCE Data Toolbox Q/C flagging scheme, and provisional value flags are also removed but a coded 'Provisional' column is added to indicate that records contain values flagged as provisional by USGS. Other flags (e.g. 'e' for estimated value) are converted to text flag columns and are automatically merged with rule-based flags assigned by attribute Q/C criteria.

2.61 to 2.62: minor enhancements, including:

updated default ClimDB/HydroDB url in 'ui_harvestclimdb' and 'ui_harvestclimdb.mat' in response to web address changes at OSU
added new function 'consolidate_cols' for consolidating data values in duplicated or otherwise redundant data columns with compatible data types/variable types
added code to 'exp_climdb' to check for and consolidate duplicate data columns (e.g. from USGS harvests) to avoid fatal harvest errors
added new item to 'Geographic Functions' menu in 'ui_editor' for looking up transect distances based on GPS coordinates (coordinates are matched to closest perpendicular point along Thalweg line reference transects stored in 'thalweg_ref.mat' and 'thalweg_bnd.mat')

2.60 to 2.61: minor enhancements to improve compatibility with MATLAB 2006a, plus inclusion of a filled vector map of the entire Georgia Coastline

2.56 to 2.60: major bug fix and enhancements, including:

added new functions and dialogs for harvesting climate and hydrographic data from the LTER All-site ClimDB/HydroDB database ('fetch_climdb_info', 'fetch_climdb_data', 'ui_harvestclimdb')
enhanced 'viewstats' dialog to include row labels in a fixed pane (to prevent scrolling offscreen) and to include a menu option for saving a full statistical summary report to disk (i.e. using ui_statreport)
rearranged fields on the 'ui_harvestusgs' dialog to improve usability
improved parsing of ASCII header fields containing spaces ('parse_header')
bug fix: corrected major flaw in 'colstats' function affecting calculation of standard errors for columns - all users are encouraged to upgrade to 2.60 to address this issue (note: SE calculated by 'aggr_stats', 'aggr_bindata' and 'aggr_datetime' were not affected by this error)

2.55 to 2.56: minor enhancements and documentation updates

2.54 to 2.55: significant enhancements and minor bug fixes, including:

added support in 'ui_editor' (Data Editor) for exporting the current data set directly to the Search Engine search results list; this allows newly imported or user-modified data sets to be integrated with data sets returned from various searches without cumbersome disk operations
added support for auto-loading of the startup screen on last window closure to all GUI dialogs
added options to plotting functions ('plotdata','plotgroups') and corresponding GUI dialogs ('ui_plotdata','ui_plotgroups') to support setting symbol size and fills
added support for axis reposition/resizing to 'plotmenu' (with new function 'plotresize')
added function 'compactcols' (with new menu option in 'ui_editor') for removing columns containing only NaN/null values
added support to 'fetch_usgs' for automatically adding USGS station metadata (coordinates, altitute, drainage area) when the generic metadata template (USGS_Generic) is used; package also includes an updated USGS stations data set ('usgs_stations.mat') containing new data fields Longitude, Latitude, Altitude and Drainage
refactored compiler code to ensure .p files are new than corresponding .m files to prevent warning messages using MATLAB R13+
fixed an index error in 'add_anomalies', which caused the incorrect columns to be listed in the processing history under certain circumstances

2.53 to 2.54: minor enhancements and bug fixes, including:

added a user-selectable option in 'ui_aboutgce' to automatically re-open the startup dialog if both the data editor and search engine dialogs are closed and no other dialogs are open (i.e. last window closed) - this will prevent GUI users from having to interact with the MATLAB command window
added support to 'add_yeardaycol' for generating a YearDay column based on multiple date part columns with or without a Year column present
added the function 'add_gce_cruise' for generating GCE-LTER cruise id columns from date/time information in the data set by looking up cruise date ranges in the data set 'gce_cruise_dates', and added this function to the 'Tools/Specialized' menu in 'ui_editor'
tightened data set validation logic and improved performance in 'gce_valid', and updated 'ui_editor' to prevent loading of invalid data sets from the command line

2.52 to 2.53: significant enhancements, including:

added support in 'dataflag' for selective analysis of q/c flagging criteria, allowing criteria to be evaluated on specified columns only to improve efficiency when analyzing large data sets; also added support for this feature in 'addcol' and 'ui_editor' to prevent unnecessary flag re-analysis when adding columns or editing flag criteria, resp.
added support in 'add_anomalies' for summarizing flagged and missing values without grouping by date for use in non-time-series data sets; also added support for this option in 'ui_editor' and 'ui_visualqc'
added support in 'search_index' and 'search_index_r13' for instantiating q/c flag columns prior to indexing (used to generate more comprehensive data set summaries on the GCE Data Portal site)
added controls to 'ui_editmetadata' for repositioning, adding and deleting individual metadata fields, and improved error handling
improved performance and consistency of q/c flagging functions 'flag_valuechange', 'flag_percentchange' and 'flag_nsigma'
improved layout of automatically-generated help text in 'help_flagfnc'

2.51 to 2.51: bug fix in 'ui_qcflags' to prevent problems entering custom QA/QC flag criteria; enhancements to 'ui_editor' and 'ui_search_data' to automatically reload the startup dialog upon exit if no other windows are open (i.e. so users will not have to use the command line or restart MATLAB to restore the startup dialog)

2.5 to 2.51: minor bug fixes in 'encode_strings', 'compass_rose' and 'mapmenu'

2.41 to 2.5: major enhancements, including:

optimized code for extracting metadata field contents ('lookupmeta'), resulting in improved performance for operations involving extensive metadata lookups (e.g. 2-3x faster performance when merging multiple data sets using 'multimerge', and 2x faster data set indexing using 'search_index')
added support for directly performing "joins" between data sets in the Search Engine dialog search results list ('ui_search_data')
added support to the Search Engine dialog ('ui_search_data') for automatically caching data sets resulting from "merge" or "join" operations and adding them to the search results list for additional analyses
added menu options for setting figure, axis and legend colors on data plots ('plotmenu')
bug fix in 'exp_ascii' to remove extra quote characters around column headings in statistics reports exported in comma-separated value format
disabled figure resizing in 'ui_datagrid' running under MATLAB 7.x to prevent GUI problems and hangs
minor bug fixes in 'listmeta', 'ui_aggrstats', 'ui_aggrdatetime', 'ui_copyflags' to address syntax errors when running under MATLAB 7

2.41 to 2.41: minor bug fixes; addition of function 'interp_missing' for interpolation of missing values in time series data with automatic flagging of interpolated values as 'E' = estimated, and user-specified limits on how many consecutive values to interpolate; addition of commands to the data set editor ('ui_editor') 'Edit' menu for deletion of empty records (i.e. deletion of records with all columns empty or all data columns empty)

2.40 to 2.41: improvements to 'encodestrings' to support reconciling integer code definitions with prior code definitions in the metadata when string columns are encoded as unique integers (either when run manually or automatically when exporting data sets in MATLAB matrix format); also added options to 'ui_editor' to support updating or appending code definitions when encoding string columns

2.33 to 2.40: bug fixes and minor enhancements, including:

bug fix in 'ui_search_data' to correct inappropriate nulling of flagged values when the option to retain flagged values is selected
bug fix in 'match_sites' to correctly handle NaN values in geographic coordinate columns
added support for logging all data value updates in the metadata to 'update_data'
added support for appending exported text files to 'exp_ascii' (and deprecated the 'exp_ascii_append' function which served the same purpose)
improved the consistency of the "csv" format option in 'exp_ascii' by wrapping all text values, flag characters, custom missing value codes, and header text in double quotes (i.e. not just those containing spaces or commas)
improved the consistency and program efficiency of support functions 'copycols', 'copyrows', 'deletecols', 'deleterows', and 'addcol'

2.32 to 2.33: significant enhancements, including improved support for log-scale axes in 'plotbuttons' and 'ui_axislabels', support for showing/hiding and setting colored borders on legends in 'plotmenu', and automatic coordinate system re-projection for map polygon data in 'poly_mgr'

2.31 to 2.32: minor enhancements to several functions, including:

improved support for parsing of formatted geographic coordinates in documentation metadata (i.e. from the Study/Coordinates field) during indexing when data sets do not contain geo-reference columns or recognized site/station columns in the data table (i.e. when GCE geographic lookups are not possible) ('search_index', 'search_index_r13')
improved consistency of PNG thumbnail sizes generated for different size figures in 'dateplot2html' and 'dateplot2template'
improved the display of long axis title labels in 'ui_axislimits'
added support for path caching to 'mapmenu' to simplify loading, saving or exporting multiple maps in different directories
added support for exporting figures and maps as 600 dpi PNG, TIFF and JPG images ('exportfig', 'plotmenu', 'mapmenu')

2.30 to 2.31: bug fix in 'ui_copyflags' to properly update the cached data structure after each flag copy operation when the 'close dialog after exporting the results' option is unchecked

2.21 to 2.30: minor bug fixes and major enhancements to mapping tools, including:

bug fix in 'querydata' to prevent problems parsing queries containing string search terms that match data set column names
bug fix in 'parse_usgs' to properly close temporary file after HTML code found and parsing halted
major enhancements to 'ui_mapdata':
- added support for plotting symbols, labels or patches on open map figures (i.e. allowing generation of multiple data series/labels from 1 or more data sets)
- added separate dialog options for plotting formatted data values as labels with optional marker symbols, allowing label font characteristics and colors to be specified separately from marker symbols and colors
- added support for deleting previously plotted symbols or labels from maps, and repositioning labels after zoom operations to adjust spacing between corresponding markers and labels
added new functions to the mapping toolbox for creating CTD station data sets at specified station intervals ('find_stations' and 'ctd2dataset') for custom plotting of CTD stations on maps using 'ui_mapdata'
added a button on the main GCE Data Toolbox startup dialog ('ui_aboutgce') and menu items on the Data Editor ('ui_editor') and Data Search Engine ('ui_search_data') dialogs for loading maps using the mapping toolbox independently of plotting data on maps

2.20 to 2.21: enhancements to data plotting functions and dialogs, including:

updated 'plotdata' and 'plotgroups' to improve handling of empty data columns to avoid generation of empty plots or empty data series, and to improve handling of large title strings (i.e. now restricted to 3 lines of text)
updated 'ui_plotdata', 'ui_plotgroups' and 'ui_mapdata' to support comprehensive data restriction queries instead of just single column restrictions; full query statements can now be entered in plotting dialogs to filter values prior to plotting, and query statements can be entered /edited manually or by using the graphical query builder dialog ('ui_querybuilder')
updated 'ui_querybuilder' to support parsing query statements to populate dialog controls and to return edited query statements to calling functions instead of executing the query and returning a data set (i.e. to support data restriction query building/editing for plotting dialogs)

2.10 to 2.20: updated 'ui_search_data' and 'multimerge' to restore support for Q/C flag removal prior to data set merging (broken in 2.01); significant enhancements to the GCE Data Search Engine components, including:

updated 'search_index', 'search_data' and 'ui_search_data' to load configuration information stored in a common file ('search_data.mat') to support end-user extension and customization of metadata content that is indexed and searched (e.g. add/edit metadata fields and field mappings, change text comparison options)
added support for simultaneously searching the contents of all indexed metadata fields (i.e. 'Any Text Contains') to 'search_data' and 'ui_search_data'
updated the metadata field selector in 'ui_search_data' to display the text comparison type along with the metadata field (e.g. 'Abstract Text Contains' or 'Accession Begins With')

2.06 to 2.10: minor bugfix in 'get_studydates' to handle text date conversion issues in MATLAB 5.3; significant enhancements to metadata management/display functions, including:

enhancements to 'ui_editmetadata' to support overlaying or overwriting metadata with only selected metadata fields from an existing data structure (i.e. instead of all fields) for fine-tuning metadata content re-use (brings 'ui_editmetadata' in line with metadata importing in 'ui_editor')
enhanced 'listmeta' by simplifying the metadata style definition syntax, adding support for metadata field placeholders in text expressions and thereby eliminating the requirement for MATLAB character array syntax when referencing multiple metadata fields (Note: unfortunately, this change also eliminates support for MATLAB text functions in style definitions for auto-generating metadata, e.g. 'datestr(now)')
updated all styles in 'metastyles.mat' to support the new 'listmeta' syntax
updated 'listhist' to improve performance when formatting large processing history arrays

2.05 to 2.06: NOTE: Version 2.06 is the first public release of version 2.x and contains various bug fixes and enhancements, including:

bug fix in 'fill_date_gaps' to prevent automatic removal of records with duplicate date/time values (specific input argument now required for duplicate removal); corresponding change to 'ui_editor' to prompt for duplicate removal if duplicate dates/times found
bug fixes and enhancements in 'flag_valuechange' and 'flag_percentchange', including new option to recursively analyze flag conditions to minimize flagging of normal-range values following flagged outliers
bug fixes in 'ui_gce_register' and 'ui_search_data' to prevent inappropriately storing user registration information in the default workspace file when 'Store my registration information on this computer' was not checked (registration is still stored in manually created workspace files)
enhancements to metadata and text viewer dialogs to support figure resizing ('ui_viewtext', 'ui_viewmeta'), text file export ('ui_viewtext', 'ui_viewmeta', 'ui_viewdocs') and text file import ('ui_viewtext') -- users can now save formatted metadata, data processing histories and toolbox documentation to disk for their own use outside of MATLAB
enhancements to 'add_anomalies' to support documenting flagged and missing values only in user-selected data columns (now set as default when function called from 'ui_editor')
Note: the public version of the toolbox (gce_datatools_206public.zip) has some GCE-specific import filters, batch scripts and metadata templates removed, but is otherwise complete

2.04 to 2.05: minor bug fix in 'joindata'; various program enhancements and additions, including:

significant performance improvements in the GCE Data Search Engine ('search_data') due to improved query optimization, resulting in up to 10-fold faster searches when numerous criteria are specified
major speed improvement in refreshing search indices due to improvements in file modification checking that minimize loading files into memory
enhanced the GCE Data Search engine dialog ('ui_search_data') to support viewing the contents of each indexed path (i.e. file name, modification date, title)
added a GUI dialog for exporting climate and hydrographic data sets as text files in the LTER ClimDB/HydroDB format, as well as viewing/editing corresponding attribute name and unit mappings ('ui_expclimdb', based on 'exp_climdb')
improvements to 'flag_valuechange' and 'flag_percentchange' routines for flagging value changes relative to prior values or running averages

2.03 to 2.04: enhancements to 'fill_date_gaps' to support time series data sets with small numbers of duplicated values, and enhancements to 'search_index' and 'search_data' to support indexing and searching on column units and variable types in addition to names and also searching on column name + units

2.02 to 2.03: major bug fix in 'add_datepartcols' to correct generation of hour=24 values after rounded seconds propagated to minutes and hours, causing difficulties with date/time aggregation due to mixed hour=0 and hour=24 values in a single data set (seconds are now truncated when not included in the output columns); added logic to 'aggr_datetime' to automatically regenerate date component columns when a serial date column is present to avoid this issue with legacy data sets containing mixed hour=24 and hour=0 values.

2.01 to 2.02: minor bug fixes in 'ui_search_data' and 'mergemeta'

2.00 to 2.01: enhancements to data integration functions and the Data Search Engine GUI application, including:

improved merging of data release dates and keywords in metadata ('mergemeta')
added support for automatic generation of a 'DataSetName' column, listing the name of each original data set when multiple data sets are merged/concatenated ('multimerge'), and added corresponding options to GUI dialogs supporting this feature ('ui_multimerge', 'ui_search_data')
added support for automatically saving the workspace on exiting the Data Search Engine dialog and automatically reloading the prior workspace on program re-start ('ui_search_data')

2.00beta to 2.00: minor enhancements to QA/QC flag functions and major enhancements to data integration tools, including:

improved handling of multiple flag conditions in QA/QC flag removal functions ('nullflags' and 'cullflags'), so all flags cleared when values are cleared using selective criteria
changed keyword search logic in 'search_data' from partial string match to beginning string match and added code to prevent reverse matches when keywords being searched are shorter than the search term (i.e. to prevent spurious results such as matching 'par' on searches for 'Spartina' or vice versa)
added new multiple dataset merge function ('multimerge') that supports merging (i.e. contatenating) data sets based on a file list and structure name list to support merging structures stored in multiple directories as well as structures contained in compound files; also supports 3 merge types (listed order, study date order, non-overlapping study date order) and various metadata merge options
added support for the new 'multimerge' function options to 'ui_multimerge' and 'ui_search_data' to provide more flexible data integration options from GUI programs
changed default startup mode of 'ui_search_data' to initially hide the Query History pane to simplify the form

1.988b to 2.00beta: Major new release, featuring significant enhancements to functions for merging multiple structures ('ui_multimerge', 'datamerge', 'merge_by_date'), improved support for plotting in MATLAB R14/v7 ('plotdata', 'plotgroups'), better standardization of command-line help, and major new functions and GUI dialogs, including:

added full support for indexing and searching GCE Data Structures stored in any number of local directories and files, as well as data sets stored on the GCE web server (i.e. publicly-accessible data catalog files and data portal files); new functions include: 'search_index' for indexing data sets, 'search_data' for parsing query statements and performing index searches, and support functions 'recurse_files' to recursively search directory trees to match files and 'split_codes' to split delimited lists of coded values to produce hash tables
added new mapping tools to support bounding box creation and site lookups ('get_bbox' and 'bbox2sites'), and now include the GCE Mapping Tools as an integral part of the GCE Data Toolbox
added a major new GUI dialog (GCE Search Engine, 'ui_search_data') for building and managing search indices, performing queries, managing cumulative search results, and performing analyses and batch operations on data sets in the search results list (e.g. view metadata, view/edit data set, create line/scatter or map plots, batch copy or export data sets in specified text or MATLAB formats with user-selectable metadata formats, header formats, and QA/QC flag handling); the search index, query fields, query history, result set and program options can all be saved and reloaded, allowing users to create any number of customized search sessions for mining data stored in GCE Data Structures; support is also built in for retrieving up-to-date indices of all publicly-accessible GCE data sets (both accessioned data in the GCE Data Catalog and ancillary data in the GCE Data Portal) and automatically logging in, retrieving, and caching data from the GCE web site, allowing users to seamlessly query and analyze both local and GCE data holdings.
added support for multiple column deletions, selective export of data columns to the base MATLAB workspace, and selective import of metadata fields from data structures or standard metadata templates to the Data Editor (ui_editor)

1.988 to 1.988b: corrects a problem with the startup script ('startup.m') which was inadvertently compiled in 1.988; functionality of the data editor ('ui_datagrid') improved by adding support for resizing the grid when the figure is resized, saving layout settings (alignment option, grid size, logging option), and viewing records with duplicated values in all columns or non-data columns (i.e. for inspection and editing/removal of duplicates); added new function 'dupe_index' for computing indices of duplicate rows without sorting the structure

1.987 to 1.988: minor enhancements to 'ui_plotdata' and 'ui_plotgroups' adding support for double-clicking to select plot columns, minor bug fixes in 'nullflags' and 'flags2cols', updated USGS metadata templates and GCE geographic information tables, and added new function 'dateplot2template' (similar to dateplot2html, except adds support for adding thumbnail plot table to a pre-existing html template for easier integration with site web pages)

1.986 to 1.987: minor enhancements to 'ui_editor', 'ui_visualqc', 'aggr_stats', 'aggr_bindata'; several new functions and GUI dialogs added, including:

added new functions for automatically summarizing QA/QC-flagged and missing values by date as data anomalies in the metadata ('add_anomalies' and 'daterange2str'), with menu/GUI support added to 'ui_editor' and 'ui_visualqc'
added new dedicated functions for performing date/time aggregation at specified intervals and automatically pre-pending appropriate interval prefixes (i.e. 'Yearly', 'Monthly', 'Daily', 'Hourly') to names and descriptions of derived statistical attributes and to data set titles ('aggr_datetime' and 'ui_aggrdatetime'; based on 'aggr_stats')

1.985 to 1.986: minor bug fixes in 'unitconv', 'ui_datagrid'; minor enhancements to 'nullflags', 'cullflags', 'plotgroups', 'ui_plotsgroups'

1.984 to 1.985: minor bug fixes in 'ui_datagrid' and 'ui_dataflag' to correct problems returning manually-assigned QA/QC flag results to the editor; minor bug fix in 'mergemeta' to correct errors meshing metadata containing different coded lists of QA/QC flags

1.983 to 1.984: minor bug fix in several GUI functions to support dependent file checking when .p files are present in addition to .m files (i.e. for proper functionality when running the compiled toolbox programs)

1.982 to 1.983: minor enhancements to 'ui_editor', including addition of new date/time functions for adding a year day column calculated from a serial date/time column or string date and time columns, addition of a new function 'plotwidgets' to add GCE plot menus, plot labels and toolbars to any MATLAB figure (i.e. provides stand-alone functionality for plots not generated by the GCE Data Toolbox).

1.981 to 1.982: enhancements to 'dateplots' to support weekly and daily plots; addition of new function 'dateplot2html' and corresponding GUI dialog 'ui_dateplots_html' for generating complete HTML web pages along with date interval images for creation of multi-column time series image displays on the web, with support for automatic generation of thumbnail images hyperlinked to larger images

1.98 to 1.981: major bug fix in 'datemerge' to address potential column ordering problems, minor bug fixes in 'cols2flags'; added new function 'dateplots' for generating multiple time series plots at a specified interval in various bitmap and vector image formats (note: supercedes 'monthplotsfig' and 'yearplotsfig' providing more graphics format and resolution options and adding support for decadal plots)

1.973 to 1.98: minor bug fix in 'aggr_sums' to correct mishandling of flagged columns, several new functions and program enhancements, including:

addition of new function 'negative2zero' for setting negative values in specified columns to 0, and logging the changes to the data anomalies section of the metadata (useful for zeroing out dark noise in light sensors prior to summation and to avoid excessive QA/QC flags)
addition of new function 'insertrows' for inserting data rows into specified columns of an existing data structure and padding corresponding rows in omitted columns with empty values
enhancements to 'ui_editor' and 'edit_importfilters' to support nesting of custom import filters in menus
enhancements to 'ui_datagrid' to support user-customized alignment of cell contents for editing
enhancements to 'ui_importfilter' to support automatic generation of custom text file import functions (.m files) based on user-specified import parameters for a text file, allowing simple generation of MATLAB programs for subsequent use and customization; new functions are also automatically added to 'imp_filters.mat' for display on the editor 'Import Data' menu
enhancements to 'imp_ascii' to improve automatic parsing of simple delimited ASCII files
enhancements to 'csi2struct' to support a wider variety of CSI data array formats

1.972 to 1.973: major bug fix in 'datamerge' to prevent loss of manually-assigned QA/QC flags when a structure is appended to another structure without any manually-assigned flags in corresponding columns (i.e. no 'manual' criteria); minor bug fix in 'ui_plotgroups' to properly pass selected flag display options to 'plotgroups'; several program enhancements, including:

addition of new GUI dialog ('ui_copyflags') for propagating composite QA/QC flags from one or more columns to one or more dependent/calculated columns, to augment or replace existing flags
modified 'add_calcexpr' to support automatically adding calculation descriptions to the Data/Calculations metadata field
added support for date/time string axis limits in 'ui_axislimits'
minor enhancements to 'fetch_usgs' and 'parse_usgs' to accomodate additional file format and flag variations in USGS data files
minor enhancements to 'listcols', 'flags2cols'

1.971 to 1.972: minor bug fix in 'dec_places'; improved handling of automatic numeric assignments in 'imp_filter' and 'imp_ascii' for data files without metadata templates; addition of URL parameter to 'fetch_usgs' and corresponding input field in 'ui_harvestusgs' to accommodate USGS URL changes; addition of a new function ('update_data') to support command-line updates to data column contents with automatic logging of data changes to the metadata (similar functionality to 'ui_datagrid' for non-GUI applications)

1.97 to 1.971: minor bug fix in 'exp_matlab' to accomodate numeric column names in structures; enhancements to 'ui_unitconv' to limit standard unit conversions to only those matching the original column units; addition of GUI dialog for harvesting real-time, daily or finalized data from the USGS web server

1.962 to 1.97: minor bug fixes in 'ui_editor', 'flags2cols', 'ui_datagrid', 'imp_ascii'; minor display enhancements and cosmetic changes to GUI dialogs; several major program enhancements, including:

addition of custom ASCII file import dialog ('ui_importfilter') to ease parsing of files with non-standard headers and missing value codes (includes a file viewer with automatic format string and column title parsing functions)
addition of data anomalies metadata field to visual QA/QC ('ui_visualqc') for easier documentation of manual flagging operations
added support for automatic calculation of flagged and missing value total columns to grouped and binned statistics functions ('aggr_stats', 'aggr_bindata') and GUI dialogs ('ui_aggrstats', 'ui_bindata'), with support for flagging criteria based on these totals -- this allows flags to be propagated to columns in derivative data sets
inclusion of new functions for harvesting data from the USGS web site ('fetch_usgs' - requires MATLAB 6.5/R13), parsing data in Campbell Scientific array-based data loggers ('csi2struct'), and formatting climate data for submission to ClimDB/HydroDB LTER databases ('exp_climdb')

1.961 to 1.962: minor bug fix in 'imp_ascii' to prevent column numerical type descriptors in ASCII file headers from been overridden by automatic value inspection

1.96 to 1.961: major bug fix in 'joindata' involving joins on multiple unsorted key fields, minor bug fixes and enhancements to 'plotdata' and 'plotgroups' to improve plots based on string date values, improvements in 'edit_unitconv', updates to cached geographic information stored in 'gce_locations.mat' and 'gce_coords_all.mat' to reflect new site designations, additional flag-handling options added to 'exp_matlab', corrections to Q/C flag references in documentation

1.951 to 1.96: bug fixes and keyboard accelerators (hot keys) added to the data structure editor (ui_editor); added a new generic import filter for USGS real-time data from any station in the US (parse_usgs); added major new geographic lookup functions, including

function for adding a study site column based on matching values in geographic coordinate columns to site polygons in the GCE geographic database (add_studysites)
function for adding geographic coordinates based on matching values in station or location code columns to entries in the GCE geographic database (add_stationcoords)
function for importing study site metadata from the GCE geographic database to support automatic lookups based on site code, station or location code, or geographic coordinate columns (add_sitemetadata)

1.95 to 1.951: bug fixes and major speed enhancements in 'joindata', enhancements in 'ui_joindata', minor changes to color selection order in 'ui_plotdata'

1.902 to 1.95: major additional features added, including:

added support for selective removal of individual values or entire rows based on specified QA/QC flags present in specified data columns, with optional logging to the 'Data|Anomalies' metadata field to aid in documentation of quality control operations (nullflags, cullflags, ui_clearflags)
added support for propagation of QA/QC flags assigned to values in one or more columns to values in other (dependent) data columns (copyflags)
added support for automatic calculation of study date metadata descriptors based on date/time information in the data set (add_studydates)
added an option to rename data editor windows based on the data set title or user-specified strings to prevent confusion when multiple editor windows are opened simultaneously (ui_editor)
added a function to automatically synchronize path caches of all open editor windows to minimize directory changes in file dialogs (syncpath)
enhanced ASCII export and statistics reports dialog to support selective flag removal and not prompt for file names before opening (ui_exportasc, ui_statreport)
minor bug fixes

1.901 to 1.902: syntax error corrected in 'struct2xml'

1.90 to 1.901: minor bug fix in 'colstats', added support for new missing value codes used by USGS to 'imp_usgshudson', 'parse_usgshudson' and 'imp_usgsdoctortown'

1.823 to 1.90: major additional features added, including functions for performing statistical analysis on binned data with optional pre-aggregation by one or more grouping columns (aggr_bindata and ui_bindata), and retrieving the top and bottom values from a vertical profile data set (e.g. CTD casts) based on a depth or pressure column after pre-aggregation by one or more columns (aggr_topbottom and ui_topbottom); minor changes to grouped statistics (aggr_stats) and column statistics (colstats) functions to prevent calculation of dispersion statistics (SD, SE) on groups containing a single data value; minor changes to Tools menu arrangement in the structure editor (ui_editor)

1.822 to 1.823: enhancements to 'add_datecol', 'add_datepartcols', and unit conversion table to support automatic time zone recognition/conversion based on datetime column units and descriptions, minor enhancements to 'parse_37sm' and 'imp_gcesonde' to support Sea-Term file type variants, minor changes to 'parse_usgshudson', 'imp_usgsdoctortown', and 'csi2struct' to support datetime conversion to GMT

1.821 to 1.822: minor bug fix to prevent errors with non-auto-scaled plots in 'plotdata', enhancements to 'parse_37sm' to support SBE file format variants, dedicated CT's, and addition of 'Pump' column to indicate whether the sonde was equipped with a pumped flow cell

1.82 to 1.821: added support for performing visual QA/QC flagging on auto-scaled plots to 'ui_visualqc'; added support for reversing plot axes directions to 'ui_axislimits' (called by 'plotbuttons')

1.81 to 1.82: significant QA/QC-related enhancements, including addition of support for multi-column flag criteria (dataflag), new functions for flagging dissolved oxygen concentrations based on deviation from temperature and salinity-dependent oxygen saturation (flag_o2saturation), support for conversion of text columns to QA/QC flags for corresponding data columns to support synchronization of manually-edited flags converted to text columns back to the flag array of the corresponding data column (cols2flags), and support for data value editing added to manual QA/QC utility (ui_dataflag)

1.803 to 1.81: minor bug fixes and enhancements in 'textfile2cell', 'ui_axislimits', 'ui_flagdefs', 'ui_qcflags', 'ui_aggrstats', 'ui_editor', 'ui_exportasc'; additional features added to 'ui_template' and 'ui_editmetadata' to facilitate importing metadata and descriptors from existing structures to create templates; added data restriction option to 'ui_mapdata'; added support for automatic conversion of text date strings (e.g. 12/1/2002) to numerical Julian date when entering date restrictions on datetime columns in 'ui_plotdata', 'ui_plotgroups', and 'ui_mapdata'

1.802 to 1.803: additional minor bug fixes in 'ui_editor', 'imp_datastruct', 'ui_flagdefs', 'ui_qcflags', 'ui_visualqc', 'ui_multimerge'; enhancements to 'add_datepartcols'

1.801 to 1.802: minor bug fixes in 'ui_editor', 'imp_ascii', 'ui_title'

1.80 to 1.801: minor bug fixes in 'ui_plotdata' and 'ui_datagrid'

1.713 to 1.80: major QA/QC-related enhancements, including addition of GUI flag criteria ('ui_qcflag') and flag code definition ('ui_flagdefs') editors and support for visual QA/QC flagging on multi-parameter and grouped scatter/line plots (flag values by left clicking/dragging with the mouse, clear flags by right clicking/dragging); also added menu accelerators (control key support) to 'ui_datagrid', enhanced the plot toolbar to add support for manual axis limit selection and custom date/time axis formats, and added a 'GCETools' menu item with support for exporting figures in various graphical formats and resolutions (without the toolbar controls visible).

1.712 to 1.713: bug fixes in 'joindata' and 'flags2cols', improved path caching in structure editor 'ui_editor', major enhancements to date axis formatting function 'dateaxis', allowing user-customized date/time formats for plots, added a new import filter for SINERR hydrographic data

1.711 to 1.712: minor enhancements to 'querydata'

1.71 to 1.711: major speed enhancements and new 'forced join' option added to join function and dialog ('joindata' and 'ui_joindata'), added support all metadata styles to matlab export function 'exp_matlab', and updates to metadata styles are automatically synchronized in all editor menus

1.70 to 1.71: minor bug fixes and enhancements to data editor (ui_datagrid), improved text field sorting, addition of a GUI editor dialog for adding and editing metadata style definitions (ui_metastyle)

1.632 to 1.70: several major enhancements to the toolbox --

Unit conversions are now applied to the QA/QC flag criteria along with the column data to prevent spurious flagging of data values after conversion (note: custom function-based criteria are not converted to prevent inappropriate parameter modifications)
Data import filters and metadata templates used by the toolbox are now fully modular, allowing users to define their own custom filters or templates that will be listed in editor menus alongside the default filters and templates. GUI editors are provided for managing filters and templates (listed under the 'Misc' menu), and sample custom import filters are available on request.

1.631 to 1.632: minor bug fixes, enhancements to ui_datagrid, ui_template to support creation of new metadata templates

1.63 to 1.631: minor bug fixes in editor menu commands (ui_editor)

1.62 to 1.63: bug fixes, enhancements to multiple-y plot dialog (ui_dataplot), addition of new function for importing ASCII data under format control with missing value substitution (imp_filter)

1.61 to 1.62: *major* speed improvements (5-10x) in delimited ASCII export for structures containing string columns or formats specifying missing value substitutions (exp_ascii), enhancements to unit conversion dialog (ui_unitconv)

1.60 to 1.61: bug fixes in multiple-merge dialog (ui_multimerge), enhancements to multiple-y plotting dialog (ui_plotdata) and plot-by-month batch function (monthplots)

1.50 to 1.60: improved support for metadata templates in the editor by allowing templates to be applied by name and edited and created using a GUI dialog (ui_template), added generic batch import function (batch_import)

1.40 to 1.50: added support for manually editing QA/QC flags for a column and setting numerical precision according to display precision setting, improved merge and join functions, miscellaneous bug fixes and enhancements

Function List

GCE Data Toolbox