Home > Informatics > Software > GCE Toolbox > Download > Version History

GCE Data Toolbox Version History

3.9.7b to 3.9.8: Minor enhancements and revisions, including:

  • Added options for merging data sets without merging documentation metadata to simplify adding data records to ongoing collections
  • Added an option to check for sparse variables when assigning Q/C rules during data aggregation to avoid excessive flagging
  • Revised the parse_usgs.m function to accommodate recent changes in the USGS NWIS for inline value flags
  • Revised interp_missing.m to correct an issue where singleton missing values were not being filled when large gaps were present in the data set
  • Revised the drift correction GUI (ui_correct_drift.m) to default to the starting and ending times of the data series
  • Revised correct_drift.m to prevent NaN values from being flagged as corrected
  • Minor fixes to plot resizing functions to accommodate syntax changes in recent versions of MATLAB

3.9.7 to 3.9.7b: Minor enhancements and revisions, including:

  • Added support for authenticated SMTP when sending harvester emails
  • Added support for code comments in Q/C criteria without breaking the parser
  • Revised ClimDB publishing workflow to default to HTTPS for compatibility with the NCO server
  • Added a sample data harvester for a NCAR CHORDS web service
  • Revised the Data Harvester Dashboard XSL stylesheets to include Q/C rules in a scrolling div
  • Fixed a bug that resulted in unspecified attribute numeric type when importing data using an incomplete metadata template
  • Updated the example data submission spreadsheet to include auto-complete fields in use at GCE

3.9.6b to 3.9.7: Enhancements and minor revisions, including:

  • Added functions and GUI dialog for shifting date/time values forward or backward by a specified date part and increment
  • Added support to the Date/Time Interval Statistics dialog for shifting date/time values prior to aggregation to account for data loggers that record time stamps that correspond with the end of the sampling period
  • Added functions for generating precise monthly and yearly ticks on time series plots
  • Added a new command-line functions for generating plots optimized for long-term data visualization (time_series_barplot.m, time_series_line_plot.m, time_series_range_plot.m)
  • Added an option for converting NaN (missing) values to 0 in the Data Set Editor
  • Added an option to the Data Set Editor for exporting data to the MATLAB workspace as a standard struct variable
  • Added a function for adjusting date/time values that differ slightly from the set interval (core/adjust_time_interval.m)
  • Added a Data Set Report tool to the Search Engine application
  • Updated gui/ui_joindata.m to support multiple selections for output variables
  • Updated web-based data retrieval functions to support the HTTPS protocol to comply with USGS, NOAA and LTER server configurations
  • Added support for nvarchar fields in database/sql2gceds.m
  • Fixed a bug in the Data Set Editor that caused column selections to be ignored when cloning data sets
  • Fixed a bug in the Data Set Editor that caused the documentation only and selected documentation options to be ignore when importing metadata
  • Fixed a bug that could lead to an "unspecified" Q/C flag code when a leading space is present in the flag column
  • Fixed a bug that caused the query builder (gui/ui_querybuilder.m) to throw an error when a space-delimited array is specified for "IN ARRAY" or "NOT IN ARRAY"
  • Additional minor fixes for syntax changes between MATLAB R2013b and R2016b

3.9.5 to 3.9.6: Major bug fixes and minor enhancements, including:

  • Corrected major issue handling time zone shifts within individual USGS NWIS rdb data files that could lead to unexpected data offsets (/parsers/parse_usgs.m)
  • Corrected issues with figure instantiation that lead to warnings and errors in recent versions of MATLAB when WindowStyle?='docked' is set by default
  • Corrected an issue where the wrong metadata field could be revised when calling /core/update_attributes.m
  • Revised the NOAA GHCND import filter to skip calculation of mean temperature from min and max when mean is already reported for a station
  • Revised /core/multimerge.m to accept any shaped array of filenames rather than just column-oriented
  • Added an offline station check to the Data Harvester Dashboard XSL style sheets that displays a prominent warning on web pages when no data have been retrieved for over 1 day (threshold can be revised by editing the XSL)
  • Added an option to the Data Set Editor to export data sets as MATLAB table objects for using associated methods
  • Other minor bugfixes

3.9.4b to 3.9.5: Bug fixes and minor enhancements, including:

  • Revised floating-point precision handling for numeric serial dates in pad_date_gaps.m to avoid generating uneven time stamps when padding large gaps
  • Prevented stripping of tilde (~) characters in Q/C rules when parsing headers of ASCII file to avoid invalidation of NOT criteria
  • Corrected an issue that prevented importing units from certain Campbell logger TOA5 files
  • Corrected an issue that prevented display of import filters in the Batch Import GUI when the subheading description was blank
  • Revised the pointer options in plot menus and the drift correction GUI for compatibility with MATLAB 2014b and higher that do not support full-screen cursors
  • Revised the batch_import.m function to automatically exclude directories (including . and ..) when a *.* wildcard specifier is used
  • Added a function (load_gce_data.m) to simplify loading file-based data structures into workflows
  • Added more descriptive error reporting to pad_date_gaps.m to facilitate debugging workflow issues
  • Added support for symbol flags in the harvest_dashboard.m and data harvester utilities to speed plotting when many Q/C flags assigned
  • Added a data tooltip option to the GCETools menu on data plots for identifying specific values graphically
  • Minor compatibility improvements for MATLAB R2016a

3.9.4 to 3.9.4b: Bug fixes, including:

  • Corrected bugs in EML package export functions that caused the file header option to be ignored and tables to be appended to existing files of the same name
  • Corrected a bug in the custom ASCII file import dialog that generated an extra variable token if leading spaces are present in the file
  • Minor compatibility changes for MATLAB R2015B

3.9.3 to 3.9.4: Enhancements and bug fixes, including:

  • Added functions and a GUI dialog for generating contour plots for vertical profile data sets containing distance and depth variables
  • Added a provisional import filter for SeapHOx instrument data (parse_seaphox.m)
  • Added an option to only import documentation metadata from a template or data structure to avoid overwriting attribute metadata
  • Revised the startup screen to place the 'Dataset Editor' button first and replace the 'Mapping' button with 'Templates' to open the metadata template editor application
  • Revised the data set editor (gui/ui_editor.m) to remove manual Q/C locks from Q/C criteria when saving metadata as a template
  • Revised the metadata template manager (gui/ui_template.m) to match attributes by column name instead of original variable name when synchronizing Q/C criteria between templates to support standardizing rules across templates for different loggers or platforms with variations in raw file variable name for the same parameter
  • Fixed a bug where very short gaps were not identified and filled by core/pad_date_gaps.m
  • Fixed a bug where date tokens in abstracts were not filled in by parsers/parse_usgs.m when a generic template is used
  • Fixed an undefined output error in core/pad_date_gaps.m

3.9.2 to 3.9.3: Enhancements and bug fixes, including:

  • Added support for calculated criteria when filtering or sub-setting data sets (gui/ui_querybuilder.m)
  • Added an Import button to the Q/C rule editor for importing rules from any metadata template variable (gui/ui_qccriteria.m)
  • Added an option to the Data Editor to display only rows with flags assigned to values in specified columns (gui/ui_datagrid.m)
  • Added support for reporting on filled missing values during a harvest (demo/gap_fill_report.m)
  • Added an option to the Dataset Editor for selectively cloning columns to create a new data structure (gui/ui_editor.m)
  • Added a command-line function for updating selected data records based on query criteria (core/update_query.m)
  • Added support for custom index file names, yearly plot intervals, and custom plot sizes in the Harvest Dashboard (workflows/harvest_dashboard.m)
  • Improved support for manually adding an array of qualifier flags to selected rows of one or more columns in a workflow (core/addflags.m)
  • Updated the query builder (gui/ui_querybuilder.m) to support expression evaluation for criteria like in the Data Editor (gui/ui_datagrid.m)
  • Updated the Add Calculated Column dialog (gui/ui_calculator.m) to automatically wrap string constants in quotes to prevent syntax errors
  • Fixed a bug that prevented repeated metadata fields from the data submission Excel template from being imported (core/imp_header.m)
  • Fixed a bug parsing query criteria containing isnull() statements (gui/ui_querybuilder.m)
  • Fixed a bug that could lead to uneven time steps when padding date gaps due to floating-point precision issues (core/pad_date_gaps.m)

3.9.1 to 3.9.2: Major bug fixes and minor enhancements, including:

  • Added support for serializing complete GCE Data Structures as text files, then loading these files back into the toolbox without loss of information for archiving and data exchange outside of MATLAB
  • Updated gui/ui_import_filter.m to support up to 10 user arguments for filter m-files to accommodate optional arguments in Campbell import functions (previously only 6 were supported)
  • Updated gui/ui_datagrid.m to automatically evaluate mathematical expressions, function calls and date strings typed into numeric cells to simplify adding calculated values to data sets
  • Updated workflows/harvest_dashboard.m and the related stylesheets in xml to include min/mean/max below each generated time interval plot
  • Updated xml/eml2gce.m to improve parsing and display of methods and instruments from EML metadata methods trees
  • Fixed a bug in plotting/plotdata.m that returned an error when all-NaN Y column(s) were selected for plotting
  • Fixed a bug in gui/ui_metastyles.m that caused the EML metadata style options to be deleted from data set editor menus for the current session if any style definitions were edited
  • Fixed a bug in parsers/parse_usgs.m that resulted in empty personnel metadata subfields being retained after a metadata template was applied
  • Fixed a bug in parsers/imp_ascii.m that could cause column titles specified in custom import filters to be ignored if an explicit format string is not defined (auto format option)
  • Fixed a bug in gui/ui_importfilter.m that resulted in a syntax error in the generated custom import filter m-file if the default title contained an apostrophe

3.9.0 to 3.9.1: Minor bug fixes

3.8.1 to 3.9.0: Major enhancements and bug fixes

  • added support for generating comprehensive Ecological Metadata Language XML metadata from a GCE Data Structure, including documentation and attribute metadata plus STMML unit definitions optionally mapped to the EML Unit Dictionary
  • added a new GUI dialog (ui/exp_eml_data.m) for generating EML-described delimited text files with user-specified date/time and flag column formatting, as well as packageId, entity name and data distribution URL for the data file for submission to data archives
  • added support for contact information sub-fields (Name, Position, Organization, Address, ...) in all personnel fields in structure metadata
  • added support for concatenating and enumerating unique text terms when summarizing data sets containing text columns (core/ui_aggr_stats_text.m) and added this as an option in gui/ui_aggrstats.m
  • revised the metadata editor and other text editing dialogs to handle excessively large strings without Java Swing memory errors
  • extended core/flags2cols.m to support instantiating flags for all data and calculation columns as well as any other columns with flags assigned, added added this new option to gui/exp_ascii.m, gui/ui_editor.m and core/exp_matlab.m
  • fixed a bug on core/normal_cols.m to could return an invalid data structure in certain circumstances
  • fixed a bug in parsers/imp_ascii.m that caused valid header metadata to be ignored is any unmapped attribute data types or variable types are present
  • fixed a bug in parsers/imp_campbell_toa5.m that resulted in inappropriate date recalculation in files already containing date part columns as part of the .dat file
  • fixed a bug in core/addflags.m that could result in an empty data structure as output with no error message
  • fixed a bug in gui/querybuilder.m that generated the incorrect query syntax when "=NaN" was specified for a numeric column
  • fixed a bug in plotting/plotdata.m that caused an incorrect symbol to be displayed in the legend when the option to display flags as symbols was selected

3.8.0 to 3.8.1: Minor enhancements and bug fixes

  • added support in the Data Harvester Dashboard for displaying variables with red/yellow status condition at the top of the page to highlight sensor issues
  • added support for generating Time_Min_* and Time_Max_* columns when aggregating time-series data sets
  • added options for specifying the statistics to calculate in data aggregation dialogs
  • added an option for plotting Q/C flags as red symbols on top of the data values instead of text characters to improve plotting performance with large data sets containing many flagged values
  • added an option in the /demo/data_harvester.m workflow for reprocessing an existing data structure to generate indices and plots without importing new raw data
  • added support for registering new flags when calling /core/addflags.m and naming a new column when calling /core/unit_convert.m to simplify use in automated workflows
  • added an alternative interpolation algorithm (/core/interp_missing2.m) for interpolating time-series data with a strong diurnal pattern using flanking values measured at the same time on preceding and following days
  • fixed a bug in ui_fetch_ncdc_ghcnd.m that prevented entering certain start/end dates

3.7.1 to 3.8.0: Minor enhancements, bug fixes and directory changes

  • removed all GCE-LTER specific workflows, data files and settings from the SVN and public distributions
  • added support for exporting data structures to the table objects introduced in MATLAB R2013B
  • fixed a bug in gui/ui_expclimdb.m that prevented saving changes to ClimDB variable mappings
  • fixed a bug in parsers/fetch_ncdc_ghcnd.m that resulted in temporary file saving errors when empty filenames and pathnames were specified

3.7.0 to 3.7.1: Minor enhancements and bugfixes

  • enhanced drift correction GUI to support graphically selecting the value offset in addition to date range (ui_correct_drift.m)
  • added support for splitting CSI array data into separate data structures on import (split_csi_arrays.m)
  • added support for synchronizing Q/C criteria changes across multiple templates (ui_template.m)
  • added support for naming exported data structure variables in files (exp_matlat.m)
  • added feature for abstracting system paths in workflows to facilitate use on multiple systems (localpath.m)
  • added a menu item in the Dataset Editor (ui_editor.m) for padding date gaps in high frequency data (up to 20Hz)
  • enabled use of the Q/C flag definition editor when creating/editing metadata templates using the "All Metadata" option
  • revised startup.m to check the MATLAB path for each directory individually to ensure stale entries in the permanent path don't prevent registering required directories
  • fixed bug in ui_fetch_dataturbine.m that prevented changes made to the target server from being used in calls to DTlist.m

3.6.0 to 3.7.0: Major enhancements and minor bugfixes

  • added data harvesting support functions and menu items to the Dataset Editor application ('ui_editor.m') for managing timed execution of data harvests and logging all harvest operations automatically
  • added support for flagging aggregated data based on maximum number of consecutive missing values
  • added support for choosing which entities in multi-entity EML data sets to download (fetch_eml_data.m, EMLdataset2mfile.xsl, EMLdatasetEntities.xsl)
  • added EML units to unit conversion database, including 1:1 mappings to conventional unit strings
  • added support for specifying an array of numeric missing value codes to remove in addition to NaN in no_nan.m to support filtering values in Q/C rules
  • updated database/sql2struct.m to always return a multi-dimensional structure regardless of setdbprefs option
  • updated core/pad_date_gaps.m to support flagging inserted values
  • added addition Thalweg reference data to settings/thalweg*.* in the GCE distributions
  • updated gui/ui_visualqc.m to re-sort data by the X column before returing data from plotgoups.m

3.5.1 to 3.6.0: Major toolbox reorganization, bugfixes and enhancements

  • revised and expanded data mining tools for NOAA NCDC Global Historic Climate Network data to include 85,986 stations around the world
  • added support for editing geographic databases included with the toolbox (gce_locations.mat, gce_polygons.mat)
  • moved toolbox dialog preference files, reference databases and other support data from /userdata to /settings to provide better separation of toolbox data files and user data files
  • removed default metadata templates (imp_templates.mat) and import filters (imp_filters.mat) from /userdata in distribution files; if local copies of these files are not provided by the user, these files will be auto-generated from settings/imp_templates_default.mat and settings/imp_filters_default.mat
  • added support for exporting data in MATLAB struct format, with fields named based on columns and records in individual struct dimensions (i.e. nx1 struct variable, where n = number of data rows)
  • added support for trimming insignificant white-space from all or selected text columns (core/trim_textcols.m and gui/ui_editor.m)
  • added support for reformatting date columns during text file export (gui/ui_exportasc.m)
  • fixed performance issues with lookup joins on text columns (core/joindata.m)
  • fixed character escaping issues in workflows/harvest_datapages_xml.m that could cause XML validation issues
  • fixed a bug in core/concat_cols.m that removed the combined text column in addition to the original columns when the option to delete original column was specified

3.5.0 to 3.5.1: Minor bugfixes and enhancements

  • added support for exporting data structures in KML, XML and HTML format (xml/gceds2kml.m, xml/gceds2xml.m and xml/gceds2html.m, resp.) and added menu items in the Dataset Editor (gui/ui_editor.m)
  • added support for cloning data structures in the Dataset Editor (gui/ui_editor.m)
  • added a new function for generating a batch of single-variable date plots for use with real-time data harvesting web pages (plotting/multi_dateplots.m)
  • updated the M-file generator in ui_import_filter.m to support user-defined unit strings
  • added support in gce_fastinsert.m and gce_insert.m for performing inserts in batches to prevent Java heap memory errors with large data sets
  • corrected an issue in the demo/gce2odm.m workflow function that caused columns to be omitted from the output
  • revised core/copy_cols.m and gui/ui_editor.m to address validation issues when adding and deleting columns in template-editing mode
  • corrected an issue in gui/ui_editor.m that prevented code changes in gui/ui_editcodes.m from being saved
  • corrected an issue in gui/edit_unitconv.m that prevent changes to English<->Metric unit mappings from being saved

3.4.1 to 3.5.0: Major enhancements

  • added support for importing channel data from a Data Turbine server (requires the DTMatlabTK beta - contact  Wade Sheldon for details)
  • added support for uploading data from GCE Data Structures to relational databases (database/gce_fastinsert.m and database/gce_insert.m)
  • added support for authentication when retrieving EML-described data tables from PASTA or servers that support basic authentication over HTTPS
  • added new functions for exporting data from GCE Data Structures to relational databases (database/gce_insert.m, database/gce_fastinsert.m, core/gceds2cell.m)
  • improved performance of string date to serial date conversions when the date units match standard MATLAB datestr formats
  • improved EML metadata parsing in xml/EMLdataset2mfile.xsl by escaping apostrophes in all free-text elements to avoid syntax errors in the generated m-file
  • added new GUI form for editing custom import filters (gui/ui_edit_filters.m) that includes function syntax lookup and function-specific argument field labels as in gui/ui_batch_import.m
  • major updates to database utilities to improve data type support and struct as well as cell array resultsets (database/sql2gceds.m, database/sql2struct.m)
  • major enhancements to Q/C flag management, including support for multi-column manual flag editing in the Data Editor (all or selected flags plus all data columns; gui/ui_manual_flags.m) and specifying custom column name prefixes when instantiating flags as columns (core/flags2cols.m)
  • added support for multiple column selections to GUI forms for normalizing (gui/ui_normalizecols.m) and de-normalizing (gui/ui_splitseries.m) data sets to simplify selecting large numbers of columns
  • added support for importing selected fields from a struct (parsers/imp_struct.m)
  • added support for splitting compound values into multiple columns on a specified character (core/split_cols.m)
  • added a new QA/QC function (qaqc/flag_total.m) for flagging values based on excessive accumulation over a specified time period (e.g. precipitation, snowfall)
  • added a new function (core/convert_date_format.m) to support explicit date/time conversion to any format supported by MATLAB (including time-only formats)
  • added a new 'Convert Date/Time Format' option to the dataset editor Date Functions menu for converting the format of 1 or more selected datetime columns (gui/ui_editor.m)
  • added support for custom extensions to the toolbox (extensions/extensions.m) so that site-specific workflows and plotting tools can be added to the dataset editor menus without altering gui/ui_editor.m code; extension functions can optionally return data to the calling editor instance to support editing operations in addition to export/plotting operations
  • updated addmeta.m to support an append option to allow new metadata content to be merged with existing content on a field-by-field basis

3.4.0 to 3.4.1: Bugfixes

  • fixed errors in ui_importfilter.m parsing column names, units and formats from text files when empty fields are present
  • revised EMLdataset2mfile.xsl to escape apostrophes in additional elements to avoid syntax errors in the generated mfile, and to use the packageId as the mfile name for consistency with the mfile names generated by the VCR statprog web service and fetch_eml_data.m
  • revised ui_editcodes.m to check for pipe separators before semi-colons to minimize the chance of inappropriately splitting code definitions rather than multiple code lists, thereby preventing proper code parsing
  • Updated gui/ui_importfilter.m to avoid splitting column names containing spaces when tab or comma delimiters are used and to check for empty fields when parsing the format to avoid array-indexing errors
  • updated csi2struct.m to correct a syntax error generating an array of NaNs when no date column is present
  • Updated parsers/imp_struct.m to correct a syntax error initializing an index variable

3.3.7 to 3.4.0: Major enhancements and minor bugfixes

  • added new GUI dialog for filling in missing values with calculated values (ui_calc_missing.m)
  • enhanced the GUI dialog for sensor drift correction (ui_correct_drift.m) by adding a "Pick" button for choosing the date range on a data plot
  • enhanced the GUI dialog for adding calculated columns (ui_calculator.m) to include fields for assigning variable type and units and adding support for saving calculations for re-use
  • enhanced the GUI dialog for defining and editing metadata templates (ui_template.m) to support editing template metadata in a specialized instance of the Data Structure Editor (ui_editor.m), importing and exporting selected templates between toolbox instances, and to redesign and relabel GUI buttons to clarify their functions
  • corrected minor bugs in confirmdlg.m that prevented proper operation in MATLAB 2012B and removed a legacy version in \mapping
  • re-labeled several menu items in ui_editor.m for clarity

3.3.6 to 3.3.7: Minor enhancements

  • added new GUI dialog for retrieving EML-described data tables (ui_fetch_eml_data.m)
  • updated EML support files (EMLdataset2mfile.xsl, fetch_eml_data.m, eml2gce.m) to parse comprehensive metadata from EML to document the downloaded data
  • updated GUI data editor (ui_editor.m) to improve support for retrieving multi-entity EML data sets and to add 'Close Other' and 'Close All' options on the 'Window' menu
  • added 'workflows' and 'demo' directories to include generalized data harvesting code and samples to toolbox distributions
  • added new data harvest support functions, including harvest_check.m for emailing harvest quality reports, stop_harvesters.m to shut down all or selected harvest timer objects, and a data_harvesters.m sample workflow
  • added import_templates.m for importing templates from another toolbox template database (imp_templates.mat)
  • added support for opening selected datasets in ui_multimerge.m to inspect files prior to merging

3.3.5 to 3.3.6: Minor bugfixes and enhancements

  • corrected bug parsing column titles from tab-delimited files in ui_importfilter.m
  • changed default option for creating a custom text import filter in ui_importfilter.m to yes
  • restored ability to import text files automatically when only a single-line header is present and the first column is non-numeric
  • added split_toa5_daily.m to split daily files from large CSI TOA5 tables for faster parsing and loading
  • updated csi2struct.m to replace gmtoffset argument with timezone for broader compatibility
  • updated eml2gce.m to avoid errors with space-containing column names when determining variable type resulting from multiple inlist matches, and refined variable type logic and added more code comments
  • added calc_missing_vals.m for filling in missing values in one or more columns based on a calculated expression (or constant value)
  • updated ui_batch_import.m to include a path Sync button and a browse button for files to import
  • updated imp_ascii.m to fix issue replacing spaces with commas when automatically determining column formats by inspecting the first 5 data rows to restore ability to automatically parse text files with combined data and time columns
  • updated aggr_stats.m to skip SD/SE calculations for mean-only option for performance and to clean up function help and code documentation
  • updated aggr_stats.m, aggr_datetime.m, aggr_bindata.m and aggr_movingdatewindow.m to support options for mean only and total only statistics and full stats without total

3.3 to 3.3.5: Major enhancements

  • revised and optimized text file importing function imp_ascii.m and supporting function imp_filter.m to take advantage of textscan on MATLAB 7+ for improved performance, compatibility, and support for non-standard missing value code removal without file re-writing (note: legacy textread parsing is still supported for MATLAB 6.x)
  • added new GUI dialog (ui_batch_import.m) for batch processing data files for importing into the GCE toolbox; this dialog prompts for import filter arguments (based on filters registered in imp_filters.mat) and runs batch_import.m for all or selected files in a directory for conversion to GCE data structures
  • extended text file export dialog (ui_exportasc.m) to support batch-exporting all or selected files in a directory with the specified settings
  • updated ITIS taxonomic data parsing (fetch_itis.m and parse_itis.m) to reflect recent changes to HTML markup that broke the parsing logic
  • added new mapping utility (gps2thalweg.m) for automatically building Thalweg reference data from sparse GPS coordinates along a river channel at the specified resolution, and function (update_thalweg.m) for updating existing Thalweg reference data in thalweg_bnd.mat and thalweg_ref.mat; extended GCE Thalweg reference data for GA coastal rivers
  • consolidated XML utilities and stylesheets into an /xml directory, and added experimental stylesheets and functions for retrieving and parsing EML-described data tables (EMLdateset2mfile.xsl, fetch_eml_data.m, eml2gce.m)
  • added support to csi2struct.m for parsing CSI array data that span multiple years but do not include a Year variable (i.e. only Yearday)

3.20.3 to 3.30: Major enhancements

  • added support in exp_ascii.m and ui_exportasc.m for specifying different line terminators for text files (e.g. CR, LF, CR/LF)
  • updated add_sitemetadata.m to support automatically looking up bounding polygons when stations/locations are matched
  • added imp_ncdc_ghcnd.m and ncdc_ghcnd_parameters.mat for parsing NCDC GHCN-D files
  • added fetch_ncdc_ghcnd.m to retrieve GHCN-D files from the NOAA NCDC
  • added codes2dataset.m to generate code definition datasets from primary data sets with coded columns
  • added add_itis_tsn.m to support automatic ITIS TSN lookup for nominal or coded species name columns
  • minor bugfixes and code documentation changes

3.20.2 to 3.20.3: Minor enhancements

  • added functions correct_drift.m and ui_correct_drift.m for correcting data for sensor drift using various methods
  • added replace_flags.m and extended ui_search_replace.m to support search/replace on flag assignments
  • updated data set editor (ui_editor.m) to add options for correcting sensor drift and replacing flags
  • updated and improved documentation of GUI tools
  • revised updata_data.m to support flagging updated data values
  • revised imp_filter.m to improve parsing speed with very wide data tables

3.20.1 to 3.20.2: Bug fixes and minor enhancements

  • fixed bug in ui_templates.m that caused data editor menu corruption after editing metadata template definitions
  • fixed bug in dataflag.m that caused an array index error when manually adding flags to columns with other flags already assigned (e.g. via daterange2flags.m)
  • fixed callback errors in ui_editor.m that prevented adding a calculated YearDay column with rounding or fixing digits
  • added a work-around for a regex bug in MATLAB R13 that could lead to invalidation of text columns after search/replace
  • updated parse_usgs.m to account for new colon separators when multiple qualifier flags are assigned
  • added support for storing variable code definitions in metadata templates for automatically updating definitions in data set metadata when attribute metadata are applied
  • added new function calc_yearday_stats.m for generating daily stats for time-series data for use in q/c rules or other analyses

3.20 to 3.20.1: Minor enhancements

  • added support for saving data structures as named variables in .mat files from the data set editor (ui_editor.m)
  • added support for importing metadata content from existing templates in the metadata editor (ui_editmetadata.m) to simplify creating new templates or documenting new data sets
  • added support for filtering records based on NaN/null values for both numeric and string variables

3.12.1 to 3.20: Major enhancements and bugfixes

  • enhanced ui_datagrid.m support for large data sets (>1 million records)
  • enhanced imp_filter.m to improve file writing speed and to use regular expression pattern matching to prevent partial substitutions of missing values codes by checking for flanking delimiters or whitespace
  • removed calls to deprecated MATLAB 'findstr' and 'strmatch' functions, updating syntax to support MATLAB 2011B
  • major enhancements to csi2struct.m, splitcsi.m, and imp_campbell_toa5.m for importing Campbell data logger files
  • major enhancements to matchsites.m for dramatically faster polygon inclusion matching (2-3 orders of magnitude faster)
  • added monthticks.m function for setting x-axis range and labels of time-series plots for even month intervals
  • minor code improvements to other functions and dialogs

3.12 to 3.12.1: Minor enhancements and bugfixes

  • added support for selective copying of Q/C flags as text or numeric coded data columns (flags2cols_selected.m)
  • updated dialog for removing flagged values or flag assignments (ui_clearflags.m) to support new flag instantiation function as well
  • updated Q/C function menus in data set editor (ui_editor.m) to support new Q/C flag options
  • corrected loop counter bug in edit_importfilters.m
  • minor code improvements to other functions and dialogs

3.11 to 3.12: Minor enhancements

  • updated fetch_ncdc_data.m to accomodate variable back-end server end-points for data retrieval
  • added support for parsing data and attribute metadata from Campbell Scientific Instruments TOA5 ASCII table format files (parse_campbell_toa5.m)
  • updated NERR CDMO parse (parse_nerr_cdmo.m) to accommodate string missing value codes in additional fields
  • add support for search and replace of numeric values in data columns (including NaN) for parity with text search/replace (num_replace.m, ui_num_replace.m)
  • updated string_replace.m and negative2zero.m to use update_data.m for more comprehensive logging of value changes in the metadata, with support for flagging revised data
  • updated interp_missing.m, interp_missing2.m and ui_interp_missing.m to support user-specified flags (or no flags) for interpolated values
  • updated listmeta.m to dramatically speed up metadata rendering when very large field contents are present (10-fold increase)
  • other minor code improvements

3.1 to 3.11: Minor enhancements

  • updated parse_usgs.m to accomodate new USGS NWIS RDB format for time zone values in instantaneous data files
  • updated plotmenu.m to support changing the pointer type to aid in Q/C flagging
  • updated extract.m to improve support for multi-column array extraction
  • updated meta_fields.m to improve efficiency and correct an error applying a named style specification provided as input
  • updated split_dataseries.m to disable metadata merging for self joins to speed up the function

3.01 to 3.1: Minor bugfixes and enhancements

  • added support for decoding numeric and text code columns in the data editor (ui_editor.m) by matching codes to code lists in the metadata to generate a categorical text column of code definitions
  • added support for copying one or more columns in the data editor (ui_editor.m) prior to editing or search/replace
  • added support for IN and NOT IN criteria to the query builder (ui_querybuilder.m) for filtering data set records based on numeric or text value lists (and for parity with the Q/C criteria builder)
  • updated ui_unitconv.m to throw an error when conversion is attempted on string variables
  • updated viewtext.m to pre-convert character arrays to cell arrays before performing tab replacement to prevent errors
  • added cell2textfile.m for saving cell arrays containing scalar character and numeric values as a delimited text file
  • added imp_schlumberger.m for parsing data from Schlumberger groundwater loggers
  • updated meta2struct.m to properly parse coordinates from metadata that contains both bounding boxes and point locations
  • updated add_sitemetadata.m to support multiple site/location columns and to return the unmodified structure and warning when site/location lookups fail
  • updated search_index.m and search_index_r13.m to include column datatype and description in index
  • other minor bugfixes to correct syntax or runtime errors

3.00 to 3.01: Major bugfixes and enhancements

  • fixed a major bug in the data grid application ('ui_datagrid.m') that could cause data corruption when editing values using recent versions of MATLAB; due to recent changes in Java-based uicontrol event handling, if a slider was moved after editing a value in a grid cell but before enter was pressed or focus was changed to another cell, the slider position metadata was updated before the update callback was fired, resulting in an offset in grid position that could lead to edits being applied to the wrong cell. A completely new callback approach is now used to ensure updates are always applied before slider events are processed regardless of event order.
  • added new QA/QC functions for flagging numeric values are are either present or absent in an array of allowed values ('inarray.m', 'flag_inarray.m' and 'flag_notinarray.m'); these functions parallel QA/QC functions for text list checks and are useful for QA/QC on numeric code columns
  • added support for new numeric array functions in the GUI QA/QC criteria editor ('ui_qccriteria.m'); options for building rules for numeric columns now include 'IN' and 'NOT IN' as for text columns.
  • added new functions ('codes2criteria.m') and GUI data editor menus for generating QA/QC rules automatically from code definitions in the metadata (e.g. 'marsh = marsh site, creek = creek site, forest = forest site' in metadata results in the QA/QC rule flag_notinlist(x,'marsh,creek,forest')='I'
  • updated GCE XML metadata implementation ('listmeta.m') to include column descriptions and nest value codes with other attribute descriptors under data/columns/column

3.0rc to 3.00: Minor bugfixes and enhancements

  • corrected path handling logic in ui_editor, ui_search_data and supporting functions to prevent double file separators in save paths
  • added 'batch_transform_xml.m' for performing batch XSL transformations on XML files in a directory
  • added 'copy_attribute_meta.m' for copying all or select attribute metadata between columns in two data structures
  • revised code documentation and added GPLv3 notice to several functions not updated in 3.0rc

2.998 to 3.0rc: Complete re-factoring of the toolbox directory structure, bugfixes and enhancements, including:

  • m-file functions reorganized into directories by function, including
    • build: release building utilities (SVN only)
    • core: core command-line GCE Toolbox functions and utilities
    • database: database toolbox support functions
    • docs: documentation files (SVN only)
    • gui: GUI dialog functions
    • gui_devel: GUI dialogs in development (SVN only)
    • harvest: data harvesting workflow functions (SVN only)
    • mapping: mapping functions and utilities
    • parsers: data import filters and parsing functions
    • plotting: graphics and plotting functions
    • qaqc: QA/QC custom flag functions
    • search_indices: search index files for search engine
    • search_temp: temporary search files for search engine
    • search_utils: search utilities (SVN only)
    • search_webcache: cache files for search engine
    • specialized: specialized GCE workflows and functions (SVN only)
    • support: general support functions (not requiring GCE Toolbox)
    • userdata: user preferences and stored data
  • added GPLv3 license, and added GPLv3 notice and reference to header of all GCE Toolbox m-files
  • added more comprehensive GUI dialog for harvesting NCDC data ('ui_fetch_ncdc'), and improved error handling in NCDC data mining functions ('fetch_ncdc_data')
  • added dataset describing >30k NCDC climate stations ('fetch_ncdc.mat')
  • improved automatic metadata generation for NCDC data ('fetch_ncdc')
  • improved help text for many mapping functions

2.997 to 2.998: Major enhancements, including:

  • added functions for harvesting data for any number of NOAA NCDC weather stations over the Internet ('fetch_ncdc_climate_url', 'fetch_ncdc_climate_data', 'batch_import_ncdc.m'); Note: NOAA only permits direct data requests originating from .edu domains
  • added support for retrieving USGS station inventory reports from the USGS NWIS to update the list of stations in the USGS data mining dialog ('fetch_usgs_inventory', 'imp_usgs_stations')
  • enhanced parsing of USGS RDB files to support importing data sets containing attributes not pre-defined in metadata templates
  • added support for copying columns to the data editor dialog ('ui_editor')
  • added a command-line function for decoding coded data columns using code definitions in the metadata ('decodecols')
  • revised fontsizes in several GUI dialogs for improved display in Linux environments
  • various minor syntax improvements and bugfixes

2.996 to 2.997: Enhancements, including:

  • updated 'interp_missing' and added 'interp_missing2' for interpolation of missing values in compound data series (i.e. interpolation within grouped data)
  • added 'ui_interp_missing' dialog for filling gaps in data series by value interpolation using 'interp_missing' and 'interp_missing2'
  • added 'string_replace' and corresponding 'ui_string_replace' dialog for replacing text in string columns using full or partial string matches
  • updated 'ui_editor' and documentation to include new interpolation and search/replace dialogs
  • enhanced support for loading variables from the base MATLAB workspace in 'ui_editor'
  • enhanced 'fill_date_gaps' to dramatically improve speed for large data sets (up to 100-fold)
  • various minor syntax improvements to toolbox functions for improved efficiency and compatability with MATLAB 2009 and above

2.995 to 2.996: Bug fixes and enhancements, including:

  • fixed bug in 'ui_qccriteria' that could corrupt custom q/c criteria containing '=='
  • updated 'listcols' to report data type information
  • updated 'get_type' to support retrieving attribute metadata for multiple columns
  • added 'tide_high_low' command-line function for empirically identifying times and amplitudes of peak high and low tides based on depth or pressure in mooring time series data (i.e. for use in tidal binning)
  • added 'date2weeknumber' command-line function for calculating week numbers based on several conventions (ISO, US, and simple) for use in aggregating data by week

2.994 to 2.995: Bug fixes and enhancements, including:

  • fixed bug in 'listmeta' that prevented some column metadata from being included when xml format was specified
  • fixed bug in 'ui_qccriteria' that prevented rules containing equals conditionals ("=") from being parsed correctly
  • updated error handling in 'deg2utm' and 'utm2deg', and updated spheroid constants for NAD83 and NAD27 datum
  • added 'concat_cols' function to generate a concatenated text column from text and numeric data columns for display or plotting, and added as an option in the data set editor "Edit" menu ('ui_editor')
  • added support for additional date/time formats to 'add_datecol' and 'ui_editor'
  • enhancements and bugfixes in 'add_datecol' and 'add_datepartcols' to improve speed and handling of year day values
  • improved error handing in 'cols2flag', 'imp_ascii', 'trimstr' and 'ui_searchdata'
  • added support for loading variables and data structures directly from the base MATLAB workspace to 'ui_editor', to update or overwrite existing column values

2.99 to 2.994: Bug fixes and enhancements, including:

  • fixed major bug in 'imp_ascii' introduced in 2.99 that prevented importing comma-delimited files with extensions other than .csv
  • updated GCE geographic locations database file ('gce_locations.mat') to correct inaccurate coordinates for some weather and hydrogaphic monitoring stations
  • updated 'messagebox' to support an optional 'Cancel' button
  • updated 'joindata' and 'ui_joindata' to support disabling units checking for key columns before performing joins (although unit checks are still performed by default to ensure proper key matching)

2.98 to 2.99: Bug fixes and enhancements, including:

  • fixed minor bug in 'imp_matlab' that might have prevented importing arrays if unsupported variable types were included in the .mat file, and enhanced function to support passing an array of variable names instead of choosing variables from a GUI dialog
  • fixed path validation bug in 'ui_multimerge'
  • fixed column sizing bug in 'ui_datagrid' that prevented proper grid sizing and resizing
  • revised code for handling legends in 'plotdata' to prevent figure resizing quirks in MATLAB 7+
  • revised code for disabling control docking for GUI dialogs in MATLAB 7+ to circumvent error if docking was enabled for new figures by default
  • revised the GCE XML metadata schema to improve encoding of column characteristics; all attribute metadata fields are now nested within Metadata/Data/Columns/column tags (e.g. Metadata/Data/Columns/column/name)
  • revised 'xml2file' to specify ISO-8859-1 instead of UTF-8 encoding to avoid parsing errors in XML editors when special characters are present
  • improved support for parsing CSV files with quoted fields in 'imp_ascii' and 'imp_filter'
  • enhanced 'ui_importfilter' to support defining a new metadata template during creation of a custom import filter, and improved automatic parsing of column names and format strings from the file preview window
  • added new documentation for data import and export features

2.97 to 2.98: Minor enhancements, including:

  • revised 'update_data' and added 'update_dataset' to support updating values in multiple columns (and adding new columns) based on data in a separate data structure, with value changes and column additions logged to the original data set metadata
  • updated 'listcols' to display column data type in addition to name and units
  • updated 'extract' to support extracting multiple data set columns into an array in one step
  • updated default maximum points to interpolate in 'interp_missing' to support wider data gaps for high frequency data sets

2.96 to 2.97: Bug fixes and enhancements, including:

  • fixed serious bug in 'ui_qccriteria' that caused edits to QA/QC criteria rules to not be saved in some circumstances
  • fixed bug in 'inlist' that prevented the use of single match criteria
  • fixed bugs in 'convert_datatype', 'split_dataseries' and 'ui_editor' caused by recent changes to 'inlist' syntax
  • added new function 'running_mean' for calculating a running mean along a user-specified window for smoothing data
  • added new parser for SeaBird Electronics CTD data ('parse_seabird') and custom flag function ('flag_ctd_soak_period') and templates for automating CTD data processing
  • enhanced 'ui_clearflags' to support removing flag assignments (calling 'clearflags') in addition to removing values or records assigned specified flags (calling 'nullflags' or 'cullflags', resp.)
  • added new 'Q/C Flag Functions' menu option to data editor for removing flag assignments

2.95 to 2.96: Bug fixes and updates, including:

  • replaced corrupted version of 'add_anomalies' in 2.95 that interfered with the operation of several import filters and prevented proper documentation of anomalies in data set metadata
  • updated 'ui_qccriteria' and 'ui_template' to support editing QA/QC criteria in metadata templates using a GUI dialog (in addition to the data grid view)
  • Note that beginning with version 2.96 the project version of the toolbox is no longer pre-compiled into .p files. However, the code is still copyrighted and permission must be obtained to distribute GCE software outside of the project

2.93 to 2.95: Various minor enhancements, including

  • added new function 'aggr_movingdatewindow' and 'ui_aggrmovingdate' for generating a smoothed time series data set by statistically summarizing values along a sliding date window
  • added support for defining common text Q/C rules in 'ui_qccriteria' using uicontrols ('IS', 'IS NOT', 'IN' and 'NOT IN' rules for Q/C checks against string values or lists)
  • expanded support for filtering on text criteria to 'ui_querybuilder' ('IS', 'IS NOT', 'IN' and 'NOT IN', for parity with 'ui_qccriteria')
  • 'startup.m', 'contents.m', and documentation in 'gce_datatools.mat' now auto-generated during relase builds for formatting consistency and accuracy
  • standardized help text for all functions for improved parsing during release builds

2.92 to 2.93: Added new function 'normalize_cols' and corresponding GUI dialog 'ui_normalizecols' for normalizing a data set by merging multiple columns to form combined parameter name and parameter value columns with records in other specified columns repeated for each group of parameters (i.e. converts a dataset with repeating groups to a normalized database table). Note that this function complements 'split_dataseries' (and 'ui_splitseries'), which generates individual columns from normalized parameter name, parameter value columns.

2.91 to 2.92: Bug fixes and updates, including:

  • bug fixes and enhancements to 'nullflags' to improve performance with large data sets and prevent potential removal of flags from incorrect columns when discontinuous column ranges are specified
  • significant performance improvements in 'dataflag' (2-3x speedup with large data sets)
  • added support for selective flag removal plus one additional QA/QC rule option to statistical aggregation tool GUIs ('ui_aggrstats', 'ui_aggrdatetime', 'ui_aggrbindata')
  • added new command-line function for retrieving syntax info for toolbox functions ('parse_gce_syntax'), and used this function to develop a new GUI dialog for building QA/QC criteria based on function calls ('ui_flagfunctions')
  • updated QA/QC reference documentation

2.90 to 2.91: Bug fixes and updates, including:

  • bug fix in 'cols2flags' to prevent truncation of pre-existing flags when codes in a text flag column (e.g. data provider-specified qualifiers) are converted to Q/C flags under certain conditions
  • updates to ClimDB/HydroDB XSLT sheets to accommodate commas in station descriptions that were causing parsing failures
  • updates to USGS NWIS data parsing function ('parse_usgs') to support retention of individual 'P' (provisional value) flags
  • added function 'clearflags' to allow specific assigned flags to be cleared (e.g. to suppress display of 'P' flags on plots)
  • added function 'multi_meta' to support applying multiple, date-dependent metadata templates to data sets to accommodate changes in parameter metadata over time (also supports automatic unit conversions to harmonize attributes)
  • additional 

2.83 to 2.90: Major enhancements to statistical functions and dialogs to support more flexible assignment of Q/C rules for derived data columns ('aggr_stats', 'ui_aggrstats', 'aggr_datetime', 'ui_aggrdatetime', 'aggr_bindata', 'ui_bindata'). Multiple rule sets can now be specified to flag derived values based on number or percentage of flagged and/or missing values in the source data; therefore, multiple thresholds can be defined for flagging statistics derived from incomplete or flagged source data (e.g. flag as Q if >5% missing values, flag as I if >25% missing values, ...).

2.82 to 2.83: Performance enhancements to 'dataflag' and minor modification to 'aggr_data' to prepend 'Total_' instead of 'Sum_' to column names for aggregate totals for compatibility with attribute name conventions used in ClimDB and elsewhere (also made corresponding name mapping changes to several metadata templates and the 'exp_climdb.mat' conversion mappings)

2.81 to 2.82: Minor bugfix in 'imp_matlab' and minor refinements to several GUI forms

2.80 to 2.81: Bug fix in 'imp_datastruct' to correct path handling errors introduced in version 2.80 that prevented merges and joins to data structure files on disk

2.70 to 2.80: Major usability enhancements to the data editor ('ui_editor'), including:

  • relabeled commands in the File menu to improve consistency with the Search Engine menus and commands used by other common applications
  • added a 'Window' menu and new commands for arranging multiple editor windows on the console (cascade, tile and stack windows), as well as selecting an open window to view from a menu listing the titles of all open data sets
  • added new commands to the 'Edit' menu for transforming data columns between data types, casting existing values into the new type using appropriate conversion functions and filters (logging the transform function used in the metadata)
  • added support for choosing open data sets for merge and join operations, allowing data to be integrated on the fly without first saving files to disk or transferring results to the search engine
  • added an automated date/time join option, which identifies compatible date/time columns in two time-series data sets to automatically create join conditions, simplifying the process

    Note: beginning with version 2.80, MATLAB 6.5 (release 13) or higher will be required due to increased dependence on networking and XML features that are not available in prior versions

2.63c to 2.70: Bug fix in 'ui_harvestclimdb' to properly enable the 'Proceed' button after all fields are filled in when the dialog is run for the first time without a saved settings file. Significant enhancements and new features, including:

  • added new commands to the Editor window 'Q/C Functions' menu for locking/unlocking automatic recalculation of Q/C flags after data set changes ('ui_editor')
  • reconfigured options on the Editor window 'Tools' menu to simplify finding dialogs for various data transformation and analysis tasks ('ui_editor')
  • added a new function and Editor window command for splitting a compound data series on values in a specified column, and then joining the split columns on other specified data columns to create a normalized data table (e.g. splitting records from a merged data set on column 'Site', creating separate 'Salinity' and 'Temperature' columns for each value in 'Site', and joining split 'Salinity' and 'Temperature' columns on date/time columns 'Year','Month','Day','Hour','Minute' to create a tabular data set with separate columns for observations from each site) ('split_dataseries')
  • added new command-line data mining functions for retrieving National Estuarine Research Reserve data from the NOAA/NWS Hydrometeorological Data System ('fetch_nerr_data', 'parse_nerr_data')
  • numerous usability improvements, including automatically recalling the most recent settings used on plotting dialogs and automatically prompting users to download a pre-compiled search index for GCE web-based data sets when starting the 'Search Engine' application for the first time

2.63b to 2.63c: Minor bug fixes and refinements to 'imp_ascii' to prevent problems parsing ClimDB information files; changes to Q/C rules for generated date component columns in 'add_datepartcols' to improve flagging logic; enhancements to 'ui_visual_qc' to support closing all plotting dialogs after manually-updating QA/QC flags (i.e. closing dialogs containing cached copies of the outdated structure); updates to USGS metadata templates to remove duplicate attribute metadata references; bug fixes in polygon editing functions in mapping tools 'edit_poly' and 'poly_mgr'

2.63a to 2.63b: Modifications to USGS data parsing  function ('parse_usgs') and metadata templates to accommodate additional USGS NWIS data format revisions

2.63 to 2.63a: bug fix in 'dataflag' to correct an issue that prevented clearing residual flags after flagged values were changed or removed under certain circumstances

2.62 to 2.63: Modifications to USGS data processing functions ('parse_usgs', 'fetch_usgs', 'ui_harvestusgs') and metadata templates to accommodate revised format of daily data provided by the USGS NWIS web applications (i.e. flagging of all data values as accepted (_A) or provisional (_P) in addition to other flags such as _e for estimated value, etc). Note that accepted flags are automatically removed from processed data for compatibility with the GCE Data Toolbox Q/C flagging scheme, and provisional value flags are also removed but a coded 'Provisional' column is added to indicate that records contain values flagged as provisional by USGS. Other flags (e.g. 'e' for estimated value) are converted to text flag columns and are automatically merged with rule-based flags assigned by attribute Q/C criteria.

2.61 to 2.62: minor enhancements, including:

  • updated default ClimDB/HydroDB url in 'ui_harvestclimdb' and 'ui_harvestclimdb.mat'  in response to web address changes at OSU
  • added new function 'consolidate_cols' for consolidating data values in duplicated or otherwise redundant data columns with compatible data types/variable types
  • added code to 'exp_climdb' to check for and consolidate duplicate data columns (e.g. from USGS harvests) to avoid fatal harvest errors
  • added new item to  'Geographic Functions' menu in 'ui_editor' for looking up transect distances based on GPS coordinates (coordinates are matched to closest perpendicular point along Thalweg line reference transects stored in 'thalweg_ref.mat' and 'thalweg_bnd.mat')

2.60 to 2.61: minor enhancements to improve compatibility with MATLAB 2006a, plus inclusion of a filled vector map of the entire Georgia Coastline

2.56 to 2.60: major bug fix and enhancements, including:

  • added new functions and dialogs for harvesting climate and hydrographic data from the LTER All-site ClimDB/HydroDB database ('fetch_climdb_info', 'fetch_climdb_data', 'ui_harvestclimdb')
  • enhanced 'viewstats' dialog to include row labels in a fixed pane (to prevent scrolling offscreen) and to include a menu option for saving a full statistical summary report to disk (i.e. using ui_statreport)
  • rearranged fields on the 'ui_harvestusgs' dialog to improve usability
  • improved parsing of ASCII header fields containing spaces ('parse_header')
  • bug fix: corrected major flaw in 'colstats' function affecting calculation of standard errors for columns - all users are encouraged to upgrade to 2.60 to address this issue (note: SE calculated by 'aggr_stats', 'aggr_bindata' and 'aggr_datetime' were not affected by this error)

2.55 to 2.56: minor enhancements and documentation updates

2.54 to 2.55: significant enhancements and minor bug fixes, including:

  • added support in 'ui_editor' (Data Editor) for exporting the current data set directly to the Search Engine search results list; this allows newly imported or user-modified data sets to be integrated with data sets returned from various searches without cumbersome disk operations
  • added support for auto-loading of the startup screen on last window closure to all GUI dialogs
  • added options to plotting functions ('plotdata','plotgroups') and corresponding GUI dialogs ('ui_plotdata','ui_plotgroups') to support setting symbol size and fills
  • added support for axis reposition/resizing to 'plotmenu' (with new function 'plotresize')
  • added function 'compactcols' (with new menu option in 'ui_editor') for removing columns containing only NaN/null values
  • added support to 'fetch_usgs' for automatically adding USGS station metadata (coordinates, altitute, drainage area) when the generic metadata template (USGS_Generic) is used; package also includes an updated USGS stations data set ('usgs_stations.mat') containing new data fields Longitude, Latitude, Altitude and Drainage
  • refactored compiler code to ensure .p files are new than corresponding .m files to prevent warning messages using MATLAB R13+
  • fixed an index error in 'add_anomalies', which caused the incorrect columns to be listed in the processing history under certain circumstances

2.53 to 2.54: minor enhancements and bug fixes, including:

  • added a user-selectable option in 'ui_aboutgce' to automatically re-open the startup dialog if both the data editor and search engine dialogs are closed and no other dialogs are open (i.e. last window closed) - this will prevent GUI users from having to interact with the MATLAB command window
  • added support to 'add_yeardaycol' for generating a YearDay column based on multiple date part columns with or without a Year column present
  • added the function 'add_gce_cruise' for generating GCE-LTER cruise id columns from date/time information in the data set by looking up cruise date ranges in the data set 'gce_cruise_dates', and added this function to the 'Tools/Specialized' menu in 'ui_editor'
  • tightened data set validation logic and improved performance in 'gce_valid', and updated 'ui_editor' to prevent loading of invalid data sets from the command line

2.52 to 2.53: significant enhancements, including:

  • added support in 'dataflag' for selective analysis of q/c flagging criteria, allowing criteria to be evaluated on specified columns only to improve efficiency when analyzing large data sets; also added support for this feature in 'addcol' and 'ui_editor' to prevent unnecessary flag re-analysis when adding columns or editing flag criteria, resp.
  • added support in 'add_anomalies' for summarizing flagged and missing values without grouping by date for use in non-time-series data sets; also added support for this option in 'ui_editor' and 'ui_visualqc'
  • added support in 'search_index' and 'search_index_r13' for instantiating q/c flag columns prior to indexing (used to generate more comprehensive data set summaries on the GCE Data Portal site)
  • added controls to 'ui_editmetadata' for repositioning, adding and deleting individual metadata fields, and improved error handling
  • improved performance and consistency of q/c flagging functions 'flag_valuechange', 'flag_percentchange' and 'flag_nsigma'
  • improved layout of automatically-generated help text in 'help_flagfnc'

2.51 to 2.51: bug fix in 'ui_qcflags' to prevent problems entering custom QA/QC flag criteria; enhancements to 'ui_editor' and 'ui_search_data' to automatically reload the startup dialog upon exit if no other windows are open (i.e. so users will not have to use the command line or restart MATLAB to restore the startup dialog)

2.5 to 2.51: minor bug fixes in 'encode_strings', 'compass_rose' and 'mapmenu'

2.41 to 2.5: major enhancements, including:

  • optimized code for extracting metadata field contents ('lookupmeta'), resulting in improved performance for operations involving extensive metadata lookups (e.g. 2-3x faster performance when merging multiple data sets using 'multimerge', and 2x faster data set indexing using 'search_index')
  • added support for directly performing "joins" between data sets in the Search Engine dialog search results list ('ui_search_data')
  • added support to the Search Engine dialog ('ui_search_data') for automatically caching data sets resulting from "merge" or "join" operations and adding them to the search results list for additional analyses
  • added menu options for setting figure, axis and legend colors on data plots ('plotmenu')
  • bug fix in 'exp_ascii' to remove extra quote characters around column headings in statistics reports exported in comma-separated value format
  • disabled figure resizing in 'ui_datagrid' running under MATLAB 7.x to prevent GUI problems and hangs
  • minor bug fixes in 'listmeta', 'ui_aggrstats', 'ui_aggrdatetime', 'ui_copyflags' to address syntax errors when running under MATLAB 7

2.41 to 2.41: minor bug fixes; addition of function 'interp_missing' for interpolation of missing values in time series data with automatic flagging of interpolated values as 'E' = estimated, and user-specified limits on how many consecutive values to interpolate; addition of commands to the data set editor ('ui_editor') 'Edit' menu for deletion of empty records (i.e. deletion of records with all columns empty or all data columns empty)

2.40 to 2.41: improvements to 'encodestrings' to support reconciling integer code definitions with prior code definitions in the metadata when string columns are encoded as unique integers (either when run manually or automatically when exporting data sets in MATLAB matrix format); also added options to 'ui_editor' to support updating or appending code definitions when encoding string columns

2.33 to 2.40: bug fixes and minor enhancements, including:

  • bug fix in 'ui_search_data' to correct inappropriate nulling of flagged values when the option to retain flagged values is selected
  • bug fix in 'match_sites' to correctly handle NaN values in geographic coordinate columns
  • added support for logging all data value updates in the metadata to 'update_data'
  • added support for appending exported text files to 'exp_ascii' (and deprecated the 'exp_ascii_append' function which served the same purpose)
  • improved the consistency of the "csv" format option in 'exp_ascii' by wrapping all text values, flag characters, custom missing value codes, and header text in double quotes (i.e. not just those containing spaces or commas)
  • improved the consistency and program efficiency of support functions 'copycols', 'copyrows', 'deletecols', 'deleterows', and 'addcol'

2.32 to 2.33: significant enhancements, including improved support for log-scale axes in 'plotbuttons' and 'ui_axislabels', support for showing/hiding and setting colored borders on legends in 'plotmenu', and automatic coordinate system re-projection for map polygon data in 'poly_mgr'

2.31 to 2.32: minor enhancements to several functions, including:

  • improved support for parsing of formatted geographic coordinates in documentation metadata (i.e. from the Study/Coordinates field) during indexing when data sets do not contain geo-reference columns or recognized site/station columns in the data table (i.e. when GCE geographic lookups are not possible) ('search_index', 'search_index_r13')
  • improved consistency of PNG thumbnail sizes generated for different size figures in 'dateplot2html' and 'dateplot2template'
  • improved the display of long axis title labels in 'ui_axislimits'
  • added support for path caching to 'mapmenu' to simplify loading, saving or exporting multiple maps in different directories
  • added support for exporting figures and maps as 600 dpi PNG, TIFF and JPG images ('exportfig', 'plotmenu', 'mapmenu')

2.30 to 2.31: bug fix in 'ui_copyflags' to properly update the cached data structure after each flag copy operation when the 'close dialog after exporting the results' option is unchecked

2.21 to 2.30: minor bug fixes and major enhancements to mapping tools, including:

  • bug fix in 'querydata' to prevent problems parsing queries containing string search terms that match data set column names
  • bug fix in 'parse_usgs' to properly close temporary file after HTML code found and parsing halted
  • major enhancements to 'ui_mapdata':
    • added support for plotting symbols, labels or patches on open map figures (i.e. allowing generation of multiple data series/labels from 1 or more data sets)
    • added separate dialog options for plotting formatted data values as labels with optional marker symbols, allowing label font characteristics and colors to be specified separately from marker symbols and colors
    • added support for deleting previously plotted symbols or labels from maps, and repositioning labels after zoom operations to adjust spacing between corresponding markers and labels
  • added new functions to the mapping toolbox for creating CTD station data sets at specified station intervals ('find_stations' and 'ctd2dataset') for custom plotting of CTD stations on maps using 'ui_mapdata'
  • added a button on the main GCE Data Toolbox startup dialog ('ui_aboutgce') and menu items on the Data Editor ('ui_editor') and Data Search Engine ('ui_search_data') dialogs for loading maps using the mapping toolbox independently of plotting data on maps

2.20 to 2.21: enhancements to data plotting functions and dialogs, including:

  • updated 'plotdata' and 'plotgroups' to improve handling of empty data columns to avoid generation of empty plots or empty data series, and to improve handling of large title strings (i.e. now restricted to 3 lines of text)
  • updated 'ui_plotdata', 'ui_plotgroups' and 'ui_mapdata' to support comprehensive data restriction queries instead of just single column restrictions; full query statements can now be entered in plotting dialogs to filter values prior to plotting, and query statements can be entered /edited manually or by using the graphical query builder dialog ('ui_querybuilder')
  • updated 'ui_querybuilder' to support parsing query statements to populate dialog controls and to return edited query statements to calling functions instead of executing the query and returning a data set (i.e. to support data restriction query building/editing for plotting dialogs)

2.10 to 2.20: updated 'ui_search_data' and 'multimerge' to restore support for Q/C flag removal prior to data set merging (broken in 2.01); significant enhancements to the GCE Data Search Engine components, including:

  • updated 'search_index', 'search_data' and 'ui_search_data' to load configuration information stored in a common file ('search_data.mat') to support end-user extension and customization of metadata content that is indexed and searched (e.g. add/edit metadata fields and field mappings, change text comparison options)
  • added support for simultaneously searching the contents of all indexed metadata fields (i.e. 'Any Text Contains') to 'search_data' and 'ui_search_data'
  • updated the metadata field selector in 'ui_search_data' to display the text comparison type along with the metadata field (e.g. 'Abstract Text Contains' or 'Accession Begins With')

2.06 to 2.10: minor bugfix in 'get_studydates' to handle text date conversion issues in MATLAB 5.3; significant enhancements to metadata management/display functions, including:

  • enhancements to 'ui_editmetadata' to support overlaying or overwriting metadata with only selected metadata fields from an existing data structure (i.e. instead of all fields) for fine-tuning metadata content re-use (brings 'ui_editmetadata' in line with metadata importing in 'ui_editor')
  • enhanced 'listmeta' by simplifying the metadata style definition syntax, adding support for metadata field placeholders in text expressions and thereby eliminating the requirement for MATLAB character array syntax when referencing multiple metadata fields (Note: unfortunately, this change also eliminates support for MATLAB text functions in style definitions for auto-generating metadata, e.g. 'datestr(now)')
  • updated all styles in 'metastyles.mat' to support the new 'listmeta' syntax
  • updated 'listhist' to improve performance when formatting large processing history arrays

2.05 to 2.06: NOTE: Version 2.06 is the first public release of version 2.x and contains various bug fixes and enhancements, including:

  • bug fix in 'fill_date_gaps' to prevent automatic removal of records with duplicate date/time values (specific input argument now required for duplicate removal); corresponding change to 'ui_editor' to prompt for duplicate removal if duplicate dates/times found
  • bug fixes and enhancements in 'flag_valuechange' and 'flag_percentchange', including new option to recursively analyze flag conditions to minimize flagging of normal-range values following flagged outliers
  • bug fixes in 'ui_gce_register' and 'ui_search_data' to prevent inappropriately storing user registration information in the default workspace file when 'Store my registration information on this computer' was not checked (registration is still stored in manually created workspace files)
  • enhancements to metadata and text viewer dialogs to support figure resizing ('ui_viewtext', 'ui_viewmeta'), text file export ('ui_viewtext', 'ui_viewmeta', 'ui_viewdocs') and text file import ('ui_viewtext') -- users can now save formatted metadata, data processing histories and toolbox documentation to disk for their own use outside of MATLAB
  • enhancements to 'add_anomalies' to support documenting flagged and missing values only in user-selected data columns (now set as default when function called from 'ui_editor')
  • Note: the public version of the toolbox (gce_datatools_206public.zip) has some GCE-specific import filters, batch scripts and metadata templates removed, but is otherwise complete

2.04 to 2.05: minor bug fix in 'joindata'; various program enhancements and additions, including:

  • significant performance improvements in the GCE Data Search Engine ('search_data') due to improved query optimization, resulting in up to 10-fold faster searches when numerous criteria are specified
  • major speed improvement in refreshing search indices due to improvements in file modification checking that minimize loading files into memory
  • enhanced the GCE Data Search engine dialog ('ui_search_data') to support viewing the contents of each indexed path (i.e. file name, modification date, title)
  • added a GUI dialog for exporting climate and hydrographic data sets as text files in the LTER ClimDB/HydroDB format, as well as viewing/editing corresponding attribute name and unit mappings ('ui_expclimdb', based on 'exp_climdb')
  • improvements to 'flag_valuechange' and 'flag_percentchange' routines for flagging value changes relative to prior values or running averages

2.03 to 2.04: enhancements to 'fill_date_gaps' to support time series data sets with small numbers of duplicated values, and enhancements to 'search_index' and 'search_data' to support indexing and searching on column units and variable types in addition to names and also searching on column name + units

2.02 to 2.03: major bug fix in 'add_datepartcols' to correct generation of hour=24 values after rounded seconds propagated to minutes and hours, causing difficulties with date/time aggregation due to mixed hour=0 and hour=24 values in a single data set (seconds are now truncated when not included in the output columns); added logic to 'aggr_datetime' to automatically regenerate date component columns when a serial date column is present to avoid this issue with legacy data sets containing mixed hour=24 and hour=0 values.

2.01 to 2.02: minor bug fixes in 'ui_search_data' and 'mergemeta'

2.00 to 2.01: enhancements to data integration functions and the Data Search Engine GUI application, including:

  • improved merging of data release dates and keywords in metadata ('mergemeta')
  • added support for automatic generation of a 'DataSetName' column, listing the name of each original data set when multiple data sets are merged/concatenated ('multimerge'), and added corresponding options to GUI dialogs supporting this feature ('ui_multimerge', 'ui_search_data')
  • added support for automatically saving the workspace on exiting the Data Search Engine dialog and automatically reloading the prior workspace on program re-start ('ui_search_data')

2.00beta to 2.00: minor enhancements to QA/QC flag functions and major enhancements to data integration tools, including:

  • improved handling of multiple flag conditions in QA/QC flag removal functions ('nullflags' and 'cullflags'), so all flags cleared when values are cleared using selective criteria
  • changed keyword search logic in 'search_data' from partial string match to beginning string match and added code to prevent reverse matches when keywords being searched are shorter than the search term (i.e. to prevent spurious results such as matching 'par' on searches for 'Spartina' or vice versa)
  • added new multiple dataset merge function ('multimerge') that supports merging (i.e. contatenating) data sets based on a file list and structure name list to support merging structures stored in multiple directories as well as structures contained in compound files; also supports 3 merge types (listed order, study date order, non-overlapping study date order) and various metadata merge options
  • added support for the new 'multimerge' function options to 'ui_multimerge' and 'ui_search_data' to provide more flexible data integration options from GUI programs
  • changed default startup mode of 'ui_search_data' to initially hide the Query History pane to simplify the form

1.988b to 2.00beta: Major new release, featuring significant enhancements to functions for merging multiple structures ('ui_multimerge', 'datamerge', 'merge_by_date'), improved support for plotting in MATLAB R14/v7 ('plotdata', 'plotgroups'), better standardization of command-line help, and major new functions and GUI dialogs, including:

  • added full support for indexing and searching GCE Data Structures stored in any number of local directories and files, as well as data sets stored on the GCE web server (i.e. publicly-accessible data catalog files and data portal files); new functions include: 'search_index' for indexing data sets, 'search_data' for parsing query statements and performing index searches, and support functions 'recurse_files' to recursively search directory trees to match files and 'split_codes' to split delimited lists of coded values to produce hash tables
  • added new mapping tools to support bounding box creation and site lookups ('get_bbox' and 'bbox2sites'), and now include the GCE Mapping Tools as an integral part of the GCE Data Toolbox
  • added a major new GUI dialog (GCE Search Engine, 'ui_search_data') for building and managing search indices, performing queries, managing cumulative search results, and performing analyses and batch operations on data sets in the search results list (e.g. view metadata, view/edit data set, create line/scatter or map plots, batch copy or export data sets in specified text or MATLAB formats with user-selectable metadata formats, header formats, and QA/QC flag handling); the search index, query fields, query history, result set and program options can all be saved and reloaded, allowing users to create any number of customized search sessions for mining data stored in GCE Data Structures; support is also built in for retrieving up-to-date indices of all publicly-accessible GCE data sets (both accessioned data in the GCE Data Catalog and ancillary data in the GCE Data Portal) and automatically logging in, retrieving, and caching data from the GCE web site, allowing users to seamlessly query and analyze both local and GCE data holdings.
  • added support for multiple column deletions, selective export of data columns to the base MATLAB workspace, and selective import of metadata fields from data structures or standard metadata templates to the Data Editor (ui_editor) 

1.988 to 1.988b:  corrects a problem with the startup script ('startup.m') which was inadvertently compiled in 1.988; functionality of the data editor ('ui_datagrid') improved by adding support for resizing the grid when the figure is resized, saving layout settings (alignment option, grid size, logging option), and viewing records with duplicated values in all columns or non-data columns (i.e. for inspection and editing/removal of duplicates); added new function 'dupe_index' for computing indices of duplicate rows without sorting the structure

1.987 to 1.988: minor enhancements to 'ui_plotdata' and 'ui_plotgroups' adding support for double-clicking to select plot columns, minor bug fixes in 'nullflags' and 'flags2cols', updated USGS metadata templates and GCE geographic information tables, and added new function 'dateplot2template' (similar to dateplot2html, except adds support for adding thumbnail plot table to a pre-existing html template for easier integration with site web pages)

1.986 to 1.987: minor enhancements to 'ui_editor', 'ui_visualqc', 'aggr_stats', 'aggr_bindata'; several new functions and GUI dialogs added, including:

  • added new functions for automatically summarizing QA/QC-flagged and missing values by date as data anomalies in the metadata ('add_anomalies' and 'daterange2str'), with menu/GUI support added to 'ui_editor' and 'ui_visualqc'
  • added new dedicated functions for performing date/time aggregation at specified intervals and automatically pre-pending appropriate interval prefixes (i.e. 'Yearly', 'Monthly', 'Daily', 'Hourly') to names and descriptions of derived statistical attributes and to data set titles ('aggr_datetime' and 'ui_aggrdatetime'; based on 'aggr_stats')

1.985 to 1.986: minor bug fixes in 'unitconv', 'ui_datagrid'; minor enhancements to 'nullflags', 'cullflags', 'plotgroups', 'ui_plotsgroups'

1.984 to 1.985: minor bug fixes in 'ui_datagrid' and 'ui_dataflag' to correct problems returning manually-assigned QA/QC flag results to the editor; minor bug fix in 'mergemeta' to correct errors meshing metadata containing different coded lists of QA/QC flags

1.983 to 1.984: minor bug fix in several GUI functions to support dependent file checking when .p files are present in addition to .m files (i.e. for proper functionality when running the compiled toolbox programs)

1.982 to 1.983: minor enhancements to 'ui_editor', including addition of new date/time functions for adding a year day column calculated from a serial date/time column or string date and time columns, addition of a new function 'plotwidgets' to add GCE plot menus, plot labels and toolbars to any MATLAB figure (i.e. provides stand-alone functionality for plots not generated by the GCE Data Toolbox).

1.981 to 1.982: enhancements to 'dateplots' to support weekly and daily plots; addition of new function 'dateplot2html' and corresponding GUI dialog 'ui_dateplots_html' for generating complete HTML web pages along with date interval images for creation of multi-column time series image displays on the web, with support for automatic generation of thumbnail images hyperlinked to larger images

1.98 to 1.981: major bug fix in 'datemerge' to address potential column ordering problems, minor bug fixes in 'cols2flags'; added new function 'dateplots' for generating multiple time series plots at a specified interval in various bitmap and vector image formats (note: supercedes 'monthplotsfig' and 'yearplotsfig' providing more graphics format and resolution options and adding support for decadal plots)

1.973 to 1.98: minor bug fix in 'aggr_sums' to correct mishandling of flagged columns, several new functions and program enhancements, including:

  • addition of new function 'negative2zero' for setting negative values in specified columns to 0, and logging the changes to the data anomalies section of the metadata (useful for zeroing out dark noise in light sensors prior to summation and to avoid excessive QA/QC flags)
  • addition of new function 'insertrows' for inserting data rows into specified columns of an existing data structure and padding corresponding rows in omitted columns with empty values
  • enhancements to 'ui_editor' and 'edit_importfilters' to support nesting of custom import filters in menus
  • enhancements to 'ui_datagrid' to support user-customized alignment of cell contents for editing
  • enhancements to 'ui_importfilter' to support automatic generation of custom text file import functions (.m files) based on user-specified import parameters for a text file, allowing simple generation of MATLAB programs for subsequent use and customization; new functions are also automatically added to 'imp_filters.mat' for display on the editor 'Import Data' menu
  • enhancements to 'imp_ascii' to improve automatic parsing of simple delimited ASCII files
  • enhancements to 'csi2struct' to support a wider variety of CSI data array formats

1.972 to 1.973: major bug fix in 'datamerge' to prevent loss of manually-assigned QA/QC flags when a structure is appended to another structure without any manually-assigned flags in corresponding columns (i.e. no 'manual' criteria); minor bug fix in 'ui_plotgroups' to properly pass selected flag display options to 'plotgroups'; several program enhancements, including:

  • addition of new GUI dialog ('ui_copyflags') for propagating composite QA/QC flags from one or more columns to one or more dependent/calculated columns, to augment or replace existing flags
  • modified 'add_calcexpr' to support automatically adding calculation descriptions to the Data/Calculations metadata field
  • added support for date/time string axis limits in 'ui_axislimits'
  • minor enhancements to 'fetch_usgs' and 'parse_usgs' to accomodate additional file format and flag variations in USGS data files
  • minor enhancements to 'listcols', 'flags2cols'

1.971 to 1.972: minor bug fix in 'dec_places'; improved handling of automatic numeric assignments in 'imp_filter' and 'imp_ascii' for data files without metadata templates; addition of URL parameter to 'fetch_usgs' and corresponding input field in 'ui_harvestusgs' to accommodate USGS URL changes; addition of a new function ('update_data') to support command-line updates to data column contents with automatic logging of data changes to the metadata (similar functionality to 'ui_datagrid' for non-GUI applications)

1.97 to 1.971: minor bug fix in 'exp_matlab' to accomodate numeric column names in structures; enhancements to 'ui_unitconv' to limit standard unit conversions to only those matching the original column units; addition of GUI dialog for harvesting real-time, daily or finalized data from the USGS web server

1.962 to 1.97: minor bug fixes in 'ui_editor', 'flags2cols', 'ui_datagrid', 'imp_ascii'; minor display enhancements and cosmetic changes to GUI dialogs; several major program enhancements, including:

  • addition of custom ASCII file import dialog ('ui_importfilter') to ease parsing of files with non-standard headers and missing value codes (includes a file viewer with automatic format string and column title parsing functions)
  • addition of data anomalies metadata field to visual QA/QC ('ui_visualqc') for easier documentation of manual flagging operations
  • added support for automatic calculation of flagged and missing value total columns to grouped and binned statistics functions ('aggr_stats', 'aggr_bindata') and GUI dialogs ('ui_aggrstats', 'ui_bindata'), with support for flagging criteria based on these totals -- this allows flags to be propagated to columns in derivative data sets
  • inclusion of new functions for harvesting data from the USGS web site ('fetch_usgs' - requires MATLAB 6.5/R13), parsing data in Campbell Scientific array-based data loggers ('csi2struct'), and formatting climate data for submission to ClimDB/HydroDB LTER databases ('exp_climdb')

1.961 to 1.962: minor bug fix in 'imp_ascii' to prevent column numerical type descriptors in ASCII file headers from been overridden by automatic value inspection 

1.96 to 1.961: major bug fix in 'joindata' involving joins on multiple unsorted key fields, minor bug fixes and enhancements to 'plotdata' and 'plotgroups' to improve plots based on string date values, improvements in 'edit_unitconv', updates to cached geographic information stored in 'gce_locations.mat' and 'gce_coords_all.mat' to reflect new site designations, additional flag-handling options added to 'exp_matlab', corrections to Q/C flag references in documentation

1.951 to 1.96: bug fixes and keyboard accelerators (hot keys) added to the data structure editor (ui_editor); added a new generic import filter for USGS real-time data from any station in the US (parse_usgs); added major new geographic lookup functions, including

  • function for adding a study site column based on matching values in geographic coordinate columns to site polygons in the GCE geographic database (add_studysites)
  • function for adding geographic coordinates based on matching values in station or location code columns to entries in the GCE geographic database (add_stationcoords)
  • function for importing study site metadata from the GCE geographic database to support automatic lookups based on site code, station or location code, or geographic coordinate columns (add_sitemetadata)

1.95 to 1.951: bug fixes and major speed enhancements in 'joindata', enhancements in 'ui_joindata', minor changes to color selection order in 'ui_plotdata'

1.902 to 1.95: major additional features added, including:

  • added support for selective removal of individual values or entire rows based on specified QA/QC flags present in specified data columns, with optional logging to the 'Data|Anomalies' metadata field to aid in documentation of quality control operations (nullflags, cullflags, ui_clearflags)
  • added support for propagation of QA/QC flags assigned to values in one or more columns to values in other (dependent) data columns (copyflags)
  • added support for automatic calculation of study date metadata descriptors based on date/time information in the data set (add_studydates)
  • added an option to rename data editor windows based on the data set title or user-specified strings to prevent confusion when multiple editor windows are opened simultaneously (ui_editor)
  • added a function to automatically synchronize path caches of all open editor windows to minimize directory changes in file dialogs (syncpath)
  • enhanced ASCII export and statistics reports dialog to support selective flag removal and not prompt for file names before opening (ui_exportasc, ui_statreport)
  • minor bug fixes

1.901 to 1.902: syntax error corrected in 'struct2xml'

1.90 to 1.901: minor bug fix in 'colstats', added support for new missing value codes used by USGS to 'imp_usgshudson', 'parse_usgshudson' and 'imp_usgsdoctortown'

1.823 to 1.90: major additional features added, including functions for performing statistical analysis on binned data with optional pre-aggregation by one or more grouping columns (aggr_bindata and ui_bindata), and retrieving the top and bottom values from a vertical profile data set (e.g. CTD casts) based on a depth or pressure column after pre-aggregation by one or more columns (aggr_topbottom and ui_topbottom); minor changes to grouped statistics (aggr_stats) and column statistics (colstats) functions to prevent calculation of dispersion statistics (SD, SE) on groups containing a single data value; minor changes to Tools menu arrangement in the structure editor (ui_editor)

1.822 to 1.823: enhancements to 'add_datecol', 'add_datepartcols', and unit conversion table to support automatic time zone recognition/conversion based on datetime column units and descriptions, minor enhancements to 'parse_37sm' and 'imp_gcesonde' to support Sea-Term file type variants, minor changes to 'parse_usgshudson', 'imp_usgsdoctortown', and 'csi2struct' to support datetime conversion to GMT

1.821 to 1.822: minor bug fix to prevent errors with non-auto-scaled plots in 'plotdata', enhancements to 'parse_37sm' to support SBE file format variants, dedicated CT's, and addition of 'Pump' column to indicate whether the sonde was equipped with a pumped flow cell

1.82 to 1.821: added support for performing visual QA/QC flagging on auto-scaled plots to 'ui_visualqc'; added support for reversing plot axes directions to 'ui_axislimits' (called by 'plotbuttons') 

1.81 to 1.82: significant QA/QC-related enhancements, including addition of support for multi-column flag criteria (dataflag), new functions for flagging dissolved oxygen concentrations based on deviation from temperature and salinity-dependent oxygen saturation (flag_o2saturation), support for conversion of text columns to QA/QC flags for corresponding data columns to support synchronization of manually-edited flags converted to text columns back to the flag array of the corresponding data column (cols2flags), and support for data value editing added to manual QA/QC utility (ui_dataflag)

1.803 to 1.81: minor bug fixes and enhancements in 'textfile2cell', 'ui_axislimits', 'ui_flagdefs', 'ui_qcflags', 'ui_aggrstats', 'ui_editor', 'ui_exportasc'; additional features added to 'ui_template' and 'ui_editmetadata' to facilitate importing metadata and descriptors from existing structures to create templates; added data restriction option to 'ui_mapdata'; added support for automatic conversion of text date strings (e.g. 12/1/2002) to numerical Julian date when entering date restrictions on datetime columns in 'ui_plotdata', 'ui_plotgroups', and 'ui_mapdata'

1.802 to 1.803: additional minor bug fixes in 'ui_editor',  'imp_datastruct', 'ui_flagdefs', 'ui_qcflags', 'ui_visualqc', 'ui_multimerge'; enhancements to 'add_datepartcols'

1.801 to 1.802: minor bug fixes in 'ui_editor', 'imp_ascii', 'ui_title'

1.80 to 1.801: minor bug fixes in 'ui_plotdata' and 'ui_datagrid'

1.713 to 1.80: major QA/QC-related enhancements, including addition of GUI flag criteria ('ui_qcflag') and flag code definition ('ui_flagdefs') editors and support for visual QA/QC flagging on multi-parameter and grouped scatter/line plots (flag values by left clicking/dragging with the mouse, clear flags by right clicking/dragging); also added menu accelerators (control key support) to 'ui_datagrid', enhanced the plot toolbar to add support for manual axis limit selection and custom date/time axis formats, and added a 'GCETools' menu item with support for exporting figures in various graphical formats and resolutions (without the toolbar controls visible).

1.712 to 1.713: bug fixes in 'joindata' and 'flags2cols', improved path caching in structure editor 'ui_editor', major enhancements to date axis formatting function 'dateaxis', allowing user-customized date/time formats for plots, added a new import filter for SINERR hydrographic data

1.711 to 1.712: minor enhancements to 'querydata'

1.71 to 1.711: major speed enhancements and new 'forced join' option added to join function and dialog ('joindata' and 'ui_joindata'), added support all metadata styles to matlab export function 'exp_matlab', and updates to metadata styles are automatically synchronized in all editor menus

1.70 to 1.71: minor bug fixes and enhancements to data editor (ui_datagrid), improved text field sorting, addition of a GUI editor dialog for adding and editing metadata style definitions (ui_metastyle)

1.632 to 1.70: several major enhancements to the toolbox --

  • Unit conversions are now applied to the QA/QC flag criteria along with the column data to prevent spurious flagging of data values after conversion (note: custom function-based criteria are not converted to prevent inappropriate parameter modifications)
  • Data import filters and metadata templates used by the toolbox are now fully modular, allowing users to define their own custom filters or templates that will be listed in editor menus alongside the default filters and templates. GUI editors are provided for managing filters and templates (listed under the 'Misc' menu), and sample custom import filters are available on request.

1.631 to 1.632: minor bug fixes, enhancements to ui_datagrid, ui_template to support creation of new metadata templates

1.63 to 1.631: minor bug fixes in editor menu commands (ui_editor)

1.62 to 1.63: bug fixes, enhancements to multiple-y plot dialog (ui_dataplot), addition of new function for importing ASCII data under format control with missing value substitution (imp_filter)

1.61 to 1.62: *major* speed improvements (5-10x) in delimited ASCII export for structures containing string columns or formats specifying missing value substitutions (exp_ascii), enhancements to unit conversion dialog (ui_unitconv)

1.60 to 1.61: bug fixes in multiple-merge dialog (ui_multimerge), enhancements to multiple-y plotting dialog (ui_plotdata) and plot-by-month batch function (monthplots)

1.50 to 1.60: improved support for metadata templates in the editor by allowing templates to be applied by name and edited and created using a GUI dialog (ui_template), added generic batch import function (batch_import)

1.40 to 1.50: added support for manually editing QA/QC flags for a column and setting numerical precision according to display precision setting, improved merge and join functions, miscellaneous bug fixes and enhancements

 Function List GCE Data Toolbox

This material is based upon work supported by the National Science Foundation under grants OCE-9982133, OCE-0620959 and OCE-1237140. Any opinions, findings, conclusions, or recommendations expressed in the material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.