Legacy Trac Support Tickets (non-working links)

Reported by wsheldon, 8 weeks ago.

A matrix dimension mismatch error results when adding a multi-column array of flags to a data set column that does not have any flags assigned.

Reported by wsheldon, 8 years ago.

Workflow scripts must currently be developed and run outside of the GUI data editor. Adding a GUI workflow/script dialog and a database of parameterized scripts (similar to the EEM Calculator in the Fluorescence Toolbox) would significantly aid extension of the toolbox by end-users.

Reported by wsheldon, 7 years ago.

In order to better support import/export plugins, a specification and tools are needed to define and manage function arguments and prompt for and validate arguments (by type) at runtime.

Reported by wsheldon, 7 years ago.

Adding support for the LTER Data Portal API to ui_search_data.m would enable retrieval of EML-described data using fetch_eml_data.m. This would provide seamless searching for GCE and LTER data and facilitate use of the toolbox for LTER synthesis projects.

Reported by guest, 9 years ago.

The current plot types are limited to line/scatter and map types. Box and whisker statistics plots and bar plots would be useful for categorical and other data.

Reported by wsheldon, 9 years ago.

The USGS station harvest list is currently managed independently of ClimDB as a local data file at GCE, but it would simplify adding and removing stations if this list was dynamically retrieved from the ClimDB database via a web service or other query. Some local information will still need to be cached (date of last finalized data, historic discharge maxima), but the list could be synchronized with ClimDB preceding each harvest to add/remove stations.

Reported by wmsheldon@earthlink.net, 9 years ago.

Adding support for generating R scripts to load exported data into a data frame would facilitate advanced statistical analysis. Existing EML2R XSLT sheets developed by TERN and John Porter could be leveraged for this purpose.

Reported by adam.kennedy@oregonstate.edu, 7 years ago.

LoggerNet? has the ability to send data directly from the LoggerNet? server to a database. It would be great to develop this LNDB as an import source to the GCE-Toolbox.

Reported by adam.kennedy@oregonstate.edu, 3 years ago.

daterange2flags() and correctdrift() could be enhanced to allow date range flagging on grouped data, for example, siteID.

Reported by georam@nmsu.edu, 6 weeks ago.

In line 84 of the file add_year_yearday_hours.m there was an error, the last component of the formula, the one that is transforming the seconds to hours, is incorrect. It is diving by 360 instead of 3600.

Reported by wsheldon, 6 years ago.

For long-term data it is often impractical to generate limit check rules because expected ranges vary over the year, so many qualified rules would need to be managed to effectively Q/C the data. In these cases it would be simpler to manage the Q/C ranges in a separate reference data set and then reference this data set once in a rule. The data set would need to be specified, along with column to look up and column(s) containing the limit check criteria. This use case was reported by Aaron Stephenson (NTL-LTER).

Reported by wsheldon, 6 years ago.

The ui_plotdata.m GUI requires that plot parameters be selected before adding the variable to the selection list, which is not intuitive for new users. This GUI should be redesigned to allow variables to be selected and then plot parameters assigned and revised separately. This issue was reported by John Porter (VCR-LTER) and Mary Martin (HBR-LTER).

Reported by wsheldon, 11 months ago.

A Wiki page is needed to walk people through the steps to define and manage custom import filters.

Reported by wsheldon, 11 months ago.

A Wiki page should be added to describe use of the harvest timer management tools.

Reported by wsheldon, 11 months ago.

A Wiki page should be added describing how to register new menu items using extensions.m

Reported by wsheldon, 5 weeks ago.

When new Q/C criteria is applied using update_attributes.m the update fails with an invalid data structure error.

Reported by wsheldon, 9 years ago.

Currently, data can only be aggregated by year, month, day or hour in ui_aggrdatetime. Options should be added for sub-hourly (e.g. 15 min) and weekly aggregation using ISO or conventional week numbering schemes. This will require re-factoring the date parsing logic and adding derived columns for hour fraction or week number column to datasets for aggregation.

Reported by wsheldon, 6 years ago.

The Visual Q/C tool (ui_visualqc.m) restricts flag/unflag operations to one variable at a time to prevent misapplication of flags when trendlines overlap. However, synchronizing flag edits across multiple variables could be useful in some cases. An option could be added to synchronize flag/unflag operations across all plotted variables to support this use case.

Reported by wsheldon, 5 years ago.

When core/pad_date_gaps.m is called, any date part columns (Year, Month, Day, ...) are automatically regenerated to fill in the NaN/empty values after filling. In cases where columns of those names are included that don't derive exactly from the primary date/time column inappropriate overwriting may occur. An option should be added to skip regenerating date part columns for those cases.

Reported by wsheldon, 5 years ago.

The statistical summary functions only list total flags assigned, and the harvest_check() function returns a formatted string summarizing flag assignments. A function that generates a structured report of individual flags assigned by variable would be useful in debugging flag rules and assessing station status.

Reported by wsheldon, 5 years ago.

Google Earth KML files are frequently used to exchange or enter geographic coordinates. It would be useful to have a KML parser in the toolbox for getting site names, descriptions and coordinates from KML files.

Reported by wsheldon, 4 years ago.

More options should be provided for managing multiple flags assigned to data values. Options to remove redundant flag assignments and remove all but the first-assigned (highest priority) flag would be useful.

Reported by wsheldon, 2 years ago.

A Wiki page should be added describing how to add calls to harvest_dashboard.m to new or existing data harvesting workflows.

Reported by adam.kennedy@oregonstate.edu, 21 months ago.

Adds data link from the dashboard view to the data source file and the data source index page.

ie. from: https://andrewsforest.oregonstate.edu/sites/default/files/lter/data/weather/portal/CENMET/dash/233_a_105/index.xml

to: https://andrewsforest.oregonstate.edu/sites/default/files/lter/data/weather/portal/CENMET/data/cenmet_233_a_5min_2017.html

and to: https://andrewsforest.oregonstate.edu/sites/default/files/lter/data/weather/portal/CENMET/data/index.html

Reported by wsheldon, 9 months ago.

The EMLUnitDictionary.mat file distributed with the toolbox should be updated after version 2.2 is officially released to include all the new standard units and the UDUnits synonym field contents. A utility function and XSLT transform should be added for parsing the .xml file to simplify future updates, preserving any custom units.

Reported by wsheldon, 5 weeks ago.

When calling the function date2range2flags.m and specifying a flag that isn't already defined in the data structure metadata no definition is created, preventing the flag from show up in Q/C dialogs. A flagdef argument should be added to support registering the flag definition if necessary to prevent this situation.

Reported by adam.kennedy@oregonstate.edu, 5 weeks ago.

Let's say we have a binary column value that indicates when a sensor is online (1 for online, 0 for removed). Then the field tech arrives at the station, they'll remove the sensor then go to the logger and manually type in a "0" to indicate the sensor has been removed. This tells the robots that the sensor has been removed. Because there is a delay between the time the sensor is removed and the "0" value entered, the post processing will miss a couple values that should be flagged.

Similarly, when the sensor is replaced, it will go through a short acclimation period where it will transfer from the temperature of the truck to the outside air temperature, for example.

Both of these situations could be addressed by adding two arguments to the num_replace() - 'pre-flag' and 'post-flag' duration after a criteria is met and the flag "Q", let's say. The additional flag argument is important because the values may be valid but will need closer inspection.

%AIR_150 and associated to flag sensor removal col = name2col(data,{'AIRTEMP_MEAN_150_0_04'});

if ~isempty(col) [data,msg] = num_replace(data, ... {'AIRTEMP_MEAN_150_0_04','AIRTEMP_MAX_150_0_04','AIRTEMP_MIN_150_0_04','DEWPT_MEAN_150_0_04','DEWPT_MIN_150_0_04','DEWPT_MAX_150_0_04' ... 'RELHUM_MEAN_150_0_04'}, ... 'AIRTEMP_150_DIAG = 0', ... NaN, ... 1, ... 'M' ... ); end

Reported by wsheldon, 5 weeks ago.

The binned statistics function aggr_bindata.m adds columns for [var]_Bin_Top, [var]_Bin_Middle and [var]_Bin_Mean. An additional column fro [var]_Bin_Bottom should be added to provide more complete options for characterizing the aggregation interval.

Reported by wsheldon, 2 weeks ago.

When the function flag_novaluechange is used with default threshold values of 0,0 and a large window (e.g. 5), all values in a typical data series are flagged. This behavior appears to be related to floating-point math issues with comparisons, because changing thresholds to a very small number like 0.00001 corrects the over-flagging. A more comprehensive fix should be added.

Reported by wsheldon, 3 months ago.

When calling daterange2flags.m and specifying specific columns to flag (cols <> []) the flags are assigned to the unspecified columns and not to those specified.

Reported by wsheldon, 8 weeks ago.

Applying a new metadata template could be an effective way to map a data set to a controlled attribute vocabulary. Allowing units to be converted to the target vocabulary units when the template is applied would streamline this use case. The apply_template.m function should support an option to convert units if original and template units don't match instead of just returning the original units and a warning.

Reported by wsheldon, 3 months ago.

The options for tallying Q/C flags and setting Q/C rules for summarized data do not discriminate based on flag semantics, so it may not be desirable to include informational flags that don't indicate problematic values (e.g. C = corrected, V = verified, etc). Having an option to clear specific flags in addition to clearing flagged values prior to aggregation would prevent inappropriate flagging of derived values in these cases.

Reported by wsheldon, 2 months ago.

Cell-to-character conversion errors occur when an empty field is present in the structure.

Reported by wsheldon, 2 months ago.

The parser for Schlumberger groundwater logger files (imp_schlumberger.m) includes specific checks and workflow steps for GCE-LTER Cera-diver, Baro-diver, and CTD-diver instruments. The parser should be more generic, and use GCE-specific metadata templates and workflows to apply project-specific information

Reported by wsheldon, 8 weeks ago.

A general function should be available for mapping/converting data sets to match a controlled vocabulary of attribute names, units and qualifier flags, similar to exp_climdb.m. The function should accept the filename of a mapping data set (e.g. Var_Original, Var_New, Units_New) and array of flags to match. Unit conversions should be performed and unmapped variables and flags should be cleared.

Reported by wsheldon, 7 weeks ago.

An error occurs when opening an instance of the Fetch ClimDB Data dialog (gui/fetch_climdb.m) if a prior instance of the dialog is already open. The existing dialog should just be given focus in this case.

Reported by wsheldon, 7 weeks ago.

Existing flag columns in data sets retrieved from ClimDB are persisted as columns with a space between "Flag" and the described column name (e.g. "Flag AirTemp?"), preventing conversion to native GCE Data Toolbox flag arrays. The flag column prefix should be changed to "Flag_" on import to facilitate this conversion.

Reported by wsheldon, 7 weeks ago.

When the unit conversions database is opened for editing using edit_unitconv.m, the table is always sorted by unit1 (source units). But it is sometimes useful to sort by destination units (unit2) when checking for conversions that need to be added. An option to support this use case should be added.

Reported by wsheldon, 12 months ago.

Sometimes opening the Correct Drift dialog (gui/ui_correct_drift.m) take a long time (several seconds) to complete.

Reported by wsheldon, 11 months ago.

When a sonde file from an undefined location is parsed using imp_gcesonde.m and the site cannot be found in parse37_sm.mat, the variable locstr is undefined and a runtime error occurs.

Reported by anonymous, 2 years ago.

The ability to assign observed bad values as NaN and assign an 'M' flag would speed removing impossible values from a dataset. An option to convert a value to NaN and assign a custom flag 'M' in this case might make the Visual Qc/QA tool more efficient.

Reported by anonymous, 2 years ago.

Function to allow flagging of non-sequential individual dates. This would be similar to flag_daterange() but allow a more succinct method to add dates without using a date pair range. AK

Reported by adam.kennedy@oregonstate.edu, 22 months ago.

Function to export QC rules from one GCE ds and import those rules into another GCE ds. This is different than the export metadata since the other GCE ds may have different Variable names. This new function would match the QC rule on the Name field in the new data structure.

This would allow users to pass around a single GCE ds, modify its QC rules independent of the original ds. Once modifications are complete, the QC rules may be exported and then imported into the original ds, thus updating the original ds's QC rules.

Reported by wsheldon, 12 months ago.

Some sensor or logger program code changes can lead to unit or scale shifts that require correction by multiplying (or dividing) by a correction factor. A multiplier option should be added to correct_drift.m (and ui_correct_drift.m) to accommodate this use case.

Reported by wsheldon, 11 months ago.

From Adam Kennedy: Is anyone using the multimerge() with a e:\path\to\matfiles\*.mat in place of the filelist cell array?

Reported by anonymous, 11 months ago.

Add file compression option to exp_ascii .csv to reduce file size of exported GCE ascii file. Akennedy

Reported by wsheldon, 9 months ago.

When core/multimerge() is called and the file list variable only contains a single structure reference the function returns an empty structure and error. To support workflows that automatically concatenate files in a directory it would be better to return the single structure as output instead, with a warning in msg. That would support blindly concatenating files in a directory without first inspecting the list to see if it contained more than one file.

Reported by wsheldon, 8 months ago.

The ClimDB/HydroDB harvest dialog should include station type (Meteorological, Gauging, etc) to indicate what data are being retrieved. This will require changes to the XSLT for parsing the stations report and the GUI (gui/ui_fetch_climdb.m).

Reported by wsheldon, 8 months ago.

The core/cols2flags.m function is supposed to use column adjacency to determine which flag columns to convert to value flags when redundant column names are present in a data set. However this is not working, and some flag columns are not processed. This is a poor data management practice, but should be handled as described and then sorted out elsewhere in the workflow.

Reported by wsheldon, 6 months ago.

The function core/multimerge.m generates a title for the merged structure containing a list of the base filenames incorporated ("Merged data from x, y, z, ..."), requiring manual updating by the user or workflow. More control should be provided by supporting retention of the first structure title for adding data to a cumulative file or adding a custom title when the function is called.

Reported by wsheldon, 6 months ago.

Many services are now requiring TLS 1.2 for secure server interaction. Procedures to enable TLS 1.2 protocol in Java network libraries using a java.opts file should be documented.

Reported by wsheldon, 4 months ago.

When options to generate flag columns for all data/calculation columns plus any non-data columns with flags assigned are specified when calling gceds2eml() the flag columns are not documented in the EML. EML-described data packages therefore fail PASTA congruency checks and cannot be uploaded to EDI.

Reported by asapp, 16 months ago.

When large gaps exceeding the threshold value are present in a data set and all the other interpolated values are singleton missing values, the interp_missing.m will fail to interpolate any values and return the original structure with a warning. This appears to be due to the test for any contiguous runs of interpolated values within bounds. The code should check for this contingency.

Reported by wsheldon, 19 months ago.

The 'Resize Plots' option under the 'Special' or 'GCETools' menus added by support/plotmenu.m fail and throw an error using recent versions of MATLAB due to the deprecation of the selectmoveresize function and replacement with plotedit.

Reported by wsheldon, 18 months ago.

The function support/fetch_itis.m returns an error referring to an invalid variable 'html'.

Reported by wsheldon, 18 months ago.

When the Correct Sensor Drift GUI (gui/ui_correct_drift.m) is initialized the starting and end dates for corrections should default to the full date range of the data set for global corrections.

Reported by wsheldon, 17 months ago.

The correct_drift.m function assigns C flags by default to corrected values, including NaN values that were not modified. NaN values should be excluded when flags are assigned.

Reported by wsheldon, 17 months ago.

When missing criteria are included in qcrules when aggr_stats.m is called, flags can be assigned to all values in sparse variables. An option to check for this condition should be added.

Reported by wsheldon, 14 months ago.

An option should be added to core/mergemeta.m and core/merge_by_date.m to skip merging metadata after merging the data columns for parity with multimerge.m and to support cases where multiple data sets are manually merged (e.g. in different directories). The options should also be added to gui/ui_editor.m.

Reported by wsheldon, 14 months ago.

The USGS NWIS system includes inline flags in place of numeric values during station recalibration or maintenance events that prevent parsing in MATLAB. The parsers/parse_usgs.m function has to remove these values using regex to properly parse the data. Recently USGS started including spelling variants of some codes in rdb files (e.g. DIS instead of Dis) that breaks the parsing. The regex logic should be changed to allow case-insensitive matching.

Reported by wsheldon, 23 months ago.

When a data file is imported and a metadata template is specified that does not contain complete attribute descriptors, the resulting data structure can contain columns for Numerical Type = 'unspecified'.

Reported by wsheldon, 22 months ago.

The ClimDB/HydroDB server at the LTER National Communications Office was revised in early 2017 to require HTTPS for initiating harvests. The workflows/publish_climdb.m function defaults to HTTP, and the resulting 401 redirect error breaks the function.

Reported by wsheldon, 6 years ago.

The EMLdataset2mfile.xsl stylesheet should be extended to include all content supported by the GCE toolbox, including project and associatedParty.

Reported by wsheldon, 5 years ago.

Over-precise time values that differ between stations can often interfere with automatic date/time joins (e.g. data logged at 10-15 minutes, but timestamps vary in seconds). This issue can be overcome by generating date component columns, deleting unneeded columns, then re-generating the date or timestamp column to revise the precision. A single function and corresponding GUI menu item that performs these steps would be a useful addition to the toolbox, and avoid issues determining which columns to use when calling add_datepartcols() and then add_datecol() from the GUI.

Reported by Adam.Kennedy, 3 years ago.

Campbell Scientific data logger compute hourly statistics on values differently than normal database functions. For example, an hourly 1100 average from a CSI data logger output table would include the values sample between 1001 and 1100, if collected at 1 minute sample frequency. This option would be an enhancement to the current GCE aggregation method such that it would allow a direct comparison between GCE averaged values and CSI data logger averaged values.

Reported by anonymous, 2 years ago.

When there are many QC rules used for a variable, the amount of text dominates the Dashboard view cell size and becomes unreadable.

Allow the user to disable QC rule display on Dashboard or add "Show QC" link to the dashboard layout.

Reported by wsheldon, 2 years ago.

From Adam Kennedy at AND: "...This creates a lot of qc rules in the import templates. This is fine, however, the view on the harvest dashboard and data description page become cluttered with this new information. Having an option to remove the QC rules and QA/QC Flags for variables would help, tweaking the formatting on the data description page would help, adding an extra link to get to the qc rules/flags would help."

Reported by wsheldon, 22 months ago.

The file /demo/data_submission_template.xls is outdated. A new version used at GCE-LTER (data_submission_template.xlxs) includes drop-down menus for attribute metadata fields and other refinements. The version in the toolbox distribution should be replaced.

Reported by wsheldon, 22 months ago.

NCAR has developed a light-weight cloud-based data service called CHORDS that uses simple HTTP GET commands for uploading and retrieving data. A sample parser and harvest script should be added for demoing this functionality.

Reported by wsheldon, 22 months ago.

Including code comments along with Q/C criteria (%xxx) can cause the interpreter to error and therefore skip the rule in certain circumstances. This option should be explicitly supported for documenting rule rationale.

Reported by wsheldon, 22 months ago.

Most email servers now require authentication for SMTP, but by default MATLAB used unauthenticated SMTP for sendmail.m, requiring a relay. Java can be used to handle the authentication as documented by Mathworks, but this requires support functions for usability. These should be added to the toolbox.

Reported by wsheldon, 2 years ago.

The LTER ClimDB/HydroDB website was revised to use the HTTPS protocol, so parsers/fetch_climdb_info.m, parsers/fetch_climdb_data.m and gui/ui_fetch_climdb.m need to be revised to use this protocol for climhy.lternet.edu to avoid permanent redirect errors.

Reported by wsheldon, 2 years ago.

From Jennifer Morse:

We use campbell loggers at most of our sites (as I’m sure many of you do). The timestamp from the campbell logger’s refers to the data before the timestamp. As in 13:00 refers to the data from 12:50 to 13:00.

It seems like the toolbox is set up to consider the timestamp as the data in front of the hour. As in, 13:00 refers to the data from 13:00 to 13:10?

This seems to be creating an issue in our data set when we use the toolbox to aggregate ten minute into hourly data, where the toolbox hour 1:00pm aggregated from ten minute data is actually referring to data from 12:50pm to 1:50pm.

I realize to some extent this is just semantics… but is seems awkward to leave this as is for users.

Wondering if there were any ideas on how to deal with this in the toolbox…. other than moving each time stamp forward ten minutes before processing the data?

Reported by wsheldon, 3 years ago.

When adding a calculated column using core/add_calcexpr.m or gui/ui_calculator.m there should be an option to automatically add composite flags from all the referenced data columns as manually-assigned flags. Flags can be propagated manually, but an automatic option would save effort and ensure Q/C information isn't lost.

Reported by wsheldon, 3 years ago.

When metadata is imported from an existing data structure in the Data Set Editor (ui_editor.m) using File > Import Metadata > Existing Data Structure, the Documentation Only and Selected Documentation Only options are ignored and column attributes are imported as well.

Reported by wsheldon, 3 years ago.

When a space-delimited array is specified as the value for "IN ARRAY" or "NOT IN ARRAY" instead of a comma-delimited array an error is generated about multi-line text and the textbox disappears.

Reported by wsheldon, 2 years ago.

The USGS NWIS will start requiring HTTPS protocol be used for all web queries beginning Dec 15 2016. The USGS data retrieval functions in the toolbox (fetch_usgs.m and fetch_usgs_dates.m) will need to be revised to take this change into account, because MATLAB's urlread() and urlwrite() functions will not follow HTTP 301 redirects.

Reported by wsheldon, 3 years ago.

When multiple columns are selected and the user selections File > Clone Data Structure > Selected Columns a listbox is displayed for choosing columns with none of the prior selections highlighted. This is confusing since other editor menu options for selected columns use the main column list multiple selections.

Reported by wsheldon, 3 years ago.

From Adam Kenedy:

Quick question on the Dashboard views (ie. http://andrewsforest.oregonstate.edu/lter/about/weather/portal/RS/RS89/dash/99_a_105/index.xml).http://andrewsforest.oregonstate.edu/lter/about/weather/portal/RS/RS89/dash/99_a_105/index.xml).

We're setting up manual entry fields via 10key at our logger stations to verify field measurement with known/calibrated standards. Since we'll only enter values into these fields every three weeks, there will be a lot of missing or NaN values along side the regularly collected 5min values. The Dashboard functions recognize these missing values and color codes the plots as expected.

Is there a way to set the dashboard function up so that on these selected fields the color coded plot indicators aren't triggered?

Reported by wsheldon, 3 years ago.

Time series data sometimes deviate from the specified sampling interval due to clock errors, numeric date rounding issues by data loggers or software, or issues starting the logging precisely at the desired time. For example, 30 min data series may contain records at 29 min, 59 min, 1 min, etc. These deviations can complicate generating a monotonic time-series data set for time series analysis or result in excessive missing records when calling pad_date_gaps.m. A function for adjusting these minor deviations around the specified interval without re-sampling the data would be useful for these cases.

Reported by wsheldon, 3 years ago.

From Adam Kennedy, including a fix:

I have a nvarchar database field and it appears sql2gceds.m was failing when it saw this. I added it to the lookup list in that function and the import succeeded, but I thought I would point this out as there might be a better way to accomodate this field. 'nvarchar' is used by the HANDAR sonic anemometer (Vaisala WS425).

if mlversion >= 7 %use MATLAB 7+ syntax

lookup = {'bit','d','logical','discrete','double(datacol)'; ...

'boolean','d','logical','discrete','str2num(char(strrep(strrep(datacol,true,1),false,0)))'; ... 'numeric','f','data','continuous',; ... 'decimal','f','data','continuous',; ... 'real','f','data','continous',; ... 'double','f','data','continuous',; ... 'float','f','data','continuous',; ... 'integer','d','data','discrete',; ... 'int','d','data','discrete',; ... 'int identity','d','ordinal','discrete',; ... 'bigint','d','data','discrete',; ... 'smallint','d','data','discrete',; ... 'tinyint','d','data','discrete',; ... 'date','s','datetime','none',; ... 'datetime','f','datetime','continuous',; ... 'datetime2','f','datetime','continuous',; ... 'time','s','datetime','none',; ... 'timestamp','s','datetime','none',; ... 'char','s','nominal','none',; ... 'nvarchar','s','nominal','none',; ... 'varchar','s','nominal','none',};

Reported by wsheldon, 3 years ago.

In a recent legacy data integration project a Q/C flag ended up with a blank leading space before a flag code, resulting in separate entries for 'Q' and ' Q' in the list of used flags to define when flags2cols.m was used to instantiate flags as data columns. How this unsupported flag configuration happened is unclear, but this case should be mitigated in flags2cols.m to avoid generating " = unspecified" code definitions and EML validation errors.

Reported by wsheldon, 3 years ago.

The ability to generate a data set report from the search index would be helpful for reporting. Fields should include filename, accession, title, authors, study dates, file size, records, etc.

Reported by wsheldon, 3 years ago.

When changing the data type of an integer column from integer (d) to floating-point (f), the precision cannot be changed at the same time. Precision remains a 0 with an error message.

Reported by wsheldon, 3 years ago.

The fill_date_tokens.m function is useful for generating filenames for harvesting scenarios, but it's hard to use for re-harvesting legacy data because it always uses the current system time (now()) as the date basis. Adding a reference date argument would allow more general use in workflows.

Reported by Adam.Kennedy, 3 years ago.

When performing an automatic Join on Date/Time on a structure that has hundreds of variables, it is repetitive to click Add for each variable when selecting columns. An "add all" or better still click - shift - click to select all or a portion of variables in the Available columns from A and from B improves this process.

Reported by wsheldon, 2 years ago.

The support/fetch_itis.m function can no longer connect to ITIS due to HTTP 302 redirects from the HTTP address to HTTPS.

Reported by wsheldon, 2 years ago.

HBR has requested advice for changing the order of plots generated by harvest_webplots_xml.m so the most recent data are at the top of the page. This could be accomplished by an XSLT change.

Reported by wsheldon, 2 years ago.

The native MATLAB plotting tools available through the desktop work better with struct variables and table objects, because multiple columns can be selected. An option to export a struct variable to the workshop should be added to support this functionality.

Reported by wsheldon, 2 years ago.

The harvest_dashboard() function could be leveraged for generating comparative web plots for variables in any time-series data with a modified style sheet. Being able to split a compound data set into separate variable plots by factor (e.g. site, treatment, ...) and plotting each selected variable for each factor on the same scale would benefit this use case.

Reported by wsheldon, 2 years ago.

Adding an option to call the nan2zero.m function from the Edit menu of the data set editor would be useful.

Reported by wsheldon, 2 years ago.

An option for generating exact yearly x-axis ticks on Jan 1 should be provided on the GCETools menu for long time series plots, parity with monthticks.m for yearly plots.

Reported by wsheldon, 2 years ago.

An option to plot the range of daily or monthly values as a min-to-max bar with a mean line plot on top would aid in visualizing trends in widely-ranging variables like estuarine salinity.

Reported by wsheldon, 2 years ago.

The EMLdataset2mfile.xsl stylesheet generates an m-file for loading EML metadata, downloading dataTable entities, and importing them into the MATLAB environment. Due to issues with self-signed certificates in earlier implementations of PASTA, the generated code was originally written to use cUrl (see http://curl.haxx.se/) for handling the HTTPS protocol. However, MATLAB's HTTPS support in urlwrite has improved and PASTA now has a valid SSL certificate so this dependency is no longer required. The stylesheet should be updated to only call cUrl as a fall-back if HTTPS fails.

Reported by wsheldon, 2 years ago.

In recent MATLAB versions, e.g. 2016b, setting fontsize for the axis also overrides font size for child objects including ylabel, xlabel and title. As a consequence plots generated by plotdata.m have smaller 10-point fonts for title and axis labels than in previous MATLAB versions.

Reported by wsheldon, 3 years ago.

A user reported errors running the toolbox under MATLAB R2008b. Testing confirmed that this issue is caused by the use of the recommended null output variable syntax in recent versions of MATLAB (e.g. [~,x] = ...). Support for that syntax was introduced after R2008b so that and earlier versions cannot successfully run the toolbox. This syntax should be rolled back to [nullvar,...] or similar to restore functionality.

Reported by wsheldon, 3 years ago.

The fn (filename) argument in /xml/gceds2eml_table.m is listed as optional, but the function call returns an unassigned input argument error if it is not provided.

Reported by wsheldon, 3 years ago.

When typos are identified in metadata it can be very tedious to manually make the necessary changes in data set and template metadata fields. A function to automate this process would simplify this maintenance.

Reported by wsheldon, 3 years ago.

When revisions to numbertype are specified in calls to core/update_attributes (or workflows/batch_update_attributes), the changes are made to the variabletype field instead of numbertype, corrupting the data structure with unsupported metadata.

Reported by wsheldon, 3 years ago.

When USGS data files are parsed using parsers/parse_usgs.m the time zone is retrieved from the first record and added to the units and description of the calculated serial date column (and used by unit conversions). However when a station reports local time and daylight savings time is observed the time zone can shift within a single data file. As a result conversions to UTC or standard time in workflows can result in hidden 1 hour shifts in processed instantaneous (sub-daily) data files around daylight savings time transition dates.

Reported by wsheldon, 3 years ago.

When a row-oriented list of filenames is used for the filelist argument when calling multimerge() an error is returned. Only column-oriented lists are accepted but this is not documented. The function should accept any valid cell array of file names.

Reported by wsheldon, 3 years ago.

There is a syntax error in core/gceds2table.m that prevents use - an unreferenced variable dim.

Reported by wsheldon, 3 years ago.

When a data set is serialized in Toolbox ASCII Input File format or generated from the Excel Data Submission Template placeholder tilde (~) characters may be included in header fields to aid parsing. Those placeholders are removed in most header fields, but not removed in Q/C criteria fields to avoid invalidating rules containing NOT criteria. This case should be tested for explicitly.

Reported by wsheldon, 3 years ago.

MATLAB now supports a table object that parallels GCE Data Structures in some ways. The function core/gceds2table.m supports generating a table object from a GCE Data Structure, but there is no option for running it from the GUI Data Set Editor.

Reported by wsheldon, 3 years ago.

The NCDC GHCN-D import filter (parsers/imp_ncdc_ghcnd.m) automatically calculates TMEAN from TMIN and TMAX to provide mean daily temperature when only max/min thermometers are used. However modern weather stations report measured daily average temperature so this estimate is not necessary. The filter should check for TAVG before calculating TMEAN.

Reported by wsheldon, 3 years ago.

The ClimDB harvest trigger workflow used at GCE for the USGS harvesting service could be useful to other sites automating data submissions to ClimDB/HydroDB. This function should be added to the base toolbox.

Reported by wsheldon, 3 years ago.

The Harvest Dashboard web page displays the date indexed, but does not display the date/time of the last data record retrieved or warn when no no data are present. An XML tag is generated for that purpose (/root/temporal_coverage/hours_lapsed), but the code needs to be added to XSL stylesheets to display the information.

Reported by jsheldon@uga.edu, 3 years ago.

Running Matlab R2016a. These are probably not the only lines that will trigger these warnings. Version check and setting windowstyle is after position, resize, etc. in the code and is triggering warnings like the following in several places where dialogs are created. Warning: Cannot set Position while WindowStyle? is 'docked'

In ui_aboutgce (line 105)

Warning: Cannot set Resize to 'off' while WindowStyle? is 'docked'.

In ui_aboutgce (line 105)

Warning: Cannot set Position while WindowStyle? is 'docked'

In ui_editor (line 128)

Warning: Cannot set Resize to 'off' while WindowStyle? is 'docked'.

In ui_editor (line 128)

Reported by dam.kennedy@oregonstate.edu, 3 years ago.

Slow processing times are noticed when a variable has large number of flagged values.

I noted this priority as a blocker, but it's really a bottleneck. And defect might be a bit too harsh.

Note: The machine matlab is running on is a 2.5Ghz 4GB ram 64bit Windows 7 virtual machine. Ranks as one of the lowest performing platforms ML will run on.

Reported by wsheldon, 3 years ago.

When pad_date_gaps.m is run in MATLAB R2016a, a different number of records is generated than using previous versions resulting in uneven time steps (e.g. 15 minute data with steps at 14, 44, 59, ...).

Reported by wsheldon, 3 years ago.

From Jody Potter: "If I want to fill empty records in a text column with the same text throughout, what would I do? I’ve tried the Search/Replace String function, but do not know what would be inserted in Old Text when the records are empty?"

Reported by wsheldon, 3 years ago.

The tilde character (~) can be used as a placeholder for an empty text field in ASCII file headers to simplify parsing files using parseheader.m. The tildes are then replace with spaces during parsing. However, if tildes are included in Q/C criteria to represent the MATLAB NOT operator then the rules can be altered by this string substitution.

Reported by janderso@nmsu.edu, 3 years ago.

Installed: MATLAB R2013b (8.2.0.701) 32-bit Installed: GCE Toolbox V3.9.4b (Public distribution, complete package) Installed: Windows 7 Ultimate

Units are not importing using Campbell TOA5 import filter. But they do import using File > Import Data > Delimited Text File (ASCII) > Custom parsing

Steps taken: Data Structure Editor: File > Import Data > Campbell Scientific TOA5 Data > Any Station (generic template)

Result: In Data Structure Editor, the Date column and Column Names were correctly created, but the units were not imported from row 3 and are not listed after the column name nor listed in the data set view created with Edit>View/Edit data. In Data Structure Editor Column List, does not even show "(none)" for units after column name.

First 6 lines from sample data: met_15min_toa5.dat "TOA5","CR1000","CR1000","5203","CR1000.Std.10","CPU:SAPMLMET_V6.0_041310.CR1","48958","SWMP" "TIMESTAMP","RECORD","ATemp","MaxTemp?","MaxTempT","MinTemp?","MinTempT","RH","BP","WSpd","Wdir","SDWDir","MaxWSpd","MaxWSpdT","TotPrcp?","TotPAR","AvgVolt?","CumPrcp?","TotRad?" "TS","RN","C","C","hhmm","C","hhmm","%","mb","m/s","Deg","Deg","","hhmm","mm","mmoles/m^{2","Volts","mm","mmoles/m}2" "","","Avg","Smp","Smp","Smp","Smp","Avg","Avg","WVc","WVc","WVc","Smp","Smp","Tot","Tot","Avg","Smp","Tot" "2010-11-11 11:45:00",18060,21.50763,21.71759,"11:30",21.27892,"11:42",45.07585,1024.826,5.093334,56.84837,18.13199,7.2,"11:41",0,1049.867,13.21411,0.254,348.5052 "2010-11-11 12:00:00",18061,21.47951,21.71759,"11:55",-1.24518,"11:52",45.40335,1024.735,4.833333,53.9375,16.6154,7.65,"11:48",0,1066.515,13.21167,0.254,358.3602

Reported by wsheldon, 4 years ago.

The batch import GUI (gui/ui_batch_import.m) actively excludes any filter entries in imp_filters.mat that do not have a subheading defined to screen out legacy entries. This behavior is unnecessary given how gui/ui_editor.m now renders import filter entries, so this restriction should be removed.

Reported by wsheldon, 3 years ago.

A new version of MATLAB is about to be released, so compatibility testing should be performed.

Reported by wsheldon, 3 years ago.

Management changes sometimes require making bulk changes to documentation metadata (e.g. contact info changes) or attributes (e.g. unit or Q/C criteria changes). A function to automate and document these changes would be useful.

Reported by wsheldon, 3 years ago.

The harvest_dashboard.m function plots Q/C-flagged values as text over the corresponding value, but when data are dense with many flags assigned this can bog down the plot speed. An option should be added to support plotting flags as symbols like in plotdata.m.

Reported by wsheldon, 3 years ago.

The '*.*' is used as the filemask for batch_import.m, directories and directory aliases (. and ..) are matched, requiring the user to cancel a file prompt dialog when the files can't be loaded. Directories should automatically be excluded when using wildcards to select files. This issue was reported by Kristin Vanderbilt at SEV-LTER.

Reported by wsheldon, 3 years ago.

As reported by Nicholas Grant at HBR-LTER, the data interpolation function interp_missing() and corresponding GUI dialog only returns the message "Warning: no values were interpolated for the selected column" when interpolation fails due to an error condition, e.g. trying to fill an invalid time series (e.g. with duplicated timestamps). Details of the error should be reported to help identify the nature of the problem.

Reported by wsheldon, 3 years ago.

Support for the full screen crosshair pointer (fullcrosshair) was removed in MATLAB 2014, resulting in a warning message when used. Several toolbox functions called by the Correct Sensor Drift GUI (e.g. get_yaxis_range.m, get_xaxis_range.m) use this option and should be revised.

Reported by wsheldon, 3 years ago.

Reported by Adam Kennedy:

"Is there a method to change the default Data Cursor output on a plot generated with the Dataset Editor to show a full date string or a full serial date number that a user can convert in the workspace?"

An option should be added to the GCETools menu to use the datacursormode function to create a data tooltip with formatted date/time.

Reported by wsheldon, 4 years ago.

MATLAB 2014 introduced an over-hauled handle graphics engine, with major syntax and object property changes. Some toolbox plotting functions are not currently compatible with these changes and result in errors (e.g. legend-handling code, axis checking, zbuffer rendering for plots). Version checking code and syntax changes are required for full compatibility with MATLAB 2014 and higher.

Reported by wsheldon, 3 years ago.

When any header option other than 5-line is chosen in the EML package export GUI or when calling gceds2eml.m or gceds2eml_table.m the option is ignored. In addition, the EML metadata correctly describes the requested header format resulting in a non-congruent data table.

Reported by wsheldon, 4 years ago.

In some cases, core/pad_date_gaps.m does not correctly identify short date gaps due to floating-point decimal rounding issues.

Reported by wsheldon, 4 years ago.

The core/apply_template.m function and corresponding option in gui/ui_editor.m provide several options for importing documentation metadata from templates or other data structures, but attribute metadata are always updated. Opening the metadata in the metadata editor application is currently the only way to refresh documentation without affecting attributes. Adding an option to update documentation metadata without revising attribute metadata would simplify both interactive and scripted documentation updates.

Reported by wsheldon, 4 years ago.

The the 'Parse Names' or 'Parse Units' buttons are used in gui/ui_importfilter.m to parse a space-delimited ASCII file, and extra variable (none) is pre-pended to the column list if leading spaces are present (e.g. right-justified data columns and heading). When the format is parsed the extra white-space is ignored, however, resulting in a mismatch between column names and data, resulting in an invalid data structure.

Reported by wsheldon, 6 years ago.

It is occasionally necessary to apply multiple metadata templates to a data source based on date-specific changes to parameters (e.g. calibration or unit changes). This can be handled using multi_templates.m in a workflow, but a GUI for specifying templates and starting/ending dates would make this function more accessible.

Reported by wsheldon, 4 years ago.

It would be useful to develop a GUI dialog to parameterize the plotting/plot_vertprofile.m function for generating 3D contour plots of cruise vertical profile data.

Reported by wsheldon, 4 years ago.

When imp_ascii.m is called without specifying a format string, the subfun_parseascii sub-function is called to determine the header and data format. However when the titlestr argument is specified when calling imp_ascii.m it is not used resulting in a default title of "Data imported from the text file...".

Reported by wsheldon, 4 years ago.

An option should be added to plotting/plot_vertprofile.m to display the distances and depths where actual measurements were made and not interpolated. Adding a vertical line for distance with points for depths would be an effective way to convey this information.

Reported by wsheldon, 4 years ago.

The dt_add output variable is not defined in core/pad_date_gaps.m prior to initial validation checks, so if the function is aborted an undefined output error occurs.

Reported by wsheldon, 4 years ago.

When no template or the generic USGS_General template is specified for parsers/parse_usgs.m, mindate and maxdate variables may not be defined depending on the date/time format of the Date column. Therefore the starting and ending dates in the abstract will be blank, and a syntax error regarding array concatenation may be displayed.

Reported by wsheldon, 4 years ago.

GCE and other coastal/marine LTER sites are starting to work with SeapHOx and SeaFET sensor packages. Import filters should be added to the toolbox so data processing workflows can be developed.

Reported by wsheldon, 4 years ago.

When File > Save Metadata Template is used in gui/ui_editor.m to create a metadata template from the open data structure, manual tokens in Q/C criteria (locked rules) from interactive flag editing are retained, preventing the rules from being applied to new data when the template is applied. The manual tokens should be removed when saving.

Reported by wsheldon, 4 years ago.

When Q/C rules are synced in gui/ui_template, attributes should be matched by Name and not Variable to facilitate synchronization across templates for different data sources. For example, to standardize rules for common variables measured using different loggers with different program-assigned variable names.

Reported by wsheldon, 4 years ago.

Photographic images are increasingly being used for research. A utility to index images and parse EXIF metadata for web display would be useful.

Reported by wsheldon, 3 years ago.

MATLAB R2015B now includes preferences for handling high DPI monitors for font graphics scaling, and a warning is generated when the command set(0,'ScreenPixelsPerInch?',96) is executed in startup.m.

Reported by wsheldon, 4 years ago.

When a text import file is created using the data_submission_template.xls spreadsheet, and multiple method or personnel entries are present, only the last entry is retained when the metadata are imported.

Reported by wsheldon, 4 years ago.

From Adam Kennedy:

I’m looking at an example where it appears that the padded data time stamp fall out of sync with the observations. See below.

"GSWS08_108",2013-11-06 13:50:00,161169,108,2086.0,40.0200,"",7.64,"",0.2040,"" "GSWS08_108",2013-11-06 13:55:00,161170,108,2086.0,40.0200,"",7.64,"",0.2040,"" "GSWS08_108",2013-11-06 14:00:00,161171,108,2086.0,40.0200,"",7.65,"",0.2040,"" "GSWS08_108",2013-11-06 14:05:00,161172,108,2086.0,40.0400,"",7.65,"",0.2040,"" "GSWS08_108",2013-11-06 14:10:00,161173,108,2086.0,40.0300,"",7.66,"",0.2040,"" "GSWS08_108",2013-11-06 14:15:00,161174,108,2086.0,40.0300,"",7.66,"",0.2030,"" "GSWS08_108",2013-11-06 14:20:00,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 14:25:00,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 14:30:00,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 14:35:00,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 14:40:00,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" ​ ​So it looks like it goes to NaN at 14:20. Then about a day or so in, the time stamp falls out of sync "GSWS08_108",2013-11-06 20:05:00,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 20:09:59,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 20:14:59,NaN,108,2086.0,NaN,"",NaN,"",NaN,"" "GSWS08_108",2013-11-06 20:19:59,NaN,108,2086.0,NaN,"",NaN,"",NaN,"""GSWS08_108",2013-11-06 20:24:59,NaN,108,2086.0,NaN,"",NaN,"",NaN,""

Has anyone seen this happen in their datasets that could possibly offer a solution or explanation?

Reported by wsheldon, 6 years ago.

Users should be able to save custom Q/C rule statements for re-use similarly to calculations for calculating missing values and adding calculated columns.

Reported by wsheldon, 4 years ago.

The Query Builder GUI (ui_querybuilder.m) supports automatic calculation of numeric dates, but not any other expressions. Criteria should be evaluated for expressions for parity with the Data Editor (ui_datagrid.m).

Reported by wsheldon, 4 years ago.

From Adam Kennedy: For snow data, when we verify no-snow periods, ie. May 30 – Nov 2, I’d like a feature that would set the observed no snow values to zero. Similar to this function below. I’m just curious if there is a better way to do this.

if ~isempty(data)

col = name2col(data,{'SWE'});

if ~isempty(col)

[data,msg] = num_replace(data, ...

{'SWE'}, ... %columns to update

'Date >=datenum(datebegin)& 'Date <=datenum(dateend)', ... %criteria to match

0, ... %new value to assign

0, ... %change log option

... %flag to assign for revised values ( for none)

);

end

end

Reported by wsheldon, 4 years ago.

When a criteria expression containing the term isnull() is unparsed in gui/ui_querybuilder.m, it should resolve to = NaN for a numeric field. However, an invalid option is returned that results in an error rendering the drop-down menu and rendering the GUI inoperable.

Reported by wsheldon, 4 years ago.

When reviewing Q/C results for data sets with many columns it would be useful to be able to filter by flagged records for only specified columns rather than any column. A pick list of columns could be generated when this option is selected with multiple selection enabled.

Reported by wsheldon, 4 years ago.

Currently, the only way to check for inserted records by pad_date_gaps.m in a data harvester function is to look for missing values in the final data set, but that does not distinguish between records added by padding and intrinsically missing values in the logger file. An option for getting a report of padded values would be useful.

Reported by wsheldon, 4 years ago.

The Harvest Dashboard web page generated by workflows/harvest_dashboard.m and the associated stylesheets does not support custom names for the index files, and does not support yearly intervals for plots. This limits use of this function for long-term data sets and makes it impossible to generate links to .html versions if multiple index pages are generated.

Reported by wsheldon, 4 years ago.

Request from Dominik Schneider:

Does anyone have a use case where they are replacing missing strings with NaN instead of , as is the default behavior for date_pad_gaps()? For example, we have data columns that are time stamps of max wind speed that we'd like to fill with NaN instead of .

Reported by wsheldon, 4 years ago.

From Adam Kennedy:

I'm looking for a function in the dataset editor to filter a data structure by column name. Is this something that is available on the GUI?

For example, let's say I load a 800 column GCE data structure into the dataset editor. I'd like to do two things: 0) clone the dataset 1) create a separate GCE data structure that only contains 4 of those 800 columns.

So yes, exclude or include selected columns when the data structure is cloned or after it's cloned.

Reported by wsheldon, 4 years ago.

The function core/add_datepartcols.m only supports numeric indices or abbreviations for date/time components to include (e.g. 'yr','mo','dy','hr','mi','sc'). Full date/time component names should also be supported (i.e.g 'Year','Month','Day') for usability and parity with other functions.

Reported by wsheldon, 4 years ago.

When a text string is added to an expression in ui_calculator.m (Add Calculated Column dialog), the string is not quoted resulting in a syntax error. Automatically adding quotes would simplify adding static text (e.g. station ids) or text parameters in function calls.

Reported by wsheldon, 4 years ago.

When a Y column contains no data (all NaN) an array indexing error results because the empty column is removed from ycols but num_ycols and the plot attribute arrays are not adjusted accordingly.

Reported by wsheldon, 4 years ago.

John Chamblee report problems with column titles being ignored when a custom import filter was generated by gui/ui_impfilter.m:

I’m attaching some fake data to see if you can give me some insight into a problem I’m having with the Toolbox during a project I’m working on. The reason I’m sending it is that, in general the text files go in very well using the automatic parser, but the custom import tool and the import filters are acting squirrely.

The package includes the following:

1. a reasonably filled out template-based text file

2. a variable-title-only, single line header text file that I created by copying data out of the spreadsheet these guys are using to collect data (don’t get me started) and pasted into a text editor.

3. the metadata template I’ve created using the full-header template text file

4. the import filter I created doing a custom import with the variable-title-only file.

5. The matlab file I was able to create by using automatic import of the header only file and then applying the template, but without using any import filters.

The problem I’m having is that the both text files go in fine with automatic import, but when I create an import filter for the using the single row header file and select the metadata template I created using the text file, *none* of the columns receive metadata. I’ve compared the headers pretty carefully and I can’t see any reason why this would be. Moreover, when I use the import filter I created, the field names get dropped and just the column order variables get used.

Reported by wsheldon, 4 years ago.

When switching between filters in gui/ui_import_filter, unneeded argument fields populated by other filters are not cleared, resulting in inappropriate argument options that do not match the selected function syntax. This could lead to syntax errors when the m-file function is called.

Reported by wsheldon, 5 years ago.

From Adam Kennedy: Do you think that adding a Min/Mean/Max table below each of the dashboard plots would be worthwhile?

From a practical perspective, we’d get the min/max at each plot resolution. So we could say what the min air temp was last week, or the max ppt intensity last month, or what the mean snow depth sensor was reading last week under no snow conditions, etc.

Since there is some room on under the table, little reformatting would be required other than just adding fields and the associated calculations.

Reported by wsheldon, 5 years ago.

When EML metadata from fetch_eml_data.m are imported by eml2gce.m the methods and instruments content in s_raw.methods are lumped into big strings and not parsed into discrete method steps. This prevents gceds2eml.m from properly parsing methods and instruments into nested methodStep elements.

Reported by wsheldon, 5 years ago.

When and empty string is specified for the col_balloon argument in xml/gceds2kml.m a syntax error results.

Reported by wsheldon, 4 years ago.

Campbell Scientific Instruments logger files often contain time of max/min columns as integers in hhmm format or times in h:m format which are not compatible with MATLAB's date/time functions. Adding a conversion to/from these proprietary time formats and standard time formats would be useful.

Reported by wsheldon, 4 years ago.

When USGS data are imported using fetch_usgs() or parse_usgs(), empty personnel subfields are present in addition to personnel fields in the metadata template.

Reported by wsheldon, 4 years ago.

Currently the only practical way to generate tokenized headers for imp_ascii.m is to use the Metadata Submission Excel template. It would be useful if headers could be generated from existing data sets to simplify parsing of similar data. This would also provide a simple means of serializing metadata and data content for archiving data sets in text format, then loading them back into the toolbox reliably.

Reported by wsheldon, 4 years ago.

When changes to metadata styles are made using gui/ui_metastyles.m (Misc > Add/Edit Metadata Styles), the EML metadata style menus are removed when menus are updated. Same for the MATLAB file export submenus. The editor needs to be restarted to restore the EML options.

Reported by wsheldon, 4 years ago.

Several import functions, including imp_ascii and fetch_data_turbine.m, support more than 6 user arguments (i.e. after fn,pn). However, gui/ui_editfilters.m and the current imp_filters_default.mat only support 6 user arguments.

Reported by wsheldon, 4 years ago.

The data editor (gui/ui_datagrid.m) should evaluate mathematical expressions and dates typed into numeric fields and automatically perform the calculations or call datenum, resp. to simplify data entry.

Reported by wsheldon, 5 years ago.

If the cols parameter is used in nullflags or cullflags, and no columns are matched, an empty array is returned. It would be better to return the original structure with a warning for consistency with other functions, and to prevent typos in workflows from breaking the entire processing pipeline.

Reported by wsheldon, 5 years ago.

For some data sets it would be useful to generate multiple plots for the same parameter(s), sub-setting the data by some other factor like treatment. This could be accommodated by extending harvest_webplots_xml.m to support a 'query' field.

Reported by wsheldon, 5 years ago.

When multiple Study.Description.StudyElement?.Design sections are present in parsed metadata, gceds2eml.m throws an array dimension error when populating the output structure.

Reported by wmsheldon@earthlink.net, 9 years ago.

Adding support for EML metadata generation will broaden the utility of the toolbox for LTER and Ecoinformatics users.

Reported by wsheldon, 5 years ago.

Mary Martin reported a serious issue processing Campbell TOA5 files that contain date part columns: We have some campbell files that have a timestamp and also a Year/Month/Day/H/M/S column. The latter are different from the timestamp and indicate the initiation of a measurement. There is a lag between that and the write time, which is the TIMESTAMP. When I harvest these in GCE, these non-timestamp, date related columns are altered (see attached). It seems as though matlab must recognize these as data components and not just plain old variables, and is doing something with them. the strange thing is that they aren't changed to match the timestamp, so I end up with the timestamp, a decimal hour, and changes to the Hour/Minute columns, and none of the three match. What a mess. Is there something that I can do to preserve the values in our 'date component' columns (those columns after 'RECORD')? I'm not sure if I'm explaining this well, but in the attached you will see the header and a record from campbell, and then the output from GCE.

Reported by wsheldon, 5 years ago.

When the flags = 2 option (i.e. "plotted as red symbols on top of data values" )is specified in plotting/plotdata.m, the symbol of the flag is shown for the variable in the legend instead of the primary symbol.

Reported by wsheldon, 5 years ago.

When a metadata field contains a huge string (e.g. >1 million characters), MATLAB can hang for many minutes while initializing an editbox control on a GUI figure. This situation can occur when merging metadata from many individual files, or for fields like Data/Anomalies when concatenating files repeatedly in a harvesting scenario. Vulnerable GUI dialogs include gui/ui_editmetadata.m, gui/ui_visualqc.m and gui/ui_flagdefs.m.

Reported by wsheldon, 5 years ago.

Under some circumstances the /core/addflags.m function may return an empty data structure without an error message. This bug is due to an error generating a padded character array of flags to match the dimensions of the existing assigned flags that results in an incorrect flag array size, invalidating the structure. The resultant invalid structure is returned as an empty array when addmeta() is called as the last step, therefore no error message is generated.

Reported by wsheldon, 5 years ago.

The normalize_cols() function can return an invalid data structure when no flags are assigned to any of the columns being combined.

Reported by wsheldon, 5 years ago.

The meta2struct() function used for XML and EML metadata looks for a list of supported subfields in every contact information field in the metadata array (e.g. Name, Position, Organization, Address, Email, ...), but those fields are not included by default when new data structures are instantiated or metadata fields are added using the GUI metadata editor. Hence most users are unaware of this useful feature.

Reported by wsheldon, 5 years ago.

When aggregating data sets values in text columns are only counted. It would be useful to have an option to concatenate and enumerate unique text terms in each group to preserve information about coded values and other textual data when synthesizing data sets.

Reported by wsheldon, 5 years ago.

When the criteria "= NaN' is specified in the query builder GUI, the expression ~isnull(...) is generated instead of isnull(...).

Reported by sheldon@uga.edu, 5 years ago.

Plotting functions in /plotting should all support an option to disable console display when running workflows that operate unattended or in the background.

Reported by wsheldon, 5 years ago.

When a text file generated from the GCE data submission template is imported using imp_ascii.m, and the file includes documentation metadata but one or more data types are mapped as 'u' (undefined) due to typo or omission, the doc metadata is overwritten when the file is evaluated for automatic typing.

Reported by wsheldon, 5 years ago.

The flags2cols() function and related options in exp_ascii() and exp_matlab() support instantiating flags as columns for all columns, all data/calc columns or just columns with flags assigned. However, sometimes it is ideal to display flags for all data/calc columns *plus* any other columns regardless of type with flags assigned to avoid masking flag assignments on geographic coordinate or code columns, for example. An additional option combining the automatic and 'alldata' options would be useful.

Reported by wsheldon, 5 years ago.

A GUI dialog for calling /xml/gceds2eml.m to generate complete EML data packages would improve visibility and usability of that toolbox feature.

Reported by wsheldon, 7 years ago.

The statistics calculated for aggregated data using aggr_stats.m and aggr_datetime.m are currently set coarsely using an enumerated argument, leading to auto-generating of unnecessary or confusing summary statistics (e.g. totals of rates, not just counts or sums). Support should be added for fine-grained lists of statistics for each attribute being summarized, and support for selecting stats should be added to ui_aggrstats.m and ui_aggrdatetime.m.

Reported by wsheldon, 5 years ago.

The date input validation is buggy in ui_fetch_ncdc_ghcnd.m, preventing entering some valid dates for filtering data records.

Reported by sheldon246, 6 years ago.

The function fetch_ncdc_ghcnd accepts an argument to delete temporary download files after successful import, which would be very useful, however there is no actual code that implements such functionality.

Reported by wsheldon, 5 years ago.

It would be useful to be able to register new flag codes and definitions by calling addflags.m in a workflow, e.g. when running a novel algorithm for Q/C or to identify events, or to add flags as part of a multi-step operation.

Reported by wsheldon, 5 years ago.

The functions for interpolating missing values (core/interp_missing.m and core/interp_missing2.m) use all non-missing values in a data series for interpolation. When interpolating time-series data with a strong diurnal pattern, e.g. air temperature or salinity, this approach can lead to significant truncation or distortion of diurnal/semi-diurnal peaks in the data. A different algorithm is need to base interpolation only on data collected at the same time of day to preserve the underlying diurnal pattern when interpolating data prior to summarization.

Reported by Adam.Kennedy, 5 years ago.

I am looking for a function that will re-run dataharvester() .mat files after all auto-updates have been completed that will re-generate all data harvester output.

For example: the AND renames GCE data structures annually, so only the current year is processed. However, when manually QC’ing the previous year data, it is not possible to re-harvest the file – since the dataharvester() stops when no new data are present.

Suggestions:

1) Modify the dataharvester() (or add a switch) to process (create new plots, export all files, etc) the .mat regardless if there is no new data available?

2) Add a menu item that allows a user to process an existing .mat file according to its respective dataharvester().

Reported by wsheldon, 5 years ago.

It would be helpful if variables with status = yellow or red in the last hour or day were displayed automatically at the top of the harvest dashboard webpage to alert monitoring technicians.

Reported by wsheldon, 5 years ago.

The /core/unit_convert.m function supports adding the converted data as a new column, but does not include an option to specify the name. A name is automatically generated based on the original name with new units appended, after scrubbing unsupported spaces and characters. The function should include an option for specifying the new column name to simplify use in workflows.

Reported by wsheldon, 5 years ago.

The harvest dashboard plots and metrics should only reflect variable values up to the maximum reporting interval (e.g. day, month) for setting axis limits and reporting total missing and flagged values.

Reported by Adam.Kennedy, 5 years ago.

From Adam Kenedy: "I’m putting together a daily stats output and am looking for a gce function to add the time of Max to a data field. Is there a function for this?". Adding an option to calculate the times of Min/Max values in an aggregate would provide this functionality.

Reported by wsheldon, 5 years ago.

Displaying flags on plots of large data sets can be very slow due to the large number of text objects MATLAB has to manage. An option should be added for plotting flags as symbols to improve performance.

Reported by wsheldon, 5 years ago.

All GCE-LTER workflows and data files should be removed from the SVN trunk to complete the transition from a site-specific to a generic environmental data management tool. A single public distribution should be offered and the SVN and Trac should be reconfigured to allow public read. Site-specific workflows could be offered as separate downloads apart from the toolbox for sharing and reference.

Reported by wsheldon, 5 years ago.

Reported by a toolbox user:

I am trying to use the 'GCE MATLAB toolbox' to extract GHCN daily rainfall data from the NCDC archive. I am specifically using the function 'fetch_ncdc_ghcnd' with the following syntax (as an example):

[s, msg, s_raw] = fetch_ncdc_ghcnd('USC00097808', 'NCDC_GHCND', [], [], 1, [], [], 1, ' ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/');

I have downloaded the toolbox in my directory and run the 'startup' file. However, I am getting the following error message:

Error using urlreadwrite>getFileOutputStream (line 224) Could not open output file "C:\Users\Fitsum Woldemeskel\Desktop".

Error in urlreadwrite (line 64)

[file,outputStream] = getFileOutputStream(inputs.filename);

Error in urlwrite (line 38) [f,status] = urlreadwrite(mfilename,catchErrors,url,filename,varargin{:});

Error in fetch_ncdc_ghcnd (line 123)

[fqfn,status] = urlwrite([baseurl,'all/',stationid,'.dly'],[pn_temp,filesep,fn_temp]);

I might have made simple mistake that i could not figure out. Could you help me to identify the reason for the above error ?

Reported by wsheldon, 5 years ago.

MATLAB R2013b adds a new 'table' object that is similar in concept to a GCE Data Structure, with named variables as columns and metadata properties for variable names, units, descriptions and a title. Support should be added for exporting GCE Data Structures to table objects to support new table-oriented functions in future versions of MATLAB

Reported by wsheldon, 5 years ago.

Changes made to ClimDB variable mappings in ui_expclimdb.m are not saved after returning revisions to the dialog.

Reported by wcsheld, 6 years ago.

On line 307 in pad_date_gaps.m there is a missing underscore in a reference to the str_flags variable, which throws errors upon execution

Reported by csebranek, 6 years ago.

From Chad Sebranek:

"I installed version 3.7, put the toolkit and your functions into /extensions (the ones you gave me from your thumb drive), added the odm stuff to /demo and removed all paths, then issued the startup command to make sure all paths were added. However, still having issues. Can you see all the sources from the Fetch Data Turbine GUI?

In the Fetch Data Turbine GUI, the problem is only TroutBog?, _Metrics and _Log come up as sources. I can't seem to fetch other data. I can harvest it from the command line, though. "

Reported by dosc3612, 6 years ago.

From Dominik Schneider (NWT): "I was looking at using the sensor drift tool but was wondering if a tool existed that could pick and match the endpoints of the discontinuity to calculate the constant offset?"

Reported by wsheldon, 6 years ago.

The function csi2struct.mat parses multiple arrays from a Campbell Scientific array-based data logger but saves them all as named variables in a single .mat file. It would be helpful to split these out into individual data structure files to support using multi_merge.m and other batch processing tools.

Reported by Adam.Kennedy, 6 years ago.

From Adam Kennedy: "Is there a clever trick to split away the QC rules from the metadata template? I have one metadata template per site, 20 sites, the QC rules are the same for each variable at all sites. When I want to tweak a QC rule, I have to replicate the effort across metadata templates, instead of one QC template. Anyway, not a priority, just wondering if there has been discussion on this. It’s tricky keeping all the QC rules consistent across sites for a given variable."

Reported by jsheldon@uga.edu, 6 years ago.

Under the menu File:Export Data: Matlab File:(submenus), each of the submenus has one further level of submenu with one option: No Metadata. It's already a long menu tree, and that last No Metadata option is not needed since there are no alternatives. Suggest you remove the No Metadata menu levels.

Reported by jsheldon@uga.edu, 6 years ago.

When exporting a dataset as a Matlab structure using File: Export Data: Matlab File: Struct Variable, it would be nice to have an opportunity to rename the struct something other than the default 'data'. If they are all called 'data' then you can't work with more than one at once in the Matlab workspace, so you have to rename and resave all the files anyway.

Reported by wsheldon, 6 years ago.

When a multi-line character array is passed to core/add_harvest_log.m as an entry to log, it is inserted as-is into a GCE Data Structure field. This causes errors in gui/ui_datagrid.m and potentially other functions.

Reported by wsheldon, 6 years ago.

From Adam Kennedy (AND):

"I want to pull out all file path specific code form the data harvester workflow so that I can move the harvesters back and forth between servers. I started doing it with some of the path names, but seemed to run into glitches with some files.

Can you advise on this? Would creating a path_config.m file that is called on every workflow script set me up for this? I would have one path_config.m per installation."

Reported by wsheldon, 6 years ago.

When the "Edit All Metadata" option is selected in the Metadata Template Editor application (ui_template.m), the 'Metadata > Add/Update Q/C Flag Definitions, Data Anomalies' menu is not available in the Data Structure Editor window, making it hard to revise Q/C flag definitions. This option should be enabled.

Reported by wsheldon, 6 years ago.

The dataset editor (gui/ui_editor.m) uses minimum interval criteria of 1 min when calling pad_date_gaps.m, so there is no option for padding gaps in high frequency data (10-20Hz). A high frequency option should be added that calls pad_date_gaps.m with a suitable min_interval argument.

Reported by wsheldon, 6 years ago.

Few of the standard EML units are in the unit conversion database, so it's not convenient to perform unit conversions when EML-described data tables are imported. Providing an EML-to-conventional mapping function like Metric-English conversions would also be useful.

Reported by wsheldon, 6 years ago.

The current toolbox functions and stylesheets for parsing EML and loading data blindly download all MATLAB-compatible data objects. This is fine for small data sets, but problematic for really large ones. There's also no reason non-MATLAB-parsable data can't be downloaded for use in other programs. A second stylesheet and associated functions/GUI should be developed that first parses the EML to display info about all downloadable objects then fetches and parses the data as a separate step.

Reported by wsheldon, 6 years ago.

The /database/sql2struct.m function returns different structure layouts depending on which database toolbox 'DataReturnFormat?' option is defined using setdbprefs. When the 'structure' option is specified, sql2struct.m returns a scalar struct with each field containing an array of column values. However, when the 'cellarray' option is specified a multi-dimensional struct is returned with each database row in separate dimension with scalar values in each field, as documented in the function help.

For example:

'structure' option:

s =

Site: {655x1 cell}

Location: {655x1 cell}

Type: {655x1 cell}

TransectDistance?: {655x1 cell}

Description: {655x1 cell}

Lon: [655x1 double] Lat: [655x1 double]

'cellarray 'option:

s =

655x1 struct array with fields:

Site Location Type TransectDistance? Description Lon Lat

These struct layouts are very different and require different syntax when extracting data values.

Reported by wsheldon, 6 years ago.

If carriage returns are entered into the multi-line Description edit box in the Dataset Editor GUI (ui_editor.m), a multi-line character array results that causes errors in other toolbox applications. For example, trying to open the data set using View/Edit Data (ui_datagrid.m) throws an error because the descriptions are added as a toolbox for column labels, and multi-line character arrays are not supported.

Reported by wsheldon, 7 years ago.

Tools for managing harvest timers are incomplete, because timers can be parameterized by editing harvest_timers.mat and the started from the command line with start_harvesters.m, but there are no functions for monitoring and starting/stopping running timers. A GUI should be added to call start_harvesters.m and to enable viewing, starting and stopping running timers.

Reported by wsheldon, 7 years ago.

Developing a harvester log function and data spec would enable MATLAB and web dashboard development to aid in monitoring the status of harvester operations. Each harvest script can call the function to add a log record with status info and links to plots (if applicable), and the function can generate MATLAB and XML output that can be used for dashboard development using MATLAB or XSLT.

Reported by wsheldon, 6 years ago.

Dominik Schneider reported an issue with closing figure windows in MATLAB running Mac OS/X that occasionally prevents him from starting ui_editor.m:

"There are a number of MATLAB bugs on OSX that are known and longstanding. It includes poor figure handling. The issue I was having is related to a general problem I have had working in matlab and creating multiple figures. I've scoured the internet and talked to a few folks here, one of whom talked to math works tech support, and the consensus is that all figure windows should be closed with close gcf or close all or close <figure handle> instead of closing the window through the OSX window system."

The figure close event handling in toolbox GUIs should be reviewed to mitigate this issue if possible, and it should be noted in the user guide.

Reported by wsheldon, 6 years ago.

When pad_date_gaps.m is called to fill in records to create a monotonic time series, the missing values that are inserted cannot be distinguished from missing values present in the raw data file. This may be an important factor for some QA/QC processing or data documentation, so an option should be included to assign a flag to empty values that are inserted by this function.

Reported by wsheldon, 6 years ago.

When a grouped plot is generated using Tools > Plotting > 2D Line/Symbol (single Y, split by groups) and the Visual QC tool is used to revise flagged values, the data set returned to the Dataset Editor is still sorted by the grouping value. This can lead to problems if the data are exported or plotted without being re-sorted to remove the grouping.

Reported by wsheldon, 6 years ago.

If numeric missing value codes are present in a data set (e.g. -9999), it is difficult to write Q/C rules containing statistical checks. A function similar to no_nan.m should be added to support stripping numeric missing values prior to calculations.

Reported by wsheldon, 6 years ago.

The aggregated statistics functions (aggr_data.m, aggr_datetime.m, etc) support flagging based on total number or percentage of missing values in an aggregate, but NWS and other monitoring programs also flag daily summaries based on the maximum number of consecutive missing values. It would be useful to add this option to the data aggregation functions in the GCE Data Toolbox.

Reported by wsheldon, 9 years ago.

Currently geographic data structures in the toolbox mirror the GCE_Metabase data tables and are not user editable in the toolbox GUIs. Point locations data (gce_locations.mat) should be in data structure format, and a GUI editor should be created for updating polygons (gce_coords*.mat). Additionally, GCE branding should be removed from the editor menus pertaining to geographic coordinate lookups.

Reported by wsheldon, 6 years ago.

Including imp_templates.mat, imp_filters.mat and unit_convert.mat in the /userdata directory of full distributions makes it too easy for users to accidentally overwrite their localized versions, potentially losing critical work. These files should be removed and replaced with *_default.mat versions, and the corresponding toolbox functions should copy and rename these defaults if a localized version is not available.

Reported by wsheldon, 6 years ago.

Performing a lookup join with text key columns is very slow taking many minutes if there are >1000 unique values in the key column. Joins should be optimized to speed up these types of operations.

Reported by wsheldon, 6 years ago.

Only United State Cooperative Observer (COOP) stations are listed in the NOAA NCDC GHCN database included with the GCE Data Toolbox (userdata/ncdc_stations.mat). Data for over 87,000 GHCN stations are available from the NCDC site, and these should all be listed in the form to provide access to all this world-wide data.

Reported by wsheldon, 6 years ago.

Hard returns in anomalies entries made in the Visual Q/C tool (gui/ui_visualqc.mat) result in multi-line character arrays in metadata value fields, which generate errors when the metadata are rendered as text or opened in the metadata editor. Multi-line strings should be converted to character arrays before being saved to metadata, as in gui/ui_flagdefs.m.

Reported by wsheldon, 6 years ago.

When running core/concat_cols.m with deleteoption=1 to remove the original columns after concatenation, the new column is omitted as well. It is also not possible to specify units and descriptions of the combined column when calling this function, requiring additional steps in a workflow.

Reported by wsheldon, 6 years ago.

A number of toolbox settings files (USGS and NOAA station lists, default maps, GUI preferences) are stored in /userdata along with templates and import filters. Setting up a separate /settings or equivalent directory would make the distinction between files easier for users to understand when migrating content between toolbox installations.

Reported by wsheldon, 6 years ago.

The support/trimstr.m function is much slower that strtrim.m available in recent versions of matlab when used with large cell arrays of strings. The newer function should be used when available to improve performance.

Reported by wsheldon, 6 years ago.

Sometimes leading or trailing whitespace may be present when text columns are imported. A function should be added for compressing all or selected text columns, and a menu item added to the dataset editor's Edit menu.

Reported by wsheldon, 6 years ago.

From Adam Kennedy:

A user can’t use an ‘&’ in any of the harvest_info or harvest_plot_info nav or title strings. The XML parser chokes. You probably already knew this, but I’m writing this just in case you didn’t.

This:

%station-specific navigation label and url to append to nav_base nav = [nav_base, ...

{'WS1 Met & Flux',' http://hja.fsl.orst.edu/weather/GCE-HJA/LNDB_TEST/PORTAL/WS1MET/data/'}];

Reported by wsheldon, 6 years ago.

When exporting data sets to text formats, MATLAB numeric dates should be converted to a more appropriate date string. It would be convenient to add a date format option to the ASCII export dialog (ui_exportasc.m) to support this option, so users do not have to temporarily convert dates prior to export.

Reported by wsheldon, 6 years ago.

Support should be added for exporting data sets as standard MATLAB struct variables in addition to matrices and individual arrays, for parity with import options. This would also simplify saving geographic data structures (geo_locations.mat) and similar settings files after editing.

Reported by wsheldon, 6 years ago.

The gce2odm.m workflow for re-factoring GCE Data Structures for inserting into a CUAHSI ODM database fails to include data from the last several columns.

Reported by wsheldon, 6 years ago.

When saving changes to English<>Metric unit conversion mappings, the error 'Warning - Could not return data to original editor' occurs and changes are not saved.

Reported by wsheldon, 6 years ago.

If very large data sets are inserted into a database using gce_fastinert.m (or gce_insert.m), Java heap memory errors can result. There should be an option to insert rows in batches to lower the memory requirements for inserting large data tables.

Reported by wsheldon, 6 years ago.

When code definitions are edited in the Dataset Editor (ui_editor.m), the changes are not saved to the metadata.

Reported by wsheldon, 6 years ago.

Functions for uploading data from GCE Data Structure columns to relational databases should be added for parity with sql2gceds.m and sql2struct.m to support inter-operability between the toolbox and relational databases.

Reported by wsheldon, 6 years ago.

The custom ascii file import dialog (ui_imp_filter.m) does not include the units string in the generated m-file, relying on metadata templates. Extra code should be included to add units when a template is not specified.

Reported by Dominik, 6 years ago.

Open a metadata template with edit all metadata. Add a column, delete this column. Hitting Return Data will cause an error: 'Empty structure can not be returned'. Work around is to fill out information for the undesired column, return data, use 'Edit Attributes', select the row that contains the bad column and use Remove Selected Row.

Reported by wsheldon, 6 years ago.

It would be useful to generate an HTML table for displaying harvested data on the web, or as part of a dashboard application.

Reported by wsheldon, 9 years ago.

A tutorial or walk-through of common toolbox operations with screen-shots would make it easier for new toolbox users to get started. This could be added to the documentation or included as a separate PDF or video file. A sample data set used for the tutorial should also be included in the public distribution.

Reported by wsheldon, 6 years ago.

The joindata.m function requires key-only matches to prevent data duplication, so it cannot be used for filling in values from lookup lists (e.g. adding information from log files or expanding codes). A new function for adding columns by looking up values in another data set that supports 1:many relationships should be developed.

Reported by wsheldon, 6 years ago.

A function for splitting text columns on a delimiter character to create multiple columns should be added for parity with concat_cols.m. This would allow columns that contain multiple treatment codes to be separated into individual treatment factors, as reported by Kristin Vanderbilt (SEV-LTER).

Reported by wsheldon, 6 years ago.

Changes made to attribute metadata fields in the dataset editor (ui_editor.m) other than column names are not individually logged in the structure processing history. All attribute metadata changes (other than descriptions) should be logged in detail to support evaluating the specific changes that were made.

Reported by wsheldon, 6 years ago.

When apply_template.m is used to add metadata to a data structure, an empty structure and error message will be returned if there are any data type mismatches between the template and existing data columns. This prevents applying templates after date string/numeric inter-conversions or other column transformations. Column datatype mismatches should be handled like unit differences, where updates are skipped and a warning message returned.

Reported by wsheldon, 7 years ago.

A GUI dialog should be created for coallesce_cols.m to provide finer control of column ordering. This dialog should also be grouped with ui_interp_missing.m and a new dialog for calc_missing_vals.m to provide multiple options for filling in missing values in a data column.

Reported by jsheldon, 7 years ago.

When a matrix is imported from the Matlab workspace, it is automatically split into columns and each column is automatically named. It would be useful to be able to specify another variable (a cell array of strings) that could be used as column names.

Reported by wsheldon, 7 years ago.

The data editor GUI ui_editor.m contains a few GCE-specific specialty function calls in the Tools menu. Specialty functions should be managed using configuration files like import filters so GCE specialized functions are not hard-coded in the menus and users can add their own workflows

Reported by wsheldon, 7 years ago.

The GUI tools for normalizing (ui_normalizecols.m) and splitting compound (ui_splitseries.m) data columns only support single-column selection. For data sets with many columns this presents a usability issue (reported by D. Blankman). Multi-select support should be added.

Reported by Adam.Kennedy, 6 years ago.

Do we continue to update our .zip files or is there a way to update just the new/changed files on the SVN to our local Matlab directory?

Reported by wsheldon, 6 years ago.

The imp_struct.m function imports all fields of a structure so all must have matching dimensions. Some structures include a mix of data arrays and metadata, so support should be added for choosing which fields to import.

Reported by wsheldon, 6 years ago.

The functions fetch_nerr_data.m and parse_nerr_data.m are actually generic for any data retrieved from the NOAA Hydrometeorological Automated Data System ( http://amazon.nws.noaa.gov/hads/interactiveDisplays/displays.html). These functions should be renamed to fetch_noaa_hads.m and parse_noaa_hads.m to avoid confusion.

Reported by Dominik Schneider, 6 years ago.

Just a reminder from the workshop, csi2struct should be made more robust to allow various field names for the date/time column header in the old array-based data files.

Reported by wsheldon, 6 years ago.

The full-range plot generated by harvest_webplots_xml.m prior to rendering specified time interval plots is not saved or indexed. An option should be provided for including this plot along with the partial plots.

Reported by jchamblee, 6 years ago.

The current version of imp_campbell_toa5 assigns standard text to all files imported as part of a data harvest routine. This could be remedied by making a coy of the imp_campbell_toa5 file and rewriting the hard coded abstract and title script, but, given that many data harvester scripts rely on Metadata templates, a possible enhancement might be to look for titles and abstracts in metadata templates and either replace or append harvest text from the imp_campbell_toa5 file to the text provided by the abstract.

Reported by wsheldon, 6 years ago.

The data editor (ui_datagrid.m) displays values assigned flags highlighted in red and displays the flag(s) as a tooltip. Some users may find it easier to review and edit flags if they were instantiated as columns in the editor similarly to the 'Manual QC' button on the data set editor (ui_editor.m). Being able to toggle between the current behavior (highlighted cells) and flags as columns would be a nice feature.

Reported by wsheldon, 6 years ago.

When q/c flags are instantiated as data columns in the data set editor (ui_editor.m) multiple times using 'Edit > Q/C Flag Functions > Convert Q/C Flags to Text Columns > ...', multiple flag columns are created with the same time but potentially different contents. This is confusing and potentially problematic. This operation should overwrite prior flag columns with user prompting, or should fail because flag columns already exist and issue an error message.

Reported by wsheldon, 6 years ago.

The date/time inter-conversion functions in the dataset editor ('ui_editor.m') determine date/time columns to use by metadata inspection and column name conventions, so they cannot reliably be used to reformat date/time columns for all data sets particularly if multiple date/time columns are present. A more specific option should be provided to support format conversion for selected columns. Additionally, conversion of dates to time-only formats should be supported to remove date portions if desired (requested by NWT for time of maximal value columns in CSI data).

Reported by akennedy, 6 years ago.

From Adam Kennedy: "I’m looking for a way to flag snow depth (or precip) values when the total accumulation during the previous hour, day, month exceeds some expected range."

A flag_accumulation.m function could be added to address this use case, with a window size based on number of records back (like flag_valuechange.m) or calendar interval.

Reported by Dominik, 6 years ago.

I installed the .4.2 beta patch and the directions led me to overwrite my existing userdata folder. I recovered it but patches probably shouldn't contain a userdata folder so as not to overwrite it as your instructions for installation say to allow.

Reported by wsheldon, 6 years ago.

The custom import filters registered in imp_filters.mat using Tools > Add/Edit Import Filters require a filemask and file prompt, which precludes registering import filters that retrieve data over the WWW or other network protocols. Adding entries to the list of supported WWW data sources requires editing gui/ui_editor.m because they are hard-coded links to GUI forms. An option should be provided for user-registration of WWW-based import filters so links to NOAA and USGS stations can be included in menus.

Reported by wsheldon, 6 years ago.

The 'Add/Edit Import Filters' command in the data set editor simply opens the filter database in an instance of the data editor (ui_datagrid.m) for revision. However, this requires that users know the exact syntax of mfiles to properly fill out Argument1, Argument2, etc. A dialog that creates meaningful argument labels and displays function help like ui_batch_import.m would be more usable.

Reported by wsheldon, 6 years ago.

No version information is stored for metadata templates, so it is not possible to determine whether the same template content was applied to data sets imported at different times. Adding a 'date' field to the templates structure in imp_templates.mat would allow logging a revision date in the processing history when a template is applied.

Reported by wsheldon, 6 years ago.

As part of an NSF SI2 project at SDSC, support is being improved for connecting to Data Turbine using MATLAB (DTMatlabTK library). This software should be leveraged to support importing data from Data Turbine sources into the GCE Data Toolbox to support data QA/QC, metadata generation and archiving.

Reported by wsheldon, 6 years ago.

Data can be imported from relational databases via sql2struct.m and sql2gceds.m, but no parallel functions are provided from inserting data from GCE Data Structures back into a relational database.

Reported by wsheldon, 6 years ago.

In some sensor network scenarios, data values need to be revised based on readings from other sensors (e.g. setting all light sensor values to 0 when a master sensor reads below a threshold value). The current num_replace.m function only supports replacing values based on exact matches to a scalar value so it cannot be used for this purpose.

Reported by wsheldon, 7 years ago.

A function that sends email alerts when harvests don't retrieve new data or too many flagged/missing values are present would be very useful for production data harvesting workflows.

Reported by wsheldon, 7 years ago.

The GCE portal_datapages_xml.m and portal_webplots_xml.m functions require lots of arguments (e.g. xml file names, starting and ending dates, etc) that require lots of code in harvester workflows or maintaining scripts to automate post-harvest processing. Generalizing these functions to use configuration profiles and derive file names and dates from the source data file would simplify development of harvest scripts and enable configuration information for plots to be stored in 1 place to simplify adding new stations.

Reported by wsheldon, 7 years ago.

The SBE import filter parse_seabird.m is generic, but includes some GCE dependencies (cruise ids, calls to parse_cruise_log.m, etc) that are not appropriate for general use. Removing GCE-specific code and moving it to a separate workflow (e.g. parse_cruise_log.m) would fix this issue.

Reported by wsheldon, 7 years ago.

From Richard Cary: " I want to determine the daily max and min temperature values for data from a site with hourly data. I tried to use Tools > Statistics > Statistics for Date Time Intervals, but it is only giving me the option of yearly statistics. The odd thing is that if I run the Statistics for Moving Date Interval tool on the data, I can create daily statistics from the new dataset using the “Statistics for Date Time Intervals” tool. Any idea why I would be limited to only yearly statistics on the Statistics for Date Time Intervals tool?"

Reported by wsheldon, 7 years ago.

The GUI data editor (ui_editor.m) just calls the generic ui_text_prompt.m dialog to prompt for an EML URL which isn't very user-friendly. A dedicated dialog should be developed to support canned URL patterns for Metacat, PASTA and GCE as well as custom URLs.

Reported by wsheldon, 7 years ago.

The new calc_missing_vals.m function would benefit from a GUI dialog for column selection, equation entry and flag definitions to parallel ui_interp_missing.m.

Reported by wsheldon, 7 years ago.

The fetch_eml_data.m function only parses select elements from an EML document beyond the entity and attribute metadata (title, abstract, codes, creator). A more complete metadata structure should be defined to store all general content for populating a GCE data structure using eml2gce.m.

Reported by wsheldon, 7 years ago.

All metadata templates are stored in one file per toolbox instance (imp_templates.mat), but as users start upgrading and sharing more templates it would be useful to support importing or merging templates stored in multiple copies of imp_templates.mat. This should be added to ui_template.m as a new button or menu option.

Reported by wsheldon, 7 years ago.

The menu options for calculating or converting date formats in ui_editor.m can be confusing to new users. These options should be simplified for clarity, adding redundant menu items for functions with multiple functionality like add_datecol.m (i.e. calculate date from date components or convert format of existing date column).

Reported by wsheldon, 7 years ago.

The normalize_cols.m function can generate parameter-name, parameter-value pairs from standard tabular data sets, but merging columns is restricted to columns with matching data types and units. An option should be provided to automatically cast columns to a specified datatype (e.g. floating-point) and precision and to ignore unit differences to support exporting to skinny data models like CUASI ODM where units and scale are associated with the parameter id.

Reported by wsheldon, 7 years ago.

The imp_matlab.m function excludes structures and fails on numeric arrays that are not type 'double'. This function should call imp_struct.m to import structure fields and support other numeric types (e.g. uint8, uint16). This would be particularly useful for importing structures returned from the database toolbox.

Reported by wsheldon, 7 years ago.

The Q/C criteria GUI (ui_qccriteria.m) only supports form-based definition of limit-check criteria and custom flag function criteria. Form support should be extended to include defining multiple-column criteria as well.

Reported by wsheldon, 7 years ago.

Metadata can be exported from ui_editor.m to create a new template, but existing templates cannot be updated the same way. An option to update attribute metadata, documentation metadata or both in a template from an open data set should be added for completeness.

Reported by wsheldon, 7 years ago.

Reported by Joan Sheldon: The o2_saturation.m function includes code to calculate air saturation in mg/l or ml/l, but the function cannot be called without oxygen concentration to return only air saturation at a given T and S. This function should be split so either property can be returned separately depending on needs.

Reported by wsheldon, 7 years ago.

The GUI title editor ui_title.m includes code to add or update date ranges in data set titles. A command-line version would allow this functionality to be used in scripted workflows.

Reported by wsheldon, 6 years ago.

It would be useful to have an option for exp_ascii.m and ui_expascii.m to specify a date format for matlab serial dates, similar to the option in gce_distribfiles.m.

Reported by wsheldon, 6 years ago.

The "Correct Drift" dialog requires starting and ending date/time values to specify the data range to correct, but filling in these values is difficult without seeing the data on a plot. Adding a plot tool with the ability to click on starting and ending data values would simplify using this tool.

Reported by wsheldon, 7 years ago.

When a new data set is loaded into ui_editor.m, the undo cache structure isn't reset so selecting undo loads the prior data set.

Reported by wsheldon, 7 years ago.

Batch exporting data structures is currently supported in the GUI search engine (ui_search_data.m), but it would also be useful to process all files in a directory without first indexing them. Expanding the text file export dialog ui_exportasc.m to support batch exporting would be useful.

Reported by wsheldon, 7 years ago.

Ascii files with a single-line header fail to parse when the first column contains string values because the header length cannot be reliably determined. Previous versions of imp_ascii.m allowed for this pattern, but the latest re-write to support textscan-based parsing apparently broke the behavior.

Reported by jsheldon, 7 years ago.

After defining custom text parsing options the dialog closes automatically on success, so if the user hasn't selected the option to save a custom import filter when opening the dialog they have to repeat the entire process. The save option should be checked by default, then the user has 2 options for opting out of saving a new filter (unchecking the box and cancelling the m-file save).

Reported by wsheldon, 7 years ago.

LTER NIS web services require HTTPS-based authentication, but MATLAB's native urlread/urlwrite only support HTTP and FTP. An open source network library (cURL) needs to be wrapped to support interaction with the NIS via WS interfaces.

Reported by wmsheldon@earthlink.net, 9 years ago.

Adding support for parsing metadata and downloading data tables described using EML will allow leveraging of the GCE Data Toolbox for a broader array of LTER data sets.

Reported by wsheldon, 7 years ago.

The ui_importfilter.m dialog for custom text file parsing does not include a field for specifying the field delimiter, but failing to provide a delimiter to the new textscan-based parser in imp_ascii.m can result in parsing failures (e.g. all NaN for numeric fields).

Reported by wsheldon, 7 years ago.

Batch processing raw data files requires running batch_import.m from the command line. All other parts of the parsing process can be performed from GUI dialogs, so it would be helpful to run batch importing from the GUI as well for parity.

Reported by wsheldon, 7 years ago.

The textread.m function is now deprecated in MATLAB, and this functionality is now provided by textscan.m. In addition, textscan supports an array of user-specified missing value codes (emptyValue argument), obviating the need to rewrite text files using imp_filter.m in many cases which should dramatically speed parsing. The imp_ascii.m and related functions should be rewritten to support textscan when present (i.e. MATLAB 7+).

Reported by wsheldon, 7 years ago.

Calls to imp_filter.m in import filters and GUI dialogs should be rewritten to use the revised imp_ascii.m function so the optimal MATLAB routine can be called to avoid unnecessary text file rewriting.

Reported by wsheldon, 8 years ago.

When creating a new metadata template, documentation metadata content cannot currently be imported directly from another template without first saving it to a file. A dynamic menu of metadata templates should be added to ui_editmetadata.m for parity with ui_editor.m to facilitate this.

Reported by wsheldon, 7 years ago.

NOAA stopped updating the legacy "Summary of the Day" reports for NWS COOP stations in Dec 2010, preventing fetch_ncdc* functions from retrieving the latest observations. Functions for parsing data from the Global Historic Climate Network daily files should be developed to replace this functionality.

Reported by wsheldon, 9 years ago.

Value codes are currently stored in a documentation metadata field (Data/ValueCodes) and not as dedicated attribute metadata fields. However, it would be easier to manage value codes along with other attribute metadata in metadata templates. Relevant codes could be synchronized to Data/ValueCodes metadata field when the template is applied, therefore avoiding the need for global changes to handling of code values in the toolbox.

Reported by wsheldon, 9 years ago.

Currently data structures can be loaded from .mat files containing multiple data sets, but compound files cannot be created from ui_editor directly. Support should be added for saving named variables to files.

Reported by wsheldon, 8 years ago.

The query builder dialog (ui_querybuilder.m) used to search or filter records does not support filtering by missing values (NaN or null strings). The query parser should be extended to catch NaN or null values and generate calls to isnull or ~isnull as appropriate to support these use case.

Reported by wsheldon, 8 years ago.

The update_data.m function logs value changes but does not assign flags to revised values. Support for flagging updates should be added.

Reported by wsheldon, 8 years ago.

Sensor drift is a common problem with environmental data, so support should be added for correcting values over specified date ranges using constant offset, linearly-varying offsets or custom offsets and flagging the revised values.

Reported by wsheldon, 7 years ago.

Q/C rules are currently not re-evaluated after performing unit conversions, under the assumption that range-check criteria are also converted so recalculation is unnecessary. However, if custom functions or algorithms are defined referencing the column then these criteria should be re-evaluated so that flags correctly reflect the updated values.

Reported by wsheldon, 7 years ago.

Exporting data in CSV format using exp_ascii.m results in quotes around all text values whether they include commas or leading/trailing spaces or not. Quotes around datetime values prevent proper parsing using the LTER NIS Data Management Library, though, so they should not be used.

Reported by wsheldon, 8 years ago.

Harvest timer scripts should be included in the toolbox distribution files and the process for managing timed harvests should be documented. The harvest timer structures should also be refactored as GCE data structures to allow GUI editing.

Reported by wsheldon, 8 years ago.

Importing array-based Campbell Scientific Instruments logger files required running 'csi2struct' from the command line. An import filter should be developed to wrap this function to simplify importing CSI array data into the GUI data editor.

Reported by wsheldon, 8 years ago.

The hammock well import filters (imp_aquatroll, imp_minitroll, imp_schlumberger, imp_hydrolabs) use add_well_metadata to add hammock and well ids, latlon, and well metrics to hammock well data. Now that surface elevation is available from RTK surveys add_well_metadata should be revised to include sensor elevation relative to NAVD88 datum and functions should be written to calculate water level from corrected water depth and sensor elevation

Reported by wsheldon, 8 years ago.

The utilities for importing and exporting metadata between GCE Data Structures and the GCE_Metabase database (i.e. imp_gcemetadata.m, exp_datasetfiles.m, exp_datasetvariables.m) are not fully compatible with the GCE_Metabase2 schema. In particular, the new DataSetEntities? portion of the model that allow multiple entities to be included for a single data set. Changes to overall program logic (e.g. support for named entities) and SQL statements are required.

Reported by wsheldon, 8 years ago.

The Campbell Scientific Instruments table-oriented ASCII format (TOA5) is commonly used for environmental monitoring, and a dedicated filter that parses column names and formats would simplify loading CSI data into the toolbox.

Reported by wsheldon, 8 years ago.

Support for USGS data in the toolbox is currently limited to NWIS streamflow and water quality data. Adding support for nutrient data from lab/analytical sampling would greatly enhance the data mining capabilities of the toolbox.

Reported by wsheldon, 8 years ago.

The cols2flags_multi.m function is currently only accessible from the command line. This should be added as a menu option in the data set editor GUI.

Reported by wsheldon, 8 years ago.

The toolbox currently supports text search/replace, but functions should also be included to support numeric value search/replace (including NaN values) as well.

Reported by wsheldon, 8 years ago.

The function flags2cols.m currently generates flag columns for all data columns with any assigned flags, and does not support filtering which flags are converted. A more flexible function allowing specification of columns and flags is needed for parity with other Q/C flag functions (like copyflags.m and clearflags.m).

Reported by wsheldon, 8 years ago.

Interpolation functions interp_missing and interp_missing2.m currently assign and define an E flag when filling in data values. This should be user-selectable.

Reported by wsheldon, 8 years ago.

When data set metadata includes lots of content (e.g. from large multi-merge operations), rendering the metadata can be very slow.

Reported by wsheldon, 8 years ago.

Geographic site (polygon) and location (point) names can contain typos and other errors that prevent generation of geographic metadata. Q/C flag functions should be added to check place names against registered names in geo_polygons.mat and geo_locations.mat databases.

Reported by wsheldon, 8 years ago.

Some data sets include geographic location names (or GPS coordinates), but not site/transect (polygon) names which may be useful for data integration and search. A function should be added to look up site codes from location names and add the matched sites as a data set column.

Reported by wsheldon, 9 years ago.

In the current XML implementation some attribute metadata, like code definitions and calculations, are listed in a separate node from the attributes they describe. All attribute descriptors should be nested in the same node structure.

Reported by wsheldon, 9 years ago.

The NCDC data harvester (fetch_ncdc_climate_url.m) and parser (imp_ncdc_climate.m) only support the simplified summary of the day format for air temperature and precipitation. Some stations with WBAN ids also include additional climate variables, which require the advanced summary of the day format be used.

The harvester and parser functions should be revised to support either format.

Reported by wsheldon, 9 years ago.

Currently GUI dialog preferences and other user data (registration data, unit conversions, metadata templates) are stored in directories along with the corresponding function. These files should be stored in one place, ideally a dedicated directory, to facilitate toolbox version upgrades and migration to other systems.

Reported by wsheldon, 8 years ago.

The GUI query builder (ui_querybuilder.m) does not include the IN and NOT IN constructs used the the GUI Q/C criteria editor for list-based selection. These should be added to simplify adding restrictions based on lists of text or numeric values.

Reported by wsheldon, 8 years ago.

The decodecols.m function to convert coded columns to nominal string columns should be added as an option to the data editor (ui_editor.m)

Reported by wsheldon, 8 years ago.

Search indices generated by search_index.m and search_index_r13.m omit information about column data type and description. This information is needed to generate dataset detail web pages (e.g. in the GCE Data Portal), and to perform more advanced searches based on column characteristics.

Reported by wsheldon, 8 years ago.

The functions for uploading metadata to the GCE Metabase need to be revised to support the new structure of entity and attribute tables in GCE_Metabase2.

Reported by wsheldon, 9 years ago.

The GCE Data Toolbox will be released under a GPL license beginning with version 3. This requires inclusion of the GPL license statement in file headings below the copyright statement, and inclusion of a license file.

Reported by wsheldon, 9 years ago.

Storing all core toolbox files in a single directory is potentially overwhelming for new users. Files should be organized into subdirectories by type, e.g. core, util, gui, qaqc, custom.

Reported by wsheldon, 9 years ago.

The NCDC data import command in the data editor dialog currently opens a station selection dialog. A GUI dialog should be developed for parity with ClimDB and USGS, allowing selection of date range and metadata template.

Reported by wsheldon, 9 years ago.

Currently parse_usgs.m mines some metadata from the USGS header, but relies on metadata templates to supply attribute metadata and qa/qc flag definitions. When an attribute is not matched in the template the import fails because a valid datatype code is not supplied in the pre-filtered text file. The NWIS RDP format now includes more information about attributes, formats and flags and this information should be parsed and applied as metadata first, then augmented with template metadata to prevent parsing errors for files with new variables.

Reported by wsheldon, 9 years ago.

The USGS station dataset (usgs_stations.mat) is several years old. Updating this file should be dynamic like ClimDB station data using http/xml/xslt to retrieve and parse NWIS station lists.

Reported by wsheldon, 9 years ago.

Harvesting data from the NOAA NCDC web site requires several form posts to retrieve a data URL, then a delay before downloading the data. A harvesting program to automate this process would make it much easier to retrieve NWS data from this database.

Reported by wsheldon, 9 years ago.

Currently to copy a column using the data editor GUI (ui_editor.m) users have to use the 'Add calculated column' dialog, which is awkward. An option to copy the selected column(n) should be added to the Edit menu.