Data Turbine Harvesting

Background

The Data Turbine MATLAB Toolkit (DTMatlabTK) developed in 2013 provides convenient functions for retrieving data and information from OSDT servers within the MATLAB environment. These functions were found to be very robust in real-world testing; however, several potential barriers to using them in production for data harvesting and management were identified:

  • The date/time convention used by most OSDT servers and DTMatlabTK is seconds since 1970 (Unix/Java time), whereas MATLAB date/times are in days AD/CE
  • Data are returned from the primary DTget.m function as a nested struct array, with independent time and data arrays for each channel; time ranges and array sizes may not always match, depending on source configuration and data loading issues
  • No functions are provided for integrating channel data from multiple retrievals

Software Development

To address these issues, utility functions were developed by Wade Sheldon for inter-operating between the OSDT and MATLAB date/time conventions (date2sec.m, sec2date.m, getUTCoffset.m). Functions were also developed for retrieving data from OSDT by calendar date (DTlatest.m), returning aligned channel data in a simplified structure more amenable to use with standard MATLAB plotting and data analysis programs (DTalign.m), and integrating data from multiple retrievals (DTcombine.m, DTappend.m).

These extensions were then leveraged to develop a comprehensive data harvesting application (DTharvest.m) that supports fully automated data retrieval and integration using native MATLAB timer objects. Options are supported for console and email alerts, operation logging, harvest job management (list/start/stop/resume) and execution of post-harvest workflows defined by the user. A schematic representation of this functionality is provided below, and all of these functions will be released as part of the DTMatlabTK library from the Open Source Data Turbine web site.

Harvesting Schematic

Data Turbine Harvesting
Figure. Sensor data harvesting using Data Turbine and the extended DTMatlabTK