Overview

The ARM Data Integrator (ADI) is a suite of tools, C libraries, structures, and interfaces developed to simplify and decrease the costs associated with the development of algorithms that analyze time-series data. To the extent possible, ADI is intended to minimize the resources needed to implement an algorithm.

These algorithms are referred to as Value Added Products (VAPs) because they apply additional processing to existing data to create a new product with added value greater than that of the individual input data products. The initial goal in developing ADI was to minimize the development activities to that of only the scientific algorithm, by eliminating the need for the algorithm developer to implement repetitive and time consuming pre- and post- data processing tasks. Once complete, it was realized that independent of a need to develop a scientific algorithm, the architecture and functionality of ADI is essentially that of a generic time-series data integrator that can consolidate diverse datasets into a one or more new data products without the need to write any code.

The core components of ADI include a graphical interface through which a VAP’s general characteristics, inputs, outputs, and specifics of any transformations to be applied are gathered, a database that stores this information, and a framework with the modules and supporting functions needed to automate the general tasks involved in data processing. The graphical interface used to collect details of the process is referred to as the Process Configuration Manager (PCM), and the database is referred to as the Data System Data Base (DSDB).

VAP algorithms access the information in the database through a web service and process the data through the ArmADI data framework. Products of this processing include the output data products, logs, and metrics.

The Data Consolidator is a supporting application that is a special instance of a VAP that does not include any additional value added logic. It retrieves, transforms, and creates the data products specified in the PCM. These actions, common to most time-series data processing, are executed by modules residing in the Data Processing Framework component of ADI, shown in green in the following figure.

A VAP utilizes these same core components, but also injects its own processing into the data processing pipeline through one or more of the user hooks shown in blue.

A code generation Templater application is provided to produce an initial set of code that executes the main library modules and has empty hooks, as is a library of functions to access and manipulate the data stored in ADI’s internal data structures.