ARM Data Integrator (ADI) Documentation

The ARM Data Integrator (ADI) is a suite of tools, C libraries, structures, and interfaces developed to simplify the development of algorithms to analyze time-series data and decrease the costs associated with such development. The architecture and functionality of ADI is designed to consolidate diverse time-series datasets into one or more new data products without the need to write any code.

Using ADI to develop an algorithm:

  • Enforces ARM standards, thereby enabling easy search and discovery of products and on-the-fly data integration

  • Documents dependencies, metrics, status, and logs

  • Automates reprocessing

  • Captures provenance

  • Significantly decreases the amount of resources needed to implement an algorithm

  • Produces more robust processes through the use of heavily used and well tested libraries and functions.

_images/overview_ADI_architecture.png

The core components of ADI include a graphical interface, a database that stores information, and a framework to automate tasks such as reading input, consolidating datasets, and writing output. An algorithm can be integrated with the ADI framework by following these steps:

  • Design the desired output data product

    • datastream name(s), dimensions, variables, global attributes

  • Enter process information

    • defining the location, processing interval, output data products

  • Enter variable retriever information into ADI interface

    • variables to retrieve and data source from which to retrieve them

    • map retrieved variables to output data products as appropriate

    • retrieve companion QC as appropriate

  • Setup coordinate transformation in interface

    • type of transformation to apply

    • parameters to apply to transformation

  • Run data consolidation application and review output

  • Implement and run user designed algorithm.

The following languages are supported through the ADI interface:

  • C (gcc 4.4.7+)

  • Python 2.7+

  • IDL, version 8.2+.

ADI is available to use on OS-X and can be downloaded at https://github.com/ARM-DOE/adi-macosx

For questions about ADI, please contact us at adisupport@arm.gov

***Draft Document***

This document is under construction. Some sections are incomplete, and some contain information that is in the process of being revised.

Table of Contents