The Matsu Wheel: a reanalysis framework for Earth satellite imagery in data commons

Maria T. Patterson, Nikolas Anderson, Collin Bennett, Jacob Bruggemann, Robert L. Grossman, Matthew Handy, Vuong Ly, Daniel J. Mandl, Shane Pederson, James Pivarski, Ray Powell, Jonathan Spring, Walt Wells, John Xia

Research output: Contribution to journalArticlepeer-review

Abstract

Project Matsu is a collaboration between the Open Commons Consortium and NASA focused on developing open source technology for the cloud-based processing of Earth satellite imagery and for detecting fires and floods to help support natural disaster detection and relief. We describe a framework for efficient analysis and reanalysis of large amounts of data called the Matsu “Wheel” and the analytics used to process hyperspectral data produced daily by NASA’s Earth Observing-1 (EO-1) satellite. The wheel is designed to be able to support scanning queries using cloud computing applications, such as Hadoop and Accumulo. A scanning query processes all, or most, of the data in a database or data repository. In contrast, standard queries typically process a relatively small percentage of the data. The wheel is a framework in which multiple scanning queries are grouped together and processed in turn, over chunks of data from the database or repository. Over time, the framework brings all data to each group of scanning queries. With this approach, contention and the overall time to process all scanning queries can be reduced. We describe our Wheel analytics, including an anomaly detector for rare spectral signatures or anomalies in hyperspectral data and a land cover classifier that can be used for water and flood detection. The resultant products of the analytics are made accessible through an API for further distribution. The Matsu Wheel allows many shared data services to be performed together to efficiently use resources for processing hyperspectral satellite image data and other, e.g., large environmental datasets that may be analyzed for many purposes.

Original languageAmerican English
Pages (from-to)251-264
Number of pages14
JournalInternational Journal of Data Science and Analytics
Volume4
Issue number4
DOIs
StatePublished - Dec 1 2017
Externally publishedYes

ASJC Scopus subject areas

  • Information Systems
  • Modeling and Simulation
  • Computer Science Applications
  • Computational Theory and Mathematics
  • Applied Mathematics

Keywords

  • Data commons
  • Earth satellite data
  • Reanalysis framework

Fingerprint

Dive into the research topics of 'The Matsu Wheel: a reanalysis framework for Earth satellite imagery in data commons'. Together they form a unique fingerprint.

Cite this