The Near Real Time Processing Effort

Algorithm employing multispectral MODIS data enhances dust depiction over both land and water. (NRL/MRY)

The Near Real Time Processing Effort

Jim O'Neal

The Aerospace Corporation played a key role in helping NASA provide data from environmental research satellites to operational weather organizations. These efforts have made a significant contribution to military and civil programs for monitoring global weather and climate.

NASA's Earth Observing System (EOS) is a research mission designed to study Earth and its environment. Operating in sun-synchronous polar orbit, the EOS satellites, Terra and Aqua, rely on two primary remote-sensing instruments. The first, the Moderate-resolution Imaging Spectroradiometer (MODIS), has 36 spectral bands for measuring visible and infrared radiation; it can detect dust over land and ocean, detect low clouds at night, and generate high-resolution true-color imagery. The second instrument, the Atmospheric Infrared Sounder (AIRS), is a grating spectrometer containing 2378 infrared channels and four visible/near-infrared channels; it's designed to obtain highly accurate temperature profiles of Earth's atmosphere along with a variety of other environmental parameters.

NASA recognized that data from EOS could be useful to civilian and military weather services; however, EOS data latency (or timeliness) was too slow to permit use in operational activities. NASA initiated a program known as the Near Real Time Processing Effort (NRTPE) to try to reduce data latency. The self-imposed latency goal was 3 hours from time of observation by the satellite to product delivery to the users.

Terra satellite data latencies for March 2004. The diamonds depict average daily latency, and the horizontal line corresponds to the NRTPE 3-hour latency goal. (NRL/MRY)

Challenges

NASA asked Aerospace to assemble the NRTPE team. In addition, an Aerospace representative was appointed lead systems engineer, providing day-to-day management of the entire effort. The NRTPE team included members from NASA, the National Oceanic and Atmospheric Administration (NOAA), the Air Force Weather Agency, the Naval Research Laboratory, and the Naval Oceanographic Office.

The first challenge involved the identification, procurement, and installation of the high-speed processing and distribution systems needed to achieve the 3-hour latency goal. NASA allocated $3 million for the project—not much, considering the sort of infrastructure that would be needed. That meant that NRTPE would have to make maximum use of existing systems; however, it was clear from the outset that additional high-speed processing capability would have to be procured. Hence, the bulk of NASA funding was used to acquire high-speed computers, a control terminal, data storage, and system maintenance. An active NASA contract enabled the rapid purchase and installation of the processing capability. The system was then loaded with NASA software for generating weather-data packages.

While the enhanced processing capability was being procured, the NRTPE team began implementation of the networks needed for distribution of data from Terra and Aqua. This effort was complicated because users were scattered at varied locations and had widely differing system capabilities. With virtually no funding to purchase a new communication/distribution system for all participants, the NRTPE team had to make use of the several different existing systems. The group evaluated various alternatives and determined that the best solution for the military was the Defense Research and Engineering Network. For civil customers, a combination of internal networks and shared processing protocols would be used.

Latency Reduction

Once the new processing systems were integrated with the old data distribution networks, testing began to determine their ability to achieve the 3-hour latency goal. In 2002, initial testing revealed latencies in the 8–10 hour range—well beyond that needed for most operational weather applications. The NRTPE team began a series of activities designed to optimize the flow of satellite data throughout the system, including modification of existing ground processing equipment and procedures. Several of these efforts led to a significant reduction in data latency.

Top-level schematic of the EOS Terra/Aqua data flow. Key: GSFC—Goddard Space Flight Center; NESDIS—National Environmental Satellite, Data, and Information Service; NRL/MRY—Naval Research Laboratory, Monterey; AFWA—Air Force Weather Agency; FNMOC—Fleet Numerical Meteorology/Oceanography Center; NAVOCEANO—Naval Oceanographic Office.

For example, Terra satellite data are uplinked to the Tracking and Data Relay Satellite System (TDRSS) and then downlinked to the receiving station in White Sands, New Mexico. Prior to NRTPE, the operational procedure was to downlink the Terra data once per orbit. Thus, because of the orbital period, some of the data were already 100 minutes old when downlinked. Aerospace and NASA arranged for a dual TDRSS contact schedule, which reduced data latency by nearly 50 minutes.

The Atmospheric Infrared Sounder (AIRS) is a grating spectrometer containing 2378 infrared channels and four visible/near-infrared channels designed to obtain highly accurate temperature profiles of Earth's atmosphere and related information. As a result of the NRTPE, NOAA is processing and distributing global AIRS calibrated and navigated data for use by numerical weather prediction centers in various products, including forecast models. Shown here is an example product using AIRS data. (NOAA/NESDIS)

The NRTPE team also expedited the return link from White Sands. Previously, the data playback from White Sands to NASA's Goddard Space Flight Center was done manually. With additional funding provided by NASA, this process was automated, significantly reducing data playback time.

Further gains were made via data streaming. Prior to NRTPE, data from Terra would be held at White Sands until the complete orbital data set was received. The information was then forwarded to Goddard Space Flight Center. The NRTPE defined and implemented a new procedure to immediately stream the data back to Goddard.

Conclusion

These latency improvements, coupled with many other processing and procedural upgrades, resulted in data latencies of 2–3 hours. Data provided by the NRTPE system have made and continue to make significant contributions to military and civil applications. As a result of NRTPE, NOAA is processing and distributing global AIRS calibrated and navigated data with short lag time for use by numerical weather prediction centers in various products, including forecast models. By providing MODIS data to the operational weather centers, NRTPE enabled new applications based on previously unavailable spatial and spectral information. Notably, the NRTPE system was in place before the conflict in Iraq, enabling generation of new products to support the war effort.