Paper Info
Title
Deriving and Managing Data Products in an Environmental Observation and Forecasting System
Abstract
Large-scale scientiflc work∞ows can perform many computationally intensive tasks and generate large volumes of derived data prod- ucts. These systems pose many challenges to both creating and managing data products, includinge-cientlyexecutingtasksandtrack- ing data product lineage and metadata. In thispaperwedescribeourexperiencesimple- menting an experimental data-product man- agementsystemtoaddressthesechallengesfor the CORIE Environmental Observation and Forecasting System. We present a novel ar- chitecture to store both data products and the tasks that create them. Our system in addition supports tasks to automatically per- form system maintenance, and enables data- intensivetaskstoexecuteonmultiplenodesof a Grid. We present several challenges to exe- cutingexistingscientiflcwork∞owsonaGrid, and propose several techniques to improve task scheduling in this environment. Prelim- inary performance results show the potential beneflts of these techniques.
Year
Venue
Field
2005
CIDR
Derived Data,System maintenance,Data mining,Metadata,Architecture,Computer science,Scheduling (computing),Data products,Database,Grid
DocType
Citations 
PageRank 
Conference
5
0.59
References 
Authors
9
2
Name
Order
Citations
PageRank
Laura Bright117617.34
David Maier256391666.90