pax

Data warehousing technology supports information management for decision making by integrating data from operational systems and external sources in a separate database, the data warehouse. In contrast to operational systems which store detailed, atomic and current data accessed by OLTP (on-line transactional processing) applications, data warehousing technology aims at providing integrated, consolidated and historical data for OLAP (on-line analytical processing) applications. As time advances and the sources from which warehouse data is extracted change, data warehouse contents must be ãsynchronizedä with the underlying sources. Thus, after an initial loading phase, warehouse data must be regularly refreshed, i.e., modifications of operational data since the last warehouse refreshment must be propagated into the warehouse such that warehouse data reflect the state of the underlying systems. Data warehouse refreshment is a complex process comprising several tasks like monitoring, extracting, transforming, integrating and cleaning data, deriving new data, building histories and loading the data warehouse. This thesis proposes an approach which allows for the modeling and execution of the data warehouse refreshment process at an intermediate layer between operational sources and the target data warehouse, and provides support for several tasks related to warehouse refreshment in data warehouse environments consisting of a wide variety of heterogeneous sources, and independently of how target warehouse data is stored persistently. The contribution of the thesis is twofold. First, the thesis proposes a metamodel which includes a set of modeling constructs for the definition of metadata required for executing the data warehouse refreshment process, such as the description of operational and target warehouse data, the definition of mappings between operational and warehouse data, including refreshment tasks like transformation, cleaning, integration, process scheduling, etc. Second, the thesis describes a methodology and an appropriate operational infrastructure for defining and executing concrete refreshment processes based on the above mentioned metamodel. The definition of the refreshment process is supported by a metadata administration tool which provides facilities for creating, modifying and checking metamodel instances stored in an object-oriented metadata repository. Furthermore, the thesis describes the way various monitoring techniques can be integrated in a data warehouse environment in order to detect updates in operational systems and refresh the warehouse incrementally. An advanced key management concept is provided which ensures the correctness of keys propagated into the target warehouse, thus improving data quality in the target warehouse. Finally, target warehouse mapping and history management techniques are introduced which consider the specific database design techniques used for data warehouses, as well as the multidimensional and temporal character of warehouse data.

A metadata-driven approach for data warehouse refreshment

A metadata-driven approach for data warehouse refreshment

actions