data lineage

Data lineage is a representation of the path along which data flows from the point of its origin to the point of its usage. Data lineage is used to design and describe processes of data transformation and processing.

Data lineage is recorded by representing a set of interlinked components such as data (elements), business processes, IT systems and applications, data controls. These components can be presented at different levels of abstraction and detail.

Scope of Data Lineage

The scope of a data lineage project is usually based on the following:

• A list of critical data elements (CDE). Critical data elements are those that derive from the scope of your data management initiative and are those that make the greatest impact on performance and outcomes.
• The "length" of your horizontal data lineage. Horizontal data lineage represents the path along which data flows starting from its point of origin to the point of its usage.
• The "depth" of your vertical data lineage. Links between the components of data lineage on these different levels (conceptual, logical, and physical) are very often called vertical data lineage.

Out-of-the-box, the Orion Governance EIIG Data Fabric can identify and govern critical data elements, and produce fine-grained, highly detailed, automated data lineage of both horizontal and vertical data lineage, as well as mapping between physical and logical levels.

Data Warehouse and BI Use Case

Data warehouses and BI reporting are examples where lineage information information is essential.   It is common that business users discover inaccurate information in critical reports and lose precious time waiting for the BI group to respond. 

Without automated data lineage, BI groups have to manually trace the data and analyze which data warehouse table and ETL processes are involved in generating the inaccuracy in the reports—a very expensive, error-prone, and time-consuming process.

Metadata-driven automated data lineage has been shown to:

• Cut the search and tracing costs by more than 50%
• Zero disruption or production issues when solving the problems
• Much faster time to improve the accuracy of the BI reports