Why Data Lineage matters
As you keep adding pipelines while building and managing pipelines in Foundry with code, at some point it gets hard to see at a glance “where did this Dataset come from, and where does it get used.” Data Lineage tracks the entire creation, transformation, and consumption history of a piece of data in real time. It’s less a static after-the-fact record and more of a living data map wired into Foundry’s metadata system.
Core features
- Bidirectional tracing (upstream/downstream): for a given Dataset, see both where it came from (upstream) and where it’s being used (downstream). Before editing pipeline code, you can check ahead of time which dashboards or applications the change would affect (impact analysis).
- Column-level lineage: beyond the table level, trace which transformation logic a specific column went through, all the way down to field level, to reach the final Dataset. Especially useful for checking how sensitive data flows across an entire pipeline.
- Automatic inheritance of security markings: security markings or access permissions set on source data are automatically inherited down the lineage to derived Datasets and downstream analysis results.
- Integration with build history: wired into Foundry’s Git-like branching model, so you can trace which code change at which point in time caused a given issue.
- Health-check overlay: each Dataset’s build status and quality-check results are displayed directly on the lineage graph, so you can immediately see exactly where a problem originated.
How it’s used in practice
- Check impact before changing a pipeline: before editing code, look at downstream first to see which dashboards and apps would be affected.
- Trace sensitive data at the column level: when personal information or other sensitive fields are involved, check how far they’ve propagated using column-level lineage instead of table-level.
- Open lineage first when a quality issue shows up: if a specific Dataset’s values look wrong, trace back through the lineage graph to find which upstream Dataset or Transform step the problem started at.
Common mistakes
- Editing a pipeline before checking the impact of the change: modifying an upstream Dataset can affect several downstream applications at once. Skip checking downstream first, and outages show up in places you didn’t expect.
- Arbitrarily lowering a security marking partway through a pipeline: an inherited security marking is usually there for a reason. If the marking drops somewhere along the lineage, data that was supposed to stay protected can end up exposed further downstream.
- Only opening lineage after something breaks: if you haven’t already built a mental map of the lineage structure, it takes time to even figure out where to start looking once an incident actually happens. It helps to get in the habit of skimming the lineage of your key pipelines ahead of time.
Next steps
To revisit the pipeline logic itself, go back to Pipeline Builder or Code Repository. To check how this data maps onto the ontology, revisit Ontology Manager.
Frequently Asked Questions
Q. Is Data Lineage recorded automatically?
Yes. Lineage is recorded automatically as part of Foundry’s pipeline and ontology connection structure — there’s no separate documentation step required.
Q. When is checking lineage actually useful in practice?
It’s especially useful when a pipeline’s output looks wrong and you need to trace the cause backward, or when you want to check ahead of time which downstream Datasets and applications would be affected before changing a source data schema.
Q. Is Lineage the same thing as an audit log?
No. Lineage shows the transformation path data has traveled through; an audit log records who did what and when. Regulatory compliance often requires both.
Q. What if the lineage graph gets too complex?
It’s common to use the filter feature to narrow the upstream/downstream range around a specific Dataset or Object Type.
Q. What’s a common mistake?
Opening Lineage for the first time only after something has already broken. It’s better to build the habit of checking structure through the Lineage view starting from the pipeline design stage.