# Lineage **Lineage** gives you a built-in, visual map of how data flows through your project: which jobs feed each [table](/en/product/lakehouse-manager/tables/index.md), which tables they produce in turn, and how everything connects from source to destination. The graph is built from the [OpenLineage](https://openlineage.io/) events your project records as jobs run, with nothing to install or configure. Lineage is recorded **automatically** for Load and Aggregate actions (Python and PySpark), and is **opt-in** for Custom actions and notebooks through the SDK. Lineage for Trino queries (Explorer, Analytics Manager) is coming soon. See [what generates lineage events](#what-generates-lineage-events) below. ![lineage-overview](picts/lineage-overview.png) * [Open the lineage view](#open-the-lineage-view) * [Explore the graph](#explore-the-graph) * [Table nodes](#table-nodes) * [Job nodes](#job-nodes) * [Graph controls](#graph-controls) * [Column-level lineage](#column-level-lineage) * [What generates lineage events](#what-generates-lineage-events) * [Use your own lineage tool](#use-your-own-lineage-tool) --- ## Open the lineage view There are two ways to reach the lineage of a table: - From the **Lineage** section of the Lakehouse Manager: the page opens on a **search bar** where every table of your project is searchable. Start typing the name of the table whose lineage you want to see and select it to display its graph. - From a **table's details view**: click the **Lineage** button to jump to the lineage page with that table already displayed. ![lineage-search](picts/lineage-search.png) --- ## Explore the graph The graph starts out from the table you selected: your **central node**, marked with a blue outline so you never lose track of where the graph begins. Around it, the graph shows the tables and jobs it connects to: the jobs and tables your table is built from **upstream**, and everything built from it **downstream**. The graph is made of two kinds of nodes: **table nodes**, which list their columns, and **job nodes**, the runs that link tables together by reading and writing them. ?> The graph is never empty: a table without lineage yet is simply displayed on its own, without connections. Lineage appears as soon as a job run finishes, so a fresh run shows up immediately. ![lineage-graph](picts/lineage-overview.png) ### Table nodes Click a table node to see the table's details: its **dataset**, its **columns**, the **producing job(s)** that write it, and the **consumer job(s)** that read it. Clicking another table node shows its details but does not re-center the graph: to start out from a different table, select it from the **Lineage** search bar instead. ![lineage-table-node](picts/lineage-table-node.png) ### Job nodes Click a job node to see the run behind the connection: | Detail | Meaning | | :--- | :--- | | **Job name** | The name of the job that produced the lineage event | | **Kind** | **DPE** (an action run), **Lakehouse** (the automatic load when a table is built), or **SDK** (custom tracking emitted from your own code) | | **Action type** | **Load**, **Aggregate**, or **Custom** | | **Last run** and **status** | When the job last ran and how it ended | | **Input tables** and **output tables** | What the job read and what it wrote | For **DPE** and **Lakehouse** jobs, a direct link at the bottom of the details opens the job itself. **SDK** jobs have no link, since they are lineage you emitted yourself through the [SDK helpers](/en/technical/sdk/dpe/3E.lineage.md). ![lineage-job-node](picts/lineage-job-node.png) ### Graph controls The controls at the top right adjust what the graph displays: - **Full graph** toggle: also shows the **sibling tables**, on top of the producers and consumers already displayed. - **Compact nodes** toggle: a minimal view of the nodes, without their columns. - **Zoom in**, **zoom out**, and **fit to screen**. - **Depth**: how many upstream and downstream levels the graph displays, from **1** to **10** (default: **2**); increase it to follow the flow further. --- ## Column-level lineage Beyond table-to-table flows, Lineage also tracks how **columns** connect across tables: for a given column, which upstream columns it is computed from, and which downstream columns are computed from it. Click a column on a table node in the graph, or a column in the table details, to switch to its column lineage. When lineage is available, the graph connects the columns across tables to show how they relate. A column with no lineage of its own is shown as independent, with a blue outline marking it as solo. ![lineage-columns](picts/lineage-columns.png) ?> Column-level lineage is not available while **Compact nodes** is enabled: switch the toggle off to click through columns. --- ## What generates lineage events | Source | Lineage | | :--- | :--- | | **Load** and **Aggregate** actions (Python and PySpark) | Automatic, with column-level detail | | **Custom** actions (Python and PySpark) | Opt-in: emit events from your code with the [SDK lineage helpers](/en/technical/sdk/dpe/3E.lineage.md) | | **Notebooks** (Python and PySpark) | Opt-in, using the same [SDK lineage helpers](/en/technical/sdk/dpe/3E.lineage.md) | | **Trino queries** (Explorer, [Analytics Manager](/en/product/am/index.md)) | Coming soon | --- ## Use your own lineage tool The same OpenLineage events can be forwarded to an external lineage solution such as [Marquez](https://marquezproject.ai/). Configure the [OpenLineage consumer](/en/product/connectors/consumers/openlineage/index.md) and schedule the [Send OpenLineage Events](/en/product/dpe/actions/send-openlineage-events/index.md) action to stream them to your backend. --- ### Need help? 🆘 > At any step, you can create a ticket to raise an incident or if you need support at the [OVHcloud Help Centre](https://help.ovhcloud.com/csm/fr-home?id=csm_index). Additionally, you can ask for support by reaching out to us on the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559). There is a step-by-step guide in the [support](/en/support/index.md) section.