For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt, and this page is available as Markdown at https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md.
  • 🇬🇧 English

    • Forward lineage events to an external tool with OpenLineage

    The OpenLineage consumer lets you forward data lineage events generated by your Data Platform project to an external lineage solution of your choice

    Objective

    The OpenLineage consumer lets you forward data lineage events generated by your Data Platform project to an external lineage solution of your choice. Once connected, the platform continuously streams OpenLineage-compliant events to your lineage backend so you can visualize and explore your data lineage graph there.

    Open lineage consumer
    Info

    Data Platform also includes a built-in lineage viewer: see Lineage in the Lakehouse Manager. The OpenLineage consumer is for teams that want the same events in their own lineage solution, see compatible tools below.

    Warning

    Compatibility notice: At this time, Marquez is the only lineage solution that has been validated to work reliably with Data Platform's OpenLineage events. Other OpenLineage-compatible tools may receive events but could provide a degraded experience. We recommend using Marquez for the best results.

    Compatible lineage solutions

    The following lineage solution has been validated to work with Data Platform:

    ToolDescription
    MarquezOpen-source metadata service built natively for OpenLineage

    Any tool that exposes an OpenLineage HTTP API endpoint can technically be used, but only Marquez is officially supported at this time.

    Configure the OpenLineage consumer

    If you haven't done so already, add a new OpenLineage consumer in the Connectors. Else, just edit it.

    You will need to provide the following values:

    FieldRequiredDescription
    Endpoint URLYesThe HTTP(S) URL of your lineage solution's OpenLineage API (e.g. https://marquez.example.com/api/v1/lineage)
    Auth tokenNoA bearer token used to authenticate requests to your lineage backend. Leave empty if your backend does not require authentication
    Keep EventsNoIf true, events are stored in the bucket after being sent for history. Otherwise they are deleted. (Default: false)
    Disable Trino eventsNoIf true, lineage events generated by Trino queries (Explorer, Analytics Manager) are not forwarded to your lineage backend. Only events from DPE actions (Load, Aggregate, Custom) are sent. Use this to reduce noise if you only care about batch lineage. (Default: false)
    Configure the OpenLineage consumer — Open lineage consumer configure

    Once a lineage consumer is configured, every lineage events will be stored on the Lakehouse Manager Bucket DWH under openlineage/events/queued. Events will be forwarded to your lineage solution each time the Send OpenLineage Events DPE action runs.

    Send lineage events from the DPE

    Lineage events are forwarded by the Send OpenLineage Events action in the Data Processing Engine. You can configure it to run:

    • As a cron job: on a fixed schedule (e.g. every hour, every night)
    • As an always-up action: continuously, to keep your lineage solution in sync in near real time

    See the Send OpenLineage Events documentation for setup details.

    Info

    For Custom actions, lineage tracking is opt-in. See Track data lineage in a Custom action to learn how to emit lineage events from your own Python code.

    Go further

    If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

    Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

    If you need support with your OVHcloud services, create a request in our Help Centre.

    Join our community of users.