For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt, and this page is available as Markdown at https://docs.dataplatform.ovh.net/connectors-sources-kafka.md.
  • 🇬🇧 English

    • Stream Apache Kafka events into Data Platform

    Apache Kafka is a distributed event store and stream-processing platform developed by the Apache Software Foundation

    Objective

    Apache Kafka is a distributed event store and stream-processing platform developed by the Apache Software Foundation. Using the Kafka connector, you can connect to your Kafka server and integrate in real-time the data you receive on the server to your Project.

    Kafka in store
    Info

    In this article, we assume that you know the basic concepts of Apache Kafka, such as messages and offsets. If you don't, just know that Apache Kafka servers store the data they receive in different topics and that this data comes in small chunks called messages. To keep track of the order of arrival of the messages, each one has an offset number, the lower it is, the earliest the message arrived.

    Add a Kafka source on the Platform

    Configuration screen overview

    Once you have found Kafka in the Platform store, click on Select and you will be able to see the configuration screen as shown below :

    Kafka configuration screen

    To connect to your server, you will have to give the server address and, if required by the server, username as well as password.

    Configuring your topics

    After successfully establishing the connection with your server, you can select which topics to get data from by checking the boxes to their left.

    Kafka topics screen

    If a topic is checked, you can hover over it and click on the Pen Icon to configure how the data it receives will be transferred to the Platform (the default offset policy).

    Kafka topics config

    There are two default options of offset policy:

    • Latest: reads data in real-time as it arrives. Older messages will be ignored.
    • Earliest: start reading the data from the earliest message in the topic. Once all the messages in the topic have been read, all messages will read in real-time (similar to latest).
    Info

    You also have the possibility to start reading from a custom offset, but this option isn't configured at the connector level but at the processing job level. Therefore it is explained in the Apache Kafka complete tutorial

    Whenever you load data from a Kafka source into Lakehouse Manager table using Data Processing Engine, the latest read offset is stored in the destination table's metadata. This means:

    • if you resume ingesting messages into this same table, the ingestion will start at the latest message read
    • If you start ingesting messages into a new destination table, the ingestion will start at the default parameter configured either at the connector level (latest/earliest offset) or job level (custom offset)

    Supported message formats

    The Platform supports the following types of input messages for metadata analysis and ingestion:

    Flat JSON formats

    {
'key':'value',
'anotherkey':'anothervalue'
}

    Unsupported message formats

    Currently, the Platform doesn't support the following formats:

    Simple strings

    'example message'

    Nested messages

    {
'key':'value',
'nestedkey':{
'a':'b'
}
}
    Info

    Feel free to reach out to us by sending us a request if the format you are using is not yet supported! 😊

    Using data from Apache Kafka in the Platform

    If you are interested in seeing how to use the Kafka connection in the rest of the of the platform, you may refer to our dedicated Apache Kafka tutorial.

    Go further

    If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

    Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

    If you need support with your OVHcloud services, create a request in our Help Centre.

    Join our community of users.