For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt, and this page is available as Markdown at https://docs.dataplatform.ovh.net/developers-python-sdk-connectors.md.
  • 🇬🇧 English

    • Connectors and Connection Strings

    In Data Platform's Project, all available data are defined and configured in the Lakehouse Manager

    Objective

    In Data Platform's Project, all available data are defined and configured in the Lakehouse Manager.

    In the Python SDK, all those data sources are named by a unique connection string and accessible through connectors returned by the connect() function. The returned Connector object as well as its available methods depend on the data source's type.

    A simple example to list tables from the default dataset:

    from forepaas.dwh import connect

# connect to default dataset
cn_prim = connect('dwh/default_dataset/')
# list databases tables
tables = cn_prim.list()
print(tables)
    Info

    Connection strings are used in all Project's components to reference data sources, for instance in Data Processing Engine Load and Aggregate actions or when applying segmentation.

    Buckets

    Buckets are Data Platforms' built-in object store. Below, the connection string and its respective Connector object returned by the connect function.

    connection stringConnectordescription
    connect("data_store")DataStoreConnects to the buckets
    connect("data_store/my-bucket")BucketConnects to a specific bucket

    Read more about DataStore and Bucket connectors.

    Storage Engines

    Projects are linked to Storage Engines as main databases to store and query data at the heart of your Project.

    A Project can be defined with multiple datasets, but by default you will receive 1 dataset: default_dataset. You can also create custom internal and external datasets.

    connection stringConnectordescription
    connect("dwh/default_dataset/")DatasetConnects to the default dataset - default_dataset
    connect("dwh/exampleDataset/")DatasetConnects to the custom dataset created by you - exampleDataset

    Read more about Database connector.

    Query Engine (Trino)

    connection stringConnectordescription
    connect("query_engine")TrinoConnects to the query engine (Trino)
    connect(f"query_engine/{catalog}/{schema}")TrinoConnects directly to a schema in a catalog

    Sources

    You may also connect to external data sources declared in Connectors's Sources. Each external source has an associated connection string based on the initial name given to the source. For instance, a new data source named My first data will have the connection string equal to dwh/my_first_data.
    The technical name is displayed on the main Connectors Source's page.

    External sources may be split in a few types:

    • Databases connectors have same methods as storage engines, see Database connectores
      (i.e. Snowflake, PostgreSQL, Oracle, Microsoft SQL Server, Amazon Redshift, MongoDB...)

    • Protocol connectors when accessing file storage or object store (Data Platform Buckets, Data Platform File Upload, SFTP, Amazon S31, ...)
      Those connectors have same methods as Bucket connector

    • Stream (i.e. Kafka, MQIT, Amazon Kinesis) or External APIs (i.e. Google Analytics, Facebook, Twitter, Mailchimp..)
      Those connectors are mainly designed to be used from Connectors Analyzer and Data Processing Engine Load Actions.

    Go further

    If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

    Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

    If you need support with your OVHcloud services, create a request in our Help Centre.

    Join our community of users.

    1: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.