---
url: https://docs.dataplatform.ovh.net/index.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform
## What is Data Platform?
Data Platform is a managed platform for building data products, with the infrastructure decisions already made: there is no cluster to size, no orchestrator to run and no deployment pipeline to build. Connecting sources, storing and modelling data, processing it, analyzing it and exposing the result as an API or an application are all provisioned and operated by the platform, behind four levels of automation, so effort goes into business logic rather than logistics.
A [project](https://docs.dataplatform.ovh.net/manage-projects.md) is the unit of work and the unit of isolation. It holds the sources, tables, buckets, pipelines, queries, dashboards, APIs and apps that make up one data system, together with the access rules over them. Creating a project deploys more than 60 microservices and provisions machines in the selected cloud region in under eight minutes, with no further input.
:::info
Access to a project is usually shared collaboratively along role lines: one group works on ingestion and transformation, another on analytics, another on the application layer. The [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md) decides who can read and change what, down to individual table attributes.
:::
If Data Platform is new to you, the fastest way in is the six-step guide that builds a working application on sample data. From there, the [services](https://docs.dataplatform.ovh.net/landing-page-services.md) documentation covers each component in the order data moves through it.
[### Getting started→
Build a first end-to-end application in six steps.
](/landing-page-getting-started)[### Tutorials→
Complete worked use cases, from Kafka to Superset.
](/landing-page-tutorials)[### Services→
The seven services a project is made of, in data order.
](/landing-page-services)[### Developers→
Public APIs and the SDKs behind them.
](/landing-page-developers)[### Manage your platform→
Organizations, projects, pricing and finding your way around.
](/landing-page-manage)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-getting-started.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Get started with your analytics application
## What you will build
The Getting Started guide will teach you **how to create an analytics project using the Platform**, through concrete examples and best practices. At the end of this tutorial, you will be completely self-sufficient. You’ll be able to create and deploy your own customized analytics application with sophisticated data processing.
:::info
**Before you start.** Access to Data Platform is set up once per organization: see [Sign up and sign in](https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md). Then [create a project](https://docs.dataplatform.ovh.net/manage-projects-create.md), which takes about eight minutes to provision. The tutorial below assumes both are done.
:::
The guide uses sample data, and demonstrates how to build an analytics application with predictive insights correlating subway traffic in the city of Chicago with weather variables and the time of the year.
_Photo by Skyler Gerald on Unsplash_
The guide walks through building that application step by step, from connecting the sample data to deploying the finished app.
This guide takes about **1 hour** to complete.
:::info
We recommend using **Chrome** as your default browser. If you're using a different browser and face any issues, check out the link to our support portal to get in touch.
:::
## Setting-up your environment
Before you start working with data, you will need to set-up a working project environment. Your working environment is called a **Project**. It includes all the necessary tools and dedicated compute and storage resources you’ll need to build your analytics application.
When asked, choose the template to create an **end-to-end project**. This will instantiate all the services on the Platform to be ready to use in your project.
Proceed with the creation. Once you have spinned up a Project, open it to [move on to creating your first application](https://docs.dataplatform.ovh.net/landing-page-getting-started.md)!
## Build your analytics app from scratch on the Platform
The goal of this tutorial is to get you familiar with the Platform so that you can start creating your own applications as soon and as quickly as possible.
We'll walk you through the main steps to **build a data visualization app in less than 1 hour**.
The application to be built serves to visualize the rides in Chicago's railway stations. It will look like this by the end of this tutorial:
You will use sample datasets, and all the features and pre-written code provided for you by the Platform.
## Six easy steps
After having [created a Project](https://docs.dataplatform.ovh.net/landing-page-getting-started.md#setting-up-your-environment), your data application will be created in 6 easy steps.
[
](https://docs.dataplatform.ovh.net/getting-started-visualize-data.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/getting-started-connect-sources.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Define, analyze and clean your data sources
## Objective
In this step, you will learn how to **connect your first data source** to your Project, **analyze** and **clean** it in a consistent way so that relations between your data can be understood at a glance.
:::info
This step assumes a project is open. If you do not have one yet, [create a project](https://docs.dataplatform.ovh.net/manage-projects-create.md) first.
:::
The **Connectors** is the first component you should use when starting out on your Project. It is the component that controls:
- Data sources collection
- Data analysis & cleaning
The data source consists of two CSV files containing data about Chicago weather and public transport.
## Connect your first source
Let’s go get your data!
:::info
To get your data, download the following files [here](https://eu.dataplatform.ovh.net/getting-started/chicago_files.zip). Once unzipped, you will find 2 _.csv_ files:
:::
- _chicago\_calendar\_full.csv_ - a table with weather conditions by day where each row represents a day.
- _station\_rides.csv_ - a table with railway usage information per station per day where each row represents a day on a given station.
Inside the Connectors, go to the menu **Sources** (1) and click on the **New Source** (2) button on the top right-hand corner to connect your very first data source.
There are many connectors supported by the Platform. In this tutorial, you will use a simple file upload.
Look for _File upload_ in the search bar, click on the icon and then click on the green _Select_ button or you can quickly click the _File Upload_ button on the bottom right.
Give a name to your source: for example _chicago\_files_.
Then you can **drag & drop or upload the 2 files** provided at the beginning of the tutorial, and click on **Create**.
Your files are now loaded to a [bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md)!
Now, let's move on to extracting their metadata.
## Analyze your source data
You should still be inside the Connectors component. Click on **Analyzer** from the left-hand menu. Here you will extract metadata (i.e. information about your data) from your sources and choose which columns you want to display.
The two source files you just uploaded appear on the sidebar. Click on the respective source name (i.e. _chicago\_files_) to expand it. Then, click on each of the files to automatically launch a metadata extraction.
The metadata extraction should take around 30 seconds.
:::info
The Analyzer is also where you can add blueprint rules (by clicking on the blue button _New Rule_). The Platform automatically applies a formatting transformation on dates to set them in a standard _yyyy-mm-dd_ format.
:::
[↪ Learn more about the Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md)
It's time to head to the Lakehouse Manager to see what's next.
[Create and Build your data models](https://docs.dataplatform.ovh.net/getting-started-organize-data.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/getting-started-organize-data.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create and Build your data models
## Objective
The **Lakehouse Manager** is the next component you will be using. It is the component that controls:
- Data warehousing
- Policy Tag Management
Using the Lakehouse Manager let us plan out your data structure at the **conceptual level**. This will be done with the help of the _Tables_ functionality.
:::info
This tutorial is meant for **Standard Datasets** using the _Lakehouse Manager Engine_. To know how to follow this same step with **External Datasets** click [here](https://docs.dataplatform.ovh.net/tutorials-external-datasets.md).
:::
## Organize your data in tables
### Create your primary schema
#### Add tables to the data model
Once that your metadata has been extracted, it's time to head to the Tables dashboard. This is where you will **build a unified and queryable view of all your data**.
The empty Tables page should look like this.
The _All tables_ tab is where you have access to the entirety of your data. The _New View_ tab allows you to create views of only a part of your data in order to collaborate better in large teams. Since this is a simple tutorial, you should work in the _All tables_ tab.
Now let’s concentrate on creating your primary tables and their attributes.
First, hover your cursor over the blue ➕ **NEW TABLE** button on the right-hand side of the screen. This will reveal the create options :
- Upload a file
- Create from Connectors source
- Create an empty tables
:::info
For the purpose of this tutorial, we shall proceed with **Create from a Connectors source.** To create a table using upload a file refer to the [create tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#create-a-table) documentation.
:::
Once you click on _create from a connectors source_, you will be shown a list of sources from the previous step. Click on source you want to add and proceed further by clicking on _Next_. No need to change the default settings here.
:::info
The options _Build the table, load the table once, and generate Load action for later_ can be disabled and each step can be done individually as well. This can be explored in detail [here](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#create-a-table)
:::
Press **Create** and proceed to repeat the same with the second table as well.
:::info
Note that each time you make a change on the Tables page, your visual configuration is **automatically saved**.
:::
At this stage, your Tables page should look like this.
:::info
Note that the Platform uses the metadata information captured in the Analyzer step to automatically create tables and assign attribute names & types. The source files provided are good to go, however in a real Project you should use the Analyzer to check the data sources, before dragging and dropping them in the Tables page.
:::
### Create your aggregate table
Now, you will aggregate all important data from the sources (namely rides, dates and temperatures) into one table that will be used in the final application.To create your first _Aggregate_ table, click on the blue ➕ icon. Select _Create an empty table_, and you will be shown a new table configuration. Set a name (_dataset\_history_ for example) and save it.
Now, before you start adding attributes to the new empty table you have created you will need to change the view from list to canvas on the bottom right side of the screen to make the next steps easier.
Move the attributes below by drag and drop to _dataset\_history_:
| Original table | Attributes to drag-and-drop |
| :-------------------------: | ------------------------------------------------------------------------ |
| **stations\_rides** | _date_ / _lat_ / _lng_ / _rides_ / _station\_id_ / _station\_name_ |
| **chicago\_calendar\_full** | _month_ / _temperature_ / _week\_day_ / _week\_day\_label_ / _weekend_ / |
Finally, you will need to create one new attribute to help translate numerical temperature data into understandable categories (cold, hot, ..).
Start by clicking on the ➕ icon that appears at the top of the table _dataset\_history_ when you click on it. You can then create or edit an attribute inside a table.
Define the attribute as such:
| Attribute name | Type | Nature |
| :------------------: | ------ | --------- |
| **cat\_temperature** | String | Dimension |
:::info
For the moment, this attribute is not physically specified. This will be done later in another component: the Data Processing Engine.
:::
[↪ Learn more about Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md)
### Finalize the build
:::warning
One last quick check! Make sure to **double-check that your data model looks exactly like the one in the screenshots** before moving to the next step. If some attributes are missing you will get stuck in later steps of the tutorial.
:::
Now, click on the _Build_ icon (under the blue ➕ icon) to effectively create/update the tables and attributes in your dataset. This still doesn't load the data into the tables (which will be done in the next article), it simply applies the logical schema to tables and attributes in your underlying dataset.
:::info
While the visual logical schema of the data is automatically saved, changes to your tables won't be visible in the rest of the Platform as long as they are not built.
:::
The build task for this tutorial shouldn't take more than a few minutes to run. Once it is done, you can move on.
### Add relevant metrics with Virtual Attributes
Before moving on to the physical processing (ETL/ELT) of the data into this model, let's prepare additional metrics for analytics later on. The final application that you are building following this tutorial includes a chart with the **number of rides per day on a given station**:
However, you do not have the necessary data to build that chart directly on the primary sources. You will need a metric that gives you the average number of rides per day for a given station and that can be used in queries and dashboards.
But how do you compute it using the Platform? One way to do this is to create a **virtual attribute**. Virtual attributes allow you to calculate SQL formulas that will be **computed on the fly** and won't be stored in the database. They can be used in a query or a chart in your final dashboard.
:::info
Adding or editing virtual attributes does not require rebuilding the schema.
:::
Switch to the **Attributes** page. This page lists all physical and virtual attributes in your data model, and the lineage in your Project.
Click on the _New Attribute_ button to create a virtual attribute.
In the creation window, make sure to select _Virtual_ as the realm.
Now add these two attributes and their respective SQL code:
| Attribute name | SQL |
| :------------------------------------: | -------------------------------------------------------------------------------------- |
| **avg\_rides\_per\_day\_per\_station** | SUM(rides)/COUNT(DISTINCT CONCAT(CAST(date AS VARCHAR), CAST(station\_id AS VARCHAR))) |
| **yearmonth** | SUBSTR(CAST(date as VARCHAR),1,7) |
The **yearmonth** attribute gives you the year and month in the format _yyyy-mm_. You will use it later.
:::info
Notice how you just used two different methods to generate new attributes/metrics from imported data: **adding an attribute to a Aggregate table** and **virtual attributes**.
:::
- Adding a new physical attribute takes up storage and requires you to physically define them in the Data Processing Engine, but this makes them more rigorous as their specifications can then be edited without changing the whole data model.
- Virtual attributes are a quick win but can become hard to manage if you need to modify them when scaling.
It's time to head to the Data Processing Engine to see what's next.
[Prepare your data pipelines with the Data Processing Engine](https://docs.dataplatform.ovh.net/getting-started-prepare-data.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/getting-started-prepare-data.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure data imports and transformations
## Objective
The Data Processing Engine (DPE) is the component that lets you **create and execute data pipelines outside and within the Platform**. Everything is automated, from the management and deployment to the scalability, to help you focus on the business logic of your data Project rather than its logistics.
While the Lakehouse Manager is where you plan out your data structure at the **conceptual level**, the Data Processing Engine is where you execute actions to **physically impact your data**.
:::info
This tutorial is meant for **Standard Datasets** using the _Lakehouse Manager Engine_. To know how to follow this same step with **External Datasets** click [here](https://docs.dataplatform.ovh.net/tutorials-external-datasets.md).
:::
## Actions
An [action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) consists of a unitary physical operation on the data. Actions can be organized in stages in order to produce automated data processing pipelines called [workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md).
Click on the **Actions** menu of your Data Processing Engine. You should see the two _Load_ actions that were[ automatically generated in the previous step](https://docs.dataplatform.ovh.net/getting-started-organize-data.md#organize-your-data-in-tables):
These Load actions will physically extract the data from your sources and load it into your data warehouse, following the schema made in the Lakehouse Manager.
### Create more actions
:::info
Our Marketplace gives you access to a dozen of curated actions to kick-start your data processing Projects: _load_ actions, _aggregate_ actions, _delete_ actions, etc. If you cannot find what you need in the catalog, you can always resort to a _custom_ action that allows you to **run any piece of Python 3+ code** as part of your data pipelines.
:::
[↪ Learn more about custom actions](https://docs.dataplatform.ovh.net/dpe-actions-custom.md).
For this tutorial, you will create an action used to aggregate your data into the _dataset\_history_ table you created in the previous part.
Click on **New action** and select the _Aggregate action_ template from the Store.
There will be 3 simple steps to configure the aggregate action:
- **(1)** Select the source table : _stations\_rides_
- **(2)** Select the destination table: _dataset\_history_
- **(3)** Select the source table : _chicago\_calendar\_full_
Change the join condition to an _INNER join_ using the dropdown menu. This will ensure that you do not have any null fields in the records of your dataset\_history table. Also make sure to copy and paste the following sql code to let the aggregate action know how the tables are going to be linked:
```
stations_rides.date = chicago_calendar_full.date
```
:::warning
Note this is required because the concept of _joining tables using primary keys_ does not exist for **standard datasets**. If you are using an **external dataset**, then it will be automatically generated.
:::
The attributes need to be manually mapped out. Refer the screenshot below and make sure all the attributes are mapped correctly.
Make sure that the all the attribute are in the **SELECT** function.
:::warning
If you are using an **external dataset**, then the attributes will be automatically mapped out. Only the attribute _rides_ needs to be switched to **SUM**.
:::
Finally, let's define the categorical attribute _cat\_temperature_ that you created earlier. Click on the **\< map >** (short for "mapping") option in blue option dropdown as shown below and switch it to **\< sql >**.
Simply copy-paste the SQL command below:
```
CASE
WHEN chicago_calendar_full.temperature < 40 THEN 'very cold'
WHEN chicago_calendar_full.temperature < 48 THEN 'cold'
WHEN chicago_calendar_full.temperature < 55 THEN 'medium'
WHEN chicago_calendar_full.temperature < 62 THEN 'hot'
ELSE 'very hot'
END
```
:::warning
**Leaving a destination attribute unmapped in the Aggregate action configuration will trigger an error when the action is launched**. If you'd rather leave the destination field empty, make sure to simply remove it from the list of mapped attributes.
:::
[↪ Learn more about Aggregate actions](https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md)
Click on **Create** at the top-right.
**You have now generated all the actions** required for this tutorial.
:::info
Of course, your real-life Project will probably have more than 3 actions. You can organize your actions in folders and rename them if needed. You can also use more than one repository, especially if you are working in collaboration with teammates. Repositories of actions can be versioned and also synced with external Git repositories. Check-out the [dedicated product documentation](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#manage-actions) page to learn more about how to do that!
:::
[↪ Learn more about Actions in the Product Documentation](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md)
Before you proceed to the next step in creating workflows, we will create one final action - **Delete**.
Its use is very straightforward, to delete the contents of the selected table(s).
Create a new delete action and add all the tables we have created so far - the idea is simple, if you are loading a table with new data or loading the table again with old data then you will need to clean the table before doing so because its best practice.
## Workflows
A workflow defines the order in which your actions run.
Within a workflow, actions are organized in sequential stages. Within a stage, all actions will be run in parallel while stages will always run one after the other. The same action can be used multiple times in the same workflow. A workflow, just like an action, can either be launched manually, set up to run on a schedule or triggered through an API call.
:::info
Note that it is important to remember that **stages are run one after the other** in the order you planned them while **actions contained in a stage are all run all at the same time** regardless of the order. In short, the order of the actions inside a stage does not matter while the stages’ order inside a workflow does.
:::
To create your first workflow, you'll need to head to the _Workflow_ tab and click on **New Workflow**. Head to preferences or double-click on the header name to set a new name _Import Chicago Data_.
Let's start by defining three different stages by clicking on **Add a stage**. Then, add actions in each stage using the dropdown search selector following the screenshot provided as a guide for each stage.
After creating the workflow (**create**), press **Play**.
:::info
Please note that workflows might take a few minutes to run when you launch them for the first time. The total time should not exceed 10 minutes - if it does, please reach out to our support team.
:::
While the workflow is running, something you can do is **schedule it to run daily** using a trigger.
Head to the _Preferences_ tab of your workflows and scroll to the Triggers widget on the bottom left side. Click **+Add**.
Select the trigger type as _CRON_ and mode as _Simple_. Navigate to the **Daily** tab and in the list of options select _Every 1 day(s)_ as shown in the picture below:
Hit the **Confirm** button to create the new trigger event with the name of your choice and it will append it below the _Launch Endpoint_ present by default in the Trigger event table.
:::info
There is a lot more you can configure in a workflow's preferences. Namely, you can [scale horizontally and vertically](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) any processing job, use [workload segmentation](https://docs.dataplatform.ovh.net/tutorials-segmentation.md) to accelerate the data processing and even save all these configurations for repeated use thanks to [environments](https://docs.dataplatform.ovh.net/dpe-environments.md).
:::
[↪ Learn more about configuring the execution preferences.](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md)
:::warning
Make sure to click on the **Save** button on the top right of the screen whenever you make a modification to your workflows. Actions are stored in repositories which can be versioned which is not the case for workflows or environments. _Autosave_ is therefore disabled for both workflows & environments.
:::
## Jobs
To wrap up this section, here are a few words about the last tab of the Data Processing Engine component: jobs.
The Jobs tab summarizes **all executions triggered in the Data Processing Engine** and includes advanced metrics reports. Jobs are listed under three main categories: running, queued and past executions. Having a look at the last jobs executed, you can verify the status of the workflow you just launched.
:::info
Resource allocation on the Platform is done through **proprietary units called [DPU](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units)**, Data Platform Units. You can change the amount of DPU dedicated to an action or workflow in their [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
:::
You now have completed the Data Engineering section of the _Getting Started_ tutorial.
A good way to make sure your data properly loaded is to go back to the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) and look at the number of rows loaded into the table. Just open the **list view mode** and check the _rows_ column, if the field has a number (indicating how many rows have been loaded) then everything worked well.
Remember you are setting up a **production-grade system**, hence why you need both a Lakehouse Manager and a Data Processing Engine. Don't hesitate to reach out at the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559) if you're running into any issues or you'd like help to speed up your data pipelines, optimize their performance, adding custom exotic data sources or creating custom scripts!
Now let's move on to the next component in your data system: the Analytics Manager.
[Create insightful queries with the Analytics Manager](https://docs.dataplatform.ovh.net/getting-started-create-queries.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/getting-started-create-queries.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create queries on your data
## Objective
The [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) allows you, thanks to a **low-code approach**, to prepare and centralize the queries matching your business KPIs. This component constitutes the interface between your data scientist team and your application developer team.
## Pre-requisites
Beforehand, you need to make sure all the previous steps of the Project are up and running. Your Lakehouse Manager schema was built properly and your workflow ran smoothly.
In this tutorial, you will build your first queries.
## Build your first queries
Now, you will create your first queries in order to make data ready for visualization:
- _rides\_per\_month_ to understand the average number of rides each month. You'll use it to build your first graph.
- _rides\_per\_cat\_temperature_ to understand the average number of rides depending on temperature.
### Create the first query
Open **Queries** in the sidebar and then click on **New Query**. This will open the Query window where you can give your query a name, parametrize the data as well the scale, order the results and add filters. In this screen, you can also view the results of your queries with tables or charts.
To begin with, name your query `rides_per_month`.
Then start typing 'avg\_' in the **Data search bar** and click on the _avg\_rides\_per\_day\_per\_station_ attribute.
The compute mode is **select** by default, you can change it by clicking on the arrow and changing the selected options. For the purpose of this tutorial, leave it as it is by default here.
Then, click on the **Scale search bar**. Here, you want to display the average number of rides **each month** per date. Select _yearmonth_ as a scale.
Then, click on the green button **Run** to run the query. You'll see the result of the query at the bottom panel, by default it is on table view.
You can change the display format: table, line chart, bar chart, area chart or pie chart. Choose by clicking on the buttons of the bottom panel (as indicated in the image below).
:::info
When you build a query in the Analytics Manager, you don't have to specify the table names. The Platform will **automatically choose the most suitable table**. But you can still force a specific table by clicking the table icon next to _Data_.
:::
[↪ Learn more about how the Analytics Manager works.](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#choice-of-table)
### Create a second query
Queries aren't meant to be used in a dashboard only, they also provide an easy environment to explore your data using an array of data visualization options.
Create a new query:
- Name it `rides_per_cat_temperature`.
- Select _avg\_rides\_per\_day\_per\_station_ with compute mode **select**.
- Select _cat\_temperature_ as your scale.
- Select _avg\_rides\_per\_day\_per\_station_ as the **Order** (ascending order by default).
:::warning
If you remember, _avg\_rides\_per\_day\_per\_station_ is a virtual attribute you have set in a previous step of the tutorial (Collect data). You cannot use any other compute mode than **Select** with this virtual attribute as it contains an SQL aggregate function. Adding **Sum** as compute mode would end up imbricating two SQL aggregate functions, resulting in an error.
:::
Just like the screenshot below, click on **bar chart icon** to automatically display the data as a bar chart.
You have created two queries in the Analytics Manager. The next step is to expose that data through an API, which you will create and deploy for use in your application.
[Expose your data by deploying an API](https://docs.dataplatform.ovh.net/getting-started-expose-api.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/getting-started-expose-api.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Deploy your first API
## Objective
An API is crucial as it is the interface between the external world (namely your applications) and the rest of your Project (i.e. your Lakehouse Manager and queries). You will have to call this API when building your application, in the next and final tutorial.
## Create a new API
From your Project's home page, click on **+** inside the API module.
You will be able to choose between **an existing API template** from the internal _Store_, or to **import your own code** by linking a Git repository.
Since you're just starting, we've made a simple NodeJS template just for you. Proceed by selecting the only template available: _API NodeJS_.
Choose a name for your API, and enable the option **auto build** & **auto deploy**. This will download the source code of your API, automatically build it and deploy it for you.
From there, give it a couple of seconds as the creation of the API may take a moment. The build takes a few minutes.
:::info
The Platform uses a **[blue-green](https://en.wikipedia.org/wiki/Blue-green_deployment) deployment methodology** ensuring 100% availability of your data. You can therefore seamlessly create & deploy new versions of your API without risking any downtime!
:::
When the deployment is complete, the **Open** button appears.
When you open your API, you will land on the screen below confirming that your API is alive & healthy. It's that easy!
[↪ Learn more about your Project APIs](https://docs.dataplatform.ovh.net/sdk/api.md)
The next and final step will consist of creating and deploying your first application on the Platform in under 5 minutes.
[Create your first app](https://docs.dataplatform.ovh.net/getting-started-visualize-data.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/getting-started-visualize-data.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build and deploy a front-end application
## Objective
In this tutorial you'll **learn how to build and deploy your first application**. To make it easier for you, you won't start from scratch. We've prepared a template to accelerate the process, thanks to pre-built graphs.
Initiate your application from **Application Services** by clicking **Create an application**.
Select the template _ReactJS - Getting Started_ from the store. This template produces a complete BI dashboard.
Fill in the different fields as shown in the screenshot below and press **Confirm**.
Click on **Confirm**, your app will automatically build and deploy. The process should take a couple of minutes, in the meantime feel free to read through the pre-requisites below.
## Requirements
Every previous step of the Getting Started guide must have been completed:
- The API is up and running.
- Queries in the Analytics Manager are properly requesting your data warehouse.
- The Lakehouse Manager shows a properly built logical schema. Virtual attributes are created and tables linked.
- Your Data Processing Engine workflow has been correctly run.
Let's create your first dashboard.
## Set up your canvas
Click on the **Dashboard** menu. In an app, dashboards are its pages.
A first dashboard, _Rides Analytics_, has been added by default when you selected the template at the beginning. It comes with styles and logos!
Click on the **house** icon to set it as the default landing page of your app.
### Set up a header
Head to the **Menus** menu. It is the page where you can define items such as header menus and filters for your whole app. It has already been pre-populated with several pages and items due to the template you used to initiate the app.
You should land on the header page. This means every item you add here will be embedded in the header of your app (i.e the upper part). You can switch to the sidebar page (to embed items in the left sidebar of your app) at the top right-hand corner.
Switch back to the header page. Expand **tabs** to see the container's items. Click on _Add an item_ to add a tab in your header. Select **link**.
Fill in the fields as shown below. You can also define a logo icon for your app header!
### Set up a dynamic parameter
Dynamic parameters are widgets to **filter your data on your pages**. Here, you will add one dynamic parameter to filter on the date.
Click on **Sidebar** on the top right-hand corner and create a container.
:::info
If you are unfamiliar with [containers](https://docs.dataplatform.ovh.net/app-services-apps-menus.md), think of them as folders for items.
:::
Drag the new container on top of the footer (which has to be at the bottom).
Open the container then add an item: select **Parameters**.
Pick the _Select box_ one, then head to the **options** of the new parameter.
Fill in the options as shown in the screenshot below.
In the **Infos** tab, give it the following name: _select\_stations_, and press Confirm.
:::info
Note that every dynamic parameter like the date picker and select boxes can be customized. To learn how to do that, head to the technical documentation for application components!
:::
[↪ Learn more about dynamic parameters](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters.md).
Your canvas configuration is now done. With a menu and a filter, you can now create your first graph.
## Set up a chart
Go to the **Dashboard** tab and click on the **Edit** icon on your _Ride Analytics_ dashboard.
You can see that the menu and the filter we've just created are well set up here!
#### Step 1. Create a panel
Click on the ➕ icon to create a panel (**Containers > Panel**).
Once it's created, you can resize it so it spans across the screen.
#### Step 2. Give a title to your chart
**Inside your new panel**, click on the ➕icon and select **Basic UI > HTML**.
Name it _Rides history_ and apply the H1 style by clicking on the H button.
#### Step 3. Generate your chart
Inside your panel, click on the ➕icon and select **Chart**. Pick the _Recharts - Bar_ type.
Click on **Request**, and select **Query selector** to pick the _rides\_per\_month_ query you've created in the Analytics Manager.
To complete the setup, you have to connect this chart with the filter we've already created. Go to the **Infos** tab, in the **Linked dynamic parameters** section, and add _select\_stations_.
You can now test if your filter is properly working. Enter the read-only mode by clicking on the **Play** icon, and modify dates or holiday type to see how your data is changing. Your first chart is done!
:::info
Your first dashboard is now complete. You can also improve its appearance. In our _Getting Further guide series_, we have released more tutorials for you to check out: [click here to learn more about the app builder's capabilities](https://docs.dataplatform.ovh.net/tutorials-customize-first-app.md)!
:::
## Build and deploy the new version of your app
One last step: put your dashboard into production.
First, don't forget to **Save** your dashboard to make sure all your edits are consistently saved ( icon in the top middle bar).
Then go back to the **Overview** tab of the Application Manager.
Changes brought to your application since you first created it **must be builded** in order to be visible by your end-users. You could stop the currently deployed version to build those changes, but the Platform also allows you to do it [without interruption of service](https://en.wikipedia.org/wiki/Blue-green_deployment).\
First, **Copy** the currently deployed
version of the application.
Then you need to name the new version (_v2_ in the screenshot below).
Now, click on **Build**, and then **Deploy**.
:::info
The **build** process usually takes a couple of minutes. If you're curious about what is going on or if the build fails, open the log console by expanding the version panel and inspect the logs. Feel free to send logs to our support team if you're not sure how to troubleshoot it yourself!
:::
Your app is **live** and can now be accessed online! The link can be obtained in the Overview tab by clicking on **Open**.
:::info
The [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md) lets you configure access rights to your app and your Project administration.
:::
That's it! Your dashboard is live and accessible by your users. Its data is refreshed automatically every day, stored in the cloud and will always be reliable.
**You've completed the Platforms' Getting Started tutorial!** You're now ready to go live on the platform, explore by yourself and take on your own Projects by yourself.
Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed.
The next step will consist of discover advanced features of the app builder.
[Customize your first app further](https://docs.dataplatform.ovh.net/tutorials-customize-first-app.md)
Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-tutorials.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform tutorials
## How the tutorials are organized
From streaming a Kafka topic to publishing a Superset dashboard, each tutorial here is self-contained, and most ship with the code and sample data they need. They are grouped below by where they sit in the data journey.
:::info
The tutorials assume a working project. If you do not have one yet, the [Getting started](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) guide builds a first end-to-end application, and these tutorials extend from there.
:::
They fall into four groups.
**Project setup and configuration** covers the work around a project: building your own app, exporting and importing configuration, generating API keys, and versioning.
**Data ingestion and transformation** covers getting data in and reshaping it: Kafka, IoT fleets, the Python SDK, SQL, external datasets, and segmentation.
**Analytics and BI** covers reading data back out: Trino, PySpark, AI Endpoints, billing analysis, and Apache Superset.
**Data export and management** covers the edges: dated export folders, bucket role conditions, and custom event handling.
Start with the [project checklist](https://docs.dataplatform.ovh.net/tutorials-project-checklist.md) if you are setting up a new project and want the shape of the work before the detail.
[### Your project checklist→
The shape of the work before the detail.
](/tutorials-project-checklist)[### Project setup & configuration→
Apps, configuration, API keys and versioning.
](/landing-page-tutorials-project-setup)[### Data ingestion & transformation→
Get data in, then reshape it.
](/landing-page-tutorials-data-engineering)[### Analytics & BI→
Trino, PySpark, AI Endpoints and Superset.
](/landing-page-tutorials-analytics-bi)[### Data export & management→
Exports, permissions and inbound events.
](/landing-page-tutorials-data-export)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-project-checklist.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Project checklist and best practices
## Objective
You have decided to go along and you have just spun up your Project! Great!
What's next? Where to start?
As the first users of our platform we, have accumulated several years of experience in carrying out data Projects. We have curated a list of best practices, learned through our services practice over the years, which we are excited to share with you.
Projects on **Data Platform** allow you to gather a team and work together on the same data Project.
You can start with some basic file uploading and seamlessly switch to near real-time data processing, keeping your methodology and easily scaling as your Project grows.\
So start, iterate, and grow your Project step-by-step: your Project will meet your growing needs.
A typical Project often goes through the following **main milestones**:
- A quick [checklist](#checklist) on the use case and Project scope
- Give your [teammates access](#user-management)
- Do iteratively, and as needed:
- Adapt the [Project resources](#manage-resources) to your needs
- Define [Data Sources](#collect-data)
- Design [Tables](#organize-data)
- Build [Actions and Workflows to process data](#process-data)
- Build [Queries and Dashboards](#write-queries) to visualize your data
- Build [Apps & API](#build-applications)
- Give [access to end-users](#give-access-to-your-application)
- [Monitor Jobs & Deployments](#monitor)
- Get feedback
## Checklist
:::warning
We recommend you go through our [Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) so you get a good understanding of the key Project components and Data Platform concepts.
:::
Before take-off, here are some basic checks on your use case scope and agenda that we share with you:
- Data sources
- Which data do you need?
- Is data available? How? Is data history available?
- Storage sizing: how much data volume and compute will this Project require?
- How often should it be refreshed?
- Describe your own personal data book:
- The "data story": where does it come from? How has it been produced? Your data flow history?
- Data fields: what does it mean? what is it for?
- Detailed Business Rules (about data)
- KPI: how to compute them? what do they mean?
- What is to be produced?
- UX Storyboard
- API for external use
- Exporting files
- Risk detection and mitigation
- What could go wrong? What is intricate in your Project?
- Identify/anticipate possible bottlenecks or hurdles
- Identify & Leverage short iterations
- View and check data as early as possible
- Release early, release often!
## User Management
Projects make it easy to work as a team on a Project. There are two levels of team management:
- [Members](https://docs.dataplatform.ovh.net/manage-organizations.md) of your Data Platform organization are your teammates. They have created a Data Platform account
- [Users](https://docs.dataplatform.ovh.net/iam-users.md) in your Project are either teammates working on this Project, or end-users of your API and apps
- Members of an organization are automatically added as users in all Projects in this organization. Permissions must be granted manually to non-admins.
After [adding your team to your organization](https://docs.dataplatform.ovh.net/manage-organizations.md), check [how to give them access rights in each Project's **Identity Access Manager**](https://docs.dataplatform.ovh.net/iam-users.md#manage-project-iam-users) according to their skills and permissions.
Your end-users access rights are also managed in the Identity Access Manager.
## Manage resources
Projects were designed from scratch to scale easily as the requirements of your data Project evolve. You can independently **scale horizontally and vertically** any component of your choice in just a few clicks, for example your [apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md).
Each component in your Project will consume [resources](https://docs.dataplatform.ovh.net/manage-pricing.md) when it is actively running (either through a job or a deployment).
:::info
Even when it is idle, your Project uses a handful of DPUs by default. You can manage this amount in the [Control Center](https://docs.dataplatform.ovh.net/control-center-monitoring.md#infrastructure-monitoring-panels).
:::
[↪ Learn more about resource consumption](https://docs.dataplatform.ovh.net/manage-pricing.md)
## Collect data
Take a look at [available connectors](https://docs.dataplatform.ovh.net/connectors-sources-configure.md) and remember that it is always possible to write a [custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) to do whatever is not already possible through the interface.
If you need to manage files in your Project, we recommend using the [Lakehouse Manager buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md):
- organize buckets either **by topic** (referentials for instance), **by type of processing** or **by providers** depending on your needs
- within a bucket, you can use **folders** to better organize your files
- when defining a [source](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md), the _Data Platform Buckets_ connector allows you to select a bucket then select files as sources
- if you have multiple files in the same bucket/folder:
- if each file is different, you can [analyze](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) them separately and use them as a source for [Data Processing Engine actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md)
- if all files need to be processed the same way: you just need to select one to **analyze**, you will use it as a source and [use segmentation over a set of files](https://docs.dataplatform.ovh.net/tutorials-segmentation-files.md) to process them
- check out [how to send files to the buckets with S31-compatible tools](https://docs.dataplatform.ovh.net/tutorials-bucket-s3-access.md) to integrate in your in-house process
## Analyze data and design blueprint rules
Once you have set up your sources, take the time to [analyze](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) them and configure basic source cleaning rules called [blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md). This can help fix some common issues such as:
- date formatting
- number formatting
- clean up strings
- skip incomplete lines
- ... and much more
In the long run, this step will save you a lot of time!
## Organize data
The [Tables page](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) is where you [prepare your database schema](https://docs.dataplatform.ovh.net/getting-started-organize-data.md#organize-your-data-in-tables).
### Tables
- We recommend using a prefix depending on the type of data:
- `ref_`: for referentials and stable data, without any temporal information
- `raw_`: data imported _as is_ with minimal changes. This can be used to trace back data issues
- `prm_`: primary tables, i.e. improved quality data from `raw_`
- `fct_`: consolidating `facts` from several `prim_` and `ref_` in a common `scale`
- `agr_`: aggregation of data on a different `scale` (for instance `datetime` to `date` or `date` to `year`)
- they are usually designed to optimize queries
- build one `agr_` table per **scale granularity** adding all needed attributes
- then add attributes names that indicate granularity (`agr_station_yearmonth`, `agr_movie_date`)
- `tmp_`: temporary tables used for processing (mark them as **not queryable**)
- `mlm_`: for tables used in a machine learning model
- Give clear names that describe what one row of data represents
- Define [indexes](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md) to speed up data operations
### Attributes
Use [Attributes tab](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) to review all attributes, in which tables they are used, which DPE Actions uses it, etc.
Here are some best-practices about attributes (i.e. field names in tables):
- attribute type: choose between _dimension_ or _measure_
- _dimension_: qualitative information that can be scaled by, filtered by, etc but not summed.
- _measure_: numerical information that can, most of the time, be summed, counted, averaged, etc.
- attribute naming:
- use a _prefix_ to distinguish them (e.g. `client_`, `fb_`, `site_`)
- name `measure` attributes so they give a hint on a temporal window:
- `_nb_` : number for that day
- `_total_` : value for this day
- `_diff_` : difference from previous day
- ...
- clearly indicate what it is (i.e. `_amount_`, `_ticket_`, `_fans_`)
- give unit of measure if not obvious (i.e. `_g`, `_kg`, `_wh`, `_kwh`, `_eur`, `_usd`)
:::warning
It is very important to **use the same attribute name and type** across all tables to **name the same piece of information**. Thus when playing a query in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md), it dynamically checks the most suitable object for required fields.
:::
### Explorer
Use [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) to quicly check table contents.
## Process data
The [Data Processing Engine (DPE)](https://docs.dataplatform.ovh.net/landing-page-dpe.md) is used to build [actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) which are the heart of your physical data flow. They can run alone or organized into groups named [workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md).
### Actions
Make sure you have a deep look at provided [actions types](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md).
If you need to go further, start writing a [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) using our [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) to interact with the rest of your Project leveraging **Python** or **PySpark**.
:::info
Use the [notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md) to quickly prototype a Custom Action, then export it to the Data Processing Engine
:::
- For [Aggregate Actions](https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md):
- `scale`: must be the same granularity as the primary key of the destination table
- **double check your rules for joining tables**: this is the main source of errors
- Set **log level** appropriately during lifecycle (i.e. `debug` if needed, `info` otherwise)
- Use **Auto flush all** to flush all caches and visualize the updates brought by the action on the datasets
### Use Repositories and Versions
As your Project grows, don't forget you can organize actions in [repositories using versions](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#manage-actions) according to your needs.
:::info
Actions are listed in alphabetical order: you can add prefixes such as numbers to better organize them.
:::
### Use segmentation
To leverage the full power of the Data Platform's DPE, use [Segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md) to parallelize computing tasks at an action's level.
### Use a perimeter
To limit an action or a workflow on a subset of your data, e.g. for last 30 days, use the [Perimeter](https://docs.dataplatform.ovh.net/dpe-jobs-perimeter.md) option.
## Write queries
Use folders and repositories to organize saved [queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
For instance:
- _Data Quality_: queries to check Data Quality
- _Data Analysis_: queries to view basic features
- _App-xxx_: prepare queries to use inside your Application
- This allows you to update the database schema and the queries without having to deploy a new app version
- Give clear, descriptive names to your queries
### Query Engine
- unified SQL Engine over multiple databases
- [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) in Lakehouse Manager
- [Build and Publish Dashboards](https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md)
- [Access from tools like PowerBI](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md)
## Build applications
Take the time to understand [Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services.md) and the possible [settings](https://docs.dataplatform.ovh.net/app-services-apps-deploy.md#application-settings).
### Helpful tips
- Design a Storyboard before starting your Project
- Make sure your `dynamic parameters` are present in all tables you need to request
- Use **object's label** in **Tables** for automatic translation
- Leverage [Formatters](https://docs.dataplatform.ovh.net/app-services-apps-formatter.md) and [Translations](https://docs.dataplatform.ovh.net/app-services-apps-translations.md)
- Leverage [Style templates](https://docs.dataplatform.ovh.net/sdk/app-charts-template.md) to streamline your Apps development
A typical application development lifecycle is:
1. Design a storyboard
- check data is available
- identify `dynamic parameters`
- identify needed _charts_ and _custom components_
2. Analytics Manager: prepare and check needed queries
3. Implement a first version in the UI
- set up `dynamic parameters`
- set up `dashboards` and `menus`
- set up available `charts` as needed
- set up `dictionaries`, `formatters` and `translations`
4. Improve that version
- Adapt CSS as needed
- Leverage [Style templates](https://docs.dataplatform.ovh.net/sdk/app-charts-template.md)
- Implement your own [custom chart](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md) or [custom component](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md) as needed
5. Test and iterate again
6. Give access to more users
### Customize your application further
Application is based upon the ReactJS framework. We strongly recommend you go through our [guide on advanced app customization](https://docs.dataplatform.ovh.net/tutorials-app-development.md).
### Git repositories
It is usually recommended to [synchronize your application with a Git repository](https://docs.dataplatform.ovh.net/app-services-apps-git-integration.md) and deploy two applications:
- `my-develop`: synchronized with your **develop** branch
- `my-master`: synchronized with your **master** branch
### Custom Domain Names
By default, your application is available at :
```yaml
https://my-project.eu.dataplatform.ovh.net/my-develop-app/
```
You may change it easily in the [App's settings](https://docs.dataplatform.ovh.net/app-services-apps-deploy.md#application-settings) to be :
```yaml
https://my-beautiful-app.eu.dataplatform.ovh.net/
```
If you need further customization, you may register [your own domain and provide your own SSL certificates](https://docs.dataplatform.ovh.net/app-services-apps-custom-domain.md) :
```yaml
https://my-beautiful-app.example.com/
```
## Give access to your application
By default, every `Application` created in **Application Services** has its counterpart in the **Identity Access Manager**.
- [Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services.md)
- it manages the instance that will be deployed and run
- it is linked to an **Identity Application Manager's Application**
- it will use that **IAM App** configuration about **Auth Providers**, **Users**, etc
- [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-application-authentication.md), you can define:
- Logo
- Application Title
- Allowed Authentication Provider
Add all the [end-users](https://docs.dataplatform.ovh.net/iam-users.md) that will have access to your application (using [authentication providers](https://docs.dataplatform.ovh.net/iam-auth-providers.md) if your company has a SSO provider), and use [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) and [groups](https://docs.dataplatform.ovh.net/iam-groups.md) to manage data access control.
## Front API
Default Front API checks authentication, process App's requests adding **dynamic parameters** and **implementing ACLs**.
Here are some scenarios where custom API is needed.
- Implement transformers
- Implement custom "evolution"
- Implement complex ACL
## Monitor
[Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md) gives access to:
- [monitoring](https://docs.dataplatform.ovh.net/control-center-monitoring.md) resource usages and health
- [logs of all deployments and jobs](https://docs.dataplatform.ovh.net/control-center-logs-explorer.md)
- [creating alerts on deployments and jobs](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md)
- a complete view of all [jobs executions and scheduling](https://docs.dataplatform.ovh.net/control-center-jobs-central.md) at Project and Organization's level
## Project Backup
### Project Configuration
Learn more about how to [backup and restore your Project configuration](https://docs.dataplatform.ovh.net/tutorials-export-import-config.md).
### Exporting data
Using a [Data Processing Engine custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), you can export all your data to a [bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) in the Lakehouse Manager.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/landing-page-tutorials-project-setup.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Project setup & configuration
## Structure, versions and credentials
Customize your first app picks up where the Getting started tutorial ends: a richer chart, units on the values, and a rebuilt version of the same app.
Building your own application is the longest tutorial here, a multi-part guide through importing code from a Git repository, working with the config files, creating components and adding custom ones. It is the path to take when the low-code app builder is not enough.
The rest are shorter and independent. Exporting and importing a configuration moves a project's setup between environments, which is how a staging project becomes a production one. Generating API keys creates the credentials an external system uses to authenticate against the project. Versioning basics covers the platform's own version control on action repositories, and how to sync a repository with Git instead.
:::warning
The application development tutorial is being reworked, with no date set. It remains accurate for the current platform.
:::
[### Customize your first app→
Pick up where Getting started ends. A richer chart, units, and a rebuilt version of the app.
](/tutorials-customize-first-app)[### Develop your own app→
Six parts, from a Git import to custom components. Rework pending, no date.
](/tutorials-app-development)[### Export / import a configuration→
Move a project's setup between environments.
](/tutorials-export-import-config)[### Generate API keys for external access→
Credentials for an outside system.
](/tutorials-generate-api-keys)[### Learn versioning basics→
Platform version control, or Git on a repository.
](/tutorials-versioning-basics)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-customize-first-app.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Customize your first application further
## Objective
In this tutorial you'll discover advanced features of the visual app builder to **customize the visual aspects** of a simple application.
:::info
This tutorial contains extended content for the _Getting Started_ section called [Build and deploy a front-end application](https://docs.dataplatform.ovh.net/getting-started-visualize-data.md).
:::
## Requirements
This tutorial assumes that you have been through **all five sections** of the [Getting Started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). Your Data Processing Engine, your Lakehouse Manager, your Analytics Manager have been configured as described in their respective pages, and you have deployed an API as well as an APP which you are going to customize.
## Create a more advanced chart
First, let's discover one more chart type: a horizontal bar chart to show the top 7 stations in terms of total traffic.
Instead of choosing a query that has already been prepared in the Analytics Manager, you can also **create new queries on the fly** when creating a chart. To do this, shrink down the first panel and **create** a second one next to it.
Inside the second panel, click on the ➕icon and select **HTML**. Name it _Top 7 Stations_ with style **H1**.
Inside the second panel, click on the ➕icon and select Chart. Pick the _Recharts - Bar Horizontal_ type.
In the Request tab, choose the _Request_ mode and enter _rides_ as data (Sum and Descending) and _station\_name_ as scale.
One last step! Queries can be fully customized using the advanced JSON editor. More specifically, you need to add a custom parameter to **limit the number** of stations to 7. To do that, head to the advanced mode of your query as shown below and simply add `"limit":7,` below line 2 in the "data" JSON object:
So far, your dashboard should look something like this. Don't forget to **save**.
## Customizing further
There is a lot that you can do in the app builder to design your app exactly the way you want it.
### Units
Click on the **play** button at the top of the dashboard to switch to read-only mode. Hover over the bars of the _Top 7 stations_ chart.
These units could be formatted in millions for better clarity. To do that, go to the **Formatter** tab. A measure called _RIDES_ was already added by choosing the Getting Started template.
:::info
More generally, if you need to add a new measure, it must have the **same name as the attribute** that you are trying to format.
:::
As for the formatting parameters, enter _Mpax_ for unit, _0_ for round (to display integers), and _0.000001_ for multiplicator (to display millions).
:::info
The Formatter runs in autosave mode: your changes are saved as you type, and there is no Save button to click.
:::
Here's the result:
## Rebuild and deploy the new version of your app
One last step: publish the recent dashboard changes to production. To do this, don't forget to save and then go back to the **Overview** tab of the Application Manager. You should first **Stop** the currently deployed application. As soon as it has shut down, click on **Build**, and then **Deploy**.
:::info
The **build** process usually takes a couple of minutes. If you're curious of what is going on or if the build fails: open the log console by expanding the version panel and inspect the logs. Feel free to send logs to our support team if you're not sure how to troubleshoot it yourself!
:::
Your app is **live** and can now be accessed online!\
The link can be obtained in the Overview tab by clicking on **Open**.
Your figures are now available in the application.
**You've completed the Advanced application builder tutorial!** You are now ready to build complex data applications on the Platform.
Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed.
Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications your have in mind.
## Looking for more flexibility?
The visual app builder is great for getting started quickly, but if you are a developer looking for **full control over your application code**, we provide additional templates built with modern front-end frameworks.
These templates connect to the same Front API and let you build fully custom applications using your own development workflow.
[Build a custom app with React, Vite & Shadcn](https://docs.dataplatform.ovh.net/tutorials-app-development-react-shadcn.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-app-development.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Develop your own application
## Objective
In this advanced step-by-step guide, you will learn how to make your own custom application components, whether charts and dynamic filters, customize them, embed them in the platform and much more.
Here, you will continue the development of the application you created in the [first part of the Getting Started Guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md).
The goal? Provide you with all the tools to create custom code based off the API references & boilerplates provided.
:::warning
This is an advanced guide aimed at developers, front-end engineers, or application architects. We recommend being familiar with basic front-end development concepts.
:::
Below is a **quick preview of what we will be able to achieve by the end of this guide**. As you can see you'll add a custom map component, more complex filter options & cleaner styling for all other charts.
## Let's get rolling!
👉Here are the steps that you'll need to follow:
[
Build
How to build, deploy & manage versions using external code
Add custom components (or extensions) as dynamic parameters and charts
](https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-app-development-build.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build your app with an external repository
## Objective
Here, you will learn how to set-up a React application and run it locally.
By the end of this guide, you will be able to edit a Data Platform application by running it **locally**, **connecting** it to the Platform' ecosystem and **deploying** it with a git repository.
## Requirements
There are a few things you need before starting :
- Follow the [Analytics App Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) up until [this](https://docs.dataplatform.ovh.net/getting-started-create-queries.md) point. This is where you expose your API.
- Install [git](https://git-scm.com/).
- Install [Node.js](https://nodejs.org/en/).
- Install [yarn](https://yarnpkg.com/) (recommended) or [npm](https://www.npmjs.com/).
- Install [npx](https://www.npmjs.com/package/npx).
- Have a **text editor**. You'll be writing HTML, CSS, and JavaScript.
:::info
The point of communication between your **Data Platform Project** and your app, is the **Data Platform API** that you exposed in Application Services. That is why it is required to complete the [Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) until the part where you expose your data and queries on the API.
:::
## Set-up an application & connect it to Data Platform
### Create an app using an existing template
Clone this [repository](https://github.com/forepaas/getting-started/) and open it:
```bash
git clone https://github.com/forepaas/getting-started.git
cd getting-started/
```
Switch to the branch release/basic
```bash
git checkout release/basic
```
Before installing the Node modules, you'll need to ensure our map-related dependencies are configured for the current environment. Open your package.json file and add the following lines to your dependencies section, or update them if they already exist:
```json
"dependencies": {
// ... other existing dependencies
"leaflet": "^1.6.0",
"react-leaflet": "^2.7.0",
"react-leaflet-markercluster": "^2.0.0"
},
```
Install node modules :
```bash
yarn
```
:::warning
If you get an error concerning **certificates** when installing yarn, you should contact your IT department to solve the issue.
:::
Install npx (Skip this step if you have already installed it globaly):
```bash
yarn global add npx
```
Install the Platform modules
```bash
npm run fppm i
```
:::warning
If you run into problems installing the modules, it may be due to your NodeJS configurations. Uninstall and reinstall NodeJS to fix the issue.
:::
:::warning
If the issue persists, it could be due to NodeJS version compatibility. Install NodeJS v16.20.2 to fix the issue.
:::
Run your app :
```bash
yarn start
```
:::info
Open `http://localhost:3333` to check your app. If you see no error but a blank screen, you still need to adjust the app configuration before the panels appear. The blank screen means your app is up & running.
:::
### Connect a git repository to the platform (optional)
To link a git repository to the platform, please follow this [guide](https://docs.dataplatform.ovh.net/app-services-apps-git-integration.md)
:::info
Now that you've learned how to create and launch your app, it's time to understand its architecture.
:::
[Understand app architecture and configuration files](https://docs.dataplatform.ovh.net/tutorials-app-development-config.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-app-development-config.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover the app configuration files
## Objective
This guide covers the configuration files of a Data Platform application: what each directory holds, then how to edit the global configuration, an API attribute, the sidebar's date-picker and a dashboard.
## Understand the app files
### Understand the app architecture
You have now a complete application. Your app tree should look like this:
```bash
├── config
├── node_modules
├── forepaas
├── public
├── scripts
└── src
```
Let's clarify the content of each directory:
- `config`: This folder holds most of the application configuration. In this folder, you can define and update your configuration files for dashboards, menus, styles, translations etc... The most important file here is `global.json`, it allows you to declare and use all of these configuration files. (explained right after)
- `forepaas`: This is the equivalent of `node_modules` for the platform. It contains all the basic components needed to embed your application to Data Platform. All the packages listed in /forepaas.json are installed here.
- `node_modules` : The node\_modules folder contains libraries downloaded from npm. It is based on /package.json.
- `public`: This folder contains all the static files (fonts, icons, images...). You can access a file stored in this folder through this url `/assets/{subfolder}/{name}.{extension}`.\
For example, the url to access `favicon.png` (and display it in a `` tag) located in `assets/img` is \`/assets/img/favicon.png
- `scripts`: This folder contains some useful scripts that you can use to generate components, dashboards or zip this Project to upload it easily on the platform.
- `src`: In this folder, you can create or update your own custom components and services. We will teach you how to do it in the next steps of this tutorial.
### Understand the configuration files
The global configuration file is `global.json`.
This file represents the entry point of your application.\
It is where you declare your environment variables, routes, dashboards, and many more important fields.
Open `global.json` located in the `config` folder.\
Initially, it looks like this:
```json
{
"authentication": "{{ENV.CAM_URL}}",
"api": "{{ENV.API_URL}}",
"application_name": "{{ENV.APP_NAME || 'ForePaaS'}}",
"debug": "{{ENV.DEBUG}}",
"loader": "spinner",
"lang": "fr",
"version": 3,
"storage": {
"type": "sessionStorage"
},
"root": "/rides-analytics",
"menu": [
"file://config/menus/header.json",
"file://config/menus/sidebar.json"
],
"dashboarding": {
"/rides-analytics": "file://config/dashboards/rides-analytics.json"
},
"style": "file://config/style.json",
"reportings": "file://config/reportings.json",
"i18n": {
"en": "file://config/i18n/en.json"
},
"formatter": "file://config/formatter.json"
}
```
Let's explain the most important fields below :
- `authentication` : refers to the authentication url, it should have this shape: `https://{api-subdomain}.eu.dataplatform.ovh.net/cam?app_id={app_id}`.\
By default it is defined as an environment variable. It is important to keep it this way to deploy your app on production. Please refer to this [part](#overwrite-the-global-configuration-file) to understand how to overwrite this field for local development purposes.
- `api` : Same as above, but, refers to the api URL, it should have this shape: `https://{api-subdomain}.eu.dataplatform.ovh.net/{api-alias}`.
- `application_name` : Your application title
- `root` : The root path of your application.
- `menu` : An array containing your menus configuration files.\
The header and the sidebar are already defined by default.
- `dashboarding` : An object to define your dashboards. The attribute is the dashboard url, and the value is the dashboard configuration file.
- `style` : The style configuration file. It allows you to style the components.
Some components are already styled there.
- `i18n` : An object to define all your supported languages. The attribute is the language alias, and the value is the translation configuration file associated to the language.
- `formatter` : The formatter configuration file. It allows you to format the data, round numbers, add units, etc...\
Some fields are already formatted by default.
## Edit the configuration files
Now that you are more familiar with app architecture, you can finally edit the app!
### Overwrite the global configuration file
The first step is to link your local application to the authentication service and to the API created previously on your Project.
Since we only need these changes to happen locally, we don't want to directly edit `global.json`. We will create another file `global-override.json` to override it instead.
- Create `global-override.json` in the `config` folder
- Copy and paste this json into your new file:
```json
{
"authentication": "https://{api-subdomain}.eu.dataplatform.ovh.net/cam?app_id={app_id}",
"api": "https://{api-subdomain}.eu.dataplatform.ovh.net/{api-alias}"
}
```
There are three variables that you need to replace:
- `{api-subdomain}` : Log into the platform, go to your API settings, this variable is referred to as the "Subdomain" in the Domain Name System panel
- `{api-alias}` : This variable can be extracted from the "Domain" by removing `eu.dataplatform.ovh.net/`
- `app_id` : Go to the Identity Access Manager page. In the Application tab, go to edit your app. This variable is referred to as "Id" in the application information.
:::warning
Remember that it is necessary to modify authentication privileges to allow to fetch your online data from your local build. Do it by adding your local host (`http://localhost:3333` by default) to the CORS origin in [_Identity Access Manager_](https://docs.dataplatform.ovh.net/landing-page-iam.md) > _Settings_ > _CORS origin_ > _Add_.
:::
Save all files. The locally running app now displays a bit more of an advanced view of what you had at the end of the Getting Started guide.
The following image shows how it will look on the first start after configuring the `global-override.json` but as you can see there seems to be an issue with the holiday select box and the rides history data display. The following section explains how to resolve these problems.
### Add the cat\_holiday attribute to your API
You may notice that in the sidebar the **holidays selector** isn't displayed correctly. That is the case because you have no exposed attribute related to holidays as part of the getting-started guide.
:::info
In the app template provided to you, that attribute is named _cat\_holiday_. We encourage you to try to understand why this problem appears by looking into the _sidebar.json_ file for mentions of the dictionary name.
:::
To address this, you have to:
- Add a string attribute named _cat\_holiday_ to the table _dataset\_history_.
- Build the _dataset\_history_ table and generate the respective DPE Action again.
- In the DPE, change the table's Aggregate Action to correct create the _cat\_holiday_ attribute with the code below.
```sql
CASE
WHEN chicago_calendar_full.christmas_holidays=1 THEN 'Christmas Holidays'
WHEN chicago_calendar_full.summer_holidays=1 THEN 'Summer Break'
WHEN chicago_calendar_full.spring_holidays=1 THEN 'SpringBreak'
WHEN chicago_calendar_full.public_holiday=1 THEN 'Bank Holiday'
WHEN chicago_calendar_full.autumn_holidays=1 THEN 'Fall Holidays'
WHEN chicago_calendar_full.sport_holidays=1 THEN 'Sport Holidays'
ELSE 'off leave'
END
```
- Run the respective workflow again.
- Flush the cache in Application Services.
Now, just rerun your app and the **Holidays Selector** should be looking good.
### Edit the range of the date-picker in sidebar.json
The reason the rides-history chart shows "no data" is that the date-picker is not set to match the dataset’s actual range. Currently, the sidebar’s date-picker only displays a date range from today to one year ago because of its configuration. Since our Chicago files dataset covers the years 2015 to 2020, we need to update the `sidebar.json` file by modifying specific fields to ensure the date-picker includes the full range of our data.
To address this, you have to:
- go to `sidebar.json`
- find the fields for startAt, minDate, maxDate, endAt.
- replace the above section with the code below.
```json
"startAt": "2015-01-01",
"minDate": "2015-01-01",
"maxDate": "2020-12-31",
"endAt": "2020-12-31"
```
Now, just save and rerun your app and the **data-picker** should be having the range between 2015 and 2020.
Now that the date picker is fixed, let's test it out! Select a range, for example, from **January 1st, 2015, to January 1st, 2016**.
You should now see the rides-history chart populate with data. If the chart looks scattered or incorrect, it's probably because the data isn't ordered by date. To sort this, open `rides-analytics.json` and add the following snippet to the `rides-history` request definition:
```json
"order": {
"yearmonth": "asc"
}
```
### Change the attribute used for stations picker in the sidebar
Currently, when you interact with the 'Stations' selector in the sidebar, you'll see a list of raw station IDs. While these are technically correct, they don't offer much in terms of user-friendliness! Let's enhance this by displaying the actual station names instead, making the selection process much clearer.
To achieve this:
- Navigate to the `sidebar.json` configuration file.
- Locate the component definition that has the ID `dyn-stations`.
- Update the attribute property within this component's dictionary to **station\_name**. This tells the component to fetch and display the human-readable names from your dataset.
- Next, you might observe that the **Top 7 Stations** chart is currently displaying `station_id` values instead of more readable station names. To correct this, apply a similar change to the one you made earlier:
- open your `rides-analytics.json` file and locate the configuration for the 'Top 7 Stations' chart.
- Then, update its relevant field **scale** to use `station_name`.
## Edit a dashboard
As you saw in `global.json`, your application already has a dashboard called `rides-analytics`.
In this part, you will see how to edit the dashboard configuration file to change the layout and the positions of the elements.
Let's start by opening `rides-analytics.json` in `config/dashboards`.
You will first create a panel and control the height, the width and the position of an item.
To create a panel, append these few lines to the end of the array `items`
It will be a placeholder for your future search bar, so let's make `search-bar` its id.
```json
{
"type": "panel",
"id" : "search-bar",
"sizeX": 100,
"sizeY": 100,
"row": 0,
"col": 0,
"items": []
}
```
- _type:_ Represents the item's type. It can be a component provided by the Platform by default or a specific component.
- _id:_ The id of your item.
- _size X:_ Represents the width of the item. The value must be between 0 and 100 as it represents a percentage of the containing item's width.
- _size Y:_ Represents the height of the item.
- _row:_ Represents the row offset.
- _col:_ Represents the column offset.
After adding the panel, the application looks like this:
Now, edit the panel to reduce its height by changing the value of _sizeY_ to 19.\
After that, move **all the other panels** to the bottom by changing the value of _row_ for each of them. Add 20 to the current value to push down the other panels.
Now, to ensure the best possible viewing experience when we introduce our custom interactive map later in the tutorial, we'll make a strategic update to the dashboard's layout. This isn't just for aesthetics; it's designed to allow the map to co-exist gracefully with the rest of your dashboard. As part of this, we'll also add a new panel to serve as the perfect placeholder for our interactive map.
Replace the entire content of your rides-analytics.json file with the configuration provided below. Upon saving, you'll observe the updated dashboard layout.
:::details rides-analytics.json
```json
{
"name": "Rides analytics",
"width": 100,
"height": 100,
"margin": [
10,
10
],
"description": "",
"items": [
{
"type": "panel",
"id" : "search-bar",
"sizeX": 100,
"sizeY": 19,
"row": 0,
"col": 0,
"items": [
]
},
{
"type": "panel",
"id": "custom-map",
"sizeX": 75,
"sizeY": 99,
"row": 20,
"col": 0,
"items": []
},
{
"type": "panel",
"id": "top-stations-container",
"sizeX": 24,
"sizeY": 99,
"row": 20,
"col": 76,
"items": [
{
"type": "chart",
"chart": {
"id": "chart-7843335",
"component": "recharts.bar-horizontal",
"options": {},
"request": {
"data": {
"limit": 7,
"fields": {
"rides": [
"sum"
]
}
},
"scale": {
"fields": [
"station_name"
]
},
"order": {
"rides": "desc"
}
}
},
"sizeX": 100,
"sizeY": 86,
"row": 6,
"col": 0
},
{
"type": "html",
"content": "Station top 7",
"customclass": "chart-title",
"sizeX": 100,
"sizeY": 5,
"row": 0,
"col": 0
}
]
},
{
"type": "panel",
"id": "rides-history-container",
"sizeX": 75,
"sizeY": 70,
"row": 120,
"col": 0,
"items": [
{
"type": "html",
"content": "Rides history",
"customclass": "chart-title"
},
{
"type": "chart",
"chart": {
"id": "chart-4992931",
"component": "recharts.line",
"request": {
"data": {
"fields": {
"avg_rides_per_day_per_station": [
"select"
]
}
},
"scale": {
"fields": [
"yearmonth"
]
},
"filter": {},
"order": {
"yearmonth": "asc"
}
},
"options": {
"yAxis": {
"domain": [
0,
6000
]
}
},
"dynamic-parameters": [
"dynamic-date",
"dynamic-holidays",
"dynamic-stations"
]
},
"sizeX": 100,
"sizeY": 55,
"col": 0,
"row": 9
}
]
},
{
"type": "panel",
"id": "weekly-seasonality-container",
"sizeX": 24,
"sizeY": 135,
"row": 120,
"col": 76,
"items": [
{
"type": "chart",
"chart": {
"id": "chart-3259339",
"component": "recharts.donut",
"request": {
"data": {
"fields": {
"avg_rides_per_day_per_station": [
"select"
]
}
},
"scale": {
"fields": [
"week_day_label"
]
}
},
"options": {}
},
"sizeX": 103,
"sizeY": 61,
"row": 29,
"col": 0
},
{
"type": "html",
"content": "Weekly seasonality",
"customclass": "chart-title",
"sizeX": 93,
"sizeY": 12,
"row": 0,
"col": 0
}
]
},
{
"type": "panel",
"id": "rides-temperature-container",
"sizeX": 37,
"sizeY": 64,
"row": 191,
"col": 0,
"items": [
{
"type": "chart",
"chart": {
"id": "chart-6930201",
"component": "recharts.bar",
"request": {
"data": {
"fields": {
"avg_rides_per_day_per_station": [
"select"
]
}
},
"scale": {
"fields": [
"cat_temperature"
]
}
},
"options": {}
},
"sizeX": 100,
"sizeY": 54,
"row": 6,
"col": 0
},
{
"type": "html",
"content": "Rides per Temperature",
"customclass": "chart-title",
"sizeX": 100,
"sizeY": 8,
"row": 0,
"col": 0
}
]
}
],
"url": "/rides-analytics",
"path": "",
"tags": []
}
```
:::
The layout of your rides analytics page is now ready.
Now we'll learn how to add a new dashboard to our application.
### Generate a dashboard
To generate a new dashboard, create a new json file in `config/dashboards`.\
You can speed up the process by using our script
```bash
yarn generate
```
Using the arrow keys choose **dashboard**, then type the name of your new dashboard ("learn" in this case)
To link your dashboard to your app, edit `global.json` by adding a new entry in the _dashboarding_ section.
```json
"/learn": "file://config/dashboards/learn.json"
```
The key represents the url path of your new dashboard.\
The value represents the path to the .json file associated with this dashboard.
Your new dashboard page should now be accessible at `http://localhost:3333/#/learn`.
As a last step, we will need to enable access to this dashboard from the UI.
To do so, go to `config/menus/header.json` and add this entry in the _items_ array of the _tabs_ container:
```json
{
"type":"link",
"text":"Rides",
"url":"/rides-analytics"
},
{
"type": "link",
"text": "Learn",
"url": "/learn"
}
```
This will allow us to easily switch between the two dashboards we have.
## You're making good progress!
You are now familiar with the app architecture and how to work on it locally.\
Let's introduce the topic of the next page: customizing your application. There are several ways you can use to style your components:
- Use [style templates](https://docs.dataplatform.ovh.net/sdk/app-charts-template.md): It allows you to define templates which will be applied by default.
- Style at the component level: The style will be only applied to a specific component. You can achieve this by passing some props when declaring a chart or a dynamic parameter.\
Example: Go to `config/dashboards/rides-analytics.json` and see the options field inside the line chart inside _rides-history-container_ panel.
- Style at the module level (for dynamic parameter only): The style will be applied to all the dynamic parameter of this type.
Example: properties passed inside `select-box` in `style.json`
:::info
**Note**: We recommend you use [style templates](https://docs.dataplatform.ovh.net/sdk/app-charts-template.md) for charts and `style.json` for dynamic parameters.
:::
:::info
**Deprecated for chart**:
:::
- Style at the module level (eg. recharts): The style will be applied to all the components of the module.\
Example: properties passed inside `recharts` in `style.json`
- Style at the component type level (eg. recharts-bar): The style will be applied to all the components that share the specified component type\
Example: properties passed inside `recharts-bar` in `style.json`
Next, learn how to create and edit charts.
:::info
[Edit and create custom charts](https://docs.dataplatform.ovh.net/tutorials-app-development-components.md)
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-app-development-components.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create and edit components
## Objective
In this tutorial, you will learn how to create and edit your first charts, and add them to your dashboards.
## Create and edit a chart component
### Add a component
You can easily add a new component to your Rides Analytics dashboard.
To do so, append this json object to the array _items_ in `rides-analytics.json`
```json
{
"type": "panel",
"id": "rides-holidays-container",
"sizeX": 37,
"sizeY": 64,
"row": 191,
"col": 38,
"items": [
{
"type": "chart",
"chart": {
"id": "chart-6930201",
"component": "recharts.bar",
"request": {
"data": {
"fields": {
"avg_rides_per_day_per_station": ["select"]
}
},
"scale": {
"fields": ["cat_holiday"]
}
},
"options": {}
},
"sizeX": 100,
"sizeY": 54,
"row": 6,
"col": 0
},
{
"type": "html",
"content": "Rides variability on Holidays",
"customclass": "chart-title",
"sizeX": 100,
"sizeY": 5,
"row": 0,
"col": 0
}
]
}
```
This json defines a panel that contains 2 items, the **chart** itself and **its title**.
A **chart** is a special item that needs a request to work. Once the chart component is loaded, the request is performed and the result is displayed differently according to the chart's type.\
The request has the same format as the [Analytics Manager's queries](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md). You can also create your request in the Analytics Manager and put the request id instead of the request itself. In this case, you will have to replace `request` by `requestId` in the json above.
The type of the chart is defined with the `component` key. There are many types of charts available natively in Data Platform, but you can also create your own components, we will show you how to achieve this in the next part of this guide.\
In this example, we've defined a new bar chart using the [recharts](https://recharts.org/en-US/) library. The request computes the average number of rides per day for each holiday.
Initially, your chart looks like this:
### Customize a chart
Since this chart is made with the recharts library, there are many properties that you can change.
:::info
The full list is available in [Recharts Bar documentation page](https://recharts.org/en-US/api/BarChart).
:::
To pass these properties from your chart configuration json, you simply have to edit the `options` key relative to this chart.\
For example, to change the size of the bars, add a `barSize` property in the `options` as below:
```json
"options": {
"barSize": 20
}
```
You can also customize the axes by adding the `xAxis` or `yAxis` property. You can then pass the properties available in [Recharts Axis documentation page](https://recharts.org/en-US/api/XAxis)\
For example, change the domain and scale of your YAxis:
```json
"options": {
"barSize": 20,
"yAxis": {
"domain": [
2000,
3800
],
"scale": "linear"
}
}
```
After these few changes, your chart looks now like this:
Let's backtrack a minute to the chart configuration file in order to explain the **title configuration**.\
This component is also natively available with Data Platform. It allows you to write HTML code in the `content` property. It will be wrapped in a div. You can associate a custom class to this div by using the `customclass` property.
The html code associated to the json defined above is:
```html
Rides variability on Holidays
```
:::info
Learn more about custom charts in the [**Technical Guide**](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md)!
:::
## Create and edit a dynamic parameter
A dynamic parameter allows the user to interact with one or more charts by changing their requests on the fly.
To create a new dynamic parameter, open `config/menus/sidebar.json` and add the following json object to the _containers_ object, right above the _footer_ item.
```json
{
"id": "dyn-weekday",
"items": [
{
"type": "html",
"content": "Weekday",
"customclass": "dyn-title"
},
{
"type": "dynamic-parameter",
"dynamic-parameter": {
"type": "filter",
"id": "dynamic-weekday",
"dictionary": "week_day_label",
"component": "selectbox",
"placeholder": "Select Weekday",
"multi": true
},
"style": {
"paddingBottom": "10px"
}
}
]
}
```
This json defines a dynamic parameter that allows you to filter chart requests by one or more week days.
It follows the same logic as the one followed by charts.
Here, we detail a bit this configuration:
- `type`: indicates how the dynamic parameter will be used, as a filter in this case. It means that any chart associated with this dynamic parameter will see its results be filtered to display only those matching the selected weekdays.
- `dictionary`: sets the field linked to this dynamic-parameter.
- `component`: sets the component that this dynamic-parameter is associated with, in here, the _selectbox_ component.
- `id`: serves to reference this object inside the platform and is commonly used to link this dynamic parameter to charts.
- `placeholder`: specific to the _selectbox_ component, specifies the text displayed inside the selectbox by default.
- `multi`: specific to the _selectbox_ component, it's a boolean property that allows selecting several options when set to true.
After copy-pasting the json object above defining this dynamic parameter, your new component should now appear in the sidebar:
To actually use this newly created dynamic-parameter, you have to link it to a chart.
You can do this by editing the charts configuration.
- Open `config/dashboards/rides-analytics.json`
- In the json object defining the **rides history chart**, append `"dynamic-weekday"` to the dynamic-parameters array.
- Reload your page, your dynamic parameter is now linked to the rides history chart.
- Select a day to filter the chart by weekday and watch it refresh with new data.
:::info
Learn more about dynamic parameters in the [**Technical Guide**](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters.md)!
:::
You have learned how to use existing components. You are now ready to take on [creating your own custom components and use them to make unique dynamic parameters and charts!](https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Add custom components
## Objective
With the React framework, you can **create your own components** from scratch. These components can be used for two things: [dynamic parameters](https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md#use-a-custom-component-as-a-dynamic-parameter) and [charts](https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md#use-a-custom-component-as-a-chart).
:::info
At any point, feel free to refer to the [**Technical Guide**](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md) to learn more about custom components!
:::
## Use a custom component as a dynamic parameter
### Generate a component
Let's see how to create a custom component and how to use it as a dynamic parameter.
As an example, we will show step-by-step you how to create this checkbox group component:
To generate a component, you can use our pre-build script by running the command below:
```bash
yarn generate
```
Select the **component** generator, name it `DynamicParameterCheckbox` and answer "No" to the last question as shown below:
:::warning
It is important that your component name starts with **DynamicParameter** to register it as a dynamic parameter
:::
This script generates a folder in `src/components` containing three files :
- `DynamicParameterCheckbox.jsx`
- `DynamicParameterCheckbox.less`
- `index.js`
### Declare a component
Before you start working on the component itself let's take a minute to see how to declare it and use it.
Follow these steps to declare the new component:
- Go to `src/components/index.js`
- Import the component by adding this line at the top of the file :
```js
import DynamicParameterCheckbox from "./DynamicParameterCheckbox";
```
- Declare it by adding its name to the components object
```js
components: {
...
DynamicParameterCheckbox
},
```
You can now call this component in your configuration files by specifying its **alias**. To obtain a component alias, transform its name from camelCase to kebab-case (`DynamicParameterCheckbox` becomes `dynamic-parameter-checkbox`), then remove "dynamic-parameter" and/or "chart" from it.
Here, the component's alias is `checkbox`. Now go to `config/menus/sidebar.json` and change the _component_ property of _dynamic-weekday_ to `"checkbox"` (i.e. the alias).
Go to your application page, notice that **the dynamic-weekday component has changed**.
The component is displayed but not yet functional: its behavior is defined in the next section.
### Edit a component
The framework used in this guide is going to be React.
:::info
We will guide you step by step to make your components, but if you are having any trouble because you are not familiar with this framework you can check the documentation [here](https://reactjs.org/docs/getting-started.html).
:::
Also, since we are working with functional components, we will use one of latest features available, **React Hooks**. You can check this [page](https://reactjs.org/docs/hooks-intro.html) for more information.
Let's start editing `DynamicParameterCheckbox.jsx`
Since this component is declared as a dynamic-parameter, some props are already automatically passed to it by Platforms data flow logic. These props can be logged when the component renders the first time, by making use of the [useEffect](https://reactjs.org/docs/hooks-reference.html#useeffect) hook.
Start by importing `useEffect` and `useState` (you will need this one later) by changing the first line of the file:
```js
import React, { useEffect, useState } from 'react'
```
Then add `props` to the arguments of the functional component:
```jsx
const DynamicParameterCheckbox = (props) => {
```
Finally, add the useEffect hook inside this function right before the return statement.
Your functional component should be defined as below:
```jsx
const DynamicParameterCheckbox = (props) => {
useEffect(() => {
console.log(props);
}, []);
return
DynamicParameterCheckbox
;
};
```
Open your browser's console panel, you should be able to see the props.
As you can see, most of these props are passed in `sidebar.json` when defining the dynamic-parameter.\
The only exception is `items` which is the result of an internal request querying the field specified in `dictionary` (`week_day` in this case)
Now let's create the checkboxes.
To achieve this, you will need to install a small package: `react-checkbox-group`. Do so by running:
```bash
yarn add react-checkbox-group
```
This package allows you to use a checkbox group component. You can now import it by adding this line at the top of the file:
```js
import CheckboxGroup from 'react-checkbox-group'
```
Add a boolean variable `expanded` to the component state to be able to display or hide the `CheckboxGroup` with a little arrow icon:
```js
const DynamicParameterCheckbox = (props) => {
const [expanded, setExpanded] = useState(true)
...
```
Change the function return statement like this:
```jsx
return (
)
```
The title section is using _props.title_. This means that you need to pass a `title` prop in **dynamic-weekday** definition in `config/menus/sidebar.json`, as shown below: (the HTML element has also been removed since it's now redundant)
```json
{
"id": "dyn-weekday",
"items": [
{
"type": "dynamic-parameter",
"dynamic-parameter": {
"type": "filter",
"id": "dynamic-weekday",
"dictionary": "week_day_label",
"component": "checkbox",
"title": "Days"
},
"style": {
"paddingBottom": "10px"
}
}
]
}
```
Finally, style your component by changing `DynamicParameterCheckbox.less`, using the code snippet below.
:::details DynamicParameterCheckbox.less
```less
// DynamicParameterCheckbox stylesheet
@import "../../styles/_variables.less";
.checkbox-header {
display: flex;
flex-direction: row;
align-items: center;
}
.dyn-checkbox-container {
display: flex;
align-items: flex-end;
margin: 3px;
}
.dyn-checkbox {
-webkit-appearance: none;
background-color: #fafafa;
border: 1px solid #cacece;
box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05), inset 0px -15px 10px -12px rgba(0, 0, 0, 0.05);
padding: 7px;
border-radius: 3px;
display: inline-block;
position: relative;
}
.dyn-checkbox:hover {
cursor: pointer;
box-shadow: 0 1px 2px @primary-color, inset 0px 1px 3px @primary-color;
border: 1px solid @primary-color;
}
.dyn-checkbox:active,
.dyn-checkbox:checked:active {
box-shadow: 0 1px 2px @primary-color, inset 0px 1px 3px @primary-color;
}
.dyn-checkbox:checked {
background-color: @primary-color;
border: 1px solid @primary-color;
box-shadow: 0 1px 2px @primary-color, inset 0px -15px 10px -12px
@primary-color, inset 15px 10px -12px @primary-color;
color: white;
}
.dyn-checkbox:checked:after {
content: "\2714";
font-size: 12px;
position: absolute;
top: -1px;
left: 2px;
color: white;
}
.dyn-checkbox-label {
cursor: pointer;
font-family: "Source Sans Pro";
font-size: 14px;
letter-spacing: 0;
color: #97a7b7;
line-height: 17px;
margin-left: 5px;
}
.fa.expand-icon {
color: #97a7b7;
font-size: 12px;
cursor: pointer;
margin-left: 5px;
}
.fa.expand-icon:hover {
color: #3e4550;
cursor: pointer;
}
```
:::
Your dynamic-parameter looks good now, but it's not working properly yet.
There is a problem, you cannot check any box!
The reason is that you need to bind CheckboxGroup to our dynamic-parameter component. To do so, you have to add an array to the state of your component. It will keep track of the selected options, so name it `selectedOptions`:
```jsx
const DynamicParameterCheckbox = (props) => {
const [expanded, setExpanded] = useState(true)
const [selectedOptions, setSelectedOptions] = useState([])
...
```
Add a function `onChange` to update the array you have just defined:
```jsx
const onChange = options => {
setSelectedOptions(options);
};
```
Finally, you need to add the two properties (`onChange` and `value`, an array containing `selectedOptions`) to CheckboxGroup. Bind them by changing the return statement:
```jsx
return (
...
...
```
The last step is to bind this dynamic-parameter to the [redux global store](https://redux.js.org/api/store) to allow it to affect charts requests. We'll use React Redux hooks for a more modern and streamlined approach. Add these lines to the imports at the top of the file:
```jsx
import { useSelector, useDispatch } from 'react-redux'
import { set } from 'forepaas/store/querystring/action'
```
Next, inside your DynamicParameterCheckbox functional component, access the Redux state and dispatch function using hooks. Place these lines, inside your component function:
```jsx
const options = useSelector((state) => state.querystring[props.id])
const dispatch = useDispatch()
```
- `useSelector`: This hook allows you to extract data from the Redux store state. Here, we're fetching the value associated with props.id from the querystring part of your Redux state. This options variable will automatically update whenever the Redux state changes for this specific ID.
- `useDispatch`: This hook returns a reference to the dispatch function from the Redux store. You'll use this dispatch function to send actions to update the store.
Inside your `DynamicParameterCheckbox` functional component, create a helper function `updateModel` to dispatch your actions to the global store:
```jsx
const updateModel = (model) => {
if (props.id) {
let value = (model || []).map((item) => item && item.value ? item.value : item)
dispatch(set(props.id, value.length ? value : null))
}
}
```
This function now uses the dispatch function obtained from `useDispatch` to send the set action, updating the querystring in the Redux store.
Then, update your onChange function to simply call updateModel with the new options:
```jsx
const onChange = (options) => {
updateModel(options)
}
```
To ensure your component's internal state (selectedOptions) is synchronized with the Redux store's state (especially after a page refresh or external changes to the Redux state for this parameter), update your `useEffect` hook like this:
```jsx
useEffect (() => {
setSelectedOptions(options)
}, [options])
```
This `useEffect` hook will run whenever the options value (which comes directly from your Redux store via `useSelector`) changes. It keeps your component's local `selectedOptions` state in sync with the global Redux state, providing a consistent view of the dynamic parameter.
Your dynamic parameter is now connected to the global store. You can make sure of this because the charts connected to dynamic-weekday are affected by this dynamic parameter.
You have created your first custom dynamic parameter.\
The final code is available [here](https://github.com/forepaas/getting-started/tree/release/advanced/src/components/DynamicParameterCheckbox).
### Adding a search bar
Let's add a second dynamic parameter to be able to filter by station.
Generate a new component:
```bash
yarn generate
```
Select the **component** generator. Name it `DynamicParameterAutocompleteBox` and answer "No" to the last question.
Change the content of the newly generated .jsx file to this:
:::details DynamicParameterAutocompleteBox.jsx
```jsx
import React, { useState, useEffect } from 'react'
import Select, { createFilter } from 'react-select'
import { useSelector, useDispatch } from 'react-redux'
import { set } from 'forepaas/store/querystring/action'
const searchIcon = () => ({
alignItems: 'center',
display: 'flex',
':before': {
fontFamily: 'FontAwesome',
content: '"\f002"',
marginRight: 8
}
})
const customStyles = {
control: (base, state) => ({
...base,
height: '57px',
borderWidth: '0px',
boxShadow: '5px 5px -2px rgba(0, 0, 0, 0.1)'
}),
option: (base, state) => ({
...base,
color: state.theme.colors.neutral50,
fontSize: '13px',
height: '48px',
display: 'flex',
alignItems: 'center',
fontWeight: 500,
letterSpacing: 0
}),
menu: (base, state) => ({
...base,
zIndex: 2000
}),
multiValue: (base, state) => ({
...base,
borderRadius: '21px',
backgroundColor: state.theme.colors.primary75,
color: state.theme.colors.neutral0,
height: '34px',
alignItems: 'center'
}),
multiValueLabel: (base, state) => ({
...base,
color: state.theme.colors.neutral0,
display: 'flex',
alignItems: 'center',
marginLeft: '10px',
fontWeight: 600,
letterSpacing: 0
}),
placeholder: (base, state) => ({
...base,
color: state.theme.colors.neutral20,
fontSize: '14px',
...searchIcon()
}),
multiValueRemove: (base, state) => {
return ({
...base,
color: state.theme.colors.neutral0,
borderRadius: '10px',
padding: '0px',
margin: '10px',
border: `1px solid ${state.theme.colors.neutral0}`,
':hover': {
backgroundColor: state.theme.colors.neutral0,
color: state.theme.colors.primary75,
cursor: 'pointer'
}
})
}
}
const DynamicParameterAutocompleteBox = ({ items, ...props }) => {
const [selectedOptions, setSelectedOptions] = useState([])
const selectedValues = useSelector((state) => state.querystring[props.id])
const dispatch = useDispatch()
items.sort((a,b) => (a.label > b.label) ? 1 : ((b.label > a.label) ? -1 : 0))
const customTheme = theme => ({
...theme,
colors: {
...theme.colors,
primary: props.primaryColor || '#00CCF9',
primary25: props.primary25Color || 'rgba(16,182,233,0.06)',
primary50: props.primary50Color || 'rgba(16,182,233,0.20)',
primary75: props.primary75Color || 'rgba(16,182,233,0.8)',
neutral20: props.placeholderColor || '#97A7B7',
neutral50: props.menuColor || '#485465'
}
})
useEffect (() => {
const mappedOptions = items.filter(item => (selectedValues || []).includes(item.value))
setSelectedOptions(mappedOptions)
}, [selectedValues, items])
const updateModel = (model) => {
if (props.id) {
let values = (model || []).map((item) => item && item.value ? item.value : item)
dispatch(set(props.id, values.length ? values : null))
}
}
const onChange = (options) => {
if (options && !Array.isArray(options)) {
options = [options]
}
updateModel(options)
}
return (
)
}
export default DynamicParameterAutocompleteBox
```
:::
[Declare it](#declare-a-component) in `src/components/index.js`
by importing your new component and adding it to the `components` object defined in the file.
Go to `config/dashboards/rides-analytics.json`, look for the panel whose id is `search-bar` and **replace it with a dynamic-parameter** as such:
```json
{
"type": "dynamic-parameter",
"id": "search-bar",
"dynamic-parameter": {
"type": "filter",
"id": "dynamic-stations",
"component": "autocomplete-box",
"placeholder": "Search for a station",
"isMulti": true,
"dictionary": "station_name"
},
"sizeX": 100,
"sizeY": 19,
"col": 0,
"row": 0,
"items": []
}
```
Finally, remove the whole _dyn-stations_ object from `config/menus/sidebar.json`, as it is redundant with the new search bar.
## Use a custom component as a chart
In this section, you will learn how to create a custom component and how to use it as a chart.
As an example, you will create an interactive map component using [react-leaflet](https://react-leaflet.js.org/) library. The final result will look like this:
Let's start by creating the component using the script as usual.
```bash
yarn generate
```
Select the **component** generator, name it `ChartMap` and answer "No" to the last question.
:::warning
It is important that your chart component name starts with 'Chart', so it is recognized and registered properly.
:::
[Declare it](#declare-a-component) in `src/components/index.js`.
You can now use it in your analytics dashboard. Go to `rides-analytics.json`, and add the new chart and its title to the `custom-map` panel:
```json
{
"type": "panel",
"id": "custom-map",
"sizeX": 75,
"sizeY": 99,
"row": 20,
"col": 0,
"items": [
{
"type": "html",
"content": "Map view",
"customclass": "chart-title",
"sizeX": 100,
"sizeY": 5
},
{
"type": "chart",
"chart": {
"id": "chart-map",
"component": "map",
"dynamic-parameters": [
"dynamic-date",
"dynamic-holidays",
"dynamic-stations",
"dynamic-weekday"
],
"request": {
"data": {
"fields": {
"rides": ["avg"]
}
},
"scale": {
"fields": ["station_id", "station_name", "lat", "lng"]
},
"filter": {},
"order": {}
}
},
"sizeX": 100,
"sizeY": 88,
"row": 6,
"col": 0
}
]
}
```
Like dynamic-parameters, charts also automatically receive some properties when they are mounted. Let's see what they look like.
As usual, import `useEffect` and `useState` by **changing the first line** of the `ChartMap.jsx` file:
```jsx
import React, { useState, useEffect } from 'react'
```
Then add `props` to the arguments of the functional component:
```jsx
const ChartMap = (props) => {
```
Finally, add a useEffect hook inside this function right before the return statement.
Your final functional component should look like this:
```jsx
const ChartMap = (props) => {
useEffect(() => {
console.log(props);
}, []);
return
ChartMap
;
}
```
Open your browser's console panel, you should be able to see the props passed to this component.
Most of these props come from the chart definition in `config/dashboards/rides-analytics.json`.\
The exception is `data` which is a JSON representing the result of the request associated with the chart. This JSON is meant to be as generic as possible, so it can be a bit tedious to manipulate.
In this example -- and in most cases -- the JSON object is too complex so we use this helper function to flatten it and make it way easier to handle:
```jsx
const flattenChartResults = results => {
let flattenedChartResults = [];
results.forEach(result => {
let fieldEntries = [];
Object.entries(result.data).forEach(([fieldName, computeModes]) => {
Object.entries(computeModes).forEach(([computeModeName, rows]) => {
computeModeName === "select"
? fieldEntries.push([computeModeName, rows[0].value])
: fieldEntries.push([
`${fieldName}_${computeModeName}`,
rows[0].value
]);
});
});
let flattenedObject = {
...result.scales,
...Object.fromEntries(fieldEntries)
};
flattenedChartResults.push(flattenedObject);
});
return flattenedChartResults;
}
```
Copy and paste the code snippet above between imports and ChartMap, then update your useEffect hook content to visualize the new flattened results:
```jsx
useEffect(() => {
const results = flattenChartResults(props.chart.data.results);
console.log(results);
}, [props.chart.data.results]);
```
To display these results on the map, you need to store this data in the component's state. Declare an array inside the `ChartMap` component like this:
```jsx
const ChartMap = (props) => {
const [stations, setStations] = useState([])
...
```
Update useEffect to change this array instead of logging the results:
```js
useEffect(() => {
const results = flattenChartResults(props.chart.data.results);
setStations(results);
}, [props.chart.data.results]);
```
Now that you've prepared the data, let's see now how to make the map.
Now import react-leaflet components by adding this line to the top of your file:
```js
import { Map as LeafletMap, CircleMarker, Popup, TileLayer } from 'react-leaflet'
```
Go to `ChartMap.less` and change the stylesheet using the code snippet below:
:::details ChartMap.less
```less
// ChartMap stylesheet
@import "~leaflet/dist/leaflet.css";
@import "~react-leaflet-markercluster/dist/styles.min.css";
.leaflet-container {
height: 100%;
}
.leaflet-popup-content-wrapper,
.leaflet-popup-tip {
opacity: 0.85;
background-color: #3e4550;
box-shadow: inset 0 -3px 0 0 rgba(0, 0, 0, 0.13);
color: #fff;
}
.leaflet-popup-content-wrapper {
text-align: center;
border-radius: 1px;
}
.popup-title {
letter-spacing: 0;
font-size: 13px;
}
.popup-value {
font-size: 13px;
font-weight: 600;
}
```
:::
Go back to `ChartMap.jsx` and define your default map parameters (center, zoom, minZoom) right below your state:
```jsx
const center = [41.89, -87.6297982]
const zoom = 14
const minZoom = 7
```
Change your function return statement to display an empty map centered on Chicago:
```jsx
return (
)
```
Now add circle markers inside `LeafletMap` right below the `TileLayer` to represent stations:
```jsx
{
stations.map(station => (
))
}
```
:::info
You can customize your circle markers by adding some props to control their size, their weight, their color, etc. You can find the complete documentation [here](https://leafletjs.com/reference-1.6.0.html#circlemarker)
:::
Below a code snippet changing the circles' color and making their size proportional to the rides.
```jsx
```
Next, add a popup to display the station name and the number of rides associated to it when clicking a circle.
Add it inside `CircleMarker` as shown below:
```jsx
{station.station_name}
{station.rides_avg}
```
You can make the average number of rides follow the formatting rules specified in `config/formatter.json` for `rides` by using `FpMeasure`.
Import FpMeasure:
```js
import { FpMeasure } from 'forepaas/formatter'
```
Change the popup value:
```jsx
```
If you play a little with your map, you will notice that it keeps reloading when you change your dynamic parameters. You can disable this behavior by adding a `noReload` property into your chart configuration options in `config/dashboards/rides-analytics.json`:
```json
"chart": {
"id": "chart-map",
"component": "map",
"noReload": true,
"dynamic-parameters": [
...
],
"request": {
...
}
},
```
Reorganize your circles by clusters to avoid overlapping, it's pretty straightforward thanks to the `react-leaflet-markercluster` library.
Import the MarkerClusterGroup component:
```js
import MarkerClusterGroup from 'react-leaflet-markercluster'
```
Then **wrap your circle markers** in `MarkerClusterGroup` like this:
```jsx
{stations.map(station => (
))}
```
You have created your first custom component.\
The final code is available [here](https://github.com/forepaas/getting-started/tree/release/advanced/src/components/ChartMap).
:::info
Learn more about creating your custom components in the [**Technical Guide**](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md)!
:::
## Publish your local app on Data Platform
Now that you've completed your app locally, it's time to plug it into the platform.\
You can either do it by [uploading the app as a zip file](#import-your-app-as-a-zip-file), or by [connecting it to a git repository](#connect-your-app-to-a-git-repository).
### Import your app as a zip file
First, you need to export your app as a zip. Your zipped file should not contain the following folders: `node_modules`, `forepaas`, `webpack`, `.git`.
You can also use our pre-build script to zip your app, to generate a zip file ready to be uploaded to your Project, simply run this command.
```bash
yarn zip
```
A zip file with the same name as your application folder will be generated.
Then, you can either create a new application using this .zip file by following [these steps](https://docs.dataplatform.ovh.net/getting-started-visualize-data.md#build-and-deploy-a-front-end-application) or you can edit an existing application and upload the zip file to create a new version of your application:
### Connect your app to a git repository
To link a git repository to the platform, please follow this [guide](https://docs.dataplatform.ovh.net/app-services-apps-git-integration.md).
Once your app is uploaded, build it then deploy it.
Your fancy app is now deployed on Data Platform!
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-app-development-react-shadcn.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build a custom Front App with React, Vite & Shadcn
## Objective
This guide introduces the **React Shadcn Vite** application template, a developer-oriented starter kit for building fully custom front-end applications on top of the Data Platform [Front API](https://docs.dataplatform.ovh.net/sdk/api.md).
This template gives **full control over application code**, allowing developers to build tailored data applications using modern front-end technologies. For a low-code drag-and-drop experience, use the [visual app builder](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md).
:::warning
This is an advanced template for developers comfortable with **React, TypeScript, and modern front-end tooling**. For a no-code approach, head to the [Getting started guide on building an app](https://docs.dataplatform.ovh.net/getting-started-visualize-data.md).
:::
## Requirements
To use this template, first complete the following steps on the Platform:
- A **Lakehouse Manager** schema has been built and populated.
- A **Data Processing Engine** workflow has been executed successfully.
- **Queries** have been created in the Analytics Manager.
- A **Front API** has been [deployed](https://docs.dataplatform.ovh.net/getting-started-expose-api.md) and is running.
- [Node.js](https://nodejs.org/) (v22 or above) has been installed on your machine.
## What's included
### Tech stack
| Technology | Purpose |
| ----------------------------------------------- | ------------------------------------- |
| [React](https://react.dev/) 19 | UI framework |
| [TypeScript](https://www.typescriptlang.org/) 5 | Type-safe development |
| [Vite](https://vite.dev/) 6 | Build tool & dev server |
| [Tailwind CSS](https://tailwindcss.com/) 4 | Utility-first CSS framework |
| [Shadcn/UI](https://ui.shadcn.com/) | Component library (based on Radix UI) |
| [React Router](https://reactrouter.com/) 7 | Client-side routing |
| [TanStack Query](https://tanstack.com/query) 5 | Server state & data fetching |
| [Zustand](https://zustand.docs.pmnd.rs/) | Client state management |
| [Recharts](https://recharts.org/) | Data visualization |
| [i18next](https://www.i18next.com/) | Internationalization (EN/FR) |
### Modules
The template comes with three built-in modules:
#### `ts-sdk`: Data Platform SDK
The core module that handles communication with the Platform. It provides:
- **`DataPlatformApi`**: An API service that manages two Axios instances, one for the Identity Access Manager (IAM) and one for the Front API.
- **Automatic token injection** via Axios interceptors on every request.
- **Query execution** through `POST /qb/query` to retrieve data from your Analytics Manager queries.
- **Application preferences** fetching for branding and authentication configuration.
- **Environment configuration** loaded from `public/environments-vars.json`.
#### `rts-authentication`: Authentication
A complete authentication system providing:
- **Login page** with support for standard credentials and OAuth providers.
- **Multi-Factor Authentication (MFA)**: email, SMS, and authenticator app.
- **Session management** with automatic refresh (every 15 minutes) and token expiration handling.
- **Password change** flow with strength validation.
- **Zustand-based session store** persisted in `localStorage`.
#### `rts-charts`: Chart components
Ready-to-use data visualization components built on Recharts:
- **``**: A wrapper component that accepts a `QueryRequest`, executes the query against the Front API, and renders the result.
- Supported chart types: **Bar chart**, **Line chart**, **Pie chart**.
- **`
`**: A data table component for tabular query results.
- Built-in **loading and error states**.
- **Dictionary support** for translating field values into readable labels.
## Project structure
```
├── public/
│ └── environments-vars.json # API & IAM endpoints configuration
├── src/
│ ├── main.tsx # Application entry point
│ ├── routes.tsx # Route definitions
│ ├── components/
│ │ └── ui/ # Shadcn/UI components
│ ├── config/
│ │ └── i18n/ # Translation files (EN/FR)
│ ├── contexts/ # React contexts (Auth, Dictionaries, etc.)
│ ├── forepaas/ # Data Platform modules
│ │ ├── ts-sdk/ # SDK module
│ │ ├── rts-authentication/ # Authentication module
│ │ └── rts-charts/ # Charts module
│ └── pages/
│ ├── Home.tsx # Landing page
│ └── dashboard/
│ ├── DashboardOne.tsx # Example dashboard with PieChart + BarChart
│ └── DashboardTwo.tsx # Example dashboard with LineChart + BarChart + Table
├── forepaas.json # Data Platform deployment configuration
├── vite.config.ts # Vite configuration
└── tailwind.config.ts # Tailwind CSS theming
```
## Getting started
### 1. Create the application on the Platform
From your Project's home page, open **Application Services** and click **Create an application**.
Select the **React Shadcn Vite** template from the store, fill in the application name, and confirm.
### 2. Configure the environment
The template connects to the Platform through two endpoints defined in `public/environments-vars.json`:
```json
{
"IAM_URL": "https://.eu.dataplatform.ovh.net/cam?app_id=",
"API_URL": "https://.eu.dataplatform.ovh.net/"
}
```
| Variable | Description |
| --------- | --------------------------------------------------------------------------------------------- |
| `IAM_URL` | The Identity Access Manager endpoint, including your application ID. Used for authentication. |
| `API_URL` | The Front API endpoint. Used for executing queries and fetching data. |
:::info
When running locally, the SDK automatically looks for `environments-vars-override.json` first, allowing you to set local development endpoints without modifying the main config.
:::
### 3. Run locally
```bash
npm install
npm run dev
```
The dev server starts on `http://localhost:3333`.
### 4. Build and deploy
```bash
npm run build
```
This generates a production-ready bundle in the `/production` folder. You can also use `npm run zip` to create a deployable archive.
On the Platform, the build and deploy process works the same way as described in the [app deployment guide](https://docs.dataplatform.ovh.net/getting-started-visualize-data.md#build-and-deploy-the-new-version-of-your-app).
## Working with queries
The template uses `QueryRequest` objects to fetch data from the Front API. A query request contains:
| Field | Description |
| -------------------- | ---------------------------------------------------------------------- |
| `data.fields` | The attributes to retrieve, with compute modes (e.g. `select`, `sum`). |
| `scale.fields` | The dimensions to group by. |
| `filter` | Filter conditions to narrow the results. |
| `dynamic_parameters` | Parameters for dynamic filtering (e.g. date ranges). |
| `order` | Sort order for results. |
| `data.limit` | Maximum number of results to return. |
### Example: using the Chicago dataset
If you followed the [Getting Started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md), the template includes example dashboards compatible with the Chicago bike rides dataset.
**Average rides per day of the week** (PieChart):
```typescript
const query: QueryRequest = {
data: {
fields: { avg_rides_per_day_per_station: ["select"] }
},
scale: {
fields: ["week_day"]
},
filter: {},
dynamic_parameters: []
};
```
**Top 5 stations by total rides** (BarChart):
```typescript
const query: QueryRequest = {
data: {
fields: { rides: ["sum"] },
limit: 5
},
scale: {
fields: ["station_name"]
},
order: { rides: "desc" },
filter: {},
dynamic_parameters: []
};
```
These queries are rendered using the `` component:
```tsx
```
:::info
The queries above use the same attributes (`avg_rides_per_day_per_station`, `rides`, `station_name`, `week_day`, `cat_temperature`) created during the [Analytics Manager step](https://docs.dataplatform.ovh.net/getting-started-create-queries.md) of the Getting Started tutorial.
:::
## Customizing the template
This template is a **starting point**. You are expected to develop your own pages, components, and business logic on top of it. Here are a few common tasks:
### Add a new page
1. Create your component in `src/pages/`.
2. Add a route in `src/routes.tsx`.
3. Add a navigation link in `src/components/Menu.tsx`.
:::info
The template includes a [Plop](https://plopjs.com/) generator to scaffold new dashboard pages. Run `npm run plop` and follow the prompts.
:::
### Add a new chart
Use the `` component with your own `QueryRequest`:
```tsx
import Chart from "@forepaas/rts-charts/components/Chart";
```
Supported types: `bar`, `line`, `pie`. For tabular data, use the `
` component.
### Add translations
Translation files are located in `src/config/i18n/`. Add your keys to both `en.json` and `fr.json`, then use them with the `useTranslation` hook from i18next:
```tsx
const { t } = useTranslation();
return
{t("my.translation.key")}
;
```
### Customize the theme
Tailwind CSS theming is configured in `tailwind.config.ts`. The template uses CSS variables (HSL-based) for colors, supporting both light and dark modes.
Shadcn/UI components can be customized or extended using the [Shadcn CLI](https://ui.shadcn.com/docs/cli).
## Go further
- [Front API documentation](https://docs.dataplatform.ovh.net/sdk/api.md): Learn how to customize and extend your API.
- [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md): Configure authentication providers and access rights.
- [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md): Create and manage your queries.
- [Apps documentation](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md): Learn about the visual no-code alternative.
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-export-import-config.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Export or import a Project's configuration
## Objective
It is possible to **export and/or import your Projects' configurations** for each component. Configurations are a bundle of files (usually JSON & code files) that contain all the information about how a certain component was set up in your Project. This is particularly useful when you're trying to reuse a set of actions or queries from one Project to another.
:::info
We recommend using a API development tool like [Postman](https://www.postman.com/) or [Insomnia](https://insomnia.rest/) to run cURL calls. This particular guide uses [Insomnia](https://insomnia.rest/), a free & secure program to test API calls.
:::
## Getting a temporary access token
To run all import and export scripts you will need a **temporary access token**.
To get an access token easily, start by logging-in to the platform via the interface, and **open any component**, like here the Data Processing Engine. Right-click on your screen and select **Inspect**.
This will open your browser’s console window: **navigate to the Network tab**. If no calls are visible, simply _press F5 or refresh your page_. You should see a list of calls available like in the screenshot above.
Right-click on the one starting with “workflows” or “actions”, navigate to the option **Copy** and select **Copy as cURL**. You have now copied a cURL call which already contains your secret access token to your clipboard.
Now open Insomnia and paste the cURL inside the main header at the top of the window (on the right side of the Insomnia title next to the _Get_ dropdown button):
When you copy-paste the cURL, Insomnia will **automatically detect all the parameters of the call and fill the form** for you. The only thing left for you to do is simply to change the main API call, instead of the one that you’ve just copy-pasted, and keeping the same list of _Header_ parameters.
## How to export/import a configuration with an API call
This section will demonstrate how to use an API call to export or import the configuration of a specific component in your Project.
:::info
The detailed [list of calls per component](#exportimport-api-calls-per-component) is available in the next section.
:::
As a general rule make sure to:
- use a _GET_ method when doing an _Export_ call
- use a _POST_ method when doing an _Import_ call
The following example will go through the whole procedure to export and then import the configuration for the Data Processing Engine.
### How to export a configuration
First, _export the configuration_ by copy-pasting the [corresponding URL](#exportimport-api-calls-per-component), change to _GET_ and click on _Send_.\
If the call is successful you will see on the right side the _TGZ_ file in the return response which you can download on your local machine. You can see on the screenshot below that the _TGZ_ (tar gzipped) file is not readable because its compressed.
### How to import a configuration
Now, let's **import the same TGZ file** which contains the Data Processing Engine's configuration. Change the _GET_ method to _POST_ and change the _Body_ type to _Multipart Form_ like shown on the screenshot below.
By choosing the _Multipart Form_ you have the option to add parameters in the body and in particular load the _TGZ_ file that you exported earlier as part of the request.
In the first input, type in `archive`, and on the dropdown arrow next to value select the _File_ option. This will let you select directly a file from your computer to load.
Select the _TGZ_ file you want to load into your Project:
Press **Send**
That’s it! Now refresh your Project page and you should see all the component configuration updated.
## Export/Import API calls per component
Here are listed the various API calls to export/import the configuration of each component in your Project.
As a general rule, make sure to:
- use a _GET_ method when doing an _Export_ call
- use a _POST_ method when doing an _Import_ call
:::warning
Don't forget to change `datademo` with your Project name and access token with its actual value. You can find the [slugified](https://en.wikipedia.org/wiki/Clean_URL#Slug) version of your Project name in the [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md).
:::
### **Connectors and Lakehouse Manager**
#### Export
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/dwh/v4/configuration/export' \
--request GET \
--header 'Authorization: Bearer TOKEN' \
--output dwh-settings.tgz
```
#### Import
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/dwh/v4/configuration/import' \
--request POST \
--header 'Authorization: Bearer TOKEN' \
--form archive=@dwh-settings.tgz
```
### **Data Processing Engine**
#### Export
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/dpe/v3/configuration/export' \
--request GET \
--header 'Authorization: Bearer TOKEN' \
--output dpe-settings.tar
```
_Query parameters_:
| Parameter | Description |
| :-------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filters` | The type of resources you want to extract (actions, workflows or environments), and the resources you want to export by name. By default all resource types will be exported. Example: `filters : {"action" : ["action1", "action2"]}` |
| `notebook_data` | **Optional.** If set to `true`, all files stored in the notebook's data store will be exported in addition to the `*.ipynb` files. By default, this parameter is `false`, and only `*.ipynb` notebook files are exported. A limit of 100 MB is applied to the total size of notebook files (including `notebook_data` if exported). |
:::info
It is also possible to call `/dpe/v3/configuration/export` as `POST` and pass parameters in a JSON object. This is useful for `filters` and `notebook_data` when the URL becomes too long or for security reasons.
:::
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/dpe/v3/configuration/export' \
--request POST \
--header 'Authorization: Bearer TOKEN' \
--header 'Content-Type: application/json' \
--data '{"filters":{"action" : ["action1", "action2"]}, "notebook_data":true}' \
--output dpe-settings.tar
```
#### Import
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/dpe/v3/configuration/import' \
--request POST \
--header 'Authorization: Bearer TOKEN' \
--form archive=@dpe-setting.tar
```
_Query Parameters_:
| Parameter | Description |
| :------------ | :--------------------------------------------------------------------------- |
| `sourceType` | Can be `tar` (default), `tgz` (compressed tar), or `url` |
| `sourceParam` | When using `sourceType=url`, the URL of the archive to import |
| `response` | By default, DPE import is asynchronous. Set to `true` to make it synchronous |
_Body parameters_
| Parameters | Description |
| :------------ | :---------------------------------------------------------------------- |
| `sourceParam` | When using `sourceType=url`, the URL of the archive to import |
| `archive` | The archive containing your configurations if `sourceType=tar` or `tgz` |
#### Notebook Manager (Standalone Export/Import)
For exporting and importing notebooks specifically, you can also use the Notebook Manager's dedicated endpoint. This endpoint currently does not support filters like the main DPE export.
**Export**
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/notebook-manager/v1/configuration/export' \
--request GET \
--header 'Authorization: Bearer TOKEN' \
--output notebooks-settings.tar
```
_(This endpoint only supports the `notebook_data` query parameters like the main DPE export.)_
**Import**
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/notebook-manager/v1/configuration/import' \
--request POST \
--header 'Authorization: Bearer TOKEN' \
--form archive=@notebooks-settings.tar
```
_(Import parameters (`sourceType`, `sourceParam`, `response`) are the same as for the main DPE import endpoint.)_
### **Analytics Manager**
#### Export
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/query-admin/v3/configuration/export' \
--request GET \
--header 'Authorization: Bearer TOKEN' \
--output am-settings.tgz
```
#### Import
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/query-admin/v3/configuration/import' \
--request POST \
--header 'Authorization: Bearer TOKEN' \
--form archive=@am-settings.tgz
```
### **Identity Access Manager - IAM**
#### Export
```bash
curl --globoff --url 'https://datademo.eu.dataplatform.ovh.net/iam/v4/backup/export?filters[applications]=true&filters[configuration]=true&filters[groups]=true&filters[mails]=true&filters[roles]=true&filters[authentication_providers]=true&filters[users]=true&filters[service_accounts]=true' \
--request GET \
--header 'Authorization: Bearer TOKEN' \
--output iam-settings.json
```
The response is a complete JSON of all configurations of the IAM (format .json).
It is possible to **select subsets of the configuration** using `filters[]` in the _Query parameters_.
#### Import
:::warning
**Please make caution when importing IAM configuration** as it will: **reset Project's private key**, **reset current sessions**, **all passwords** (using Project Auth Provider), and **all API Key / Secret Key**.
:::
This will **remove existing configurations, applications, users, groups, etc**.
This is done on the basis of the configuration imported, so it may concern only a **configuration subset** selected when exporting.
```bash
curl --url 'https://datademo.eu.dataplatform.ovh.net/iam/v4/backup/import' \
--request POST \
--header 'Authorization: Bearer TOKEN' \
--header "Content-Type: application/json"
--data @iam-settings.json
```
The JSON to import is directly in the _Body_ part.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Generate API and secret keys for a user / service account
## Objective
If you want to connect to the Data Platform from outside of the platform, you will need to generate **API and secret keys** to authenticate. The _API key_ is the public key associated to the user who is connecting to an API. The _secret key_ is the private key known only to the user. The combination of the two is used to authenticate, through the generation of temporary [authentication tokens](#generate-an-authentication-token).
API and secret keys can be generated per [user](https://docs.dataplatform.ovh.net/iam-users.md) or [service account](https://docs.dataplatform.ovh.net/iam-service-accounts.md).
## Generate a new set of API and secret keys
### For a user
Navigate to the **Users** tab of the Identity Access Manager. Search for the user for which you want to generate API & Secret keys, using the search bar on the top right if needed. Edit the user.
In the user's settings, scroll down to the **API Access** panel and click on _Add access_.
Select the lifetime of the keys (i.e. the expiration date from today) and whether you want to send an email to the user with the API & secret key information. Then press **Create**.
:::info
The _Default_ value can be changed in your Identity Access Manager settings.
:::
A new window will open with the API and secret key values.
:::warning
After closing the window, there is no way to view the secret key again. Make sure you keep a record a the keys either by sending an email to the user or storing them in your code.
:::
To delete a key, hover over the key that you would like to delete and click on the **trash** 🗑️ icon. Once the key is deleted, the credentials for the keys will become inactive and cannot be use to authenticate anymore.
### For a service account
Navigate to the **Service Accounts** tab of the Identity Access Manager. Search for the service account for which you want to generate API & Secret keys, using the search bar on the top-right if needed.
In the service account settings, locate the **API Access** panel and click on _Add access_.
After that, the [process is the same as for users](#for-a-user). Note that if you want to send an email with a record of the API and secret keys, you need to fill in an email address for the service account.
## Generate an authentication token
### Generation scripts
Once you've created API & secret keys, you need to generate a dynamic authentication token in order to authenticate to Data Platform via an API endpoint. Authentication tokens are only active for a short time: this duration can be configured in your IAM settings.
To generate a token, simply run the command samples below.
**cURL**
```bash
curl --request POST \
--url https://{project_subdomain}.eu.dataplatform.ovh.net/iam/login \
--header 'Content-Type: application/json' \
--data '{
"auth_mode": "apikey",
"apikey": "",
"secretkey": ""
}'
```
**Python 3+**
```python
import requests
url = "https://{project_subdomain}.eu.dataplatform.ovh.net/iam/login"
payload = {
"auth_mode": "apikey",
"apikey": "",
"secretkey": ""
}
response = requests.request("POST", url, data=payload)
print(response.text)
```
:::warning
Make to update the **\{project\_subdomain}** else your calls won't go through.
:::
[How to find your Project subdomain.](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-versioning-basics.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Learn versioning and repositories basics
## Objective
This tutorial demonstrates how to use the Platform's native versioning system or how to setup a Git repository to track the changes in your [Data Processing Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md).
## Introduction
### Requirements
To follow this tutorial, you need to be familiar with the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) and you should preferably have done the [Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). The essential is that you know what [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) are for and how to use them.
### Concepts overview
Editing actions is always on autosave mode. When you are trying to iterate on existing actions, it is recommended to create several versions of the actions' code. This lets you incrementally introduce variations to experiment and test your work, while keeping the version in production always intact.
**On the Platform, this versioning happens at the level of the repositories**. The repositories are the tabs you see at the top of the Actions' tree view.
You cannot version an individual action, but rather a whole repository of actions. The versioning panel can be accessed at the top right of the repository header:
There are 2 versioning systems on the Platform:
- [Data Platform version control system](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#use-the-platform-version-control-system): the default versioning system on all repositories
- [Git versioning](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#use-a-repository-linked-to-git): if you decide to synchronize your repository with Git
Now let's see how all of this works in practice!
## Use the Platform version control system
By default, each repository of actions on the platform has an integrated version control system which allows you to manually manage the different versions of the code (i.e. actions) they contain.
Repositories all have:
- a _deployed version_, which is the version served when [executing the job](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md).
- an _editing version_, which is the version currently being edited through the editor panel.
:::info
If your repository only has _one version_, the active version is the same as the deployed version, which means that you are editing your actions in production.
:::
If your repository has _two versions or more_, the deployed version cannot be edited. It can be viewed in read-only mode.
Versions need to be created manually. To create a new version for the repository, open the versioning panel at the top right and click on the **+** icon.
Choose which version to duplicate to create the new version.
After creating the new version, the _deployed version_ will not change. You are able to edit the new version directly.
Click on the **Play** button to set a new version as the deployed version: the version of the code used when actions of the repo are executed.
## Use a repository linked to Git
Each repository can be linked to an external Git repository in which case versions are synced with Git commits. This allows you to update & test actions continuously without affecting the version deployed in production.
### Linking a repository to Git
To link your repository on the platform to Git, you must click on the gear icon to edit an existing repository or create a new a one. Either of this actions will open the repository configuration window.
Click on _Connect to Git_ to expand the window.
Here the first thing you need to enter is the Repository SSH URL.
It is generally found on repository cloning options of common Git solutions. On GitHUB, for instance, it can be found on Code > Clone > SSH.
Next click on _FETCH_ to get the distant branches and select the branch you desire to use.
Finally, copy the Public SSH key given at the bottom and paste it in the list of SSH keys associated to your account on your Git solution. For GitHUB, you can follow [this tutorial](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account).
### Committing and pushing
After changing an object in a repo, your repo will display the uncommitted changes as shown in the image below. You can commit and push them by clicking on the arrow pointing upwards indicated in the image.
:::info
If there is a conflict when you push, your changes will not be pushed to the branch you configured, instead they will be pushed to a new branch with same name of the current branch and a suffix of _\_conflictN_, where N is the number of the conflict. For example, if your branch is named _current-branch_ and your push causes a conflict, the new branch will be named _current-branch\_conflict1_. If there is a conflict again when you push, the new branch will be named _current-branch\_conflict2_.
:::
### Pulling and merging
Pull distant changes by clicking on the arrow pointing downwards indicated in the image, a popup window will prompt you to confirm your decision. All uncommitted changes will be lost and both branches will be merged together.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-tutorials-data-engineering.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data ingestion & transformation
## From raw source to modelled table
Ingestion comes first. Streaming from Apache Kafka connects a broker as a source and keeps a table current from a topic. The IoT fleet-monitoring pipeline is the longer worked example, taking device messages through to dashboards in two parts, and is the closest thing here to a full reference architecture.
Transformation follows. The Python SDK tutorials show how a Custom action reads a table, changes it in a dataframe and writes it back. SQL transformation does the same work in SQL on the Trino engine. Data models with external datasets builds tables over data the platform never copies. Segmentation is the performance chapter: splitting one job into parallel tasks by date, attribute, file set or account, across seven short guides covering each mode.
Read them in order for a full pipeline, or pick the one that matches the source you have.
[### Stream data from Apache Kafka→
Keep a table current from a topic.
](/tutorials-kafka)[### Build an IoT fleet-monitoring pipeline→
Device messages through to dashboards, in two parts.
](/tutorials-iot-fleet-monitoring)[### Transform data using the Python SDK→
Read a table, change it in a dataframe, write it back.
](/tutorials-python-sdk-transform)[### SQL transformation→
The same work in SQL on the Trino engine.
](/tutorials-sql-transformation)[### Data models with external datasets→
Model over data the platform never copies.
](/tutorials-external-datasets)[### Apply segmentation to your jobs→
Split one job into parallel tasks, seven modes covered.
](/tutorials-segmentation)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-kafka.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Stream data from Apache Kafka to the Platform
## Objective
This tutorial demonstrates how to use the data from an Apache Kafka Broker in the platform.\
The first section is dedicated to creating some test data on your Kafka server. If you already have messages on your Broker, you can skip this step.
## Introduction
### Requirements
To follow this tutorial, you need to have a Kafka Broker up-and-running. The sample code provided was written for a server that did not require any particular authentication (i.e. anyone with the IP address can read messages). If your Kafka server has its authentication configured differently, you need to adapt the code used here to deal with that.
Additionally, we recommend having done at least the [first Getting Started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) before doing this one. Here we assume you are comfortable using Data Platform and familiar with the main components of the platform.
### Concepts overview
Data Platform integrates to Apache Kafka through the [Kafka Connector in the Connectors](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md). This connector allows to retrieve data from one or more topics in the same server to Data Platform.
Typically, data is then ingested into [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) tables. One [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) will store data from one topic and **each message in a topic becomes a line** of data in the corresponding table.
:::info
**About nested fields...** currently Data Platform only supports JSON-formatted messages without nesting. Therefore only fields that are at the root level of the JSON representation are taken into account.
:::
Once you've configured your connection to Kafka topics in [Connectors](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md) and setup your Lakehouse Manager tables, you will need to **load the data** from the topics to the tables by launching a [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) using [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md).
After loading the data, your messages will be automatically loaded into Lakehouse Manager tables as long as your action is running. Please note that your actions will be executed in [Serverless execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#serverless) by default, which has a timeout. That is why we recommend you to use the [Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up) if you are using the [Kafka Connector](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md).
Now let's see how all of this works in practice!
## Setting up test data (optional)
To send messages to your Kafka broker for test purposes, you can configure a Producer in Data Platform using a [DPE Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md). Create a Custom Action in the Data Processing Engine, select _start with a boilerplate_ and replace the boilerplate code by the one below (a simple name guessing game):
```python
from forepaas.dwh.connect import connect
import logging, time, json, random
from kafka import KafkaProducer # kafka-python
logger = logging.getLogger(__name__)
TOPIC = "sample"
# The next two lines MUST be replaced by your own Kafka address and port
KAFKA_PORT="9092"
KAFKA_ADDRESS=["10.152.1.186","10.152.1.187","10.152.7.65"]
def generate_bootstrap():
servers = [f"{x}:{KAFKA_PORT}" for x in KAFKA_ADDRESS]
return ",".join(servers)
def customfunc(event):
logger.info("Begin function customfunc")
i = 0
# Name guessing game:
# - Win 5 to 15 points for guessing the correct name
# - Lose 5 to 15 points for guessing the bad name
# - No points for guessing the other names
# generates list of names
nameslist = ["helene", "francoise", "lea", "lorene", "claire", "lise", "karen", "elise", "elia", "annabele"]
# select a "bad" and "correct" name
correct_name = nameslist[0]
bad_name = nameslist[1]
logger.info("Correct name:" + correct_name)
logger.info("Bad name:" + bad_name)
try:
producer = KafkaProducer(bootstrap_servers=generate_bootstrap())
while True:
i +=1
name = random.choice(nameslist)
# distinguish the bad and good names, against the rest
if name == correct_name:
points = random.randint(5,15)
elif name == bad_name:
points = random.randint(-15,-5)
else:
points = 0
value = {
"index":i,
"points":points,
"name":name,
}
# Kafka connection, and publish to broker
producer.send(TOPIC, json.dumps(value).encode("utf-8"))
if i % 1000 == 0:
logger.info("SENT 1000 records")
# Sleep for 1 second every 1000 messages
time.sleep(1)
if i ==10000:
return
logger.info("END function customfunc")
except Exception as err:
logger.critical(err)
```
The code above represents a name guessing game, it will simply send a messages
representing guesses. Each message contains an index, a name (the guess) and the points gained for the guess. Once 10 0000 messages have been sent, the action will stop and you should have some messages in your Broker.
:::warning
Do not forget to add the `kafka` module to the Python dependencies of your Custom action.
:::
Now, run the action to populate your topic with the test data and stop the action running once a few thousand records have been sent.
## Connecting your Kafka server to Data Platform
### Configure your Kafka connection
The first thing to do is configure your connection to a Kafka server and choose a topic to read data from. If you need help, you can check our dedicated [Apache Kafka Connector article](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md).
### Configure your schema
Now that your connection and topic are properly configured, you are ready to access your messages. To do it, you need to go to the [Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) tab and **extract the metadata** of the topic in your connection.
The connection will appear on the left sidebar and the topic will be displayed upon clicking on the connection. Select the topic and click on the _Extract metadata_ button.
After the metadata extraction, the messages will appear in the preview panel where each line corresponds to a message. Check the boxes on the metadata panel to configure which fields of the message will be included when you use your message in Data Platform.
:::info
You may notice that there are **additional fields** in your message. The timestamp, date as well as the offset are given by the Broker and correspond to your messages arrival timestamp, arrival date and topic offset respectively. They can be useful for some use cases but, if you don't want to include them in your data Project, simply uncheck them in the Metadata panel and they will be ignored by the rest of the platform.
:::
### Create and build your table
Before loading your data into Data Platform, you need to create and build the table that will store it. If you are not familiar with these concepts you can check our [Tables article](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md). You will probably want to read the [Create a new table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#create-a-table) and [Build all tables](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#build-table) sections.
## Loading your data into Data Platform
### Configuring the Load Action
In comparison to other connectors, there are a few differences when creating a [Load Action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) with a streaming source such as Apache Kafka.
To get started, select the table related to your topic as _Source_ when configuring the action (if you already generated the action while creating the table then you will not need to select the table).
### Execution modes
When running a [Load Action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) connected to a Kafka source we strongly recommend you to select the [Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up).
Regardless of the Execution Mode you use, your action will run until some data arrives. Once it does, it will be loaded into Data Platform.
If you are using the [Serverless execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#serverless), the action will stop running once [timeout](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#timeout-options) is reached (by default 2 hours). Given that new data can arrive anytime in your Kafka Broker, this means that you need to re-run this action after it stopped if you want to keep feeding data from your Kafka to Data Platform. That is why **we recommend the use of the [Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up)**, specially for production environments.
:::info
If you don't want to use the [Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up), you can alternatively set [Time-based triggers](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#time-based-trigger) to automatically run your action according to preset time interval.
:::
### Automatic segmentation
When using a Kafka connector, you can benefit from a faster execution time by using the [Automatic Segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md#7-automatic-segmentation) feature. This option is available in the [Preferences of your action](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md), use it for heavy workloads!
### Custom offset
One other option available to you is to start reading your messages from a custom offset instead of [configuring your topics](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md#configuring-your-topics) to reading from the earliest or latest message.
To override the _Latest_ or _Earliest_ offset policy you configured on your topic and start reading messages from a defined offset you have to use the Actions' [Advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#advanced-mode). Simply add a field inside the `paras.load_from` field as the following example (starts reading from offset 7):
```json
"params": {
"load_from": [
{
"offset_number": 7,
...
}
],
...
```
:::info
Note that this will override the _Latest_ or _Earliest_ offset policy you configured on your topic.
:::
### Technical considerations
Another consideration you should be aware of is that, if you are on _Earliest_ mode and you change your destination table after reading the earliest messages, upon a running the Load action again, all messages in the topic will be read again.
This happens because the offset of the last message consumed by Data Platform is stored in the metadata of the destination table of the Data Platform database. If you use a new destination table, the offset will start back from the earliest.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-iot-fleet-monitoring.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build an IoT fleet-monitoring pipeline
## Objective
This tutorial walks you through a complete, real-time pipeline on the Data Platform: streaming simulated IoT sensor telemetry from a Custom action into Apache Kafka, ingesting it into a Lakehouse Manager table, and visualizing fleet health on a live Superset dashboard.
By the end you will have a working dashboard that shows device health, environmental trends, anomalies, and downtime, driven by a scripted failure scenario so the data is always telling a story.
## Introduction
### Requirements
To follow this tutorial you need:
- An OVHcloud Data Platform project with Data Processing Engine, Lakehouse Manager, and a Superset/Trino setup available.
- An **Apache Kafka** broker you can write to. This tutorial uses an OVH managed Kafka service with client-certificate (mTLS) authentication, but the approach works with any broker once you adapt the connection settings.
- A Kafka topic named `iot_readings` created on your broker (topic auto-creation is often disabled).
We recommend completing the [first Getting Started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) and the [Stream data from Apache Kafka](https://docs.dataplatform.ovh.net/tutorials-kafka.md) tutorial first. This guide assumes you are comfortable with the main components of the platform.
### What you will build
```mermaid
flowchart TD
P[DPE Custom action: producer] --> K[Apache Kafka topic: iot_readings]
K --> C[Kafka source connector]
C --> L[DPE Load action: Always-up]
L --> T[Lakehouse Manager table: iot_readings]
T --> TR[Trino]
TR --> S[Superset dashboards]
classDef process fill:#E6F3FA,stroke:#0089C0,color:#0a2a36;
classDef store fill:#0089C0,stroke:#0089C0,color:#fff;
class P,C,L,TR,S process;
class K,T store;
```
| Stage | Component | Role |
| --------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Produce | [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) | Simulates sensors, emits flat-JSON readings to Kafka every 5 seconds |
| Transport | Apache Kafka | Single topic `iot_readings` |
| Ingest | [Kafka source connector](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md) | Streams the topic into a table |
| Store | [Lakehouse Manager table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) `iot_readings` | One message becomes one row |
| Load | [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) | Consumes the stream into the queryable table |
| Visualize | Superset over [Trino](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) | Live dashboards |
### The scenario
The producer simulates a fleet of environmental sensors (temperature, humidity, CO₂, PM2.5) across four sites in three regions. A per-device state machine drives a realistic failure lifecycle so the data is never flat:
```mermaid
stateDiagram-v2
[*] --> OK
OK --> DEGRADED: scripted trigger or battery < 15%
DEGRADED --> FAULT: after 120s
FAULT --> OFFLINE: after 90s, stops emitting
OFFLINE --> OK: repaired after 60s
```
Two devices are scripted to fail on a fixed schedule, so every run reliably shows the full story. Readings follow a compressed ten-minute "daily cycle" with random noise, so trends look believable. When a device goes `OFFLINE` it stops emitting entirely, which shows up later as a genuine gap in the data.
:::info
**About message format.** The Kafka connector reads only root-level JSON fields, and it infers the table schema by sampling messages. Every message therefore emits every field with stable types (numerics are always floats, `error_code` is the string `"NONE"` when healthy). A field that is sometimes missing, or sometimes an integer and sometimes a float, would break the inferred schema.
:::
## Step 1: Store your Kafka credentials in a bucket
Rather than pasting certificates or passwords into the action source, store them in a [Lakehouse Manager bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) and pull them at runtime. A Custom action is self-authenticated to its project, so it can read the bucket without any extra credentials, and nothing sensitive lives in your code.
1. In Lakehouse Manager, create a bucket named `iot-demo-certs`.
2. Upload your three mTLS files into it:
| File | Role | Producer setting |
| ---------------------- | ------------------ | ---------------- |
| `certificate.txt` | CA certificate | `ssl_cafile` |
| `user-certificate.txt` | client certificate | `ssl_certfile` |
| `user-access-key.txt` | client key | `ssl_keyfile` |
The producer downloads these at startup using the [bucket connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-bucket.md) in the SDK.
:::warning
Never commit certificate or key files to a repository, and never paste them into the action source. Keep them in the bucket only.
:::
## Step 2: Create the producer Custom action
Download the producer and add it as a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md):
:::info
[producer.py
](/images/public-cloud/data-platform/getting-further/iot-fleet-monitoring/resources/producer.py)
:::
1. Go to **Data Processing Engine > Actions > New > Custom**.
2. Upload the downloaded `producer.py` directly.
3. Set the entry function to `customfunc` in the action's Information panel.
4. Add **`kafka-python`** to the action's Python Requirements.
5. Set the execution mode to [Always-up](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up). The producer loops continuously, so [Serverless mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#serverless) would time out.
:::warning
Use `kafka-python`, not `kafka`. The bare `kafka` package on PyPI is an abandoned Python-2-only distribution and fails on the workers with `invalid syntax (simple.py, line 54)`. The `kafka-python` package is what provides the `from kafka import ...` namespace.
:::
### Parameters you can adjust
All configuration lives at the top of the file. The ones you are most likely to change:
| Parameter | Default | What it controls |
| ------------------------------ | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `BOOTSTRAP` | placeholder | Your Kafka broker endpoint, as `host:port`. Required. |
| `KAFKA_MODE` | `"ssl"` | Auth method: `"ssl"` for mTLS, `"sasl_ssl"` for SASL/SCRAM, or `"noauth"`. |
| `TOPIC` | `"iot_readings"` | The Kafka topic to produce to. |
| `MAX_RUNTIME_SECS` | `None` | `None` runs forever (live dashboard). Set a number of seconds for a finite run that exits with SUCCESS instead of a manual "stopped". |
| `CERT_BUCKET` | `"iot-demo-certs"` | The Lakehouse bucket holding your cert files from Step 1. |
| `SITES` and `DEVICES_PER_SITE` | 4 sites, 3 each | Size and shape of the simulated fleet. |
| `METRIC_PROFILE` | temperature, humidity, co2, pm25 | Baseline, daily swing, and noise for each metric. |
| `PHASE_*` durations | 120 / 90 / 60 / 90 s | How long each failure phase lasts. |
| `time.sleep(5)` in the loop | 5 s | Emit interval per device. Lower it for a faster stream. |
### Code highlights
You do not need to read the whole file to run it, but a few parts are worth knowing.
**Connection block.** Set your broker endpoint and auth method here, and choose whether the run is finite:
```python
KAFKA_MODE = "ssl"
TOPIC = "iot_readings"
BOOTSTRAP = ":"
MAX_RUNTIME_SECS = None # set seconds for a finite, SUCCESS run
```
**Credentials from a bucket.** The certificates are pulled at runtime, never hard-coded, so nothing sensitive lives in the action:
```python
CERT_SOURCE = "bucket"
CERT_BUCKET = "iot-demo-certs"
# resolve_certs() downloads the CA, client cert, and key to /tmp via the SDK
```
**The fleet.** Change the sites or the per-site count to resize the simulation:
```python
SITES = [
("paris-dc1", "EU-W", "indoor-air"),
("london-dc2", "EU-W", "indoor-air"),
# ...
]
DEVICES_PER_SITE = 3 # 4 sites x 3 -> 12 devices
```
**The failure scenario.** Two devices fail on a fixed schedule, and these durations set the pace of the OK to DEGRADED to FAULT to OFFLINE story:
```python
PHASE_DEGRADED, PHASE_FAULT, PHASE_OFFLINE, PHASE_REPAIRED_COOLDOWN = 120, 90, 60, 90
```
Run the action. Every message carries all twelve fields with stable types, so the schema is complete as soon as any data flows. The first scripted device degrades after about fifteen seconds, so `DEGRADED` and `FAULT` samples appear quickly. Watch the action logs for the heartbeat line: `sent ~N messages; fleet: {...}`.
## Step 3: Connect Kafka with the source connector
Go to **Connectors > Sources > Apache Kafka** and create a connection. Point it at your broker endpoint and the `iot_readings` topic.
For an mTLS broker, authenticate with the connector's **SSL fields**: upload your CA certificate, client certificate, and client key, and leave username and password blank.
:::info
The connector form exposes SSL fields even when the broker uses client certificates. Those are what make the mTLS handshake work. For full details see the [Kafka connector reference](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md).
:::
## Step 4: Extract metadata and create the table
Once data is flowing, open the [Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) and [extract the metadata](https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md).
You will see sixteen attributes: the twelve fields from the producer, plus four envelope columns added by Kafka (`timestamp`, `date`, `offset_r`, `partition`). For a simple append-only table you can skip all four envelope columns. Keep `partition` and `offset_r` only if you want upsert or deduplication semantics.
The producer emits `ts` as an ISO-8601 string. If it is inferred as a string, you can either set it to a Timestamp type in the table, or keep it as a string and parse it in Trino (see [Good to know](#good-to-know)). Build the [Lakehouse Manager table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) `iot_readings` from the resulting schema.
The complete column list:
| Attribute | Type | Notes |
| ------------- | --------- | ------------------------------------------------------ |
| `ts` | Timestamp | ISO-8601 UTC event time |
| `device_id` | String | for example `paris-dc1-sensor-01` |
| `site` | String | denormalized dimension |
| `region` | String | `EU-W`, `EU-C`, `EU-E` |
| `device_type` | String | `indoor-air` or `outdoor-air` |
| `temperature` | Number | °C |
| `humidity` | Number | % |
| `co2` | Number | ppm |
| `pm25` | Number | µg/m³ |
| `battery_pct` | Number | declines over device life, drives failure |
| `status` | String | `OK`, `DEGRADED`, `FAULT` (never `OFFLINE`, see below) |
| `error_code` | String | `NONE`, `E_DRIFT`, `E_BATTERY`, `E_STUCK`, `E_SPIKE` |
:::info
`OFFLINE` never appears as a row value. An offline device stops emitting, so downtime shows up as a gap (no rows for that `device_id`). You detect it by comparing each device's latest `ts` against the current time.
:::
## Step 5: Load the stream into your table
Create a [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) in Data Processing Engine, mapping the `iot_readings` topic to your `iot_readings` table. Run it in [Always-up mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up) so it keeps consuming the live stream.
:::warning
The Load action keeps running as long as data exists in Kafka, up to its timeout. That is expected for a streaming load.
:::
A Load action that completes normally runs a metadata update automatically, so the table's row count refreshes on its own. In this streaming setup the action runs Always-up and is stopped manually or times out rather than ending cleanly, so that automatic update may not run. When that happens, run an [Update Metadata action](https://docs.dataplatform.ovh.net/dpe-actions-maintenance.md#update-metadata) so the row count in Lakehouse Manager reflects the data that was ingested.
## Step 6: Query the table from Superset
Connect Superset to your table through [Trino](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md). If you have not deployed Superset yet, follow [Deploy Apache Superset](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md).
In Superset, add a database connection with a SQLAlchemy URI of the form `trino://@:/`. Test the connection before building charts. This is the step where end-to-end setups most often break.
If `ts` arrived as a string, parse it in Trino with `from_iso8601_timestamp(ts)`, which returns a `TIMESTAMP(3) WITH TIME ZONE`. Create a virtual dataset that exposes the parsed column and mark it as the dataset's main temporal column, so it becomes available as the X-axis of time-series charts.
## Step 7: Build the dashboards
The finished dashboard is a single-screen view of the whole fleet.
A practical set of panels:
| Panel | Chart type | What it shows |
| -------------------- | ------------------ | ------------------------------------------------------------- |
| KPI row | Big Number | Devices online, fleet size, devices in alert, average CO₂ now |
| Fleet health | Donut | Live status split across the fleet |
| Battery levels | Bar | Per-device battery, lowest first, for predictive maintenance |
| Readings trend | Line (time-series) | CO₂, temperature, PM2.5 over time, per site |
| Readings by site | Bar | Average CO₂ and PM2.5 compared across sites |
| Error-code breakdown | Bar | Which fault types are occurring |
| Downtime table | Table | Devices whose latest reading is stale, highlighted |
For the full virtual datasets and the exact Trino SQL and configuration of every panel, follow the companion page:
[Build the Superset dashboards](https://docs.dataplatform.ovh.net/tutorials-iot-fleet-monitoring-superset-dashboards.md)
A few Superset tips that save time:
- On **bar charts**, put the category in the X-axis and leave the Dimensions box empty. Dimensions is only for splitting each bar into sub-series.
- Threshold lines, such as a battery warning level, are available on time-series charts through annotation layers. On a categorical bar chart, sort ascending instead. On a table, use conditional formatting.
- CO₂ values (around 480) dwarf temperature (around 21) and PM2.5 (around 9) on a shared axis. Chart CO₂ separately, or use a secondary axis, so the small metrics stay readable.
- The downtime table is empty when the fleet is healthy, which is correct. Build it without a filter so it lists every device, then highlight stale rows with conditional formatting, so the panel never looks broken.
## Good to know
- **The producer never ends by design.** It is an infinite loop in Always-up mode. The action's default timeout is around two hours. If you stop it manually it shows as "stopped" rather than a success, because there is no natural end. To get a clean success run, set `MAX_RUNTIME_SECS` to a number of seconds. The loop then flushes, closes, logs a total, and exits. Leave it as `None` for a continuously live dashboard.
- **Refresh the row count with Update Metadata.** A Load action that ends normally updates the table metadata automatically. In this streaming setup the action runs Always-up and is stopped or times out instead of ending cleanly, so as noted in Step 5 you may need to run an Update Metadata action for the row count to reflect what was ingested.
- **Parse `ts` for time-series.** If the column is a string, parse it with `from_iso8601_timestamp(ts)` and mark it as the temporal column in Superset.
- **Guard against unparseable rows.** If a test message left a row whose `ts` is not a valid timestamp, `from_iso8601_timestamp` throws `INVALID_FUNCTION_ARGUMENT`. Wrap the parse in `try(from_iso8601_timestamp(ts))` so the bad row resolves to null, or delete the row.
## What you have built
You now have a full streaming pipeline: a Custom action producing simulated telemetry, Kafka transporting it, a connector and Load action ingesting it into Lakehouse Manager, and Superset visualizing it live over Trino. As the scripted failures play out, the dashboard moves through the whole story: a healthy fleet, a device degrading and faulting, a window of downtime that surfaces in the downtime table and as a gap in the trend, and finally a repair that returns the device to normal.
From here you can extend the model with a separate device dimension table, add more metrics, or adapt the producer to replay your own real sensor data.
:::info
**This pattern scales.** The same building blocks compose into a much larger pipeline. Because one topic maps to one table, you handle multiple data streams by repeating the pieces: add more Kafka topics, extract metadata for each into its own Lakehouse Manager table, and run one Load action per topic-to-table mapping. A single producer can emit to several topics, and Superset can join across the resulting tables through Trino. So a multi-topic, multi-table fleet (for example, separate streams for environmental readings, energy usage, and device events) is just this tutorial applied several times over, feeding one set of dashboards.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-iot-fleet-monitoring-superset-dashboards.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build the Superset dashboards
## Objective
This page is the detailed companion to Step 7 of the [IoT fleet-monitoring pipeline](https://docs.dataplatform.ovh.net/tutorials-iot-fleet-monitoring.md) tutorial. It gives the virtual datasets and the exact chart configuration for every panel of the fleet-monitoring dashboard.
Superset queries the Lakehouse through [Trino](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), so all SQL on this page is in the **Trino dialect**. If you have not deployed Superset yet, follow [Deploy Apache Superset](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) first.
The finished dashboard you will build:
:::info
**About `ts` parsing.** The producer emits `ts` as an ISO-8601 string. If the table column is `VARCHAR`, every query parses it with `from_iso8601_timestamp(ts)`. If you set the column to a real `TIMESTAMP` in Lakehouse Manager, you can drop the parse and use `ts` directly. Either way, mark the parsed temporal column as the dataset's main temporal column in Superset so time filters and time-series charts work.
:::
:::info
**Guard against unparseable rows.** If a test message left a record whose `ts` is not a valid timestamp (for example `device_id = 'local-test'`), `from_iso8601_timestamp` throws `INVALID_FUNCTION_ARGUMENT: Invalid format`. The queries below wrap the parse in `try()` so that row resolves to `null` and is filtered out. The clean fix is to purge it once with `DELETE FROM iot_readings WHERE device_id = 'local-test'`, after which the `try()` wrappers are optional.
:::
## Setup
1. **Connect Trino as a database** in Superset under **Settings > Database Connections > +**. The SQLAlchemy URI has the form `trino://@:/`. Test the connection before going further.
2. **Add the physical dataset** `iot_readings` under **Datasets > +**, picking the catalog, schema, and table.
3. **Create the virtual datasets** below in SQL Lab (run the query, then **Save > Save dataset**). The "current state per device" logic is reused by several charts, so save it once as `iot_current_state`.
## The iot\_current\_state virtual dataset
This returns the latest row per device, which several panels build on. Save it as `iot_current_state`.
```sql
SELECT *
FROM (
SELECT
device_id, site, region, device_type,
temperature, humidity, co2, pm25,
battery_pct, status, error_code,
ts_parsed AS ts,
row_number() OVER (
PARTITION BY device_id
ORDER BY ts_parsed DESC
) AS rn
FROM (
SELECT *, try(from_iso8601_timestamp(ts)) AS ts_parsed
FROM iot_readings
)
WHERE ts_parsed IS NOT NULL
)
WHERE rn = 1
```
The inner `try()` plus `WHERE ts_parsed IS NOT NULL` drops any unparseable row, which also prevents a bad row from winning the `row_number()` ordering and being treated as a device's current state.
## Panel 1: KPI row
Four headline Big Number charts across the top of the dashboard.
| Chart | Type | Dataset | Metric |
| ---------------- | ---------- | ------------------- | ------------------------------------------- |
| Devices online | Big Number | virtual (below) | distinct devices seen in the last 2 minutes |
| Fleet size | Big Number | `iot_current_state` | `COUNT(DISTINCT device_id)` |
| Devices in alert | Big Number | `iot_current_state` | `COUNT_IF(status <> 'OK')` |
| Avg CO₂ now | Big Number | `iot_current_state` | `AVG(co2)` |
Save the **Devices online** virtual dataset:
```sql
SELECT count(DISTINCT device_id) AS devices_online
FROM iot_readings
WHERE try(from_iso8601_timestamp(ts)) >= current_timestamp - interval '2' minute
```
:::info
"Online" means reported in the last 2 minutes. This is how an offline device drops out of the count. There is no `OFFLINE` row to count, only an absence of recent rows.
:::
## Panel 2: Fleet health donut
The current status distribution across the fleet, one slice per device rather than per row.
- Chart type: **Pie / Donut**
- Dataset: `iot_current_state`
- Dimension: `status`. Metric: `COUNT(DISTINCT device_id)`
- Suggested colors: OK green, DEGRADED amber, FAULT red. `OFFLINE` rarely shows here because offline devices stop emitting, so they fall out of the current state. See [Panel 5](#panel-5-downtime-table).
## Panel 3: Battery levels
The "you could have seen it coming" chart, sorted so the most at-risk devices are first.
- Chart type: **Bar Chart**. Bar Orientation **Horizontal** under Customize.
- Dataset: `iot_current_state`
- **X-axis**: `device_id`. **Metrics**: `MAX(battery_pct)`. Leave **Dimensions** empty.
- Sort by `MAX(battery_pct)` ascending, row limit 15.
:::warning
A threshold line at the 15% failure trigger is **not** available on a categorical bar chart. Annotation layers exist only on time-series charts. To convey the threshold, either rely on the ascending sort so the lowest batteries are obvious, or build this as a **Table** instead and use **Customize > Conditional formatting** to colour `MAX(battery_pct)` red below 15.
:::
:::info
On bar charts, put the category in **X-axis** and leave the **Dimensions** box empty. Dimensions is only for splitting each bar into sub-series, which you do not want here.
:::
## Panel 4: Readings trend
The headline trend: average metrics over time, sliceable by site or region.
- Chart type: **Line Chart** (time-series)
- Dataset: a dataset whose temporal column is the parsed `ts` (see the parsing note at the top). Create a virtual dataset that exposes `try(from_iso8601_timestamp(ts)) AS ts` and mark it as the main temporal column.
- Time grain: 1 minute (or 30 seconds)
- **X-axis**: `ts`. **Metrics**: `AVG(co2)`, `AVG(pm25)`, `AVG(temperature)`. **Dimensions**: `site` to get one line per site.
- Add dashboard filters on `region` and `device_type`.
:::warning
CO₂ values (around 480) dwarf temperature (around 21) and PM2.5 (around 9) on a shared axis. For all three to be readable, chart CO₂ separately or put the smaller metrics on a secondary Y-axis.
:::
For a per-device anomaly close-up that shows the FAULT stuck-at values and the PM2.5 spike clearly, duplicate this chart, set the series dimension to `device_id`, and filter to one of the scripted devices:
```sql
SELECT
try(from_iso8601_timestamp(ts)) AS ts,
device_id, pm25, status
FROM iot_readings
WHERE device_id = 'paris-dc1-sensor-01'
ORDER BY ts
```
## Panel 5: Downtime table
Detects true downtime: a device whose latest reading is stale because it stopped emitting.
- Chart type: **Table**
- Dataset: the virtual dataset below
Save this dataset. It returns every device with the time since its last reading, so the panel is never empty:
```sql
SELECT
device_id,
site,
region,
max(try(from_iso8601_timestamp(ts))) AS last_seen,
date_diff('second', max(try(from_iso8601_timestamp(ts))), current_timestamp) AS seconds_since_last
FROM iot_readings
WHERE device_id <> 'local-test'
GROUP BY device_id, site, region
ORDER BY seconds_since_last DESC
```
Then in the Table chart, use **Customize > Conditional formatting** to colour `seconds_since_last` red when it is greater than 30.
:::info
A threshold of 30 seconds assumes the 5 second emit interval, so a device missing about 6 cycles is considered down. Tune it to taste. If you would rather list only down devices, add `HAVING date_diff('second', max(try(from_iso8601_timestamp(ts))), current_timestamp) > 30` to the query. Note that this version is empty when the fleet is healthy, which can look broken in a live demo.
:::
## Panel 6: Error-code breakdown
Which kinds of faults are occurring over a window.
- Chart type: **Bar Chart**
- Dataset: `iot_readings`
- **X-axis**: `error_code`. **Metrics**: `COUNT(*)`. Leave **Dimensions** empty.
- Filter: `error_code <> 'NONE'` and time range set to the last 15 minutes.
## Panel 7: Readings by site
Compare environmental readings across the fleet.
- Chart type: **Bar Chart**
- Dataset: `iot_current_state`
- **X-axis**: `site` (or `region`). **Metrics**: `AVG(co2)`, `AVG(pm25)`. Leave **Dimensions** empty.
:::warning
As in Panel 4, CO₂ dwarfs PM2.5 on a shared axis. Chart them separately or use a secondary Y-axis if you need both readable.
:::
## Dashboard layout
A practical arrangement of the seven panels:
```
+-----------------------------------------------------------------+
| Devices online | Fleet size | Devices in alert | Avg CO2 (now) | KPI row
+------------------------------+----------------------------------+
| Fleet health donut | Battery levels |
+------------------------------+----------------------------------+
| Readings trend over time, by site |
+------------------------------+----------------------------------+
| Downtime table | Error-code breakdown |
+------------------------------+----------------------------------+
```
Add dashboard filters for `region`, `site`, `device_type`, and a time range. Set the dashboard auto-refresh interval to 30 seconds (under **Edit dashboard**) for a live feel.
As the scripted failures play out, the dashboard moves through the whole story: a healthy fleet, a device degrading and faulting, a window of downtime that surfaces in the downtime table and as a gap in the trend, and finally a repair that returns the device to normal.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-python-sdk-transform.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Transform data from sources using the Python SDK
## Objective
We have prepared detailed guides and sample scripts for common use cases to help you get started quickly and efficiently. These examples demonstrate how to leverage **Custom Actions** in different scenarios, allowing you to extract, transform, and load (ETL) data across various components of your data platform.
## 1. Custom Action with a Lakehouse Manager Table
### Overview:
This short example shows how to extract data from a table in Lakehouse Manager, transform it and then insert it or update the Lakehouse Manager.
The code is written in [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) context and it uses the _stations\_rides_ table from the [Getting Started](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) Tutorial. If you did that tutorial, you can just copy and paste the code below to test it, otherwise you need to adapt it to your tables and data sources.
### Example Application:
:::warning
Do not forget to build the table you use in the Lakehouse Manager and then load it with a [DPE Load Action](https://docs.dataplatform.ovh.net/dpe-actions-load.md). You need to load your table before using the code below, otherwise it will not work.
:::
```python
import sys
import pandas as pd
import logging
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
logger = logging.getLogger(__name__)
def customfunc(event):
try:
logger.notice("Begin function")
# make connection to the default dataset
cn = connect("dwh/default_dataset/")
# option 1 : extract data from the table with no SQL required
df = cn.select("stations_rides")
# option 2 : extract data with custom SQL
df = cn.query("SELECT station_id, date, rides, station_name FROM stations_rides")
# perform your custom transform in the dataframe
df.loc[df["station_name"] == 'Harlem-Lake', "rides"] = 0
# reinsert your dataframe in the destination table
stats = bulk_insert(cn, "stations_rides", df)
# show insertion statistics (if DBMS compatible)
logger.info(stats)
# delete rows where station name is "Davis"
cn.delete("stations_rides", {"station_name":"Davis"})
# update rows set rides to 0 where station_id=40040
cn.update("stations_rides", {"rides":0}, {"station_id":40040})
# when finished, disconnect cn
del cn
logger.notice("END function")
except Exception as err:
raise Exception("err:{} L:{}".format(err,sys.exc_info()[2].tb_lineno))
```
## 2. Custom Action with Data Platform Buckets
### Overview:
Sometimes you need to handle a complex file format beyond our [Load Action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) capabilities.
In this case, we advise you to store and manipulate files with the [Data Platform Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) in your Project.
:::info
In the current Data Platform SDK, the **Datastore connector** is used to interact with the Data Platform Buckets. You may think of the Datastore simply as a bucket container.
:::
### Example Applications:
```python
import sys
import pandas as pd
from logging import getLogger
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
logger = getLogger(__name__)
def extract_func(event):
try:
# we get data from a bucket and we will archive them in another bucket
bucket_source_name = "your_source_bucket_name_here"
bucket_archives_name = "your_source_bucket_name_here"
# create a connector to handle bucket
bucket_connector = connect("data_store/{}".format(bucket_source_name))
# list files from bucket
files = bucket_connector.list()
# retrieve a file from Data Store bucket to temporary local folder
bucket_filepath = "stations_rides.csv"
local_filepath = "/tmp/stations_rides.csv"
bucket_connector.fget(bucket_filepath, local_filepath)
# read then transform the file as you need
# here the date column format is simply adjusted for compatibility reasons
df = pd.read_csv(local_filepath, sep=';')
df['date'] = pd.to_datetime(df['date'])
# load the dataframe into a project table named 'raw_file'
cn = connect("dwh/default_dataset/")
bulk_insert(cn, "stations_rides_artur", df)
del cn
# option 1 : copy the file into the archives bucket
bucket_archive_filepath = "archives/stations_rides.csv"
bucket_connector.fcopy_to(bucket_archives_name, bucket_archive_filepath, bucket_filepath)
# option 2 : put a file into the archives
bucket_archives = connect("data_store/{}".format(bucket_archives_name))
bucket_archives.fput(bucket_archive_filepath, local_filepath)
del bucket_archives
# delete file from source bucket
bucket_connector.delete(bucket_filepath)
# disconnect from datastore
del bucket_connector
except Exception as err:
raise Exception("err:{} L:{}".format(err,sys.exc_info()[2].tb_lineno))
```
Below is an example code which uploads an image to a bucket from a simple URL.
```python
from forepaas.dwh import connect
data_store = connect('data_store')
# Get bucket and upload image from URL to path uploads/test.jpg.
# And finally get the image from the bucket
bucket_test = data_store.get_bucket('test')
lists = bucket_test.list(recursive=True)
bucket_test.put_request("https://i.stack.imgur.com/r8jTK.jpg", path='uploads/test.jpg')
data = bucket.get('hello/test.jpg')
# Create a bucket if it does not already exists
if data_store.bucket_exists('test-exists') is False:
data_store.create_bucket('test-exists')
# Connect directly to the bucket test and remove the file
bucket_test2 = connect('data_store/test')
bucket_test2.delete('hello/test.jpg')
```
:::tip
This also works with any **Object Store** that you may define as **S31 compatible source** in the Connectors.
:::
## 3. Custom Action with a Connectors Source
### Overview:
This example will show you how to get a file directly from a [Connectors Source](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md), treat it and put the treated data into a [Lakehouse Manager Table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
The code is written in the [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) context and it uses the _chicago\_files_ source from the [Getting Started Tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). If you did that tutorial, you can just copy and paste the code below to test it, otherwise you need to adapt it to your tables and data sources.
:::tip
This works with any `Source` protocol such as [FTP](https://docs.dataplatform.ovh.net/connectors-sources-ftp.md), [Dropbox](https://docs.dataplatform.ovh.net/connectors-sources-dropbox.md), etc.
:::
### Example Applications:
```python
import sys
from forepaas.dwh.connect import connect
from forepaas.dwh import bulk_insert
import logging
logger = logging.getLogger(__name__)
def customfunc(event):
try:
# here we are connecting to a source named 'chicago_files'
source_address = "dwh/chicago_files_artur/"
# specify unsupported filename from list of files in source
filename_w_extension = "stations_rides.csv"
# connecto to file toget the address
source_file_connector = connect(source_address + filename_w_extension)
# connect to file directly
file_address = source_file_connector.get()
file_connector = connect(file_address)
# Extract and treat the file so it is usable
df = file_connector.extract(return_type='dataframe')
# Treat the data
# - - - -
# connect to the Lakehouse Manager
dm_connector = connect("dwh/default_dataset/")
# insert into an existing destination table
stats = bulk_insert(dm_connector, "chicago_calendar_full", df)
logger.info(stats)
# disconnect from datastore and remote source
del source_connector
del dm_connector
except Exception as err:
raise Exception(f"err:{err} L:{sys.exc_info()[2].tb_lineno}")
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-sql-transformation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# SQL Transformations within the OVHcloud Data Platform
## Objective
Welcome to the guide on performing **SQL Transformations within the OVHcloud Data Platform**. SQL (Structured Query Language) remains the most widely used and efficient language for querying and transforming structured data. This document will introduce you to the core concepts of using SQL for data manipulation and aggregation directly within our platform, leveraging its powerful capabilities for data processing.
Whether you're looking to clean, reshape, filter, or aggregate large datasets, SQL provides a robust and intuitive way to achieve your data transformation goals. Its declarative nature allows you to focus on what you want to achieve with your data, rather than how the operations are performed, making it accessible to a wide range of data professionals.
## Why SQL for Data Transformation?
SQL is an indispensable tool in the data transformation pipeline for several key reasons:
- **Universality:** It's a standard language, highly adopted across various databases and data platforms, making skills easily transferable.
- **Readability & Simplicity:** Its English-like syntax makes complex operations relatively easy to understand and write.
- **Performance:** SQL engines are highly optimized for relational operations, often outperforming custom code for large-scale data manipulation.
- **Declarative Nature:** You specify the desired end-state of your data, and the engine determines the most efficient way to achieve it.
- **Integration:** Seamlessly integrates with data warehousing, business intelligence, and reporting tools.
## Core Concepts of SQL Transformations
At its heart, SQL transformation involves using standard SQL commands to manipulate data. This can include:
### 1. Data Cleaning and Preparation
- **Filtering:** Using `WHERE` clauses to select specific rows based on conditions.
- **Selecting/Projecting:** Using `SELECT` statements to choose specific columns and rename them (`AS`).
- **Type Conversion:** Using `CAST` or `TRY_CAST` functions to change data types. `TRY_CAST` is particularly useful in Trino for gracefully handling conversion errors (returning `NULL` instead of crashing).
- **Handling Missing Values:** Using `COALESCE` or `CASE` statements to replace `NULL` values.
- **String Manipulation:** Functions like `SUBSTRING`, `LENGTH`, `UPPER`, `LOWER`, `TRIM` to clean and standardize text data.
### 2. Data Aggregation and Summarization
- **Grouping:** Using `GROUP BY` to aggregate rows that have the same values in specified columns.
- **Aggregate Functions:** Applying functions like `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` to summarized groups.
- **Filtering Aggregations:** Using `HAVING` clauses to filter results of `GROUP BY` operations.
### 3. Data Reshaping and Restructuring
- **Joins:** Combining data from two or more tables based on related columns (`INNER JOIN`, `LEFT JOIN`, `RIGHT JOIN`, `FULL OUTER JOIN`).
- **Unions:** Combining the result sets of two or more `SELECT` statements (`UNION`, `UNION ALL`).
- **Pivoting/Unpivoting:** Transforming rows into columns (pivot) or columns into rows (unpivot) to change the data's structure. In Trino SQL (commonly used in DPE), pivoting is often achieved using `SUM` with `FILTER` or `CASE` statements, and unpivoting with `CROSS JOIN UNNEST`.
- **Window Functions:** Performing calculations across a set of table rows that are related to the current row, without collapsing rows (`ROW_NUMBER()`, `RANK()`, `LEAD()`, `LAG()`, `SUM() OVER()`, `AVG() OVER()`).
## SQL Transformations on OVHcloud Data Platform (Practical Guide)
This section provides a practical, end-to-end example of performing SQL transformations within the OVHcloud Data Platform's DPE (Data Processing Environment) Notebooks, specifically using the SDK and focusing on the **dirty\_cafe\_sales** dataset.
### Prerequisites
1. Download the dirty\_cafe\_sales.csv
2. Upload it to the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md) and extract the metadata using the [Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md).
3. Create a new [Table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) from source in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)
**Dataset Information:**
We will be working with a simulated cafe sales dataset named `dirty_cafe_sales`. It contains the following columns and known data quality issues:
| Column Name | Description | Known Issues |
| :----------------- | :-------------------------------------------------- | :-------------------------------------------------------------------------- |
| `transaction_id` | Unique identifier for each transaction | None |
| `item` | Name of the item sold | None |
| `quantity` | Number of units sold in the transaction | Contains 'UNKNOWN' strings |
| `price_per_unit` | Price of a single unit of the item | None |
| `total_spent` | Total amount spent for the item line in transaction | Contains 'ERROR' strings |
| `payment_method` | Method used for payment (e.g., Credit Card, Cash) | None |
| `location` | Sales location (e.g., In-store, Takeaway) | Contains 'ERROR' strings and empty strings `''` |
| `transaction_date` | Date of the transaction | Contains non-date strings (e.g., 'ERROR'), needs casting and error handling |
### 1. Create a New DPE Notebook
- Navigate to Data Processing Environment (DPE) → Notebooks.
- Click **+ New Notebook** and for the purpose of this guide we will continue with the **Base Notebook**.
- Give your notebook a meaningful name (e.g., `Cafe_Sales_SQL_Transformations`).
- Click on **Create**.
- Once JupyterLab is open, click on the Python3 Notebook to create a new `.ipynb` file. This is where we are going to run all the following steps in cells.
### 2. Connect to the SDK
We'll use the SDK provided in your DPE environment. This SDK provides methods to connect to the Lakehouse Manager and interact with tables.
```python
# Import necessary modules from SDK
from forepaas.dwh import connect, bulk_insert
from forepaas.core.settings import CONFIG
from forepaas.dwh.logical import LogicalObject
import pandas as pd #useful for displaying dataframes
print("SDK modules imported successfully.")
```
### 3. List Tables from the Dataset
Before connecting to `dirty_cafe_sales`, it's good practice to list available tables to confirm its presence and exact name within the specified data path.
```python
# Connect to the Lakehouse Manager
# Connect to the default Lakehouse Manager dataset
connector = connect("dwh/default_dataset/")
print("Listing tables in 'dwh/default_dataset/':")
available_tables = connector.list()
for table_name in available_tables:
print(f"- {table_name}")
if "dirty_cafe_sales" in available_tables:
print("\n'dirty_cafe_sales' table found!")
else:
print("\nWARNING: 'dirty_cafe_sales' table not found. Please check the table name or path.")
```
### 4. Connect to the Table and Inspect Data
Now, let's connect to the `dirty_cafe_sales` table using `connector.select()` and print its information and descriptive statistics. This step visually confirms the data, including the 'ERROR' and 'UNKNOWN' values we need to clean.
```python
# Connect to the 'dirty_cafe_sales' table
df_raw_sales = connector.select("dirty_cafe_sales")
print("\nDataset information:")
df_raw_sales.info()
print("\nDataset details:")
df_raw_sales.describe()
print("\nSample of Raw Data (first 5 rows):")
display(df_raw_sales.head())
```
### 5. Run SQL Commands (Transformations)
This is the core of our transformation. We'll define two SQL queries: a simple one for basic exploration and a more complex one for thorough cleaning and detailed aggregation.
:::info
**Important Note on Semicolons:** When executing SQL through an SDK or API in a programmatic environment like a notebook, **do not include a trailing semicolon (`;`)** at the very end of your SQL query string. The API typically expects a single SQL statement without an explicit delimiter at the end. Including it can lead to common errors like `mismatched input ';'` or `syntax error near ';'`.
:::
#### 5.1 Simple SQL Query: Daily Total Sales per Location (Initial Exploration)
This query demonstrates a basic aggregation, initially showing how raw data issues can lead to errors, and then fixing them using `TRY_CAST` for robustness. It also cleans the `location` field.
```python
print("--- Running Simple SQL Query ---")
SIMPLE_SQL_QUERY = """
SELECT
CAST(valid_transaction_date AS DATE) AS sale_date,
-- Handle 'ERROR' and empty strings in location
CASE
WHEN location = 'ERROR' THEN 'Unknown'
WHEN TRIM(location) = '' THEN 'Unknown'
ELSE location
END AS clean_location,
SUM(TRY_CAST(total_spent AS DOUBLE)) AS gross_revenue_dirty
FROM
(
SELECT
TRY_CAST(transaction_date AS DATE) AS valid_transaction_date,
location,
total_spent
FROM
dirty_cafe_sales
) AS subquery_sales
WHERE
valid_transaction_date IS NOT NULL
GROUP BY
CAST(valid_transaction_date AS DATE),
CASE
WHEN location = 'ERROR' THEN 'Unknown'
WHEN TRIM(location) = '' THEN 'Unknown'
ELSE location
END
ORDER BY
sale_date DESC, clean_location
""" # No semicolon at the end here!
try:
# Execute the query using your connector's method.
df_simple_result = connector.query(SIMPLE_SQL_QUERY)
# --- Pandas Post-Processing for Data Types ---
# Convert 'sale_date' to datetime objects for proper date operations
df_simple_result['sale_date'] = pd.to_datetime(df_simple_result['sale_date'])
# --- End Pandas Post-Processing ---
print("\nSimple Query Results (first 10 rows):")
display(df_simple_result.head(10))
print("\nSimple Query Results Schema:")
df_simple_result.info()
except Exception as e:
print(f"Error executing Simple SQL query: {e}")
print("\nFailed SQL Query:\n", SIMPLE_SQL_QUERY)
```
#### 5.2 Complex SQL Query: Detailed Cleaned Daily Item Performance
This query will perform robust data cleaning, calculate accurate sales metrics, and aggregate them by date, item, and location. It directly addresses the data quality issues identified (`UNKNOWN` in `quantity`, `ERROR` in `total_spent`, bad `transaction_date` strings, and problematic `location` entries).
```python
print("\n--- Running Complex SQL Transformation Query ---")
COMPLEX_SQL_TRANSFORMATION_QUERY = """
WITH cleaned_and_corrected_sales AS (
SELECT
transaction_id,
item,
-- Clean and cast quantity: 'UNKNOWN' becomes NULL, then cast to INTEGER.
-- TRY_CAST handles non-numeric strings safely by returning NULL.
TRY_CAST(NULLIF(quantity, 'UNKNOWN') AS INTEGER) AS quantity_cleaned,
-- Ensure price_per_unit is numeric, handling potential non-numeric entries safely.
TRY_CAST(price_per_unit AS DOUBLE) AS price_per_unit_cleaned,
payment_method,
-- Clean location: 'ERROR' and empty strings become 'Unknown'.
CASE
WHEN location = 'ERROR' THEN 'Unknown'
WHEN TRIM(location) = '' THEN 'Unknown'
ELSE location
END AS clean_location,
-- Use TRY_CAST for transaction_date to handle bad date strings safely, then filter later.
TRY_CAST(transaction_date AS DATE) AS sale_date_raw
FROM
dirty_cafe_sales
),
final_calculated_sales AS (
SELECT
transaction_id,
item,
quantity_cleaned AS final_quantity,
price_per_unit_cleaned AS final_price_per_unit,
-- Recalculate total_spent based on cleaned quantity and price_per_unit.
quantity_cleaned * price_per_unit_cleaned AS calculated_total_spent,
payment_method,
clean_location,
sale_date_raw AS sale_date
FROM
cleaned_and_corrected_sales
-- Filter out rows where crucial values (quantity, price_per_unit, or sale_date)
-- couldn't be cleanly converted, ensuring only valid data proceeds.
WHERE
quantity_cleaned IS NOT NULL
AND price_per_unit_cleaned IS NOT NULL
AND sale_date_raw IS NOT NULL
)
SELECT
sale_date,
item,
clean_location AS location,
COUNT(DISTINCT transaction_id) AS number_of_transactions,
SUM(final_quantity) AS total_items_sold,
SUM(calculated_total_spent) AS total_revenue_cleaned,
AVG(final_price_per_unit) AS average_item_price_per_unit,
-- Pivot revenue by payment method using Trino's FILTER clause
SUM(calculated_total_spent) FILTER (WHERE payment_method = 'Credit Card') AS revenue_credit_card,
SUM(calculated_total_spent) FILTER (WHERE payment_method = 'Cash') AS revenue_cash,
SUM(calculated_total_spent) FILTER (WHERE payment_method = 'Digital Wallet') AS revenue_digital_wallet,
-- Calculate average quantity per transaction for this group
CAST(SUM(final_quantity) AS DOUBLE) / CAST(COUNT(DISTINCT transaction_id) AS DOUBLE) AS avg_quantity_per_transaction
FROM
final_calculated_sales
GROUP BY
sale_date,
item,
clean_location
ORDER BY
sale_date DESC, total_revenue_cleaned DESC
""" # No semicolon at the end here!
try:
# Execute the complex transformation query
df_transformed = connector.query(COMPLEX_SQL_TRANSFORMATION_QUERY)
# --- Pandas Post-Processing for Data Types ---
# Convert 'sale_date' to datetime objects for proper date operations
df_transformed['sale_date'] = pd.to_datetime(df_transformed['sale_date'])
# --- End Pandas Post-Processing ---
print("\nTransformed Cafe Sales Data (Sample - first 5 rows):")
display(df_transformed.head())
print(f"\nTransformed Data Schema:")
df_transformed.info()
print(f"\nTotal rows in transformed data: {len(df_transformed)}")
except Exception as e:
print(f"Error during Complex SQL transformation: {e}")
print("\nFailed SQL Query:\n", COMPLEX_SQL_TRANSFORMATION_QUERY)
```
### 6. Create a Physical Table from the Transformed Data (CTAS)
After successfully performing the complex transformation and verifying the results, the next logical step is to persist this cleaned and aggregated data into a new physical table in your database. This is typically done using a `CREATE TABLE AS SELECT` (CTAS) statement. This new table can then be used for reporting, further analysis, or as a source for other data processes, without needing to re-run the complex cleaning logic every time.
:::info
**Important Note on SQL Execution:** Some database connectors or APIs expect only one SQL statement per `query()` call. To execute `DROP TABLE` and `CREATE TABLE AS SELECT`, we'll send them as separate commands. Also, ensure there are no trailing semicolons at the very end of each query string.
:::
```python
print("\n--- Creating Physical Table from Complex SQL Transformation Query ---")
# Define the name of your new cleaned table. This variable can be reused across cells.
NEW_CLEANED_TABLE_NAME = "cleaned_cafe_sales_daily_summary"
# 1. DROP TABLE statement (removes the table if it already exists, for idempotent runs)
# Note: No trailing semicolon at the very end of the string.
DROP_TABLE_QUERY = f"DROP TABLE IF EXISTS {NEW_CLEANED_TABLE_NAME}"
# 2. CREATE TABLE AS SELECT statement
# This uses the same logic from the COMPLEX_SQL_TRANSFORMATION_QUERY
# Note: No trailing semicolon at the very end of the string.
CTAS_CORE_QUERY = f"""
CREATE TABLE {NEW_CLEANED_TABLE_NAME} AS
WITH cleaned_and_corrected_sales AS (
SELECT
transaction_id,
item,
TRY_CAST(NULLIF(quantity, 'UNKNOWN') AS INTEGER) AS quantity_cleaned,
TRY_CAST(price_per_unit AS DOUBLE) AS price_per_unit_cleaned,
payment_method,
CASE
WHEN location = 'ERROR' THEN 'Unknown'
WHEN TRIM(location) = '' THEN 'Unknown'
ELSE location
END AS clean_location,
TRY_CAST(transaction_date AS DATE) AS sale_date_raw
FROM
dirty_cafe_sales
),
final_calculated_sales AS (
SELECT
transaction_id,
item,
quantity_cleaned AS final_quantity,
price_per_unit_cleaned AS final_price_per_unit,
quantity_cleaned * price_per_unit_cleaned AS calculated_total_spent,
payment_method,
clean_location,
sale_date_raw AS sale_date
FROM
cleaned_and_corrected_sales
WHERE
quantity_cleaned IS NOT NULL
AND price_per_unit_cleaned IS NOT NULL
AND sale_date_raw IS NOT NULL
)
SELECT
sale_date,
item,
clean_location AS location,
COUNT(DISTINCT transaction_id) AS number_of_transactions,
SUM(final_quantity) AS total_items_sold,
SUM(calculated_total_spent) AS total_revenue_cleaned,
AVG(final_price_per_unit) AS average_item_price_per_unit,
SUM(calculated_total_spent) FILTER (WHERE payment_method = 'Credit Card') AS revenue_credit_card,
SUM(calculated_total_spent) FILTER (WHERE payment_method = 'Cash') AS revenue_cash,
SUM(calculated_total_spent) FILTER (WHERE payment_method = 'Digital Wallet') AS revenue_digital_wallet,
CAST(SUM(final_quantity) AS DOUBLE) / CAST(COUNT(DISTINCT transaction_id) AS DOUBLE) AS avg_quantity_per_transaction
FROM
final_calculated_sales
GROUP BY
sale_date,
item,
clean_location
ORDER BY
sale_date DESC, total_revenue_cleaned DESC
"""
try:
# Execute DROP TABLE first
print(f"Dropping table {NEW_CLEANED_TABLE_NAME} if it exists...")
connector.query(DROP_TABLE_QUERY)
print("Drop table command executed.")
# Then execute CREATE TABLE AS SELECT
print(f"Creating table {NEW_CLEANED_TABLE_NAME}...")
# For DDL operations like CREATE TABLE, connector.query() might return an empty DataFrame or None.
connector.query(CTAS_CORE_QUERY)
print(f"\nSuccessfully created table: {NEW_CLEANED_TABLE_NAME}")
# Optional: Verify the table was created by querying its schema or a few rows
print(f"\nVerifying schema of new table: {NEW_CLEANED_TABLE_NAME}")
df_verify = connector.query(f"SELECT * FROM {NEW_CLEANED_TABLE_NAME} LIMIT 5")
display(df_verify)
df_verify.info()
except Exception as e:
print(f"Error during CTAS operation for {NEW_CLEANED_TABLE_NAME}: {e}")
if "DROP TABLE" in str(e) and DROP_TABLE_QUERY in str(e):
print("\nFailed SQL Query (DROP TABLE):\n", DROP_TABLE_QUERY)
elif "CREATE TABLE" in str(e) and CTAS_CORE_QUERY in str(e):
print("\nFailed SQL Query (CREATE TABLE AS SELECT):\n", CTAS_CORE_QUERY)
else:
print("\nFailed SQL Query:\n", e)
```
### 7. Explore the New Physical Table
You can use standard SQL metadata commands to explore your newly created physical table. Remember to replace `` and `` with your actual values.
```python
# You'll need to know your catalog and schema names.
# Example: your_catalog_name = "default_dataset", your_schema_name = "sales_data"
# To retrieve catalog and schema name you can execute the following commands in the Lakehouse Manager Explorer
# CATALOG LIST: show catalogs
# SCHEMA LIST: show schemas from {catalog_name}
your_catalog_name = "your_main_catalog" # <<< IMPORTANT: Replace with your actual catalog name!
your_schema_name = "your_schema_name" # <<< IMPORTANT: Replace with your actual schema name!
your_table_name = "cleaned_cafe_sales_daily_summary"
print(f"\n--- Exploring the New Table: {your_table_name} ---")
try:
# Describe your new table
df_describe = connector.query(f"DESCRIBE {your_catalog_name}.{your_schema_name}.{your_table_name}")
print(f"\nDescription of table '{your_table_name}':")
display(df_describe)
# Select some data from your new table
df_sample_data = connector.query(f"SELECT * FROM {your_catalog_name}.{your_schema_name}.{your_table_name} LIMIT 5")
print(f"\nSample data from '{your_table_name}':")
display(df_sample_data)
except Exception as e:
print(f"Error exploring table '{your_table_name}': {e}")
print("Please check the full table path (catalog.schema.table) and permissions.")
```
### 8. Create a Logical Object from the Physical Table
When you create a new table through SQL, it exists only as a physical table in the database. However, the Tables section in the UI operates at a logical level. It displays tables that are registered as logical objects within the platform.
To make your newly created physical table visible and usable in the UI, you’ll need to create a corresponding logical object. This logical representation acts as a bridge between the database and the platform interface, allowing you to interact with the table’s schema and data directly from the UI.
```python
print(f"\n--- Creating LogicalObject for '{NEW_CLEANED_TABLE_NAME}' ---")
try:
logical_cleaned_sales = LogicalObject().create_from_physical(NEW_CLEANED_TABLE_NAME)
print(f"Successfully created logical object for: {NEW_CLEANED_TABLE_NAME}")
except NameError:
print("Error: 'LogicalObject' is not defined. Please ensure you have imported the correct library/class or that it's globally accessible.")
except Exception as e:
print(f"Error creating logical object: {e}")
```
:::info
if you want to remove a LogicalObject that you have created, you can directly delete the table from the UI or else use the method - `LogicalObject().remove("table_name")` - this will remove the table at both logical and physical levels.
:::
### Conclusion
You have successfully navigated through an end-to-end SQL transformation process on the OVHcloud Data Platform. Starting from a dirty raw dataset, you have applied various SQL cleaning, aggregation, and reshaping techniques to produce a clean, summarized, and highly usable dataset. This transformed data was then persisted into a new physical table and represented as a logical object for seamless integration into your Python workflows. This foundational knowledge empowers you to tackle more complex data preparation challenges and build robust data pipelines.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-external-datasets.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create and Build your data models - External Datasets
## Objective
This tutorial will help you in using the **Lakehouse Manager** for the second step and the **data processing engine** for the third step of the _getting started tutorial_ with the _External Datasets_.
## Organize your data in tables
### Create your primary schema
#### Add tables to the data model
:::warning
You need to make sure that you have created an external dataset before proceeding. The steps to create an external dataset can be found [here](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md).
:::
Once that your metadata has been extracted, it's time to head to the Tables dashboard. This is where you will **build a unified and queryable view of all your data**.
The empty Tables page should look like this.
The _All tables_ tab is where you have access to the entirety of your data. The _New View_ tab allows you to create views of only a part of your data in order to collaborate better in large teams. Since this is a simple tutorial, you should work in the _All tables_ tab.
Now let’s concentrate on creating your primary tables and their attributes.
First, hover your cursor over the blue ➕ icon on the left-hand side of the screen. This will reveal the create options :
- Upload a file
- Create from Connectors source
- Create an empty tables
:::info
For the purpose of this tutorial, we shall proceed with **Create from a Connectors source.**
:::
Once you click on _create from a connectors source_, you will be shown a list of sources from the the previous step. Click on source you want to add and proceed further by clicking on _Next_. Here we can choose the one of the external datasets you have created. No need to change any of the other default settings here.
:::info
The options _Build the table, load the table once, and generate Load action for later_ can be disabled and each step can be done individually as well. This can be explored in detail in the [Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) guide.
:::
Press **Create** and proceed to repeat the same with the second table as well.
:::info
Note that each time you make a change on the Tables page, your visual configuration is **automatically saved**.
:::
At this stage, your Tables page should look like this.
You may notice that some attributes in your tables are written **in bold**. The platform has automatically detected that they were **primary keys**, i.e. the attribute (or set of attributes) used to uniquely identify each data row in the table.
:::info
You can manually change the primary keys for any table by hovering your mouse cursor over the attribute and clicking on the **star** ⭐ icon.
:::
:::warning
You may notice that while using _External Datasets_ some attributes in your tables are written **in bold**. The Platform has automatically detected that they were **primary keys**, i.e. the attribute (or set of attributes) used to uniquely identify each data row in the table. **The Lakehouse Manager Engine** is more big data engine, and by default the concept of primary keys do not exist. Hence you will not be seeing any attributes **in bold**.
:::
:::info
Note that the Platform uses the metadata information captured in the Analyzer step to automatically create tables and assign attribute names & types. The source files provided are good to go, however in a real Project you should use the Analyzer to check the data sources, before dragging and dropping them in the Tables page.
:::
#### Join tables together
You need to tell the Lakehouse Manager how your tables relate to each other.
You will join tables that relate to each other in a parent/ child relationship. A **parent** is a table that has detailed information about a particular subject: for example, the weather in Chicago at a certain date. A **child** table references multiple parent tables. Another way to look at it is that a child table inherits values from the parent table (just like children in real life, except that a child can have more than two parents!)
The table _stations\_rides_ is a child table. It contains ridership information by date and train station. But it does not have detailed information about the days. Both your tables have _date_ as a primary key. So, if you link the _chicago\_calendar\_full_ parent table to the _stations\_rides_ child table, you will automatically have added weather information to your ridership, for each day of the year.
First, click once on _chicago\_calendar\_full_ parent table to select it, then click on the white circle at the bottom of the table:
Simply drag the arrow to the child table _stations\_rides_ and drop/unclick. Your screen should be like this:
### Create your aggregate table
Now, you will aggregate all important data from the sources (namely rides, dates and temperatures) into one table that will be used in the final application. Aggregate tables consolidate data from multiple source tables.
To create your first aggregate table, click on the blue ➕ icon. Select _Create an empty table_, and you will be shown a new table configuration where you set a name (_dataset\_history_ for example) and save it.
Then, drag and drop _date_ and _station\_id_ from the source table _stations\_rides_ into the table _dataset\_history_ and set these attributes as primary keys.
Move the attributes below to _dataset\_history_ and do not set them as keys (since they are simple data of each row and are not unique):
| Original table | Attributes to drag-and-drop |
| :-------------------------: | ------------------------------------------------------------------------ |
| **stations\_rides** | _lat_ / _lng_ / _rides_ / _station\_name_ |
| **chicago\_calendar\_full** | _month_ / _temperature_ / _week\_day_ / _week\_day\_label_ / _weekend_ / |
Finally, you will need to create one new attribute to help translate numerical temperature data into understandable categories (cold, hot, ..).
Start by clicking on the ➕ icon that appears at the top of the table _dataset\_history_ when you click on it. You can then create or edit an attribute inside a table.
Define the attribute as such:
| Attribute name | Type | Nature |
| :------------------: | ------ | --------- |
| **cat\_temperature** | String | Dimension |
:::info
For the moment, this attribute is not physically specified. This will be done later in another component: the Data Processing Engine.
:::
[↪ Learn more about Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md)
### Finalize the build
:::warning
One last quick check! Make sure to **double-check that your data model looks exactly like the one in the screenshots** before moving to the next step. If some attributes are missing you will get stuck in later steps of the tutorial.
:::
Now, click on the _Build_ icon (under the blue ➕ icon) to effectively create/update the tables and attributes in your dataset. This still doesn't load the data into the tables (which will be done in the next article), it simply applies the logical schema to tables and attributes in your underlying dataset.
:::info
While the visual logical schema of the data is automatically saved, changes to your tables won't be visible in the rest of the Platform as long as they are not built.
:::
The build task for this tutorial shouldn't take more than a few minutes to run. Once it is done, you can move on.
### Add relevant metrics with Virtual Attributes
Before moving on to the physical processing (ETL/ELT) of the data into this model, let's prepare additional metrics for analytics later on. The final application that you are building following this tutorial includes a chart with the **number of rides per day on a given station**:
However, you do not have the necessary data to build that chart directly on the primary sources. You will need a metric that gives you the average number of rides per day for a given station and that can be used in queries and dashboards.
But how do you compute it using the platform? One way to do this is to create a **virtual attribute**. Virtual attributes allow you to calculate SQL formulas that will be **computed on the fly** and won't be stored in the database. They can be used in a query or a chart in your final dashboard.
:::info
Adding or editing virtual attributes does not require rebuilding the schema.
:::
Switch to the **Attributes** page. This page lists all physical and virtual attributes in your data model, and the lineage in your Project.
Click on the _New Attribute_ button to create a virtual attribute.
In the creation window, make sure to select _Virtual_ as the realm.
Now add these two attributes and their respective SQL code:
| Attribute name | SQL |
| :------------------------------------: | -------------------------------------------------------------------------------------- |
| **avg\_rides\_per\_day\_per\_station** | SUM(rides)/COUNT(DISTINCT CONCAT(CAST(date AS VARCHAR), CAST(station\_id AS VARCHAR))) |
| **yearmonth** | SUBSTR(CAST(date as VARCHAR),1,7) |
The **yearmonth** attribute gives you the year and month in the format _yyyy-mm_. You will use it later.
:::info
Notice how you just used two different methods to generate new attributes/metrics from imported data: **adding a physical attribute to a table** and **virtual attributes**.
:::
- Adding a new physical attribute takes up storage and requires you to physically define them in the Data Processing Engine, but this makes them more rigorous as their specifications can then be edited without changing the whole data model.
- Virtual attributes are a quick win but can become hard to manage if you need to modify them when scaling.
## Actions
An [action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) consists of a unitary physical operation on the data. Actions can be organized in stages in order to produce automated data processing pipelines called [workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md).
Click on the **Actions** menu of your Data Processing Engine. You should see the two _Load_ actions that were[ automatically generated in the previous step](https://docs.dataplatform.ovh.net/getting-started-organize-data.md#organize-your-data-in-tables):
These Load actions will physically extract the data from your sources and load it into your data warehouse, following the schema made in the Lakehouse Manager.
### Create more actions
:::info
Our Marketplace gives you access to a dozen of curated actions to kick-start your data processing Projects: _load_ actions, _aggregate_ actions, _delete_ actions, etc. If you cannot find what you need in the catalog, you can always resort to a _custom_ action that allows you to **run any piece of Python 3+ code** as part of your data pipelines.
:::
[↪ Learn more about custom actions](https://docs.dataplatform.ovh.net/dpe-actions-custom.md).
For this tutorial, you will create an action used to aggregate your data into the _dataset\_history_ table you created in the previous part.
Click on **New action** and select the _Aggregate action_ template from the Platform Store.
There will be 2 simple steps to configure the aggregate action:
- **(1)** Select a source table : _stations\_rides_
- **(2)** Select the destination table: _dataset\_history_
After a couple of seconds, the Data Processing Engine will automatically find all the join conditions required, and map attributes.
Change the join condition to an _INNER join_ using the dropdown menu. This will ensure that you do not have any null fields in the records of your dataset\_history table.
Attributes have been mapped automatically. Find the attribute _rides_, which is the metric you are looking to aggregate, and switch it to a **SUM** function. Everywhere else, leave the **MAX** function as it is: most DBMS require you to apply an aggregate clause to attributes that aren't part of the _GROUP BY_ clause.
Finally, let's define the categorical attribute _cat\_temperature_ that you created earlier. Click on the **\< map >** (short for "mapping") option in blue option dropdown as shown below and switch it to **\< sql >**.
Simply copy-paste the SQL command below:
```
CASE
WHEN MAX(chicago_calendar_full.temperature)<40 THEN 'very cold'
WHEN MAX(chicago_calendar_full.temperature)<48 THEN 'cold'
WHEN MAX(chicago_calendar_full.temperature)<55 THEN 'medium'
WHEN MAX(chicago_calendar_full.temperature)<62 THEN 'hot'
ELSE 'very hot'
END
```
:::warning
**Leaving a destination attribute unmapped in the Aggregate action configuration will trigger an error when the action is launched**. If you'd rather leave the destination field empty, make sure to simply remove it from the list of mapped attributes.
:::
[↪ Learn more about Aggregate actions](https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md)
Click on **Create** at the top-right.
**You have now generated all the actions** required for this tutorial.
:::info
Of course, your real-life Project will probably have more than 3 actions. You can organize your actions in folders and rename them if needed. You can also use more than one repository, especially if you are working in collaboration with teammates. Repositories of actions can be versioned and also synced with external Git repositories. Check-out the [dedicated product documentation](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#manage-actions) page to learn more about how to do that!
:::
[↪ Learn more about Actions in the Product Documentation](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md)
## Workflows
A workflow defines the order in which your actions run.
Within a workflow, actions are organized in sequential stages. Within a stage, all actions will be run in parallel while stages will always run one after the other. The same action can be used multiple times in the same workflow. A workflow, just like an action, can either be launched manually, set up to run on a schedule or triggered through an API call.
:::info
Note that it is important to remember that **stages are run one after the other** in the order you planned them while **actions contained in a stage are all run all at the same time** regardless of the order. In short, the order of the actions inside a stage does not matter while the stages’ order inside a workflow does.
:::
To create your first workflow, you'll need to head to the _Workflow_ tab and click on **New Workflow**. Head to preferences or double-click on the header name to set a new name _Import Chicago Data_.
Let's start by defining two different stages by clicking on **Add a stage**. Then, add actions in each stage using the dropdown search selector following the screenshot provided as a guide for each stage.
After creating the workflow (**create**), press **Play**.
:::info
Please note that workflows might take a few minutes to run when you launch them for the first time. The total time should not exceed 10 minutes - if it does, please reach out to our support team.
:::
While the workflow is running, something you can do is **schedule it to run daily** using a trigger.
Head to the _Preferences_ tab of your workflows and scroll to the Triggers widget on the bottom left side. Click **+Add**.
Select the trigger type as _CRON_ and mode as _Simple_. Navigate to the **Daily** tab and in the list of options select _Every 1 day(s)_ as shown in the picture below:
Hit the **Confirm** button to create the new trigger event with the name of your choice and it will append it below the _Launch Endpoint_ present by default in the Trigger event table.
:::info
There is a lot more you can configure in a workflow's preferences. Namely, you can [scale horizontally and vertically](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) any processing job, use [workload segmentation](https://docs.dataplatform.ovh.net/tutorials-segmentation.md) to accelerate the data processing and even save all these configurations for repeated use thanks to [environments](https://docs.dataplatform.ovh.net/dpe-environments.md).
:::
[↪ Learn more about configuring the execution preferences.](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md)
:::warning
Make sure to click on the **Save** button on the top right of the screen whenever you make a modification to your workflows. Actions are stored in repositories which can be versioned which is not the case for workflows or environments. _Autosave_ is therefore disabled for both workflows & environments.
:::
## Jobs
To wrap up this section, here are a few words about the last tab of the Data Processing Engine component: jobs.
The Jobs tab summarizes **all executions triggered in the Data Processing Engine** and includes advanced metrics reports. Jobs are listed under three main categories: running, queued and past executions. Having a look at the last jobs executed, you can verify the status of the workflow you just launched.
You now have completed the Data Engineering section of the _Getting Started_ tutorial.
A good way to make sure your data properly loaded is to go back to the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) and look at the number of rows loaded into the table. Just open the **list view mode** and check the _rows_ column, if the field has a number (indicating how many rows have been loaded) then everything worked well.
If you want to continue with the getting started tutorial, you can proceed directly to [step-4: the Analytics Manager](https://docs.dataplatform.ovh.net/getting-started-create-queries.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Apply segmentation to your jobs
## Objective
[Segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md) will allow you to unleash the Data Processing Engine's power of scalability. When executed, your [actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) will be **split into multiple tasks**, each task will manage a smaller dataset.
:::info
With great power comes with great responsibility: segmentation options should be configured carefully. Depending on your segmentation choices, your action may be much faster... or much slower.
:::
## Segmentation over the values of a table attribute
[↪ Learn how to use segmentation over the values of a table attribute](https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md)
## Segmentation over a predefined set of values
[↪ Learn how to use segmentation over a predefined set of values](https://docs.dataplatform.ovh.net/tutorials-segmentation-predefined-values.md)
## Segmentation over the date perimeter set in the workflow
[↪ Learn how to use segmentation over the date perimeter set in the workflow](https://docs.dataplatform.ovh.net/tutorials-segmentation-workflow-dates.md)
## Segmentation by chunking a source
[↪ Learn how to use segmentation by chunking a source](https://docs.dataplatform.ovh.net/tutorials-segmentation-source-chunks.md)
## Segmentation over same structured files
[↪ Learn how to use segmentation over same structured files](https://docs.dataplatform.ovh.net/tutorials-segmentation-files.md)
## Segmentation over source accounts
[↪ Learn how to use segmentation over source accounts](https://docs.dataplatform.ovh.net/tutorials-segmentation-source-accounts.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover segmentation "Based on an attribute of a Project table"
## Objective
Your action will be split into multiple tasks, each task will manage a smaller set of values coming from a **chosen attribute from a [given table from the Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md)**.
:::info
You can find the actual product documentation page for segmentation, detailing the feature's behavior and specifications [on that page](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md).
:::
## Requirements
Before using this segmentation type, some points **must be checked**.
### 1. Is the `var_name` attribute indexed in the source table ?
If not, the extraction query will be much slower.
### 2. Is there enough CPU on the source and destination databases?
- Keep in mind that `select` and `insert` operations are CPU intensive.
- For example, if your database has only 1 CPU, it's probably a bad idea to set 6 workers to select and insert simultaneously against the DBMS instance...
:::info
General Rule: 1 CPU should be available for each worker running simultaneously.
:::
### 3. Avoid generating too many tasks in the same stage.
To ensure the Data Processing Engine (DPE) performs well, we advise against having stages with more than 500 tasks.
This is not a hard limit but performance degradations can be observed with increasing number of tasks above 500.
In order to reduce the number of tasks, you can set a higher bucket size, so that each task will manage more values (meaning less tasks at the end).
## How to use this type of segmentation?
For the example, let's use an `aggregate` action on the following `prim_ticket` table, in order to aggregate all tickets income per `date`, and insert the result in the table agg\_date.
To optimize execution time of this action, the workload will be split into multiple tasks based upon each date value.
### Understanding advanced parameters
Before going further, here is how the _Preferences_ tab matches with the JSON fields in _Advanced mode_.\
For the sake of clarity, we will use the **technical JSON names** below.
:::info
Please note that the blue squares are only used by the UI for clarity. The green parameters are the ones used by the DPE.
:::
**Source's attribute / var\_name:** SQL attribute that will be used for the filtering of the source.
**Reference Attribute / Values:** The address from where the set of values is taken.
must be formatted as **"dwh/TABLE\_NAME/ATTRIBUTE\_NAME"** if from a dataset table.
or **"dwh/SOURCE\_NAME/TABLE\_NAME/ATTRIBUTE\_NAME"** if from a source.
**Bucket size / Chunksize:** Number of values to be filtered for each task.
### Choosing the right segmentation attribute
Choosing the right attribute for the segmentation is the key for a successful and fast data pipeline. The characteristics of a good segmentation attribute are:
#### 1. Cardinality
The cardinality of the attribute should not be too high regarding the number of rows of the source table.
:::info
You can check out the cardinality with a simple [COUNT DISTINCT query](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#aggregation-functions) in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
#### 2. Rows distribution
Ideally, distribution of the number of rows for each value should be approximately equal.
#### 3. The values should not be long text...
...or it can overload the job controller.
#### 4. For `aggregate`, `diff` and `delete_diff` actions
:::warning
The segmentation attribute must be part of the **primary key** of the **destination table**.
:::
If not, you will have eventually incomplete data for each groups.
For example, if you use segmentation over `ticket_id` while aggregating per `date`, task1 will insert the first ticket income, then task2 will update the income with ticket 2 that occurred within the same date...
#### 5. Good candidates tips
- The attribute that describes the date of your facts is often a good candidate :
- it is often in the primary key of your aggregate tables
- it may have a low to middle cardinality
- it may have a pretty good distribution of rows over time.
- Attributes that are part of your main referential tables can be good candidates too.
### Other tips
#### 1. You can use SQL formulas
For `segmentation.values` (in the attribute last part) and for `segmentation.var_name`, you can use SQL formulas, as long as they are compatible with your DBMS. Please note that the DBMS can change over time and so you might have to recheck and correct those SQL formulas if you decide to use them.
For instance, you may have :
```json
"segmentation":
{
"active": true,
"type": "dwh_attributes",
"values": "dwh/default_dataset/tickets/DATE(datetime)",
"var_name": "DATE(datetime)",
"chunksize": 1
}
```
#### 2. `segmentation.values` table can be different than the source table
If your set of `segmentation.values` is contained in another table than your source table, you can indicate a table that is not the same as the source table used in the action.\
This can be useful to narrow down some elements and not recalculate your whole table each time.
For instance, `segmentation.values` could come from a referential table, or a temporary table, to manage only the data that just came in.
## How does it work behind the scenes?
If your action has a segmentation **Based on an attribute of a Project table**,
when executing the action, or the action in a workflow, it will execute:
1. A hidden pre-stage gets all the distinct values from the `table/attribute` given in `segmentation.values` field.
1. The Job Controller splits the action into multiple tasks, each with a different value (or set of values, depending on the `chunksize` configuration) amongst values found in step 1.
2. Then each **worker** executes **each task one by one**.\
This means that there is no issue for having hundreds of tasks, your action will be parallelized regarding the number of workers.
### Technical explanations
Let's review exactly how each part of the segmentation configuration is used.
With the following segmentation configuration:
```json
{
"segmentation": {
"active": true,
"type": "dwh_attributes",
"values": "dwh/default_dataset/tickets/date",
"var_name": "date",
"chunksize": 1
}
}
```
1. The pre-stage task gets all the distinct `date` values from `prim_ticket` and replaces the `segmentation.values` address by the actual values.\
Please note how the `segmentation.values` address is built.\
This address may be different from the actual `source table` selected in the action.
2. The Job Controller creates tasks according to the length of `segmentation.values` and the `chunksize` (number of values that each task should handle)
3. Each worker handles the tasks one by one, here is an example with an aggregate action.
Of note; `segmentation.var_name` is used as the attribute to filter in the WHERE clause for the extraction query. So, it may be different from the one used in `segmentation.values`.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation-workflow-dates.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover segmentation based on the date perimeter configured in the workflow
## Objective
Often, for a daily usage, you will need to ask for your [Data Processing Engine (DPE) actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) to manage data corresponding to the X last days until today.
Some other times, you will need your actions to act only on a historical set of dates.
This segmentation type basically works like the [“Segmentation over the values of an attribute of a table.”](https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md), except that the values comes from the dates you set in the [workflow configuration](https://docs.dataplatform.ovh.net/dpe-jobs-perimeter.md), rather than coming from a [Lakehouse Manager table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
### Concrete examples of usage
- Get data from a source only from X days ago to today. (daily relative dates usage)
- Get data from a source between 2 precise dates (get historical data, oneshoot usage)
- Recalculate an aggregate table only between 2 dates (daily relative dates usage or oneshoot usage )
If your source data contains a date attribute, it is possible to split the action in separate processes which will individually address a specific date or a group of dates.
In this case, each task will apply a filter on the date that it needs to extract.
:::info
You can find the actual product documentation page for segmentation, detailing the feature's behavior and specifications [on that page](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md).
:::
## Requirements
Before using this segmentation type, some points **must be checked**:
### 1. Is the `var_name` attribute indexed in the source table?
If not, the extraction query will be much slower.
### 2. Is there enough CPU on the source and destination databases?
- Keep in mind that `select` and `insert` operations are CPU intensive.
- For example, if your database has only 1 CPU, it's probably a bad idea to set 6 workers to select and insert simultaneously against the DBMS instance...
:::info
General Rule: 1 CPU should be available for each worker running simultaneously.
:::
### 3. Avoid generating too much tasks in the same stage.
To keep the Data Processing Engine (DPE) performs well, we advice to avoid setting stages of more than 500 tasks.
This is not a hard limit as it won't bug right after this threshold, but performance degradations can be observed.
In order to reduce the number of tasks, you can set a higher bucket size, so that each task will manage more values. (meaning less tasks at the end).
#### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle; **API**: Facebook, Twitter, Google Analytics, YouTube Private, YouTube Public, HTTP REST |
## How to use this type of segmentation?
Here is a quick view between the visible mode and the advanced JSON mode.
**Source's attribute / var\_name:** SQL attribute that will be used for the filtering of the source.
**Bucket size / Chunksize:** Number of values to be filtered for each task.
### Other tips
#### You can use SQL formulas
For `segmentation.var_name` (in the attribute last part), you can use SQL formulas, as long as they are compatible with your DBMS. Please note that the DBMS can change over time and so you might have to recheck and correct those SQL formulas if you decide to use them.
For instance, you may have :
```json
"segmentation":
{
"active": true,
"type": "workflow_dates",
"var_name": "DATE(datetime)",
"chunksize": 1
}
```
## How does it work behind the scenes?
This segmentation mode works the same as [segmentation based on an attribute of a Project table](https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md#how-does-it-work-behind-the-scenes). The only difference is that the values comes from the dates perimeter of the workflow, instead of a Lakehouse Manager table attribute, so there is no pre-stage needed.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation-predefined-values.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover segmentation based on a predefined set of values
## Objective
Your actions can be executed on a set of values of your choice while segmenting on them at the same time.
Examples of usage :
- Segmenting on a precise preset of shopping sites or items
- Applying a business rule A to subset of data A with aggregate action A, and a business rule B to a subset of data B in an aggregate action B.
:::info
You can find the actual product documentation page for segmentation, detailing the feature's behavior and specifications [on that page](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md).
:::
## Requirements
Before using this segmentation type, some points **must be checked**:
### 1. Is the `var_name` attribute indexed in the source table?
If not, the extraction query will be much slower.
### 2. Is there enough CPU on the source and destination databases?
- Keep in mind that `select` and `insert` operations are CPU intensive.
- For example, if your database has only 1 CPU, it's probably a bad idea to set 6 workers to select and insert simultaneously against the DBMS instance...
:::info
General Rule: 1 CPU should be available for each worker running simultaneously.
:::
### 3. Avoid generating too much tasks in the same stage.
To ensure the Data Processing Engine (DPE) performs well, we advise against having stages with more than 500 tasks.
This is not a hard limit but performance degradations can be observed with increasing number of tasks above 500.
In order to reduce the number of tasks, you can set a higher bucket size, so that each task will manage more values. (meaning less tasks at the end).
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshi |
## How to use this type of segmentation?
Here is a quick view between the visible mode and the advanced JSON mode.
**Source's attribute / var\_name:** SQL attribute that will be used for the filtering of the source.
**Bucket size / Chunksize:** Number of values to be filtered for each task.
**Predefined set / values:** Set of hard-coded values to filter from.
### Other tips
#### You can use SQL formulas
For `segmentation.var_name` (in the attribute last part), you can use SQL formulas, as long as they are compatible with your DBMS. Please note that the DBMS can change over time and so you might have to recheck and correct those SQL formulas if you decide to use them.
For instance, you may have :
```json
{
"segmentation": {
"active": true,
"type": "predefined_set",
"var_name": "DATE(datetime)",
"values": ["2020-01-01","2020-01-02"]
"chunksize": 1
}
}
```
## How does it work behind the scenes?
This segmentation mode works the same as [segmentation based on an attribute of a Project table](https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md#how-does-it-work-behind-the-scenes). The only difference is that the values are already set in the configuration, rather than a Lakehouse Manager table attribute, so there is no pre-stage needed.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation-source-chunks.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover segmentation by chunking source rows
## Objective
Sometimes, a CSV file might be huge to absorb. In this case you can choose to segment on the number of rows to improve total load speed.
## Requirements
Before using this segmentation type, some points **must be checked**:
### 1. Avoid generating too many tasks in the same stage.
To ensure the Data Processing Engine (DPE) performs well, we advise against having stages with more than 500 tasks.
This is not a hard limit but performance degradations can be observed with increasing number of tasks above 500.
In order to reduce the number of tasks, you can set a higher bucket size, so that each task will manage more rows. (meaning less tasks at the end).
Keep in mind that for each task, the source CSV file will be uploaded again from a datastore temporary bucket into each worker. so if your csv file is huge, you might want to keep the number of tasks low. **Ideally the number of tasks should be the number of workers the job possesses.**
### 2. Is there enough CPU on the destination database?
- Keep in mind that `insert` operations are CPU intensive for your DBMS.
- For example, if your database has only 1 CPU, it's probably a bad idea to set 6 workers to insert simultaneously against the DBMS instance...
:::info
General Rule: 1 CPU should be available for each worker running simultaneously.
:::
### Compatibility
| Action Types | Source Types |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Load; Custom | **File Formats**: CSV**Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle, SQLServer |
## How to use this type of segmentation?
Here is a quick view between the visible mode and the advanced JSON mode.
**Bucket size / Chunksize:** Set how many rows will be absorbed for each task.
### Load actions
The [Load action source](https://docs.dataplatform.ovh.net/dpe-actions-load.md) will be the base of the segmentation.
### Custom actions
Make sure that the key `params.load_from[0].source` indicates the address of the source:
_dwh/SOURCE\_NAME/FILE\_NAME_ or _TABLE\_NAME_
:::info
Please check the previous screenshot for an example of the advanced JSON mode.
:::
## How does it work behind the scenes?
If your action has a segmentation **Based on a fixed number of lines**,
when executing the action, or the action in a workflow, it will execute:
1. A hidden pre-stage that gets the CSV source file, count its number of rows, then put it in an object-store temporary bucket.
2. The Job Controller splits the action into multiple tasks, each with a different row chunk to ingest.
3. Then each **worker** executes **each task one by one**.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation-files.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover segmentation over a set of files
## Objective
For sources with the same structure in a distant source folder, this segmentation mode allows the Data Processing Engine (DPE) to automatically find all files in a folder, then create tasks with a specific number of files. It will replace the source file in “load\_from.source” actions for each files.
Example: Every day, there will be new file(s) coming in the source. you need your load action to absorb all those files incoming, not knowing in advanced their exact name.
- rides\_chicago\_2020-01-01.csv
- rides\_paris\_2020-01-01.csv
- rides\_tokyo\_2020-01-01.csv
Each task will manage different files, allowing for parallelization of work amongst different workers.
## Requirements
Before using this segmentation type, some points **must be checked**:
- All the files must have the same format and structure
- All files must be in the same folder
- If other files with different structures are present in the folder, the files must have specific naming such as it can be filtered out by a RegExp.
- All the attributes/columns present in the schema, and also in each file, must:
- Be present (except for semi-structured formats such as XML or JSON: they will be replaced by NULL values if not present) ;
- have the same type or structure. (example: datetime structures)
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| ------------ | -------------------------------------------------------------- |
| Load; Custom | S31 (file upload, datastore, AWS, etc..); FTP; SFTP |
## How to use this type of segmentation?
To configure this segmentation option select _Based on files_ in the "Segmentation Type" dropdown.
**RegExp / Values:** You can set a regexp to identify the filenames you want to get. the star (\*) character permit to search for any characters (it will be replaced in the regexp by: .\* ).
It can be a list of RegExps as well. or a hard-coded list of precise filenames.
**Bucket size / Chunksize:** Set how many files will be absorbed for each task.
it is recommended to set this value to 1, unless you have hundreds of files, you will want to put a higher number of files for each task to keep the number of tasks reasonable (under 100 tasks).
### Blueprints rules and mapping
In the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md), you will need to [analyze](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) one file that will serve as the blueprint for all the other files.
It's important that you keep the definition of this file in the connectors to keep the [blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md).
In the DPE, creating the [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md), you will use the analyzed file as a source.
In doing so, all the loaded files will have the same blueprint rules applied.
## How does it work behind the scenes?
If your action has a segmentation **Based on an fixed number of files**,
when executing the action, either on its own or via a workflow, it will execute:
1. A hidden pre-stage gets all the filenames in your source corresponding to the RegExp set in the configuration.
1. The Job Controller splits the action into multiple tasks, each with a different value (or set of values, depending on the `chunksize` configuration) amongst filenames found in step 1.
2. Then each **worker** executes **each task one by one**.\
This means that there is no issue for having hundreds of tasks, your action will be parallelized regarding the number of workers.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-segmentation-source-accounts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover segmentation over source accounts
## Objective
For sources like social media which are user account specific, it is possible to configure multiple accounts for the same source within the Lakehouse Manager component. You can then leverage these accounts to split an DPE action using that source in several tasks, where each task will retrieve the data from one or more accounts.
## Requirements
Before using this segmentation type, some points **must be checked**:
- You already configured a source in the Connectors having some accounts such as below
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| ------------ | --------------------------------------------------------------------------------------- |
| Load; Custom | **API**: Facebook, Twitter, Google Analytics, YouTube Private, YouTube Public, LinkedIn |
## How to use this type of segmentation?
To configure this segmentation option select _Based on the source's accounts_ in the "Segmentation Type" dropdown.
**Bucket size / chunksize:** Set how many accounts will be managed for each task.
(default = 1)
### Load actions
The [Load action source](https://docs.dataplatform.ovh.net/dpe-actions-load.md) will be the base of the segmentation.
### Custom actions
Make sure that the key `params.load_from[0].source` indicates the address of the source:
_dwh/SOURCE\_NAME/ENDPOINT\_NAME_
:::info
Please check the previous screenshot for an example of the advanced JSON mode.
:::
### Other tips
#### Load from only one set of accounts.
You can hard-code a list of account(s) you want to load, adding a key "values" as shown below.
It can be useful if you need to retrieve an history of a newly created account without retrieving all data from all accounts set.
The configuration would look like this:
```json
"environment": {
"params": {
"segmentation": {
"active": true,
"type": "accounts",
"values": ["account_jp","account_fr"]
"chunksize": 1
}
}
}
```
The values must correspond to the **account keys** set in the source, as you can see in the first use-case screen. (in the example : account\_fr, account\_jp, account\_us).
if "values" is not present, the job will retrieve data from all set accounts in the source that are enabled.
## Behind the scenes
If your action has a segmentation **Based on the source's accounts**,
when executing the action, or the action in a workflow:
1. The Job Controller splits the action into multiple tasks, each with a different account (or set of accounts, depending on the `chunksize` configuration) amongst the accounts set in the source
2. Then each **worker** executes **each task one by one**.\
This means that there is no issue for having hundreds of tasks, your action will be parallelized regarding the number of workers.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-tutorials-analytics-bi.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Analytics & BI
## Querying and visualizing what you built
[Trino](https://trino.io/) is the distributed SQL engine behind interactive analytics on the platform, and four tutorials cover it: extracting data from one project into another, connecting external tools with a manual connection string, the type and cast rules that trip up cross-source queries, and Metabase as a worked example.
PySpark handles the analysis that does not fit in a single query. The NYC taxi series works through a real public dataset in four parts, from one table to a multi-dataset comparison, and ends with a cheat sheet worth keeping open.
Two tutorials go further afield. [AI Endpoints](https://endpoints.ai.cloud.ovh.net/) applies models to project data. OVHcloud billing analysis builds a cost dashboard from the OVHcloud API as a source, which doubles as a FinOps starting point.
Apache Superset covers the presentation layer: deploying it against a project, then two use cases built on top, FinOps reporting on FOCUS-format data and a sales dashboard for an online store.
[### Extract project data with Trino→
Move data from one project into another.
](/tutorials-trino-connector)[### Integrating Trino with external tools→
Connect an external client with a manual connection string.
](/tutorials-trino-external-tools)[### Trino data types and casts→
The mapping that causes most cross-source query errors.
](/tutorials-trino-types-and-casts)[### Using Metabase with Trino→
A worked external-BI connection.
](/tutorials-metabase-trino)[### NYC taxi data analysis with PySpark→
A real public dataset in four parts, cheat sheet included.
](/tutorials-pyspark)[### Data analysis with AI Endpoints→
Apply models to project data.
](/tutorials-ai-endpoints)[### OVHcloud billing data analysis→
Build a cost dashboard from the OVHcloud API.
](/tutorials-billing-analysis)[### Deploy Apache Superset→
Run Superset against a project.
](/tutorials-install-apache-superset)[### FOCUS and Apache Superset use case→
FinOps reporting on FOCUS-format data.
](/tutorials-finops-focus-superset)[### Analyse online store sales→
Prepare two years of orders, then build a sales dashboard in Superset.
](/tutorials-ecommerce-sales)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-trino-connector.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract project data with Trino
## Objective
This tutorial demonstrates how to use the [Trino Connector](https://docs.dataplatform.ovh.net/connectors-sources-trino.md) to extract data from one project into another on the Platform.
## Introduction
### Requirements
To follow this tutorial, you need to be familiar with the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md), [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md), [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md) and you should preferably have done the [Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). The essential is that you know what [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) are for and how to use them.
### Concepts Overview
It is common to have multiple projects running across the platform. In the scenario where you would want to extract and transfer data from an existing project into another, the _Trino Connector_ does the work.
This is made possible thanks to the **managed Trino query engine** which is part of the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) service. It allows to query the datasets of the project - and can be connected to from the external world.
This tutorial will follow a two-step process:
- Make the required changes at the **source project** to allow the extraction of data.
- Use the Trino Connector at the **destination project** as a source to move the data.
Now let's see how all of this works in practice!
## Use the Trino Connector
### Steps to Follow at the Source Project
These initial steps leverage the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) and the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
1. **Identify Source Catalog and Schema:**
Find the catalog and schema you intend to connect to in Lakehouse Manager [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md)
Execute the following SQL commands to retrieve the available catalogs and schemas:
```sql
SHOW CATALOGS
```
Copy and save the desired catalog name as `CATALOG_NAME`.
```sql
SHOW SCHEMAS FROM CATALOG_NAME
```
:::info
**Note:** Replace `CATALOG_NAME` with the actual catalog you selected in the previous step.
:::
This command will display a list of available schemas within your chosen catalog.
Copy and save the desired schema name as `SCHEMA_NAME`.
:::info
Upon creation of a new project, the `default dataset` is provided. If you have not established any additional datasets within your project, this is the appropriate selection.
:::
2. Open the _Identity Access Manager_ and create a new service account for the purpose of this transfer.
:::info
Make sure to add _Platform Admin_ as the group for this service account
:::
Once the service account has been successfully create, you need to generate a key for the _API Access_. Click on the add button as shown in the picture below, followed by create.
You will be provided with the _API Key_ and the _Secret Key_. Make sure to copy and save them securely as you wont be able to access the _Secret Key_ once you leave this page.
You have successfully completed all the steps required at the source project; the next part of the tutorial will continue on the destination project.
### Steps to follow at destination project
1. Proceed to the _Connectors_ and add a new Trino Source.
2. Fill in the details as given below and click on connect
- **Host:** replace by your _**source**_ _Project\_Name_ + `-query.eu.dataplatform.ovh.net`
- **Port:** 443
- **Catalog:** replace with your _CATALOG\_NAME_
- **Schema:** replace with your _SCHEMA\_NAME_
- **Username:** replace with _API\_KEY_
- **Password:** replace with _SECRET\_KEY_
- **SSL:** activated
3. Once the Trino Connection has been successfully established, the list of available tables will be populated. Click on the box next to table name you desire to extract from the source and click create.
## Using the extracted data
Once all the previous steps have been completed, you can visit the _Analyzer_ tab within the _Connectors_ to verify the extracted tables.
To use this extracted data you can follow **step three** of the [Getting Started](https://docs.dataplatform.ovh.net/getting-started-prepare-data.md) Tutorial.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Integrating OVHcloud Data Platform with External Tools using Trino
## Objective
:::info
**Looking for the quick way?** You can now generate a ready-to-use Trino connection string in a few clicks with a [**Trino consumer**](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md). This guide covers the **manual method**: useful when you want to set up the service account, role, and connection string yourself, or need fine-grained control over the catalog and permissions.
:::
This guide provides a comprehensive walkthrough for creating a Trino integration to be used in a third-party tool. You will generate the necessary credentials and construct a connection string to access your data via Trino.
## Steps to Establish a Trino Connection
### Step 1: Create a New Service Account
1. Log in to the Data Platform.
2. Navigate to **IAM** (Identity and Access Manager).
3. Go to the **Roles** section and click **New Role**.
4. Provide a name for the role (e.g., `TrinoServiceAccountRole`).
5. Assign the **Advanced Data Access Control** permissions to this role. This role will be assigned to your service account to grant the necessary Trino access.
6. Go to **Service Accounts**.
7. Create a new Service Account and give it a name (e.g., `Apache Superset Service Account` or `Metabase Service Account`) and an email address.
8. Assign the newly created role (`TrinoServiceAccountRole`) to this account.
9. Click **Create**.
:::info
For better security and access management, create a dedicated service account when allowing external tools to access your data.
:::
### Step 2: Generate an API Key and Secret Key
1. Once you have created the _Service Account_, you will see the section for **API Access**. Click on **+ ADD**.
2. Here you can modify the duration of the keys and also if you want to send a copy to your registered email.
3. **Critical**: Once the modal is closed, the keys cannot be retrieved again. Be sure to save them immediately in a secure location, such as a password manager. These API Key and Secret Key will serve as your `Username` and `Password` for the Trino connection.
### Step 3: Prepare the connection string
The connection string you will be able to use in your third-party tools will be as follows:
```text
trino://{Username}:{Password}@{Project}-query.eu.dataplatform.ovh.net:{Port}/{Catalog}
```
- **Project**: Replace `{Project}` with your specific project name. You can find this name by navigating to any of your existing projects, then clicking the Information icon represented by ℹ️ in the top right corner of the interface. Your project name will be displayed there.
- **Port**: Replace `{Port}` with the default Data Platform port (`443`) or a custom port provided by your setup.
- **Catalog**: Replace `{Catalog}` with the catalog you want to query. You can find the catalog you want by executing this SQL command in the explorer of the Lakehouse Manager: `SHOW CATALOGS`.
- **Example `SHOW CATALOGS` output:**
```text
Catalog
-------
tpcds
tpch
db_m8obb5ch_default_dataset
```
In this example, `db_m8obb5ch_default_dataset` is a valid catalog name.
- **Username**: Use the _API Key_ generated earlier from IAM in Step 2.
- **Password**: Use the _Secret Key_ generated earlier from IAM in Step 2.
**Example:**
```text
trino://XXXX:YYYYYY@demo-query.eu.dataplatform.ovh.net:443/db_m8obb5ch_default_dataset
```
### Step 4: Use the connection string in your third-party tool
Once you have prepared your connection string, you can use it to connect your preferred third-party tool (e.g., Apache Superset) to Trino on OVHcloud Data Platform.
**Returning to a Specific Tool Guide?**
- If you came from the Apache Superset integration guide, click [here](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) to continue.
- If you came from the Metabase integration guide, click [here](https://docs.dataplatform.ovh.net/tutorials-metabase-trino.md) to continue.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-trino-types-and-casts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Trino data types and casts
## Objective
Whatever path you take to query your project over [Trino](https://trino.io/), the SQL editor of the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md), a [Trino consumer](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), or an external tool, the types you manipulate are the **Trino types**, not the logical types you see in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md). Most of the time the two line up, but a few cases require an explicit `CAST` at `INSERT` time.
## Data types: Lakehouse to Trino
The table below maps each Lakehouse type to the Trino type it is exposed as:
| Lakehouse type | Trino type | Notes |
| ---------------- | --------------- | --------------------------------------------- |
| `integer` | `INTEGER` | 32-bit signed integer |
| `bigint` | `BIGINT` | 64-bit signed integer |
| `real` / `float` | `REAL` | 32-bit floating point |
| `double` | `DOUBLE` | 64-bit floating point |
| `decimal` | `DECIMAL(p, s)` | Precision and scale are preserved |
| `varchar` | `VARCHAR` | Unbounded unless a length was set at creation |
| `boolean` | `BOOLEAN` | |
| `date` | `DATE` | |
| `timestamp` | `TIMESTAMP` | Timezone-naive |
:::info
The exact Trino type of a column, including any length or precision, is whatever the table was **created** with. When in doubt, run `SHOW CREATE TABLE ..
;`. Any `CAST` in an `INSERT` should target that type, not the logical type you see in the Lakehouse Manager.
:::
### Implicit coercions (no CAST needed)
Trino widens these types automatically, so you can insert the source value directly:
- `INTEGER` → `BIGINT`
- `REAL` → `DOUBLE`
- `DECIMAL` → a wider `DECIMAL`
- `VARCHAR(n)` → unbounded `VARCHAR`: a literal such as `'foo'` is a `VARCHAR(3)` and coerces to an Iceberg string without any change
- `DATE` → `TIMESTAMP`
### Casts you must write explicitly
Trino will **not** coerce these on its own. An `INSERT` without a `CAST` returns `TYPE_MISMATCH`:
- `VARCHAR` → numeric or date: `CAST('1' AS INTEGER)`, `CAST('2026-05-26' AS DATE)`
- numeric → `VARCHAR`: `CAST(x AS VARCHAR)`
- `TIMESTAMP` ↔ `TIMESTAMP WITH TIME ZONE` (any change in timezone-awareness)
- `DECIMAL(p1, s1)` → `DECIMAL(p2, s2)` when the precision or scale differs and truncation is possible
- Complex types: there is no coercion between `ROW` values of different structures, you have to rebuild the `ROW(...)` field by field
## Inserting typed values
### Use a valid literal form
`INT '1'` is **not** valid Trino syntax. `INT` is accepted as an alias of `INTEGER` in a _column declaration_, but not in the _typed-literal_ form `TYPE 'value'`.
To insert an integer, use any of:
```sql
INSERT INTO t VALUES (1); -- native integer literal
INSERT INTO t VALUES (INTEGER '1'); -- typed literal, canonical type name
INSERT INTO t VALUES (CAST('1' AS INTEGER)); -- explicit cast
```
:::info
Reading the same column never raises an error, because no typed literal is evaluated on `SELECT`. Only the `INT 'x'` form breaks the parser.
:::
### Troubleshooting TYPE\_MISMATCH on VARCHAR inserts
`CAST(x AS VARCHAR)` produces an unbounded `VARCHAR`, which is compatible with an Iceberg string column. If you still get a `TYPE_MISMATCH`, check the following, in order:
1. **The target column may not be unbounded.** If the table was created through Trino with `VARCHAR(50)`, the connector keeps that length constraint. Confirm with `DESCRIBE catalog.schema.table;` or `SHOW CREATE TABLE catalog.schema.table;`. If the column is `varchar(50)`, cast to `CAST(x AS VARCHAR(50))`.
2. **Column order or count.** Without an explicit column list, Trino maps values by position and raises `TYPE_MISMATCH` as soon as one downstream column no longer matches, and the error sometimes points at the wrong expression. Always prefer:
```sql
INSERT INTO t (col_a, col_b, col_c) VALUES (...);
```
3. **Complex types.** `CAST(... AS VARCHAR)` on a `ROW`, `ARRAY` or `MAP` fails. Convert field by field, or use `json_format(CAST(... AS JSON))`.
4. **Generated columns and timestamps.** A `VARCHAR` does not coerce implicitly to `TIMESTAMP` or `DATE`. A `TYPE_MISMATCH` that appears to be on a `VARCHAR` can actually come from another column in the same `VALUES` clause.
:::warning
If the mismatch persists, collect `SHOW CREATE TABLE`, the full `INSERT` statement and the exact error message before opening a support ticket. That is enough to pinpoint the faulty column.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-metabase-trino.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connecting Metabase to Data Platform via Trino
## Objective
This guide provides a step-by-step walkthrough to set up a connection between Metabase and Data Platform using Trino. The instructions include configuration details, common troubleshooting tips, and relevant screenshots to ensure a seamless setup.
**Before you begin:**
You will need the connection details for Trino on OVHcloud Data Platform (such as `host`, `port`, `catalog`, `username (API Key)`, and `password (Secret Key)`). The easiest way is to create a [**Trino consumer**](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), which generates a connection string containing all of these values. Alternatively, assemble them manually by following [Integrating Trino with External Tools](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md).
:::info
For the purpose of this tutorial, we will use a locally hosted instance of Metabase running on [Docker](https://docs.docker.com/desktop/). This setup allows for easy deployment and isolation while keeping the environment consistent across different systems.
:::
## Steps to Connect Metabase to Data Platform via Trino
### Step 1: Set Up a Self-Hosted Metabase with Docker-Compose
**1. Prepare the Environment**
Ensure you have **Docker** and **Docker Compose** installed on your server or local machine.
**2. Create a Docker Compose File**
Create a `docker-compose.yml` file in your working directory and add the following configuration:
```yaml
version: "3.8"
services:
metabase:
image: metabase/metabase:latest
container_name: metabase
platform: linux/amd64
ports:
- "3000:3000"
volumes:
- ./data/metabase:/metabase-data # Align with other paths
- ./data/metabase/plugins:/plugins
environment:
- MB_DB_FILE=/metabase-data/metabase.db
```
**3. Create the Directory Structure**
Before running `docker-compose up`, you need to create the directory structure that the _docker-compose.yml_ file expects. This ensures that Metabase can properly store its data and plugins.
Create the following directories in your working directory:
:::info
1. ./data/metabase
2. ./data/metabase/plugins
:::
You can create these directories using the following commands in your terminal:
```bash
mkdir -p ./data/metabase
mkdir -p ./data/metabase/plugins
```
This will create the necessary directory structure for Metabase to persist its data and store plugins.
To start Metabase using Docker, run the following command:
```sh
docker-compose up -d
```
Once started, you can access Metabase at `http://localhost:3000` or replace `localhost` with your server's IP/domain.
:::info
**First-Time Setup**: If you are running Metabase for the first time, the setup process may take some time. You can monitor the logs to check when Metabase is fully up and running by using:
:::
```sh
docker logs -f metabase
```
This command will display real-time logs, allowing you to verify that the service has started successfully.
### Step 2: Download and Add the Starburst Plugin
Visit the [**Starburst plugin page**](https://github.com/starburstdata/metabase-driver/releases) and download the latest **JDBC driver** for Starburst. This driver enables Metabase to connect to Trino (formerly Starburst/PrestoSQL).
Locate the downloaded JDBC file (e.g., `starburst-6.1.0.metabase-driver.jar`) and copy it to the Metabase plugin directory (`./data/metabase/plugins` as configured in `docker-compose.yml`).
Ensure the Metabase container has access to the plugin directory, and **restart Metabase** if necessary, using:
```sh
docker-compose restart metabase
```
### Step 3: Configure Trino in Metabase
:::info
Once Metabase is running, you will need to complete a few initial steps to create your account and set up the application before accessing your admin dashboard.
:::
1. Go to **Admin Settings > Databases**.
2. Click on **Add Database**.
3. Choose **Starburst** as the database type from the available options.
### Step 4: Provide Connection Details
Using the Trino connection details from your [Trino consumer](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) connection string (or built manually via the [integration guide](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md)), fill in the Metabase connection form with the following values:
- **Host**: The Trino host, e.g., `{ProjectName}-query.eu.dataplatform.ovh.net`.
- **Port**: The Trino port, typically `443`.
- **Catalog**: The Trino catalog name you wish to connect to.
- **Schema (optional)**: Enter the desired schema name, or leave it blank to allow access to all schemas within the catalog. (Note: In Metabase, 'Schema' refers to a logical grouping of tables, not a single table.)
- **Username**: Your Trino API Key.
- **Password**: Your Trino Secret Key.
### Step 5: Advanced Settings
Expand the **Advanced Options** and configure the following:
- **Enable SSL**: Check this option to use secure communication. This is highly recommended for production environments.
- **Extra JDBC Parameters**: Include any additional necessary parameters for your specific setup. Leave blank if unsure.
### Step 6: Finalize and Test the Connection
After entering all the necessary details, click the **Save** button to finalize the database connection. Metabase will attempt to connect to your OVHcloud Data Platform Trino instance using the provided credentials.
You can now exit the Admin panel.
### Step 7: Allow Time for Database Synchronization
After saving the connection, Metabase will begin synchronizing with your OVHcloud Data Platform database. This process can take some time, especially if you have a large number of tables.
**Recommendation**: Allow Metabase sufficient time to complete the synchronization process. The duration depends on the size and complexity of your database. You can monitor the progress in the background or check the Metabase logs (`docker logs -f metabase`) for completion.
### Step 8: Explore and Utilize Metabase Features
Once the database synchronization is complete, you can fully explore and utilize Metabase's features:
- **Data Exploration**: Browse your OVHcloud Data Platform databases and tables directly within Metabase.
- **Querying**: Use the Metabase query builder to create custom SQL queries and retrieve specific data.
- **Dashboard Creation**: Build interactive dashboards with visualizations to monitor key metrics and trends.
- **Reporting**: Generate and schedule reports to share insights with your team.
- **Alerting**: Set up alerts to notify you of important changes in your data.
With the connection successfully established, you can now leverage Metabase's powerful features to analyze and visualize your data stored in OVHcloud Data Platform via Trino.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-pyspark.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# NYC Taxi Dataset Analysis with PySpark on OVHcloud Data Platform
## Objective
Welcome to our comprehensive guide for analyzing New York City's taxi and ride-sharing data using **PySpark** on the **OVHcloud Data Platform**. This collection of tutorials demonstrates how to harness the power of distributed computing to extract meaningful insights from one of the world's most fascinating urban transportation datasets.
## Understanding the NYC Taxi Dataset
### What is the NYC Taxi Dataset?
The [NYC Taxi and Limousine Commission (TLC) Trip Record Data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) represents one of the most comprehensive urban transportation datasets available publicly. This massive collection captures millions of taxi and ride-sharing trips across New York City, providing unprecedented insights into urban mobility patterns.
### Dataset Components
The dataset encompasses four distinct transportation modes, each serving different segments of NYC's transportation ecosystem:
1. **🚕 Yellow Taxi Trip Records**
- **Service Type**: Traditional street-hail yellow cabs
- **Primary Coverage**: Manhattan and airport connections
- **Trip Volume**: \~3.5 million trips per month
- **Key Features**: Pickup/dropoff locations, trip duration, distance, fare details, passenger count
2. **🟢 Green Taxi Trip Records**
- **Service Type**: Street-hail taxis for outer boroughs
- **Primary Coverage**: Brooklyn, Queens, Bronx, Staten Island
- **Trip Volume**: \~48,000 trips per month
- **Regulatory Purpose**: Designed to improve taxi service in underserved areas
3. **🚗 For-Hire Vehicle (FHV) Records**
- **Service Type**: App-based rides (Uber, Lyft, etc.)
- **Primary Coverage**: All five boroughs
- **Trip Volume**: \~1.9 million trips per month
- **Characteristics**: Includes longer trips and airport services
4. **🚙 High Volume FHV Records**
- **Service Type**: High-frequency ride-sharing providers
- **Primary Coverage**: Citywide with exceptional density
- **Trip Volume**: \~20.4 million trips per month
- **Market Position**: Dominates the modern ride-sharing landscape
### Data Richness and Scope
Each record contains detailed information including:
- **Temporal Data**: Precise pickup and dropoff timestamps
- **Geographic Data**: Location IDs mapped to 265 taxi zones across NYC
- **Trip Characteristics**: Distance, duration, passenger count
- **Economic Data**: Fare amounts, tips, taxes, and total charges
- **Service Details**: Payment methods, rate codes, and service types
This wealth of information enables analysis of urban mobility patterns, economic impacts, and transportation efficiency across different service types and geographic areas.
## Why PySpark for NYC Taxi Data Analysis?
### Scale and Performance Challenges
The NYC Taxi dataset presents significant computational challenges:
- **Volume**: Combined datasets exceed 25 million trips per month
- **Variety**: Multiple data formats (Parquet, CSV) with different schemas
- **Complexity**: Multi-dimensional analysis across time, geography, and service types
### PySpark Advantages
1. **Distributed Computing Power**
PySpark leverages Apache Spark's distributed computing framework to process large datasets across multiple nodes, dramatically reducing processing time compared to traditional single-machine approaches.
2. **Memory-Optimized Processing**
Unlike traditional disk-based processing, PySpark keeps data in memory between operations, enabling faster iterative analysis and machine learning workflows.
3. **Scalability**
As your data grows, PySpark scales horizontally by adding more compute nodes, ensuring consistent performance regardless of dataset size.
4. **Rich Ecosystem**
PySpark integrates seamlessly with:
- **MLlib**: Machine learning algorithms optimized for distributed computing
- **Spark SQL**: SQL-like querying for data exploration
- **GraphX**: Network analysis capabilities
- **Streaming**: Real-time data processing
5. **Python Integration**
Combines Spark's performance with Python's rich ecosystem of data science libraries (Pandas, Matplotlib, Seaborn, Scikit-learn).
### OVHcloud Data Platform Benefits
The OVHcloud Data Platform provides:
- **Managed Infrastructure**: No need to configure Spark clusters
- **Integrated Ecosystem**: Seamless connection between Connectors, Lakehouse Manager, and Processing Engine
- **Scalable Resources**: Automatic scaling based on workload requirements
- **Cost Efficiency**: Pay-as-you-use model for compute resources
## Two Comprehensive Analysis Examples
We've created two detailed tutorials that demonstrate different aspects of NYC Taxi data analysis, each showcasing specific PySpark capabilities and analytical approaches.
## **Example 1: Single Dataset Deep Dive**
[NYC Yellow Taxi Dataset Analysis with PySpark](https://docs.dataplatform.ovh.net/tutorials-pyspark-single-dataset.md)
### Objective
Perform comprehensive exploratory data analysis and build machine learning models to predict trip duration using Yellow Taxi data.
### What You'll Learn
- **Data Engineering**: Loading, cleaning, and preprocessing large Parquet files
- **Exploratory Data Analysis**: Discovering patterns in taxi usage across time and geography
- **Feature Engineering**: Creating meaningful predictors from raw trip data
- **Machine Learning**: Building and comparing regression models (Linear Regression, Random Forest, Gradient Boosted Trees)
- **Performance Optimization**: Leveraging PySpark's distributed computing for efficient model training
### Key Analytical Insights
- **Trip Patterns**: Peak usage hours and seasonal variations
- **Geographic Hotspots**: High-demand pickup zones and borough-specific patterns
- **Duration Prediction**: Achieving 80.6% variance explained (R² = 0.806) with Gradient Boosted Trees
- **Feature Importance**: Trip distance accounts for 77.4% of duration prediction accuracy
### Technical Highlights
- Processing 2.78 million trip records
- Handling data quality issues and outlier detection
- Implementing distributed machine learning algorithms
- Creating interactive visualizations with Matplotlib and Seaborn
## **Example 2: Multi-Dataset Comparative Analysis**
[NYC Taxi Comparative Analysis Across Types with PySpark](https://docs.dataplatform.ovh.net/tutorials-pyspark-multi-dataset.md)
### Objective
Compare usage patterns, market share, and operational characteristics across all four NYC transportation modes to understand the competitive landscape.
### What You'll Learn
- **Data Integration**: Combining multiple datasets with different schemas
- **Schema Standardization**: Harmonizing column names and data types across datasets
- **Comparative Analytics**: Analyzing market share and usage patterns
- **Geographic Analysis**: Understanding service distribution across NYC boroughs
- **Temporal Analysis**: Comparing peak hours and usage patterns by service type
### Key Analytical Insights
- **Market Dominance**: High Volume FHV services lead in absolute trip volume
- **Geographic Specialization**: Yellow Taxis dominate Manhattan; Green Taxis serve outer boroughs effectively
- **Service Characteristics**: FHV services handle longer trips; traditional taxis optimize for shorter urban rides
- **Temporal Patterns**: All services show similar rush-hour patterns but differ in off-peak utilization
### Technical Highlights
- Processing 25+ million combined trip records
- Handling schema heterogeneity across datasets
- Implementing advanced data cleaning and standardization techniques
- Creating comprehensive comparative visualizations
## Getting Started
### Prerequisites
- Access to OVHcloud Data Platform
- Basic understanding of Python and data analysis concepts
- Familiarity with Jupyter notebooks
### Tutorial Path
1. **Start with Example 1** if you're new to PySpark or want to understand single-dataset analysis.
2. **Progress to Example 2** to learn multi-dataset integration and comparative analysis.
3. **Combine techniques** to build your own custom analyses.
### Data Setup
Both tutorials include complete step-by-step instructions for:
- Uploading datasets to the Connectors
- Creating tables in the Lakehouse Manager
- Configuring PySpark notebooks in the Data Processing Engine
## Real-World Applications
### Urban Planning
- **Traffic Flow Analysis**: Understanding peak congestion patterns
- **Infrastructure Planning**: Identifying high-demand areas for transportation improvements
- **Service Optimization**: Optimizing public transportation routes based on taxi usage patterns
### Business Intelligence
- **Market Analysis**: Competitive positioning across transportation modes
- **Demand Forecasting**: Predicting service needs by location and time
- **Revenue Optimization**: Dynamic pricing strategies based on demand patterns
### Research and Policy
- **Transportation Policy**: Evaluating the impact of regulations on service distribution
- **Economic Impact**: Understanding the economic footprint of ride-sharing services
- **Sustainability Analysis**: Analyzing trip efficiency and environmental impact
## Technical Architecture
### Data Flow
1. **Data Ingestion**: Raw trip records from NYC TLC
2. **Connectors**: Metadata management and schema discovery
3. **Lakehouse Manager**: Structured data storage with Iceberg tables
4. **PySpark Processing**: Distributed analysis and machine learning
5. **Visualization**: Interactive charts and statistical summaries
### Performance Optimization
- **Caching**: Strategic DataFrame caching for iterative operations
- **Partitioning**: Optimized data partitioning for parallel processing
- **Broadcasting**: Efficient joins with lookup tables
- **Resource Management**: Dynamic resource allocation based on workload
## Next Steps
After completing these tutorials, consider exploring:
### Advanced Analytics
- **Time Series Forecasting**: Predicting future demand patterns
- **Clustering Analysis**: Identifying customer segments and trip patterns
- **Network Analysis**: Understanding pickup-dropoff relationships
- **Anomaly Detection**: Identifying unusual patterns in trip data
### Integration Opportunities
- **Weather Data**: Analyzing weather impact on transportation choices
- **Event Data**: Understanding how events affect transportation demand
- **Economic Indicators**: Correlating economic conditions with transportation usage
### Machine Learning Extensions
- **Deep Learning**: Neural networks for complex pattern recognition
- **Reinforcement Learning**: Optimizing taxi dispatch strategies
- **Ensemble Methods**: Combining multiple models for improved accuracy
**Quick Reference:** For a handy guide to PySpark syntax and common operations, check out our **[PySpark Cheat Sheet](https://docs.dataplatform.ovh.net/tutorials-pyspark-cheat-sheet.md)**.
Ready to dive into the world of urban data analytics? Choose your starting point and begin exploring the fascinating patterns hidden within NYC's transportation data!
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-pyspark-single-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# NYC Yellow Taxi Dataset Analysis with PySpark Notebook
## Objective
This tutorial demonstrates how to efficiently use **PySpark Notebook** on the **OVHcloud Data Platform** to analyze the _Yellow Taxi Trip Records_, stored as a Parquet file, alongside the _Taxi Zone Lookup Table_ stored as a CSV file. We’ll perform **exploratory data analysis (EDA)** to uncover patterns in taxi usage, create **visualizations** to interpret these patterns, and build a machine learning model to predict trip duration based on features like trip distance and pickup location.
The tutorial is designed for beginners, with detailed explanations for each step to help you understand how to use PySpark in a Jupyter notebook on OVHcloud’s Data Platform. It assumes you’ve completed the **[Getting Started](https://docs.dataplatform.ovh.net/landing-page-getting-started.md)** guide, which includes setting up your environment, accessing the Connectors, and configuring a PySpark-enabled Jupyter notebook.
By the end of this tutorial, you’ll know how to:
- Load and clean large datasets using PySpark.
- Join datasets to add geographical context.
- Perform EDA to identify trends in taxi trip data.
- Visualize results using Matplotlib and Seaborn.
- Build a simple machine learning model to predict trip duration.
## Setting Up Data Sources and Tables for the PySpark Tutorial
Before analyzing the **Yellow Taxi Trip Records** and **Taxi Zone Lookup Table** using PySpark on the OVHcloud Data Platform, you need to prepare the data environment. This involves:
1. Creating data sources in the **[Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md)** to register the Parquet and CSV files.
2. Creating tables in the **[Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)** to store the data in a structured format.
3. Configuring a _Load Action_ in the **[Data Processing Engine (DPE)](https://docs.dataplatform.ovh.net/landing-page-dpe.md)** to populate the Yellow Taxi table with data.
These steps ensure the data are accessible in the Lakehouse Manager for analysis in a Jupyter notebook with PySpark.
### Prerequisites:
- Access to the OVHcloud Data Platform with permissions to manage the Connectors, Lakehouse Manager, and DPE.
- Completed the **OVHcloud Data Platform Getting Started guide**.
- The files `yellow_tripdata_YEAR-MONTH.parquet` and `taxi_zone_lookup.csv` available from [NYC TLC Trip Record data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
### Step 1: Create a new source in Connectors
- Create a new _File Upload_ source and name it "NYC-taxi".
- Upload the `yellow_tripdata_YEAR_MONTH.parquet` and `taxi_zone_lookup.csv` files, then click create.
- Go to the Analyzer tab and perform meta extraction to infer schemas for the uploaded files.
### Step 2: Create Tables in Lakehouse Manager
- Go to _Tables_ tab → click on **Create from a Connectors source**.
- Select the `yellow_tripdata_YEAR_MONTH.parquet` file → and only select the option "Automatically build the table".
- Repeat for the `taxi_zone_lookup.csv` file, select all the options since this table is smaller and can be fully loaded by Iceberg quickly.
### Step 3: Use Load PySpark action in the DPE
- Go to _Actions_ tab → Create new action → Use **Load PySpark**.
- Set:
- **Source**: `yellow_tripdata_YEAR_MONTH`
- **Destination**: `yellow_tripdata_YEAR_MONTH` Table in the Lakehouse Manager.
- Ensure schema mapping is correct (the source columns should align with the table’s schema)
- Save and run the action.
### Step 4: Create a Jupyter Notebook in the DPE
- Go to the "Notebooks" tab → New notebook → Select **PySpark Notebook**.
- Name it as `NYC yellow taxi notebook`.
After launching, the platform will take a few moments to initialize a JupyterHub instance. Once it's ready, click the **Open in Jupyter** button to access the notebook interface.
With the data environment now fully configured and all sources loaded into the Lakehouse, we’re ready to dive into the PySpark notebook and begin our analysis.
## PySpark Notebook Walkthrough
The notebook below includes the full PySpark code, structured step by step with inline comments and explanations. You can download it and upload it directly into your Jupyter environment on the OVHcloud Data Platform. Each step is explained in detail to help you follow the analysis process, and a summary is provided at the end for quick reference.
:::info
[NYC_Yellow_Taxi_Dataset_Analysis.ipynb
](/images/public-cloud/data-platform/getting-further/pyspark/resources/NYC_Yellow_Taxi_Dataset_Analysis.ipynb)
:::
### What’s Inside the Notebook?
The notebook walks you through:
- Connecting to the Lakehouse Manager and loading the dataset
- Cleaning and joining trip data with zone lookups
- Performing EDA with Matplotlib and Seaborn
- Building and comparing three ML models (Linear Regression, Random Forest, GBT)
- Outputting performance metrics and insights
Below are a few sample outputs from the notebook:
## NYC Taxi Trip Duration Prediction – Summary
### Objective
The objective of this analysis was to predict `trip_duration` using machine learning models trained on NYC Yellow Taxi data. These predictions can help optimize taxi dispatch strategies, improve fare estimates, and support operational decision-making.
### Models Compared
We evaluated three regression models to understand their effectiveness on this task:
- **Linear Regression** – a simple and interpretable baseline
- **Random Forest** – an ensemble method that captures non-linear relationships
- **Gradient Boosted Trees (GBT)** – a boosting-based approach aimed at maximizing predictive accuracy
### Data Preparation
To ensure high-quality input data and better model performance, the following steps were performed:
- Removed extreme outliers (trips shorter than 60 seconds or longer than 3600 seconds)
- Encoded categorical variables (e.g., `pickup_borough`)
- Scaled and assembled numerical features into a unified feature vector
### Features Used
- `trip_distance`
- `pickup_hour`
- `day_of_week`
- `pickup_borough` (indexed)
### Data Split
- **Training set:** 80% (\~2.2 million records)
- **Test set:** 20% (\~556,000 records)
### Model Performance
| Model | RMSE (seconds) | R² |
| ----------------- | -------------- | ----- |
| Linear Regression | 373.1 | 0.675 |
| Random Forest | 318.6 | 0.763 |
| GBT | 288.7 | 0.806 |
**Interpretation:**\
GBT achieved the best performance overall, with the lowest Root Mean Squared Error (\~4.8 minutes) and the highest R² score (80.6% variance explained).
### Feature Importance (Random Forest)
| Feature | Importance (%) |
| ---------------------- | -------------- |
| `trip_distance` | 77.4 |
| `pickup_borough_index` | 19.8 |
| `pickup_hour` | 2.4 |
| `day_of_week` | 0.4 |
**Insight:**\
Trip distance was by far the most influential predictor. Pickup borough had some impact, while time-based features (hour and weekday) had minimal effect on trip duration.
### Impact of Data Cleaning
- Removed records with negative or unrealistic values
- Reduced noise and improved model reliability
- Final dataset (\~2.78 million records) was more representative
- Resulted in lower RMSE and higher R² across all models, especially GBT
## Conclusion
This tutorial demonstrated how PySpark and the OVHcloud Data Platform can be used effectively to process, explore, and model large-scale taxi trip data. Through structured data preparation, visual analysis, and model evaluation, we identified key drivers of trip duration and built a predictive model with strong performance.
Among the models tested, Gradient Boosted Trees (GBT) offered the best results in both accuracy and robustness, highlighting its value for real-world forecasting tasks.
The workflow outlined here, covering ingestion, transformation, exploration, and machine learning, can serve as a reusable framework for similar large-scale analytics projects across different domains.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-pyspark-multi-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# NYC Taxi Comparative Analysis Across Types with PySpark
## Objective
This tutorial extends our previous analysis to compare trip patterns across four different NYC transportation datasets for **January 2025**: Yellow Taxi, Green Taxi, For-Hire Vehicle (FHV), and High Volume FHV Trip Records. We'll perform a comprehensive comparative analysis to understand the differences in service usage across NYC's ride-sharing ecosystem.
These datasets represent distinct segments of NYC's transportation network:
- **Yellow Taxi**: The iconic yellow cabs, primarily street-hail services operating mainly in Manhattan
- **Green Taxi**: Street-hail taxis specifically focused on outer boroughs (Brooklyn, Queens, Bronx, Staten Island)
- **FHV**: For-hire vehicles (including Uber, Lyft) booked via mobile apps, covering all boroughs
- **High Volume FHV**: High-volume providers like Uber and Lyft with exceptionally large trip counts
By comparing trip volume, average trip duration, and pickup zones across boroughs, we'll uncover key differences in service usage patterns. This analysis provides valuable insights for urban planners, taxi operators, and ride-sharing companies analyzing market dynamics and operational strategies.
The tutorial builds on cleaning and joining techniques from our previous Yellow Taxi analysis, performs comprehensive **exploratory data analysis (EDA)**, and creates **visualizations** to highlight usage patterns across different transportation modes.
## Requirements
Before starting this comparative analysis, ensure you have:
- **Data Setup**: All four datasets (`yellow_tripdata_2025_01.parquet`, `green_tripdata_2025_01.parquet`, `fhv_tripdata_2025_01.parquet`, `fhvhv_tripdata_2025_01.parquet`) and the `taxi_zone_lookup.csv` file are loaded in your Connectors and accessible in the Lakehouse Manager. You can download these datasets from the [official NYC TLC Trip Record Data website](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
- **Environment**: A PySpark-enabled Jupyter notebook configured on the OVHcloud Data Platform
- **Previous Tutorial**: Completed the **[NYC Yellow Taxi Dataset Analysis](https://docs.dataplatform.ovh.net/tutorials-pyspark-single-dataset.md)** tutorial for foundational understanding
If you haven't completed the data setup, follow **Steps 1-3** from the previous tutorial to:
1. Upload datasets to the Connectors
2. Create tables in the Lakehouse Manager
3. Configure load actions in the Data Processing Engine (DPE)
## Step-by-Step Tutorial
The complete PySpark code is provided below with detailed explanations for each step. Copy and paste this into a new Jupyter notebook on the OVHcloud Data Platform. Each section includes comprehensive comments and explanations to guide you through the comparative analysis process.
### Step 1: Environment Setup and Data Connection
We initialize our PySpark session and establish connections to access all four transportation datasets.
```python
import logging
from forepaas.dwh import connect
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, col, hour, dayofweek, unix_timestamp, avg, count, sum, when
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
# Configuration variables
DATASET = "default_dataset"
PROJECT_ID = "PROJECT_ID" # Replace with your actual Project ID
YEAR = "2025"
MONTH = "01"
# Set up logging for debugging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Initialize SparkSession
try:
spark = SparkSession.builder.appName("NYC_Taxi_Comparative_Analysis").getOrCreate()
logging.info(f"Spark Version: {spark.version}")
except Exception as e:
logging.error(f"Failed to initialize SparkSession: {e}")
raise
# Connect to Lakehouse
try:
cn_prim = connect(f"dwh/{DATASET}/")
logging.info(f"Connected to Lakehouse - Dataset: {DATASET}")
except Exception as e:
logging.error(f"Failed to connect to Lakehouse - Dataset: {DATASET} | {e}")
raise
```
**What this does:**
- Imports necessary libraries for data processing and visualization
- Configures logging for debugging and monitoring
- Initializes a PySpark session optimized for large-scale data processing
- Establishes connection to the OVHcloud Data Platform Lakehouse
### Step 2: Data Loading and Initial Inspection
We load all four transportation datasets and the taxi zone lookup table, then verify successful loading and inspect their schemas.
```python
# Load all datasets from the Lakehouse
yellow_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.yellow_tripdata_2025_01")
green_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.green_tripdata_2025_01")
fhv_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.fhv_tripdata_2025_01")
fhvhv_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.fhvhv_tripdata_2025_01")
taxi_zones_df = cn_prim.query(f"SELECT LocationID, Borough, Zone FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.taxi_zone_lookup")
# Cache DataFrames for improved performance
yellow_df.cache()
green_df.cache()
fhv_df.cache()
fhvhv_df.cache()
taxi_zones_df.cache()
# Verify successful data loading
logging.info(f"Yellow Taxi Records: {yellow_df.count():,}")
logging.info(f"Green Taxi Records: {green_df.count():,}")
logging.info(f"FHV Records: {fhv_df.count():,}")
logging.info(f"High Volume FHV Records: {fhvhv_df.count():,}")
logging.info(f"Taxi Zones Records: {taxi_zones_df.count():,}")
# Inspect dataset schemas
print("=== DATASET SCHEMAS ===")
print("\nYellow Taxi Schema:")
yellow_df.printSchema()
print("\nGreen Taxi Schema:")
green_df.printSchema()
print("\nFHV Schema:")
fhv_df.printSchema()
print("\nHigh Volume FHV Schema:")
fhvhv_df.printSchema()
print("\nTaxi Zones Schema:")
taxi_zones_df.printSchema()
```
**Expected Output:**
- Yellow Taxi: \~3,475,226 records
- Green Taxi: \~48,326 records (notably smaller, possibly partial dataset)
- FHV: \~1,894,659 records
- High Volume FHV: \~20,405,666 records (largest dataset)
- Taxi Zones: 265 records
**Why this matters:**
- Caching improves performance for repeated operations, especially critical for the High Volume FHV dataset
- Schema inspection reveals column differences that need standardization
- Record counts help understand the relative scale of each transportation mode
### Step 3: Data Cleaning and Standardization
We clean each dataset to remove invalid records and standardize column names and data types for consistent analysis across all transportation modes.
```python
# Clean Yellow Taxi DataFrame
yellow_df_clean = yellow_df.filter(
(col("tpep_pickup_datetime").isNotNull()) &
(col("tpep_dropoff_datetime").isNotNull()) &
(col("passenger_count").isNotNull()) &
(col("passenger_count") > 0) &
(col("trip_distance") > 0) &
(col("fare_amount") > 0)
).withColumn("pickup_datetime", col("tpep_pickup_datetime")) \
.withColumn("dropoff_datetime", col("tpep_dropoff_datetime")) \
.withColumn("trip_duration", unix_timestamp("tpep_dropoff_datetime") - unix_timestamp("tpep_pickup_datetime")) \
.withColumn("pickup_hour", hour("tpep_pickup_datetime")) \
.withColumn("day_of_week", dayofweek("tpep_pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Clean Green Taxi DataFrame
green_df_clean = green_df.filter(
(col("lpep_pickup_datetime").isNotNull()) &
(col("lpep_dropoff_datetime").isNotNull()) &
(col("passenger_count").isNotNull()) &
(col("passenger_count") > 0) &
(col("trip_distance") > 0) &
(col("fare_amount") > 0)
).withColumn("pickup_datetime", col("lpep_pickup_datetime")) \
.withColumn("dropoff_datetime", col("lpep_dropoff_datetime")) \
.withColumn("trip_duration", unix_timestamp("lpep_dropoff_datetime") - unix_timestamp("lpep_pickup_datetime")) \
.withColumn("pickup_hour", hour("lpep_pickup_datetime")) \
.withColumn("day_of_week", dayofweek("lpep_pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Clean FHV DataFrame (Note: PULocationID is string, cast to double)
fhv_df_clean = fhv_df.filter(
(col("pickup_datetime").isNotNull()) &
(col("dropoff_datetime").isNotNull()) &
(col("pulocationid").isNotNull())
).withColumn("trip_duration", unix_timestamp("dropoff_datetime") - unix_timestamp("pickup_datetime")) \
.withColumn("pickup_hour", hour("pickup_datetime")) \
.withColumn("day_of_week", dayofweek("pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Clean High Volume FHV DataFrame (uses trip_miles instead of trip_distance)
fhvhv_df_clean = fhvhv_df.filter(
(col("pickup_datetime").isNotNull()) &
(col("dropoff_datetime").isNotNull()) &
(col("trip_miles").isNotNull()) &
(col("trip_miles") > 0) &
(col("pulocationid").isNotNull())
).withColumn("trip_duration", unix_timestamp("dropoff_datetime") - unix_timestamp("pickup_datetime")) \
.withColumn("trip_distance", col("trip_miles")) \
.withColumn("pickup_hour", hour("pickup_datetime")) \
.withColumn("day_of_week", dayofweek("pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Apply consistent outlier filtering and capping for all datasets
yellow_df_clean = yellow_df_clean.filter(
(col("trip_duration") >= 60) &
(col("trip_distance") >= 0.1) &
(col("trip_distance").isNotNull()) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))) \
.withColumn("trip_distance", when(col("trip_distance") > 50, 50).otherwise(col("trip_distance")))
green_df_clean = green_df_clean.filter(
(col("trip_duration") >= 60) &
(col("trip_distance") >= 0.1) &
(col("trip_distance").isNotNull()) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))) \
.withColumn("trip_distance", when(col("trip_distance") > 50, 50).otherwise(col("trip_distance")))
fhv_df_clean = fhv_df_clean.filter(
(col("trip_duration") >= 60) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration")))
fhvhv_df_clean = fhvhv_df_clean.filter(
(col("trip_duration") >= 60) &
(col("trip_distance") >= 0.1) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))) \
.withColumn("trip_distance", when(col("trip_distance") > 50, 50).otherwise(col("trip_distance")))
# Verify cleaned data counts
logging.info("=== CLEANED DATA COUNTS ===")
logging.info(f"Cleaned Yellow Taxi Records: {yellow_df_clean.count():,}")
logging.info(f"Cleaned Green Taxi Records: {green_df_clean.count():,}")
logging.info(f"Cleaned FHV Records: {fhv_df_clean.count():,}")
logging.info(f"Cleaned High Volume FHV Records: {fhvhv_df_clean.count():,}")
```
**Key Cleaning Steps:**
- **Remove invalid records**: Null datetimes, zero distances, negative passenger counts
- **Standardize columns**: Create consistent `pickup_datetime`, `trip_duration`, `pickup_hour`, `day_of_week`, and `PULocationID`
- **Handle schema differences**: Cast `PULocationID` to double for FHV datasets, rename `trip_miles` to `trip_distance` for High Volume FHV
- **Cap outliers**: Limit trip durations to 1 hour (3600 seconds) and distances to 50 miles to reduce impact of extreme values
- **Filter edge cases**: Remove trips shorter than 60 seconds or distances less than 0.1 miles
### Step 4: Geographic Context with Taxi Zone Joins
We join each cleaned dataset with the Taxi Zone Lookup table to add borough and zone information for geographic analysis.
```python
# Join all datasets with taxi zones for pickup location context
yellow_df_clean = yellow_df_clean.join(taxi_zones_df, yellow_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
green_df_clean = green_df_clean.join(taxi_zones_df, green_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
fhv_df_clean = fhv_df_clean.join(taxi_zones_df, fhv_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
fhvhv_df_clean = fhvhv_df_clean.join(taxi_zones_df, fhvhv_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
# Filter out records with invalid or unknown geographic information
yellow_df_clean = yellow_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
green_df_clean = green_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
fhv_df_clean = fhv_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
fhvhv_df_clean = fhvhv_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
# Verify final cleaned data after geographic filtering
logging.info("=== FINAL CLEANED DATA COUNTS ===")
logging.info(f"Final Yellow Taxi Records: {yellow_df_clean.count():,}")
logging.info(f"Final Green Taxi Records: {green_df_clean.count():,}")
logging.info(f"Final FHV Records: {fhv_df_clean.count():,}")
logging.info(f"Final High Volume FHV Records: {fhvhv_df_clean.count():,}")
```
**Why Geographic Context Matters:**
- **Market Analysis**: Understanding which boroughs each service primarily serves
- **Operational Insights**: Identifying high-demand zones for resource allocation
- **Competitive Analysis**: Comparing service penetration across different areas
- **Urban Planning**: Supporting transportation infrastructure decisions
### Step 5: Comparative Exploratory Data Analysis
Now we perform comprehensive comparative analysis across all four transportation modes to identify usage patterns, market share, and operational characteristics.
#### Analysis 1: Trip Volume Comparison by Borough
```python
# Add trip_type identifier to each dataset
yellow_df_clean = yellow_df_clean.withColumn("trip_type", lit("Yellow Taxi"))
green_df_clean = green_df_clean.withColumn("trip_type", lit("Green Taxi"))
fhv_df_clean = fhv_df_clean.withColumn("trip_type", lit("FHV"))
fhvhv_df_clean = fhvhv_df_clean.withColumn("trip_type", lit("High Volume FHV"))
# Create unified dataset for comparative analysis
# Note: Excluding trip_distance as FHV dataset lacks this field
combined_df = yellow_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type") \
.union(green_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type")) \
.union(fhv_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type")) \
.union(fhvhv_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type"))
# Calculate trip volume by borough and taxi type
trip_volume_by_borough = combined_df.groupBy("trip_type", "pickup_borough") \
.agg(count("*").alias("num_trips")) \
.orderBy("pickup_borough", "trip_type")
# Convert to Pandas for visualization
trip_volume_by_bpd = trip_volume_by_borough.toPandas()
# Create comprehensive visualization
plt.figure(figsize=(14, 8))
sns.barplot(data=trip_volume_by_bpd, x="pickup_borough", y="num_trips", hue="trip_type", palette="Set1")
plt.xlabel("Pickup Borough", fontsize=12)
plt.ylabel("Number of Trips", fontsize=12)
plt.title("Trip Volume Comparison by Borough and Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.xticks(rotation=45, ha='right')
plt.legend(title="Transportation Type", bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Log key insights
logging.info("=== TRIP VOLUME INSIGHTS ===")
for borough in trip_volume_by_bpd["pickup_borough"].unique():
borough_data = trip_volume_by_bpd[trip_volume_by_bpd["pickup_borough"] == borough]
dominant_service = borough_data.loc[borough_data["num_trips"].idxmax()]
logging.info(f"{borough}: Dominant service is {dominant_service['trip_type']} with {dominant_service['num_trips']:,} trips")
```
#### Analysis 2: Average Trip Duration Comparison
```python
# Calculate average trip duration by transportation type
duration_by_type = combined_df.groupBy("trip_type") \
.agg(avg("trip_duration").alias("avg_duration_seconds")) \
.orderBy("avg_duration_seconds", ascending=False)
# Convert to Pandas and add minutes column
duration_by_type_pd = duration_by_type.toPandas()
duration_by_type_pd["avg_duration_minutes"] = duration_by_type_pd["avg_duration_seconds"] / 60
# Create visualization
plt.figure(figsize=(12, 6))
bars = plt.bar(duration_by_type_pd["trip_type"], duration_by_type_pd["avg_duration_minutes"],
color=['#FF6B6B', '#4ECDC4', '#45B7D1', '#96CEB4'])
plt.xlabel("Transportation Type", fontsize=12)
plt.ylabel("Average Trip Duration (Minutes)", fontsize=12)
plt.title("Average Trip Duration by Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.xticks(rotation=45, ha='right')
plt.grid(axis='y', linestyle='--', alpha=0.7)
# Add value labels on bars
for bar, value in zip(bars, duration_by_type_pd["avg_duration_minutes"]):
plt.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.2,
f'{value:.1f}', ha='center', va='bottom', fontweight='bold')
plt.tight_layout()
plt.show()
# Log duration insights
logging.info("=== TRIP DURATION INSIGHTS ===")
for _, row in duration_by_type_pd.iterrows():
logging.info(f"{row['trip_type']}: Average duration {row['avg_duration_minutes']:.1f} minutes")
```
#### Analysis 3: Top Pickup Zones by Transportation Type
```python
# Calculate top pickup zones for each transportation type
zone_trips_by_type = combined_df.groupBy("trip_type", "pickup_zone") \
.agg(count("*").alias("num_trips")) \
.orderBy("trip_type", "num_trips", ascending=[True, False])
# Select top 5 zones per transportation type using window function
windowSpec = Window.partitionBy("trip_type").orderBy(col("num_trips").desc())
zone_trips_top5 = zone_trips_by_type.withColumn("rank", row_number().over(windowSpec)) \
.filter(col("rank") <= 5) \
.drop("rank")
# Convert to Pandas for visualization
zone_trips_top5_pd = zone_trips_top5.toPandas()
# Create comprehensive visualization
plt.figure(figsize=(16, 10))
sns.barplot(data=zone_trips_top5_pd, x="num_trips", y="pickup_zone", hue="trip_type", palette="Set3")
plt.xlabel("Number of Trips", fontsize=12)
plt.ylabel("Pickup Zone", fontsize=12)
plt.title("Top 5 Pickup Zones by Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.legend(title="Transportation Type", bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(axis='x', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Log top zones for each service
logging.info("=== TOP PICKUP ZONES ===")
for trip_type in zone_trips_top5_pd["trip_type"].unique():
top_zones = zone_trips_top5_pd[zone_trips_top5_pd["trip_type"] == trip_type].head(3)
logging.info(f"\n{trip_type} - Top 3 zones:")
for _, zone in top_zones.iterrows():
logging.info(f" {zone['pickup_zone']}: {zone['num_trips']:,} trips")
```
#### Analysis 4: Hourly Usage Patterns
```python
# Analyze hourly usage patterns by transportation type
hourly_usage = combined_df.groupBy("trip_type", "pickup_hour") \
.agg(count("*").alias("num_trips")) \
.orderBy("trip_type", "pickup_hour")
# Convert to Pandas
hourly_usage_pd = hourly_usage.toPandas()
# Create line plot showing hourly patterns
plt.figure(figsize=(14, 8))
for trip_type in hourly_usage_pd["trip_type"].unique():
data = hourly_usage_pd[hourly_usage_pd["trip_type"] == trip_type]
plt.plot(data["pickup_hour"], data["num_trips"], marker='o', linewidth=2, label=trip_type)
plt.xlabel("Hour of Day", fontsize=12)
plt.ylabel("Number of Trips", fontsize=12)
plt.title("Hourly Usage Patterns by Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.legend(title="Transportation Type")
plt.grid(True, linestyle='--', alpha=0.7)
plt.xticks(range(0, 24))
plt.tight_layout()
plt.show()
# Identify peak hours for each service
logging.info("=== PEAK HOUR ANALYSIS ===")
for trip_type in hourly_usage_pd["trip_type"].unique():
data = hourly_usage_pd[hourly_usage_pd["trip_type"] == trip_type]
peak_hour = data.loc[data["num_trips"].idxmax()]
logging.info(f"{trip_type}: Peak hour is {peak_hour['pickup_hour']}:00 with {peak_hour['num_trips']:,} trips")
```
## Key Findings and Insights
### Market Share Analysis
**Manhattan Dominance:**
- **Yellow Taxis**: Maintain strong presence in Manhattan, particularly in Midtown areas
- **High Volume FHV**: Significant market penetration across all boroughs, with highest absolute numbers
**Outer Borough Patterns:**
- **Green Taxis**: Concentrated in Brooklyn and Queens as intended by regulation
- **FHV Services**: Provide important connectivity to areas less served by traditional taxis
### Service Characteristics
**Trip Duration Patterns:**
- **Shortest**: Green Taxis (optimized for local trips in outer boroughs)
- **Longest**: FHV services (often include airport and longer-distance trips)
- **Medium**: Yellow Taxis and High Volume FHV (balanced mix of trip types)
**Peak Usage Times:**
- **Morning Rush**: 8-9 AM across all services
- **Evening Rush**: 6-7 PM with variation by service type
- **Late Night**: High Volume FHV maintains stronger presence than traditional taxis
### Geographic Insights
**High-Demand Zones:**
- **Airport Access**: JFK and LaGuardia dominate FHV pickup zones
- **Transit Hubs**: Penn Station, Grand Central prominent across services
- **Business Districts**: Midtown Manhattan remains critical for Yellow Taxis
## Conclusion
This comparative analysis reveals distinct usage patterns and market positioning across NYC's transportation ecosystem. **High Volume FHV services** dominate by sheer volume, while **Yellow Taxis** maintain their traditional Manhattan stronghold. **Green Taxis** successfully serve outer borough markets, and **FHV services** provide crucial connectivity for longer trips and airport access.
The analysis demonstrates how different transportation modes complement each other, serving distinct geographic areas, trip purposes, and time patterns. This insight is valuable for:
- **Urban Planners**: Understanding transportation demand patterns
- **Service Operators**: Optimizing fleet deployment and pricing strategies
- **Policy Makers**: Evaluating the effectiveness of transportation regulations
- **Researchers**: Analyzing the evolution of urban mobility
The PySpark-based approach showcases how large-scale transportation data can be efficiently processed and analyzed to extract meaningful insights for data-driven decision making in urban transportation planning.
## Next Steps
To extend this analysis further, consider:
- **Temporal Analysis**: Comparing patterns across different months or seasons
- **Demand Forecasting**: Building predictive models for trip volume by zone and time
- **Network Analysis**: Examining origin-destination patterns and flow dynamics
- **Economic Analysis**: Incorporating fare and revenue data for financial insights
- **Weather Impact**: Analyzing how weather conditions affect different transportation modes
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-pyspark-cheat-sheet.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PySpark Cheat Sheet: Essential and Advanced Functions on OVHcloud Data Platform
## Objective
This cheat sheet provides a concise guide to essential and advanced PySpark functions for data processing on the **OVHcloud Data Platform**. It uses the NYC Yellow Taxi Trip Records for January 2025 (`yellow_tripdata_2025_01.parquet`, \~3.5M records) and the Taxi Zone Lookup Table (`taxi_zone_lookup.csv`, 265 records) as practical examples.
Aimed at intermediate users, it covers core functions (`filter`, `select`, `groupBy`, `join`, `udf`) and advanced ones (`window`, `pivot`, `approx_count_distinct`, `collect_list`, `explode`, `regexp_replace`) for complex transformations and analytics. These examples demonstrate practical applications, making this a versatile reference for any dataset.
## Requirements
Before diving in, ensure you have:
- **Datasets**: (`yellow_tripdata_2025_01.parquet`, `taxi_zone_lookup.csv`) available in your Connectors and accessible in the Lakehouse Manager. You can download these from the [official NYC TLC Trip Record Data website](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
- **Notebook**: A PySpark-enabled Jupyter notebook.
## Setup Instructions
1. **Connectors**: Create a source named "NYC-taxi", upload your files (`yellow_tripdata_2025_01.parquet`, `taxi_zone_lookup.csv`), and extract their schemas.
2. **Lakehouse Manager**: Create corresponding tables in the Lakehouse Manager.
3. **DPE (Data Processing Engine)**: Ensure the data is loaded into these tables within the DPE.
4. **Notebook**: Start a PySpark Jupyter notebook within the OVHcloud Data Platform.
## PySpark Functions Cheat Sheet
### Step 1: Initializing Spark and Loading Data
**Code Block**
```python
from forepaas.dwh import connect
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, unix_timestamp, hour, when, count, avg, udf, approx_count_distinct, collect_list, explode, regexp_replace, row_number
from pyspark.sql.types import FloatType
from pyspark.sql.window import Window
from forepaas.dwh.common import request as dwh_request, DwhRequestException
import io
import logging
# Setup logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Set up variables (DATASET, PROJECT_ID, YEAR, MONTH)
DATASET = "default_dataset"
PROJECT_ID = "PROJECT_ID" # Make sure to replace this with your actual PROJECT_ID
YEAR = "2025"
MONTH = "01"
# Initialize SparkSession
spark = SparkSession.builder.appName("PySpark_Advanced_Cheat_Sheet").getOrCreate()
logging.info(f"Spark Version: {spark.version}")
# Connect to Lakehouse
cn_prim = connect("dwh/default_dataset/")
# Load data
taxi_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.yellow_tripdata_{YEAR}_{MONTH}")
zones_df = cn_prim.query(f"SELECT LocationID, Borough, Zone FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.taxi_zone_lookup")
taxi_df.cache()
zones_df.cache()
logging.info(f"Taxi Records: {taxi_df.count()}")
logging.info(f"Zones Records: {zones_df.count()}")
```
**Functions Used in this Code Block:**
- `SparkSession.builder.appName().getOrCreate()`:
- **What**: Initializes the PySpark session and DataFrame API, which is the entry point for using Spark functionalities.
- **Benefit**: Sets up the environment for distributed data processing.
- `cn_prim = connect("dwh/default_dataset/")`:
- **What**: Establishes a connection to your specified Lakehouse dataset using the `forepaas.dwh` library.
- **Benefit**: Allows you to query and interact with tables stored in your Lakehouse.
- `cn_prim.query(sql_query)`:
- **What**: Executes an SQL query against tables accessible via the `cn_prim` connection. It retrieves the data as a Spark DataFrame.
- **Benefit**: Provides a simple way to load data from your Connectors and Lakehouse into a Spark DataFrame.
- `DataFrame.cache()`:
- **What**: Marks the DataFrame to be cached in memory the first time it's computed. Subsequent actions on this DataFrame will read from cache.
- **Benefit**: Significantly speeds up operations on the same DataFrame, especially useful for iterative algorithms or multiple transformations on a large dataset (\~3.5M records).
- `DataFrame.count()`:
- **What**: Triggers the computation and returns the total number of rows in the DataFrame.
- **Benefit**: Used here to quickly verify that the data has been loaded and to check the record count.
**Output of this Code Block:**
- Spark Version (e.g., `Spark Version: 3.4.1`)
- `Taxi Records: ~3,500,000` (actual count may vary slightly)
- `Zones Records: 265`
### Step 2: Clean and Transform Data
**Code Block**
```python
# Clean and transform
cleaned_df = taxi_df \
.filter(
(col("tpep_pickup_datetime").isNotNull()) &
(col("fare_amount") > 0) &
(col("trip_distance") > 0)
) \
.select(
"tpep_pickup_datetime",
"tpep_dropoff_datetime",
"trip_distance",
"fare_amount",
col("pulocationid").cast("double").alias("pulocationid"),
regexp_replace(col("store_and_fwd_flag"), "^[Yy]$", "Yes").alias("store_and_fwd_flag")
) \
.withColumn(
"trip_duration",
unix_timestamp("tpep_dropoff_datetime") - unix_timestamp("tpep_pickup_datetime")
) \
.withColumn(
"pickup_hour",
hour("tpep_pickup_datetime")
) \
.withColumn(
"trip_duration",
when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))
) \
.filter(col("trip_duration") >= 60)
logging.info(f"Cleaned Records: {cleaned_df.count()}")
```
**Functions Used in this Code Block:**
- `DataFrame.filter(condition)`:
- **What**: Filters rows of the DataFrame based on a given condition, returning a new DataFrame with only the rows that satisfy the condition.
- **Benefit**: Essential for data cleansing, removing invalid or irrelevant records.
- `pyspark.sql.functions.col(column_name)`:
- **What**: References a column in a DataFrame, allowing you to apply various transformations and operations to it.
- **Benefit**: Provides a way to build expressions involving DataFrame columns.
- `DataFrame.select(columns)`:
- **What**: Projects a set of expressions (columns or column-based transformations) and returns a new DataFrame containing only these selected columns.
- **Benefit**: Useful for column subsetting and reducing the DataFrame's size by selecting only relevant fields.
- `Column.cast(dataType)`:
- **What**: Converts the data type of a column to the specified `dataType`.
- **Benefit**: Ensures data type compatibility for calculations or downstream processes (e.g., casting `pulocationid` to `double`).
- `pyspark.sql.functions.regexp_replace(column, pattern, replacement)`:
- **What**: Replaces all occurrences of a string pattern within a column's text data with a specified replacement string.
- **Benefit**: Excellent for data standardization and cleaning string fields (e.g., converting "Y" to "Yes").
- `DataFrame.withColumn(colName, col)`:
- **What**: Returns a new DataFrame by adding a new column or replacing an existing column with the specified expression.
- **Benefit**: Key for feature engineering and creating new derived columns on the fly.
- `pyspark.sql.functions.unix_timestamp(timestamp_column)`:
- **What**: Converts a timestamp string or timestamp column into Unix timestamp (seconds since 1970-01-01 00:00:00 UTC).
- **Benefit**: Facilitates numerical time calculations, such as computing trip durations by subtracting timestamps.
- `pyspark.sql.functions.hour(timestamp_column)`:
- **What**: Extracts the hour component from a timestamp column.
- **Benefit**: Useful for temporal analysis, allowing you to identify hourly patterns or trends in the data.
- `pyspark.sql.functions.when(condition, value).otherwise(other_value)`:
- **What**: Implements conditional logic. If the `condition` is true, the column takes `value`; otherwise, it takes `other_value`. Can be chained.
- **Benefit**: Effective for handling outliers, applying business rules, or categorizing data based on specific conditions (e.g., capping `trip_duration`).
**Output of this Code Block:**
- `Cleaned Records: ~2,700,000 – ~2,800,000` (actual count may vary based on data quality).
### Step 3: Custom Logic with UDF (User-Defined Function)
**Code Block**
```python
# UDF for fare efficiency (fare per minute)
def fare_efficiency(fare, duration):
return fare / (duration / 60) if duration > 0 else 0.0
fare_efficiency_udf = udf(fare_efficiency, FloatType())
# Apply UDF
transformed_df = cleaned_df \
.withColumn("fare_efficiency", fare_efficiency_udf(col("fare_amount"), col("trip_duration")))
transformed_df.show(5)
```
**Functions Used in this Code Block:**
- `udf(func, returnType)`:
- **What**: Registers a Python function as a User-Defined Function (UDF) in PySpark. This allows you to apply custom Python logic to Spark DataFrame columns.
- **Benefit**: Enables tailored, complex calculations that might not be available in native PySpark functions (e.g., calculating "fare per minute" using custom logic).
- `FloatType()` (from `pyspark.sql.types`):
- **What**: Specifies the return data type of the UDF as a Float (single-precision floating-point number).
- **Benefit**: Ensures that the output of your custom function is correctly typed in the Spark DataFrame.
- `DataFrame.withColumn(colName, col)`:
- **What**: (Re-used from Step 2) Adds a new column or replaces an existing one based on the result of an expression.
- **Benefit**: Here, it's used to apply the newly defined `fare_efficiency_udf` to create the `fare_efficiency` column.
**Output of this Code Block:**
- A `DataFrame.show(5)` output, displaying the first 5 rows of `transformed_df`, including the newly added `fare_efficiency` column.
- Example: If `fare_amount` is 15.0 and `trip_duration` is 600 seconds (10 minutes), `fare_efficiency` would be 1.5 ($/min).
### Step 4: Advanced Window Functions and Joins
**Code Block**
```python
# Define window for ranking trips by fare within borough
window_spec = Window.partitionBy("pickup_borough").orderBy(col("fare_amount").desc())
# Join with zones and rank trips
joined_df = transformed_df \
.join(
zones_df,
transformed_df.pulocationid == zones_df.LocationID,
"left"
) \
.withColumnRenamed("Borough", "pickup_borough") \
.drop("LocationID") \
.filter(col("pickup_borough").isNotNull()) \
.withColumn("fare_rank", row_number().over(window_spec))
logging.info(f"Joined Records: {joined_df.count()}")
joined_df.filter(col("fare_rank") <= 3).show()
```
**Functions Used in this Code Block:**
- `Window.partitionBy(*cols).orderBy(*cols)`:
- **What**: Defines a window specification. `partitionBy` divides the rows into groups, and `orderBy` defines the logical order of rows within each partition.
- **Benefit**: Crucial for enabling advanced analytical operations (like ranking, lead/lag, cumulative sums) that operate on a defined subset of rows.
- `pyspark.sql.functions.row_number()`:
- **What**: A window function that assigns a unique, sequential number to each row within its partition, based on the ordering defined in the window specification.
- **Benefit**: Perfect for ranking records (e.g., identifying the top N records based on a metric, like top fares).
- `DataFrame.join(other_df, on=None, how=None)`:
- **What**: Combines two DataFrames based on a specified join condition (`on`) and join type (`how`, e.g., "inner", "left", "right").
- **Benefit**: Enriches data by bringing together related information from different sources (e.g., joining taxi trip data with zone lookup data).
- `DataFrame.withColumnRenamed(existing, new)`:
- **What**: Returns a new DataFrame by renaming an existing column.
- **Benefit**: Helps clarify schema and improves readability, especially after joins where column names might be ambiguous.
- `DataFrame.drop(*cols)`:
- **What**: Returns a new DataFrame with the specified columns dropped.
- **Benefit**: Helps manage DataFrame size and complexity by removing unnecessary columns, saving memory.
**Output of this Code Block:**
- `Joined Records: ~2,600,000 – ~2,700,000` (actual count may vary).
- A `DataFrame.show()` output, displaying rows where `fare_rank` is less than or equal to 3, showing the top 3 fares per borough.
### Step 5: Aggregate and Pivot Data
**Code Block**
```python
# Aggregate: unique zones and trips per borough
agg_df = joined_df \
.groupBy("pickup_borough") \
.agg(
approx_count_distinct("pulocationid").alias("unique_zones"),
count("*").alias("num_trips")
)
# Pivot: avg fare by hour and borough
pivot_df = joined_df \
.groupBy("pickup_hour") \
.pivot("pickup_borough") \
.agg(avg("fare_amount")) \
.orderBy("pickup_hour")
agg_df.show()
pivot_df.show()
```
**Functions Used in this Code Block:**
- `DataFrame.groupBy(*cols)`:
- **What**: Groups the DataFrame by one or more specified columns, preparing for aggregate computations.
- **Benefit**: Enables summarization and analysis of data based on distinct categories or dimensions.
- `DataFrame.agg(*exprs)`:
- **What**: Applies aggregate functions to the grouped data, computing summary statistics.
- **Benefit**: Used to calculate metrics like counts, sums, averages, etc., for each group.
- `pyspark.sql.functions.approx_count_distinct(column)`:
- **What**: Returns an approximate count of distinct items in a group. It uses the HyperLogLog++ algorithm.
- **Benefit**: Significantly faster and more memory-efficient than `countDistinct` for very large datasets when an exact count isn't strictly necessary.
- `pyspark.sql.functions.count(column)`:
- **What**: Counts the number of non-null values in a column or, with `count("*")`, counts all rows in a group.
- **Benefit**: Tallies the size of each aggregated group or the occurrences of specific values.
- `DataFrame.pivot(pivot_column)`:
- **What**: Rotates (pivots) a DataFrame, transforming unique values from a specified column into new columns. Requires a subsequent aggregation.
- **Benefit**: Creates wide tables that are often more suitable for reporting and cross-sectional analysis, allowing direct comparison of values across categories.
- `pyspark.sql.functions.avg(column)`:
- **What**: Computes the average value of a numerical column.
- **Benefit**: Provides a central tendency measure for quantitative data within each group.
- `DataFrame.orderBy(*cols, ascending=True)`:
- **What**: Sorts the rows of the DataFrame based on one or more columns in ascending or descending order.
- **Benefit**: Organizes the output for better readability and to present data in a logical sequence.
**Output of this Code Block:**
- `agg_df.show()`: Displays a table with `pickup_borough`, `unique_zones` (approximate), and `num_trips` (e.g., Manhattan might show \~60 unique zones and \~2M trips).
- `pivot_df.show()`: Displays a pivoted table showing `pickup_hour` as rows and `pickup_borough` as columns, with the average `fare_amount` in each cell.
### Step 6: Collect and Explode Lists
**Code Block**
```python
# Collect zones per borough
list_df = joined_df \
.groupBy("pickup_borough") \
.agg(collect_list("Zone").alias("zones_list"))
# Explode zones list
exploded_df = list_df \
.select("pickup_borough", explode(col("zones_list")).alias("zone"))
exploded_df.show(10)
```
**Functions Used in this Code Block:**
- `pyspark.sql.functions.collect_list(column)`:
- **What**: An aggregation function that gathers all non-null values from a specified column within each group into a Python list.
- **Benefit**: Useful for creating array-like structures where each element corresponds to a record from the original group.
- `pyspark.sql.functions.explode(array_column)`:
- **What**: Transforms a column containing arrays (lists) or maps into individual rows for each element in the array/map. If an array has `N` elements, it creates `N` rows for that original row.
- **Benefit**: Flattens nested data structures, allowing individual elements to be processed or viewed as separate records.
**Output of this Code Block:**
- A `DataFrame.show(10)` output, displaying a table with `pickup_borough` and an `exploded` `zone` column, where each distinct zone for a borough gets its own row (e.g., for Manhattan, you'd see multiple rows like "Manhattan | Midtown", "Manhattan | Upper East Side", etc.).
### Step 7: Save DataFrame to Bucket and Create Table (Advanced)
**Code Block**
```python
def create_table_from_this_dataframe(dataframe, dataset, table_name, bucket, source_bucket):
logging.info(f"We will create a source (bucket) - {source_bucket} where we will store the new table - {table_name} - and automatically load it")
# Creating bucket to store the table
cn_datastore = connect('data_store')
logging.info(f"{cn_datastore.list()} - Before creating new bucket")
cn_datastore.create_bucket(bucket)
logging.info(f"{cn_datastore.list()} - After adding new bucket")
cn_bucket = connect('data_store/' + bucket)
get_dbs = dwh_request(f"v4/databases", method="GET")
data = get_dbs.json()
db_exist = next((item['_id'] for item in data if item.get('display_name') == source_bucket and item.get("package") == "data-store"), None)
if db_exist is None:
# Creating source where we will use the new bucket created to get access to the table
new_source = {"type":"protocol","package":"data-store","parameters":{"path":"","bucket":bucket},"default":False,"level":"source","display_name":source_bucket}
new_source_bucket = dwh_request(f"v4/databases", method="POST", json=new_source)
logging.info(f"New source added with the bucket: {bucket} - source name: {source_bucket}")
else:
logging.info("Source already exist")
# Call to API - To get dataset id
get_database_id = dwh_request(f"v4/databases", method="GET")
database_all = get_database_id.json()
# Filter to get the corresponding _id for the database
dataset_id = next((item['_id'] for item in database_all if item.get('name') == dataset), None)
logging.info(f"dataset_id : {dataset_id}")
# Convert table to Pandas and serialize to CSV in BytesIO
try:
# Convert to Pandas DataFrame
table = dataframe.toPandas()
# Create BytesIO buffer and write CSV
data = io.BytesIO()
table.to_csv(data, index=False, encoding='utf-8')
data.seek(0) # Reset buffer position
# Upload to bucket
file_path = f"{table_name}.csv"
etag = cn_bucket.put(file_path, data, data.getbuffer().nbytes)
logging.info(f"DataFrame uploaded to bucket {bucket}/{file_path} with ETag: {etag}")
# Verify bucket contents
files = cn_bucket.list()
logging.info(f"Bucket contents: {files}")
except Exception as e:
logging.error(f"Failed to save to bucket: {e}")
raise
# Call to API - To add the file into the source
table_config = {"display_name":file_path,"progress":None,"physical_status":None,"parameters":{},"filename":file_path}
res_table = dwh_request(f"v4/databases/{source_bucket}/tables/{file_path}", method="PUT", json=table_config)
# Call to API - To get the corresponding ID for the file_path added in the source
get_template_catalog_object = dwh_request(f"v4/tables", method="GET")
data = get_template_catalog_object.json()
file_path_source_id = next((item['_id'] for item in data if item.get('filename') == file_path), None)
logging.info(f"file_path_source_id : {file_path_source_id}")
# In case the table already exist and you made some modification on it
auto_build_table_DELETE = dwh_request(f"v4/logical/objects/{table_name}", method="DELETE")
# Call to API - To launch the build of the table on the corresponding dataset and load the specific data
config_build = {"database_id":dataset_id,"display_name":table_name,"name":table_name,"type":"prim","load_data":True,"build_table":True,"templated_from":"data_catalog","template_catalog_object":file_path_source_id}
auto_build_table = dwh_request(f"v4/logical/objects", method="POST", json=config_build)
logging.info(f"You can check the build of the table {table_name} on the Lakehouse Manager screen")
# Use the function create_table_from_this_dataframe:
create_table_from_this_dataframe(pivot_df,"default_dataset","taxi_pivot_table","new_bucket","new_source_bucket")
```
**Functions Used in this Code Block (and within `create_table_from_this_dataframe`):**
- `DataFrame.toPandas()`:
- **What**: Converts a Spark DataFrame to a Pandas DataFrame. This collects all distributed data to the driver node.
- **Benefit**: Enables the use of Pandas-specific functions for local data manipulation and file serialization (e.g., `to_csv`).
- **Warning**: Use with caution on very large datasets as it can cause out-of-memory errors on the driver.
- `Pandas_DataFrame.to_csv(path_or_buffer, index=False, encoding='utf-8')`:
- **What**: Writes the Pandas DataFrame to a comma-separated values (CSV) file.
- **Benefit**: Serializes the data into a standard text format suitable for storage and retrieval.
- `io.BytesIO()`:
- **What**: A class from Python's `io` module that creates an in-memory binary stream, acting like a file object.
- **Benefit**: Allows you to write and read bytes as if you were interacting with a physical file, which is useful for direct data transfer to services without saving to disk.
- `cn_datastore.create_bucket(bucket_name)` (from `forepaas.dwh`):
- **What**: Creates a new storage bucket within the OVHcloud Data Platform's data store.
- **Benefit**: Provides a dedicated location to store files, including intermediate or final processed data.
- `cn_bucket.put(file_path, data, size)` (from `forepaas.dwh`):
- **What**: Uploads data (typically from a BytesIO buffer) to a specified path within a connected bucket.
- **Benefit**: Persists your processed data (e.g., the CSV from `pivot_df`) into OVHcloud cloud storage.
- `cn_bucket.list()` (from `forepaas.dwh`):
- **What**: Retrieves a list of files and subdirectories within a connected bucket.
- **Benefit**: Used for verification that files have been successfully uploaded to the bucket.
- `dwh_request(path, method, json)` (from `forepaas.dwh.common`):
- **What**: A utility function to make direct HTTP API calls to the OVHcloud Data Platform's backend services.
- **Benefit**: This function provides granular control to automate tasks like creating data sources, registering files in the Connectors, and triggering table builds in the Lakehouse Manager programmatically, which are not always exposed via the higher-level `connect` object.
**Explanation of this Step's Process:**
This advanced step showcases how to persist a processed Spark DataFrame (`pivot_df`) into a new table in the OVHcloud Data Platform's Lakehouse. It leverages a combination of PySpark, Pandas, and direct API calls:
1. **Bucket Creation**: A new bucket (`new_bucket`) is created in the OVHcloud data store if it doesn't already exist.
2. **Source Creation**: A new data source (`new_source_bucket`) is configured and linked to this bucket, allowing the Connectors to discover files within it.
3. **Data Serialization & Upload**: The `pivot_df` is converted to a Pandas DataFrame, then serialized into a CSV format within an in-memory buffer (`io.BytesIO`). This CSV data is then uploaded to the newly created bucket.
4. **Table Registration & Build**: Via direct API calls (`dwh_request`), the uploaded CSV file is registered as a "physical table" in the Connectors, and a "logical table" (`taxi_pivot_table`) is created in the Lakehouse Manager, triggering a data load from the registered source.
:::warning
**Warning**: This is a temporary solution for saving and creating tables, relying on direct API interactions. The OVHcloud Data Platform SDK is expected to be updated to include more native and simplified SQL transformation and direct storage capabilities in the future, reducing the need for manual API calls and Pandas conversions.
:::
### Step 8: Verify Table Existence
**Code Block**
```python
# Verify table exists in Lakehouse
try:
test_result = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.taxi_pivot_table")
logging.info(f"taxi_pivot_table Records: {test_result.count()}")
except Exception as e:
logging.error(f"Failed to query table - please ensure it's the correct table")
```
**Functions Used in this Code Block:**
- `cn_prim.query(sql_query)`:
- **What**: (Re-used from Step 1) Executes an SQL query against tables in your Lakehouse through the established `cn_prim` connection, retrieving the result as a Spark DataFrame.
- **Benefit**: Used here to confirm that the `taxi_pivot_table` was successfully created and is queryable in the Lakehouse.
- `DataFrame.count()`:
- **What**: (Re-used from Step 1) Returns the total number of rows in the queried DataFrame.
- **Benefit**: Confirms that the newly created table contains the expected data and records.
**Output of this Code Block:**
- If successful: `taxi_pivot_table Records: [Number of rows in pivot_df]` (e.g., `taxi_pivot_table Records: 24` if 24 hours are present).
- If unsuccessful: An error message from the `logging.error` indicating that the table could not be queried.
### Step 9: Clean Up and Stop Spark Session
**Code Block**
```python
spark.catalog.clearCache()
spark.stop()
logging.info("Spark cache cleared and Spark session stopped.")
```
**Functions Used in this Code Block:**
- `SparkSession.catalog.clearCache()`:
- **What**: Clears the Spark's internal cache, releasing memory occupied by cached DataFrames and RDDs.
- **Benefit**: Essential for freeing up resources, especially after complex operations or when you no longer need the cached data. It helps prevent out-of-memory issues in long-running applications or interactive sessions.
- `SparkSession.stop()`:
- **What**: Terminates the SparkSession, shutting down the SparkContext and releasing all associated resources.
- **Benefit**: Ensures a clean shutdown of your Spark application, freeing up cluster resources. It's crucial to call this at the end of your Spark application to avoid resource leaks.
**Output of this Code Block:**
- `INFO - Spark cache cleared and Spark session stopped.`
- This message confirms that the cache has been cleared and the Spark session has been gracefully terminated. You will typically see further output from the notebook environment indicating that the kernel is idle or has shut down.
## Conclusion and Next Steps
This PySpark cheat sheet has provided a hands-on overview of essential and advanced functions for data processing on the OVHcloud Data Platform. You've seen how to initialize Spark, load and clean data, apply custom logic with UDFs, perform complex aggregations with window functions and pivots, and even manage data persistence to your Lakehouse.
This guide serves as a practical reference to accelerate your PySpark development. To continue your journey and apply these skills to real-world scenarios, we encourage you to:
- **Explore the Tutorials**: Dive into the comprehensive [NYC Taxi Dataset Analysis tutorials](https://docs.dataplatform.ovh.net/tutorials-pyspark.md) to build end-to-end data pipelines and machine learning models.
- **Deepen Your Knowledge**: Consult the [official PySpark Documentation](https://spark.apache.org/docs/latest/api/python/) for in-depth details on all functions.
Happy coding and data analyzing!
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-ai-endpoints.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Enhance Your Data Analysis with AI Endpoints on the Data Platform
## Objective
No more worries about imperfect datasets: import, clean, and visualize your data with the **Data Platform**. But that's not all! Benefit from detailed analysis thanks to AI and go even further by asking your LLM to find a Machine Learning rule that fits perfectly with your data.
This guide will demonstrate how to take advantage of [**AI Endpoints**](https://endpoints.ai.cloud.ovh.net/) to enhance your data analysis capabilities on the Data Platform. You'll learn how to start from scratch and generate meaningful predictions from your data.
Here you can find all the **resources** used (CSV files, Python scripts, etc.) for the purpose of this guide. Please download all required files before you begin:
| **Type** | **Name** |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Source file | dirty\_cafe\_sales.csv |
| Preprocessing Python script | custom\_action\_cafe\_sales\_processing.py |
| Preprocessing notebook & LLM request | cafe\_sales\_preprocessing.ipynb |
## Tutorial Steps
### step 1 - Upload CSV file
The first step is to access the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md), create a new source, and choose `File Upload` to upload the CSV file. For the purpose of this guide we will be using the - _Dirty Cafe Sales Dataset_
### step 2 - Analyze the source and define blueprint rules
Access the [Analyser](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) from the Connectors and click on the newly added source to extract the metadata.
Once complete you can start to define the following [blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md) for each of the mentioned attributes:
| **Attribute** | **Blueprint rule** |
| --------------------- | ---------------------------------------------------------------------------------------------------- |
| **payment\_method** | - **Skip line** if _Is null_ - **Skip line** if _Equal_ "ERROR" - **Skip line** if _Equal_ "UNKNOWN" |
| **location** | - **Skip line** if _Is null_ - **Skip line** if _Equal_ "ERROR" - **Skip line** if _Equal_ "UNKNOWN" |
| **transaction\_date** | - **Skip line** if _Is null_ - **Skip line** if _Equal_ "ERROR" - **Skip line** if _Equal_ "UNKNOWN" |
### step 3 - Create table from source
1. Open the tables section by going to the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
2. Click on **Create from a Connectors source** and select the source _`dirty cafe sales.csv`_.
3. Name it dirty\_cafe\_sales (default name)
4. Click on **Create** to build and create the table.
5. Verify that the table has been created correctly. If there are any issues, check the logs for errors.
### step 4 - Duplicate the table
Duplicate this table to get a new empty table with the same attributes and name it `clean_cafe_sales`. Launch the "Build" and check that everything's okay after completion.
:::info
As noted, the duplicated table is currently empty. This occurs because only the table attributes have been duplicated, while the corresponding data still needs to be loaded separately.
:::
### step 5 - Create an empty table for future data visualization
Create a new empty table called `clean_dataset_history` and add the following attributes:
:::info
**Tip**: You can switch to the Canvas view to easily drag and drop the attributes, instead of creating them individually.
:::
- item
- location
- payment\_method
- quantity
- total\_spent
- transaction\_date
### step 6 - Create new attributes
Head to the [Attributes](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) in the Lakehouse Manager, create new Virtual Attributes for data visualization:
| **Attribute Name** | **Virtual Attribute Definition** |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **avg\_quantity\_per\_day\_per\_item** | SUM(CAST(quantity AS DECIMAL(10,2))) / COUNT(DISTINCT CONCAT(CAST(transaction\_date AS VARCHAR), CAST(item AS VARCHAR))) |
| **avg\_sales\_per\_day\_per\_item** | SUM(CAST(total\_spent AS DECIMAL(10,2))) / COUNT(DISTINCT CONCAT(CAST(transaction\_date AS VARCHAR), CAST(item AS VARCHAR))) |
| **avg\_sales\_per\_day\_per\_location** | SUM(CAST(total\_spent AS DECIMAL(10,2))) / COUNT(DISTINCT CONCAT(CAST(transaction\_date AS VARCHAR), CAST(location AS VARCHAR))) |
| **yearmonth** | SUBSTR(CAST(transaction\_date AS VARCHAR), 1, 7) |
it should look like so:
## Processing Options
You now have two options to process and analyze your data:
### Option 1: Automated Processing (Steps 7-9)
- Automate processing with workflow and visualize data for custom analysis
- Create a Data Processing workflow with custom action using a Python script + an Aggregate action (for automation and efficient dataset processing)
- Create Queries from Analytics Manager to do data visualization using the previously created attributes
### Option 2: Step-by-Step Processing with LLM (Steps 10-11)
- Process data step-by-step and benefit from LLM API endpoints for a complete data analysis
- Use a Jupyter notebook to process your dataset step-by-step
- Benefit from powerful LLMs available on AI Endpoints to do a complete data analysis and generate a report
## Option 1: Automated Processing
### step 7 - Clean dataset using your own data processing workflow
If you want to process data quickly and be able to launch it with a single click, you can create a workflow with custom and aggregate actions from the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md).
1. **First, create a new "Custom" action:**
- Upload your Python file `custom_action_cafe_sales_processing.py`
- name the custom action as `cafe_sales_data_processing`
- Choose Python 3.9 version
- Add `numpy` and `pandas` dependencies (type in the library name and press enter) and click on **Create**
- Now you can **RUN** the custom action.
Also, you should see logs similar to below to double check if what you have done is correct:
```
2025-02-27 08:59:09 [NOTICE] "cafe_sales_data_processing" SUBMITTED
2025-02-27 08:59:09 [NOTICE] "cafe_sales_data_processing" QUEUED
2025-02-27 08:59:10 [NOTICE] "cafe_sales_data_processing" BUILD SUBMITTED
...
2025-02-27 09:00:41 [NOTICE] "cafe_sales_data_processing" Stop cleaning
2025-02-27 09:00:41 [NOTICE] "cafe_sales_data_processing" END (custom) action. Duration: 6.65 sec
...
2025-02-27 09:00:44 [NOTICE] "Flush dataplant cache" END (forepaas-flushall) action. Duration: 2.14 sec
```
2. **Then, create the "Aggregate" action as follows:**
- Configure the aggregate action settings according to your requirements
3. **Then, create the "Delete" action as follows:**
- Configure the delete action settings according to your requirements (this action will ensure you are using a clean table every time)
4. **Finally, define the processing workflow:**
- Set up your workflow to connect the delete, custom action and aggregate action
Your dataset is now clean.
### step 8 - Explore clean tables
Now, you can check if everything's okay:
1. **From the Lakehouse Manager Tables**, you should see your created tables
2. **From the Lakehouse Manager [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md)**, you can explore the `clean_cafe_sales` table
### step 9 - Data visualization using custom attributes
Now we can take an even more closer look at our data visually with the help of [queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
1. **Create quantity\_per\_item query**
- You should see the average quantity sold per day for each item
2. **Create total\_spent\_per\_item query**
- You could see the average total amount sold per day for each item
3. **Create sales\_per\_location query**
- You could see the average total amount sold per day for each location
You can go further by creating more queries and then add them to a Dashboard which you can then share with others.
## Option 2: Step-by-Step Processing with LLM
### step 10 - Clean dataset and save it into S31
bucket
If you want to follow the data cleaning process step by step, you can create a new notebook from the Data Processing service.
1. **Create a "Base notebook" with:**
- Python 3.9 version
- Install `numpy`, `pandas`, `matplotlib`, and `seaborn` dependencies
2. **Drag and drop the or click upload, to upload the file `cafe_sales_preprocessing.ipynb` into the jupyterhub file siebar**
### step 11 - One-shot data analysis with LLM AI Endpoints
A single request is enough to generate all the Python code you need for dataset analysis!
1. **Use the Llama 3.3 70B Instruct model** with the Python code that will allow you to analyze your dataset easily
2. **Generate an AI Endpoints API key** and use it in the designated line as seen below in the screenshot
- Access [here](https://endpoints.ai.cloud.ovh.net/)
3. **Use the generated Python code** to analyze your data and display graphs
The advantage of asking an LLM to generate code in Python is that you will be able to reuse it when you add new data to your coffee sales in subsequent months, ensuring a consistent analysis method.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-billing-analysis.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# OVHcloud API: Billing Data Analysis Configuration
## Objective
This guide provides step-by-step instructions on configuring multiple endpoints for the OVHcloud Billing API. You'll learn how to retrieve billing data and integrate it into your applications for accurate and streamlined financial tracking.
## Overview
The OVHcloud API supports multiple endpoint configurations, allowing you to fetch and integrate data from interconnected resources. This guide provides a comprehensive walkthrough for configuring and utilizing the OVHcloud Billing API endpoints. By following these steps, you'll be able to programmatically retrieve, process, and analyze your billing information efficiently.
## Requirements
Before beginning, ensure you have:
- An active OVHcloud account with billing access
- API credentials, specifically:
- Application Key (App Key)
- Application Secret (App Secret)
- Consumer Key
- Access to the OVHcloud API Developer Portal
- Basic understanding of API endpoint interactions
## Step-by-Step Billing Endpoint Configuration
## 1. Initialize API Connection
Configure your OVHcloud API connection by following the authentication setup process detailed in the OVHcloud API Connector Documentation. This involves obtaining and securely inputting your Application Key, Application Secret, and Consumer Key through the connector's user interface. For comprehensive instructions on acquiring and setting up these credentials, consult the [API Connection Details](https://docs.dataplatform.ovh.net/connectors-sources-ovhcloud-api.md) section in the connector documentation.
## 2. Define the billing endpoints
### 2.1. Retrieve Account Bills IDs
- Endpoint: `/me/bill`
- Purpose: Fetch a list of all bills IDs for your account
Steps:
1. Select "Add New Endpoint" in the connector
2. Enter `/me/bill` as the endpoint path
3. Configure query parameters (e.g., filtering by year) - for this example we have used key:`date.from` and value:`2023`, to get the records from the year 2023.
4. Make sure that endpoint has a unique name `bills_ids_from_2023`
5. Click "Confirm" and "Save"
:::info
We can include an additional step to confirm whether the connected endpoint is correct. While this step is not mandatory, it can be helpful the first time to get an idea of the data you are looking for.
:::
#### Validate Endpoint Data
1. Navigate to the Analyzer tab
2. Extract metadata for the new endpoint
3. Verify bill records are correctly retrieved
### 2.2. Retrieve Account Bills Details IDs
- Endpoint: `/me/bill/{billId}/details`
- Purpose: Fetch the list of IDs for each invoice line details
Steps:
1. Click "Add Endpoint"
2. Enter `/me/bill/{billId}/details` as the endpoint path
3. Configure multiple endpoint selection
4. Link to the previous bills endpoint
5. Select `{billId}` as the path parameter
6. Make sure that endpoint has a unique name `bills_details_ids_from_2023`
7. Save the configuration
:::info
Use unique endpoint names to simplify source management. This will be useful for step 4. Import your data in the lakehouse.
:::
:::warning
**Note**: Currently, only one path parameter can be dynamically selected. Manual entry might be required for complex endpoints with multiple path parameters.
:::
Returning to the main modal by clicking on the Settings tab, you'll notice that the path parameter and endpoint fields are now non-editable. This is because their values are being derived from another endpoint. If you need to make changes, simply go back to the multiple endpoints section and select 'No multiple endpoint' from the endpoint selection dropdown.
### 2.3. Retrieve Account Bills Details
- Endpoint: `/me/bill/{billId}/details/{detailsID}`
- Purpose: Fetch the details of each invoice line
Steps:
1. Click "Add Endpoint"
2. Enter `/me/bill/{billId}/details/{detailsID}` as the endpoint path
3. Configure multiple endpoint selection
4. Link to the previous bills details ID endpoint
5. Select `{billId}` and `{detailsID}` as the path parameter
6. Make sure that endpoint has a unique name `bills_details_from_2023`
7. Save the configuration
### 2.4. Retrieve services IDs
- Endpoint: `/services`
- Purpose: Fetch the IDs of all the services of your accounts
Steps:
1. Click "Add Endpoint"
2. Enter `/services` as the endpoint path
3. Make sure that endpoint has a unique name `services_ids`
4. Save the configuration
### 2.5. Retrieve services information
- Endpoint: `/services/{serviceId}`
- Purpose: Fetch information about your services to enrich the data in the pipeline
Steps:
1. Click "Add Endpoint"
2. Enter `/services/{serviceId}` as the endpoint path
3. Configure multiple endpoint selection
4. Link to the previous service IDs endpoint
5. Select `{serviceId}` as the path parameter
6. Make sure that endpoint has a unique name `services`
7. Save the configuration
The configuration of the endpoints is now over, you should see the following:
## 3. Metadata extraction
With the endpoints now established, navigate to the analyzer tool within the Connectors. Once there, select the endpoints you have recently configured. Allow some time for the system to initiate and complete the processing jobs associated with these endpoints.
:::warning
The duration of this process may vary depending on the complexity of the tasks and the volume of data being processed.
:::
## 3.1. Reformat data
The data extracted from the sources can be modified to match the exact format we want, or to bucketize it through the blueprints. This is what we are going to do for 2 endpoints:
- `bills_details_from_2023`
- `services`
### 3.1.1. Reformat bills details
The field `periodstart` from the API is a value with the format YYYY-MM-DD, with the DD that can be any day of the month, depending on the day you purchased the service. We are going to use a blueprint to override this day to be set on the first of each month.
Click on top right of the data analysis on `New Rule`, then select `Date format`.
You are now on the configuration of the blueprint, please fill as the following:
- Attribute: `periodstart`
- Current Date Format: `%Y-%m-%d`
- Replace by: `%Y-%m-01`
- Condition: `Is not null`
Click on confirm, you can now see the `periodstart` has 1 blueprint.
### 3.1.2. Reformat services
Many datetime fields from the `services` endpoint contains the timezone information. We want to remove this timezone information from the datetime, so we will reformat all the dates with the `Date Format` blue print, as seen previously.
Go to the `services` endpoint, click `New rule`, select `Date format`.
You are now on the configuration of the blueprint, please fill as the following:
- Attribute: `billing_next_billingdate`
- Current Date Format: `%Y-%m-%dT%H:%M:%S%z`
- Replace by: `%Y-%m-%dT%H:%M:%S`
- Condition: `All`
You can repeat the operation for the following attributes:
- `billing_expirationdate`
- `billing_lifecycle_current_terminationdate`
- `billing_lifecycle_current_creationdate`
- `billing_renew_current_nextdate`
## 4. Import your data in the lakehouse
With the data sources now configured, we can proceed to construct the tables. You can go into the `Lakehouse Manager`, and the `Tables` section. Click on the `+` and `Create from a Connectors source`.
The first table will be populated using the `bills_details_from_2023` data source. Click `Next`, rename the table `raws_bills_details`, then click `Create`.
You can now also import the services into a dedicated table:
## 5. Add a tag on services
In order to categorize the expenses, we want to categorize the services with tags, that can represent either your technical environments (e.g. production, dev) or your company's departments or even your customers.
In order to extract the first list of services to categorize, you can go into the `Explorer` section inside the `Lakehouse Manager`.
In the Data, select the `domain`, click on `select` and replace by `select_distinct` and then click on `RUN`.
You should have a result like the following:
You can now click on `Download` and `As a CSV`. You will now have a first CSV file with all the resources to tag. You can open it, rename the column `select_distinct_domain` by `domain`, and then add a second column `tag` and for each domain you want to tag, add the value that you want. You should end up with a CSV file like the following:
It is now time to add a new datasource in the `Connectors` with this CSV file, and create a table in the lakehouse out of it. You can go into `Connectors` -> `sources` -> `New Source` -> `File Upload`. You can name your new source `Domains Tagged` and Add the CSV file and hit the `create` button.
You can new go to the analyzer to check the data is well evaluated, and also add a blueprint on the tags, so we will replace the empty values by `unknown`. Click on `new rule`, `Replace By`. Fill the form as follow:
- Attribute: `tag`
- Replace by: `unknown`
- Condition: `Equal`
- Equal: leave empty
Confirm the blue print.
You can now return to the `Lakehouse Manager`and `Tables` section, and create a table from this new source. Be careful to name the table `domains_tags`.
## 6. Create an aggregation table
Now we have a lot of information about our services, theirs bills and also tags. We are going to build an aggregation table to link all the information together.
Inside the `Lakehouse Manager`and `Tables` section, click on the `+` and `Create a new empty table`. Name it: `enriched_bills_details`.
You should now have the following tables in your lakehouse:
- `services`
- `domains_tags`
- `raws_bills_details`
- `enriched_bills_details`
We are going to drag some fields from the tables we created from sources, and drop them into the `enriched_bills_details`. The fields to drag and drop are:
- From `raws_bills_details`:
- `domain`
- `periodstart`
- `totalprice_value`
- From `domains_tags`:
- `tag`
- From `services`:
- `resource_displayname`
- `resource_product_name`
- `resource_product_description`
Your `enriched_bills_details` table should now look like:
You can now click on `Build` to build this new table.
Now we have the structure of this new table, we are going to create a job to extract the data from the previous 3 tables and load it into this new table. We need to go now inside the `Data Processing Engine` and `Actions` section. Click on `New Action` and select `Aggregate`.
We are now going to map fields from different table into the `enriched_bills_details`, for that, you need first to select the sources and destination:
- Sources:
- `raws_bills_details`
- `services`
- `domains_tags`
- Destination: `enriched_bills_details`
For the sources, we need to explain how the data from the 3 tables are linked together through the SQL JOIN syntax. We are going to add then the following joins:
- Between `raws_bills_details` and `services`:
```sql
raws_bills_details.domain = services.resource_name
```
- Between `raws_bills_details` and `domains_tags`:
```sql
raws_bills_details.domain = domains_tags.domain
```
To finish the aggregate, we need to configure how to select and group attributes of the tables into enriched\_bills\_details. Map the attributes as shown in the screenshot below.
You can now create the aggregate, and run the action.
## 7. Prepare the queries and build the Dashboard
Now we have all the data ready to be exploited, we can go into the `Analytics Manager` and `Queries`.
Let's create some queries.
### Cost evolution vs previous month
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `periodstart`
- Order by: `periodstart desc`
For the visualization:
- As a metric
- Customization: enable all the toggle buttons
### Cost repartition by tag
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `tag`
For the visualization:
- As a pie chart
- Customization:
- Show category
- Show % of total
- Position: outside
- Hide legend
### Cost repartition by product
For the query:
- data: `totalprice_value` sum FROM `enriched_bills_details`
- scale: `resource_displayname`
For the visualization:
- As a pie chart
- Customization:
- Show category
- Show % of total
- Position: outside
- Hide legend
### Cost repartition by product type
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `resource_productdescription`
For the visualization:
- As a pie chart
- Customization:
- Show category
- Show % of total
- Position: outside
- Hide legend
### Cost per month and tag
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `periodstart` `resource_product_description` `tag`
- Filter on: `periodstart` `Not null`
- Order by: `periodstart` `asc`
For the visualization:
- As bar charts
- Customization:
- Display stacked
- Hide legend
- X-axis: regroup by periodstart
### Create the Dashboard
You can now go into the dashboard section, create a new one and put all your charts into this dashboard by clicking the + button.
On top of the dashboard, you can also add filters to ease the naviation into the data. Consider adding the following filters:
- `periodstart`
- `tag`
- `resource_product_description`
## Conclusion
By following these steps, you can effectively configure and utilize the OVHcloud Billing API endpoint to retrieve, process, and analyze billing information programmatically.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Deploying Apache Superset on OVHcloud Data Platform
## Objective
This guide provides a comprehensive walkthrough for deploying Apache Superset on the OVHcloud Data Platform, covering installation, configuration, and Trino integration for a secure and reliable setup.
**Before you begin:**
You will need an active Trino connection string. The easiest way is to create a [**Trino consumer**](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) in the Connectors; alternatively, build one manually by following [Integrating Trino with External Tools](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md). This connection string is crucial for configuring Trino within Apache Superset.
## Steps to Connect Apache Superset to Data Platform via Trino
### Step 1: Add Apache Superset into Application Services
1. Navigate to your project.
2. Click **Create an application** in the tile **Application Services**.
3. Select **Apache Superset**.
4. In the Apache Superset Store, select the Template you want (for this tutorial, we will select v5.0.0 with SQLite).
:::info
The SQLite file will be stored in the **appservice**
folder of the S31
bucket (Lakehouse Manager section). We highly recommend backing up this file regularly.
:::
5. Provide a name for the new application (e.g., `dashboard`).
6. Build the application.
7. Once built, deploy the application by clicking on the **Deploy** button.
### Step 2: Customize the Template (Editor Tab)
In the **Editor** tab of your application, you have access to the full source code of the template. You can modify it as needed.
This includes:
- The overridden **Apache Superset source code** used in the template.
- A technical **`README.md`** file that explains the structure and customization points of the template.
- The **`entrypoint.sh`** script, which manages backup and restoration of the **`sqlite.db`** file.
- The standard **`forepaas.json`** file, present in all AppService deployments, which defines:
- The version of Apache Superset being used
- The configuration for **role mapping** and runtime parameters
:::info
Modifying this template allows advanced customization of Superset behavior, layout, and deployment logic.
:::
### Step 3: Configure Trino in Apache Superset
1. Open Apache Superset by clicking on the **Open** button.
2. Click **Sign In With DataPlatformSSO**. Once redirected, you will be connected to Superset.
3. In the **Settings** menu, click **Database Connections**.
4. Click **+ Database**.
5. Select **Trino** in the **Supported databases** list.
6. Paste the **full Trino connection string** (from your [Trino consumer](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), or built manually per the [Integrating Trino with External Tools](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md) guide) into the **SQLAlchemy URI** field and validate the form.
### Step 4: Create Your First Chart
1. Click **`Datasets`** in the main menu.
2. Click the **`+ Dataset`** button.
3. Once the **New dataset** page is loaded, fill out the form with the following:
- **Database**: The Trino database you have just created.
- **Schema**: The catalog you want to query (e.g., `db_m8obb5ch_default_dataset`).
- **Table**: A table from your selected catalog (e.g., `dataset_history`).
4. Click the **Create dataset and create chart** button.
5. Once created, you will be redirected to the **Create a new chart** form.
6. Select a chart (e.g., **Pie Chart**).
7. Complete the **Query** fields (e.g., **Dimensions**: **`cat_temperature`** & **Metric**: **`Sum(rides)`**).
8. Click the **Create chart** button.
:::info
For more information, see the [Apache Superset documentation](https://superset.apache.org/docs/intro/).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-finops-focus-superset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# FinOps uses case with FOCUS and superset
## Objective
This guide provides step-by-step instructions on configuring your Data Project to take your first steps inside the FinOps with a FOCUS file.
## Overview
[FOCUS](https://focus.finops.org/) is a specification inside the FinOps world to expose data consumption throughout multiple cloud providers. This guide will provide an example on how to consume this kind of file inside the Data Platform, and leverage the power of superset to create useful dashboards.
## Requirements
Before beginning, ensure you have:
- An active OVHcloud account with billing access
## Step-by-Step configuration
## 1. Retrieve the FOCUS file
We need to have a FOCUS formatted file to develop this use case. You can easily extract a CSV file with your OVHcloud account consumption by following this link: [OVHcloud Focus Extractor](https://finops.labs.ovh.net/).
You can login on the interface through the OVHcloud SSO. This tool is a POC developped by our OVHcloud internal team meanwhile we are working to provide a more standard way to extract a FOCUS formatted file.
Once logged, you can click on "Load Services Consumption".
When the table is created, you can now click on "Export to CSV" to extract a complete file with your consumption.
You have now a FOCUS formatted CSV that can be loaded inside the Data Platform
## 2. Load the FOCUS file inside the Lakehouse
### 2.1. Create the Data Source
You can now go to the [`Data Platform`](https://eu.dataplatform.ovh.net), inside the `Project` of your choice, then `Connectors` section, and to finish `Sources` in order to create a new `File Upload` source.
You can name your source `focus`, and add the `export.csv` file you downloaded inside the previous step. You can now click on `Create`.
Now the file is loaded, you can go into the `Explorer` section, and then click on your new source `finops` and then start the analysis of the file on clicking `export.csv`. After few seconds, your file should be analyzed successfully.
### 2.3. Load the data into the Lakehouse
You can now go into the `Lakehouse Manager` section, and then `Tables`. You can hit `New Table` and `Create from a Connectors source`. Select your `export.csv` file, then hit `next`, rename the table into `consumption` and `create`.
## 3. Create the Superset Application
Now all the data is ready to be analyze in the Lakehouse, you need to create the Apache Superset Application. You can do so by following [this tutorial](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md).
Considering this tutorial, we consider you followed it until the `Step 5: Create your first Chart` and until the \`3. New Dataset' where you create the Dataset for:
- Database: `trino`
- Schema: `Default dataset`
- Table: `consumption`
You can now hit `Create Dataset and create dashboard`
## 4. Create the Charts
For now, we are going to create multiple charts, always from the same process:
- Click `Charts`, then `+ Chart`
- Chose the dataset `consumption`
We are going through multiple chart you can create:
### Effective cost - current
- Type: `Big number with Trendline`
- Temporal axis: `billingperiodstart`
- Metric: `Billed Cost` aggregate `SUM`
- Comparison Period Lag: `1`
- Show timestamp: true
### Effective cost - previous month
- Type: `Big number`
- Metric: `Billed Cost` aggregate `SUM`
- Filters: `billingperiodstart` with `Range type: advanced` as per the screenshot
- Display settings
- Subtitle: `1 month ago`
### Effective cost - 2 month
- Type: `Big number`
- Metric: `Billed Cost` aggregate `SUM`
- Filters: `billingperiodstart` with `Range type: advanced` and `- 3 month` and `-2 month`
- Display settings
- Subtitle: `2 month ago`
### Most popular region by effective cost
- Type: `Word Cloud`
- Dimension: `regionname`
- Metric: `billedcost` with aggregate `SUM`
### Total accounts
- Type: `Big number`
- Metric: `billingaccountname` aggregate `COUNT DISTINCT`
- Filters: `billingperiodstart` with `Last Month`
- Display settings
- Subtitle: `Previous month`
### Total services
- Type: `Big number`
- Metric: `resourceid` aggregate `COUNT DISTINCT`
- Filters: `billingperiodstart` with `Last Month`
- Display settings
- Subtitle: `Previous month`
### Total providers
- Type: `Big number`
- Metric: `resourceid` aggregate `COUNT DISTINCT`
- Filters: `billingperiodstart` with `Last Month`
- Display settings
- Subtitle: `Previous month`
### Effective cost per service
- Type: `mixed chart`
- X-Axis: `billingperiodstart`
- Time Grain: `Month`
- Query A:
- Metrics: 'billedcost' with aggregate 'SUM'
- Dimension: `Custom SQL` with `CONCAT(servicecategory, '-', servicesubcategory)`
- filters: `billingperiodstart`
- Query B:
- Metrics: 'billedcost' with aggregate 'SUM'
- filters: `billingperiodstart`
- Customize:
- Query A:
- Serie type: `Bar`
- Stack serie: `true`
### Cost repartition per service category and subcategory
- Type: `sunburst chart`
- hierarchy: `servicecategory` and `servicesubcategory`
- primary metric: `billed cost`
- Filters:
- `billingperiodstart`: 'Last month'
### Service Category Effective cost per region
- Type: `Sankey chart`
- Source: `region name`
- Target: `servicecategory`
- Metric: `billedcost`
- Filters: `billingperiodstart`
### Effective cost per sub account name
- Type: `Partition Chart`
- Metrics: `billedcost` aggregate by `SUM`
- Filters: `billingperiodstart`
- Dimensions: `billingaccountname`, `subaccountname`, `servicecategory`
## 5. Create the Dashboard
Now you have a lot of charts, you can put them together into a single place: the Dashboards. Click on `Dashboard`, then `Create`. You can name your dashboard `Billing Summary`, and drag and drop all the charts you previously created to finalize the view.
## Conclusion
By following these steps, you can effectively configure and utilize the OVHcloud Data Platform to implement your first advanced FinOps use case based on the FOCUS format.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Prepare online store orders for a sales dashboard
## Objective
This guide walks you through loading two years of orders from an online store (about 1 million order lines) into the Lakehouse, concatenating them into a single table and cleaning them with a Custom action. You upload the files with [Connectors](https://docs.dataplatform.ovh.net/connectors-sources-upload-files.md), create the tables in [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md), then concatenate and clean the orders with [Data Processing Engine](https://docs.dataplatform.ovh.net/dpe-actions-load.md).
By the end you will have the table `online_retail_ii_clean`, ready to query from Apache Superset. The second part of the guide, [Build a sales dashboard for an online store in Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md), turns it into a sales dashboard.
## Requirements
To follow this guide you need:
- A Data Platform project. This guide uses a project named `ecommerce-sales`.
- The orders of the store and the cleaning script of the Custom action:
| Type | Name |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Orders, December 2009 to December 2011 | [Online Retail II](https://archive.ics.uci.edu/dataset/502/online+retail+ii), UCI Machine Learning Repository |
| Cleaning Python script | online\_retail\_cleaning.py |
The UCI publishes the orders as one Excel workbook, `online_retail_II.xlsx`, with one sheet per year. Before Step 1, save each sheet as its own file:
| Sheet | File to create |
| ---------------- | ------------------------------ |
| `Year 2009-2010` | `online_retail_II_200910.xlsx` |
| `Year 2010-2011` | `online_retail_II_201011.xlsx` |
:::info
**About the data:** [Online Retail II](https://archive.ics.uci.edu/dataset/502/online+retail+ii), UCI Machine Learning Repository, is licensed under CC BY 4.0 (DOI 10.24432/C5CG6D).
:::
We recommend completing the [Getting Started guides](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) first. This guide assumes you are comfortable with the main components of Data Platform.
## What you will build
The two parts of this guide end with this dashboard in Superset:
| Stage | Component | Result |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| 1. Upload the files | [File Upload source](https://docs.dataplatform.ovh.net/connectors-sources-upload-files.md) | A source holding the two files |
| 2. Analyse the files | [Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) | Detected attributes and types |
| 3. Create the tables | [Lakehouse Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) | One table per year, one table for all orders |
| 4. Concatenate the years | [Load actions](https://docs.dataplatform.ovh.net/dpe-actions-load.md) and a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) | 1,067,371 order lines in `online_retail_ii` |
| 5. Clean the orders | [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) | 1,021,128 clean order lines in `online_retail_ii_clean` |
| 6. Connect Superset | [Trino consumer](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) and [Apache Superset](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) | A Superset dataset on the clean table |
## Instructions
### Step 1: Upload the files
1. Open **Connectors** > **Sources** and create a new source of type _File Upload_.
2. Name it `online_retail_ii`.
3. Drop `online_retail_II_200910.xlsx` and `online_retail_II_201011.xlsx` in the _Files_ area.
4. Click **Create**.
### Step 2: Analyse the files
1. Open **Connectors** > **Analyzer**.
2. Select the source `online_retail_ii`, then the file `online_retail_II_200910.xlsx`.
3. Click **Extract metadata**.
4. Do the same for `online_retail_II_201011.xlsx`.
The [Analyzer](https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md) detects 8 attributes: `invoice`, `stockcode`, `description`, `quantity`, `invoicedate`, `price`, `customer_id` and `country`. Check that `invoice` and `stockcode` are detected as **String**: invoice numbers such as `C489449` and product codes such as `85123A` are not numbers.
### Step 3: Create the tables
#### Create one table per year
1. Open **Lakehouse Manager** > **Tables**.
2. Click **New table** > **Create from a Connectors source**.
3. Select the file `online_retail_II_200910.xlsx` from the source `online_retail_ii`, then click **Next**.
4. Keep the default settings and click **Create**. The table is named `online_retail_ii_200910`, in the dataset `default_dataset`.
5. Repeat with `online_retail_II_201011.xlsx`. The table is named `online_retail_ii_201011`.
Both tables appear with 8 attributes and about 525k and 542k rows.
#### Create the table for all orders
1. [Duplicate](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#duplicate) the table `online_retail_ii_200910`. Hover over its row and click the **Duplicate** icon.
2. Name the new table `online_retail_ii`, in the dataset `default_dataset`, and click **Create**.
3. [Build](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#build-table) the table.
The table has the same 8 attributes, and no rows yet.
### Step 4: Concatenate the two years
#### Create the Load actions
1. Open **Data Processing Engine** > **Actions** and click **New action**.
2. Search for `load`, select the _Load_ action and click **Select**.
3. Fill in the action:
| Setting | Value |
| ------------ | -------------------------------------------------------------------------- |
| Source table | `online_retail_II_200910.xlsx` from `online_retail_ii` |
| Destination | `online_retail_ii` from `default_dataset` |
| Attributes | each source attribute mapped to the destination attribute of the same name |
4. Click **Create**.
5. Create a second _Load_ action, with `online_retail_II_201011.xlsx` as source and the same destination.
The list of actions shows your two _Load_ actions, next to the two actions generated automatically when you created the yearly tables.
#### Run both actions with a workflow
1. In **Data Processing Engine** > **Workflows**, click **New workflow**.
2. Name it `Concat`.
3. In _Stage 1_, add your two _Load_ actions.
4. Click **Create**, then run the workflow.
The workflow takes a couple of minutes. When its status turns green, open **Lakehouse Manager** > **Tables**: `online_retail_ii` holds the rows of both yearly tables, **1,067,371 rows**.
:::warning
Run the `Concat` workflow once. Running it again may add the same order lines a second time. After each run, check that `online_retail_ii` holds 1,067,371 rows.
To run the workflow again, you can create a [Delete action](https://docs.dataplatform.ovh.net/dpe-actions-delete.md) that empties `online_retail_ii`, and run it before the workflow.
:::
### Step 5: Clean the orders
#### Create the clean table
1. In **Lakehouse Manager** > **Tables**, duplicate `online_retail_ii`.
2. Name the new table `online_retail_ii_clean` and build it.
You now have 4 tables. `online_retail_ii_clean` has the same 8 attributes and no rows yet.
#### Create the cleaning action
1. In **Data Processing Engine** > **Actions**, click **New action**.
2. Search for `custom`, select the _Custom_ action and click **Select**.
3. Fill in the action:
| Setting | Value |
| -------------------- | ---------------------------------- |
| Action name | `online_retail_cleaning` |
| Function name | `online_retail_cleaning` |
| Development language | `Python 3.11` |
| Python requirements | `pandas` |
| File | upload `online_retail_cleaning.py` |
4. Click **Create**, then run the action.
:::info
**Note:** the function name is the entry point of the script and must be `online_retail_cleaning`. The other functions of the file are helpers: setting one of them as the function name returns `'Event' object is not subscriptable`.
:::
The action reads `online_retail_ii`, applies the cleaning rules and writes the result to `online_retail_ii_clean`. Open its logs: each rule reports how many rows it affects, and the last line gives the number of rows written to `online_retail_ii_clean`: **1,021,128 rows**.
| Rule | A row is removed when |
| ------------------------- | --------------------------------------------------------- |
| 1. Exact duplicates | it is identical to another row on the 8 columns |
| 2. Invalid invoice number | the invoice is not 6 digits, with an optional `C` |
| 3. Invalid stock code | the stock code is not 5 digits, with optional letters |
| 4. Missing description | the description is empty |
| 5. Inconsistent quantity | a sale is not positive, or a cancellation is not negative |
| 6. Zero or negative price | the price is not strictly positive |
| 7. Missing invoice date | the date is empty |
A final step, **8. Harmonized descriptions**, gives each stock code its most frequent description, so a product never shows under two names.
#### Create an action to empty the clean table
The cleaning action adds rows to `online_retail_ii_clean`. Before running it again, empty the table with a [Delete action](https://docs.dataplatform.ovh.net/dpe-actions-delete.md), as for `online_retail_ii` before running the `Concat` workflow again.
1. Click **New action**, search for `delete`, select the _Delete_ action and click **Select**.
2. In _Table list_, move `online_retail_ii_clean` to _Table to empty_.
3. Click **Create**.
To clean the data again, run this _Delete_ action first, then `online_retail_cleaning`.
### Step 6: Connect Superset
All the data is now ready in the Lakehouse. To analyse it, create an Apache Superset application:
1. Create a Trino connection by following the [Generate a Trino connection string for external tools](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) guide.
2. Create the application and connect it to Trino by following Steps 1 to 3 of the [Deploying Apache Superset on OVHcloud Data Platform](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) guide.
#### Create the dataset in Superset
1. In Superset, open **Datasets** and click **+ Dataset**.
2. Select the database connection you created, named `Trino DP` in this guide, then the schema `default_dataset` and the table `online_retail_ii_clean`.
3. Check the columns, then click **Create and explore dataset**.
Every chart of the dashboard then starts from **Charts** > **+ Chart**: choose the dataset `online_retail_ii_clean`, then the chart type.
Continue with [Build a sales dashboard for an online store in Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md).
## Good to know
- **You can connect a real store.** This guide uses a public dataset, uploaded as files. To analyse the orders of your own store, create a [Shopify](https://docs.dataplatform.ovh.net/connectors-sources-shopify.md) or [Stripe](https://docs.dataplatform.ovh.net/connectors-sources-stripe.md) source instead of the File Upload source: in **Connectors** > **Sources**, both appear in the _New source_ window. The tables then follow the attributes of that source, and the cleaning script must be adapted to them.
## What you have built
You have a raw table with every order line of the store, a clean table that follows the UCI description of the variables, and a Superset dataset on top of it.
The next part, [Build a sales dashboard for an online store in Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md), turns the clean table into a dashboard: key figures, revenue over time, best-selling products, sales by country and customer loyalty.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build a sales dashboard for an online store in Superset
## Objective
This guide walks you through turning the cleaned order lines of an online store into a Superset dashboard: key figures, revenue over time, best-selling products, sales by country and customer loyalty.
It is the second part of [Prepare online store orders for a sales dashboard](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md). Superset queries the Lakehouse through [Trino](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), so all SQL on this page is in the **Trino dialect**.
## Requirements
To follow this guide you need:
- The table `online_retail_ii_clean` in the dataset `default_dataset` of your project, produced by the cleaning Custom action in Step 5 of [Prepare online store orders for a sales dashboard](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md#step-5-clean-the-orders).
- An Apache Superset application connected to your project. If you have not deployed it yet, follow the [Deploying Apache Superset on OVHcloud Data Platform](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) guide.
- The Superset dataset `online_retail_ii_clean`, created in Step 6 of [Prepare online store orders for a sales dashboard](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md#step-6-connect-superset).
## What you will build
At the end of this part, your dashboard looks like this:
| Section | Content |
| --------------- | --------------------------------------------------------------------------------------- |
| Key figures | 4 tiles: net revenue, orders, average order value, customers |
| _Time_ tab | Net revenue by month, seasonality by year |
| _Products_ tab | Top 10 products by revenue, top 10 products by units sold |
| _Geography_ tab | United Kingdom vs rest of world, top 10 countries outside the United Kingdom, world map |
| _Customers_ tab | 3 loyalty tiles, purchase frequency, recency segments, days between purchases |
Every chart follows the same path in Superset: **Charts** > **+ Chart**, pick the dataset and the chart type, fill in the settings, click **Create chart**, then **Save** with the chart name. Leave **Add to dashboard** empty: you create the dashboard and add the charts to it in [Step 8](#step-8-create-and-assemble-the-dashboard).
:::info
**Note:** amounts are in pounds sterling. For every amount, in the _Customize_ tab, set **Currency format** to `GBP` with the `Prefix` position, so values read `£18.9M` and not `18.9M £`.
:::
## Instructions
### Step 1: Create the customer datasets in SQL Lab
The customer charts need one row per customer or per return visit, which a chart cannot compute from order lines. You prepare these rows once, with two queries saved as virtual datasets.
#### Create the virtual dataset customer\_summary
1. Open **SQL** > **SQL Lab**.
2. Select your database and the schema `default_dataset`.
3. Paste the query below and click **Run**.
4. In the query toolbar, click the **Save or Overwrite Dataset** icon and save the dataset as `customer_summary`.
```sql
WITH purchase_days AS (
SELECT DISTINCT customer_id, CAST(invoicedate AS DATE) AS purchase_day
FROM default_dataset.online_retail_ii_clean
WHERE customer_id IS NOT NULL
AND invoice NOT LIKE 'C%'
),
customers AS (
SELECT
customer_id,
COUNT(*) AS purchase_days,
MIN(purchase_day) AS first_purchase,
MAX(purchase_day) AS last_purchase
FROM purchase_days
GROUP BY customer_id
)
SELECT
customer_id,
purchase_days,
LEAST(purchase_days, 10) AS purchase_days_capped,
CASE WHEN purchase_days >= 2 THEN 1 ELSE 0 END AS is_repeat,
first_purchase,
last_purchase,
date_diff('day', last_purchase, DATE '2011-12-09') AS days_since_last_purchase,
CASE
WHEN date_diff('day', last_purchase, DATE '2011-12-09') <= 90 THEN '1. Active (90 days)'
WHEN date_diff('day', last_purchase, DATE '2011-12-09') <= 180 THEN '2. At risk (91 to 180 days)'
WHEN date_diff('day', last_purchase, DATE '2011-12-09') <= 365 THEN '3. Lapsed (181 to 365 days)'
ELSE '4. Lost (over a year)'
END AS recency_segment
FROM customers
```
The result has one row per identified customer: **5,852 rows**.
#### Create the virtual dataset purchase\_intervals
Repeat these steps with the query below and save the dataset as `purchase_intervals`.
```sql
WITH purchase_days AS (
SELECT DISTINCT customer_id, CAST(invoicedate AS DATE) AS purchase_day
FROM default_dataset.online_retail_ii_clean
WHERE customer_id IS NOT NULL
AND invoice NOT LIKE 'C%'
),
with_previous AS (
SELECT
customer_id,
purchase_day,
lag(purchase_day) OVER (PARTITION BY customer_id ORDER BY purchase_day) AS previous_purchase
FROM purchase_days
)
SELECT
customer_id,
purchase_day,
date_diff('day', previous_purchase, purchase_day) AS days_since_previous_purchase
FROM with_previous
WHERE previous_purchase IS NOT NULL
```
The result has one row per return visit: **27,026 rows**.
:::info
**Note:** in SQL Lab, select the schema first. Without it, Trino returns `Schema must be specified when session schema is not set`. You can also prefix the table, as in the queries above: `default_dataset.online_retail_ii_clean`.
:::
### Step 2: Add calculated columns to online\_retail\_ii\_clean
Open **Datasets**, edit `online_retail_ii_clean`, then go to the _Calculated columns_ tab. For each column below, click **Add item**, then fill in its name and SQL expression. Start with `revenue`, the simpler one. Click **Save** once both columns are added.
#### Add revenue
| Setting | Value |
| -------------- | ------------------ |
| Column name | `revenue` |
| SQL expression | `quantity * price` |
`revenue` is the amount of each order line. A cancellation has a negative quantity, so its revenue is negative, and a plain sum gives the net revenue. Every amount in the dashboard is built on this column.
#### Add country\_code
| Setting | Value |
| -------------- | -------------------- |
| Column name | `country_code` |
| SQL expression | the expression below |
`country_code` converts the country name to its ISO 3166-1 alpha-3 code, for the world map:
```sql
CASE country
WHEN 'United Kingdom' THEN 'GBR'
WHEN 'EIRE' THEN 'IRL'
WHEN 'Germany' THEN 'DEU'
WHEN 'France' THEN 'FRA'
WHEN 'Netherlands' THEN 'NLD'
WHEN 'Spain' THEN 'ESP'
WHEN 'Switzerland' THEN 'CHE'
WHEN 'Belgium' THEN 'BEL'
WHEN 'Portugal' THEN 'PRT'
WHEN 'Australia' THEN 'AUS'
WHEN 'Italy' THEN 'ITA'
WHEN 'Norway' THEN 'NOR'
WHEN 'Sweden' THEN 'SWE'
WHEN 'Cyprus' THEN 'CYP'
WHEN 'Finland' THEN 'FIN'
WHEN 'Austria' THEN 'AUT'
WHEN 'Denmark' THEN 'DNK'
WHEN 'Greece' THEN 'GRC'
WHEN 'Japan' THEN 'JPN'
WHEN 'USA' THEN 'USA'
WHEN 'Poland' THEN 'POL'
WHEN 'United Arab Emirates' THEN 'ARE'
WHEN 'Israel' THEN 'ISR'
WHEN 'Hong Kong' THEN 'HKG'
WHEN 'Singapore' THEN 'SGP'
WHEN 'Malta' THEN 'MLT'
WHEN 'Iceland' THEN 'ISL'
WHEN 'Canada' THEN 'CAN'
WHEN 'Lithuania' THEN 'LTU'
WHEN 'RSA' THEN 'ZAF'
WHEN 'Bahrain' THEN 'BHR'
WHEN 'Brazil' THEN 'BRA'
WHEN 'Thailand' THEN 'THA'
WHEN 'Korea' THEN 'KOR'
WHEN 'Lebanon' THEN 'LBN'
WHEN 'Bermuda' THEN 'BMU'
WHEN 'Nigeria' THEN 'NGA'
WHEN 'Czech Republic' THEN 'CZE'
WHEN 'Saudi Arabia' THEN 'SAU'
ELSE NULL
END
```
### Step 3: Build the key figures
The four tiles use the dataset `online_retail_ii_clean` and the chart type _Big Number_.
#### Net revenue
| Setting | Value |
| --------------- | --------------------- |
| Metric | `SUM(revenue)` |
| Number format | `SMART_NUMBER` |
| Currency format | `GBP`, prefix |
| Subheader | `after cancellations` |
Expected value: **£18.9M**.
#### Orders
| Setting | Value |
| ------------- | ---------------------------------------------------------- |
| Metric | _Simple_ tab: column `invoice`, aggregate `COUNT_DISTINCT` |
| Filters | _Custom SQL_: `invoice NOT LIKE 'C%'` |
| Number format | `SMART_NUMBER` |
Expected value: **39.5k**.
#### Average order value
| Setting | Value |
| --------------- | ------------------------------------------------------ |
| Metric | _Custom SQL_: `SUM(revenue) / COUNT(DISTINCT invoice)` |
| Filters | _Custom SQL_: `invoice NOT LIKE 'C%'` |
| Number format | `,.2f` |
| Currency format | `GBP`, prefix |
| Subheader | `sale invoices, before cancellations` |
Expected value: **£497.08**.
:::warning
Do not use `AVG(revenue)` for the average order value. It returns the average amount of an order line, about £18.53, not of an order.
:::
#### Customers
| Setting | Value |
| ------------- | -------------------------------------------------------------- |
| Metric | _Simple_ tab: column `customer_id`, aggregate `COUNT_DISTINCT` |
| Number format | `SMART_NUMBER` |
| Subheader | `identified customers` |
Expected value: **5.88k**.
### Step 4: Build the Time tab
#### Net revenue by month
| Setting | Value |
| --------------- | ----------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Line Chart_, or _Area Chart_ |
| X-axis | `invoicedate` |
| Time grain | `Month` |
| Metric | `SUM(revenue)` |
| Currency format | `GBP`, prefix |
#### Seasonality by year
| Setting | Value |
| --------------- | ------------------------------------------------------------------ |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Area Chart_ |
| X-axis | _Custom SQL_: `date_format(invoicedate, '%m - %M')`, label `month` |
| Dimensions | _Custom SQL_: `CAST(year(invoicedate) AS VARCHAR)`, label `YEAR` |
| Metric | `SUM(revenue)` |
| Currency format | `GBP`, prefix |
You get one area per year. The month number at the start of each label keeps January to December in order.
### Step 5: Build the Products tab
Both charts use the dataset `online_retail_ii_clean` and the chart type _Table_. The cleaning action gives each stock code a single description, so a product never appears twice.
#### Top 10 products by revenue
| Setting | Value |
| ---------- | ------------------------------- |
| Dimensions | `stockcode`, `description` |
| Metrics | `SUM(revenue)`, renamed `total` |
| Row limit | `10` |
| Sort by | `total`, descending |
In the _Customize_ tab, under **Conditional formatting**, add a rule:
| Setting | Value |
| ------------ | ----------------- |
| Column | `total` |
| Color scheme | `success` |
| Color type | background colour |
| Use gradient | yes |
#### Top 10 products by units sold
| Setting | Value |
| ---------- | ----------------------------------- |
| Dimensions | `stockcode`, `description` |
| Metric | `SUM(quantity)`, renamed `quantity` |
| Row limit | `10` |
| Sort by | `quantity`, descending |
Apply the same conditional formatting as the previous table, on the `quantity` column. Placed side by side, the two tables show that the products sold in the largest volumes are not the ones that bring in the most revenue.
### Step 6: Build the Geography tab
#### United Kingdom vs rest of world
| Setting | Value |
| ---------- | --------------------------------------------------------------------------------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Pie Chart_, with **Donut** enabled |
| Dimensions | _Custom SQL_: `CASE WHEN country = 'United Kingdom' THEN 'United Kingdom' ELSE 'Rest of world' END` |
| Metric | `SUM(revenue)` |
| Label type | `Category and Percentage` |
#### Top 10 countries outside the United Kingdom
| Setting | Value |
| ---------- | ------------------------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Pie Chart_ |
| Dimensions | `country` |
| Metric | `SUM(revenue)` |
| Filters | _Custom SQL_: `country <> 'United Kingdom'` |
| Row limit | `10` |
| Label type | `Category and Percentage` |
#### Revenue by country
| Setting | Value |
| ------------------ | --------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _World Map_ |
| Country column | `country_code` |
| Country field type | `ISO 3166-1 alpha-3 (cca3)` |
| Metric | `SUM(revenue)` |
### Step 7: Build the Customers tab
These charts cover identified customers and sale invoices, from December 2009 to December 2011. Recency is measured from 9 December 2011, the last day of the data. A customer comes back when they buy on another day: several invoices on the same day count as one purchase.
#### Repeat customer rate
| Setting | Value |
| ------------- | -------------------------------------- |
| Dataset | `customer_summary` |
| Chart type | _Big Number_ |
| Metric | _Custom SQL_: `AVG(is_repeat)` |
| Number format | `.1%` |
| Subheader | `customers with two purchases or more` |
Expected value: **71.4%**.
#### Median purchases per customer
| Setting | Value |
| ------------- | ----------------------------------------------------- |
| Dataset | `customer_summary` |
| Chart type | _Big Number_ |
| Metric | _Custom SQL_: `approx_percentile(purchase_days, 0.5)` |
| Number format | `.0f` |
| Subheader | `purchases per customer` |
Expected value: **3**.
#### Median days between purchases
| Setting | Value |
| ------------- | -------------------------------------------------------------------- |
| Dataset | `purchase_intervals` |
| Chart type | _Big Number_ |
| Metric | _Custom SQL_: `approx_percentile(days_since_previous_purchase, 0.5)` |
| Number format | `.0f` |
| Subheader | `days, across all return visits` |
Expected value: **32**.
#### Customers by number of purchases
| Setting | Value |
| ------------ | ----------------------------- |
| Dataset | `customer_summary` |
| Chart type | _Bar Chart_ |
| X-axis | `purchase_days_capped` |
| Metric | `COUNT(*)` |
| X-axis title | `Purchases (10 = 10 or more)` |
| Y-axis title | `Customers` |
If the bars are unevenly spaced, enable **Force categorical** on the X-axis.
#### Customers by recency
| Setting | Value |
| ---------- | ------------------ |
| Dataset | `customer_summary` |
| Chart type | _Bar Chart_ |
| X-axis | `recency_segment` |
| Metric | `COUNT(*)` |
The four bars add up to 5,852 customers. The number at the start of each segment keeps them in order, from active to lost.
#### Days between purchases
| Setting | Value |
| ---------- | --------------------------------------------------- |
| Dataset | `purchase_intervals` |
| Chart type | _Histogram_ |
| Column | `days_since_previous_purchase` |
| Filters | _Custom SQL_: `days_since_previous_purchase <= 365` |
| Bins | `10` |
| Normalize | enabled |
The filter leaves out the rare returns after more than a year, which would stretch the axis and squeeze every other bar to the left.
### Step 8: Create and assemble the dashboard
1. Open **Dashboards** and click **+ Dashboard**. The new dashboard opens in edit mode.
2. Click its title and rename it `Online store sales`.
3. Add a header `Key figures`, drag the 4 key figure tiles from the right panel and place them on one row.
4. Add a header `Trends and breakdowns`, then a tabs component with 4 tabs: `Customers`, `Time`, `Products` and `Geography`.
5. Drag each of the other charts from the right panel into its tab. In the _Customers_ tab, put the 3 tiles on one row, the 2 bar charts side by side and the histogram at full width.
6. Click **Save**.
## What you have built
You have a dashboard that answers the questions of an online store manager: how much the store sells and to how many customers, how sales evolve over the year, which products and countries matter, and how often customers come back.
From here you can add dashboard filters on `country` or on a date range, or apply the same pipeline to your own order data. For another Superset dashboard built on Data Platform, see the [FOCUS and Apache Superset use case](https://docs.dataplatform.ovh.net/tutorials-finops-focus-superset.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-tutorials-data-export.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data export & management
## Exports, external access and permissions
Exporting folders with a custom date builds export paths that carry the run date, which is what downstream systems and archives usually expect. Setting role conditions on one bucket narrows an IAM role to a single bucket rather than the whole Lakehouse, using CEL conditions. Custom event handling actions let an outside system trigger a workflow with a payload, which is the inbound half of the same integration problem.
Accessing buckets with S31
-compatible tools covers the other direction, reading and writing project buckets from the AWS CLI, Rclone, NodeJS or Python instead of from the interface.
Together they cover data lifecycle work: scheduling exports, shaping their layout, reaching data from outside, and keeping permissions tight around all of it.
[### Export folders with a custom date→
Export paths that carry the run date.
](/tutorials-dated-export-folders)[### Set role conditions on one bucket→
Narrow an IAM role to a single bucket with CEL.
](/tutorials-bucket-role-conditions)[### Custom event handling action→
Let an outside system trigger a workflow with a payload.
](/tutorials-custom-event-handling)[### Access your buckets with S3-compatible tools→
AWS CLI, Rclone, NodeJS and Python against a project bucket.
](/tutorials-bucket-s3-access)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-dated-export-folders.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Export folders with a custom date
## Objective
This tutorial demonstrates how to add a custom date to your folder names during the creation of the [Data Processing Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md).
:::info
**Note:** This functionality is only available to the following actions: _Load to Bucket_, _MySQL to Parquet_, and _PostgreSQL to Parquet_.
:::
## Introduction
### Requirements
To follow this tutorial, you need to be familiar with the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) and you should preferably have done the [Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). The essential is that you know what [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) are for and how to use them.
### Concepts Overview
When you create actions, you have the choice to name your folder which will hold your exported files. The _$now_ token enables you to give custom dates easily using a few basic patterns as part of the folder name.
There are 4 patterns available on the Data Platform:
- **$now(date)**
- **$now(datetime)**
- **$now(timestamp):** Timestamp will be printed in [UNIX epoch time](https://en.wikipedia.org/wiki/Unix_time)
- **$now(_insert sequence here_):** Create custom date sequences with the available [format codes](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Now let's see how all of this works in practice!
## Use the $now token
For this tutorial we shall only explore how to use the _$now_ token. To know how to create and run _Actions_, there is an in depth documentation on [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) available.
For the ease of understanding we will choose to create an action - _Load to Bucket_, which is meant to run daily for the foreseeable future - Which will in turn lead to folders being generated on a daily basis. To help sort and easily access the correct folder, the **$now** token can be used.
At the stage where the option to input the name for the folder becomes available, You can use the _$now_ token like shown in the example below.
On the picture below, you can see that the folder has been successfully generated in the _bucket_ with the naming schema we expected. It is not necessary to have additional text in the folder name like used in the example, the token can be used independently as well.
bFor the scenario mentioned above, here is an example of how the _bucket_ will look with the folders generated over the past few days. It becomes easy to quickly identify the folder created on a certain date.
## Examples
Here are some more examples of how the folder name will appear with different use cases:
- **$now(date):** 2023-08-15
- **$now(datetime):** 2023-08-15\_20-51-04
- **$now(timestamp):** 1692132664 _(Unix epoch time)_
- **$now(%a-%d-%b-%Y\_%H-%M%):** Tue-15-Aug-2023\_20-53
:::info
Since folder names shouldn't have special characters like _colon_, _slash_, _asterisk_, etc. the _dash_ character is used for the time and date format.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-bucket-role-conditions.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# How to set a role condition on a specific bucket
## Objective
The Data Platform [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md) allows you to set fine-grained conditions in your Project:
- on resources properties
- on users properties
- on data, through Advanced Data Access Control
The purpose of this tutorial is to show how to give users in your Project restricted access to only the [Lakehouse Manager bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) called _my\_bucket_.
## Requirements
You will need:
- an [IAM role](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) that grants access to the resource `bucket` (such as the default "DataStore Editor" role)
## Tutorial
To restrict access to specific [buckets in your Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md), you must set a condition on the _**technical name**_ when binding a role granting access to _Data Store buckets_.
:::info
Data Store is the original name of the buckets on Data Platform.
:::
[Bind](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#bind-a-role-to-a-user-service-account-or-group) such a role to the group/user of your choice.
Then, add a [condition](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#add-a-condition-when-binding-a-role) to the role binding.
The condition must be set on the _technical name_ of the resource, which intuitively would look like the following:
```text
Name == "my_bucket"
```
However, just adding this condition would filter **the entire role** to the resources called "my\_bucket", including resources that are not buckets (like queries, dashboards, pipelines, etc..). In other words, if your role gives access to any other resource, they will be filtered just like buckets.
Consequently, it is recommended to set conditions as follows.
### Using the visual builder
If your initial role grants access to other resources, it is important to start by adding a first **rule** which will allow access to all resources that are not buckets which are planned by the role.
:::info
Through conditions, you cannot give access to resources/actions that were not initially permitted by the role. Conditions can only filter a role further.
:::
- Field: _Resource_
- Operator: _Not equals_
- Value: _Bucket_
Then, add a **branching** with an _OR_ operator.
In the branching, add two rules linked by an _AND_ operator.
First rule:
- Field: _Resource_
- Operator: _Equals_
- Value: _Bucket_
Second rule:
- Field: _Technical name_
- Operator: _Equals_
- Value: _my\_bucket_
Press **Confirm**. The user/group will now have access only to the bucket "my\_bucket", and access to all other resources contained in the role.
:::info
This branching system can also be used to restrict _write_/_delete_ access to only some objects (for example, the ones that have the tag "dev") while allowing _read_ access to the remaining objects.
:::
### Using the CEL editor
In CEL, the condition would be the following:
```text
Resource != "bucket"
||
(
Resource == "bucket"
&&
Name == "my_bucket"
)
```
To go further, see the customization capabilities of the Identity Access Manager in our documentation.
[↪ Learn more about roles and conditions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md)
Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-custom-event-handling.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Creating Event Handling Actions
## Objective
In this article we will go through the steps required to **create an action that allows you to handle events**. This can be applied to trigger actions or workflows defined in your Data Processing Engine with a body of data outside of the platform.
## Event Handling Table
We usually recommend that when actions are called by an external event, the event call is archived for reference in your project. In this article we will use an example table in our data model called _event\_handle_
- **Table Name**: event\_handle
- **Table Description**: the table contains 4 attributes to store the content of the event.
| Attribute Name | Type | Example value |
| ------------------ | ------ | ---------------------- |
| **event\_id** | Number | 38123461 |
| **event\_name** | String | Rebecca |
| **event\_content** | String | Walked in the store |
| **date** | Date | 2020-05-21 11:34:23 PM |
:::warning
Note that the names provided in this example both for the table, its attributes as well as all the other object names below (for the action or functions) are **examples**. You can obviously choose to name your objects differently, just make sure that you update the subsequent code section which references the different objects.
:::
Generally speaking this is a good best practice to set-up if you're planning on creating custom actions which handle events because you will use it to archive all triggered events. To create such a table, head to the _Lakehouse Manager_ component, create a table with attributes following the ones outlined in the table above.
[Creating a data model in the Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)
## Event Handling Action Set-up
To create your custom action, head to the _Data Processing Engine_ component and create a new custom action and enter the **action name** as _Event handle_. The sample Python code provided below shows how to manage the body of the event when the action is triggered outside of the platform, typically via API endpoint.
In the custom action configuration, change the **function name** (which specifies which Python function is the main function to run) to _event\_handle_. Finally, copy & paste the following code sample in the custom code editor:
```python
import logging
import pandas as pd
from datetime import datetime
from forepaas.dwh import bulk_insert
from forepaas.dwh import connect
logger = logging.getLogger(__name__)
def event_handle(event):
logger.info(f"Start inserting event to default_dataset/event_handle")
logger.info(f"event_type = {event.type}, number of rows = {len(event.content)}")
if type(event.content) is not list:
raise Exception("When playing this action you need to send rows")
# Create dataframe
df = pd.DataFrame(event.content)
df["date"] = datetime.now().strftime("%Y-%m-%d")
# Connect to destination database and insert dataframe to table
destination = connect("dwh/default_dataset/")
stats, error = bulk_insert(destination, "event_handle", df)
logger.info(f"End insert of event: stats={stats}, error={error}")
```
:::info
Note that if you're more confortable in an IDE-like interface, click on the _Advanced_ mode on top of the page to switch to the online code editor. You will be able to update the raw source code of the action in the Python script and all its configuration in the JSON file.
:::
## Launching Actions and Workflows
Now that you know how to handle the incoming event information when an action is launched externally, let's look at how to set-up the trigger events. This will enable you to start a job of an action or workflow programmatically.
### Required Parameters
**There 3 parameters required to launch a job which you obviously need to set with respect to your own environment configuration**. All the parameters are outlined in the code samples using \{} brackets like \{subdomain}.
| Parameter name | Value | Example |
| -------------- | ------------------------------- | ------------------------ |
| subdomain | Project subdomain | test |
| action\_id | Unique reference of your action | 5eaf0db7f6f51d5255974217 |
| token | Authentication token | xxxxx |
#### Project Subdomain
To retrieve your project subdomain, simply look-up your project URL. For instance if you copy paste the URL when you have the DPE component open it should look like this:
```text
https://{project_subdomain}.eu.dataplatform.ovh.net/dpe/#/
```
The subdomain is the **first reference in the path of your project's URL**.
#### Action ID or Workflow ID
To retrieve the action ID of the action you want to start, you can also find it in your action's URL when editing it. To do that, click _edit_ on the action you want to trigger, the URL should look like this:
```
# Action's URL
https://{project_subdomain}.eu.dataplatform.ovh.net/dpe/#/action/{action_id}
# Workflow's URL
https://{project_subdomain}.eu.dataplatform.ovh.net/dpe/#/workflow/{workflow_id}
```
#### Authentication Token
To generate an authentication token, you will first need to create a specific API & secret key from the _Identity Access Manager_ component. To learn how to do that, check-out this dedicated article:
[Creating an API / Secret Key](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md)
### Code samples
**cURL**
```bash
curl --request POST \
--url 'https://{subdomain}.eu.dataplatform.ovh.net/dpe/v3/actions/{action_id}/start?app_id=55c1423560702d6426490f38&type=cam&token={token}' \
--header 'content-type: application/json' \
--data '[
{
"event_name": "Rebecca",
"event_id": 1,
"event_content": "imperdiet ornare."
},
{
"event_name": "Veronica",
"event_id": 2,
"event_content": "Nullam ut nisi a odio"
},
{
"event_name": "Aurelia",
"event_id": 3,
"event_content": "amet"
},
{
"event_name": "Yardley",
"event_id": 4,
"event_content": "ornare placerat, orci lacus vestibulum"
},
{
"event_name": "Dakota",
"event_id": 5,
"event_content": "nulla. Cras eu tellus eu"
}
]'
```
**Python 3+**
```python
import requests
url = "https://{subdomain}.eu.dataplatform.ovh.net/dpe/v3/actions/{action_id}/start"
querystring = {
"app_id":"55c1423560702d6426490f38",
"type":"cam",
"token":"{token}"
}
payload = [
{
"event_name": "Rebecca",
"event_id": 1,
"event_content": "imperdiet ornare."
},
{
"event_name": "Veronica",
"event_id": 2,
"event_content": "Nullam ut nisi a odio"
},
{
"event_name": "Aurelia",
"event_id": 3,
"event_content": "amet"
},
{
"event_name": "Yardley",
"event_id": 4,
"event_content": "ornare placerat, orci lacus vestibulum"
},
{
"event_name": "Dakota",
"event_id": 5,
"event_content": "nulla. Cras eu tellus eu"
}
]
response = requests.request("POST", url, params=querystring, json=payload)
print(response.text)
```
:::info
Note that **workflows and actions can be equally triggered using the same process**. To use the data sent as part of the call, make sure the workflow contains a handling event action. Also, make sure to **use the right URL for workflows** (see section on Action or Workflow ID).
:::
We hope this article was useful! Let us know if you have any questions or would like to suggest an improvement.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-bucket-s3-access.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Access your buckets with S3-compatible tools
## Objective
The [buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md)
of a project speak the S31
protocol, so any S3-compatible client can read and write them: the AWS CLI, Rclone, and the AWS SDKs for NodeJS and Python among them. This tutorial covers each in turn, with the commands and the credentials each one needs.
The Data Platform Buckets are fully compatible with **AWS S3** client.
To interact with them, you will need to [generate an API key and Secret key in your Identity Access Manager](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) from a **Data Platform Directory** account.
:::info
ℹ️ **Note:** Currently, all root buckets are named as `"project-projectID"`.\
You can find your `projectID` by clicking on the info icon at the top-right corner of the platform.\
**Example:** `project-abcdef`
:::
## Using AWS CLI
:::info
In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
1. Install the AWS CLI client from [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/)
2. Configure the AWS CLI, it is suggested to use [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
```bash
aws configure --profile myproject
aws Access Key ID [None]: your_access_key
aws Secret Access Key [None]: your_secret_key
Default region name [None]: forepaas
Default output format [None]:
```
1. Enable _AWS Signature Version 4_ for MinIO server.
```bash
aws configure set s3.signature_version s3v4 --profile myproject
```
1. A few command examples (replacing `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain) and 'mybucket' with your desired bucket name):
```bash
# list your buckets
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 ls
# list contents inside bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 ls s3://mybucket
# make a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 mb s3://mybucket
# add an object to a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 cp my-file.csv s3://mybucket
# delete an object from a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 rm s3://mybucket/my-file.csv
# remove a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 rb s3://mybucket
```
:::info
To manage entrypoint URL inside a named profile, you may look at this documentation page: [configure entry point URL per profile](https://github.com/wbingli/awscli-plugin-endpoint).
:::
## Using Rclone
:::info
In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
1. Install [rclone](https://rclone.org/) from [https://rclone.org/downloads/](https://rclone.org/downloads/)
2. Configure rclone (replacing `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain)).
```bash
rclone config create my-project s3 provider Minio region forepaas env_auth false
rclone config update my-project endpoint https://my-project.eu.dataplatform.ovh.net/datastore
rclone config update my-project access_key_id YOUR_ACCESS_KEY
rclone config update my-project secret_access_key YOUR_SECRET_KEY
```
1. Check rclone's config file location if needed
```bash
rclone config file
```
1. Check your configuration if it matches the following below
```bash
rclone config show my-project
[my-project]
type = s3
provider = Minio
region = forepaas
env_auth = false
endpoint = https://my-project.eu.dataplatform.ovh.net/datastore
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
```
1. Some simple commands (please refer to rclone documentation for more details):
```bash
# listing buckets
rclone lsd my-project:
# create a bucket
rclone mkdir my-project:test-rclone
# generate a local docs for test purpuse
rclone gendocs docs
# copy ./docs/ to test-rclone
rclone copy docs my-project:test-rclone/docs
# what is the size of a bucket or a path ?
rclone size my-project:test-rclone
Total objects: 69
Total size: 239.570 kBytes (245320 Bytes)
# displays a tree
rclone tree my-project:test-rclone
# move files in bucket
rclone move my-project:test-rclone/docs/commands my-project:test-rclone/commands
# syncing source to destination
rclone sync --dry-run docs my-project:test-rclone/docs
# listing files in a human redable way
rclone lsl my-project:test-rclone
# listing files with format options
rclone lsf --format "tsp" --recursive my-project:test-rclone
# deleting files greater than 1k
rclone --min-size 1k lsl my-project:test-rclone/commands
rclone --dry-run --min-size 1k delete my-project:test-rclone/commands
rclone --min-size 1k delete my-project:test-rclone/commands
# copy files in ./commands
rclone copy my-project:test-rclone/commands commands
# delete an entire path
rclone delete my-project:test-rclone/commands/
```
## Using NodeJS
:::info
In the base URLs in the code snippets below, you will need to replace `[PROJECT_NAME]` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
:::info
ℹ️ **Note:** Currently, all root buckets are named as `"project-projectID"`.\
You can find your `projectID` by clicking on the info icon at the top-right corner of the platform.\
**Example:** `project-abcdef`
:::
```sh
# Add aws-sdk to your application with the following command
yarn add aws-sdk
# Or
npm i -s aws-sdk
```
```js
const AWS = require('aws-sdk')
const fs = require('fs')
// Modify those values for your environment
let accessKey = '[ACCESS_KEY]'
let secretKey = '[SECRET_KEY]'
let endpoint = 'https://[PROJECT_NAME].eu.dataplatform.ovh.net'
let bucket = '[BUCKET_NAME]'
let region = 'forepaas'
// Constructs a service object. This object has one method for each API operation.
const dataStoreClient = new AWS.S3({
accessKeyId: accessKey,
secretAccessKey: secretKey,
endpoint: `${endpoint}/datastore`,
region: region,
signatureVersion: 'v4',
s3ForcePathStyle: true
})
////////////////////////////////
// Adds an object to a bucket //
////////////////////////////////
let uploadStream = fs.createReadStream('./file.csv')
dataStoreClient.putObject({
Bucket: bucket,
Key: 'file.csv',
Body: uploadStream
}, (err) => {
if (err) console.error(err)
else console.info('File uploaded')
})
/////////////////////////////////////
// Retrieves objects from a bucket //
/////////////////////////////////////
let downloadStream = fs.createWriteStream('./file_downloaded.csv')
dataStoreClient.getObject({
Bucket: bucket,
Key: 'file.csv'
})
.createReadStream()
.on('error', (err) => {
console.error(err)
})
.pipe(downloadStream)
.on('close', () => {
console.info('File downloaded')
})
/////////////////////////////////////
// Returns some or all (up to 1000) of the objects in a bucket.
/////////////////////////////////////
dataStoreClient.listObjects({
Bucket: bucket
}, (err, data) => {
if (data && data.Contents) {
data.Contents.forEach(file => {
console.info(`${file.Key} (${file.Size} bytes)`)
})
}
})
```
## Using Python
:::info
In the base URLs in the code snippets below, you will need to replace `[PROJECT_NAME]` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
:::info
ℹ️ **Note:** Currently, all root buckets are named as `"project-projectID"`.\
You can find your `projectID` by clicking on the info icon at the top-right corner of the platform.\
**Example:** `project-abcdef`
:::
```sh
# Install boto3 with the following command:
pip install boto3
```
```python
import boto3
from botocore.client import Config
import os
def customfunc(event):
# Modify these values for your environment
project_url = 'https://[PROJECT_NAME].eu.dataplatform.ovh.net'
access_key = '[ACCESS_KEY]'
secret_key = '[SECRET_KEY]'
bucket = '[BUCKET_NAME]'
region = 'forepaas'
try:
# Create the S3 client (named datastore)
datastore = boto3.client(
's3',
endpoint_url=f'{project_url}/datastore',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
config=Config(signature_version='s3v4',request_checksum_calculation='when_required'),
region_name=region
)
# List all available buckets
buckets_response = datastore.list_buckets()
print("Available buckets:")
if 'Buckets' in buckets_response:
for b in buckets_response['Buckets']:
print(b['Name'])
else:
print("No buckets found.")
# Test connection by listing objects in the specified bucket
response = datastore.list_objects(Bucket=bucket)
print("Connection successful!")
if response and 'Contents' in response:
print("Bucket contents:")
for obj in response['Contents']:
print(f"{obj['Key']} ({obj['Size']} bytes)")
else:
print("Bucket is empty or no objects returned.")
# Ensure the local file exists
local_filename = './file.csv'
if not os.path.exists(local_filename):
print(f"Local file {local_filename} not found.")
return
# Upload a file using a key that places it in an allowed subfolder.
upload_key = 'dwh/uploads/file.csv'
datastore.upload_file(
Bucket=bucket,
Filename=local_filename,
Key=upload_key
)
print(f"Uploaded local file '{local_filename}' to bucket '{bucket}' with key '{upload_key}'")
# Download a file from the bucket.
# Using a file key from your logs:
download_key = 'dwh/uploads/clean_cafe_sales.csv'
local_download_filename = './clean_cafe_sales_downloaded.csv'
datastore.download_file(
Bucket=bucket,
Filename=local_download_filename,
Key=download_key
)
print(f"Downloaded '{download_key}' from bucket '{bucket}' to local file '{local_download_filename}'")
#Here we have used a file with very specific directory using the download key, this can be done manually or either using the step above to list all the files and picking from that
#for reference here the dpe folder exists inside the bucket at root level.
except Exception as e:
print("Connection failed:", e)
# Example usage:
if __name__ == "__main__":
customfunc(event={})
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/landing-page-services.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform services
## How the services fit together
Five of the seven services form the path data takes through a project: in, stored, transformed, queried, then exposed. The other two are conditions on all of it rather than steps in it.
**Connectors** is the door. It registers every external system the project reads from, decides what structure incoming data arrives with, and exposes results back out to external BI and SQL tools. **Lakehouse Manager** is what the project knows about its data: datasets, tables, attributes, buckets, lineage, and the policy tags that govern access down to a single column. **Data Processing Engine** does the work, through actions, workflows and notebooks, and through the jobs that execute them. **Analytics Manager** turns tables into queries, charts and dashboards. **Application Services** ships the result, either as an authenticated custom API or as a deployed web application.
The two cross-cutting services come last. **Control Center** watches every job and deployment, holds the logs and the audit trail, raises alerts and scales resources. **Identity Access Manager** decides who can read and change what, at organization and project level.
The boundaries are deliberate. Registering a connection, modelling the data behind it and running the pipeline that fills it are three different jobs, and they can belong to three different teams.
[### Connectors→
Register sources, control what comes in, expose what goes out.
](/landing-page-connectors)[### Lakehouse Manager→
Serverless Iceberg storage for datasets, tables and buckets.
](/landing-page-lakehouse-manager)[### Data Processing Engine→
Actions, workflows and the jobs that run them.
](/landing-page-dpe)[### Analytics Manager→
Query the lakehouse and build dashboards on it.
](/landing-page-analytics-manager)[### Application Services→
Expose data as an authenticated API or a deployed app.
](/landing-page-app-services)[### Control Center→
Metrics, logs, alerts and resource scaling.
](/landing-page-control-center)[### Identity Access Manager→
Who can do what, at organization and project level.
](/landing-page-iam)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-connectors.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform connectors
## Where data enters the project
Connectors holds every external system the project reads from, the rules applied to incoming data, and the external tools allowed to read results back out.
Three areas divide the work. **Sources** register the connection to an external system: credentials, network access, and the objects that source exposes. The **Analyzer** inspects those objects before anything is loaded, extracting metadata and applying blueprint rules so structure and formatting are settled at the door. **Consumers** point the other way, exposing project data to external tools such as Power BI, an OpenLineage backend, or any SQL client through Trino.
The split between registering a connection and loading its data is deliberate: credentials can stay with one team and data modelling with another. Registration happens here. Loading happens in the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md), into tables held by the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
Start with the key concepts if Connectors is new to you, then register your first source.
[Discover the Connectors key concepts](https://docs.dataplatform.ovh.net/connectors-key-concepts.md)
[### Key concepts→
How sources, analysis and loading fit together across a project.
](/connectors-key-concepts)[### Sources→
Register an external system and the objects it exposes.
](/landing-page-connectors-sources)[### Analyzer→
Extract metadata and apply blueprint rules before data is loaded.
](/landing-page-connectors-analyzer)[### Consumers→
Give an external BI or SQL tool read access to project data.
](/landing-page-connectors-consumers)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-key-concepts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover the key concepts in the Connectors
## Objective
:::info
The Connectors, being the central repository for all your data assets on the platform, is one of the most critical components in your Project. With the ease to connect, extract your data,the Platform allows you to **understand your data well at an early stage**. This will help you start progressing on it, rather than spending a lot of time just to clean and manage the infrastructure of your data.
:::
The Connectors component has 4 sub-components, corresponding to its main features:
- **Overview** giving information on the various data sources (like their distribution, issues related to them, associated blueprint rules, execution analysis)
- [**Sources**](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) to connect external data from any of the supported sources. You can either import the data into your the Platform Project, or, for compatible connectors, directly query this source for analytics purposes.
- [**Analyzer**](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) to extract the metadata of your sources and apply basic cleaning rules to them
- [**Consumers**](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) allow you to connect external analytics tools to your Project and build reports and dashboards using softwares that you are familiar with.
- **Jobs** to give all users an overall overview of all the jobs and their status
Within the Connectors, once you have collected your [sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md), you can [analyze](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) them by extracting their metadata and by applying data cleaning rules (called [blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md)) to improve your data quality. These rules will be automatically applied by the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md), whenever a job on a data source is executed. Plug-in external [Consumers](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) like Power BI
Next, through the **Jobs** tab, you will be able to view the summary of your job performances. It provides you with quick hand information like status of jobs, jobs run overtime, job’s runtime, also running jobs, queued jobs, past jobs executed.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connector sources
## What is a source?
A source is a registered connection to an external system: the credentials, the network access, and the set of objects that system exposes. Registering a source moves no data. It records how to reach the system, so the [Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) can inspect it and the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) can load from it later.
Sources are chosen from the connector Store, which carries connectors for file storage systems, database engines, network protocols, streaming and IoT brokers, social platforms and analytics tools. Around twenty of them have their own guide covering the fields and permissions that connector needs. The rest follow the same configuration pattern.
Two paths open once a source is registered. The usual one imports a copy of the data into the project, by [creating](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#create-a-table) and [loading](https://docs.dataplatform.ovh.net/dpe-actions-load.md) a [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md). The alternative, available on a subset of connectors, marks the source as directly queryable so the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) runs SQL against it in place, with no copy held on the platform.
Sources are organized in folders. Registering credentials is deliberately separate from analyzing and loading the data, so the two can sit with different teams and different [IAM roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md).
[Learn how to configure your connector](https://docs.dataplatform.ovh.net/connectors-sources-configure.md)
[### Configure your connector→
The shared configuration flow, plus the IP address to allow-list.
](/connectors-sources-configure)[### OVHcloud API→
Collect data from the OVHcloud REST API.
](/connectors-sources-ovhcloud-api)[### Apache Kafka→
Stream events from an Apache Kafka topic.
](/connectors-sources-kafka)[### Data Platform Buckets→
Use files stored in your project's own buckets.
](/connectors-sources-dataplatform-buckets)[### Dropbox→
Collect files shared on a Dropbox account.
](/connectors-sources-dropbox)[### Facebook→
Collect page insights: likes, views and comments.
](/connectors-sources-facebook)[### FHIR→
Collect healthcare data from an HL7 FHIR R4 server.
](/connectors-sources-fhir)[### File Upload→
Upload your own files as a source.
](/connectors-sources-upload-files)[### FTP→
Collect files from an FTP server.
](/connectors-sources-ftp)[### Google Analytics→
Collect audience and behaviour data from GA4.
](/connectors-sources-google-analytics)[### HTTP Files→
Import files published at a URL over HTTP.
](/connectors-sources-http)[### HTTP REST→
Query any compatible REST API and import the result.
](/connectors-sources-rest)[### HubSpot→
Extract CRM, marketing and CMS objects from HubSpot.
](/connectors-sources-hubspot)[### Microsoft Azure Blob Storage→
Collect files from Microsoft Azure Blob Storage.
](/connectors-sources-blob-storage)[### MySQL→
Use the tables of a MySQL database as a source.
](/connectors-sources-mysql)[### Odoo→
Extract contacts, orders, invoices and more from Odoo.
](/connectors-sources-odoo)[### Salesforce→
Extract standard and custom objects from Salesforce.
](/connectors-sources-salesforce)[### SFTP→
Collect files from an SFTP server.
](/connectors-sources-sftp)[### Shopify→
Extract products, customers and orders from Shopify.
](/connectors-sources-shopify)[### SQL Server→
Use the tables of a SQL Server database as a source.
](/connectors-sources-sql-server)[### Stripe→
Extract customers, charges and subscriptions from Stripe.
](/connectors-sources-stripe)[### Trino→
Query external data sources through Trino.
](/connectors-sources-trino)[### Twitter→
Consolidate insights from a given X account.
](/connectors-sources-twitter)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-configure.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure your connector
## Objective
The platform connectors allow you to import any data type, regardless of its source or format, into your platform, in order to leverage and cross-reference it within analytics applications, APIs, or predictive models.
Read the following tutorials on featured pre-configured connectors of the Connectors to learn how to connect your data in a few clicks.
## Managing sources
Data sources are organized in a hierarchical way through a tree view consisting of:
- **Folders** for logical grouping of your sources
- **Sources** specifying the access to smaller **source objects** (which can be tables, files, or endpoints)
:::info
For better readability, it is recommended to group sources into folders.
:::
Each time you add a source, a new line will be added to this dashboard. The basic actions that you can
perform on the Source menu dashboard includes:
- [Creating](#creating-a-new-source) a new source
- **Searching** for an existing source
- **Grouping** sources by folder
- **Adding, duplicating, deleting** sources within an existing folder
## Creating a new source
Under Sources click on **New Source** and you will see all available connectors in the Platform Store. You can find a connector by either typing the name of the desired connector in the search bar, filtering by a specific category on the left-hand side of the Store or scrolling down the connector list
Data Platform currently supports the following sources:
:::warning
Some of the connectors mentioned below have been deprecated and we are working on updating the list. Please reach out to the support team in case of any issues faced.
:::
- **Files storage systems:** Amazon S31, Dropbox, Google Drive, Azure Blob Storage, Microsoft OneDrive, Google Cloud Storage, Shadow Drive
- **Database engines:** PostgreSQL, MySQL, SQLServer, MariaDB, Cassandra, Google BigQuery, Amazon Redshift, ElasticSearch, Heroku Postgres, Clickhouse, SingleStore, MongoDB, Oracle, Snowflake, Trino, OpenSearch, Prometheus, M3DB, CockroachDB, InfluxDB
- **Network protocols & Open Data:** FTP, HTTP Files, SFTP, HTTP REST, Open Weather
- **Real-time & IoT:** Apache Kafka, Amazon Kinesis, MQTT, RabbitMQ, Apache Druid
- **Social media:** Pinterest, Facebook, LinkedIn, Twitter, YouTube, Weibo, Instagram
- **Analytics:** Google Analytics, Appfigures, Hubspot, MailChimp
Once you have found it in the **Platform Store**, click on it and then on _Select_ to proceed to the connector configuration screen.
[Learn how to configure your connector](https://docs.dataplatform.ovh.net/connectors-sources-configure.md)
### Make your data source directly queryable
In the **Preferences** of your source, you can activate the option _Direct Query_ to mark it as directly queryable.
If you mark a source as directly queryable, you will be able to execute SQL queries on its source objects (for example, the tables in your Amazon Redshift source) from the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md), without having to import the data in the Platform first.
:::warning
This option is currently only available in [SQL queries](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md).
:::
The following sources are compatible for direct queries:
- Amazon Redshift
- BigQuery
- Cassandra
- ClickHouse
- ElasticSearch
- Google Sheets
- Amazon Kinesis
- MongoDB
- MySQL
- Oracle
- PostgreSQL
- Redis
- SingleStore
- Snowflake
- SQL Server
## Featured connector guides
- [OVHcloud API](https://docs.dataplatform.ovh.net/connectors-sources-ovhcloud-api.md)
- [Apache Kafka](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md)
- [Dropbox](https://docs.dataplatform.ovh.net/connectors-sources-dropbox.md)
- [Facebook](https://docs.dataplatform.ovh.net/connectors-sources-facebook.md)
- [File Upload](https://docs.dataplatform.ovh.net/connectors-sources-upload-files.md)
- [Data Platform Buckets](https://docs.dataplatform.ovh.net/connectors-sources-dataplatform-buckets.md)
- [FTP](https://docs.dataplatform.ovh.net/connectors-sources-ftp.md)
- [Google Analytics](https://docs.dataplatform.ovh.net/connectors-sources-google-analytics.md)
- [HTTP Files](https://docs.dataplatform.ovh.net/connectors-sources-http.md)
- [HTTP REST](https://docs.dataplatform.ovh.net/connectors-sources-rest.md)
- [HubSpot](https://docs.dataplatform.ovh.net/connectors-sources-hubspot.md)
- [Microsoft Azure Blob Storage](https://docs.dataplatform.ovh.net/connectors-sources-blob-storage.md)
- [MySQL](https://docs.dataplatform.ovh.net/connectors-sources-mysql.md)
- [Odoo](https://docs.dataplatform.ovh.net/connectors-sources-odoo.md)
- [Salesforce](https://docs.dataplatform.ovh.net/connectors-sources-salesforce.md)
- [SFTP](https://docs.dataplatform.ovh.net/connectors-sources-sftp.md)
- [Shopify](https://docs.dataplatform.ovh.net/connectors-sources-shopify.md)
- [SQL Server](https://docs.dataplatform.ovh.net/connectors-sources-sql-server.md)
- [Stripe](https://docs.dataplatform.ovh.net/connectors-sources-stripe.md)
- [Trino](https://docs.dataplatform.ovh.net/connectors-sources-trino.md)
- [Twitter](https://docs.dataplatform.ovh.net/connectors-sources-twitter.md)
- [FHIR](https://docs.dataplatform.ovh.net/connectors-sources-fhir.md)
## Data Platform IP Whitelisting for External Database Access
To ensure secure access to your external database from our platform, it may be necessary to authorize specific IP addresses. You can whitelist the following IP address in your database configuration to allow seamless and secure communication:
### IP Address to Whitelist:
`51.210.213.114`
### Steps to Whitelist:
1. **Access Database Configuration**:\
Log in to your database management or security settings where you manage network access.
2. **Locate the IP Whitelisting Section**:\
Find the section for managing authorized IP addresses, often under "Security" or "Firewall" settings.
3. **Add the IP Address**:\
Enter `51.210.213.114` into the list of allowed or trusted IP addresses.
4. **Save and Apply Changes**:\
After adding the IP address, save your changes and apply them to ensure proper access.
By whitelisting this IP, you enable secure communication between the platform and your database while maintaining robust security measures.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/connectors-sources-ovhcloud-api.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect data from the OVHcloud API
## Objective
**OVHcloud API** is a RESTful API that enables you to automate your OVH configuration by providing programmatic access to all OVHcloud products and services. With this API, you can manage your services as easily as through the OVH Control Panel.
## API Overview
The OVHcloud API follows REST principles:
- Uses standard HTTP methods (GET, POST, PUT, DELETE)
- Resources are accessed via unique URLs
- Is stateless - each request contains all necessary information
## Configure your source
To set up the OVHcloud API source, you'll need to provide the following information:
### API Connection Details
**OVH Region**: Select your OVH region (Europe, US, Canada)\
**Application key**: Your OVH API application key\
**Application secret**: The secret associated with your application\
**Consumer key**: Your consumer key for authentication
:::info
You can also find the guide here on how to [get started with OVHcloud API](https://help.ovhcloud.com/guides/manage-and-operate/api/first-steps)
:::
To get your credentials:
1. Visit the OVHcloud token generation page
2. Sign in with your OVH account
3. Fill in the application name and description
4. Set the validity period
5. Define the required access rights
### Authentication setup
The OVH API uses a 3-key authentication system:
1. **Application Key (AK)**:
- Identifies your application
- Public identifier
- Generated when you register your application
2. **Application Secret**:
- Private key used to sign requests
- Must be kept secure
- Never transmitted in API calls
3. **Consumer Key (CK)**:
- Similar to an OAuth token
- Defines your application's permissions
- Can be revoked without changing your application key/secret
Once you have entered all the details, click on connect to successfully configure your connector.
## Specify endpoints to load data from
After configuring your authentication, you'll need to set up the endpoints you want to connect to. Each endpoint will correspond to a data source object that can be loaded into a distinct table.
### Single Endpoint Configuration
To configure an endpoint, you'll need to specify:
1. **Basic Settings**:
- **Technical Name**: Enter a unique name for your endpoint
- **Method**: Select the HTTP method (GET, POST, etc.)
- **Endpoint**: Base endpoint is automatically set based on your selected region during connector setup (e.g., ovh-eu for Europe)
- **Path**: Specific API path (e.g., `/me/bill`)
2. **Parameter Types**:
- **Path Parameters**: Path parameters are automatically added to the list when you include them in the path. For example, when you enter `/me/bill/{billId}` into the path, `billId` is automatically added as a parameter below, where you can assign its value.
- **Query-String Parameters**: Add query parameters to filter or customize your request. For example: Key:`date.from` - Value:`2024` to filter bills from 2024.
- **JSON Path**: lets you filter or drill into the JSON so you only capture the fields you need.
For example:
1. A simple JSON Path which can retrieve just the prices with tax included
```json
{
"path": "$.priceWithTax[*]"
}
```
2. A more complex one where you can specify exactly which fields you need.
```json
{
"path": "$.[*]",
"fields": [
"category",
"date",
"priceWithTax.Value",
"priceWithoutTax.Value"
]
}
```
3. **Examples**: View and use predefined endpoint configurations.
- Click on any example to automatically populate all relevant fields. Click on the 'Reset" button to clear the filled in example.
- Available examples include:
- `GET - /me/bill/{billId}`
- `GET - /me/bill/{billId}/details`
- `POST - /me/document`
- `GET - /allDom/{serviceName}/domain`
### Multiple Endpoint Configuration
The multiple endpoint feature is automatically triggered when you include a path parameter in your endpoint URL (e.g., when you add "\{billId}" to your path). This feature enables you to fetch related data across multiple endpoints using the same parameter.
When triggered:
1. The interface will switch to show the **MULTIPLE ENDPOINT** tab
2. You can select previously configured endpoints
3. Map the path parameters between endpoints
:::info
**Note**: For detailed implementation examples of multiple endpoints, particularly for billing scenarios, please refer to our comprehensive [Billing data analysis guide](https://docs.dataplatform.ovh.net/tutorials-billing-analysis.md) in the documentation. This guide provides step-by-step instructions and real-world examples of how to effectively use multiple endpoints.
:::
### Best Practices
1. **Using Examples**:
- Browse the EXAMPLES tab for common endpoint patterns
- Click an example to automatically populate all fields
- Modify the pre-filled values as needed for your use case
2. **Path Parameters**:
- Adding curly braces \{} around parameters (e.g., \{billId}) automatically enables multiple endpoint functionality
- Ensure parameter names match across related endpoints
- Test parameters with sample values before confirming
3. **Technical Names**:
- Choose clear, descriptive names for your source objects
- Include relevant identifiers (e.g., year, service type)
- Avoid special characters in technical names
## API Usage Guidelines
1. **Security**:
- Never share your Application Secret
- Rotate your credentials periodically
- Use the minimum required permissions for your use case
- Store credentials securely
- Use HTTPS for all API calls
2. **Rate Limiting**:
- Global limit: 20 calls per second
- Per-API limits may apply for specific endpoints
- Implement exponential backoff for retries
- Monitor your API usage through OVH's console
3. **Error Handling**:
- 400: Bad Request - Check your request parameters
- 403: Unauthorized - Verify your credentials and permissions
- 404: Not Found - Verify the resource exists
- 429: Too Many Requests - Implement rate limiting
- 500: Internal Server Error - Contact OVH support
## OVH API Console
The **OVHcloud API Console** is an interactive web-based interface that allows you to explore and test OVHcloud API endpoints without writing code. It provides a user-friendly way to execute API calls, inspect responses, and debug requests.
### Key Features:
- **Browse API Endpoints**: View available OVHcloud API services and their parameters.
- **Execute API Calls**: Test GET, POST, PUT, and DELETE requests directly from the console.
- **Authenticate with Your Account**: Use your **Application Key**, **Application Secret**, and **Consumer Key** to make authenticated requests.
- **Inspect API Responses**: View JSON responses, HTTP status codes, and request details.
- **Generate API Requests**: Copy pre-built API request formats for integration into your applications.
### How to Use the API Console:
1. **Access the Console**: Visit OVHcloud API Console.
2. **Select an API Endpoint**: Browse the API categories or search for a specific function.
3. **Authenticate**:
- Click **Authenticate** in the top right corner.
- Enter your **Application Key** and generate a **Consumer Key** if needed.
4. **Set Request Parameters**: Fill in required fields such as path and query parameters.
5. **Execute the Request**: Click **Run** to send the request and view the response.
6. **Review the Response**: Inspect returned data, error messages, and response headers.
:::tip
The API Console is ideal for testing before automating API calls in your application.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-kafka.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Stream Apache Kafka events into Data Platform
## Objective
Apache Kafka is a distributed event store and stream-processing platform developed by the Apache Software Foundation. Using the Kafka connector, you can connect to your Kafka server and integrate in real-time the data you receive on the server to your Project.
:::info
In this article, we assume that **you know the basic concepts of Apache Kafka**, such as messages and offsets. If you don't, just know that Apache Kafka servers store the data they receive in different topics and that this data comes in small chunks called messages. To keep track of the order of arrival of the messages, each one has an offset number, the lower it is, the earliest the message arrived.
:::
## Add a Kafka source on the Platform
### Configuration screen overview
Once you have found _Kafka_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below :
To connect to your server, you will have to give the server address and, if required by the server, username as well as password.
### Configuring your topics
After successfully establishing the connection with your server, you can select which topics to get data from by checking the boxes to their left.
If a topic is checked, you can hover over it and click on the Pen Icon to configure how the data it receives will be transferred to the Platform (the default offset policy).
There are two default options of offset policy:
- **Latest**: reads data in real-time as it arrives. Older messages will be ignored.
- **Earliest**: start reading the data from the earliest message in the topic. Once all the messages in the topic have been read, all messages will read in real-time (similar to _latest_).
:::info
You also have the possibility to **start reading from a custom offset**, but this option isn't configured at the connector level but at the processing job level. Therefore it is explained in the [Apache Kafka complete tutorial](https://docs.dataplatform.ovh.net/tutorials-kafka.md#execution-modes)
:::
Whenever you load data from a Kafka source into [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) using [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md), the latest read offset is stored in the destination table's metadata. This means:
- if you resume ingesting messages into this **same** table, the ingestion will start at the latest message read
- If you start ingesting messages into a **new** destination table, the ingestion will start at the default parameter configured either at the connector level (latest/earliest offset) or job level (custom offset)
### Supported message formats
The Platform supports the following types of input messages for metadata analysis and ingestion:
#### Flat JSON formats
```json
{
'key':'value',
'anotherkey':'anothervalue'
}
```
### Unsupported message formats
Currently, the Platform **doesn't support** the following formats:
#### Simple strings
```
'example message'
```
#### Nested messages
```json
{
'key':'value',
'nestedkey':{
'a':'b'
}
}
```
:::info
Feel free to reach out to us by sending us a request if the format you are using is not yet supported!
:::
## Using data from Apache Kafka in the Platform
If you are interested in seeing how to use the Kafka connection in the rest of the of the platform, you may refer to our dedicated [Apache Kafka tutorial](https://docs.dataplatform.ovh.net/tutorials-kafka.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-dropbox.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect Dropbox files with the Data Platform connector
## Objective
Dropbox is a cloud storage service used for file sharing and collaborative work. The Platform allows you to collect data you stored on Dropbox and import it directly into your platform.
## Add a Dropbox source on the Platform
### Configuration screen overview
Once you have found _Dropbox_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below -
### Learn how to get a Dropbox token
- Go to [https://www.dropbox.com/developers/apps/create](https://www.dropbox.com/developers/apps/create)
You will see the below screen to Create an App –
- Do the following steps –
1. Choose an `API`
2. Choose the type of access you need - `App Folder` or `Full Dropbox`
3. Give your app a `Name`
4. Click on `Create app`
- Once your application has been created, you will be redirected to the settings page –
- Go into the `Permissions` tab, then make sure `files.metadata.read` and `files.content.read` permissions are enabled. Then click on Submit.
- Going back to `Settings` tab Click on `Generate` under `Generated access token`
- Set `Access token expiration` to `No expiration`
- Copy your `token`
### Configuring your source
When creating the source, you will be required to input the following information :
- Token: The access key to your app (generated in the previous step).
- Path: The path to the folder.
:::info
If the token is an application token linked to a folder, you don't need to put anything in the Path field
:::
Once you add the above details click on _Connect_ and then on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-facebook.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect Facebook page insights with the Data Platform connector
## Objective
:::warning
Uh oh! This article is outdated and needs to be rewritten. Please contact Support if you need help setting up this connector.
:::
The Facebook connector lets you collect data and insights from a Facebook page into the Data Platform, such as likes, views and comments.
## Get your access credentials
### Learn how to get a Facebook Developer app token
The following steps will guide you through the procedure of creating an app token on the Facebook Developer user interface.
#### Pre-requisites: app token
- Log into your Facebook account that has all the **Admin** permissions on the Pages you want the access tokens for.\
If you are not the page's admin, ask your admin to grant you Analyst access to the page.
- Please note that if the account that granted access to the tokens doesn’t own the admin or analyst role for the page anymore, the access token will be revoked.
#### Procedure: app token
##### Step 1. Create your Facebook Developer app
- Go on : [https://developers.facebook.com/](https://developers.facebook.com/) and click on `Get Started` at the top right corner.
- Follow the steps in the pop-up window that just opened. It is not mandatory to create a "First App", you can skip this step.
- Go on [https://developers.facebook.com](https://developers.facebook.com). Enter the `My Apps` menu at the top right corner, right next to the search bar. You should now be able to click the `Add a New App` button at the top right corner.
- Fill-in the blanks in the form that just popped up and click on `Create App ID`
##### Step 2. Retrieve your App ID and App Secret
- You will now land on you new App Dashboard. Click on `Settings` on the left and then on `Basic`, right beneath it.
- Just right to the `App Secret` field, click on `Show`.
- You can now copy your App ID and App Secret and paste them in a new text editor window.
#### How to validate your Facebook app
- Go to [https://developers.facebook.com/apps](https://developers.facebook.com/apps). Click on the app you need to review
- Go to your basic settings. Click on "Settings", then "Basic"
- Give your basic settings. Give the following information in order to submit your app.
- Add a platform. At the bottom of the settings page, click on "Add Platform". Then click on Website, enter your company website URL and click on "Save changes", at the bottom right.
- Begin the app review by simply clicking on "App Review". Then click on "Add items" in the right corner.
- Then click on "Read\_insights" and click on add 1 item.
- Add details of sumissions. Go back to "App review tab" and click on "Add details" for read\_insights.
- Add details of read\_insights submission
- Submit the review. You must read the conditions, click on "I accept..." and then click on "Submit".
- Conclusion of the review.
Facebook review can take from 2-3 days, to some weeks in peak periods.
If your review is not accepted :
If that’s because of the screencast, kindly reajust the screencast in view of the facebook notices
If that’s because of the review text, please reajust the text and gives us feedback about it
If the reason seems more blocking or go in contradiction with the present document, you can contact the
Data Platform Team and we will do our best to help.
- Business creation. In the left menu, click on "Settings", then Advanced. Scroll down until you see "Business Manager", then click on "Get started".
- Complete and validate the following form.
- Business verification. Once your app has been verified, you should receive a message in order to verify your business
Go to App Review, and click on "Provide business details". Enter them and click on the Next button.
- You should provide business documents that show the legal name of your business plus at least one of the following which matches the information you just entered :
- Business Phone Number
- Address
- Tax ID
- Provide basic information about you, then "Submit".
- You will receive an email in order to accept the legal terms
If you haven’t got the email : please check your spam box.
Or redo the last step and correct your email information, the submission will re-send an email.
Click on the link in order to accept the terms.
- On the terms page, read attentively all the terms,
Then click "I Agree" if you agree with all the terms.
You're good to move to step 3!
##### Step 3. Generate your app's token
- Go on [https://developers.facebook.com/tools/explorer/](https://developers.facebook.com/tools/explorer/)
- Click on the `Graph API Explorer`dropdown menu at the top right and select your newly created app (called `Data Platform Connector` on the following screenshot).
- Now click on the `Get token` button and select `Get User Access Token` from the dropdown menu.
- In the window that just popped up, select the different permissions you need **(don't forget to tick the `read_insights` option)** and click on the `Get Access Token` button.
:::info
Note: as stated above, you need to be logged in to Facebook with an account that has either **Admin** or **Analyst** access for this to work
:::
- Click on the `Continue as` button.
- Then on `OK` at the bottom right.
- You just generated a short-term access token. Copy it and paste it in your text editor alongside your App ID and App Secret.
- Now edit the following URL template by replacing **\{app\_id}**, **\{app\_secret}** and **\{access\_token}** with your values.
> `https://graph.facebook.com/v2.12/oauth/access_token?grant_type=fb_exchange_token&client_id={app_id}&client_secret={app_secret}&fb_exchange_token={access_token}`
- Once done, paste your URL in your web browser. Copy the result and paste it in your text editor.
- Copy this new URL and replace **\{access token}** with the content of the `"access_token"` field from last step's result:
> `https://graph.facebook.com/v2.12/me?access_token={access_token}`
- Once again, copy the result from last step in your text editor and replace the **\{account\_id}** and **\{access\_token}** with the **id** and **access\_token** fields from the previous steps in the following URL template:
> `https://graph.facebook.com/v2.12/{account_id}/accounts?access_token={access_token}`
- One last time, paste the URL in your web browser. You can now save the information displayed on your screen to a .json document.
### Learn how to find your Facebook page ID
#### Pre-requisites: page ID
- You need to be the page's Admin to get this information.
#### Procedure: page ID
##### Step 1. Log into your Facebook account that has Admin access to the page.
##### Step 2. Click on the three horizontal dots and select `Edit Page Info` in the dropdown menu.
##### Step 3. Click on `See All Information` at the bottom of the window that just popped up.
##### Step 4. Copy your Page ID from the very bottom of the page.
## Add a Facebook source on Data Platform
### Configuration screen overview
Once you have found _Facebook_ in the **Data Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below:
### Configuring your source
#### Connect to Facebook
When creating the source, you will be required to input the following information :
- **Account**: your Facebook Page ID
- **Access Token**: your Facebook Developer app token
- **Source default schema**: you can use this field to add columns to your data, if necessary. Use the following syntax _\{"name":"value"}_, where _name_ is the name of your column and _value_ is the value to be filled in all rows.
Once you add the above details click on _Connect_ to establish a connection with Facebook.
#### Add the Accounts and Endpoints
Before moving on, you have to add accounts and endpoints.
- [Accounts](#how-to-add-an-account) are Facebook pages referenced by the Page ID - just like in the previous screen - which you want to get data from.
- [Endpoints](#how-to-add-an-endpoint) are the **different types of data** about accounts that Data Platform will get using the Facebook API. Each endpoint corresponds to a final table, aggregating data for all selected _accounts_.
:::info
Data Platform will use the credentials given in the [Connect to Facebook](#connect-to-facebook) step to access the Facebook API but **you can overwrite the credentials** to have multiple accounts and endpoints in the same source. Simply add the different accounts to the Source after the [Connect to Facebook](#connect-to-facebook) step.
:::
##### How to add an account?
To add an account, click on _Add an Account_. This will open the following window:
By default, it is using the same credentials you wrote previously when establishing the connection, which you can overwrite here. The _account name_ is the internal display name.
##### How to add an Endpoint?
On Data Platform, each registered endpoint will correspond to a data source object, which you will be able to [load](https://docs.dataplatform.ovh.net/dpe-actions-load.md) into a distinct [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
To add an endpoint, click on _Add an Endpoint_. This will open the following window:
Next, select the type of Endpoint you wish to add. The Data Platform team is always adding new types of Endpoints to the connectors. You can click on the scroll-down list under the _Type_ field to see the ones available.
Most common ones are:
- _page\_data_: general data about the page.
- _day\_insights_: data about the page sorted by day.
- _post\_insights_: data about the page's posts.
Some types require you to add _Insights_ to the endpoint, which is just a way of telling the Data Platform platform which data to get. To add an insight, just start typing inside the Insight field and click _Enter_ to select the data insight you want.
:::info
Note that you can add more than one insight to the same endpoint.
:::
To finish, click on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-upload-files.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Upload files as Data Platform sources
## Objective
This simple connector allows you to upload your files as sources in the platform.
## Add a File source on the Platform
### Configuration screen overview
Once you have found _File upload_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below:
### Supported files and sizes
This connector allows you to drag and drop files from the following types: `.csv`, `.xls`, `.xlsx`, `.xml`, `.json` and `.parquet`.
:::warning
There is a **1GB size limit** on files handled by this tab. In order to process files larger than 1GB, you need to upload them through the [Data Platform Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
:::
### Configuring your source
To add a file just drag & drop it directly to the designated area in the middle of the screen or click on the indicated link to select the file from your computer.
Uploaded files will be displayed in the list below the drag & drop area, as in the picture below :
:::info
Note that you can add multiple files to a single _File Upload_ source.
:::
The encoding of the source file is automatically detected by the Platform. However, if the automatic detection fails, you have the ability to specify the file encoding by clicking on the **pen icon** next to each source object:
Select the encoding of your source object in the dropdown selection.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-dataplatform-buckets.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform Buckets
## Objective
The connector allows you to use files stored in your [Project's buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) as data sources.
:::info
Buckets are a traditional file-based storage system and allow you to organize unstructured data
:::
## Add a Buckets source
### Configuration screen overview
Once you have found _Buckets_ in the **the Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below -
### Configuring your source
When creating the source, you will be required to input the following information :
- **Bucket:** The name of the Bucket from which you want to find the source files
- **Path:** The optional path to the subfolder containing the source files
Once you add the above details click on _Connect_ and
you will see the files in your bucket. An example is shown below:
Then click on the _Create_ button on the top right-hand corner to create your source.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
The encoding of the source files is automatically detected by the Platform. However, if the automatic detection fails, you have the ability to specify the file encoding by clicking on the **pen icon** next to each source object:
Select the encoding of your source object in the dropdown selection.
### Supported files and sizes
The Datastore allows an **S31 compatible object** store, which allows you to store any unstructured data.
:::warning
You can upload files with an individual size of up to 10 GB.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/connectors-sources-ftp.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect files from an FTP server
## Objective
The File Transfer Protocol (FTP) is a standard network protocol used for the transfer of computer files between a client and server on a computer network.
The Platform allows you to connect to an FTP server, select your files on it, and import their data directly into your platform.
## Add a FTP source on the Platform
### Configuration screen overview
Once you have found _FTP_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below -
### Supported files and sizes
Before you proceed with learning how to configure an FTP source, it would be interesting to note that -
- Even though our FTP connector is able to list every file on your server, keep in mind that meta/data extraction only support the following types: .csv, .xls, .xlsx, .xml, .json, .parquet
- For the Metadata Extraction (see the Analyze part of our connectors guide), the actual limits of size for a file downloaded by our FTP connector are:
- Complete Extract: about 900 Mo
- Sample Extract: about 2.5 Go
### Configuring your source
When creating the source, you will be required to input the following information :
- Host: The URL/IP of the FTP server you want to connect to (ex: 127.0.0.1)
- Port: The port listened to by your FTP server (21 by default for most FTP servers)
- Username: The login used to authenticate
- Password: The password associated
- Path: The path needed to find your files on the server (ex: Documents/data/export\_2018.csv)
Once you add the above details click on _Connect_ and then on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-google-analytics.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect Google Analytics 4 data with the Data Platform connector
## Objective
Google Analytics 4 (GA4) is Google's next-generation analytics platform that helps you understand customer behavior across websites and apps. The Data Platform lets you integrate GA4 data to build powerful, data-driven applications.
## Requirements
Before you begin, ensure you have:
1. **Google Analytics 4 Property** - You must be using GA4 (not Universal Analytics)
2. **Google Cloud Console Access** - Ability to create projects and service accounts
3. **GA4 Admin Permissions** - Editor or Administrator role on the GA4 property
4. **Data Platform Access** - Permissions to create sources in the Connectors
:::info
**Note:** Universal Analytics (GA3) is no longer supported. If you're still using Universal Analytics, you must migrate to GA4 first.
:::
## Configuration Steps
### Step 1: Get your GA4 Property ID
The Property ID identifies your GA4 property.
1. Log into [Google Analytics](https://analytics.google.com)
2. Click **Admin** (gear icon in bottom left)
3. In the **Property** column, click **Property Details**
4. Copy your **Property ID** (format: `123456789`)
:::info
The Property ID is a numeric value, different from the old View ID format used in Universal Analytics.
:::
### Step 2: Create Google Service Account
A Service Account allows the Data Platform to securely access your GA4 data without requiring individual user credentials.
#### 2.1 Create or Open Google Cloud Project
1. Navigate to [Google Cloud Console](https://console.cloud.google.com)
2. **If you already have a project:** Click **Create or Select a project** and choose your existing project
3. **If you need to create a new project:**
- Click **Create or Select a project** → **NEW PROJECT**
- Enter a **Project name** (e.g., "DataPlatform-GA4-Integration")
- Click **CREATE**
#### 2.2 Enable Google Analytics Data API
1. In the Cloud Console, go to **APIs & Services** from the quick access menu
2. Click on **+ Enable APIs and Services**
3. Search for **"Google Analytics Data API"**
4. Select **Google Analytics Data API** (not Analytics Reporting API)
5. Click **ENABLE**
:::info
Make sure to enable the **Google Analytics Data API** for GA4, not the older Analytics Reporting API.
:::
#### 2.3 Create Service Account
1. Go to **APIs & Services** → **Credentials**
2. Click **CREATE CREDENTIALS** → **Service account**
3. Enter a **Service account name** (e.g., "dataplatform-ga4-reader")
4. Add description (optional)
5. Click **CREATE AND CONTINUE**
#### 2.4 Assign Permissions (Optional)
1. For **Role**, select **Viewer** (optional for this step)
2. Click **CONTINUE**
3. Click **DONE**
#### 2.5 Generate JSON Key
1. Find your newly created service account in the list
2. Click the **service account email** to open details
3. Go to the **KEYS** tab
4. Click **ADD KEY** → **Create new key**
5. Select **JSON** format
6. Click **CREATE**
:::info
The JSON key file will automatically download. Keep it secure - this contains credentials to access your data.
:::
#### 2.6 Copy Service Account Email
From the service account details page, copy the **email address** (format: `service-account-name@project-id.iam.gserviceaccount.com`)
You'll need this in the next step.
### Step 3: Grant Analytics Permissions
Now you need to give the Service Account permission to read your GA4 data.
1. Log into [Google Analytics](https://analytics.google.com)
2. Click **Admin** (gear icon in bottom left)
3. In the **Property** column, click **Property access management**
4. Click the **+** icon (top right) → **Add users**
5. Enter the **Service Account email** from Step 2.6
6. Uncheck **Notify new users by email**
7. Select role: **Viewer**
8. Click **Add**
:::info
You've now granted the Data Platform read-only access to your GA4 data through the Service Account.
:::
### Step 4: Configure Source on Data Platform
1. In the Data Platform **Connectors**, find **Google Analytics** in the source store
2. Click **Select** to open the configuration screen
3. Fill in the required fields:
| Field | Description | Example |
| ------------------------ | ------------------------------------------- | ------------------------- |
| **Property ID** | Your GA4 Property ID from Step 1 | `123456789` |
| **Service Account JSON** | Contents of the JSON key file from Step 2.5 | Paste entire JSON content |
4. Click **Connect** to test the connection
5. If successful, click **Create** (top right)
:::warning
Remember to name your source before creating it. The technical name cannot be changed later and will be used in SDK calls.
:::
:::info
Your Google Analytics 4 source is now connected and ready to use!
:::
## Troubleshooting
### Common Issues
**"Invalid credentials" error**
- Verify the JSON key file content is complete and valid
- Ensure no extra spaces or characters when pasting
- Check that the Service Account email is correct
**"Permission denied" error**
- Confirm the Service Account has Viewer access to the GA4 property
- Wait 5-10 minutes for permissions to propagate
- Verify you're using the correct Property ID
**"API not enabled" error**
- Ensure **Google Analytics Data API** is enabled in your Google Cloud Project
- Confirm you're in the correct Google Cloud Project
**"Property not found" error**
- Verify the Property ID is correct (numeric, no spaces)
- Ensure you're using a GA4 Property ID, not a Universal Analytics View ID
- Check that the property hasn't been deleted
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-http.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Import files from a URL over HTTP
## Objective
_HTTP_ is a standard application protocol that allows you to import files from an URL or IP address.
Please note that this connector only works if the files are directly listed on an HTML page.
In most modern usage, you will want to use the [HTTP REST connector](https://docs.dataplatform.ovh.net/connectors-sources-rest.md).
## Add an HTTP source (for files)
### Configuration screen overview
Once you have found _HTTP Files_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below:
### Supported files and sizes
Even though the HTTP connector is able to list every file on your server, keep in mind that the metadata extraction in the [Connectors Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) only supports the following file types: .csv, .xls, .xlsx, .xml, .json, and .parquet.
Additionally, there are some file size limits to inspect the data on the fly using the meta data extraction, in particular for a:
- Complete extract: \~900 MB
- Sample extract: \~2.5 GB
### Configuring your source
You will need the following parameters to configure your source:
- **URL**: IP or Host address with a port (example: `127.0.0.1:8888`).
- **SSL verified (optional)**: In case you want to setup a secure connection.
- **User/Password (optional)**: In case you need to authenticate via login and password
Click _Connect_ to test your connection. If the connection is valid, the files that have been detected will be displayed in a list as shown in the picture below:
Check the files you want to import on the Platform and click on **Create** on the top right-hand corner to create the source.
:::info
Note that you can click on _Fetch_ to refresh the file list, in case of changes on the server.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-rest.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Import data from a REST API
## Objective
**REST** is a standard application protocol that allows you to request information from any compatible API and import data from the JSON response
## Add an HTTP REST source on Data Platform
### Configuration screen overview
Once you have found _HTTP REST_ in the **store**, click on _Select_ and you will be able to see the configuration screen as shown below.
### Supported files and sizes
:::info
The HTTP REST connector currently **only** supports response of type JSON.
:::
Additionally, there are some file size limits when launching a [metadata extract job](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md).
In particular for a:
- Complete extract: \~900 MB
- Sample extract: \~2.5 GB
### Configure your source
When creating the source, you will be required to input the following information:
#### API information
**URL**: The base URL or address of the API you want to request (you will register the [endpoints](#specify-endpoints-to-load-data-from) later on)\
**SSL verify**: Activate if you want to setup a secure connection\
**Authentication mode**: Activate if your API requires to authenticate (see below)
#### Authentication information
If your REST API requires authentication, configure it on the **Authentication** section by choosing an authentication mode. You can also skip this step if the API doesn't require any authentication.
Choose the method you want (POST or GET), followed by the URL on which you must authenticate (for instance `http://localhost/auth`).
Then, enter the _token path_ to use in the response and decide whether to keep the token in _cookies_, i.e. store authentication session in cookies.
Finally, configure the optional parameters below:
- **Body**: add an optional body of type json, form-data, form-urlencoded or inline
- **Query**: add your queries to the request
- **Header**: add the headers of your request
If you need to get a token from the authentication response, you must indicate in which key in the response JSON the token is stored. (commonly : `token`)
Once you've added the above details, click on _Connect_ to establish connection with your API and move on to endpoint registration.
### Specify endpoints to load data from
On Data Platform, each registered endpoint will correspond to a data source object, which you will be able to [load](https://docs.dataplatform.ovh.net/dpe-actions-load.md) into a distinct [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
Endpoints from your API can be added from the **Endpoints to connect** section. By default, there are none and you must add them manually.
The following parameters can be configured for each endpoints:
- **Query**: add your queries to the request for this endpoint
- **Header**: add the headers of your request for this endpoint
- **Options**: configure additional options such as timeout, pagination, or setting a name for the table which will contain the data from this endpoint
:::info
Variables can be injected in the aforementioned options. Check out the [reference down below](#inject-variables) for further information.
:::
You can import your preconfigured endpoint from a _Curl_ request by using the **Advanced mode** in a new endpoint.
#### Inject variables
**Authentication token**
If you activated the authentication endpoint and need to re-use the token previously obtained, you can enter `$token` in the value of the required key in the _Query_ section, or in the _Header_ section, _Body_ section or URL.
**Segmentation values**
This section outlines how to use variable placeholders within endpoints for customization purposes. These placeholders allow users to specify values dynamically, enhancing the flexibility and adaptability of the system.
Variables are enclosed within percentage (%) signs and follow the format: `(%VARIABLE_TYPE|DEFAULT_VALUE%)`.
VARIABLE\_TYPE: Describes the type of variable.
DEFAULT\_VALUE: Represents the default value that the variable will be replaced with if not provided by the user.
Variable Types:
segmentationValue: This variable type replaces the placeholder with the first value of segmentation.
Format: Numeric or Text.
Example: (%segmentationValue|1%)
segmentationValues: This variable type replaces the placeholder with all values of segmentation, formatted as an array.
Example: (%segmentationValues|\[1,2,3]%)
Users can incorporate these variables in the following components of an endpoint:
URL: Variables can be used within the URL to dynamically insert values.
Example: GET /api/data/(%segmentationValue|1%)/details
Body (Payload): When sending a payload in a request, variables can be used to include dynamic values.
Example:
json
\{
"segmentationValue": "(%segmentationValue|1%)",
"filter": "(%segmentationValues|\[]%)"
}
Query Parameters: Variables can be included in query parameters to customize the request.
Example: GET /api/data?segment=%segmentationValue|1%\&filters=%segmentationValues|\[]%
**Relative dates**
If you need to use a date parameter that is relative to the day of the query, you can use the following special joker composed by 2 parts separated by a pipe `|` :
`(%now|date%)` will be replaced by the date when the query happens (formatted as YYYY-MM-DD)
Possible values before the pipe:
- `(%dateMin|`: will use the date min configured in the DPE workflow parameters
- `(%dateMax|`: will use the date max configured in the DPE workflow parameters
- `%now|`: will use the current date and time when running
Possible values after the pipe to indicate the date format:
- `|date%)`: as `YYYY-MM-DD`
- `|datetime%)`: as `YYYY-MM-DD HH:mm:SS`
- `|timestamp%)`: as a UNIX timestamp
- `|%Y-%m-%d%)`: as your custom formatter
:::info
To learn more about the custom format, please refer to the [Python `datetime` documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
:::
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-hubspot.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract HubSpot CRM and marketing data
## Objective
The HubSpot connector extracts data from your HubSpot account: contacts, companies, deals, tickets, pipelines, forms, marketing emails, and more.
**25 endpoint types** covering the full HubSpot API: CRM, Marketing, CMS, Automation, and Settings.
The connector returns **raw JSON** from the HubSpot API. The platform automatically flattens nested objects into columns and stores the result in the lakehouse. No manual schema definition needed.
:::info
**Prerequisites:** A HubSpot account (the free plan is sufficient for API access. No paid plan required) with at least a few test records (contacts, companies) so you have data to extract. Sign up at [app.hubspot.com/signup-hubspot/crm](https://app.hubspot.com/signup-hubspot/crm).
:::
## 1. Get an API Token
### Create a Private App
1. Log in to HubSpot
2. Go to **Settings** (gear icon) > **Integrations** > **Private Apps**
3. Click **Create a private app**
4. Give it a name (e.g. `data-connector`)
5. Go to the **Scopes** tab and select the required permissions (see next section)
6. Click **Create app** > **Continue creating**
7. Copy the generated token (starts with `pat-na1-...` or `pat-eu1-...`)
:::info
**Note:** Private Apps may appear under a "Legacy" section in HubSpot's UI, but they remain the standard for server-side API integrations. "Personal Access Keys" only work for the HubSpot CLI, NOT for the REST API.
:::
### Configure Credentials
Paste your token in the **HubSpot API Key** field:
```
pat-eu1-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
```
## 2. Recommended Scopes
### Minimum scopes (basic CRM)
To extract contacts, companies, deals, and tickets:
| Scope | Grants access to |
| ---------------------------- | ----------------------------------------------- |
| `crm.objects.contacts.read` | Contacts, calls, emails, meetings, notes, tasks |
| `crm.objects.companies.read` | Companies |
| `crm.objects.deals.read` | Deals + deal pipelines |
| `tickets` | Tickets + ticket pipelines |
| `crm.schemas.contacts.read` | Contact properties |
| `crm.schemas.companies.read` | Company properties |
| `crm.schemas.deals.read` | Deal properties |
### Additional scopes (as needed)
| Scope | Grants access to |
| --------------------------------------- | ------------------------ |
| `forms` | Forms + form submissions |
| `content` | Marketing emails |
| `crm.lists.read` | Lists and segments |
| `e-commerce` | Products |
| `crm.objects.line_items.read` | Line items |
| `crm.objects.quotes.read` | Quotes |
| `crm.objects.feedback_submissions.read` | Feedback submissions |
| `crm.objects.leads.read` | Leads (Sales Hub Pro+) |
:::tip
Start with minimum scopes. You can always add more later in your Private App settings.
:::
## 3. Available Endpoint Types
### CRM (relational data)
| Endpoint | Description | Parameters |
| ---------------------------- | ------------------------------------------------------------------ | -------------------------------------------- |
| **crm\_objects** | 27 CRM object types (contacts, companies, deals...) | object\_type, properties\_filter, max\_items |
| **associations** | Relationships between objects (contact->deal, company->contact...) | from\_type, to\_type, max\_items |
| **association\_definitions** | Available relationship types between objects | from\_type, to\_type |
| **pipelines** | Pipelines with stages (deals or tickets) | pipeline\_object\_type |
| **pipeline\_audit** | Audit log for a specific pipeline | pipeline\_object\_type, pipeline\_id |
| **properties\_meta** | Property schema for an object type | property\_object\_type |
| **property\_groups** | Property groups | property\_object\_type |
| **lists** | Lists and segments | max\_items |
| **list\_memberships** | Member record IDs for a specific list | list\_id, max\_items |
### Marketing
| Endpoint | Description | Parameters |
| --------------------- | ------------------------------- | -------------------- |
| **marketing\_emails** | Email campaigns with statistics | max\_items |
| **forms** | Form definitions | max\_items |
| **form\_submissions** | Submissions for a specific form | form\_id, max\_items |
| **campaigns** | Marketing campaigns | max\_items |
### CMS (content)
| Endpoint | Description | Parameters |
| ------------------ | ---------------------------- | ---------- |
| **blog\_posts** | Blog articles | max\_items |
| **site\_pages** | Website pages | max\_items |
| **landing\_pages** | Landing pages | max\_items |
| **hubdb\_tables** | HubDB structured data tables | max\_items |
### Automation
| Endpoint | Description | Parameters |
| ------------- | -------------------- | ---------- |
| **workflows** | Automation workflows | max\_items |
| **sequences** | Sales sequences | max\_items |
### Settings / Admin
| Endpoint | Description | Parameters |
| -------------------- | ------------------------ | -------------------------------------------------- |
| **users** | Account users | max\_items |
| **imports** | CRM import history | max\_items |
| **crm\_schemas** | Custom object schemas | max\_items |
| **custom\_events** | Behavioral events | event\_object\_type, event\_object\_id, max\_items |
| **timeline\_events** | Timeline event templates | app\_id |
## 4. Add a HubSpot source on Data Platform
1. In the Data Platform **Connectors**, find **HubSpot** in the source store and click **Select**.
2. Paste the Private App token from [step 1](#1-get-an-api-token) into the **HubSpot API Key** field.
3. Click **Connect** to verify the credentials.
4. Click **Add an Endpoint**, choose an endpoint type from the dropdown (e.g. `crm_objects`), then fill in the parameters that endpoint requires (for example `object_type: contacts`, an optional `properties_filter`, and an optional `max_items`).
5. Repeat step 4 for every additional endpoint you want to ingest in this source. Each endpoint becomes a separate table in the lakehouse.
6. Name the source and click **Create**.
:::warning
The **technical name** cannot be changed after the source is created. It is used when opening the source via the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## 5. Quick Example: Extract Contacts
1. **Endpoint**: `crm_objects`
2. **Object Type**: `contacts`
3. **Properties Filter**: `email, firstname, lastname, phone, company, jobtitle`
4. **Max Items**: `100` (or empty to extract all)
The connector returns raw JSON from the HubSpot API. The platform automatically flattens nested objects (e.g. `properties.email` becomes a column `properties_email`) and stores the result in the lakehouse.
## 6. Best Practices
### Use the properties filter
By default, the connector uses HubSpot's native default set (6 core fields for contacts: email, firstname, lastname, createdate, lastmodifieddate, hs\_object\_id). Use `properties_filter` to pick a specific list of properties.
### Start with max\_items
When setting up, set `max_items` to 10 or 100 to test quickly. Remove the limit once you're satisfied with the output.
### Pipelines as a reference table
Extract pipelines separately to get a reference table. You can then join with deals/tickets on the pipeline and stage fields.
### Use properties\_meta to discover fields
The `properties_meta` endpoint returns the full data dictionary for any object type. Useful to find out which fields exist and their types.
## 7. HubSpot Plan Limits
Some endpoints require paid plans:
| Plan | Accessible endpoints |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Free** | crm\_objects, pipelines, properties\_meta, forms, blog\_posts, site\_pages, landing\_pages, associations, property\_groups, lists, marketing\_emails, form\_submissions, pipeline\_audit, list\_memberships |
| **Pro** | + workflows, sequences, users, campaigns |
| **Enterprise** | + conversations, custom\_events, timeline\_events, crm\_schemas, hubdb\_tables, imports |
Endpoints not available on your plan or missing the required scope return a 403 error. The connector does **not** catch these errors. The extraction will fail with an `HTTPError`. Make sure your Private App has the necessary scopes enabled before selecting an endpoint.
:::info
**Note:** The plan classification above is approximate. Some 403 errors may be caused by missing **scopes** on your Private App rather than plan limitations. If an endpoint returns 403, check your Private App's scope configuration first.
:::
:::info
For detailed technical information (auth internals, full endpoint reference, scopes, pagination, output format, limitations), see the [HubSpot Technical Reference](https://docs.dataplatform.ovh.net/connectors-sources-hubspot-technical-reference.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-hubspot-technical-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# HubSpot: Technical Reference
## Objective
This is the technical companion to the main [HubSpot connector](https://docs.dataplatform.ovh.net/connectors-sources-hubspot.md) documentation. It covers authentication internals, the full endpoint reference, scopes, pagination, output format, and limitations, everything needed to integrate the connector into a data pipeline.
## Authentication
### Supported Methods
| Method | Format | Use case |
| ----------------------------------- | ------------------------------ | ------------------------ |
| **Private App Token** (recommended) | `pat-na1-xxx` or `pat-eu1-xxx` | Server-side integrations |
| **OAuth2 Access Token** | Standard OAuth bearer token | Public distributed apps |
All methods use the same Bearer header:
```http
Authorization: Bearer {token}
```
### Deprecated authentication methods
| Method | Why |
| ------------------- | --------------------------------------------- |
| API Key (`hapikey`) | Sunset since November 2022 |
| Personal Access Key | CLI only, which returns 401 on REST API calls |
### Credentials Format
```json
{
"token": "pat-eu1-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
```
## Architecture
The connector returns raw JSON from the HubSpot API. The platform takes over from there. It auto-discovers the schema from the JSON payload, flattens nested objects into dot-notation columns (slugified to lowercase with underscores), and stores the result in the lakehouse, queryable via Trino. Any new field HubSpot adds to an object appears automatically on the next extraction; you don't define a schema, list columns, or write any transformation code.
The connector itself is responsible for authentication (Bearer token), endpoint routing, pagination (cursor or offset depending on the endpoint), and rate-limit retries.
## CRM Object Types
### 27 Standard Objects
The connector exposes the 27 standard HubSpot CRM object types. Each is reachable via `/crm/v3/objects/{object_type}` (except owners, see below).
#### Core CRM
| Object Type | API Path |
| ----------- | --------------------------- |
| `contacts` | `/crm/v3/objects/contacts` |
| `companies` | `/crm/v3/objects/companies` |
| `deals` | `/crm/v3/objects/deals` |
| `tickets` | `/crm/v3/objects/tickets` |
#### Engagements
| Object Type | API Path |
| ---------------- | -------------------------------- |
| `calls` | `/crm/v3/objects/calls` |
| `emails` | `/crm/v3/objects/emails` |
| `meetings` | `/crm/v3/objects/meetings` |
| `notes` | `/crm/v3/objects/notes` |
| `tasks` | `/crm/v3/objects/tasks` |
| `communications` | `/crm/v3/objects/communications` |
| `postal_mail` | `/crm/v3/objects/postal_mail` |
#### E-commerce & Sales
| Object Type | API Path |
| ------------ | ---------------------------- |
| `products` | `/crm/v3/objects/products` |
| `line_items` | `/crm/v3/objects/line_items` |
| `quotes` | `/crm/v3/objects/quotes` |
#### Commerce
| Object Type | API Path |
| --------------- | ------------------------------- |
| `invoices` | `/crm/v3/objects/invoices` |
| `subscriptions` | `/crm/v3/objects/subscriptions` |
| `orders` | `/crm/v3/objects/orders` |
| `payments` | `/crm/v3/objects/payments` |
#### Commerce Hub
| Object Type | API Path |
| ----------- | --------------------------- |
| `carts` | `/crm/v3/objects/carts` |
| `discounts` | `/crm/v3/objects/discounts` |
| `fees` | `/crm/v3/objects/fees` |
| `taxes` | `/crm/v3/objects/taxes` |
#### Sales Hub
| Object Type | API Path | Notes |
| ----------- | ----------------------- | -------------- |
| `leads` | `/crm/v3/objects/leads` | Sales Hub Pro+ |
| `goals` | `/crm/v3/objects/goals` | |
#### Service & Scheduling
| Object Type | API Path | Notes |
| ---------------------- | -------------------------------------- | ----------- |
| `feedback_submissions` | `/crm/v3/objects/feedback_submissions` | Service Hub |
| `appointments` | `/crm/v3/objects/appointments` | Scheduling |
| `services` | `/crm/v3/objects/services` | |
### Special: Owners
Owners use a dedicated endpoint (`GET /crm/v3/owners`) instead of `/crm/v3/objects/owners`. The connector handles this automatically when `object_type=owners` is selected.
## Endpoint Reference
### crm\_objects
Extract records from any CRM object type.
| Parameter | Type | Required | Description |
| ------------------- | ------ | -------- | ---------------------------------------------------------- |
| `object_type` | select | Yes | One of 27 CRM types + `owners` |
| `max_items` | number | No | Maximum records to extract (empty = all) |
| `properties_filter` | tags | No | Specific properties to fetch (empty = HubSpot default set) |
**API**: `GET /crm/v3/objects/{object_type}?properties={props}&limit=100&after={cursor}`
**Pagination**: Cursor-based (`paging.next.after`)
**Output**: Raw JSON. Each record has `id`, `createdAt`, `updatedAt`, `archived`, and a nested `properties` dict containing all requested property values.
**Properties behavior**:
- Empty filter: fetches ALL properties via `GET /crm/v3/properties/{type}` first, then requests all of them
- With filter: requests only the specified properties
- HubSpot returns all property values as strings (even numbers and dates)
| Object Type | Approx. Default Properties |
| ----------- | -------------------------- |
| Contacts | \~370+ properties |
| Companies | \~250+ properties |
| Deals | \~200+ properties |
### associations
Extract relationships between CRM objects.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------------- |
| `from_type` | select | Yes | Source object type (14 options) |
| `to_type` | select | Yes | Target object type (14 options) |
| `max_items` | number | No | Max source records to process |
**API**: `POST /crm/v4/associations/{from_type}/{to_type}/batch/read`
**Pagination**: Cursor-based on source objects, batch POST for association lookup (max 1000 IDs per request)
**Output**: Raw JSON. Each result contains `from` and `to` objects with IDs and association metadata.
### association\_definitions
Get the available association types between two object types.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------ |
| `from_type` | select | Yes | Source object type |
| `to_type` | select | Yes | Target object type |
**API**: `GET /crm/v4/associations/{from_type}/{to_type}/labels`
**Pagination**: None (single GET)
**Output**: Raw JSON, association type definitions with category, type ID, and label.
### pipelines
Extract pipeline definitions with their stages.
| Parameter | Type | Required | Description |
| ---------------------- | ------ | -------- | -------------------- |
| `pipeline_object_type` | select | Yes | `deals` or `tickets` |
**API**: `GET /crm/v3/pipelines/{pipeline_object_type}`
**Pagination**: None (single GET, returns all pipelines)
**Output**: Raw JSON. Each pipeline contains `id`, `label`, `displayOrder`, `createdAt`, `updatedAt`, and a nested `stages` array. The platform flattens stages into separate rows automatically.
### pipeline\_audit
Audit log for a specific pipeline.
| Parameter | Type | Required | Description |
| ---------------------- | ------ | -------- | -------------------------------------------------- |
| `pipeline_object_type` | select | Yes | `deals` or `tickets` |
| `pipeline_id` | text | Yes | Pipeline ID (use `pipelines` endpoint to find IDs) |
**API**: `GET /crm/v3/pipelines/{object_type}/{pipeline_id}/audit`
**Pagination**: None (single GET)
**Output**: Raw JSON, audit entries as returned by the API.
### properties\_meta
Extract the data dictionary (property schema) for an object type.
| Parameter | Type | Required | Description |
| ---------------------- | ------ | -------- | ------------------------ |
| `property_object_type` | select | Yes | Object type (12 options) |
**API**: `GET /crm/v3/properties/{property_object_type}`
**Pagination**: None (single GET)
**Output**: Raw JSON. Each property has `name`, `label`, `type`, `fieldType`, `groupName`, `description`, and more.
### property\_groups
Extract property groups for an object type.
| Parameter | Type | Required | Description |
| ---------------------- | ------ | -------- | ------------------------ |
| `property_object_type` | select | Yes | Object type (12 options) |
**API**: `GET /crm/v3/properties/{property_object_type}/groups`
**Pagination**: None (single GET)
**Output**: Raw JSON, group definitions as returned by the API.
### lists
Extract list and segment definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------ |
| `max_items` | number | No | Maximum lists to extract |
**API**: `GET /crm/v3/lists`
**Pagination**: Cursor-based
**Output**: Raw JSON, list definitions as returned by the API.
### list\_memberships
Get member record IDs from a specific list.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------------------------------------- |
| `list_id` | text | Yes | HubSpot List ID (ILS number, found in Contacts > Lists) |
| `max_items` | number | No | Maximum members to extract |
**API**: `GET /crm/v3/lists/{list_id}/memberships`
**Pagination**: Cursor-based
**Output**: Raw JSON, membership records as returned by the API.
### marketing\_emails
Extract marketing email definitions with statistics.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------- |
| `max_items` | number | No | Maximum emails to extract |
**API**: `GET /marketing/v3/emails`
**Pagination**: Cursor-based
**Output**: Raw JSON, email campaign data with nested statistics, flattened by the platform.
### forms
Extract form definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------ |
| `max_items` | number | No | Maximum forms to extract |
**API**: `GET /marketing/v3/forms`
**Pagination**: Cursor-based
**Output**: Raw JSON, form definitions as returned by the API.
### form\_submissions
Extract submissions for a specific form.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------------------------------------ |
| `form_id` | text | Yes | Form ID (find in Marketing > Forms > form details URL) |
| `max_items` | number | No | Maximum submissions to extract |
**API**: `GET /form-integrations/v1/submissions/forms/{form_id}`
**Pagination**: **Offset-based** (v1 API, uses `offset` + `hasMore`, NOT cursor-based)
**Output**: Raw JSON, submission data as returned by the API.
### conversations
Extract conversation threads (chat, email, bot).
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | -------------------------- |
| `max_items` | number | No | Maximum threads to extract |
**API**: `GET /conversations/v3/conversations/threads`
**Pagination**: Cursor-based
**Output**: Raw JSON, thread data as returned by the API.
**Requires**: Conversations scope + appropriate HubSpot plan
### campaigns
Extract marketing campaign definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------------------- |
| `max_items` | number | No | Maximum campaigns to extract |
**API**: `GET /marketing/v3/campaigns`
**Pagination**: Cursor-based
**Output**: Raw JSON, campaign data as returned by the API.
### blog\_posts
Extract CMS blog articles.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------ |
| `max_items` | number | No | Maximum posts to extract |
**API**: `GET /cms/v3/blogs/posts`
**Pagination**: Cursor-based
**Output**: Raw JSON, blog post data (title, content, author, publish date, etc.).
### site\_pages
Extract CMS website pages.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------ |
| `max_items` | number | No | Maximum pages to extract |
**API**: `GET /cms/v3/pages/site-pages`
**Pagination**: Cursor-based
**Output**: Raw JSON, page data as returned by the API.
### landing\_pages
Extract CMS landing pages.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------ |
| `max_items` | number | No | Maximum pages to extract |
**API**: `GET /cms/v3/pages/landing-pages`
**Pagination**: Cursor-based
**Output**: Raw JSON, page data as returned by the API.
### workflows
Extract automation workflow definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------------------- |
| `max_items` | number | No | Maximum workflows to extract |
**API**: `GET /automation/v3/workflows`
**Pagination**: None (single GET, data key is `workflows`, not `results`)
**Output**: Raw JSON, workflow definitions as returned by the API.
### sequences
Extract sales sequence definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------------------- |
| `max_items` | number | No | Maximum sequences to extract |
**API**: `GET /automation/v4/sequences`
**Pagination**: Cursor-based
**Output**: Raw JSON, sequence data as returned by the API.
**Requires**: Sales Hub Pro+
### users
Extract account users.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------ |
| `max_items` | number | No | Maximum users to extract |
**API**: `GET /settings/v3/users`
**Pagination**: Cursor-based
**Output**: Raw JSON, user data as returned by the API.
### imports
Extract CRM import history.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | --------------------------------- |
| `max_items` | number | No | Maximum import records to extract |
**API**: `GET /crm/v3/imports`
**Pagination**: Cursor-based
**Output**: Raw JSON, import records as returned by the API.
### crm\_schemas
Extract custom object schema definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | -------------------------- |
| `max_items` | number | No | Maximum schemas to extract |
**API**: `GET /crm/v3/schemas`
**Pagination**: None (single GET)
**Output**: Raw JSON, schema definitions as returned by the API.
### custom\_events
Extract behavioral events for a specific CRM record.
| Parameter | Type | Required | Description |
| ------------------- | ------ | -------- | ----------------------------------------------------- |
| `event_object_type` | select | Yes | CRM object type (contacts, companies, deals, tickets) |
| `event_object_id` | text | Yes | HubSpot record ID |
| `max_items` | number | No | Maximum events to extract |
**API**: `GET /events/v3/events?objectType={type}&objectId={id}`
**Pagination**: Cursor-based (with `extra_params`)
**Output**: Raw JSON, event data as returned by the API.
**Requires**: Marketing Hub Enterprise
### timeline\_events
Extract timeline event templates for an integration app.
| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ------------------------------------------- |
| `app_id` | text | Yes | HubSpot App ID (found in developer account) |
**API**: `GET /crm/v3/timeline/{app_id}/event-templates`
**Pagination**: None (single GET)
**Output**: Raw JSON, event template data as returned by the API.
### hubdb\_tables
Extract HubDB table definitions.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------- |
| `max_items` | number | No | Maximum tables to extract |
**API**: `GET /cms/v3/hubdb/tables`
**Pagination**: Cursor-based
**Output**: Raw JSON, table definitions as returned by the API.
## Scopes by Endpoint
### CRM Objects
| Object Type | Required Scope |
| ------------------------------------- | --------------------------------------- |
| contacts | `crm.objects.contacts.read` |
| companies | `crm.objects.companies.read` |
| deals | `crm.objects.deals.read` |
| tickets | `tickets` |
| products | `e-commerce` |
| line\_items | `crm.objects.line_items.read` |
| quotes | `crm.objects.quotes.read` |
| calls, emails, meetings, notes, tasks | `crm.objects.contacts.read` |
| communications | `crm.objects.contacts.read` |
| feedback\_submissions | `crm.objects.feedback_submissions.read` |
| leads | `crm.objects.leads.read` |
| invoices | `crm.objects.invoices.read` |
| subscriptions | `crm.objects.subscriptions.read` |
| goals | `crm.objects.goals.read` |
| orders | `crm.objects.orders.read` |
| payments | `crm.objects.payments.read` |
| owners | `crm.objects.owners.read` |
### Other Endpoints
| Endpoint | Required Scope |
| ---------------------------------------- | ------------------------------------------------------------------------- |
| pipelines, pipeline\_audit | `crm.objects.deals.read` or `tickets` (depends on pipeline\_object\_type) |
| properties\_meta, property\_groups | Same scope as the target object type |
| lists, list\_memberships | `crm.lists.read` |
| marketing\_emails | `content` |
| forms, form\_submissions | `forms` |
| associations, association\_definitions | Scopes for both the source and target object types |
| campaigns | `content` |
| blog\_posts, site\_pages, landing\_pages | `content` |
For endpoints not listed above (workflows, sequences, users, imports, crm\_schemas, conversations, custom\_events, hubdb\_tables, timeline\_events), refer to [HubSpot's scope reference](https://developers.hubspot.com/docs/api/working-with-oauth#scopes) for the authoritative scope name.
## Pagination
The connector uses three pagination strategies depending on the endpoint:
### Cursor-based (most endpoints)
```http
GET /crm/v3/objects/contacts?limit=100&after=NTI1Cg==
```
Response:
```json
{
"results": [...],
"paging": {
"next": { "after": "NTI1Cg==" }
}
}
```
When `paging.next.after` is absent, all data has been fetched.
### Offset-based (form submissions only)
```http
GET /form-integrations/v1/submissions/forms/{id}?limit=50&offset=0
```
Response:
```json
{
"results": [...],
"hasMore": true,
"offset": 50
}
```
When `hasMore` is false, all data has been fetched.
### No pagination (single GET)
Some endpoints return all data in a single response: `pipelines`, `pipeline_audit`, `properties_meta`, `property_groups`, `association_definitions`, `workflows`, `timeline_events`, `crm_schemas`.
## Rate Limits
### Limits by Plan
Refer to [HubSpot's official rate limit documentation](https://developers.hubspot.com/docs/api/usage-details) for current limits. Limits vary by plan and API endpoint.
### Rate Limit Handling
The connector automatically handles `429 Too Many Requests` responses:
1. Reads the `Retry-After` header (seconds to wait)
2. Falls back to 10 seconds if header is missing
3. Retries the request after waiting
## Output Format
### Raw JSON (connector output)
The connector returns **raw JSON** from the HubSpot API via `handle_api_extraction(data, limit, return_type)`. The data is a list of dicts, exactly as returned by the API.
Example CRM object (contacts):
```json
{
"id": "123",
"createdAt": "2024-01-15T10:30:00.000Z",
"updatedAt": "2024-03-20T14:22:00.000Z",
"archived": false,
"properties": {
"email": "john@example.com",
"firstname": "John",
"lastname": "Doe",
"createdate": "2024-01-15T10:30:00.000Z"
}
}
```
### Flattened output (lakehouse)
The platform automatically flattens the raw JSON into a flat table. Nested keys become column names with underscores:
| Raw JSON path | Lakehouse column |
| ---------------------- | ---------------------- |
| `id` | `id` |
| `createdAt` | `createdat` |
| `properties.email` | `properties_email` |
| `properties.firstname` | `properties_firstname` |
Column names are **slugified**: lowercase, dots/special chars replaced by underscores, must start with a letter or underscore.
## Limitations
- **Property values are always strings**, even numbers and dates. Cast them in your downstream processing.
- **Custom objects are not supported**: the `object_type` parameter is a fixed select list of 27 standard CRM types + owners. Custom object types cannot be extracted with this connector.
- **Some object types require paid plans**: invoices, subscriptions, and goals may require Sales Hub or Commerce Hub. Leads require Sales Hub Professional+. Commerce objects (carts, discounts, fees, taxes) require Commerce Hub. The API returns 403 if unavailable.
- **Form submissions use v1 API**: the only endpoint still on the legacy v1 API. Uses offset pagination instead of cursor-based.
- **Behavioral events**: require Marketing Hub Enterprise and a specific record ID (cannot bulk-extract all events).
- **Column names in lakehouse are slugified**: `properties.email` becomes `properties_email`. This is handled by the platform, not the connector.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-blob-storage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Microsoft Azure Blob Storage
## Objective
The Microsoft Azure Blob Storage connector lets you retrieve files stored in Microsoft Azure's object storage system: [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/).
## Get your access credentials from Azure
You need to obtain the following information from Azure:
- the name of the Azure storage account to connect to
- a key for this storage account
- the name of the blob container to connect to
### Get your storage account name and key
On the Microsoft Azure console, open the **storage account** that you want to connect to and open the _Access keys_ tab.
Retrieve the following information:
- _storage account name_
- _key_
:::warning
If you are on a [Dedicated plan](https://docs.dataplatform.ovh.net/manage-pricing.md#subscriptions-and-billing-accounts) with a self-hosted cluster, you shouldn't connect to the resource group/storage account dedicated to the Platform resources.
:::
If you don't have a separate storage account yet, use the below documentation to create one.
#### Create a new storage account
To create a new [Microsoft Azure storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json) from the Microsoft Azure console, select _Storage accounts_ and click on **Create**.
Follow the steps to add a storage account filling in the required fields and custom settings. The _storage account name_ will be needed to connect from the Platform.
:::warning
If you are on a [Dedicated plan](https://docs.dataplatform.ovh.net/manage-pricing.md#subscriptions-and-billing-accounts) with a self-hosted cluster, you shouldn't use the resource group dedicated to the Platform resources.
:::
Once your storage account is created, go back to the first steps up above to retrieve its name and key.
### Get your blob container name
In the Microsoft Azure console, open the _Storage browser_ tab.
In the service account you just used, open the section _Blob containers_.
If you have an existing container that you want to use, retrieve its _name_ at this step. Else, create a new container.
#### Create a new blob container
Click on **Add container**. The _name_ you fill in will be needed to connect from the Platform.
## Add a Microsoft Azure Blob Storage source on the Platform
### Configuration screen overview
Once you have found _Microsoft Azure Blob Storage_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below:
### Configuring your source
#### Connect to Microsoft Azure Blob Storage
When creating the source, you will be required to input the following information :
- **Account name**: the name of the storage account obtained [here](#get-your-storage-account-name-and-key)
- **Account key**: the key for this storage account obtained [here](#get-your-storage-account-name-and-key)
- **Container name**: the name of the blob container obtained [here](#get-your-blob-container-name)
- **Path of files** _(optional)_: the full path of a specific folder or file to connect to
Once you add the above details click on **Connect** to establish a connection with Microsoft Azure Blob Storage.
:::info
If everything works successfully, you should see the list of parsed files appearing on the page. This may take up to a few seconds.
:::
To finish, click on the **Create** button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to access the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-mysql.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use MySQL database tables as Data Platform sources
## Objective
MySQL is an open-source relational database management system. The Platform allows you to use the tables in a MySQL database as sources for your application.
## Add a MySQL source on the Platform
### Configure your connection
When creating the source, you will be required to input the following information :
- **Host**: the host name of the MySQL database.
- **Port**: the port address of the MySQL database.
- **Database**: the name of the MySQL database.
- **Username**: the user name for the MySQL database.
- **Password**: the password for the Username you are using.
- **SSL**: if activated, encrypts the information exchanged between the Data Platform and the database.
### Select the tables you want
After establishing a connection with your MySQL database your tables will be fetched and displayed :
Select which tables you want to be included in the source by checking the boxes to the left of the table name, as indicated in the image above.
:::info
If the state of the database changes, you can use the _Fetch_ button to refresh the tables displayed and include them in your source.
:::
Once you follow the steps above, click on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-odoo.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract Odoo ERP data with the Data Platform connector
## Objective
The Odoo connector extracts data from any Odoo ERP instance: contacts, CRM leads, sales orders, invoices, products, inventory, HR, projects, and more.
**26 predefined models** covering 9 domains + a **custom model** endpoint for any Odoo model + a **model fields** endpoint for schema introspection.
The connector returns **raw JSON** from the Odoo JSON-RPC API. The platform automatically flattens nested objects into columns and stores the result in the lakehouse. No manual schema definition needed.
**Compatible with Odoo 14–19.**
## 1. Get an API Key
1. Log in to your Odoo instance
2. Click your **avatar** (top right) > **My Profile**
3. Go to the **Account Security** tab
4. Click **New API Key**
5. Enter a description (e.g., `data-platform-connector`)
6. Click **Generate Key**
7. **Copy the key immediately**: it is shown only once
:::info
**Odoo Online:** API access requires the **Custom plan**. See [Odoo Pricing](https://www.odoo.com/pricing).
:::
## 2. Configure Credentials
| Field | Required | Description |
| ------------ | -------- | ------------------------------------------------------------------------ |
| **URL** | Yes | Your Odoo instance URL (e.g., `https://mycompany.odoo.com`) |
| **Login** | Yes | Your Odoo login email |
| **API Key** | Yes | The API key generated above |
| **Database** | No | Auto-detected for `*.odoo.com` URLs. Required for self-hosted instances. |
## 3. Available Endpoint Types
### models: Predefined Models
26 commonly-used Odoo models, grouped by domain:
| Domain | Models |
| ------------- | ---------------------------------------------------------- |
| **Contacts** | res.partner |
| **CRM** | crm.lead, crm.stage, crm.team |
| **Sales** | sale.order, sale.order.line |
| **Purchases** | purchase.order, purchase.order.line |
| **Invoicing** | account.move, account.move.line, account.journal |
| **Products** | product.template, product.product, product.category |
| **Inventory** | stock.picking, stock.move, stock.warehouse, stock.location |
| **HR** | hr.employee, hr.department |
| **Projects** | project.project, project.task |
| **System** | res.users, res.company, res.country, res.currency |
**Parameters:**
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | ------------------------------------------------ |
| Model | select | Yes | Odoo model to extract |
| Fields Filter | tags | No | Specific fields to include (empty = all) |
| Domain Filter | text | No | Odoo domain filter in JSON (empty = all records) |
| Max Items | number | No | Max records to extract (empty = all) |
### custom\_model: Any Odoo Model
For models not in the predefined list (e.g., `helpdesk.ticket`, `mrp.production`, `fleet.vehicle`).
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | --------------------------------------------------- |
| Model Name | text | Yes | Full technical model name (e.g., `helpdesk.ticket`) |
| Fields Filter | tags | No | Specific fields to include |
| Domain Filter | text | No | Domain filter in JSON |
| Max Items | number | No | Max records |
### model\_fields: Schema Introspection
Returns field definitions for any model (name, type, required, help text, relation). Useful to discover available fields before extracting data.
| Parameter | Type | Required | Description |
| ---------- | ---- | -------- | -------------------------------------- |
| Model Name | text | Yes | Model to inspect (e.g., `res.partner`) |
## 4. Add an Odoo source on Data Platform
1. In the Data Platform **Connectors**, find **Odoo** in the source store and click **Select**.
2. Fill in the connection fields from [step 2](#2-configure-credentials), **URL**, **Login**, **API Key**, and (for self-hosted) **Database**.
3. Click **Connect** to verify the credentials. The connector authenticates via JSON-RPC and resolves your Odoo `uid`.
4. Click **Add an Endpoint**, choose `models`, `custom_model`, or `model_fields` from the dropdown, and fill in the parameters that endpoint requires (e.g. **Model**, optional **Fields Filter**, optional **Domain Filter**, optional **Max Items**).
5. Repeat step 4 for every additional endpoint you want to ingest in this source. Each becomes a separate table in the lakehouse.
6. Name the source and click **Create**.
:::warning
The **technical name** cannot be changed after the source is created. It is used when opening the source via the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## 5. Quick Example: Extract Contacts
1. **Endpoint**: `models`
2. **Model**: `res.partner` (Contacts)
3. **Fields Filter**: `name, email, phone, is_company, city, country_id`
4. **Max Items**: `100` (or empty to extract all)
The connector returns raw JSON from the Odoo API. The platform automatically flattens nested objects and stores the result in the lakehouse.
## 6. Domain Filter Syntax
The domain filter uses Odoo's **Polish (prefix) notation**. Enter it as a JSON array:
```json
[["state", "=", "sale"], ["amount_total", ">", 1000]]
```
Common operators: `=`, `!=`, `>`, `<`, `>=`, `<=`, `in`, `not in`, `like`, `ilike`
**OR conditions** use the `|` prefix:
```json
["|", ["type", "=", "lead"], ["type", "=", "opportunity"]]
```
Leave empty `[]` to extract all records.
## 7. Best Practices
### Use the fields filter
Without a filter, Odoo returns **all fields**: some models have 200+ fields. Use the fields filter to keep only what you need, especially for large models like `res.partner`.
### Start with max\_items
Set `max_items` to 10 or 100 to test quickly. Remove the limit once satisfied.
### Use model\_fields to discover fields
The `model_fields` endpoint returns the complete schema for any model. Use it to find field names, types, and relationships before setting up extraction.
### Stages and reference tables
Extract `crm.stage`, `account.journal`, `product.category` as reference tables. Join them with main data on the stage/journal/category ID fields.
## 8. Known Limitations
- **Model availability depends on installed apps.** Models like `crm.lead` require the CRM app, `sale.order` requires Sales, etc. Use `custom_model` to test if a model exists on your instance.
- **Some fields require specific user groups.** For example, `project.project.stage_id` requires the "Use Stages on Project" group. If you get a permission error, use `fields_filter` to exclude restricted fields.
- **Odoo Online API access** requires the Custom plan. See [Odoo Pricing](https://www.odoo.com/pricing).
:::info
For detailed technical information (auth internals, full endpoint reference, pagination, rate limits, output format, domain filter reference, version compatibility, limitations), see the [Odoo Technical Reference](https://docs.dataplatform.ovh.net/connectors-sources-odoo-technical-reference.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-odoo-technical-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Odoo: Technical Reference
## Objective
This is the technical companion to the main [Odoo connector](https://docs.dataplatform.ovh.net/connectors-sources-odoo.md) documentation. It covers authentication internals, the full endpoint reference, pagination, rate limits, output format, the domain filter language, version compatibility, and limitations, everything needed to integrate the connector into a data pipeline.
## Authentication
### Protocol
The connector uses **JSON-RPC 2.0** via `POST {url}/jsonrpc`. All requests go to a single endpoint with different JSON bodies.
### Credentials
| Field | Required | Description |
| ---------- | -------- | -------------------------------------------------------------------- |
| `url` | Yes | Odoo instance URL (e.g., `https://mycompany.odoo.com`) |
| `login` | Yes | Odoo user email |
| `api_key` | Yes | API key generated from your Odoo profile |
| `database` | No | Auto-detected from `*.odoo.com` subdomain. Required for self-hosted. |
### Auth Flow
1. `configure()` calls `authenticate(db, login, api_key)` via JSON-RPC
2. Returns `uid` (integer user ID)
3. Every subsequent call passes `(db, uid, api_key)`: stateless, no session cookie
### Odoo Online Requirement
External API access requires the **Custom plan** on Odoo Online. Free and Standard plans do not include API access. See [Odoo Pricing](https://www.odoo.com/pricing).
## Architecture
The connector returns raw JSON from the Odoo JSON-RPC API. The platform takes over from there. It auto-discovers the schema from the JSON payload, flattens nested objects into columns, and stores the result in the lakehouse, queryable via Trino. Any new field that appears in your Odoo instance shows up automatically on the next extraction.
The connector itself is responsible for authentication (JSON-RPC `authenticate` to obtain a `uid`), endpoint routing, data extraction (a single `search_read` call shared by every model), schema introspection (`fields_get`), offset-based pagination, and rate-limit retries.
Because every Odoo model is reachable through the same `search_read` JSON-RPC call, the connector uses a single extraction path for all 26 predefined models, the `custom_model` endpoint, and any model surfaced through `model_fields`. Adding a new model means adding its name to the dropdown. No per-model code is needed.
## Endpoint Reference
### models
Extract records from any of the 26 predefined Odoo models.
| Parameter | Type | Required | Description |
| --------------- | ------ | -------- | ------------------------------------------------ |
| `object_type` | select | Yes | Odoo model (26 options) |
| `fields_filter` | tags | No | Specific fields to include (empty = all) |
| `domain_filter` | text | No | Odoo domain filter as JSON (empty = all records) |
| `max_items` | number | No | Max records (empty = all) |
**API call**: `execute_kw(model, "search_read", [domain], {fields, limit, offset, order})`
**Pagination**: Offset-based (limit=80, offset increments by 80)
**Output**: Raw JSON, list of dicts as returned by the Odoo API.
**Predefined models (26):**
| Domain | Models |
| --------- | ------------------------------------------------------------------ |
| Contacts | `res.partner` |
| CRM | `crm.lead`, `crm.stage`, `crm.team` |
| Sales | `sale.order`, `sale.order.line` |
| Purchases | `purchase.order`, `purchase.order.line` |
| Invoicing | `account.move`, `account.move.line`, `account.journal` |
| Products | `product.template`, `product.product`, `product.category` |
| Inventory | `stock.picking`, `stock.move`, `stock.warehouse`, `stock.location` |
| HR | `hr.employee`, `hr.department` |
| Projects | `project.project`, `project.task` |
| System | `res.users`, `res.company`, `res.country`, `res.currency` |
### custom\_model
Extract records from any Odoo model not in the predefined list.
| Parameter | Type | Required | Description |
| --------------- | ------ | -------- | --------------------------------------------------- |
| `model_name` | text | Yes | Full technical model name (e.g., `helpdesk.ticket`) |
| `fields_filter` | tags | No | Specific fields |
| `domain_filter` | text | No | Domain filter as JSON |
| `max_items` | number | No | Max records |
**API call**: Same as `models`: `execute_kw(model_name, "search_read", ..)`
**Output**: Raw JSON, same format as `models`.
**Use cases**: `helpdesk.ticket`, `mrp.production`, `fleet.vehicle`, `event.event`, or any custom model.
### model\_fields
Returns field definitions for any Odoo model (schema introspection).
| Parameter | Type | Required | Description |
| ------------ | ---- | -------- | -------------------------------------- |
| `model_name` | text | Yes | Model to inspect (e.g., `res.partner`) |
**API call**: `execute_kw(model_name, "fields_get", [], {attributes: [string, type, required, help, readonly, relation]})`
**Output**: Raw JSON, list of dicts, each with `field_name`, `string` (label), `type`, `required`, `help`, `readonly`, `relation` (for relational fields).
**Use case**: Discover available fields and their types before setting up `fields_filter` on a `models` or `custom_model` extraction.
## Pagination
The connector uses a single pagination strategy for all models:
### Offset-based
```
Call 1: search_read(domain, {limit: 80, offset: 0, order: "id asc"})
Call 2: search_read(domain, {limit: 80, offset: 80, order: "id asc"})
Call 3: search_read(domain, {limit: 80, offset: 160, order: "id asc"})
...until returned records < 80
```
- **Page size**: 80 (Odoo recommended)
- **Order**: Always `id asc` for consistent pagination
- **Stop condition**: Fewer records returned than the page size
- **max\_items**: When set, pagination stops as soon as enough records are collected, and the result is truncated to exactly `max_items`
## Rate Limits
Refer to your Odoo instance documentation for exact limits.
| Environment | Approximate Limit |
| ------------------ | --------------------------------------------------------------------------------- |
| Odoo Online (SaaS) | Varies by plan, refer to [Odoo documentation](https://www.odoo.com/documentation) |
| Odoo.sh | Varies, refer to your instance configuration |
| Self-hosted | Depends on server resources |
The connector handles `429 Too Many Requests` automatically: reads the `Retry-After` header and waits before retrying. Falls back to 10 seconds if the header is missing.
## Output Format
### Raw JSON (connector output)
The connector returns raw JSON from the Odoo API. Each record is a dict with all requested fields.
Example (`res.partner`):
```json
{
"id": 8,
"name": "Acme Corp",
"email": "contact@acme.com",
"phone": "+33 1 23 45 67 89",
"is_company": true,
"country_id": [75, "France"],
"category_id": [1, 3]
}
```
### Relational fields
Odoo relational fields are returned as:
- **Many2one**: `[id, display_name]` (e.g., `"country_id": [75, "France"]`)
- **One2many / Many2many**: list of IDs (e.g., `"category_id": [1, 3]`)
The platform flattens these automatically.
### Flattened output (lakehouse)
The platform flattens nested objects into columns with underscores:
| Raw JSON | Lakehouse column |
| ------------ | ----------------------------------- |
| `id` | `id` |
| `name` | `name` |
| `country_id` | `country_id` (flattened from array) |
## Domain Filter Reference
Odoo uses **Polish (prefix) notation** for domain filters. The connector accepts them as a JSON string.
### Syntax
Each criterion is `[field, operator, value]`. Multiple criteria are AND-ed by default.
### Operators
| Operator | Description |
| ----------------------- | ---------------------------------------- |
| `=`, `!=` | Equals / not equals |
| `>`, `>=`, `<`, `<=` | Comparison |
| `in`, `not in` | Set membership |
| `like`, `ilike` | Pattern match (ilike = case-insensitive) |
| `not like`, `not ilike` | Negated pattern match |
| `=like`, `=ilike` | SQL LIKE without auto-wrapping |
| `child_of`, `parent_of` | Hierarchical relationships |
### Logical operators
| Operator | Arity | Description |
| -------- | ------ | ---------------------- |
| `&` | Binary | AND (implicit default) |
| `\|` | Binary | OR |
| `!` | Unary | NOT |
### Examples
```json
// Active companies
[["is_company", "=", true], ["active", "=", true]]
// Sales orders over 1000
[["state", "=", "sale"], ["amount_total", ">", 1000]]
// Leads OR opportunities
["|", ["type", "=", "lead"], ["type", "=", "opportunity"]]
// Invoices from 2024
[["invoice_date", ">=", "2024-01-01"], ["invoice_date", "<=", "2024-12-31"]]
```
## Version Compatibility
Tested and supported on **Odoo 14 through 19** (including Odoo Online saas\~19.2).
| Odoo Version | Status | API Key Support |
| ------------ | -------------------------------------- | --------------- |
| 14–18 | Fully supported | Yes |
| 19 | Fully supported (tested on saas\~19.2) | Yes |
All 26 predefined models are stable across Odoo 14–19. If a model does not exist on your instance (depends on installed apps), use `model_fields` to check availability.
## Limitations
- **No field hardcoding.** The connector returns whatever the API provides. Field names and types depend on your Odoo version and installed modules.
- **Restricted fields.** Some models have fields restricted to specific user groups (e.g., `project.project.stage_id`). Use `fields_filter` to exclude them, or grant the required group to your API user.
- **Binary fields.** Fields like `image_1920` return base64-encoded data, which can be very large. Use `fields_filter` to exclude image fields when not needed.
- **Odoo Online API access.** Requires the Custom plan. Free and Standard plans block external API access.
- **No webhook/push extraction.** The connector uses pull-based extraction only (JSON-RPC search\_read).
- **One authentication method.** API key only. OAuth2 is not supported.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-salesforce.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract Salesforce objects with the Data Platform connector
## Objective
The Salesforce connector extracts data from your Salesforce org: standard and custom sObjects (Account, Contact, Opportunity, Lead, Case, custom `*__c` objects), SOQL queries, SOSL searches, sObject metadata, org API limits, and report listings.
**7 endpoint types** covering the Salesforce REST API surface most useful for analytics.
The connector returns **raw JSON** from the Salesforce API. The OVHcloud Data Platform automatically flattens nested objects into columns and stores the result in the lakehouse. No manual schema definition needed.
## 1. Create an External Client App for API access
The connector authenticates via **OAuth 2.0 Client Credentials Flow**, which requires an External Client App (or legacy Connected App) configured in your Salesforce org.
### Steps
1. In Salesforce, open **Setup** (gear icon, top right).
2. Search for **App Manager** in the Quick Find box.
3. Click **New External Client App** (top right). If you see a legacy org with only "New Connected App", that works too, choose Connected App and enable "Client Credentials Flow" under OAuth Settings.
4. Fill in:
- **App name**: any descriptive name (e.g. `DPE Data Platform`)
- **Contact email**: your email
5. In the OAuth section:
- ☑️ **Enable OAuth**
- **Callback URL**: `https://login.salesforce.com/services/oauth2/success` (required even though unused for Client Credentials)
- **OAuth scopes**: add at minimum `Manage user data via APIs (api)`
6. **Save** the app.
7. Open the newly created app → **Policies** tab → **Edit**.
8. Enable **Client Credentials Flow** and select a **Run As** user (typically your admin user on a Developer org).
9. **Save**.
### Retrieve credentials
1. Back on the app page, open the **Settings** tab → **OAuth Settings**.
2. Click **Consumer Key and Secret**. Salesforce will ask for email verification.
3. Copy the **Consumer Key** (your `client_id`) and **Consumer Secret** (your `client_secret`).
### Find your instance URL
- Top-right profile menu in Salesforce shows your My Domain URL.
- Or: **Setup → Quick Find → My Domain** → the value under "Current My Domain URL".
Format: `https://.my.salesforce.com` (e.g. `https://orgfarm-XXXXXX-dev-ed.develop.my.salesforce.com` on a Developer Edition).
## 2. Configure the connector on the platform
In the OVHcloud Data Platform UI, connect a new Salesforce source and fill in:
| Field | Value |
| ------------------- | ------------------------------ |
| **Instance URL** | Your My Domain URL from step 1 |
| **Consumer Key** | From your External Client App |
| **Consumer Secret** | From your External Client App |
Save: the connector will exchange these for an access token automatically on every run. Tokens aren't stored; the exchange happens each time data is extracted.
## 3. Available endpoints
Add one table per endpoint type you want to extract. You can add the same endpoint type multiple times (e.g. one `sobject_records` per sObject).
### Data extraction
| Endpoint | Description | Key parameters |
| -------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| **sobject\_records** | Extract records from any standard or custom sObject (Account, Contact, `MyObject__c`…) | `sobject_name` (required), `fields_filter`, `where_clause`, `max_items` |
| **soql\_query** | Run a freeform SOQL query for complex joins, aggregations, or sub-queries | `soql` (required), `max_items` |
| **sosl\_search** | Full-text search across multiple sObjects at once | `sosl` (required), `max_items` |
### Metadata
| Endpoint | Description | Key parameters |
| --------------------- | ------------------------------------------------------------ | ------------------------- |
| **sobjects\_list** | List every sObject available in your org (standard + custom) | `max_items` |
| **sobject\_describe** | Full schema (fields, types, relationships) for one sObject | `sobject_name` (required) |
| **reports\_list** | List reports configured in your org | `max_items` |
| **limits** | Org API request limits and current usage | |
## 4. Quick example: extract Accounts
1. **Endpoint type**: `sobject_records`
2. **sObject name**: `Account`
3. **Fields filter** (optional): `Id, Name, Industry, AnnualRevenue, CreatedDate`
4. **Max items**: `100` (or leave empty to extract everything)
The connector returns raw JSON from Salesforce. The platform flattens nested fields (Salesforce responses include an `attributes` object with `type` and `url`, which becomes `attributes.type` / `attributes.url` columns) and loads the result into your lakehouse.
## 5. Best practices
### Use `fields_filter` for large sObjects
When `fields_filter` is empty, the connector introspects the sObject and selects all queryable fields. Some standard sObjects have 50–100+ fields, which makes extraction slow and the resulting table very wide. For analytics, list the 5–20 fields you actually need.
### Use `where_clause` for incremental extraction
Add a filter like `LastModifiedDate > LAST_N_DAYS:1` to pull only recently changed records. Salesforce SOQL date literals like `LAST_N_DAYS:N`, `TODAY`, `YESTERDAY`, `THIS_WEEK` are particularly handy here.
You can type the filter with or without the leading `WHERE` keyword. Both work.
### Prefer `sobject_records` over `soql_query` when possible
`sobject_records` is simpler (fill fields + optional WHERE) and works identically under the hood. Reserve `soql_query` for:
- Sub-queries (e.g. `SELECT Id, (SELECT Id FROM Contacts) FROM Account`)
- Aggregates (`SELECT COUNT(Id) FROM Case GROUP BY Status`)
- Cross-object joins
### Discover available sObjects with `sobjects_list`
Run `sobjects_list` once: it returns the full catalog of sObjects in your org, including every custom object ending in `__c`. Use it to figure out the exact API name to put in `sobject_name`.
### Start with `max_items` set low
Use `max_items: 10` when setting up a new table, then raise or clear it once the output looks right.
## 6. Limitations
- **Read-only.** The connector does not create, update, or delete records. It only extracts.
- **REST API v60.0**: modern enough to cover all documented sObjects and queries. Newer Salesforce Spring '26 features are accessible via the general REST endpoints but may not be surfaced through specialized connector endpoints.
- **No Bulk API 2.0**: for single-query extractions above a few million rows, Bulk would be more efficient. Not supported in this version.
- **No streaming / Platform Events / Pub/Sub**: this is a batch extraction connector, not a real-time event listener.
- **Reports are listed, not executed.** The `reports_list` endpoint returns the catalog of reports, not their results. To pull report data, replicate the equivalent SOQL query via `soql_query`.
- **Org-level API quotas apply.** Salesforce enforces a daily API request limit per org (different per edition). Extractions count against it, see the `limits` endpoint for current usage.
- **Errors are not silently swallowed.** If your query has a SOQL syntax error, references a field the Run As user can't see, or hits a permission issue, the extraction fails with the Salesforce error message.
:::info
For technical details (pagination, rate limits, authentication flow, full endpoint reference, output format, limitations), see the [Salesforce Technical Reference](https://docs.dataplatform.ovh.net/connectors-sources-salesforce-technical-reference.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-salesforce-technical-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Salesforce: Technical Reference
## Objective
This is the technical companion to the main [Salesforce connector](https://docs.dataplatform.ovh.net/connectors-sources-salesforce.md) documentation. It covers authentication internals, endpoint behavior, pagination, rate limits, and the output format, everything needed to integrate the connector into a data pipeline.
## 1. Authentication: OAuth 2.0 Client Credentials Flow
### Flow
The connector exchanges three credentials for a short-lived access token at the start of every run:
```yaml
POST /services/oauth2/token
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
&client_id=
&client_secret=
```
Salesforce responds with an `access_token`, the canonical `instance_url`, and `token_type: Bearer`. The connector then uses `Authorization: Bearer ` on every subsequent request.
### Tokens are not refreshed
Client Credentials Flow does not issue refresh tokens. The connector requests a fresh token on each extraction run, which fits within Salesforce's default access-token TTL (approximately 2 hours, see official docs for the value configured on your org).
### Instance URL resolution
Whatever instance URL you type in the connector config, the token response returns the canonical URL for your org. The connector uses the response value for all REST calls, so you can provide either `https://mycompany.my.salesforce.com` or `https://mycompany.develop.my.salesforce.com` without issue.
### Required OAuth scope
At minimum: `Manage user data via APIs (api)`. Other scopes are not required for data extraction. Refer to the [OAuth Tokens and Scopes](https://help.salesforce.com/s/articleView?id=sf.remoteaccess_oauth_tokens_scopes.htm) official docs.
## 2. Architecture
- **Semi-structured pattern.** Every endpoint returns raw JSON (list of dicts). The platform flattens nested fields (dot-notation) and infers the schema automatically. You don't declare columns anywhere.
- **Hybrid per-endpoint design.** Data endpoints (`sobject_records`, `soql_query`, `sosl_search`) route through a shared SOQL pagination helper. Metadata endpoints (`sobjects_list`, `sobject_describe`, `reports_list`, `limits`) each call their dedicated REST resource.
- **Describe-then-query for default fields.** When `sobject_records` is used without `fields_filter`, the connector first fetches the sObject's describe to collect all queryable field names, then builds a SOQL `SELECT` listing them explicitly. Salesforce SOQL has no `SELECT *`: explicit fields are always required.
- **Input normalization.** SOQL/SOSL inputs are trimmed of leading/trailing whitespace and any trailing `;` (Salesforce doesn't use statement terminators). The `where_clause` field accepts input with or without a leading `WHERE` keyword. The connector strips it if present before prepending its own.
## 3. Endpoint reference
All data endpoints return a list of JSON objects. Each object from `/query` includes an `attributes` sub-dict (`{"type": "", "url": "/services/data/v60.0/sobjects//"}`) alongside the requested fields.
### sobject\_records
Extract records from a standard or custom sObject via SOQL.
| Parameter | Type | Required | Description |
| --------------- | -------- | -------- | ------------------------------------------------------------------------------------------ |
| `sobject_name` | text | yes | API name of the sObject (e.g. `Account`, `Contact`, `MyObject__c`) |
| `fields_filter` | tags | no | Explicit list of field API names. Empty → all queryable fields, discovered via describe |
| `where_clause` | textarea | no | SOQL WHERE expression, without the `WHERE` keyword (optional, leading `WHERE` is stripped) |
| `max_items` | number | no | Hard cap on returned records |
- **API call**: `GET /services/data/v60.0/query?q=SELECT FROM [WHERE ]`
- **Pagination**: SOQL `nextRecordsUrl` (cursor-based, see section 4)
- **Output**: Raw SOQL response records.
### soql\_query
Freeform SOQL for advanced use cases (sub-queries, aggregates, joins).
| Parameter | Type | Required | Description |
| ----------- | -------- | -------- | ------------------------------------ |
| `soql` | textarea | yes | SOQL statement. Multi-line supported |
| `max_items` | number | no | Hard cap on returned records |
- **API call**: `GET /services/data/v60.0/query?q=`
- **Pagination**: SOQL `nextRecordsUrl`
- **Output**: Raw SOQL response records. Structure depends on the query.
### sosl\_search
Salesforce Object Search Language: full-text search across multiple sObjects.
| Parameter | Type | Required | Description |
| ----------- | -------- | -------- | ------------------------------------------------------------------------------ |
| `sosl` | textarea | yes | SOSL statement (e.g. `FIND {Acme} IN NAME FIELDS RETURNING Account(Id, Name)`) |
| `max_items` | number | no | Hard cap on returned records |
- **API call**: `GET /services/data/v60.0/search?q=`
- **Pagination**: Single-page. SOSL returns a bounded result set, typically capped server-side at 2000 records.
- **Output**: Each record in `searchRecords` includes the `attributes.type` field identifying its sObject.
### sobjects\_list
The full catalog of sObjects available to the Run As user.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------------------- |
| `max_items` | number | no | Hard cap on returned records |
- **API call**: `GET /services/data/v60.0/sobjects`
- **Pagination**: Single-page.
- **Output**: One record per sObject with metadata (`name`, `label`, `custom`, `queryable`, `createable`, etc.).
### sobject\_describe
Full schema (fields, types, relationships, picklist values) for one sObject.
| Parameter | Type | Required | Description |
| -------------- | ---- | -------- | ----------------------- |
| `sobject_name` | text | yes | API name of the sObject |
- **API call**: `GET /services/data/v60.0/sobjects//describe`
- **Pagination**: Single record returned (one-row table).
- **Output**: The entire describe object from Salesforce, deeply nested, includes `fields`, `childRelationships`, etc. The platform flattens it to dot-notation columns.
### reports\_list
List of reports stored in your org.
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ---------------------------- |
| `max_items` | number | no | Hard cap on returned records |
- **API call**: `GET /services/data/v60.0/analytics/reports`
- **Pagination**: Single-page. Returns at most the first few hundred reports.
- **Output**: One record per report (`Id`, `Name`, `DeveloperName`, `FolderName`, etc.). Reports themselves are **not executed**.
### limits
Org-level API usage and quotas.
- **API call**: `GET /services/data/v60.0/limits`
- **Pagination**: Single-page (the payload is one JSON object describing dozens of limits).
- **Output**: The connector reshapes the response into a list of records, one per named limit, with the original keys (`Max`, `Remaining`) preserved.
## 4. Pagination
SOQL-based endpoints (`sobject_records`, `soql_query`) use Salesforce's cursor-based pagination:
- First response includes `totalSize`, `done: false`, `records`, and `nextRecordsUrl` (a path like `/services/data/v60.0/query/01g...-2000`).
- The connector follows `nextRecordsUrl` until `done: true` or `max_items` is reached.
- Default page size is 2000 records (server-side default).
Non-SOQL endpoints (`sosl_search`, `sobjects_list`, `sobject_describe`, `reports_list`, `limits`) do not paginate in this version.
## 5. Rate limits & error handling
### Salesforce limits
- **Daily API Requests**: org-wide soft quota, varies by edition (e.g. 100,000/day on Enterprise base, higher on Unlimited, generous on Developer Edition). See the `limits` endpoint for the exact current value on your org.
- **Concurrent calls**: 5 on Developer Edition, 25 on Enterprise+. Each request times out at 10 minutes server-side.
See [API Request Limits and Allocations](https://developer.salesforce.com/docs/atlas.en-us.salesforce_app_limits_cheatsheet.meta/salesforce_app_limits_cheatsheet/salesforce_app_limits_platform_api.htm) for current numbers.
### What the connector does
- **HTTP 429**: rare on Salesforce REST but handled: the connector honors the `Retry-After` header and retries the same request.
- **HTTP 4xx other**: surfaced immediately as an extraction failure with the Salesforce error message in logs. The connector does not catch permission errors; if the Run As user can't see a field or sObject, the extraction fails and the error is visible.
- **HTTP 5xx**: same as 4xx, surfaced immediately. Transient server errors will need a manual re-run.
### Common Salesforce error codes
| Error code | Meaning |
| ------------------------ | ------------------------------------------------------------------ |
| `INVALID_SESSION_ID` | Access token expired, re-run extracts a fresh token |
| `MALFORMED_QUERY` | SOQL/SOSL syntax error, check the query text |
| `INVALID_TYPE` | `sobject_name` doesn't exist or isn't accessible |
| `INVALID_FIELD` | A field listed in `fields_filter` isn't visible to the Run As user |
| `REQUEST_LIMIT_EXCEEDED` | Daily API quota hit, wait or increase the org allocation |
| `INSUFFICIENT_ACCESS` | Run As user lacks object/field-level permission |
## 6. Output format
Every endpoint returns raw JSON. The platform automatically:
- Flattens nested dicts to dot-notation columns (e.g. `attributes.type`, `attributes.url`).
- Infers the schema from the first batch and extends it if later batches introduce new fields.
- Stores the result in the lakehouse under your dataset as a queryable table.
For `sobject_records` and `soql_query`, each record contains the fields you selected plus the `attributes` sub-object. For metadata endpoints (`sobject_describe`, `sobjects_list`, `limits`), the payload is the API response reshaped into one-or-more records.
Column names reflect Salesforce's API field casing (PascalCase for standards: `Id`, `Name`, `CreatedDate`. Custom fields preserve whatever casing you gave them, e.g. `MyField__c`).
## 7. Limitations
- **Salesforce REST API v60.0** is hardcoded. Newer versions (Spring '26 is v66.0 at time of writing) add features not surfaced as dedicated endpoints; they remain accessible via `soql_query` for anything queryable.
- **No write operations** (no create/update/delete).
- **No Bulk API 2.0 support**: Salesforce REST queries, while paginated, are less efficient than Bulk for multi-million-row extractions.
- **No Streaming / Platform Events / Pub/Sub API**: batch-only connector.
- **Reports are listed, not executed.** Report execution via `/analytics/reports//executeAsync` is out of scope in this version.
- **Compound fields (`address`, `location`) excluded from default field discovery.** When `fields_filter` is empty, the connector skips these types because they require sub-field queries. Use an explicit `fields_filter` listing the sub-fields (e.g. `BillingStreet, BillingCity, BillingCountry`) to extract them.
- **SOQL `FIELDS(ALL)` not used.** Salesforce's `FIELDS(ALL)` syntax requires a `LIMIT 200` and has other restrictions; the connector prefers the describe-then-query approach for predictability at scale.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-sftp.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect files from an SFTP server
## Objective
The Secure File Transfer Protocol (SFTP) is an alternative to FTP that also allows you to transfer files between a client and server on a computer network, but further adds a layer of security to the process.
The Platform allows you to connect to an SFTP server, select your files on it, and import their data directly into your platform.
## Add a SFTP source on the Platform
### Configuration screen overview
Once you have found _SFTP_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below -
### Supported files and sizes
Before you proceed with learning how to configure an SFTP source, it would be interesting to note that -
- Even though our SFTP connector is able to list every file on your server, keep in mind that meta/data extraction only support the following types: .csv, .xls, .xlsx, .xml, .json, .parquet
- For the Metadata Extraction (see the Analyze part of our Connectors guide), the actual limits of size for a file downloaded by our SFTP connector are:
- Complete Extract: about 900 Mo
- Sample Extract: about 2.5 Go
### Configuring your source
When creating the source, you will be required to input the following information :
- Host: The URL/IP of the SFTP server you want to connect to (ex: 127.0.0.1)
- Port: The port listened to by your SFTP server
- Username: The login used to authenticate
- Password: The password associated
- Server public key: Your SFTP configuration key
- Path: The path to find specific files on the server (ex: Documents/data/export\_2018.csv)
Once you add the above details click on _Connect_ and then on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-shopify.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract Shopify store data with the GraphQL Admin API
## Objective
The Shopify connector extracts data from your Shopify store via the **GraphQL Admin API**: products, customers, orders, inventory, marketing, B2B, payments, content, and more.
**40 built-in endpoint types** + a **custom query** escape hatch for arbitrary GraphQL.
The connector returns **raw JSON** from the Shopify API. The platform automatically flattens nested fields into columns and stores the result in the lakehouse. No manual schema definition needed.
## 1. Get an Admin API Access Token
You install your own **custom app** on your store. The connector then uses that app's credentials to read your data, nothing else.
Two paths to get the credentials, both supported:
### Path A: Dev Dashboard (recommended for new stores)
1. Go to [partners.shopify.com](https://partners.shopify.com) and sign in (free, no card required).
2. Open the **Dev Dashboard**, click **Create app**, name it (e.g. `data-connector`).
3. In the new app, go to **Versions** → **Create a new version**.
4. Under **Access** → **Access scopes**, paste the [recommended scope list](#3-recommended-scopes) below.
5. Open **Request access** at the top of the **Access** section → enable **Customer data** (and Order data, etc.) → check **Data analytics** → save. _On a development store this is approved instantly._
6. **Publish** the version.
7. Install the app on your store from the **Distribution** section.
8. Open **Settings** of the app → copy **Client ID** and **Client Secret**.
9. In the connector UI, fill **Shop**, leave **Access Token** empty, and paste **Client ID** + **Client Secret**. The connector will exchange them for an offline token at first run.
### Path B: Existing token (shorter, if you already have one)
If you've already obtained an Admin API access token through the legacy "Develop apps" flow (token starts with `shpat_`) or by running the OAuth `client_credentials` exchange yourself (`shpua_`), paste it directly into the **Access Token** field, leave **Client ID** / **Client Secret** empty.
## 2. Configure Credentials
The connector accepts two interchangeable auth modes:
| Field | Mode A (direct token) | Mode B (OAuth exchange) |
| ----------------- | -------------------------------------- | ----------------------------- |
| **Shop** | `my-store` or `my-store.myshopify.com` | same |
| **Access Token** | `shpat_xxx…` or `shpua_xxx…` | empty |
| **Client ID** | empty | hex string from Dev Dashboard |
| **Client Secret** | empty | starts with `shpss_` |
If both are filled, the direct token takes priority.
## 3. Recommended Scopes
Paste this comma-separated list into the **Access scopes** field of your custom app version (Dev Dashboard → Access → Access scopes):
```
read_products, read_customers, read_orders, read_draft_orders,
read_inventory, read_locations, read_discounts,
read_shopify_payments_payouts, read_shopify_payments_disputes,
read_shopify_payments_accounts, read_marketing_events,
read_content, read_users, read_shipping, read_metaobjects,
read_metaobject_definitions, read_publications, read_locales,
read_markets, read_files, read_companies, read_assigned_fulfillment_orders,
read_audit_events
```
If a scope name is rejected, drop it. Shopify renames scopes between API versions, but the connector handles missing scopes gracefully (returns `[]` with a warning instead of crashing).
For **Customer / Order / Draft Order / Abandoned Checkout** data you also need to enable the **Protected Customer Data** approval (see Path A, step 5). Without it, those endpoints return rows but with PII fields blanked out.
## 4. Add a Shopify source on Data Platform
1. In the Data Platform **Connectors**, find **Shopify** in the source store and click **Select**.
2. Fill in the connection fields using either Mode A (direct token) or Mode B (Client ID + Client Secret) from [step 2](#2-configure-credentials).
3. Click **Connect**. With Mode B, the connector exchanges the Client ID/Secret for an offline access token at this step.
4. Click **Add an Endpoint**, choose an endpoint type from the dropdown (e.g. `products`, `orders`, `custom_query`), then fill in any required parameters (e.g. `segment_id`, `query_filter`, `max_items`).
5. Repeat step 4 for every additional endpoint you want to ingest in this source. Each endpoint becomes a separate table in the lakehouse.
6. Name the source and click **Create**.
:::warning
The **technical name** cannot be changed after the source is created. It is used when opening the source via the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## 5. Available Endpoint Types
### Core commerce (14)
| Endpoint | Description | Required parameters |
| ------------------------------ | -------------------------------- | ----------------------- |
| **products** | Catalog products | |
| **product\_variants** | All variants across all products | |
| **collections** | Manual + smart collections | |
| **customers** | Customer records | |
| **customer\_segment\_members** | Members of a specific segment | `segment_id` |
| **orders** | Orders (filterable) | optional `query_filter` |
| **draft\_orders** | Pending unconverted orders | |
| **abandoned\_checkouts** | Funnel data | |
| **fulfillment\_orders** | Logistics queue | |
| **tender\_transactions** | Payment-type breakdown | |
| **locations** | Stores / warehouses | |
| **inventory\_items** | SKU + cost master data | |
| **segments** | Customer segment definitions | |
| **companies** | B2B companies | |
### B2B & markets (4)
| Endpoint | Description | Required parameters |
| ---------------------- | -------------------------- | ------------------- |
| **company\_locations** | B2B shipping addresses | |
| **price\_lists** | B2B / wholesale pricing | |
| **catalogs** | B2B catalog assignments | |
| **markets** | Multi-region configuration | |
### Promotions & loyalty (4)
| Endpoint | Description | Required parameters |
| ------------------------- | ----------------------------------------- | ------------------- |
| **discount\_nodes** | All discounts (automatic + code + manual) | |
| **code\_discount\_nodes** | Code discounts only | |
| **gift\_cards** | Gift cards (Shopify Plus only) | |
| **selling\_plan\_groups** | Subscription plans | |
### Marketing & Shopify Payments (5)
| Endpoint | Description | Required parameters |
| ------------------------- | ------------------------------------ | ------------------- |
| **marketing\_events** | Campaign tracking | |
| **publications** | Sales-channel publications | |
| **payouts** | Shopify Payments payouts | |
| **disputes** | Shopify Payments disputes | |
| **balance\_transactions** | Shopify Payments fine-grained ledger | |
### Metadata & custom data (3)
| Endpoint | Description | Required parameters |
| --------------------------- | ---------------------------------- | -------------------------------------------- |
| **metaobjects** | Custom data instances | `metaobject_type` |
| **metaobject\_definitions** | Metaobject schemas | |
| **metafield\_definitions** | Metafield schemas (per owner type) | `owner_type` (PRODUCT, CUSTOMER, ORDER, ...) |
### Content (5)
| Endpoint | Description | Required parameters |
| ------------------ | ------------------------- | ------------------- |
| **articles** | Blog posts | |
| **blogs** | Blog containers | |
| **pages** | Online store static pages | |
| **url\_redirects** | URL redirect rules | |
| **files** | Uploaded media assets | |
### Admin & operations (4)
| Endpoint | Description | Required parameters |
| ------------------------- | ---------------------- | ------------------- |
| **events** | Shop audit log | |
| **staff\_members** | Store team | |
| **delivery\_profiles** | Shipping zones & rates | |
| **fulfillment\_services** | 3PL integrations | |
### Single object (1)
| Endpoint | Description | Required parameters |
| -------- | ----------------------------------------- | ------------------- |
| **shop** | Store-level settings (returns one record) | |
### Custom GraphQL (1)
| Endpoint | Description | Required parameters |
| ----------------- | ------------------------------- | ------------------------------------------------------------------------ |
| **custom\_query** | Run any GraphQL query you write | `query` (your GraphQL), optional `variables`, optional `connection_path` |
For `custom_query`, your GraphQL **must** declare `$first: Int!` and `$after: String` as variables and paginate a connection with `pageInfo { hasNextPage endCursor }`. The connector injects cursors automatically.
## 6. Pagination & `max_items`
Shopify's GraphQL API never returns all results in one shot. Every list is **paginated**. The connector handles pagination automatically; the only knob you control is the `max_items` field in the UI.
### The `max_items` field
Every endpoint (built-in and `custom_query`) exposes a **Max Items** field. It tells the connector "stop after collecting N records, even if more are available."
| What you want | Set Max Items to | What happens |
| ----------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| All records | empty | Connector loops until Shopify says `hasNextPage: false`. Could be 1 request (small store) or 1000+ requests (large store). |
| Just the first record | `1` | One request with page size 1. Stops immediately. |
| The first 5 records | `5` | One request with page size 5. Stops. |
| The first 100 records | `100` | One request with page size 100. Stops. |
| The first 250 records | `250` | One request with page size 250 (the Shopify max). Stops. |
| The first 1000 records | `1000` | Four requests of 250 each. |
| 50 records to test, then everything | `50` first, then re-run with empty | Common pattern while iterating. |
### How it works under the hood
The connector picks the page size dynamically:
```
page_size = min(100, max_items - already_collected) # if max_items is set
page_size = 100 # if max_items is empty
```
Each request asks Shopify for that many records. Once `max_items` is reached or `hasNextPage: false`, the loop stops.
So `max_items: 5` triggers exactly **one** HTTP request (`first=5, after=null`). No pagination loop, no second call. You don't need to write any pagination logic yourself.
### "Just give me the first page"
There's no explicit "first page only" toggle. But because Shopify caps page size at 250, **setting `max_items` to anything ≤ 250 guarantees a single HTTP request**. If you want "the natural Shopify page size" (50–100 by default), just set `max_items: 50` or `100`.
### Applies to every endpoint
This works the same way for:
- The 40 built-in endpoints (`products`, `orders`, `customers`, ...).
- The `custom_query` endpoint: the connector still injects `first` and `after` into your query and respects `max_items`.
You never write `first: 5` in your GraphQL query yourself. You write `first: $first` and let the connector inject the right value based on `max_items`.
## 7. Custom Query: Examples
The `custom_query` endpoint takes any GraphQL you write against the Admin API. Three rules apply:
1. Declare `$first: Int!` and `$after: String` in the query variables.
2. Paginate one connection with `pageInfo { hasNextPage endCursor }`.
3. Either set the **Connection Path** field (e.g. `products`) or let the connector auto-detect the connection.
Here are concrete templates for common cases. Paste any of them in the **GraphQL Query** field, fill the **Variables** field if needed, and run.
### Example 1: Bare-bones products (just a few fields)
Useful when you only need IDs + titles for downstream joins, instead of the 20-field default `products` endpoint.
```graphql
query ($first: Int!, $after: String) {
products(first: $first, after: $after) {
edges {
node {
id
title
handle
vendor
createdAt
}
}
pageInfo { hasNextPage endCursor }
}
}
```
- **Variables**: leave empty
- **Connection Path**: `products` (or leave empty for auto-detect)
### Example 2: Orders with a date filter
Replicates the built-in `orders` endpoint but lets you adjust both the filter and the field selection. Pass the filter via Variables.
```graphql
query ($first: Int!, $after: String, $query: String) {
orders(first: $first, after: $after, query: $query, sortKey: CREATED_AT) {
edges {
node {
id
name
createdAt
displayFinancialStatus
totalPriceSet { shopMoney { amount currencyCode } }
customer { id email }
}
}
pageInfo { hasNextPage endCursor }
}
}
```
- **Variables**: `{"query": "created_at:>=2026-04-01 AND financial_status:paid"}`
- **Connection Path**: `orders`
The full search syntax is documented at [shopify.dev/docs/api/usage/search-syntax](https://shopify.dev/docs/api/usage/search-syntax).
### Example 3: Products with their variants and metafields
Cases where the default `products` endpoint isn't enough. This pulls each product's variants inline and a few specific metafields.
```graphql
query ($first: Int!, $after: String) {
products(first: $first, after: $after) {
edges {
node {
id
title
variants(first: 50) {
edges {
node { id sku price inventoryQuantity }
}
}
metafields(first: 10, namespace: "custom") {
edges {
node { key value type }
}
}
}
}
pageInfo { hasNextPage endCursor }
}
}
```
- **Variables**: leave empty
- **Connection Path**: `products`
:::warning
Each nested `first:` adds to the query cost. On a Standard plan (100-pt budget per query), `products(first: $first)` with `variants(first: 50)` and `metafields(first: 10)` consumes roughly `first × (50 + 10 + 1)` points. Keep the parent page size small (e.g. `max_items: 20`) for this kind of nested query.
:::
### Example 4: Order line items (a sub-resource not exposed as a top-level endpoint)
Each order's line items aren't a built-in endpoint. The only way to get them is via a custom query.
```graphql
query ($first: Int!, $after: String) {
orders(first: $first, after: $after, query: "status:any", sortKey: CREATED_AT) {
edges {
node {
id
name
lineItems(first: 50) {
edges {
node {
id
title
quantity
originalUnitPriceSet { shopMoney { amount currencyCode } }
variant { id sku }
}
}
}
}
}
pageInfo { hasNextPage endCursor }
}
}
```
- **Variables**: leave empty
- **Connection Path**: `orders`
### Example 5: Nested connection (using Connection Path)
When the connection isn't at the root, e.g. the orders of a specific Shopify Payments balance transaction. Set **Connection Path** explicitly.
```graphql
query ($first: Int!, $after: String) {
shopifyPaymentsAccount {
payouts(first: $first, after: $after) {
edges {
node {
id
status
issuedAt
summary {
chargesGross { amount currencyCode }
refundsFee { amount currencyCode }
}
}
}
pageInfo { hasNextPage endCursor }
}
}
}
```
- **Variables**: leave empty
- **Connection Path**: `shopifyPaymentsAccount.payouts` _(required, auto-detect would still work here, but explicit is clearer when there are multiple connections at different depths)_
### Tips
- **Test in the GraphiQL Explorer first.** Shopify ships an in-store GraphiQL at `https://{shop}.myshopify.com/admin/api/explorer`. Build and validate your query there before pasting it in the connector. The auto-completion and schema docs make iteration much faster.
- **Field discovery.** The full schema is documented at [shopify.dev/docs/api/admin-graphql/latest](https://shopify.dev/docs/api/admin-graphql/latest). Every type page lists the available fields and their cost.
- **Two places, two roles.** Your query string MUST declare `$first: Int!` and `$after: String` in its variable list. That's the GraphQL **declaration** so Shopify knows to expect them. The connector then injects their **values** at runtime, page after page. The **Variables** UI field is for the _values_ of additional variables your query declares (e.g. `$query`, `$ownerType`, `$segmentId`), never put `first` or `after` there, the connector overwrites those.
- **`max_items` applies just like for built-in endpoints**: see [section 6](#6-pagination--max_items).
## 8. Quick Example: Extract Products
1. **Endpoint type**: `products`
2. **Max Items**: `1000` (or empty to extract everything)
3. Run the table extraction.
The connector returns raw JSON. The platform automatically flattens nested fields (e.g. `featuredImage.url` becomes a column `featuredimage_url`).
For a focused query: e.g. only orders from the last 30 days:
1. **Endpoint type**: `orders`
2. **Query filter**: `created_at:>=2026-04-01`
3. **Max Items**: empty
The `query_filter` field accepts the [Shopify search syntax](https://shopify.dev/docs/api/usage/search-syntax).
## 9. Best Practices
### Normalize the shop name once
The connector accepts `my-store`, `my-store.myshopify.com`, or `https://my-store.myshopify.com/`. All three resolve to the same host. Pick whichever is most readable.
### Use `custom_query` for narrow extractions
The default queries return \~10–20 fields per record. If you only need a handful of fields and have millions of records, write a `custom_query` with just those fields, query cost is roughly proportional to the field count.
### Refresh tokens via Client ID + Secret
If you set up Mode B (Client ID + Client Secret), the connector re-runs the OAuth exchange at every job start. This means a leaked / rotated token can be replaced by re-issuing the secret in the Dev Dashboard, without touching the connector configuration.
### Default `query_filter` for orders
The orders endpoint uses `status:any` by default to include all orders (open, closed, cancelled). If you want only open orders, set `query_filter: status:open`.
## 10. Plan & Scope Gating
Some endpoints are gated by plan or scope:
| Endpoint | Gating |
| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `gift_cards` | Shopify Plus only |
| `payouts`, `disputes`, `balance_transactions` | Shopify Payments must be enabled on the store |
| `companies`, `company_locations`, `catalogs`, `price_lists` | B2B must be enabled |
| `customers`, `orders`, `draft_orders`, `abandoned_checkouts`, `fulfillment_orders`, `tender_transactions` | Protected Customer Data approval (see Path A step 5) |
When a scope or plan is missing, the connector logs a warning and returns an empty list for that endpoint, your other tables keep working.
:::info
For detailed technical information (auth internals, full endpoint reference, pagination, rate limits, output format, limitations), see the [Shopify Technical Reference](https://docs.dataplatform.ovh.net/connectors-sources-shopify-technical-reference.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-shopify-technical-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Shopify: Technical Reference
## Objective
This is the technical companion to the main [Shopify connector](https://docs.dataplatform.ovh.net/connectors-sources-shopify.md) documentation. It covers authentication internals, the full endpoint reference, pagination, rate limits, output format, and limitations, everything needed to integrate the connector into a data pipeline.
## Authentication
### Supported Modes
The connector supports two interchangeable authentication modes that both end up sending an `X-Shopify-Access-Token` header on every API request.
| Mode | Fields filled in the UI | What the connector does |
| ----------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| **Direct token** | `shop` + `access_token` | Sends the token as-is. |
| **OAuth client\_credentials** | `shop` + `client_id` + `client_secret` | Exchanges the credentials at the Shopify OAuth endpoint, then uses the resulting offline access token. |
If `access_token` is filled, it takes priority. The OAuth exchange is skipped.
### Token Formats
| Prefix | Source | Notes |
| --------- | ----------------------------------------------- | ------------------------------------------------------------- |
| `shpat_…` | Legacy "Develop apps" flow in the Shopify admin | Token revealed once in the UI |
| `shpua_…` | Output of the `client_credentials` OAuth grant | Offline access token, non-expiring while the app is installed |
Both prefixes are valid in the **Access Token** field and are used identically by the connector.
### Deprecated authentication methods
| Method | Why |
| --------------------------------------------- | ------------------------------------------------------------------------------ |
| API key (`?key=…` query string) | Removed by Shopify |
| Storefront access token | Different audience (customer browsing); not accepted by the Admin API |
| OAuth `authorization_code` flow with redirect | Designed for distributed public apps, not for a server-to-server data pipeline |
### Custom App Approval
For Customer / Order / DraftOrder / AbandonedCheckout / FulfillmentOrder data, Shopify requires **Protected Customer Data** approval at the app level. On a development store this is granted instantly when the merchant ticks the relevant boxes in the Dev Dashboard. On production stores distributed via the App Store, Shopify reviews the request manually.
When the approval is missing, the API returns `ACCESS_DENIED` errors on the affected fields. The connector treats these as expected and returns whatever data was successfully fetched (with the restricted fields blanked out).
## Architecture
The connector returns raw JSON from the Shopify GraphQL API. The platform takes over from there. It auto-discovers the schema from the JSON payload, flattens nested objects into dot-notation columns (slugified to lowercase with underscores), and stores the result in the lakehouse, queryable via Trino. Any new field Shopify adds to a node appears automatically on the next extraction.
The connector itself is responsible for:
| Responsibility | Behavior |
| ------------------------------ | ------------------------------------------------------------------------------------------------ |
| **Auth** | Builds the `X-Shopify-Access-Token` header from either the direct token or an OAuth exchange |
| **Endpoint routing** | Maps the selected endpoint type to a built-in GraphQL query |
| **Pagination** | Loops `first: N, after: cursor` while `pageInfo.hasNextPage` is true |
| **Cost-based back-off** | Reads `extensions.cost.throttleStatus`, sleeps if the next request would exceed available budget |
| **Throttling** | Catches `THROTTLED` GraphQL errors and HTTP 429, retries with `Retry-After` |
| **Field-level access denials** | Returns the partial data with restricted fields removed (instead of failing) |
### API Version
All requests target the `2026-04` version of the Shopify Admin GraphQL API. Update [Shopify's release notes](https://shopify.dev/changelog) before pinning to a newer version, schema changes can rename fields or change types.
## Endpoint Reference
Every paginated endpoint shares the same call shape:
```http
POST https://{shop}.myshopify.com/admin/api/2026-04/graphql.json
{ "query": "...", "variables": { "first": 100, "after": null } }
```
The connector iterates pages until `pageInfo.hasNextPage == false` or the user-specified `max_items` cap is reached.
For each endpoint below, "Output" is **Raw JSON**: the full GraphQL `node` object, with nested fields preserved. The platform flattens it downstream.
### products
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `products(first, after, sortKey: CREATED_AT)`
**Default fields:** `id`, `handle`, `title`, `description`, `vendor`, `productType`, `status`, `tags`, timestamps, `totalInventory`, `tracksInventory`, `featuredImage`, `options`, `seo`, `onlineStoreUrl`, `priceRangeV2`.
### product\_variants
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `productVariants(first, after)`: top-level since API 2022-07; no parent product ID required.
**Default fields:** `id`, `sku`, `title`, `position`, `price`, `compareAtPrice`, `barcode`, `taxable`, `inventoryQuantity`, `availableForSale`, timestamps, `selectedOptions`, parent `product` ref, `image`.
### collections
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `collections(first, after)`: unified view of manual + smart collections.
**Default fields:** `id`, `handle`, `title`, `description`, `updatedAt`, `sortOrder`, `productsCount`, `seo`, `image`.
### customers
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `customers(first, after)`
**Default fields:** `id`, `firstName`, `lastName`, `email`, `phone`, `state`, `note`, `tags`, timestamps, `verifiedEmail`, `numberOfOrders`, `amountSpent`, `defaultAddress`, `lifetimeDuration`.
PII fields (firstName, lastName, email, phone, defaultAddress.zip) require Protected Customer Data approval. Without it the rows still come through with those fields blanked out.
### customer\_segment\_members
| Parameter | Type | Required |
| ------------ | ------ | -------------------------------------------------------- |
| `segment_id` | text | **Yes**: Shopify GID, e.g. `gid://shopify/Segment/12345` |
| `max_items` | number | No |
**Root field:** `customerSegmentMembers(first, after, segmentId: $segment_id)`
Use the `segments` endpoint first to discover segment IDs.
### orders
| Parameter | Type | Required |
| -------------- | ------ | ------------------------- |
| `query_filter` | text | No (default `status:any`) |
| `max_items` | number | No |
**Root field:** `orders(first, after, query: $query_filter, sortKey: CREATED_AT)`
`query_filter` accepts [Shopify search syntax](https://shopify.dev/docs/api/usage/search-syntax) (e.g. `created_at:>=2026-01-01`, `financial_status:paid`).
**Default fields:** `id`, `name`, `legacyResourceId`, lifecycle timestamps, `displayFinancialStatus`, `displayFulfillmentStatus`, money sets (totalPrice, subtotalPrice, totalTax, totalDiscounts, totalRefunded, totalShippingPrice), `customer`, addresses, `channelInformation`.
### draft\_orders
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `draftOrders(first, after)`
**Default fields:** `id`, `name`, `status`, `email`, `note2`, timestamps, money sets, `customer`, `shippingAddress`.
### abandoned\_checkouts
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `abandonedCheckouts(first, after)`
**Default fields:** `id`, `name`, `abandonedCheckoutUrl`, timestamps, money sets, `customer`.
### fulfillment\_orders
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `fulfillmentOrders(first, after)`
**Default fields:** `id`, `status`, `requestStatus`, timestamps, `destination` (address fields), `assignedLocation`, parent `order`.
### tender\_transactions
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `tenderTransactions(first, after)`
**Default fields:** `id`, `paymentMethod`, `processedAt`, `remoteReference`, `test`, `amount`, parent `order`, `user`.
### locations
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `locations(first, after)`
**Default fields:** `id`, `name`, `isActive`, timestamps, `fulfillsOnlineOrders`, `shipsInventory`, `legacyResourceId`, `address`.
### inventory\_items
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `inventoryItems(first, after)`
**Default fields:** `id`, `sku`, `tracked`, `requiresShipping`, timestamps, `countryCodeOfOrigin`, `harmonizedSystemCode`, `unitCost`, parent `variant` + `product`.
### segments
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `segments(first, after)`
**Default fields:** `id`, `name`, `query` (segment definition expression), `creationDate`, `lastEditDate`.
### companies
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `companies(first, after)`: B2B only; returns empty if B2B is not enabled on the store.
**Default fields:** `id`, `name`, `externalId`, `note`, timestamps, `locationsCount`, `ordersCount`, `totalSpent`.
### company\_locations
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `companyLocations(first, after)`: B2B only.
**Default fields:** `id`, `name`, `externalId`, `note`, timestamps, addresses, parent `company`.
### price\_lists
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `priceLists(first, after)`: B2B / wholesale.
**Default fields:** `id`, `name`, `currency`, `parent.adjustment`, linked `catalog`.
### catalogs
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `catalogs(first, after)`: B2B catalog assignments.
**Default fields:** `id`, `title`, `status`, linked `priceList`.
### markets
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `markets(first, after)`: multi-region configuration.
**Default fields:** `id`, `name`, `handle`, `enabled`, `primary`, `webPresence.rootUrls`, `currencySettings.baseCurrency`.
### discount\_nodes
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `discountNodes(first, after)`: union of automatic + code + manual discounts.
**Default fields:** `id`, plus a typed `discount` payload using inline fragments for `DiscountAutomaticBasic`, `DiscountAutomaticBxgy`, `DiscountCodeBasic`, `DiscountCodeBxgy`, `DiscountCodeFreeShipping`, `DiscountAutomaticFreeShipping`. Each variant exposes `title`, `status`, `startsAt`, `endsAt`, usage counts.
### code\_discount\_nodes
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `codeDiscountNodes(first, after)`: code discounts only.
Same field shape as `discount_nodes`, restricted to the `DiscountCode*` types.
### gift\_cards
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `giftCards(first, after)`: Shopify Plus only.
**Default fields:** `id`, `enabled`, `expiresOn`, timestamps, `lastCharacters`, `note`, `balance`, `initialValue`, owning `customer`.
### selling\_plan\_groups
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `sellingPlanGroups(first, after)`: subscription plans.
**Default fields:** `id`, `name`, `description`, `createdAt`, `merchantCode`, `appId`, `summary`, `productsCount`, `options`.
### marketing\_events
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `marketingEvents(first, after)`: campaign tracking.
**Default fields:** `id`, `type`, `remoteId`, `startedAt`, `endedAt`, `manageUrl`, `previewUrl`, UTM fields, owning `app`.
### publications
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `publications(first, after)`: sales-channel publications.
**Default fields:** `id`, `name`, `supportsFuturePublishing`, owning `app`.
### payouts / disputes / balance\_transactions
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root path:** `shopifyPaymentsAccount.{payouts | disputes | balanceTransactions}(first, after)`. Shopify Payments must be enabled on the store. The connector returns an empty list (with a warning) when `shopifyPaymentsAccount` is null.
**Default fields (payouts):** `id`, `status`, `issuedAt`, `net`, summary breakdown.
**Default fields (disputes):** `id`, `status`, `initiatedAt`, `amount`, `reasonDetails`.
**Default fields (balance\_transactions):** `id`, `type`, `transactionDate`, `test`, `amount`, `fee`, `net`.
### metaobjects
| Parameter | Type | Required |
| ----------------- | ------ | ----------------------------------------------------------- |
| `metaobject_type` | text | **Yes**: the metaobject definition's `type` (e.g. `recipe`) |
| `max_items` | number | No |
**Root field:** `metaobjects(first, after, type: $metaobject_type)`
Use the `metaobject_definitions` endpoint first to list the available types.
**Default fields:** `id`, `handle`, `type`, `displayName`, `updatedAt`, `capabilities.publishable.status`, `fields[].{key, value, type, jsonValue}`.
### metaobject\_definitions
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `metaobjectDefinitions(first, after)`
**Default fields:** `id`, `type`, `name`, `displayNameKey`, `description`, nested `fieldDefinitions`.
### metafield\_definitions
| Parameter | Type | Required |
| ------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `owner_type` | select | **Yes**: one of PRODUCT, PRODUCTVARIANT, CUSTOMER, ORDER, COLLECTION, ARTICLE, BLOG, PAGE, SHOP, LOCATION, COMPANY, COMPANY\_LOCATION, MARKET |
| `max_items` | number | No |
**Root field:** `metafieldDefinitions(first, after, ownerType: $owner_type)`
**Default fields:** `id`, `namespace`, `key`, `name`, `description`, `ownerType`, `type.name`, `pinnedPosition`.
### articles / blogs / pages / url\_redirects / files
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
Standard paginated content endpoints. Each returns the typed `node` object, see the [Shopify GraphQL reference](https://shopify.dev/docs/api/admin-graphql/latest) for the exact field shape per type. The connector requests a sensible default field subset per endpoint (handle, title, body/summary, timestamps, publication status, ..).
For `files`, the query uses inline fragments for `MediaImage`, `Video`, and `GenericFile`, so the output includes a media-type-specific payload alongside `id`, `alt`, `createdAt`, `fileStatus`.
### events
| Parameter | Type | Required |
| ----------- | ------ | -------- |
| `max_items` | number | No |
**Root field:** `events(first, after, sortKey: CREATED_AT, reverse: true)`: shop audit log.
**Default fields:** `id`, `message`, `createdAt`, `appTitle`, `attributeToApp`, `attributeToUser`, `criticalAlert`.
### staff\_members / delivery\_profiles / fulfillment\_services
Standard endpoints with a `max_items` parameter. `fulfillment_services` is unusual. The data is nested under `shop.fulfillmentServices` and is returned as a list (not paginated). The connector handles the special path automatically.
### shop
No parameters. Returns a one-element list with the store's metadata, id, name, domain, currency, timezone, plan, billing address, primary domain, feature flags, timestamps. Useful as a small reference table.
### custom\_query
| Parameter | Type | Required |
| ----------------- | -------- | -------------------------- |
| `query` | textarea | **Yes** |
| `variables` | json | No (default `{}`) |
| `connection_path` | text | No (default = auto-detect) |
| `max_items` | number | No |
Run any GraphQL query against the Admin API. Constraints:
1. The query **must** declare `$first: Int!` and `$after: String` as variables.
2. It **must** paginate one connection with `pageInfo { hasNextPage endCursor }`.
3. Either provide `connection_path` (dot-path to the connection from the data root, e.g. `products`, `shop.metafields`) or let the connector auto-detect the first `edges`/`pageInfo` connection in the response.
The connector substitutes `$first` (defaults to 250 or `max_items` when set) and `$after` (cursor from the previous page), then loops until `pageInfo.hasNextPage` is false.
`variables` lets you pass additional parameters used by your query (e.g. `{"query": "tag:vip"}` for a customer search filter). `$first` / `$after` are reserved.
Validation runs at extract time, invalid queries (missing variables) raise a clear error before the first HTTP call.
## Pagination
All paginated endpoints use **cursor-based pagination** (Relay-style connections). The connector loops:
```json
{
edges { node { ... } }
pageInfo { hasNextPage endCursor }
}
```
With `first: 100` (default page size, capped at 250) and `after: `. The loop ends when `hasNextPage == false` or the user-specified `max_items` is reached.
Single-object endpoints (`shop`) and the nested list `fulfillment_services` skip pagination. The connector wraps the response in a one-element list.
## Rate Limits
The Shopify Admin GraphQL API uses **cost-based** rate limiting (not requests-per-second). Every query has a point cost computed from its connection sizes; each store has a per-app bucket that refills at a fixed rate.
| Plan | Bucket | Restore rate |
| ----------------------------- | --------- | ------------ |
| Standard / Shopify / Advanced | 100 pts | 100 pts/s |
| Shopify Plus | 1,000 pts | 1,000 pts/s |
| Enterprise | 2,000 pts | 2,000 pts/s |
Refer to [Shopify's official rate-limit documentation](https://shopify.dev/docs/api/usage/rate-limits) for current details.
### Throttle Handling
The connector handles throttling in three ways:
1. **Pre-emptive sleep**: after every response, it inspects `extensions.cost.throttleStatus`. If the next page's cost would exceed the currently available budget, it sleeps for the time needed to refill.
2. **GraphQL `THROTTLED` errors**: Shopify sometimes returns HTTP 200 with a `THROTTLED` error in the body. The connector retries after a short back-off.
3. **HTTP 429**: rare, but handled with `Retry-After`. The connector retries up to 5 times before raising.
## Output Format
### Raw JSON
The connector returns the contents of `edges[].node` as a list of dicts, exactly as Shopify GraphQL returned them. Nested objects (addresses, money sets, child arrays) are preserved.
Example product node (truncated):
```json
{
"id": "gid://shopify/Product/123",
"handle": "snowboard",
"title": "All-mountain snowboard",
"vendor": "Acme",
"productType": "Snowboard",
"status": "ACTIVE",
"createdAt": "2024-01-15T10:30:00Z",
"totalInventory": 42,
"featuredImage": {
"url": "https://cdn.shopify.com/.../snowboard.jpg",
"altText": "Snowboard front view"
},
"priceRangeV2": {
"minVariantPrice": { "amount": "299.00", "currencyCode": "EUR" }
}
}
```
### Flattened in the lakehouse
The platform automatically flattens nested objects into dot-notation columns, then slugifies them:
| Raw JSON path | Lakehouse column |
| ------------------------------------- | ------------------------------------- |
| `id` | `id` |
| `featuredImage.url` | `featuredimage_url` |
| `priceRangeV2.minVariantPrice.amount` | `pricerangev2_minvariantprice_amount` |
List-of-object fields (e.g. `tags`, `options`) are exploded automatically, one row per list element.
## Limitations
- **GIDs are strings, not integers.** Shopify identifiers come as `gid://shopify/Resource/12345`: keep them as strings in your downstream queries.
- **No bulk operations.** Very large extractions (millions of records) page synchronously through the GraphQL API. For multi-million-row stores, write a `custom_query` against `bulkOperationRunQuery` instead, or extract incrementally with a date filter (`query_filter`).
- **Shopify Payments endpoints require enrollment.** `payouts`, `disputes`, `balance_transactions` return empty lists if Shopify Payments isn't activated on the store.
- **Protected Customer Data field-level approval.** Without it, customer / order rows return with PII fields (firstName, lastName, email, phone, zip) blanked out, but the rows themselves are returned. Approve the relevant data categories in the Dev Dashboard to get full PII.
- **Sub-resources require `custom_query`.** Per-order transactions / refunds / fulfillments and per-product metafields aren't dedicated endpoints. They are accessible via `custom_query` if you write a nested GraphQL.
- **B2B endpoints return empty without B2B.** `companies`, `company_locations`, `catalogs`, `price_lists` return `[]` on stores without B2B enabled. No error, just empty.
- **`gift_cards` is Shopify Plus only.** Lower plans receive `ACCESS_DENIED` and the connector returns an empty list.
- **Field set is fixed per endpoint.** Each built-in endpoint has a default field selection. To customize fields (more or fewer), use `custom_query`.
- **Shopify GraphQL changelog.** The connector is pinned to API version `2026-04`. Field renames or removals in newer versions are not picked up automatically, review [Shopify's release notes](https://shopify.dev/changelog) before bumping the version.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-sql-server.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use SQL Server database tables as Data Platform sources
## Objective
Microsoft SQL Server is a relational database management system. The Platform allows you to use the tables in a SQL Server database as sources for your application.
## Add a SQL Server source on the Platform
### Configure your connection
When creating the source, you will be required to input the following information :
- **Host**: the host name of the SQL Server database.
- **Port**: the port address of the SQL Server database.
- **Database**: the name of the SQL Server database.
- **Username**: the user name for the SQL Server database.
- **Password**: the password for the Username you are using.
- **SSL**: if activated, encrypts the information exchanged between the Data Platform and the database.
### Select the tables you want
After establishing a connection with your SQL Server database your tables will be fetched and displayed :
Select which tables you want to be included in the source by checking the boxes to the left of the table name, as indicated in the image above.
:::info
If the state of the database changes, you can use the _Fetch_ button to refresh the tables displayed and include them in your source.
:::
Once you follow the steps above, click on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-stripe.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract Stripe payment data with the Data Platform connector
## Objective
The Stripe connector extracts data from your Stripe account: customers, charges, payment intents, subscriptions, invoices, products, balance transactions, events, and more.
**76 endpoint resources** grouped into 12 domain groups + 4 filtered endpoints + 3 standalone endpoints + a **custom** endpoint for any Stripe resource path.
The connector returns raw JSON from the Stripe API. The platform automatically flattens nested objects into columns and stores the result in the lakehouse. No manual schema definition needed.
## 1. Get a Stripe API Key
### Create an Account
1. Sign up at [dashboard.stripe.com/register](https://dashboard.stripe.com/register), free, no credit card required
2. Every account starts with a **sandbox** (Test mode), full API access, no real charges
### Get the Secret Key
1. Log into your Stripe Dashboard
2. Make sure the **Test mode** toggle is ON (top right of the dashboard)
3. Go to **Developers** > **API Keys**
4. Copy the **Secret key**: it starts with `sk_test_` in test mode, `sk_live_` in live mode
:::info
**Important:** Use the **Secret key** (`sk_test_...` / `sk_live_...`), not the Publishable key (`pk_*`). The Publishable key is client-side only and cannot call most API endpoints.
:::
## 2. Configure Credentials
Paste your Secret key in the **Stripe API Key** field:
```
sk_test_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```
The connector uses Bearer token authentication. No scope configuration is needed. The secret key has full API access.
## 3. Available Endpoints
The connector is organized into 20 endpoints. Most group related resources with a "Resource Type" dropdown.
### Domain Groups (12)
| Endpoint | Resources inside | Notes |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- |
| **core** | customers, charges, payment\_intents, balance\_transactions, payouts, refunds, disputes, setup\_intents, events, files, file\_links, webhook\_endpoints, payment\_method\_configurations, payment\_method\_domains | Main payment resources |
| **products** | products, prices, coupons, promotion\_codes, tax\_codes, tax\_rates, shipping\_rates | Catalog and pricing |
| **billing** | subscriptions, subscription\_schedules, invoices, invoice\_items, invoice\_rendering\_templates, credit\_notes, plans, quotes, billing\_meters, billing\_alerts, billing\_credit\_grants | Recurring billing |
| **checkout** | checkout\_sessions, payment\_links | Hosted checkout flows |
| **connect** | accounts, application\_fees, transfers, top\_ups, country\_specs | Marketplace / platform |
| **radar** | early\_fraud\_warnings, reviews, value\_lists | Fraud detection (Stripe Radar) |
| **issuing** | authorizations, cardholders, cards, transactions, disputes, personalization\_designs, physical\_bundles | Requires Stripe Issuing |
| **treasury** | transactions, transaction\_entries, outbound/inbound transfers, outbound/received payments, credits/debits, reversals | Requires Stripe Treasury + a Financial Account ID |
| **terminal** | locations, readers, configurations | Requires Stripe Terminal |
| **identity** | verification\_sessions, verification\_reports | Requires Stripe Identity |
| **climate** | orders, suppliers, products | Requires Stripe Climate |
| **reporting** | report\_runs, report\_types, sigma\_scheduled\_query\_runs | Reports and Sigma |
### Filtered Endpoints (require a parent ID)
| Endpoint | Required parameter | Purpose |
| ---------------------------------------- | --------------------------- | -------------------------------------- |
| **payment\_methods** | Customer ID (cus\_xxx) | Payment methods attached to a customer |
| **setup\_attempts** | Setup Intent ID (seti\_xxx) | Setup attempts for a setup intent |
| **subscription\_items** | Subscription ID (sub\_xxx) | Line items within a subscription |
| **financial\_connections\_transactions** | Account ID (fca\_xxx) | Transactions on a linked account |
### Standalone Endpoints
| Endpoint | Purpose |
| ------------------------------------ | ---------------------------------------------- |
| **financial\_connections\_accounts** | Linked bank accounts (Financial Connections) |
| **tax\_registrations** | Tax registrations by jurisdiction |
| **treasury\_financial\_accounts** | Treasury financial accounts (no filter needed) |
### Custom Endpoint
| Endpoint | Required parameter | Purpose |
| ---------- | ------------------ | ------------------------------------------ |
| **custom** | Resource Path | Any Stripe list endpoint not covered above |
Example Resource Path values: `issuing/settlements`, `treasury/statements`, `capital/financing_offers`. The `/v1/` prefix is optional. The connector adds it if missing.
## 4. Add a Stripe source on Data Platform
1. In the Data Platform **Connectors**, find **Stripe** in the source store and click **Select**.
2. Paste the Secret key from [step 1](#1-get-a-stripe-api-key) into the **Stripe API Key** field.
3. Click **Connect** to verify the credentials. The connector calls `GET /v1/balance` as a health check.
4. Click **Add an Endpoint**, choose an endpoint from the dropdown (e.g. `core`, `billing`, `custom`), and where applicable pick a **Resource Type** (e.g. `customers`) and any required parameters (e.g. Customer ID for `payment_methods`, Resource Path for `custom`).
5. Repeat step 4 for every additional endpoint you want to ingest in this source. Each becomes a separate table in the lakehouse.
6. Name the source and click **Create**.
:::warning
The **technical name** cannot be changed after the source is created. It is used when opening the source via the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## 5. Quick Example: Extract Customers
1. **Endpoint**: `core`
2. **Resource Type**: `customers`
3. **Max Items**: `100` (or empty to extract all)
The connector returns raw JSON from Stripe. The platform automatically flattens nested objects (e.g. `address.country` becomes a column `address_country`) and stores the result in the lakehouse.
## 6. Best Practices
### Use `max_items` when testing
When setting up a new table, set `max_items` to `10` or `100` to verify the output quickly. Stripe has generous pagination (100 per page, unlimited total), so a large extraction may take time on accounts with lots of data.
### Extract `events` for a full audit trail
The `events` endpoint in the `core` group contains every state change in your account: customer created, subscription updated, charge refunded, etc. This is the best source for an incremental audit log.
### Extract pipelines as reference tables
The `products`, `prices`, `plans`, and `coupons` endpoints give you static reference data. Extract them separately and join with `charges`, `subscriptions`, or `invoices` for analytics.
### Use `balance_transactions` for financial reporting
`balance_transactions` contains every movement on your Stripe balance, charges, refunds, fees, payouts, adjustments, all in one unified stream. This is typically the best starting point for reconciliation.
### Secret key vs Restricted key
For read-only analytics, you can create a **Restricted API key** (prefix `rk_test_` / `rk_live_`) with only `Read` permissions on the resources you need. This is safer than a full Secret key. Go to **Developers > API Keys > Restricted keys > Create restricted key**.
## 7. Limitations
- **Static type prefixes**: Every Stripe object has a typed ID prefix (`cus_`, `ch_`, `sub_`, `pi_`, `in_`, etc.), keep these to join across tables.
- **Timestamps are Unix seconds**, not ISO 8601. Cast them in your downstream processing.
- **Some endpoints require a specific Stripe product** (Issuing, Treasury, Terminal, Identity, Climate). If the product isn't activated on your account, Stripe returns a 400 error and the extraction fails. The connector does not skip or retry this error.
- **Stripe product activation** is managed from your Stripe Dashboard, outside this connector.
- **Test mode vs Live mode**: the Secret key determines which data is extracted. A `sk_test_` key only sees sandbox data; a `sk_live_` key only sees production data. They are completely isolated.
- **Reports API quirk**: `/v1/reporting/report_types` rejects the `limit` parameter. The connector automatically retries without it. No action needed on your side.
:::info
For detailed technical information (auth internals, full endpoint reference, pagination, rate limits, output format, limitations), see the [Stripe Technical Reference](https://docs.dataplatform.ovh.net/connectors-sources-stripe-technical-reference.md).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-stripe-technical-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Stripe: Technical Reference
## Objective
This is the technical companion to the main [Stripe connector](https://docs.dataplatform.ovh.net/connectors-sources-stripe.md) documentation. It covers authentication internals, the full endpoint reference, pagination, rate limits, output format, and limitations, everything needed to integrate the connector into a data pipeline.
## Authentication
### Method
Bearer token with the Stripe Secret API Key.
```yaml
Authorization: Bearer sk_test_XXXXXXXXXXXXXXXX
```
The key is set once when the connector initializes; every subsequent API call reuses the same authenticated session.
### Key Formats
| Prefix | Type | Usage |
| ----------------------- | ----------------- | ------------------------------------------ |
| `sk_test_` | Secret key (test) | Sandbox, full API, fake data |
| `sk_live_` | Secret key (live) | Production, real transactions |
| `rk_test_` / `rk_live_` | Restricted key | Scoped permissions (read-only recommended) |
| `pk_test_` / `pk_live_` | Publishable key | Client-side only, **cannot be used here** |
### Health Check
The connector's health check calls `GET /v1/balance`: a lightweight endpoint available on every Stripe account. A successful 200 response confirms the Secret key is valid.
## Architecture
### Raw JSON, platform-handled schema
The connector returns raw JSON from the Stripe API, arrays of Stripe objects with their full nested structure. The platform takes over from there:
1. Schema is auto-discovered from the JSON payload
2. Nested objects are flattened into dot-notation columns
3. Data is stored in the lakehouse, queryable via SQL
You don't define a schema, list columns, or write any transformation code, any new field Stripe adds to an object appears automatically on the next extraction.
### Uniform API, uniform connector
Stripe's list endpoints are exceptionally uniform: every one returns the same envelope, `{"object":"list","data":[..],"has_more":bool}`, and uses the same cursor-based pagination. Because of this, the connector uses a single extraction path for all 76 supported resources. Adding a new resource means adding its API path to a registry; no new extraction logic is needed.
### Endpoint resolution
Endpoints are organized in the UI in three layers:
1. **Domain groups** (`core`, `billing`, `products`, ...): you select a group, then pick the specific resource from a dropdown (e.g. `billing` → `invoices`).
2. **Standalone endpoints** (`payment_methods`, `tax_registrations`, ...): the endpoint directly identifies the resource; no dropdown needed.
3. **Custom endpoint** (`custom`): you type any Stripe API path (e.g. `issuing/settlements`) and the connector calls it with standard cursor pagination.
Whichever path you choose, the extraction behavior is identical: authenticate, paginate, return raw JSON.
### What the connector handles vs what the platform handles
| Responsibility | Owner |
| ----------------------------------------------------- | --------- |
| Authentication (Bearer token) | Connector |
| Pagination (cursor, `has_more` loop) | Connector |
| Automatic retry on 429 | Connector |
| One-time fallback when an endpoint rejects `limit` | Connector |
| Raw JSON output | Connector |
| Schema discovery from JSON | Platform |
| Flattening nested objects into columns | Platform |
| Column naming (slugification, lowercase, underscores) | Platform |
| Storage in the lakehouse | Platform |
## Endpoint Reference
All list endpoints follow the same pattern: `GET /v1/{resource}?limit=100&starting_after={cursor}`. Responses share the same envelope: `{"object":"list","data":[...],"has_more":bool,"url":"/v1/..."}`.
The tables below document the 20 endpoints exposed in the UI and the Stripe resources they cover.
### core
Main payment resources. Single domain group with a "Resource Type" dropdown (14 options).
| Resource | API path | Description |
| ------------------------------- | ----------------------------------- | ---------------------------------------------- |
| customers | `/v1/customers` | Customer profiles |
| charges | `/v1/charges` | Charges (legacy, kept for backward compat) |
| payment\_intents | `/v1/payment_intents` | Modern payment flow (recommended over charges) |
| balance\_transactions | `/v1/balance_transactions` | All balance movements |
| payouts | `/v1/payouts` | Bank payouts |
| refunds | `/v1/refunds` | Refunds |
| disputes | `/v1/disputes` | Payment disputes |
| setup\_intents | `/v1/setup_intents` | Setup flow for saving payment methods |
| events | `/v1/events` | Webhook event log |
| files | `/v1/files` | Uploaded files |
| file\_links | `/v1/file_links` | Shareable file URLs |
| webhook\_endpoints | `/v1/webhook_endpoints` | Configured webhook receivers |
| payment\_method\_configurations | `/v1/payment_method_configurations` | Which payment methods to display |
| payment\_method\_domains | `/v1/payment_method_domains` | Domain verification for methods |
**UI options:** Resource Type (required, dropdown with the 14 resources above) + Max Items (optional).
**Pagination:** cursor-based.
**Output:** Raw JSON. Stripe list envelope.
### products
Catalog and pricing (7 options).
| Resource | API path |
| ---------------- | --------------------- |
| products | `/v1/products` |
| prices | `/v1/prices` |
| coupons | `/v1/coupons` |
| promotion\_codes | `/v1/promotion_codes` |
| tax\_codes | `/v1/tax_codes` |
| tax\_rates | `/v1/tax_rates` |
| shipping\_rates | `/v1/shipping_rates` |
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based. **Output:** Raw JSON.
### billing
Recurring billing (11 options).
| Resource | API path |
| ----------------------------- | --------------------------------- |
| subscriptions | `/v1/subscriptions` |
| subscription\_schedules | `/v1/subscription_schedules` |
| invoices | `/v1/invoices` |
| invoice\_items | `/v1/invoiceitems` |
| invoice\_rendering\_templates | `/v1/invoice_rendering_templates` |
| credit\_notes | `/v1/credit_notes` |
| plans | `/v1/plans` |
| quotes | `/v1/quotes` |
| billing\_meters | `/v1/billing/meters` |
| billing\_alerts | `/v1/billing/alerts` |
| billing\_credit\_grants | `/v1/billing/credit_grants` |
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based. **Output:** Raw JSON.
**Notable quirk:** per [Stripe's API docs](https://docs.stripe.com/api/subscriptions/list), the `subscriptions` list endpoint returns **all subscriptions that have not been canceled** by default (active, trialing, past\_due, incomplete, unpaid, paused). Canceled subscriptions are excluded unless you ask for them explicitly. To include canceled ones, use the custom endpoint with `resource_path=subscriptions?status=canceled` or `?status=all`.
### checkout
Hosted checkout flows (2 options: `checkout_sessions`, `payment_links`).
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based.
### connect
Marketplace / platform resources (5 options: `accounts`, `application_fees`, `transfers`, `top_ups`, `country_specs`).
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based.
### radar
Fraud detection: requires Stripe Radar (3 options: `early_fraud_warnings`, `reviews`, `value_lists`).
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based.
### issuing
Card issuing: requires **Stripe Issuing** activation (7 options: authorizations, cardholders, cards, transactions, disputes, personalization\_designs, physical\_bundles).
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based.
**Behavior if product is not enabled:** Stripe returns `400 Bad Request` with the message "Your account is not set up to use Issuing". The extraction fails, see [Limitations](#limitations).
### treasury
Money movement: requires **Stripe Treasury** activation (9 sub-resources).
| Resource | API path |
| ------------------------------ | ---------------------------------- |
| treasury\_transactions | `/v1/treasury/transactions` |
| treasury\_transaction\_entries | `/v1/treasury/transaction_entries` |
| treasury\_outbound\_transfers | `/v1/treasury/outbound_transfers` |
| treasury\_outbound\_payments | `/v1/treasury/outbound_payments` |
| treasury\_inbound\_transfers | `/v1/treasury/inbound_transfers` |
| treasury\_received\_credits | `/v1/treasury/received_credits` |
| treasury\_received\_debits | `/v1/treasury/received_debits` |
| treasury\_credit\_reversals | `/v1/treasury/credit_reversals` |
| treasury\_debit\_reversals | `/v1/treasury/debit_reversals` |
**UI options:** Resource Type (required) + Financial Account ID (required, format `fa_xxx`) + Max Items (optional). **Pagination:** cursor-based.
### terminal
Physical card readers (3 options: `terminal_locations`, `terminal_readers`, `terminal_configurations`).
**UI options:** Resource Type (required) + Max Items (optional). **Pagination:** cursor-based.
### identity
Identity verification: requires **Stripe Identity** (2 options: `identity_verification_sessions`, `identity_verification_reports`).
### climate
Carbon removal: requires **Stripe Climate** (3 options: `climate_orders`, `climate_suppliers`, `climate_products`).
### reporting
Reports and Sigma (3 options: `report_runs`, `report_types`, `sigma_scheduled_query_runs`).
**Notable quirk:** `/v1/reporting/report_types` rejects the `limit` parameter. The connector detects this, drops the parameter, and retries automatically. No action needed on your side.
### payment\_methods (filtered standalone)
**UI options:** Customer ID (required, format `cus_xxx`) + Max Items (optional).
Lists payment methods attached to a specific customer. Stripe requires the customer filter, if the Customer ID is missing, the connector returns an empty list instead of making the API call.
### setup\_attempts (filtered standalone)
**UI options:** Setup Intent ID (required, format `seti_xxx`) + Max Items (optional).
### subscription\_items (filtered standalone)
**UI options:** Subscription ID (required, format `sub_xxx`) + Max Items (optional).
### financial\_connections\_transactions (filtered standalone)
**UI options:** Account ID (required, format `fca_xxx`) + Max Items (optional).
### financial\_connections\_accounts (standalone)
**UI options:** Max Items only. Lists all Financial Connections accounts linked to the Stripe account.
### tax\_registrations (standalone)
**UI options:** Max Items only.
### treasury\_financial\_accounts (standalone)
**UI options:** Max Items only. Note: the individual transaction endpoints are in the `treasury` domain group. This standalone endpoint is only for the accounts list itself.
### custom
**UI options:** Resource Path (required) + Max Items (optional).
Free-text Stripe API path. The connector accepts the path with or without a leading `v1/` prefix, and calls the endpoint with standard cursor pagination.
Example inputs:
- `issuing/settlements` → `GET /v1/issuing/settlements`
- `v1/capital/financing_offers` → `GET /v1/capital/financing_offers` (no double prefix)
Use this endpoint for Stripe resources that aren't covered by a named endpoint yet, or for niche/beta endpoints where a named slot isn't warranted.
## Pagination
All list endpoints use Stripe's cursor-based pagination:
```http
GET /v1/{resource}?limit=100&starting_after={last_id}
```
Response:
```json
{
"object": "list",
"url": "/v1/...",
"has_more": true,
"data": [ ... ]
}
```
Behavior you can expect:
- The connector fetches 100 records per request (Stripe's maximum page size)
- It continues paginating as long as Stripe reports `has_more: true`
- It stops when `has_more` is `false`, or when Max Items is reached
### No-limit fallback
A small number of Stripe endpoints (notably `/v1/reporting/report_types`) don't accept the `limit` parameter. The connector detects the resulting 400 error, retries the request without `limit`, and continues the extraction transparently. You don't need to configure anything. The retry is automatic and only happens on the first request.
See [Stripe Pagination docs](https://docs.stripe.com/api/pagination) for the complete specification.
## Rate Limits
Stripe documents rate limits at [docs.stripe.com/rate-limits](https://docs.stripe.com/rate-limits). Exact numbers depend on the endpoint and account type, refer to the official documentation for current limits.
### Rate Limit Handling
On `429 Too Many Requests`, the connector waits and retries automatically:
1. It reads the `Retry-After` header sent by Stripe (in seconds)
2. If the header is missing, it falls back to a 2-second wait
3. It then retries the same request and continues the extraction
All other error codes (400, 401, 403, 404, 500, etc.) propagate as failures. The connector does not catch or skip them. The extraction stops immediately with the error returned by Stripe.
## Output Format
### Raw JSON (connector output)
The connector returns raw JSON from the Stripe API, arrays of Stripe objects with their full nested structure, matching Stripe's object schemas exactly.
Every Stripe object contains:
| Field | Description |
| ---------- | ---------------------------------------------------------------------- |
| `id` | Unique ID with type prefix (`cus_`, `ch_`, `sub_`, `pi_`, `in_`, etc.) |
| `object` | Object type (`customer`, `charge`, `subscription`, ...) |
| `created` | Unix timestamp (seconds) |
| `livemode` | Boolean, test vs live mode |
| `metadata` | User-defined key-value map |
Resource-specific fields vary per object. Refer to [Stripe API reference](https://docs.stripe.com/api) for each object's schema.
### Flattened output (lakehouse)
The platform flattens the raw JSON into a flat table. Nested keys become column names with underscores:
| Raw JSON path | Lakehouse column |
| --------------------- | --------------------- |
| `id` | `id` |
| `object` | `object` |
| `created` | `created` |
| `address.country` | `address_country` |
| `metadata.custom_key` | `metadata_custom_key` |
Column names are normalized: lowercase, dots and special characters replaced by underscores, always starting with a letter or underscore.
## Limitations
- **No deletion tracking**: the connector extracts the current state of each resource. Deleted objects are typically not returned by list endpoints. For an audit trail, extract `events` (which records deletions as `*.deleted` event types).
- **Unix timestamps**: Stripe uses Unix seconds (integer), not ISO 8601 strings. Cast to timestamp in downstream queries.
- **Stripe product gating**: Issuing, Treasury, Identity, Terminal, and Climate require the corresponding Stripe product to be activated on the account. If not activated, Stripe returns a 400 error with a message like "Your account is not set up to use X" and the extraction fails. The connector does not skip or retry.
- **Permission errors are not skipped**: if Stripe returns a 403 (permission denied), for example, because your Restricted API Key lacks read access to a resource, the extraction fails immediately. The connector doesn't have special handling to turn 403s into empty results; it only retries on 429 rate-limit responses.
- **`subscriptions` default filter**: per [Stripe's API docs](https://docs.stripe.com/api/subscriptions/list), the list endpoint returns all subscriptions that have not been canceled by default. To include canceled subscriptions, use the custom endpoint with `resource_path=subscriptions?status=canceled` or `?status=all`.
- **Custom endpoint pagination**: the custom endpoint assumes Stripe's standard list response format (`{"object":"list","data":[...],"has_more":bool}`). Endpoints with a non-standard response shape (e.g. singletons like `/v1/balance`) will return an empty list.
- **Test mode vs live mode isolation**: the Secret key determines which environment the connector reads from. `sk_test_` and `sk_live_` data are completely isolated. There is no cross-environment extraction.
- **No automatic expansion**: Stripe supports `expand[]=field` to inline related objects, but the connector does not set this. Related objects appear as ID references in the output; join them downstream via a separate extraction.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-trino.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Query external data sources with the Trino connector
## Objective
Trino is an open-source distributed SQL query engine for adhoc and batch ETL queries against multiple types of data sources. The Platform allows you to use Trino to access the sources for your application.
## Add a Trino source on the Platform
### Configure your connection
When creating the source, you will be required to input the following information :
- **Host**: the Host name of the Trino server.
- **Port**: the Port address of the Trino server _(8080 by default)_.
- **Catalog**: the name of the catalog from the Trino server.
- **Schema**: the name of the Schema from the Trino server.
- **Username**: the Username for the Trino server.
- **Password**: the Password for the Username you are using.
- **SSL**: if activated, encrypts the information exchanged between the Data Platform and the database.
### Select the tables you want
After establishing a connection with your Trino Server your tables will be fetched and displayed :
Select which tables you want to be included in the source by checking the boxes to the left of the table name, as indicated in the image above.
:::info
If the state of the database changes, you can use the _Fetch_ button to refresh the tables displayed and include them in your source.
:::
Once you follow the steps above, click on the _Save_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Connect Trino to an existing Platform Project
You can use **Trino** as a connector to an existing project on the Platform. This will be useful in the scenario where there is a need to extract data from one project to another. The tutorial can be found [here](https://docs.dataplatform.ovh.net/tutorials-trino-connector.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-twitter.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect X account insights with the Data Platform connector
## Objective
The X connector is used to **consolidate insights from a given X account**.
You can for instance get data from competitors' accounts.
## Available metrics
Using the X connector, you can get information on:
- A X account:
- Date
- Account name
- Total number of followers
- Total number of tweets (tweets, retweets, replies)
- Total number of favorites
- A tweet:
- Tweet ID
- Account name
- Creation date of the tweet
- Tweet text
- Total number of retweets, including retweets from the original tweet if it's a retweet itself
- Total number of retweets, excluding retweets from the original tweet if it's a retweet itself
- Number of retweets for the original tweet (if it's a retweet)
- Number of users that added this tweet in their favorites
- Type of tweet: quote of another tweet
- Type of tweet: reply to another tweet
- Type of tweet: retweet
- Language
## Get your access credentials
To access the X API you will need first to create an application on X. Then the tokens will be generated from it.
As the connector is using X's public API, you don’t need to create this app from a specific account with specific rights.
### Step 1
Go on [X](https://apps.X.com/) and click on « Create New App ».
### Step 2
Fill the form with your own information all the mandatory fields.
_Do not forget the Developer Agreement_.
### Step 3
Go on the « Keys and Access Tokens » tab.
### Step 4
Go on « Your Access Token » and click on « Create my access token ».
### Step 5
Open a Text Editor software and copy/paste your Consumer Key, Consumer Secret, Access Token and Access Token Secret.
## Add a X source on the Platform
### Configuration screen overview
Once you have found _X_ in the **Platform store**, click on _Select_ and you will be able to see the configuration screen as shown below:
### Configuring your source
#### Connect to X
When creating the source, you will be required to input the following information :
- **Account**: your X Page ID.
- **Consumer key**: key provided by X (you copied it earlier).
- **Consumer secret**: secret associated to the Consumer key.
- **Access token**: your X API access token.
- **Access token secret**: the secret associated to the Access token.
- **Source default schema**: you can use this field to add columns to your data, if necessary. Use the following syntax _\{"name":"value"}_, where _name_ is the name of your column and _value_ is the value to be filled in all rows.
Once you add the above details click on _Connect_ to establish a connection with X's API.
#### Add the Accounts and Endpoints
Before moving on, you have to add accounts and endpoints.
- [Accounts](#how-to-add-an-account) are X pages referenced by the Page ID - just like in the previous screen - which you want to get data from.
- [Endpoints](#how-to-add-an-endpoint) are the **different types of data** about accounts that the Platform will get using the X API. Each endpoint corresponds to a final table, aggregating data for all selected _accounts_.
:::info
The Platform will use the credentials given to access the X API but **you can overwrite the credentials** to have multiple accounts and endpoints in the same source. Simply add the different accounts to the Source after the [Connect to X](#connect-to-x) step.
:::
##### How to add an account?
To add an account, click on _Add an Account_. This will open the following window:
By default, it is using the same credentials you wrote previously when establishing the connection, which you can overwrite here. The _account name_ is the internal display name.
##### How to add an Endpoint?
On the Platform, each registered endpoint will correspond to a data source object, which you will be able to [load](https://docs.dataplatform.ovh.net/dpe-actions-load.md) into a distinct [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
To add an endpoint, click on _Add an Endpoint_. This will open the following window:
Next, select the type of Endpoint you wish to add. The team is always adding new types of Endpoints to the connectors. You can click on the scroll-down list under the _Type_ field to see the ones available.
Most common ones are:
- _tweet\_insights_: general data about the page.
- _account\_insights_: data about the page sorted by day.
To finish, click on the _Create_ button on the top right-hand corner.
:::warning
Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
For more information about the X API, follow [this link](https://developer.X.com/en/docs/api-reference-index).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-sources-fhir.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Collect healthcare data from a FHIR R4 server
## Objective
HL7 FHIR Release 4 (FHIR R4) is a standard for exchanging healthcare data. The FHIR connector connects the Platform to your FHIR server. It ingests selected data into relational tables that you can use in Lakehouse Manager and processing jobs.
:::info
The FHIR connector is an initial release: supported tables and behaviour will grow over time and may not yet match every integration or FHIR workflow you have in mind. Check the [release notes](https://docs.dataplatform.ovh.net/release-notes.md) and this page periodically for updates.
:::
:::info
Before you begin, collect your FHIR base URL (for example `https://example.com/fhir`). If your server requires authentication, prepare a bearer token or basic credentials.
:::
## Add a FHIR source on the Platform
### Configure the connection
1. Open the Platform store and select the FHIR connector.
2. Enter the endpoint URL of your FHIR API. This is the server base URL the connector calls.
3. Choose an authentication mode:
- **None** if the server allows unauthenticated access.
- **Bearer** if the server expects an `Authorization: Bearer` token. Paste the token when prompted.
- **Basic** if the server uses HTTP basic authentication. Enter the username and password.
4. Complete the connection step using **Connect**.
## Select tables to ingest
After the connection step, you add data by choosing tables in the **Add tables** dialog. Each row is a relational table name (for example `patient`, `encounter`) with a short description and a category tag.
### Filter by category
The category list on the left is fixed in the Platform. It does not come from your FHIR server.
You can filter tables using:
- All
- Administrative
- Encounters
- Clinical
- Medication
- Diagnostics
- Referentials
- Transverse
Categories help you browse long lists. They do not change what your FHIR server exposes.
### Search
The search field filters rows by table name. It does **not** run ad hoc queries against the FHIR server.
### Presets
**Presets** are shortcuts that pre-select a bundle of tables for a common scenario. Examples include `PATIENT PROFILE`, `CARE PATHWAY`, `CLINICAL OBSERVATIONS`, `MEDICATIONS`, and `FULL EXPORT`.
Presets are starting points only. They help you select tables faster. They are not driven by server configuration and do not limit what you can add manually.
### Batch
Each time you add tables, you assign a batch value. The batch is a suffix appended to table names so you can group related tables inside the Platform.
You can edit the batch name before you confirm. You can also add tables to an existing batch when you need the same grouping suffix.
### Population filters
For patient-centric tables, you can narrow who is included using population filters.
- **Simple** mode offers controls such as gender and date of birth through dropdowns.
- **Advanced** mode does not add extra controls; it is for typing the filter manually as text, for example `gender=female&birthdate=...` with parameters separated by `&`. Use the syntax your FHIR server expects.
### Linked tables
Some tables depend on others. When you select a table, the UI may auto-select related tables that are required for a consistent dataset.
For example, selecting **`patient_name`** can also select **`patient`** and **`human_name`**. The exact rules follow the product implementation.
:::info
Review the full selection (including any tables added automatically) before you click **Add**. Dependencies can widen the scope of the data you ingest.
:::
When your selection is ready, click **Add**. Use **Cancel** to close without adding tables.
### Name the source
Before you finish creating the source, enter a name in the field provided by the UI.
:::warning
The **technical name** cannot be changed after you create the source. It is used when you open the source with the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Use ingested data
After you ingest data, you can work with it like other Platform sources. Load it into [Lakehouse Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) using [Data Processing Engine load actions](https://docs.dataplatform.ovh.net/dpe-actions-load.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connectors Analyzer
## What is the Analyzer?
The Analyzer is the first quality gate on incoming data. It runs against a registered source before any of that data reaches a project table, so problems in structure, typing and formatting are caught at the door rather than after a load has written them into the Lakehouse.
It does two things.
**Metadata extraction** reads the shape of the source objects: the fields they expose, their types, and the volumes involved. Every downstream [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) builds on that metadata, so a source has to be analyzed before its objects can be loaded.
**Blueprint rules** are the cleaning layer on top. A blueprint is a named set of conditions and transformations applied to a source object as its data passes through, so date formats, casing, numeric precision and null handling follow one standard across every source feeding the project.
Extract the metadata first. Blueprint rules are written against fields the Analyzer has already found.
[### Extract the metadata→
Read the fields, types and volumes a source object exposes.
](/connectors-analyzer-extract-metadata)[### Blueprint rules→
Normalize incoming values with conditions and transformations.
](/connectors-analyzer-blueprint-rules)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract the metadata of a source
## Objective
To extract the metadata of a source for the first time, simply select a source object on the left-hand panel. This will parse a sample of 1,000 rows from the source to obtain the following information:
- attribute names
- attribute types
- attribute minimum and maximum values
- attribute card., i.e. the number of unique values in the sample
- attribute null values
To extract the metadata of a source again, you can click on the **Extract Metadata** button at the top right. Further, choose _Sample data_ or _Complete data_ to extract on the first 1,000 rows or on the entire table respectively.
:::info
A _complete_ metadata extraction can take a while depending on the size of the selected table
:::
:::warning
If your metadata extraction fails, investigate why by checking out the **logs in the Jobs tab** in the Connectors.
:::
Once the meta are extracted, you can start working to set standard structure formats for your data by adding blueprint rules to any attribute.
[Learn how to add blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Clean data automatically with blueprint rules
## Objective
The Platform provides default "blueprint rules" which consist of simple cleaning logic (like dates formatting, thousands of separators, etc.) to help users run simple data preparation. You can choose from available transformations in the Platform store and apply them if the applied condition is met.
It is not mandatory to use blueprint rules when using the platform. If used, the rules will be added automatically to the [Data Processing Engine Load actions](https://docs.dataplatform.ovh.net/dpe-actions-load.md) you will define to ingest the data inside your dataset.
## How a rule works
A rule applies to each line independently and allows several things.
If a condition is specified, the rule will apply only under certain conditions expressed in `condition` and `condition_value(s)`.
If a transformation is filled, the transformation action on each line will be performed, otherwise, it is considered to be a filtering rule (`keep`).
If no _condition_ is filled, the one set by default is `ALL`, which allows any line.
If no _transformation_ nor _condition_ is filled, then we consider it to be a _default transformation_. Otherwise, if no transformation is filled and there is one _condition_, the action by default is _keep_. The action applies only to lines that respond to the rule defined by _condition_.
### Fields
| Attribute name | Required | Example | Comment |
| ----------------- | ----------------------------------------------------------------------------- | ----------------------------------- | --------------------------------- |
| attribute\_name | Yes if the format is a list, no if the format is an object | `ca_ttc`, `date` | |
| condition | No (defaults to `ALL`) | `EQUAL`, `IN` | See the condition list below |
| condition\_value | Yes, except if condition is `IN`, `NOT IN` or `ALL` | `1`, `"Categorie1"` | |
| condition\_values | Yes if condition is `IN` or `NOT IN` | `["ca_ttc", 2, 3]`, `["categorie"]` | |
| action | No | `keep`, `concat` | See the transformation list below |
| action\_value | Yes if action is `default` or `split` | `1`, `"Categorie1"` | |
| action\_values | Yes, except if action is `default`, `split`, `action_error`, `keep` or `skip` | `["ca_ttc", 2, 3]`, `["categorie"]` | |
| else\_value | If the condition is not met, replaces the value by `else_value` | `"ca2"` | |
## Configure a blueprint rule
Rules require 3 steps to be configured:
1. **Information:** Name of the attribute
2. **Transformation:** If the condition is confirmed, which transformation action to apply to the data (can be chosen from the Platform store)
3. **Condition:** which rule to apply for the condition
To add a rule on an attribute, on the relevant field, click on the following button:
### Transformations
Different **transformations** are available in the Platform store:
- _Replace by_: Replaced values by a single value
- _Substring_: Values will be truncated
- _Date format_: Replace date formats
- _Keep line_: Keep matching values
- _Skip line_: Skip matching values
- _Throw error_: Matching values will trigger an error
- _Replace substring_: Replace matching substrings
- _Apply regexp_: A new regexp will be applied to matching values
- _Stop workflow_: Stop the ongoing workflow
### Conditions
Different **conditions** are available:
The Platform provides you a variety of Conditions you can apply to your chosen transformation. The list includes the following: _Equal_, _Not Equal_, _Between_, _Not Between_, _Lower than_, _Lower or Equal than_, _Greater than_, _Greater or Equal than_, _REGEXP_, _NOT REGEXP_, _Is a number_, _Is not a number_, _Is a date_, _Is not a date_, _Is null_, _Is not null_
:::info
Once the transformation and condition are set, the rule will be applied to the selected attribute.
:::
## Reference
The names below are the values written in the rule itself, so they are the ones you use when you edit rules as JSON, for example in the [Load advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md) or the [Aggregate advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md).
:::details Condition list
| Condition's name | Description |
| ---------------- | ---------------------------------------------------------------------------------------------------------------- |
| ALL | Keeps all the lines |
| EQUAL | Valid lines: those whose _attribute\_name_ is equal to _condition\_value_ |
| NOT EQUAL | Valid lines: those whose _attribute\_name_ is not equal to _condition\_value_ |
| IN | Valid lines: those whose _attribute\_name_ is in the list _condition\_values_ |
| NOT IN | Valid lines: those whose _attribute\_name_ is not in the list _condition\_values_ |
| LTE | Valid lines: those whose _attribute\_name_ is lower or equal to _condition\_value_ |
| LT | Valid lines: those whose _attribute\_name_ is lower than _condition\_value_ |
| GTE | Valid lines: those whose _attribute\_name_ is higher or equal to _condition\_value_ |
| GT | Valid lines: those whose _attribute\_name_ is higher than _condition\_value_ |
| REGEXP | Valid lines: those whose _attribute\_name_ corresponds to the regexp _condition\_value_. Example: `^([0-9]{4})` |
| NOT REGEXP | Valid lines: those whose _attribute\_name_ does not correspond to the regexp _condition\_value_ |
| BETWEEN | Valid lines: those whose _attribute\_name_ is between _condition\_values\[0]_ and _condition\_values\[1]_ |
| NOT BETWEEN | Valid lines: those whose _attribute\_name_ is not between _condition\_values\[0]_ and _condition\_values\[1]_ |
| OR | Valid lines: those where one of the values contained in _condition\_values_ is not empty (empty, null, False, 0) |
| AND | Valid lines: those where all values contained in _condition\_values_ are not empty (empty, null, False, 0) |
| IS NULL | Valid lines: those whose _attribute\_name_ is null |
| IS NOT NULL | Valid lines: those whose _attribute\_name_ is not null |
| IS INT | Valid lines: those whose _attribute\_name_ is an integer |
| IS NOT INT | Valid lines: those whose _attribute\_name_ is not an integer |
| IS DATE | Valid lines: those whose _attribute\_name_ is a date |
| IS NOT DATE | Valid lines: those whose _attribute\_name_ is not a date |
| IS DEC | Valid lines: those whose _attribute\_name_ is a decimal |
| IS NOT DEC | Valid lines: those whose _attribute\_name_ is not a decimal |
| IS NUM | Valid lines: those whose _attribute\_name_ is a number |
| IS NOT NUM | Valid lines: those whose _attribute\_name_ is not a number |
:::
:::details Transformation list
- _keep_: Keep the line
- _skip_: Skip the line
- _default_:
Replace the value by _action\_value_.
For dates, there is:
- `$today` returns yyyy-mm-dd
- `$now` returns yyyy-mm-dd hh:mm:ss
- `$yesterday` returns yyyy-mm-dd
- _regexp\_replace_:
Replace the value corresponding to regexp _action\_values\[0]_ by _action\_values\[1]_
- _replace_:
Replace the value corresponding to _action\_values\[0]_ by _action\_values\[1]_
- _substring / substr_:
- If _action\_values\[0]_ >= 0:
Starts at the \*action\_values\[0]\*th character
- If _action\_values\[0]_ \< 0:
Only takes into account the _action\_values\[0]_ last characters
- If _action\_values\[1]_ is filled:
The 2nd figure represents the max number of characters wanted
- _date\_replace_:
Replace the date format _action\_values\[0]_ by _action\_values\[1]_.
Examples for _action\_values_:
- `["%d/%m/%Y", "%Y-%m-%d"]`
- See the date format directives below
- _add_:
Add several fields or hard numbers, automatically detecting if for each value of _action\_values_ it is an attribute available in the line, otherwise it is considered to be a number. Examples for _action\_values_:
- `["ca_ht", "taxes"]`
- `["nb_clics", 1]`
- _sub_: Substract _action\_values\[0]_ from all the other data present in _action\_values_
- _mul_: Multiply _action\_values\[0]_ by _action\_values\[1]_
- _div_: Divide _action\_values\[0]_ by _action\_values\[1]_
- _mod_: Modulo _action\_values\[0]_ by _action\_values\[1]_
- _concat_: Concatenate one or several fields with hard strings. Each element of _action\_values_ detects if it is the name of a column available in the line, if not we consider it to be a string of hard characters. Examples for _action\_values_:
- `["site_name", " - ", "country"]`
- `["civilite", " name", "lastname"]`
- _ceil_: Rounded up _action\_value_
- _floor_: Rounded down _action\_value_
- _disaggregate / split_:
Used to create multiple lines if a field contains multiple values that should be split into separate lines.
There can only be one _disaggregate_ rule per DPE action, and this rule always applies last.
The field is split on the character _action\_value_, and for each split element only non-empty values are kept.
- _action\_error_: Leads to an error in the action, and consequently considers that all the lines are false
- _stop\_workflow_: Stop the workflow
:::
:::details Date format directives
| **Directive** | **Meaning** | **Example** |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| %w | Weekday as a decimal number, where 0 is Sunday and 6 is Saturday. | 0, 1, ..., 6 |
| %d | Day of the month as a zero-padded decimal number. | 01, 02, ..., 31 |
| %m | Month as a zero-padded decimal number. | 01, 02, ..., 12 |
| %y | Year without century as a zero-padded decimal number. | 00, 01, ..., 99 |
| %Y | Year with century as a decimal number. | 1970, 1988, 2001, 2013 |
| %H | Hour (24-hour clock) as a zero-padded decimal number. | 00, 01, ..., 23 |
| %I | Hour (12-hour clock) as a zero-padded decimal number. | 01, 02, ..., 12 |
| %p | Locale's equivalent of either AM or PM. | AM, PM (en\_US); am, pm (de\_DE) |
| %M | Minute as a zero-padded decimal number. | 00, 01, ..., 59 |
| %S | Second as a zero-padded decimal number. | 00, 01, ..., 59 |
| %f | Microsecond as a decimal number, zero-padded on the left. | 000000, 000001, ..., 999999 |
| %z | UTC offset in the form +HHMM or -HHMM (empty string if the object is naive). | (empty), +0000, -0400, +1030 |
| %Z | Time zone name (empty string if the object is naive). | (empty), UTC, EST, CST |
| %j | Day of the year as a zero-padded decimal number. | 001, 002, ..., 366 |
| %U | Week number of the year (Sunday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0. | 00, 01, ..., 53 |
| %W | Week number of the year (Monday as the first day of the week) as a decimal number. All days in a new year preceding the first Monday are considered to be in week 0. | 00, 01, ..., 53 |
| %% | A literal '%' character. | % |
:::
:::details Rule examples
Replace a date format:
```json
[
{
"attribute_name": "date",
"action": "date_replace",
"action_values": ["%d/%m/%Y", "%Y-%m-%d"]
}
]
```
Add 1 to the _year_ attribute, only on the lines where it is 2015 or 2016:
```json
[
{
"attribute_name": "year",
"condition": "IN",
"condition_values": ["2015", "2016"],
"action": "add",
"action_values": ["year", 1]
}
]
```
Set _year_ to 2015 on the lines where it is lower than 2015:
```json
[
{
"attribute_name": "year",
"condition": "LT",
"condition_value": "2015",
"action": "default",
"action_value": "2015"
}
]
```
Set the _date_ attribute to today's date:
```json
[
{
"action": "default",
"action_value": "$today",
"attribute_name": "date"
}
]
```
Convert a temperature from Kelvin to Celsius:
```json
[
{
"action": "sub",
"action_values": [
"temp_min",
273.15
],
"attribute_name": "temp_min"
}
]
```
Normalise a timestamp:
```json
[
{
"action": "date_replace",
"action_values": [
"%a %b %d %H:%M:%S +0000 %Y",
"%Y-%m-%d %H:%M:%S"
],
"attribute_name": "statuses_created_at"
}
]
```
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connector consumers
## What is a consumer?
A consumer is an external tool granted read access to project data. Where [sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) bring data in, consumers expose it back out: a BI report built on project tables, a lineage backend receiving OpenLineage events, or any SQL client connecting through Trino.
Creating one is a single step. Under **New consumer**, pick the tool from the store, and the platform registers a matching [IAM application](https://docs.dataplatform.ovh.net/iam-application-authentication.md) in the project with the OAuth redirect URLs and secrets already configured for it. [Trino consumers](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) are the exception: they are created from their own **Trino consumers** section rather than that button.
Power BI and OpenLineage support one consumer each per project. Trino consumers have no such limit, so create as many as you need, each with its own connection string.
Access is then governed entirely by the project's [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md). People connecting from the external tool authenticate against project IAM and see only the tables and queries their [roles and groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md) allow, which means every external user, including people who never open the platform itself, has to exist in the project's IAM. Granting the _Analytics Consumer_ role gives full read access to the project's data.
:::warning
Permissions are not scoped per consumer. A role that grants access to a table grants it through every consumer attached to the project.
:::
[### Power BI→
Connect a Power BI workspace to project tables and queries.
](/connectors-consumers-power-bi)[### OpenLineage→
Send lineage events to an OpenLineage-compatible backend.
](/connectors-consumers-openlineage)[### Trino→
Expose a Trino endpoint for any external SQL client.
](/connectors-consumers-trino)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-consumers-power-bi.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Plug Power BI to your Data Platform
## Objective
Are you or your teammates already proficient with [Microsoft Power BI](https://powerbi.microsoft.com/en-us/getting-started-with-power-bi/)? If you want to use it as your analytics solution on top your Data Platform, this guide is for you.
The configuration is done in two parts. First you need to [configure the consumer on Data Platform](#how-to-configure-power-bi-with-your-data-platform) and then [configure the connector on Power BI Desktop](#configuration-on-power-bi).
## How to configure Power BI with your Data Platform
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) _Power BI_ consumer in the Connectors. Else, just edit it.
Since the Trino connector for Power BI is an open sourced project and not yet certified by Microsoft, you have to **download the extension**, either manually from [this link](https://raw.githubusercontent.com/CreativeDataEU/PowerBITrinoConnector/refs/heads/master/Trino/bin/AnyCPU/Debug/Trino.mez) or from the interface shown below.
Power BI users will have to be registered in a Project's Identity Access Manager (IAM) in order to access its data. They will only access the data associated to their [API Secret Key](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md).
You can **easily grant full access to a new Power BI user** by assigning them the _Analytics Consumer_ role. This is a default role that can be assigned to a group, user, or service account and provides the appropriate permissions to view data from an external consumer.
Alternatively, if you prefer to fine-tune permissions, you can go to your project's Identity Access Manager and grant the user the following specific permissions:
- _Data Manager | Source | Read_ and _Data Manager | Table | Read_: to import a table's data inside Power BI
- _Advanced Data Access Control | Read_: to access the data inside the project
:::warning
Any user who has the role of **Admin** will also have access to the Project's data
:::
## Configuration on Power BI
Once you have [downloaded](https://raw.githubusercontent.com/CreativeDataEU/PowerBITrinoConnector/refs/heads/master/Trino/bin/AnyCPU/Debug/Trino.mez) the connector _.MEZ_ file, follow this procedure to use it on **Power BI Desktop**.
### Allow your Power BI Desktop to use a custom connector
Open _Power BI Desktop_. Click on **File** in the header.
Open the **Options and settings** section, then select **Options**.
Find the **Global > Security** section, then activate the following option: _(Not Recommended) Allow any extension to load without validation or warning_.
This will allow you to get data from a source not yet certified by Microsoft, in this case the Trino Connector.
Now, **close Power BI Desktop** for the changes to be applied.
### Install the connector
Create this folder in your user directory:
```yaml
C:\Users\[Your_Windows_User_Name]\Documents\Microsoft Power BI Desktop\Custom Connectors
```
If the _Microsoft Power BI Desktop\Custom Connectors_ folders already exists, then you don't have to create it.
:::warning
Make sure that the folders have exactly the same names as above, or you won't be able to find the Trino connector inside Power BI Desktop.
:::
Copy the connector _.MEZ_ file that you have previously downloaded into this new directory.
You can now restart Power BI Desktop.
### Import your first data from Data Platform
Click on **Get Data**.
Search for _Trino_, and select the connector.
You now have to enter the [URL of the Data Platform Project](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md) you want to connect to. You can easily find this URL on the page of the consumer on the Data Platform project. The port is 443.
If you are not yet authenticated to this Project, you will be asked to authenticate at this step.
Choose the Basic mode and use **API key** for User name and **Secret key** for Password as configured to your Service-Account/User in IAM.
When you are successfully authenticated, press **connect** to continue.
Then you will see your datasets, schemas and tables in preview mode.
You can now create your first visualizations on your data from your Data Platform project using Power BI Desktop.
:::info
If the data is updated on Data Platform, graphs can be manually refreshed with the latest data by pressing the refresh button in the Power BI header.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Forward lineage events to an external tool with OpenLineage
## Objective
The **OpenLineage consumer** lets you forward data lineage events generated by your Data Platform project to an external lineage solution of your choice. Once connected, the platform continuously streams [OpenLineage](https://openlineage.io/)-compliant events to your lineage backend so you can visualize and explore your data lineage graph there.
:::info
Data Platform also includes a built-in lineage viewer: see [Lineage](https://docs.dataplatform.ovh.net/lakehouse-manager-lineage.md) in the Lakehouse Manager. The OpenLineage consumer is for teams that want the same events in their own lineage solution, see [compatible tools](#compatible-lineage-solutions) below.
:::
:::warning
**Compatibility notice:** At this time, [Marquez](https://marquezproject.ai/) is the only lineage solution that has been validated to work reliably with Data Platform's OpenLineage events. Other OpenLineage-compatible tools may receive events but could provide a degraded experience. We recommend using Marquez for the best results.
:::
## Compatible lineage solutions
The following lineage solution has been validated to work with Data Platform:
| Tool | Description |
| :------------------------------------ | :---------------------------------------------------------- |
| [Marquez](https://marquezproject.ai/) | Open-source metadata service built natively for OpenLineage |
Any tool that exposes an OpenLineage HTTP API endpoint can technically be used, but only Marquez is officially supported at this time.
## Configure the OpenLineage consumer
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) **OpenLineage** consumer in the Connectors. Else, just edit it.
You will need to provide the following values:
| Field | Required | Description |
| :----------------------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint URL** | Yes | The HTTP(S) URL of your lineage solution's OpenLineage API (e.g. `https://marquez.example.com/api/v1/lineage`) |
| **Auth token** | No | A bearer token used to authenticate requests to your lineage backend. Leave empty if your backend does not require authentication |
| **Keep Events** | No | If true, events are stored in the bucket after being sent for history. Otherwise they are deleted. (Default: false) |
| **Disable Trino events** | No | If true, lineage events generated by Trino queries (Explorer, Analytics Manager) are not forwarded to your lineage backend. Only events from DPE actions (Load, Aggregate, Custom) are sent. Use this to reduce noise if you only care about batch lineage. (Default: false) |
Once a lineage consumer is configured, every lineage events will be stored on the [Lakehouse Manager Bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) **DWH** under `openlineage/events/queued`. Events will be forwarded to your lineage solution each time the [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) DPE action runs.
## Send lineage events from the DPE
Lineage events are forwarded by the [**Send OpenLineage Events**](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) action in the Data Processing Engine. You can configure it to run:
- **As a cron job**: on a fixed schedule (e.g. every hour, every night)
- **As an always-up action**: continuously, to keep your lineage solution in sync in near real time
See the [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) documentation for setup details.
:::info
For Custom actions, lineage tracking is opt-in. See [Track data lineage in a Custom action](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md) to learn how to emit lineage events from your own Python code.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-consumers-trino.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Generate a Trino connection string for external tools
## Objective
A **Trino consumer** generates a ready-to-use Trino connection string in a few clicks. Previously, connecting an external tool to your project over Trino meant creating a service account, generating API keys, and assembling the connection string by hand. A Trino consumer does all of that for you.
You'll find them in the Connectors service, under the **Consumers** tab, in the **Trino consumers** section of **Registered consumers**.
:::info
Unlike the PowerBI and OpenLineage consumers (which you connect once), you can create as many Trino consumers as you need. Each with its own name, permission level, and credentials.
:::
## Create a Trino consumer
1. In the **Trino consumers** section, click **+ New Trino consumer**.
2. Give it a **name**, an optional **description**, and optional **tags**.
3. Set the **permission**: the access level this Trino consumer has on your data:
| Permission | Access |
| :--------------- | :-------------------------------- |
| **READ** | Query and retrieve data only. |
| **READ & WRITE** | Query, retrieve, and modify data. |
4. Click **Create**.
5. Your **Trino connection string** appears.
:::warning
Copy the connection string now and store it somewhere safe (for example, a password manager). You can't view it again after you close this window, if you lose it, you'll have to create a new Trino consumer.
:::
## Manage the linked service account
Each Trino consumer has a dedicated [service account](https://docs.dataplatform.ovh.net/iam-service-accounts.md) in your project's Identity Access Manager. After you create the consumer, click **Go to service account settings** to view and manage it.
The project **admin also receives an email** containing the Trino connector credentials.
## Use the connection string
The connection string is ready to paste into any external Trino-compatible client or BI tool. It embeds the credentials and follows this shape:
```text
trino://:@-query.eu.dataplatform.ovh.net:443/
```
Plug it into your tool of choice, for example:
[Deploy Apache Superset](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md)
[Connect Metabase to Data Platform via Trino](https://docs.dataplatform.ovh.net/tutorials-metabase-trino.md)
:::warning
Any user connecting through a Trino consumer accesses data according to the consumer's permission and the project's [IAM roles and groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md). Treat the connection string like a password.
:::
## Query a different catalog
By default, the connection string targets your project's **default dataset** catalog (the trailing `/` segment, e.g. `db__default_dataset`).
To query a **different catalog** in the project, replace that trailing `/` segment of the connection string. To list the available catalog names, run the following in the [Lakehouse Manager Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md):
```sql
SHOW CATALOGS
```
For the full breakdown of the connection string and how to build one by hand, see [Integrating Trino with External Tools](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md).
:::info
Writing data through Trino sometimes requires an explicit `CAST`. See [Trino data types and casts](https://docs.dataplatform.ovh.net/tutorials-trino-types-and-casts.md) for the mapping between Lakehouse types and Trino types, and for troubleshooting `TYPE_MISMATCH` errors.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Lakehouse Manager
## What is Lakehouse Manager?
Lakehouse Manager is where a project's data lives. It is a serverless lakehouse built on [Apache Iceberg](https://iceberg.apache.org/), unifying data lake and data warehouse storage in one service: no cluster to size, and one open table format for both raw files and analytic tables.
Data is organized in [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md), which group [tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) the way a schema groups relations. Tables carry [attributes](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) with their own types and constraints, and every write is versioned, so [Time Travel](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md) can read a table as it stood at an earlier point. [Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) hold unstructured and semi-structured files alongside those tables, and the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) browses both.
Two capabilities sit across the whole store. [Lineage](https://docs.dataplatform.ovh.net/lakehouse-manager-lineage.md) maps how data flows through the project, from the pipelines that feed a table down to the tables built from it. [Policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) apply access control down to the individual attribute, evaluated through CEL conditions against the caller's IAM identity.
Tables are populated by the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) from sources registered in [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md), and read by the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
[Discover the Lakehouse Manager key concepts](https://docs.dataplatform.ovh.net/lakehouse-manager-key-concepts.md)
[### Key concepts→
Datasets, tables and the Iceberg model behind them.
](/lakehouse-manager-key-concepts)[### Datasets→
Group tables and control what a query can reach.
](/lakehouse-manager-datasets)[### Tables→
Create tables, set identifier fields and partitions.
](/lakehouse-manager-tables)[### Attributes→
Types, constraints and defaults on table columns.
](/lakehouse-manager-attributes)[### Buckets→
Object storage for files alongside the tables.
](/lakehouse-manager-buckets)[### Explorer→
Browse tables and bucket contents directly.
](/lakehouse-manager-explorer)[### Lineage→
See what feeds a table and what is built from it.
](/lakehouse-manager-lineage)[### Policy tags→
Attribute-level access control through CEL conditions.
](/lakehouse-manager-policy-tags)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-key-concepts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover the key concepts in the Lakehouse Manager
## Objective
:::info
At the heart of your data-driven initiatives, **the Lakehouse Manager stands as the pivotal hub for all your data assets.** With the Lakehouse Manager, you're primed to jumpstart your data-driven progress, bypassing the arduous tasks of data infrastructure cleanup and management, and instead focusing on what truly matters, **unleashing the full potential of your data.**
:::
The Lakehouse Manager component has 8 submenus, corresponding to its main features:
- **Overview** giving information on the various data assets (like their distribution, issues related to them, associated tags, storage usage, execution analysis)
- [**Datasets**](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) to logically group and manage Tables easily.
- [**Tables**](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) to create tables from sources and organize them in databases
- [**Attributes**](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) to visualize attributes across your data lake / data warehouse and their lineage in the Project
- [**Buckets**](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) to allow you to stock non-structured objects (pictures for example)
- [**Explorer**](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) to inspect your data hands-on with visual or SQL queries.
- [**Policy Tags**](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) to help you with fine-grained Advanced Data Access Control
- **Jobs** to give all users an overall overview of all the jobs and their status
The Lakehouse Manager takes data modelling a step further by proposing the creation of data lakes and data warehouses under the [tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) tab. Manage all your ingestion and primary tables, and design powerful 'data marts' - i.e. the aggregations designed to answer specific data visualization or AI needs. The Lakehouse Manager further provides collaborative working (with autosave) by allowing multiple users to work together on the same set of tables under separate views which could be specific to a Project or department. Tables are composed of [attributes](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) which can be processed, queried, or used as ML features.
[Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md)
is the next subcomponent, which brings along the flexibility by allowing you to store any unstructured data in an S31
compatible object-store.
[Datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) allows you to logically group and manage Tables easily.
[Policy Tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) was developed to help the user set 'policy tags' on _tables, attributes, and datasets_ for fine-grain Advanced Data Access Control.
Lastly, through the **Jobs** tab, you will be able to view the summary of your job performances. It provides you with quick hand information like status of jobs, jobs run overtime, job’s runtime, also running jobs, queued jobs, past jobs executed.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Organize tables into Lakehouse Manager datasets
## Objective
Datasets are logical groupings of tables, which allow them to share some default parameters.the Platform supports **Standard** as well as **External** Datasets to manage your tables.
By default you will be given one standard dataset named **default\_dataset**. This standard dataset will be using the _Lakehouse Manager Engine_ which is based on [Apache Iceberg](https://iceberg.apache.org/).
:::warning
Once you create a dataset, there is a waiting period of 60 seconds before it can be used for queries.
:::
## Datasets
Datasets were designed with the mindset of encouraging users to manage the parameters of a table at a group/dataset level. You can Edit, Duplicate and Delete datasets from the list view.
### Standard Dataset
The [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) does the dual role of being the data management service but also the storage engine for the standard dataset. You can learn more about this engine by clicking on the _discover_ link during the creation of a dataset.
Click on the **New Dataset** button to start creating a new dataset, then select _standard dataset_. Fill in all the required dataset fields and click **Create**.
## How data is stored
The Lakehouse Manager separates the **logical** organisation of your data from where the bytes actually live. Three concepts are easy to confuse, so it helps to keep them apart:
- **Datasets**: a logical envelope that groups tables together. A dataset does **not** store any data itself.
- **Tables**: the physical objects, stored in [Apache Iceberg](https://iceberg.apache.org/) and managed by the Lakehouse Manager.
- **Buckets**: S31-compatible object storage for unstructured files, independent from the Lakehouse. See [Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
A dataset is therefore **not** a bucket: it is a logical grouping of tables.
:::info
Because a dataset only holds metadata, sharing or moving a dataset never copies data: it changes what the Lakehouse points to. See [Cross-project sharing](#cross-project-sharing-beta).
:::
## Dataset Management
The configuration page shows all the information you need to know about your dataset. You can edit the description and the tags for your datasets, you can also view the data residency policies - storage engine used and data residency location.
In the tables tab, you can view all the associated tables that are a part of the dataset. It is important to remember that this is only a view, and you cannot edit any of the tables from this menu.
The [policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) menu allows you assign dataset wide policy tags, which will be further applied to all the tables part of the dataset.
You can choose from the available policy tags, or even create policy tags on spot. [Here](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md#binding-policy-tags-to-data) is the step-by-step guide to binding policy tags.
## Organising your data: the medallion approach
The Data Platform does **not** prescribe a Bronze / Silver / Gold ("medallion") architecture: you are free to model your lakehouse the way that fits your use case. By default, a dataset is provisioned for you (**default\_dataset**).
That said, managing medallion layers **with datasets** is a valid pattern and works naturally with the rest of the platform. A typical pipeline looks like this:
1. **Bronze**: raw data, ingested through [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md).
2. **Silver**: cleaned, validated and deduplicated by [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) actions.
3. **Gold**: enriched, joined and aggregated by Data Processing Engine actions, ready for consumption in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) or your BI tool.
Each layer is a dataset; the transformations between layers are actions orchestrated in a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md).
## Cross-project sharing (beta)
:::info
This feature is currently in beta. Please report any issues you run into.
:::
By default, every project on the Data Platform is fully isolated: there is no implicit way to reach a dataset from another project. Cross-project sharing lets the owner of a dataset expose it, from the **source project**, to one or more **receiving projects** in a controlled way, without breaking the existing security boundaries.
### What sharing does, and does not, do
- Sharing happens **at the dataset level**, between projects **within the same organization**. You cannot share with another organization.
- A shared dataset is a **singleton, not a copy**. Reads, writes, builds, and use in notebooks and custom actions apply to the same underlying data on both the source side and the receiving side.
- For now, sharing is **full**: the entire dataset is exposed to the receiving project.
- Isolation between unrelated projects stays strictly enforced.
### Permissions and roles
Sharing is managed from the source project.
- A shared dataset is exposed in **READ & WRITE**.
- The **LM & AM Editor** role is required to share a dataset or to remove an existing share.
:::warning
Access to a shared dataset is always evaluated **in the context of the receiving project**, not the source project. When user _B_ in project _B_ reads a dataset shared from project _A_, the platform asks "does user _B_ have access to this dataset **in project B?**", never "in project A". Since user _B_ does not exist in project _A_, that second question would always return false. Your access rules therefore stay anchored in the receiving project.
:::
:::info
A finer-grained **Advanced Data Access Control** module is under study for more granular permissions on shared objects.
:::
### Share a dataset
1. From the source project, open the dataset you want to share and go to its **Configuration** tab.
2. Click **Manage share settings**.
3. Select one or more projects from the available list of projects in your organization, then confirm.
4. You will receive an email confirming that the dataset has been shared.
The source project shows usage information for each project the dataset has been shared with, so you can keep track of who is consuming the shared data.
### On the receiving project
Shared datasets appear in the **Datasets** menu of the receiving project's Lakehouse Manager, alongside the project's own datasets. They are flagged with a shared icon and the type **Lakehouse Manager (shared)**.
Opening a shared dataset shows its details in **read-only** mode: the dataset configuration and table structure stay managed from the source project. The associated tables can be previewed by clicking the preview icon.
The shared dataset can be used as a source in Connectors and in Trino, so users of the receiving project work with its data exactly as they would with a dedicated dataset.
### In the Explorer
Inside the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md), use the **list by dataset** option to find a new **Shared Datasets** category. All tables coming from datasets shared with this project are listed here.
:::info
Shared dataset tables can only be queried using the **SQL editor**. Visual builder support is not available yet.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage tables in the Lakehouse Manager
## Objective
_Tables_ give you **full control over your data lake / data warehouse** (tables and attributes) by providing a visual and logical structure of tables within datasets on a single screen.
When you access the Tables tab, you will be in the [Canvas view-mode](#discover-the-canvas-mode), selected by default. This view-mode is designed to provide a visual depiction of your data. However, this representation can be overwhelming if there are too many tables. That is why we also provide the **List view-mode**.
:::info
Note that some features are only available on one of the view-modes. For instance, [linking tables of external datasets](#link-parent-and-children-tables) is only doable in Canvas-view.
:::
## Create a table
Hover over the _+_ icon (or the _New table_ button if you are in list mode) to see the options for creating a new table on the schema:
- **Create a table by uploading a file**
- **Create a table from source**
- **Create an empty table**
The [getting started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) contains the step-by-step guide to creating a table from source and creating an empty table. Here, let's see the how create a table by uploading a file:
1. Click on ➕ and select - Upload a file
2. Upload the file using the options shown on the screen. (Currently support is present for CSV and XLSX files)
3. Set the name and other parameters for the table and click and create.
Now the uploaded file will go through the analyzer and a meta extract process and a job will be triggered which will build the table and load it with data.
## Why declare a table in the logical catalog
The **Tables** tab is not a cosmetic wrapper over Iceberg: it is the **single source of metadata** the rest of the platform relies on.
- **Identity Access Manager**: [policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) and CEL conditions (Advanced Data Access Control)
- **Cross-component lineage**: the [Attributes](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) page
- **Business semantics**: dimensions and measures, dictionaries, parent/children links
- **Discovery**: the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) and the canvas
- **Integrations**: Analytics Manager and Machine Learning Manager
A table created with a direct `CREATE TABLE` (from Superset or a SQL Custom action) does exist in Iceberg, but stays **orphaned** on the platform side: invisible in the UIs, outside governance, and outside lineage.
### Making an externally-created table visible
To bring such a table into the logical catalog, either:
- **Declare it in Tables before writing**: create the empty table (or set it as the logical target of your action), then write into it; or
- **Run an Update Metadata** after the fact to pick up an already-written table.
:::info
We are investigating whether this step can be simplified through a unified catalog. In the meantime, declaring the table up front is the reliable path.
:::
## Tables
Lakehouse Manager was built with the mindset of encouraging users to adopt the best practices regarding data management. Tables in Lakehouse Manager are almost the same as your regular SQL-database tables (in fact they are implemented using SQL).
On Lakehouse Manager interface, when you hover over a table (List-view) or click on it (Canvas-view) you will see that a series of buttons are displayed. You can use them to interact with the table; some common interactions are :
- ➕ - [add](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#add-a-new-attribute) an attribute (can be a dimension or measure).
- 🔍 - preview the table within [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md).
- ✏️ - [edit](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#edit-table) the table.
- ▶️ - [build](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#build-table) the table.
- 🗑 - [delete](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#delete) the table.
[Check out the complete detailed documentation here](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md)
### Link parent and children tables
:::info
This feature is only available in the [canvas mode](#discover-the-canvas-mode).
:::
:::warning
Linking tables is on possible for tables of external datasets
:::
Lakehouse Manager allows you to link a table (the parent) to other tables (children tables) in order to model an 'inheritance-like' system for the information in your data warehouse.
Concretely, this allows you to centralize information in one single referential table - the parent table - which in turn is referenced in many other children table via IDs. Linking tables allows to automate the configuration of [Aggregate actions](https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md) (which allow you to join multiple tables into a single aggregated table when you execute them) when you create them in the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md)
The primary key(s) of the parent table must be present in the children table - if they're not, they will be automatically created as you link both tables.
:::info
For example, if you are representing a news article in with your data. There is a table called `articles` with fields for _article\_id_, _headline_, _content_, _date_ and _author_. But instead of placing a name in the author field, you could have the _ID_ value of a user in a separate `authors` table with fields such as _author\_id_, _first\_name_, and _last\_name_. Therefore, if you need to update an author’s name, you only need to do so in the `authors` table (parent); because the `articles` table (child) only contains the ID of the corresponding author record.
:::
Link a **parent table** to a **child table** by clicking on the white circle at the bottom of the parent table (1) and dragging the arrow to the child table (2).
As a rule of thumb, you want to start from the table which contains all the reference data, a link it into a table which only has an ID referencing that initial table.
### Naming Conventions
When naming tables and attributes in Lakehouse Manager, it's important to follow certain conventions and avoid specific reserved words to prevent unexpected issues. These restrictions primarily stem from underlying SQL implementations and best practices for data integrity.
- **General Naming Guidelines:**
- **Case Sensitivity:** It's generally recommended to use lowercase and underscores (e.g., `my_table_name`, `column_id`) for consistency and to avoid potential issues with case-sensitive databases.
- **Characters:** Stick to alphanumeric characters (a-z, 0-9) and underscores (`_`). Avoid special characters, spaces, or hyphens, as these can cause problems in queries or integrations.
- **Start with a Letter:** Table and attribute names should ideally start with a letter.
- **Descriptive Names:** Choose names that clearly describe the data they contain.
### Reserved Words
There are words reserved by the underlying database system or that have special meaning within Lakehouse Manager. Using these words _alone_ as a table name or attribute name can lead to errors. However, they can typically be used as part of a larger name (e.g., `count_records` is allowed, but `COUNT` is not).
To ensure compatibility, avoid using the following reserved words:
| Reserved Word | Reserved Word | Reserved Word | Reserved Word |
| ------------- | ------------- | ------------- | ------------- |
| NULL | WITH | ALTER | CREATE |
| TABLE | AND | END | AS |
| COUNT | MAX | MIN | SUM |
| AVG | ROUND | BETWEEN | CASE |
| INSERT | UPDATE | DELETE | HAVING |
| IS | NOT | LIKE | LIMIT |
| OR | GROUP | ORDER | BY |
| INNER | OUTER | JOIN | SELECT |
| DISTINCT | WHERE | FROM | DROP |
| UNION | RANGE | ROWS | OVER |
| RECURSIVE | INTERSECT | EXCEPT | VALUE |
| OFFSET | CHECK | | |
:::tip
If you encounter an error related to a table or attribute name, consult the table above. If in doubt, try using a more descriptive name that includes a prefix or suffix (e.g., `user_count` instead of `COUNT`).
:::
## Discover the canvas mode
You can navigate the schema by clicking on any blank space in the schema and then moving your cursor. If there are a lot of tables, you can move the view window navigation (indicated by 2) by clicking on it and moving your cursor. Additionally, you can zoom in/out on your data schema (indicated by 3).
At any point of organizing your tables, Lakehouse Manager autosaves all the visual modifications of your data schema.
:::info
Use the undo-redo options (indicated by 1) to move back/forward on any change you just made in the schema. You will go back/forward to an immediate previous state of your work.
:::
### Filtered tabs
the platform allows you to choose to work on a selected set of tables (which could be specific to a team or Project within your organization). You can add a New Filtered Tab for doing this (indicated by 6).
:::info
Note - The default tab has All Tables with All Users.
:::
### Interact with a table
Hover your mouse over a table to display some table interactions and hover over the _..._ icon to display the rest.
Access our [table interactions documentation](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md) to find out more about actions available on a table.
### Interact with an attribute
In the Canvas view-mode you can also easily manipulate the attributes of each table. Hovering your cursor over an attribute will display **attribute interactions**:
- Set it as a **Primary key** by clicking on ⭐️ (_only available for tables in external datasets_)
- Edit the attribute by clicking on ✏️
- Delete the attribute 🗑
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage a table in Lakehouse Manager
## Objective
There are six table interactions in total. When in Canvas-view, display the first three options by clicking on a table and, display the other three, by clicking on the '**...**' icon after clicking on the table. In List-view, just scroll over the table to display all available interactions.
- ➕ [Add a new attribute](#add-a-new-attribute)
- [Attribute constraints](#attribute-constraints)
- ✏️ [Edit table](#edit-table)
- [Basic actions](#basic-actions)
- [Labeling and indexing](#labeling-and-indexing)
- ▶️/⏹️ [Build table/Stop building table](#build-table)
- 📑 [Duplicate](#duplicate)
- 🗑 [Delete](#delete)
## Add a new attribute
:::info
Represented by the symbol ➕.
:::
Adds a new attribute to the target table by opening the attribute creation window, shown below.
### Attribute constraints
When setting an attribute on a table, you can add the following constraints:
- **Required:** the field can never contain NULL. If a job tries to insert a null value into it, the query engine returns an error.
- **Required + Identifier:** the field is required _and_ marked as an **identifier field**, meaning it is used to define the uniqueness of a row for **upsert** operations. See the [write strategy](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#write-strategy) of the Load / Aggregate (Python and PySpark) actions for more information.
:::warning
**Once a table is built, you cannot add a Required field.** You can still _remove_ the Required property from an existing required field, but you cannot add the NOT NULL constraint to a field after the fact. The only way to add a required (or identifier) field to a built table is to fully rebuild it. See the [Rebuild action](https://docs.dataplatform.ovh.net/dpe-actions-rebuild.md).
:::
:::info
An identifier field cannot be a `double` or a `decimal`.
:::
## Edit table
:::info
Represented by the symbol ✏️.
:::
Allows the editing of the table's schema and configuration.
### Basic actions
You can change the following basic configurations :
- Name of the table
- Table type
- Table's description
- Table tags (used to easily search for it)
- Queryable (if a table isn't set as queryable, it won't be detected by the automated visual query builders of Analytics Manager and Lakehouse Manager Explorer)
### Labeling and Indexing
When editing a table you can further add:
- A **label**
- One or several **partition keys/indexes** (_under advanced options_)
#### Labels
You can choose one of the table’s attributes as an optional **label** for the primary key of this table.
This turns the table into a _dictionary_ which is going to be stored in the cache of the [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) that you deploy in your Project. The API will enforce the translation in the queries using this dictionary, and the label will automatically be displayed in your applications instead of the primary key.
:::info
**Pro Tip:** By defining labels, you don't have to store heavy additional data in aggregated tables.
:::
#### Indexes
Further, you can set **indexes** or **partition keys** to quickly locate data without having to search every row in a table.
:::info
**Pro tip:** Defining indexes improves the speed of data retrieval operations on a table at the cost of additional writes and storage space to maintain the index structure.
:::
## Build table
:::info
Represented by the symbol ▶️ if clicking to build, or ⏹️ if stopping build.
:::
Click to start or stop build job for the target table. This job corresponds to physically building your tables in the database and is **required for loading and accessing your data** in the platform.
Starting the build job will automatically open the job logs that you can monitor the job status:
## Duplicate
:::info
Represented by the symbol 📑.
:::
Creates a copy of the target table that has to be built separately.
## Delete
:::info
Represented by the symbol 🗑.
:::
Removes the target table from the logic data schema.
Removes its physically built counterpart as well if the table has already been built.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Restore previous table versions with Time Travel
## Objective
Time Travel is a powerful feature in Lakehouse Manager that enables you to **view, explore, and restore previous versions of your tables**. This capability provides complete recovery options for your data, allowing you to track changes over time and restore to any previous state when needed.
With Time Travel, every modification to a table such as data insertion, deletion, or other changes are automatically captured as a snapshot. These snapshots form a comprehensive version history that you can navigate, inspect, and use to revert your table to any previous point in time. This functionality is essential for data governance, debugging, and recovery from accidental changes.
:::info
Time Travel operates automatically in the background. Every change to your table creates a new snapshot without requiring any manual intervention or configuration.
:::
## Understanding snapshots
### What is a snapshot?
A snapshot is an immutable point-in-time representation of your table's state, capturing both the data and schema as they existed at a specific moment. Each snapshot is identified by a unique snapshot ID and includes metadata about when it was created, what changes were made, and its relationship to other snapshots in the version history.
**Snapshot retention:**
- Default retention: **14 days**
- Configurable range: **7 to 28 days**
- Can be set at dataset level or overridden per table
- After expiration, snapshots are automatically deleted
:::info
**Pro tip:** Choose your retention period based on your data change frequency and recovery requirements. Tables that undergo frequent changes or contain critical business data may benefit from longer retention periods.
:::
### How snapshots are created
Snapshots are created automatically whenever you modify your table through operations such as data insertions, updates, deletions, or schema modifications. The platform captures the table's state and creates a new snapshot representing the post-change state.
Each snapshot maintains a parent-child relationship with its predecessor, creating a linked history that shows how the table evolved over time. Snapshots are lightweight and efficient, the storage system stores only the differences (deltas) between versions rather than complete copies.
### Snapshot statuses
Snapshots can have the following statuses:
- **Current**: The most recent version of the table representing its current state _(displayed in ● blue)_
- **Parent**: The immediate predecessor of the current version _(displayed in ● light blue)_
- **Rollbacked**: A snapshot that was previously current but has been reverted from _(displayed in ● gray)_
## Accessing the Time Travel interface
To access Time Travel for a specific table:
1. Navigate to **Lakehouse Manager** from the main navigation menu
2. Select **Tables** from the left sidebar
3. Click on the table you want to view
4. Click on the **Time Travel** tab
## The Time Travel interface
The Time Travel interface is designed to provide both a high-level overview and detailed information about your table's version history. The interface is divided into two complementary sections that work together to help you understand and manage snapshots effectively.
### **Changes history visualizer**
The Changes History Visualizer is a graphical timeline that provides an at-a-glance view of your table's evolution over time. This visual representation helps you quickly understand the pattern of changes, identify important versions, and see the relationships between different snapshots.
The visualizer displays snapshots as nodes on a timeline, with each node representing a specific version of your table. The nodes are connected by lines that show the parent-child relationships between snapshots, illustrating how changes flow from one version to the next. This linear representation makes it easy to trace the history of your table chronologically.
When operations such as reverts occur, the visualizer shows branching in the timeline. A revert creates a new current version that branches from an older snapshot, while the intermediate versions remain in the history. This branching visualization helps you understand the non-linear aspects of your version history and see how reverts affect the overall structure.
#### Graph controls
The visualizer provides several control buttons to help you navigate and view your version history:
- **Zoom in**: Enlarge the graph to see more detail
- **Zoom out**: Reduce the graph size to see more snapshots at once
- **Fit view**: Automatically adjusts the zoom level to fit the entire graph in the viewport, providing a complete overview of your version history
- **Center on current node**: Instantly jumps to and centers the view on the current (most recent) snapshot
These controls are especially useful when working with tables that have extensive version histories spanning many snapshots.
#### Interactive tooltips
The visualizer is fully interactive. Hovering over any node displays a tooltip with:
- **Version status**: Current, Parent, or Rollbacked (see [Snapshot statuses](#snapshot-statuses))
- **Snapshot ID**: The unique identifier for this version
- **Snapshot timestamp**: When this snapshot was created
- **Nature of change**: Description of what changed (e.g., "Data inserted", "Data deleted", "Data updated")
Click on any node to copy its snapshot ID to your clipboard.
### **Snapshots list**
The Snapshots List provides a detailed, tabular view of all snapshots for the table, ordered from newest to oldest. This list complements the visual timeline by offering comprehensive information and direct access to actions for each snapshot.
The list is paginated to handle tables with extensive version histories, ensuring the interface remains responsive even when dealing with hundreds of snapshots. You can navigate through pages to explore the complete history of your table.
Each row in the list represents a single snapshot and contains the following information:
1. **Version Status:** Indicates the snapshot's current state (see [Snapshot statuses](#snapshot-statuses))
2. **Snapshot ID:** The unique identifier for this snapshot, used in queries, API calls, or support communications
3. **Date of Snapshot:** When this snapshot was created, displayed in your local timezone
4. **Changes from Parent Version:** Description of what changed in this snapshot. Common change types include:
- **Data inserted**: New rows were added to the table
- **Data deleted**: Existing rows were removed from the table
- **Data updated**: Existing rows were modified
- **Data replaced**: Existing data was replaced
- **Data rollbacked**: Rollback operation was performed
5. **Actions:**
- **🔍 Explore**: Opens the snapshot in the Explorer to view its data
- **↻ Rollback**: Restores the table to this snapshot's state
These actions are described in detail in the [Working with snapshots](#working-with-snapshots) section.
## Working with snapshots
### **Filtering and searching snapshots**
When working with tables that have extensive version histories, the Time Travel interface provides several tools to help you locate specific versions.
#### Search by Snapshot ID
The search bar allows you to find a specific snapshot by entering its ID. The list automatically filters as you type.
#### Filter by Change Type
View only snapshots that correspond to specific modifications:
- **Data inserted**: Show only snapshots where data was added
- **Data deleted**: Show only snapshots where data was removed
- **Data updated**: Show only snapshots where existing data was modified
- **Data replaced**: Show only snapshots where existing data was replaced
- **Data rollbacked**: Show only snapshots where rollback operation was performed
#### Filter by Time Range
Focus on snapshots created within a specific period:
- **Last 7 days**: Shows snapshots from the past week
- **Last 14 days**: Shows snapshots from the past two weeks
- **Last 21 days**: Shows snapshots from the past three weeks
- **Last 28 days**: Shows all available snapshots (within maximum retention)
You can combine multiple filters to narrow your search further.
### **Exploring a snapshot**
The explore functionality allows you to inspect the data contained in a specific snapshot without affecting your current table. This is a read-only operation that provides visibility into historical states of your data.
To explore a snapshot:
1. Click the magnifying glass icon (🔍) next to the snapshot you want to inspect
2. The Lakehouse Manager [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) opens in a new tab with a pre-configured SQL query
3. Click "Run" to execute the query and view the snapshot data
4. Interact with the data as needed: scroll, sort, filter, or export results
This exploration capability is particularly valuable for:
- Investigating when specific data values changed
- Verifying the state of data at a particular point in time
- Understanding the impact of past changes
- Retrieving data that was accidentally deleted or modified
:::info
**Pro tip:** Use the explore feature before performing a revert to verify that a snapshot contains the data you expect. This can prevent unnecessary revert operations.
:::
### **Reverting to a previous snapshot**
Reverting to a previous snapshot is a powerful capability that allows you to restore your table to a prior state. This operation creates a new snapshot that is identical to the snapshot you're reverting to, making it the new current version of your table.
Reverting is a significant operation that affects your table's current data and schema. Because of its impact, the revert process includes mandatory confirmation steps to prevent accidental changes.
**Step 1: Initiate the revert**
Click the circular arrow icon (↻) next to the snapshot you wish to restore. This opens the revert confirmation dialog.
**Step 2: Review the confirmation dialog**
A modal dialog appears with important information about the revert operation:
**Step 3: Confirm or cancel**
If you confirm, the system:
1. Reverts your table to the selected snapshot, making it the current version
2. Updates the Changes History Visualizer to show the branching that occurred
3. Displays a success message
#### Understanding the result
After a successful revert:
- Your table now contains the exact data and schema from the snapshot you reverted to
- A new snapshot has been created at the current time, preserving this state
- All previous snapshots remain available in the history
- You can revert to a different snapshot later if needed, including reverting forward to a more recent state
## Managing snapshot retention
Snapshot retention controls how long historical versions of your table remain available in the system. Proper retention management helps balance the need for historical data access with storage efficiency and cost considerations.
The current snapshot retention period for your table is always displayed in the Time Travel interface, ensuring you're aware of how long historical snapshots will remain available.
### **Configuring retention periods**
Retention can be configured at two levels:
#### Dataset-level retention (default)
All tables within a dataset inherit the dataset's retention period, set to **14 days** by default. You can modify this for an entire dataset to apply to all tables that haven't been individually configured with an override.
To configure dataset-level retention:
1. Navigate to the Datasets section in Lakehouse Manager
2. Click on the dataset to open its details page
3. Access the configuration or settings panel
4. Select your desired retention duration:
- **7 days** (minimum)
- **14 days** (default)
- **21 days**
- **28 days** (maximum)
5. The changes are automatically saved
This change applies immediately to all tables within the dataset that don't have a table-level override configured.
#### Table-level retention override
For tables that require different retention periods than their parent dataset, you can configure a table-specific override. This override takes precedence over the dataset-level setting.
To configure table-level retention:
1. Navigate to the specific table and open its details page
2. Access the configuration or settings panel
3. Enable "Override dataset retention"
4. Select the retention period (7, 14, 21, or 28 days)
5. The changes are automatically saved
6. Verify in the Time Travel tab that the override is active
To remove a table-level override, simply disable the override setting in the table configuration.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-views.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Save a SQL query as a reusable view
## Objective
**Views** let you save a SQL `SELECT` query as a named, reusable **virtual table** inside a [dataset](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md). A view looks and behaves like a [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) everywhere you query data, in [Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) queries and dashboards, and in [notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md), but it stores no data of its own: every time it is read, the result is computed live from its source tables. Change the underlying data and the view reflects it instantly, with no build or refresh step.
Typical uses: exposing a cleaned or filtered version of a raw table, pre-joining tables that are always queried together, renaming or hiding columns for consumers, or sharing one agreed-upon business definition of a metric instead of copy-pasting SQL.
## Instructions
### Create a view
A view needs three things: a **name**, the **dataset** it lives in, and the **SQL `SELECT`** statement that defines it. To create one, go to the _Views_ section in the Lakehouse Manager, then click **New View**.
- The SQL is **validated immediately** on creation. Errors come back human-readable, with the line and column in your SQL and the object names you know, for example `line 2:3: Column 'name' cannot be resolved`.
- The view's **columns are derived automatically** from the SQL, names and types included. You never declare a schema.
- The view can read tables from its own dataset (plain names) or from other datasets (qualified names), and it can read **other views**: views stack.
- A view and a table cannot share a name within a dataset. Collisions are rejected in both directions: you also cannot build a table over an existing view's name.
### View status: Valid and Broken
Every view has a status, and the platform keeps it truthful for you:
| Status | Meaning |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Valid** | The view's SQL currently resolves against its source tables and can be queried. |
| **Broken** | At least one source table or column the view depends on is missing. The view cannot be queried until the source is restored. |
- When a platform action changes a source table (a build finishes, a column is added or removed, a table is deleted or rolled back), the views reading that table are **re-checked automatically within seconds** and their status updates on its own.
- A broken view tells you **why**: if a source table is gone, the missing tables are listed by name. Opening the view re-checks it live at column level, so a dropped column is detected the moment you look.
:::info
Breakage is **not destructive**: the view definition is kept, and as soon as the source is restored (the table rebuilt, the column re-added), the view flips back to _Valid_ automatically. No manual repair needed.
:::
Changes made entirely outside the platform (for example a notebook writing straight to the storage layer) emit no signal. Those are picked up by periodic background checks instead: within minutes for a missing table, within a few hours for column-level drift.
### Edit, rename, delete
From the views list, click the **edit** icon to open a view. Its **title and SQL definition are editable in place**: as soon as an edit is in progress, a **Save** button appears. The list also shows when each view was **last updated**.
- The **SQL definition can be replaced at any time**. The new SQL is validated first and applied atomically, so a failed edit never leaves the view half-changed. Columns and status are recomputed on the spot.
- **Renaming is guarded**: if other views read this one, the rename is refused and the dependent views are listed, because renaming would break them silently otherwise.
:::warning
**Deleting is allowed even when other views depend on this one.** The dependent views are immediately marked _Broken_, with the deleted view pinpointed as the missing source.
:::
### Governance
- **[Policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md)** work on views exactly as on datasets and tables: they can be set at creation or on update, and are shown wherever the view is shown.
- **Permissions ride the dataset**: whoever can read a dataset can read its views, and whoever can edit a dataset's data model can create, edit and delete its views. No new roles or grants to learn.
- Changes to views are recorded in the **[audit trail](https://docs.dataplatform.ovh.net/control-center-auditing.md)** like other data-model changes.
### Views created outside Lakehouse Manager
Views created directly on the lakehouse by external tools (dbt, notebooks, BI tools) are **discovered automatically** and appear alongside the ones created in the Lakehouse Manager. Their status is tracked the same way, and their SQL is shown read-only, as stored in the catalog.
### Good to know
- A view is computed at query time: a heavy view is a heavy query. For expensive transformations that are read often, a built (materialized) table may be the better tool. Views are for definitions that should always be fresh.
- Creating or editing a view takes a few seconds, because the SQL is really executed against the query engine for validation. Listing and opening views is instant.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage table attributes in the Lakehouse Manager
## Objective
The Attributes page groups all the attributes in your lakehouse Manager by **unique attribute names**. Each of those attributes are part of a realm: _physical_ or _virtual_.
## Physical attributes
Physical attributes are the attributes physically stored in [Lakehouse Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md). You can edit their category, type, and other information at the level of the table.
They are grouped by name on the Attributes page, letting you watch at a glance the different categories and types each attribute takes across all of your data warehouse / data lake.
Each attribute can be expanded in order to get full lineage over the objects in your Project that use this attribute:
- Lakehouse Manager [tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md)
- Data Processing Engine [actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md)
- Analytics Manager [queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md)
Some attributes can be a key for a **dictionary**. In that case, another attribute is used as a label for this attribute, to be automatically displayed in applications instead of the key.
Labels can be set at the level of a table (containing the dictionary key as its primary key) in the [Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) screen.
### Supported attribute types
Attributes support the following primitive types:
`integer`, `bigint`, `float`, `real`, `double`, `decimal`, `varchar`, `boolean`, `timestamp`, `date`.
There is no native `GEOMETRY` type, and complex or nested structures are not stored natively. The patterns below cover the most common needs in the meantime.
:::info
Custom business types with native validation in the schema (SIRET, IBAN, GPS coordinates) are not supported. Store a SIRET or an IBAN as a `varchar`, without schema-level validation.
:::
### Working with unstructured or complex data
**Documents (PDF, Word, ...)**
1. Store the raw file in a [bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
2. Extract its content in a [Custom action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) (Python, with libraries such as `pypdf` or `python-docx` installed through _Python Requirements_).
3. Write the metadata and the extracted text into a table so it can be queried.
**Geographic objects**
There is no native path today. Two options:
- Store the geometry as **WKT** text in a `varchar` attribute, or as **WKB** in a `VARBINARY` attribute.
- Run spatial computations outside the schema, with **Apache Sedona** (added as a Git dependency in a PySpark Custom action) or with `geopandas` / `shapely` in a Notebook.
:::info
Trino exposes the standard spatial functions (`ST_*`) when the deployed version includes them. Confirm availability for your project before relying on them in a query.
:::
**Semi-structured JSON**
- Use a `ROW` / `MAP` field in Iceberg when the schema is stable.
- Use a `varchar` attribute plus Trino's `json_*` functions (`json_extract`, `json_parse`) when the schema varies.
**Full-text / vector search**
There is no native path. The recommended pattern is to export to an external service (a managed Elasticsearch, a vector database) through a Custom action on the output side.
:::info
A **PostgreSQL** storage engine is on the roadmap and will, in time, be the best answer for geographic data through PostGIS.
:::
### Constraints
A physical attribute can carry constraints that are applied to the table when it is built:
- **Required:** the field can never contain NULL.
- **Required + Identifier:** the field is required _and_ used to define the uniqueness of a row for [upsert](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#write-strategy) operations. An identifier field cannot be a `double` or a `decimal`.
You set these when [adding or editing an attribute](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints) on a table.
:::info
Constraints apply to physical attributes only. Virtual attributes cannot be Required or Identifier.
:::
## Naming conventions
Attribute names are **unique per table**, but the platform aggregates them **by name at the Project level** on the Attributes page. This is useful to spot inconsistencies, for example the same `code` attribute typed as `integer` in one table and `bigint` in another, but it does **not** prevent them.
The general naming guidelines and the list of reserved words apply to attributes as well as tables: see [Naming Conventions](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#naming-conventions) and [Reserved Words](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#reserved-words) on the Tables page.
:::warning
`VALUE` is on the reserved words list. Prefer `amount`, `metric_value` or `entity_value` instead.
:::
### Handling homonym attributes
When attributes such as `code`, `libelle` or `value` come from different sources with different types, use the following practices:
1. **Prefix by domain or source.** Use `clt_code`, `clt_libelle`, `clt_value` for customers, `prd_code`, `prd_libelle` for products, `cmd_code` for orders, and so on. A prefix turns `code` into `entity_code` and removes any ambiguity, both when reading and in auto-detected formulas.
2. **If renaming is not possible, harmonise the type across tables.** Pick the widest type (`decimal` or `bigint`) and `CAST` at `INSERT` time in the source tables. This removes at least the type inconsistency on the Attributes page.
3. **Document with tags and a description** to distinguish usages when a name is ambiguous.
4. **Reserve short generic names** (`code`, `libelle`, `value`, `id`, `name`, `status`) for cross-cutting use, or avoid them altogether.
## Documenting shared attributes: the data dictionary
The Data Platform does not ship a full external data catalog, but several built-in mechanisms already act as a functional data dictionary:
- **The Attributes page.** One entry per attribute name across the whole Project, a breakdown of the types and categories it takes across tables (ideal for spotting `code` as `integer` vs `bigint` vs `decimal`), and full lineage over the objects that consume it, as described [above](#physical-attributes).
- **Policy tags.** Although [policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) were designed for access control, they can also be reused to categorise your data: `domain: finance`, `sensitivity: high`, `pii: true`, `glossary_term: client_id`. Policy tags are [inherited](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md#inheritance) from dataset to table to attribute, so a tag set on a dataset applies to every untagged attribute below it. This lets you set a baseline and override it where needed.
- **Dictionaries.** The dictionary mechanism (key → label) described [above](#physical-attributes) handles code → label reference tables.
Enriched per-attribute descriptions, a business glossary, ownership, and integration with an external catalog (Collibra, DataHub) are not available. For those needs, a complementary external tool is still required.
## Virtual attributes
Virtual attributes are used to compute new formulas on your data. They are not stored in the database but are computed on the fly for a [query](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) or a dashboard/restitution chart.
### Virtual attribute formula
The most important element of a virtual attribute is its SQL definition.
Write a formula which contains a combination of SQL keywords and [physical attributes](#physical-attributes).
:::info
The formula should be written using **ANSI-compliant** SQL syntax to ensure the portability of your Project. Engine-specific syntax is not supported by Data Platform.
:::
#### Example of virtual attribute formula
`SUM(rides)/COUNT(DISTINCT CONCAT(CAST(date AS VARCHAR), CAST(station_id AS VARCHAR)))`
Here, _rides_, _date_, and _station\_id_ are physical attributes coming from a Lakehouse Manager table. The Platform will automatically detect them, along with the most relevant table to query from (if it is not explicitly defined in your formula).
### Virtual attribute options
#### Required attributes
the platform automatically detects the [physical attributes](#physical-attributes) required to run the virtual attribute formula.
You can edit them in the advanced options of the virtual attribute. Reasons for editing the type include (but are not limited to): a physical attribute in the formula is detected as a SQL keyword instead of a required attribute, or the vice-versa.
#### Type
The type of the virtual attribute is automatically inferred when you create it. This type allows the platform to automate the planning of [queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) which use the virtual attribute.
You can edit it in the advanced options of the virtual attribute. Reasons for editing the type include (but are not limited to): filter values based on a virtual attribute are automatically converted to another type resulting queries to fail in the [Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-queries.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Store unstructured data in S3-compatible Data Platform buckets
## Objective
Data Platform Buckets are **S31 compatible object stores** which allow you to store any unstructured data. Think of buckets as different workspaces, for instance, a repository of photos or videos.
## Create a bucket
To create a new bucket, click on the _+ New Bucket_ button on the bottom left side of the screen.
Then, name your bucket and click on _Confirm_.
## Upload files
Once you have created your bucket, you will see a _+ Upload file_ button. Just click on it and drag-and-drop your file to upload it.
:::warning
You can upload files with an individual size of up to **10 GB**.
:::
## Delete a bucket
To delete a bucket, click on _Remove Bucket_ on the top righ-hand corner of the screen. Then you will have to write the input the name of the bucket and click on _Confirm_ to delete the bucket.
## Iceberg bucket
The Buckets screen also lists your project's **Iceberg bucket**: the object storage holding the data of your [Lakehouse Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
:::info
The Iceberg bucket is **read-only**. You can browse its files, but you cannot upload, modify, or delete anything in it, and the bucket itself cannot be deleted. To work with the data it contains, use your project's [tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) and the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md).
:::
## Manage permissions on buckets
You can also manage how each user is allowed to interact with the buckets. These permissions are managed on the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
For details refer to the following article:
- [Roles and conditions: Add a role condition to a specific bucket](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#add-a-role-condition-on-a-specific-bucket)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Explore your data with the Lakehouse Manager Explorer
## Objective
The Explorer is a powerful and flexible tool designed to **perform exploratory data analysis** in an easy-to-use interface, using either a no-code visual query tool, or SQL.
It provides an easy way to check table contents and get the insights you need to decide what to build next, including data you have [loaded through the Data Processing Engine](https://docs.dataplatform.ovh.net/dpe-actions-load.md) into the Lakehouse Manager, and data you reach through the entry points below.
## Finding tables in the Explorer
In the Explorer you can list and open tables in three ways:
- **Folders**: browse the folder hierarchy.
- **Datasets**: browse by dataset. A **Shared Datasets** category is also listed here when other projects in your organization have [shared datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md#cross-project-sharing-beta) with this project. Shared dataset tables can only be queried using the SQL editor.
- **Direct query**: when a data source has **direct query** enabled, it appears in this list. You can select it and explore tables **directly** in the Explorer, without importing or loading those tables into the Lakehouse Manager first.
The Explorer is a collaborative tool. You and your team can work on multiple tabs at the same time, and you can see the queries of your teammates by switching to their tabs. You can only edit the tabs that you created: your teammates' tabs are in read-only mode.
:::info
The Explorer only displays results in the form of a table. If you want to create charts and visualizations, use the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
:::
## How to explore your data using the visual builder
The visual builder interface offered in the Explorer is the same as the one used to build queries in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md). Discover how to use it on the link below.
:::info
[**Use the visual builder**](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md)
:::
## How to explore your data using the SQL editor
The SQL editor interface offered in the Explorer is the same as the one used to build queries in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md). Discover how to use it on the link below.
:::info
[**Use the SQL editor**](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md)
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-lineage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Trace how data flows through your project with lineage
## Objective
**Lineage** gives you a built-in, visual map of how data flows through your project: which jobs feed each [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md), which tables they produce in turn, and how everything connects from source to destination. The graph is built from the [OpenLineage](https://openlineage.io/) events your project records as jobs run, with nothing to install or configure.
Lineage is recorded **automatically** for Load and Aggregate actions (Python and PySpark), and is **opt-in** for Custom actions and notebooks through the SDK. Lineage for Trino queries (Explorer, Analytics Manager) is coming soon. See [what generates lineage events](#what-generates-lineage-events) below.
## Open the lineage view
There are two ways to reach the lineage of a table:
- From the **Lineage** section of the Lakehouse Manager: the page opens on a **search bar** where every table of your project is searchable. Start typing the name of the table whose lineage you want to see and select it to display its graph.
- From a **table's details view**: click the **Lineage** button to jump to the lineage page with that table already displayed.
## Explore the graph
The graph starts out from the table you selected: your **central node**, marked with a blue outline so you never lose track of where the graph begins. Around it, the graph shows the tables and jobs it connects to: the jobs and tables your table is built from **upstream**, and everything built from it **downstream**.
The graph is made of two kinds of nodes: **table nodes**, which list their columns, and **job nodes**, the runs that link tables together by reading and writing them.
:::info
The graph is never empty: a table without lineage yet is simply displayed on its own, without connections. Lineage appears as soon as a job run finishes, so a fresh run shows up immediately.
:::
### Table nodes
Click a table node to see the table's details: its **dataset**, its **columns**, the **producing job(s)** that write it, and the **consumer job(s)** that read it.
Clicking another table node shows its details but does not re-center the graph: to start out from a different table, select it from the **Lineage** search bar instead.
### Job nodes
Click a job node to see the run behind the connection:
| Detail | Meaning |
| :------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------- |
| **Job name** | The name of the job that produced the lineage event |
| **Kind** | **DPE** (an action run), **Lakehouse** (the automatic load when a table is built), or **SDK** (custom tracking emitted from your own code) |
| **Action type** | **Load**, **Aggregate**, or **Custom** |
| **Last run** and **status** | When the job last ran and how it ended |
| **Input tables** and **output tables** | What the job read and what it wrote |
For **DPE** and **Lakehouse** jobs, a direct link at the bottom of the details opens the job itself. **SDK** jobs have no link, since they are lineage you emitted yourself through the [SDK helpers](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md).
### Graph controls
The controls at the top right adjust what the graph displays:
- **Full graph** toggle: also shows the **sibling tables**, on top of the producers and consumers already displayed.
- **Compact nodes** toggle: a minimal view of the nodes, without their columns.
- **Zoom in**, **zoom out**, and **fit to screen**.
- **Depth**: how many upstream and downstream levels the graph displays, from **1** to **10** (default: **2**); increase it to follow the flow further.
## Column-level lineage
Beyond table-to-table flows, Lineage also tracks how **columns** connect across tables: for a given column, which upstream columns it is computed from, and which downstream columns are computed from it.
Click a column on a table node in the graph, or a column in the table details, to switch to its column lineage. When lineage is available, the graph connects the columns across tables to show how they relate. A column with no lineage of its own is shown as independent, with a blue outline marking it as solo.
:::info
Column-level lineage is not available while **Compact nodes** is enabled: switch the toggle off to click through columns.
:::
## What generates lineage events
| Source | Lineage |
| :--------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------ |
| **Load** and **Aggregate** actions (Python and PySpark) | Automatic, with column-level detail |
| **Custom** actions (Python and PySpark) | Opt-in: emit events from your code with the [SDK lineage helpers](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md) |
| **Notebooks** (Python and PySpark) | Opt-in, using the same [SDK lineage helpers](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md) |
| **Trino queries** (Explorer, [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)) | Coming soon |
## Use your own lineage tool
The same OpenLineage events can be forwarded to an external lineage solution such as [Marquez](https://marquezproject.ai/). Configure the [OpenLineage consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md) and schedule the [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) action to stream them to your backend.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Control data access with policy tags
## Objective
Policy tags enable fine-grained data access control by assigning key-value labels to datasets, tables, and attributes. Combined with [CEL conditions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions) in the Identity Access Manager, policy tags determine which users can read or write specific data.
## Table of Contents
1. [Introduction](#introduction)
2. [How Access Evaluation Works](#how-access-evaluation-works)
3. [Creating a New Policy Tag](#creating-a-new-policy-tag)
4. [Policy Tag Configuration](#policy-tag-configuration)
5. [Binding Policy Tags to Data](#binding-policy-tags-to-data)
6. [Next Steps](#next-steps)
## Introduction
Policy tags allow you to label datasets, tables, and attributes with key-value pairs that represent access categories (e.g., `sensitivity: high`, `department: finance`). These labels are then referenced in [CEL conditions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions) attached to IAM role bindings to control which users can access which data.
A policy tag consists of two parts:
- **Key**: A unique identifier (e.g., "sensitivity", "department").
- **Value**: A key can have multiple values (e.g., "high", "medium", "low").
:::warning
**Important**: Assigning a policy tag to a resource does _not_ automatically restrict access. You must also configure a [CEL condition](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md) on the user's IAM role binding that references the tag. Without a CEL condition, the tag has no effect on access control.
:::
## How Access Evaluation Works
When a user attempts to access data, the system evaluates their permissions through a **sequential gate chain**. Understanding this chain is essential for correctly configuring policy tags and CEL conditions.
### The Gate Chain
The system checks the user's [CEL condition](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions) at three resource levels, **in order**:
1. **Dataset**: Does the user's CEL condition pass for this dataset?
2. **Table**: Does the user's CEL condition pass for this table?
3. **Attribute**: Does the user's CEL condition pass for this attribute?
Each level must pass before the next level is evaluated. If any level fails, access is **denied immediately**: the system does not continue to the next level.
```mermaid
flowchart LR
D[Dataset check] -->|PASS| T[Table check]
D -->|FAIL| X1[DENIED]
T -->|PASS| A[Attribute check]
T -->|FAIL| X2[DENIED]
A -->|PASS| G[ACCESS GRANTED]
A -->|FAIL| X3[DENIED]
style G fill:#28a745,color:#fff
style X1 fill:#dc3545,color:#fff
style X2 fill:#dc3545,color:#fff
style X3 fill:#dc3545,color:#fff
```
:::warning
**Key insight**: The CEL condition is the **sole enforcement mechanism**. The system does not independently check policy tags. It only evaluates the CEL. If the CEL includes a pass-through for a resource level (e.g., `Resource == "attribute"`), that level's policy tags are **not enforced**. To require a tag at a given level, the CEL must explicitly check it. See [Writing CEL Conditions Manually](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#writing-cel-conditions-manually) for details.
:::
:::info
Policy tags at different levels are **cumulative requirements**, not overrides. If a dataset requires tag A and a table within it requires tag B, the user's CEL must check for A at the dataset level AND B at the table level. Tag B does not replace or override tag A.
:::
### Inheritance
If a resource does not have a policy tag explicitly assigned, it inherits the tag from its parent:
- **Tables**: If a table has no policy tag, it inherits the tag from its parent dataset.
- **Attributes**: If an attribute has no policy tag, it inherits the tag from its parent table (which may itself have inherited from the dataset).
- **Datasets**: Datasets are the top of the hierarchy and do not inherit from anything.
Inheritance only flows **downward**. A table's tag is never propagated upward to its parent dataset.
:::info
Inheritance means that tagging a dataset effectively tags all its untagged children. This is the simplest way to set a baseline access level.
:::
### Cumulative Access Requirements
Because each level in the gate chain is evaluated independently, tags at different levels **stack**. The example below uses one tag key, `access_level`, with values `level 1`, `level 2`, and `level 3` applied at the dataset, table, and attribute levels.
**Setup:**
- **Dataset:** `access_level: level 1`
- **Table** `chicago_calendar_full`: `access_level: level 2`
- **Attribute** `bridge` (in `chicago_calendar_full`): `access_level: level 3`
Other tables in the dataset have no explicit tags and inherit `level 1` from the dataset.
| User's tags | Can access dataset? | Can access `chicago_calendar_full`? | Can access `bridge`? |
| --------------------------------- | ------------------------------- | ----------------------------------- | ----------------------- |
| `level 1` only | Yes | No, needs `level 2` too | No |
| `level 1` + `level 2` | Yes | Yes | No, needs `level 3` too |
| `level 1` + `level 2` + `level 3` | Yes | Yes | Yes |
| `level 2` only | No, needs `level 1` for dataset | No | No |
| `level 2` + `level 3` | No, needs `level 1` for dataset | No | No |
:::warning
**Warning**: Tag values (`level 1`, `level 2`, `level 3`) have no inherent hierarchy or ordering in the product. They are labels you define. `level 2` does not automatically include `level 1`. Each value required at a resource level must be explicitly granted to the user.
:::
:::info
**Note**: A user who cannot pass the dataset gate cannot access _anything_ in that dataset, regardless of table- or attribute-level tags. Always ensure users have the dataset-level tag as a baseline.
:::
- **User\_A (`level 1` only)**: Can access untagged tables (they inherit `level 1`). Cannot access `chicago_calendar_full`: passes the dataset gate but fails the table gate (`level 2` required). Cannot access `bridge`.
- **User\_B (`level 1` + `level 2`)**: Can access untagged tables and `chicago_calendar_full`. Whether they can read `bridge` depends on their CEL: if the CEL checks tags at the attribute level, `bridge` is denied (`level 3` required). If the CEL uses a pass-through for attributes, `bridge` may still be accessible. See [Common CEL Patterns](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#common-cel-patterns).
- **User\_C (`level 1` + `level 2` + `level 3`)**: Has every tag value required in this setup, can access the dataset, `chicago_calendar_full`, and `bridge` when the CEL checks policy tags at each level (attribute pass-throughs are covered in the Important note below).
:::warning
**Important:** Whether attribute-level tags are enforced depends entirely on the CEL condition. To restrict `bridge`, the CEL must explicitly check `PolicyTags.access_level` at `Resource == "attribute"`. A pass-through like `|| Resource == "attribute"` will **skip** the tag check and allow access to all attributes.
:::
:::info
To verify this scenario with SQL in the Explorer, follow [Testing & Troubleshooting](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-testing.md).
:::
## Creating a new policy tag
To create a new policy tag, follow these steps:
1. Click on the **New Policy Tag** button.
2. Enter the **Key** (e.g., "department"), the first **Value** (e.g., "finance"), and a **Description** for the policy tag.
3. Save the policy tag and review it in the list of available tags.
## Policy tag configuration
After creating the policy tag, you can configure it by adding more values (e.g., tech, product) or updating its description in the `Tag Information` section. Policy tags should be configured thoughtfully to match the security needs of the data.
:::info
You can also view the configuration menu for the required policy tag by clicking on the **Edit** button.
:::
The `Associated Data Resources` section details the data resources – tables, datasets, and attributes – to which this policy tag has been applied.
## Binding policy tags to data
Policy tags can be bound to various data components:
### Datasets
1. Navigate to the **Datasets** section.
2. Select the dataset you want to apply the policy tag to.
3. Bind the appropriate policy tag to the dataset by selecting it from the available tags.
### Tables
1. Navigate to the **Tables** section.
2. Select the table you want to apply the policy tag to.
3. Bind the policy tag to the table by following similar steps as for datasets.
### Attributes
1. Navigate to the **Tables** section and select the desired table.
2. You can either click on the **POLICY TAGS** section or click on the **EDIT** icon for a specific attribute.
3. Locate the specific attribute (column) you want to apply the policy tag to or if you chose a specific attribute by clicking on the **EDIT** icon then you directly add the policy tag.
4. Bind the appropriate policy tag to the attribute. This might involve selecting from a dropdown or using a tagging interface.
## Next Steps
Now that you understand how policy tags work and how to bind them to data, continue with:
- [CEL Conditions & Access Patterns](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md): Grant Access UI, writing CEL conditions, common patterns
- [Testing & Troubleshooting](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-testing.md): Step-by-step access control tests, common errors
- [Example: Team Access with Groups](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-groups-example.md): Real-world walkthrough using groups for onboarding
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# CEL Conditions & Access Patterns
## Objective
This page covers how to grant users access to policy-tagged data using CEL (Common Expression Language) conditions. For an overview of policy tags, the gate chain, and how to bind tags to data, see [Policy Tags Overview](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md).
## Table of Contents
1. [Granting Access with CEL Conditions](#granting-access-with-cel-conditions)
2. [Best Practices](#best-practices)
## Granting Access with CEL Conditions
Granting a user access to policy-tagged data is a two-part process:
1. **Assign the policy tag value to the user** via the Grant Access UI in Lakehouse Manager
2. **Verify or edit the CEL condition** on the user's role binding in the Identity Access Manager
### Using the Grant Access UI
The Grant Access UI provides a quick way to assign policy tag values to users:
1. Open the **Grant Access** menu.
2. Select the policy tag you want to assign and specify the access type: **Read** for read-only access, or **Read and Write** for both.
3. Assign the policy tag values (e.g., `access_level: level 1`) to the user or group.
4. Confirm and save the access settings.
:::warning
**Important limitation**: The Grant Access UI generates a CEL condition without Resource-level scoping. This means the condition is evaluated at **every level** of the gate chain (dataset, table, attribute). If the policy tag is only applied at the table level, the dataset-level check will fail and the user will be denied access. **You must manually edit the CEL condition** in the IAM to add Resource scoping. See [Writing CEL Conditions Manually](#writing-cel-conditions-manually) below.
:::
:::info
**Note**: Users without **Advanced Data Access Control** permissions cannot read or write to any dataset, table, or attribute by default. This can be verified in the [IAM settings](https://docs.dataplatform.ovh.net/landing-page-iam.md) for the user.
:::
### Writing CEL Conditions Manually
To correctly scope policy tag access, you must edit the CEL condition on the user's role binding in the [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions).
The Grant Access UI generates a CEL condition like this:
```text
Service == "adac" && (PolicyTags.sensitivity == "high")
```
This condition checks `PolicyTags.sensitivity == "high"` at **every** resource level. If the dataset doesn't have `sensitivity: high` (only the table does), the dataset check fails and the user is denied.
To fix this, edit the CEL condition to scope the policy tag check to specific resource levels using `Resource`:
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
This condition means:
- At the **dataset** level: always pass (no tag check needed)
- At the **table** level: require `sensitivity: high`
- At the **attribute** level: always pass (no tag check needed)
To edit the CEL condition:
1. Open the [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions) and navigate to the user or group.
2. Find the ADAC role binding (the one with Service `adac`).
3. Click **Edit** on the condition.
4. Switch to the **CEL editor** and replace the generated condition with the Resource-scoped version.
5. Save.
:::info
Note the comment `// Permissions are managed in Lakehouse Manager. Do not update here!` on line 1. This is auto-generated by the Grant Access UI. You can safely replace the CEL content below it with your Resource-scoped condition.
:::
### Common CEL Patterns
Below are ready-to-use CEL patterns for common scenarios. In each pattern, replace tag keys and values with your own.
:::warning
**Critical:** Any resource level with a plain pass-through (e.g., `|| Resource == "dataset"`) has **no tag enforcement** at that level. Only use pass-throughs for levels where you intentionally want to skip tag checks. All patterns below list conditions in gate chain order: dataset → table → attribute.
:::
**1. Enforce tags at ALL levels (dataset, table, and attribute), recommended for full access control:**
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "attribute")
)
```
This checks tags at every level. A user with `level 1` + `level 2` can access the dataset, tables tagged up to `level 2`, and attributes tagged up to `level 2`. Attributes tagged `level 3` will be denied.
**2. Enforce tags at table level only (dataset and attribute pass-through):**
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
:::info
Use this when you only tag tables and don't need dataset-level or attribute-level restrictions. Note: attribute-level tags are **not enforced** with this pattern.
:::
**3. Grant access to tables that do NOT have a specific tag:**
```text
Service == "adac" && (
Resource == "dataset"
|| (!has(PolicyTags.sensitivity) && Resource == "table")
|| Resource == "attribute"
)
```
**4. Enforce at dataset level only (all tables and attributes pass through):**
```text
Service == "adac" && (
(PolicyTags.sensitivity == "high" && Resource == "dataset")
|| Resource == "table"
|| Resource == "attribute"
)
```
:::info
Use this when you only need to gate access at the dataset level. All tables and attributes within the dataset are accessible if the dataset gate passes.
:::
**5. Enforce at dataset and table levels (attribute pass-through):**
```text
Service == "adac" && (
(PolicyTags.department == "finance" && Resource == "dataset")
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
:::info
Attribute-level tags are **not enforced** with this pattern.
:::
**6. Enforce at attribute level only:**
```text
Service == "adac" && (
Resource == "dataset"
|| Resource == "table"
|| (PolicyTags.sensitivity == "high" && Resource == "attribute")
)
```
**7. Combine multiple tag values (user needs any of the listed values):**
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity in ["high", "medium"] && Resource == "table")
|| Resource == "attribute"
)
```
:::info
For more information on CEL syntax, operators, and the visual condition builder, see [Roles and Conditions, CEL Conditions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions).
:::
## Best Practices
- **Always tag at the dataset level first.** Since the gate chain starts at the dataset, an untagged dataset with tagged tables creates a confusing setup. Set a baseline tag on the dataset.
- **Use Resource-scoped CEL conditions.** Never rely on the default CEL generated by the Grant Access UI for production access control. Always edit the CEL to include `Resource` scoping.
- **Order CEL checks as dataset → table → attribute.** This matches the gate chain evaluation order and makes conditions easier to read and debug.
- **Keep tag structures simple.** Use a single tag key (e.g., `sensitivity`) with a small number of values (e.g., `public`, `internal`, `restricted`). Avoid deeply nested tag combinations.
- **Grant cumulative tags.** Remember that users need tags for every level in the chain. If a user needs `restricted` table access, they also need whatever tag is on the parent dataset.
- **Test with a dedicated user.** Before rolling out access policies, create a test user and verify each level of the gate chain independently. See [Testing Access Control](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-testing.md).
- **Audit CEL conditions regularly.** As policy tags change, CEL conditions on role bindings may become stale. Review them when modifying tag bindings.
- **Document your tag schema.** Keep a record of which tag keys and values are in use, what they mean, and which resources they're bound to.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-testing.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Testing & Troubleshooting
## Objective
This page covers how to test policy tag access control and troubleshoot common errors. For an overview of policy tags and the gate chain, see [Policy Tags Overview](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md). For CEL condition patterns, see [CEL Conditions & Access Patterns](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md).
## Table of Contents
1. [Testing Access Control](#testing-access-control)
2. [Troubleshooting and Common Errors](#troubleshooting-and-common-errors)
## Testing Access Control
This walkthrough uses the dataset from the [Getting Started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) and the `access_level` policy tag.
### Prerequisites
You need a user account with the following roles assigned:
1. **Organization Level**: `Organization Viewer`: Allows the user to view the platform and projects.
2. **Project Level**: `AM Admin`: Grants permissions to access the Explorer and run SQL queries.
### Setup
Bind the following policy tags:
- **Dataset level:** `access_level: level 1` on the entire dataset.
- **Table level:** `access_level: level 2` on the table `chicago_calendar_full`.
- **Attribute level:** `access_level: level 3` on the `bridge` attribute in `chicago_calendar_full`.
- **No Policy Tag:** The `stations_rides` table has no tag (it will inherit `level 1` from the dataset).
Verify the bindings in the policy tag menu:
### Test 1: User with `level 1` only
Grant the user `access_level: level 1` using the [Grant Access UI](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#using-the-grant-access-ui).
Then, edit the CEL condition in the IAM to enforce tags at **all three levels**:
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level == "level 1" && Resource == "table")
|| (PolicyTags.access_level == "level 1" && Resource == "attribute")
)
```
This CEL checks `level 1` at every level. Since the dataset has `level 1` and untagged tables/attributes inherit it, this user can access inherited resources but not `level 2` or `level 3` resources.
Open the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) in SQL mode as the test user.
**Show catalogs:**
```
SHOW CATALOGS
```
Expected: The dataset appears in the catalog list.
**Read an untagged table (inherits `level 1` from dataset):**
```sql
SELECT * FROM stations_rides
```
Expected: **Success**. The table has no tag, inherits `level 1` from the dataset, and the user's CEL matches `level 1` at all levels.
**Read the `level 2` table:**
```sql
SELECT * FROM chicago_calendar_full
```
Expected: **Error**. The user passes the dataset gate (`level 1`) but fails the table gate (table has `level 2`, CEL only checks for `level 1`).
### Test 2: User with `level 1` + `level 2`
Grant the user `access_level: level 2` in addition to `level 1`.
Update the CEL condition to include `level 2` at the table and attribute levels:
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "attribute")
)
```
This CEL enforces tags at all three levels. At the attribute level, it checks for `level 1` or `level 2`, but `bridge` has `level 3`, so it will be denied.
**Read the `level 2` table with SELECT \*:**
```sql
SELECT * FROM chicago_calendar_full
```
Expected: **Error**. The user passes the dataset and table gates, but `SELECT *` includes the `bridge` attribute which has `level 3`. The CEL only allows `level 1` or `level 2` at the attribute level.
**Read the table excluding the restricted attribute:**
```sql
SELECT date, public_holiday, christmas_holidays, weekend, sport_holidays, summer_holidays, month, week_day_label, autumn_holidays, spring_holidays, temperature, cloud_cover, humidity, week_day, wind_speed FROM chicago_calendar_full
```
Expected: **Success**. All selected attributes either have `level 1` or `level 2` tags (or inherit from the table), and the CEL matches.
**Read only the restricted attribute:**
```sql
SELECT bridge FROM chicago_calendar_full
```
Expected: **Error**: `bridge` has `level 3`, and the CEL only allows `level 1` or `level 2` at the attribute level.
### Test 3: Demonstrating that pass-throughs disable enforcement
To understand the difference between enforced and pass-through levels, try this CEL with the same user (`level 1` + `level 2`):
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| Resource == "attribute"
)
```
Note the **attribute level is now a pass-through** (`|| Resource == "attribute"`).
```sql
SELECT bridge FROM chicago_calendar_full
```
Expected: **Success**, even though `bridge` has `level 3` and the user doesn't have `level 3`, the CEL does not check tags at the attribute level. The pass-through means all attributes are accessible.
:::warning
This test demonstrates why pass-throughs must be used carefully. If you need attribute-level restrictions, the CEL **must** check tags at `Resource == "attribute"`.
:::
### Further testing
Try these variations:
- Grant `level 3` to the user (updating the CEL to include `level 3` at all levels) and verify `SELECT *` works on `chicago_calendar_full`.
- Remove `level 1` from the user's CEL and verify they **cannot access anything**, not even the `level 2` table, because the dataset gate requires `level 1`.
- Try accessing a table with only `level 2` (no `level 1`) to confirm the gate chain blocks at the dataset level.
## Troubleshooting and Common Errors
### User cannot access a table despite having the correct tag
**Most common cause:** The CEL condition is missing Resource scoping. The Grant Access UI generates a CEL without `Resource` constraints, so the tag check runs at every level of the gate chain. If the tag is only on the table (not the dataset), the dataset-level check fails.
**Fix:** Edit the CEL condition in the IAM to include Resource scoping. See [Writing CEL Conditions Manually](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#writing-cel-conditions-manually).
### User has the table-level tag but gets denied
**Cause:** The gate chain is cumulative. The user must also satisfy the dataset-level tag. Having only the table tag is not sufficient. The dataset gate must pass first.
**Fix:** Ensure the user's CEL covers all levels in the chain, or grant the user the dataset-level tag as well.
### `SELECT *` fails but selecting specific columns works
**Cause:** One or more attributes in the table have a higher-level policy tag that the user doesn't have. `SELECT *` includes all attributes, so the restricted attribute's gate fails.
**Fix:** Explicitly list the columns the user has access to, excluding the restricted attribute(s). Or grant the user the required attribute-level tag.
### User can see the dataset in `SHOW CATALOGS` but cannot query any tables
**Cause:** The dataset-level gate may pass, but every table in the dataset has tags that the user doesn't have.
**Fix:** Verify the user has tags matching both the dataset and the target table(s). Remember that untagged tables inherit the dataset's tag.
### CEL condition uses `PolicyTags.` but the resource has no such tag
**Cause:** If the CEL checks `PolicyTags.sensitivity == "high"` and the resource has no `sensitivity` tag at all, the check fails (the attribute doesn't exist).
**Fix:** Use `has(PolicyTags.sensitivity)` to check for existence first, or scope the check to specific resource levels with `Resource ==`.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-groups-example.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Example: Team Access with Groups
## Objective
This walkthrough demonstrates how to use [policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) together with [IAM groups](https://docs.dataplatform.ovh.net/iam-groups.md) and [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) to manage data access for team members. By assigning a CEL condition to a group's role binding, every user added to the group automatically inherits the correct access level.
## Table of Contents
1. [Scenario](#scenario)
2. [Step 1: Design the Tag Schema](#step-1-design-the-tag-schema)
3. [Step 2: Create and Bind Policy Tags](#step-2-create-and-bind-policy-tags)
4. [Step 3: Create the IAM Group](#step-3-create-the-iam-group)
5. [Step 4: Configure the CEL Condition on the Group](#step-4-configure-the-cel-condition-on-the-group)
6. [Step 5: Add Users to the Group](#step-5-add-users-to-the-group)
7. [Step 6: Verify Access](#step-6-verify-access)
8. [Extending the Setup](#extending-the-setup)
## Scenario
Your company has a dataset called **employee\_data** containing the following tables:
| Table | Description | Who should access |
| --------------------- | ----------------------------------- | ----------------- |
| `office_locations` | Office addresses, floor plans | Everyone |
| `team_directory` | Employee names, titles, departments | Everyone |
| `salary_details` | Compensation, bonuses, equity | HR only |
| `performance_reviews` | Review scores, manager notes | Management only |
The `salary_details` table has an attribute `national_id` that contains personally identifiable information (PII) and should only be accessible to senior HR staff.
**Goal:** New team members (newcomers) should be able to access `office_locations` and `team_directory`, but **not** `salary_details` or `performance_reviews`. HR staff should access salary data but not see `national_id` unless they are senior HR.
## Step 1: Design the Tag Schema
Create a policy tag key `access_level` with three values:
| Value | Purpose |
| ------------ | ------------------------------------------- |
| `general` | Baseline access. All employees |
| `hr` | HR-restricted data (salaries, benefits) |
| `management` | Management-restricted data (reviews, notes) |
Additionally, create a tag key `pii` with one value:
| Value | Purpose |
| ------------ | --------------------------------------------------- |
| `restricted` | Personally identifiable information, senior HR only |
## Step 2: Create and Bind Policy Tags
[Create the policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md#creating-a-new-policy-tag) `access_level` (with values `general`, `hr`, `management`) and `pii` (with value `restricted`).
Then [bind them to data](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md#binding-policy-tags-to-data):
| Resource | Tag |
| ------------------------------------------ | -------------------------- |
| **Dataset** `employee_data` | `access_level: general` |
| **Table** `salary_details` | `access_level: hr` |
| **Table** `performance_reviews` | `access_level: management` |
| **Attribute** `salary_details.national_id` | `pii: restricted` |
Tables `office_locations` and `team_directory` have no explicit tags. They inherit `access_level: general` from the dataset.
```mermaid
flowchart TD
DS["employee_data access_level: general"]
DS --> T1["office_locations inherits: general"]
DS --> T2["team_directory inherits: general"]
DS --> T3["salary_details access_level: hr"]
DS --> T4["performance_reviews access_level: management"]
T3 --> A1["national_id pii: restricted"]
T3 --> A2["other columns inherits: hr"]
style DS fill:#0089C0,color:#fff
style T1 fill:#e8f4f8
style T2 fill:#e8f4f8
style T3 fill:#f0ad4e,color:#fff
style T4 fill:#d9534f,color:#fff
style A1 fill:#d9534f,color:#fff
style A2 fill:#f0ad4e,color:#fff
```
## Step 3: Create the IAM Group
In the [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-groups.md), create three groups:
| Group | Purpose |
| --------------- | ----------------------------------------------- |
| **New Joiners** | New team members, general access only |
| **HR Team** | HR staff, access to salary data (excluding PII) |
| **Senior HR** | Senior HR, full access including PII |
For each group, assign the **ADAC Reader** role (or the appropriate role that includes Advanced Data Access Control read permissions).
:::info
For step-by-step instructions on creating groups and assigning roles, see [Groups](https://docs.dataplatform.ovh.net/iam-groups.md).
:::
## Step 4: Configure the CEL Condition on the Group
For each group's ADAC role binding, set the CEL condition in the [IAM](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions). Remember: conditions must follow the gate chain order, dataset → table → attribute.
**New Joiners**: access to `general` data only:
```text
Service == "adac" && (
(PolicyTags.access_level == "general" && Resource == "dataset")
|| (PolicyTags.access_level == "general" && Resource == "table")
|| (PolicyTags.access_level == "general" && Resource == "attribute")
)
```
This CEL enforces `access_level: general` at every level. New joiners can access `office_locations` and `team_directory` (which inherit `general` from the dataset) but not `salary_details` (`hr`) or `performance_reviews` (`management`).
**HR Team**: access to `general` + `hr` data, but NOT PII attributes:
```text
Service == "adac" && (
(PolicyTags.access_level == "general" && Resource == "dataset")
|| (PolicyTags.access_level in ["general", "hr"] && Resource == "table")
|| (PolicyTags.access_level in ["general", "hr"] && !has(PolicyTags.pii) && Resource == "attribute")
)
```
This CEL:
- **Dataset level:** requires `access_level: general`: passes for `employee_data`
- **Table level:** allows `general` or `hr`: passes for all tables except `performance_reviews` (`management`)
- **Attribute level:** allows `general` or `hr` BUT also requires the `pii` tag to **not exist**: denies `national_id` (which has `pii: restricted`)
:::info
The `!has(PolicyTags.pii)` check ensures that any attribute tagged with the `pii` key is denied, regardless of its value. This is useful for blanket PII protection.
:::
**Senior HR**: full access to `general` + `hr` data including PII:
```text
Service == "adac" && (
(PolicyTags.access_level == "general" && Resource == "dataset")
|| (PolicyTags.access_level in ["general", "hr"] && Resource == "table")
|| (PolicyTags.access_level in ["general", "hr"] && Resource == "attribute")
)
```
This CEL is the same as HR Team but **without** the `!has(PolicyTags.pii)` restriction. Senior HR can access `national_id`.
:::warning
To edit these CEL conditions, navigate to the group's role binding in the IAM and switch to the CEL editor. See [Writing CEL Conditions Manually](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#writing-cel-conditions-manually) for instructions.
:::
## Step 5: Add Users to the Group
When a new team member joins:
1. [Create their user account](https://docs.dataplatform.ovh.net/iam-users.md) in the IAM.
2. Assign them the required base roles:
- **Organization Level:** `Organization Viewer`
- **Project Level:** `AM Admin` (or the appropriate role for their work)
3. [Add them to the **New Joiners** group](https://docs.dataplatform.ovh.net/iam-groups.md#assign-users-or-service-accounts-to-a-group).
That's it. The group's CEL condition automatically grants them access to `general`-tagged data. No need to configure individual CEL conditions.
When they move to the HR team:
1. Remove them from **New Joiners**.
2. Add them to **HR Team**.
Their access updates immediately: they can now see `salary_details` (except `national_id`) in addition to the general tables.
## Step 6: Verify Access
Use the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) to verify access for a user in each group:
**As a New Joiner:**
| Query | Expected Result |
| ----------------------------------- | ---------------------------------- |
| `SELECT * FROM office_locations` | Success, inherits `general` |
| `SELECT * FROM team_directory` | Success, inherits `general` |
| `SELECT * FROM salary_details` | Error, table requires `hr` |
| `SELECT * FROM performance_reviews` | Error, table requires `management` |
**As HR Team member:**
| Query | Expected Result |
| ------------------------------------------------ | ---------------------------------------------- |
| `SELECT * FROM office_locations` | Success |
| `SELECT * FROM salary_details` | Error, `SELECT *` includes `national_id` (PII) |
| `SELECT name, salary, bonus FROM salary_details` | Success. No PII columns |
| `SELECT national_id FROM salary_details` | Error, `national_id` has `pii: restricted` |
| `SELECT * FROM performance_reviews` | Error, table requires `management` |
**As Senior HR:**
| Query | Expected Result |
| ----------------------------------- | ---------------------------------- |
| `SELECT * FROM salary_details` | Success, including `national_id` |
| `SELECT * FROM performance_reviews` | Error, still requires `management` |
## Extending the Setup
### Adding a Management group
To give managers access to `performance_reviews`, create a **Management** group with this CEL:
```text
Service == "adac" && (
(PolicyTags.access_level == "general" && Resource == "dataset")
|| (PolicyTags.access_level in ["general", "management"] && Resource == "table")
|| (PolicyTags.access_level in ["general", "management"] && Resource == "attribute")
)
```
### Users in multiple groups
A user can belong to multiple groups. If someone is in both **HR Team** and **Management**, they get the union of both groups' permissions. They can access both `salary_details` and `performance_reviews`.
:::info
When a user belongs to multiple groups, each group's role binding is evaluated independently. If **any** role binding's CEL condition passes, the user is granted access. This makes multi-group setups additive.
:::
### Service accounts
[Service accounts](https://docs.dataplatform.ovh.net/iam-service-accounts.md) can also be added to groups. If you have a reporting pipeline that needs to read general data, add its service account to the **New Joiners** group rather than configuring individual CEL conditions.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-row-filters.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Restrict which rows a user can see with row filters
## Objective
**Row filters** decide which rows of a [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) a given user or group can see. The table stays one table: same name, same columns, one copy of the data. Two people querying it can get different results, because each of them carries their own filters.
A filter that says _"Marie only sees rows where `country = 'France'`"_ means that when Marie queries that table, France rows come back and everything else is simply absent. She gets no error, no warning, and no notice that rows were hidden: the table looks to her like a smaller table. The counts, aggregates and joins she runs are computed on her visible rows only. Nobody else's view changes: a colleague with no filter on that table still sees everything.
One filter covers one table and applies to the users and groups you assign to it. You will find them in the Lakehouse Manager, in the **Row filters** section, which lists every filter in the project.
:::warning
Row filters are enforced on the **Trino query path**, which is what [Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) queries and dashboards and [Trino consumers](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) use. Someone reading the same table another way is **not filtered and sees every row**: a [notebook](https://docs.dataplatform.ovh.net/dpe-notebooks.md) using PySpark or PyIceberg, or direct catalog and object storage access.
Row filters are therefore a control over how data is consumed through SQL, not a substitute for withholding access to the underlying storage. Anyone whose access must genuinely be limited must not hold direct storage credentials for that table. Tell the teams running notebooks or ETL against a table before you filter it, so nobody assumes a protection that is not there on their path.
:::
## Instructions
### Create a row filter
Go to the _Row filters_ section in the Lakehouse Manager and click **+ New filter**. The modal has two steps: the filter itself, then the people it applies to.
Step 1 asks for three things:
- a **Filter name**, so the list reads as intent rather than as a list of expressions,
- the **target table**, picked as a [dataset](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) then a table in it,
- the **filter condition**: rows not matching it will not be visible to the assigned users and groups.
You have two ways to write the condition: the **builder mode**, which is the default, and the **SQL mode**.
### Build a condition in builder mode
Builder mode is the simple mode, and the one to use unless you need something it cannot express. Pick a column from the dropdown (its type is shown next to its name), pick an operator, and enter the value. The **generated SQL** appears below the condition as soon as the value is entered, so you always see the condition you are actually saving.
Click **+ Add condition** for a second line. From two conditions on, you choose how they combine: **Match ALL conditions (AND)** or **Match ANY condition (OR)**. That choice applies to every condition in the filter, so builder mode gives you one AND group or one OR group, never a mix of the two. A condition that needs both, or any other nesting, belongs in SQL mode.
Which operators you can pick depends on the column's type:
| Column type | Available operators |
| ----------------- | ---------------------------------------------------------------------------------------- |
| **Text** | equals, not equals, like, is one of, is not one of, is empty, is not empty |
| **Number** | equals, not equals, `<`, `≤`, `>`, `≥`, is one of, is not one of, is empty, is not empty |
| **Date and time** | same as number: ordering a date is as meaningful as ordering a number |
| **Boolean** | equals, is empty, is not empty |
**Empty values** have their own operators. You cannot find blank rows by comparing a column to an empty value, and the platform tells you so rather than saving a filter that quietly matches nothing. _Is empty_ and _is not empty_ split a table exactly: their row counts always add up to the total.
**Values are checked against the column as you write them.** Compare a date column against `01/02/2024` and you are told immediately what format that column needs, instead of saving a filter that looks fine and then breaks every query the restricted person runs. Dates want `2024-01-01`, timestamps want `2024-01-01 10:30:00`, times want `09:30:00`, and a value carrying a time zone is refused rather than silently shifted: send the local wall-clock value the column actually holds.
A condition has room for:
| Limit | Value |
| ------------------------------ | ----- |
| Comparisons in one condition | 200 |
| Values in a single _is one of_ | 100 |
| Characters in a text value | 500 |
### Write a condition in SQL mode
For conditions the builder cannot express, click **Switch to SQL mode** and write the condition directly as a SQL expression, for example `country = 'France' AND lower(dept) = 'sales'`. Whatever you built in the builder is carried over, so the usual path is to start in the builder and switch when you hit its limits.
This is a plain condition: no `WHERE`, no semicolon, no statement. It is checked before it is saved, and it is refused if it references another table, uses a subquery, contains a comment or a statement separator, or calls a function outside the allowed list. The allowed functions are the cheap per-row ones: text handling, `coalesce`, `cast`, date truncation and similar. Aggregates, regular expressions and JSON functions are not available, because a row filter runs on every row of every query against that table and has to stay cheap.
Column names behave the way they do in SQL: unquoted names are case-insensitive, and you quote a name to reach a column whose case actually matters.
:::warning
**Switching back from SQL mode to builder mode discards your SQL.** The builder cannot represent every condition SQL can, so it starts from an empty condition rather than from an approximation of yours. The platform warns you before it happens.
:::
### Assign the filter and check its impact
Click **Next** for step 2. It opens with a **filter summary**: the name, the dataset, the table and the condition, exactly as they will be saved.
Below it, **Impact** reports what the condition does to that table right now, for example _"227,712 of 528,363 rows would be visible with this filter, 300,651 rows would be hidden"_. It is counts only, never row data, so it is safe to look at on a table whose contents you should not be reading. Read it before you assign anyone: it is the cheapest way to catch a condition that keeps nothing, or everything.
Then pick the **principals**, the users and groups this filter applies to. A filter can name several of each. Group membership is resolved every time the filters are published, not when you write the filter, so someone added to a restricted group next month is filtered from then on with no need to touch the filter.
Finish with **Save and activate**, or with **Save as inactive** to park the filter until you want it to bite.
### How several filters combine
If more than one filter applies to the same person on the same table, **all of them apply at once, and every one of them must be satisfied** for a row to be visible. Filters never widen what someone sees: adding a second one can only ever take more rows away.
This is the most common source of surprise. Two filters that each look reasonable, _"EMEA only"_ and _"North America only"_, intersect to nothing and the person sees an empty table. Being targeted twice by the same condition, named directly and through a group you belong to, is harmless: the same condition applied twice hides the same rows.
### Manage filters day to day
The _Row filters_ list covers the whole project and shows, for every filter, its active toggle, name, principals, dataset, table and condition, with **edit**, **duplicate** and **delete** actions.
- **Deactivating** a filter with the toggle stops the restriction from applying and keeps the filter on the list, ready to be switched back on.
- **Editing** a filter changes its condition or its principals.
- **A filter covers one table.** To apply the same condition to another table, duplicate the filter and change the target table on the copy.
Row filters are part of the project's configuration, so they travel with a [configuration export](https://docs.dataplatform.ovh.net/tutorials-export-import-config.md) and can be imported into another project alongside the datasets they protect.
### Permissions
Two things are gated separately, and they do not imply each other: managing filters says something about the policy, while the impact count says something about the data.
| To do this | You need |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| See, create, edit, duplicate and delete row filters | The _Row filter_ permissions (read, write, bind, delete) in the [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-roles-conditions.md), listed under _Data Manager_ |
| Read the impact count on a table | Read access to the dataset |
### Good to know
- **Saving is not instant.** Filters are published to the query engine, which picks them up on its own cycle, so a new, changed, deactivated or deleted filter takes a few seconds before queries reflect it. If you save a filter and immediately query as the restricted person, you may still see unfiltered rows: retry rather than concluding the filter did not work.
- **A filter that cannot be published restricts nobody.** If a filter targets a group that can no longer be resolved, the platform cannot publish a restriction for it and the targeted people see every row. The filter still appears in the list and still reads as active. When a restriction genuinely matters, confirm it by querying as one of the restricted people rather than by reading the list.
- **A column dropped after the filter was saved is not covered.** On save, the columns a condition names are checked against the table as it physically exists, so a column the catalog knows about but the table does not is refused, and re-syncing the table is the fix. A column dropped later is not re-checked on its own: opening the filter recomputes its impact, which is what surfaces the problem.
### Troubleshooting
| Symptom | Where to look |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| "I see no rows at all" | Check every filter reaching that person on that table. Several filters combine narrowly, and two sensible ones can intersect to nothing. |
| "I see no rows, and only one filter applies" | The condition is being applied correctly and no row satisfies it. That is a data question, not a filter question. |
| "The filter did not apply" | Wait a few seconds and retry: publication is not instantaneous. |
| "The filter still does not apply" | Confirm the person is reading through Trino. A notebook using PySpark or PyIceberg, or direct storage access, is not filtered. |
| "The filter is active but nobody is restricted" | Its principals may no longer resolve. Verify by querying as one of the restricted people. |
| "It saved, then their queries started failing" | The condition may name a column that has since been dropped. Open the filter to re-check it. |
| "Someone unrestricted sees fewer rows than expected" | Row filters only apply to the people a filter names. Look at their dataset permissions instead. |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-dpe.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Processing Engine
## What is the Data Processing Engine?
The Data Processing Engine builds and runs the data pipelines of a project, from raw sources through to the tables analysts query. It covers extraction and loading from a registered source, transformation of data already held in the project, and the scheduling and triggering around both.
Four objects make up the model:
- An [action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) is one unit of work: load a table, aggregate it, run SQL, run custom Python or PySpark.
- A [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) chains actions into stages with a given execution order.
- An [environment](https://docs.dataplatform.ovh.net/dpe-environments.md) is a reusable set of execution settings, covering resources, perimeter, segmentation and variables, shared across actions and workflows.
- A [job](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md) is one execution of an action or a workflow.
Execution is job-based rather than long-running. When an action or workflow is triggered, the platform builds a worker with the resources requested, runs the job, then shuts the worker down, so compute is billed for the execution only. An always-up execution mode is available for workloads that cannot wait for provisioning.
Every job runs as a [service account](https://docs.dataplatform.ovh.net/iam-service-accounts.md), and that identity's [IAM roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) decide what the job can read and write. [Notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md) are the interactive counterpart: Jupyter instances for exploring data before its logic becomes an action. For PySpark jobs, the [Spark History Server](https://docs.dataplatform.ovh.net/dpe-spark-history-server.md) keeps the Spark UI of past runs available, so a finished job can still be inspected stage by stage.
[### Actions→
One unit of work: load, transform, export or clean up.
](/landing-page-dpe-actions)[### Workflows→
Chain actions into stages with an execution order.
](/dpe-workflows)[### Environments→
Reusable execution settings shared across actions.
](/dpe-environments)[### Jobs→
Every execution, its state, resources and history.
](/landing-page-dpe-jobs)[### Notebooks→
Jupyter instances for exploring data interactively.
](/dpe-notebooks)[### Spark History Server→
Inspect a finished Spark run and its DataFlint analysis.
](/dpe-spark-history-server)[### Settings→
Engine-level defaults and quotas.
](/dpe-settings)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Processing Engine actions
## What is an action?
An action is one unit of work in the Data Processing Engine: read from somewhere, do something, write somewhere. Around twenty action types cover the common cases, from loading a registered source into a Lakehouse table to running arbitrary Python or PySpark.
They group by what they do. **Loading** brings data in: Load, Load PySpark, Load to Bucket, File Transfer, CSV to PostgreSQL. **Transformation** reshapes data already held in the project: Aggregate, Aggregate PySpark, SQL, Custom, Custom PySpark, Diff. **Export** sends data back out: Export Table to Source, MySQL to Parquet, PostgreSQL to CSV, PostgreSQL to Parquet. **Housekeeping** keeps the project tidy: Delete, Delete Diff, Delete Bucket, Rebuild, Maintenance actions, Send OpenLineage Events.
Every action is configured by a JSON document. The graphical editor writes that document for you, and advanced mode edits it directly, which is where fields with no UI equivalent live. Load, Aggregate and [Workflows](https://docs.dataplatform.ovh.net/dpe-workflows-advanced-mode.md) each document their JSON separately.
Actions are held in repositories, and the repository is what gets versioned, either through the platform's own version control or by syncing the repository with Git. An action always runs inside a [job](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md), under the preferences and resources set on the action, on its workflow, or on a shared [environment](https://docs.dataplatform.ovh.net/dpe-environments.md).
Creating an action, versioning a repository and syncing it with Git are covered together in [Create and manage actions](https://docs.dataplatform.ovh.net/dpe-actions-manage.md).
[### Create and manage actions→
Create one, version the repository, sync it with Git, and know what triggers a rebuild.
](/dpe-actions-manage)[### Load→
Extract, map and load data from a source into a Lakehouse Manager table.
](/dpe-actions-load)[### Load PySpark→
The same load, distributed with PySpark for large datasets.
](/dpe-actions-load-pyspark)[### Load to Bucket→
Load raw data straight into an object storage bucket.
](/dpe-actions-load-to-bucket)[### File Transfer→
Copy files between buckets, or from a source to a bucket, keeping the path structure.
](/dpe-actions-file-transfer)[### CSV to PostgreSQL→
Ingest a CSV file from any source into an existing PostgreSQL table.
](/dpe-actions-csv-to-postgresql)[### Aggregate→
Read a table, apply transformations, write the result to another table.
](/dpe-actions-aggregate)[### Aggregate PySpark→
Distributed aggregation between Lakehouse Manager tables.
](/dpe-actions-aggregate-pyspark)[### SQL→
Run SQL against datasets and tables on Trino, including DML and DDL.
](/dpe-actions-sql)[### Custom→
Run your own Python for ETL or validation logic.
](/dpe-actions-custom)[### Custom PySpark→
Run your own PySpark code on a Spark cluster.
](/dpe-actions-custom-pyspark)[### Diff→
Build a change log between datasets over time.
](/dpe-actions-diff)[### Export Table to Source→
Push tables back out to a compatible external source.
](/dpe-actions-export-table-to-source)[### MySQL to Parquet→
Transfer a MySQL table into a bucket as Parquet files.
](/dpe-actions-mysql-to-parquet)[### PostgreSQL to CSV→
Transfer a PostgreSQL table into a bucket as CSV files.
](/dpe-actions-postgresql-to-csv)[### PostgreSQL to Parquet→
Extract from PostgreSQL and store the result as Parquet.
](/dpe-actions-postgresql-to-parquet)[### Delete→
Delete an object entirely, such as a table.
](/dpe-actions-delete)[### Delete Diff→
Delete rows that no longer exist in a differential comparison.
](/dpe-actions-delete-diff)[### Delete Bucket→
Remove files, folders or entire contents from a bucket.
](/dpe-actions-delete-bucket)[### Rebuild→
Rebuild a table to change constraints, repartition and clean metadata.
](/dpe-actions-rebuild)[### Maintenance actions→
Refresh project metadata and flush component caches.
](/dpe-actions-maintenance)[### Send OpenLineage Events→
Forward lineage events to an external OpenLineage backend.
](/dpe-actions-send-openlineage-events)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-manage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create and manage actions
## Objective
Creating an action, keeping several versions of it, and syncing those versions with Git are one workflow rather than three. This page covers that workflow, plus what triggers a rebuild and what is not possible today.
Which action type to create is a separate question, answered on the [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) page.
## Create an action
When a repository is empty, the first thing you'd like to do is **create an action**. From the main page, click the _New action_ button on the top right of the screen. You will then be prompted to choose the type of action you wish to perform from the Platform Store.
:::info
Note that editing actions is always on autosave mode. Use [versions](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#versioning-repositories-of-actions) if you don't want to tamper with your deployed production actions.
:::
### Advanced mode
Actions in the Data Processing Engine are all configured using a set of inputs specified via a JSON configuration file. There are 2 ways for users to update an action's inputs:
- either using the graphical user interface
- or using an **advanced mode** which lets you directly edit the JSON configuration file
Using the "advanced mode" offers a bit more flexibility and options which might come in handy for advanced users.
:::info
Note that the advanced mode is also very practical for [Custom actions](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) as it offers an IDE-like experience in the web. This will allow you to code custom Python script directly inside of the platform without managing files locally on your computer.
:::
## Manage actions
As your Project matures, structure and organization becomes increasingly important. To facilitate this, the Platform provides several methods to better organize and manage your actions:
- [Repositories](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#versioning-repositories-of-actions): Allow compartmentalizing your actions into different sections that can be versioned
- [Versions](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#versioning-repositories-of-actions): the platform offers an online versioning experience for each repository of actions individually. The platform has its own version control system, but you can also synchronize each repository with Git
- **Descriptions & Tags**: Every action can be assigned a _description_ and/or _tags_. This allows you to document your Project, particularly useful when collaborating in large teams.
- **Lineage**: Every action contains a lineage sub-tab, accessible when editing an action. It displays every [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) that contains the action. Workflows allow you to orchestrate the execution of multiple actions and launch them in a specific order using stages.
### Versioning repositories of actions
Editing actions is always on autosave mode. When you are trying to iterate on existing actions, it is recommended to create several versions of the actions' code. This lets you incrementally introduce variations to experiment and test your work, while keeping the version in production always intact.
**On the platform, this versioning happens at the level of the repositories**. The repositories are the tabs you see at the top of the Actions' tree view.
You cannot version an individual action, but rather a whole repository of actions. The versioning panel can be accessed at the top right of the repository header:
:::info
This functionality is also available on other components that implement repositories, such as [Analytics Manager queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#versioning-queries). It works in the same way, just look for the equivalent UI elements in the Queries screen.
:::
There are 2 versioning systems on the Platform:
- [Data Platform version control system](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#use-the-platform-version-control-system): the default versioning system on all repositories
- [Git versioning](https://docs.dataplatform.ovh.net/dpe-actions-manage.md#use-a-repository-linked-to-git): if you decide to synchronize your repository with Git
Now let's see how all of this works in practice!
### Use the Platform Version Control System
By default, each repository of actions on the platform has an integrated version control system, allowing you to manually manage different versions of the code (i.e., actions) they contain.
Repositories now have:
- a **deployed version**, which is the version served when [executing the job](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md).
- a **latest version**, which is the current version being edited through the editor panel.
:::info
If your repository has _only one version_, that version is both the deployed and editable version, meaning changes affect production immediately.
:::
If your repository has _two or more versions_, only the latest version can be edited. All other versions, including the deployed version, will be in read-only mode.
To create a new version for the repository, open the versioning panel at the top right and click on the **+** icon.
Choose which version to duplicate to create the new version.
After creating the new version, the _deployed version_ will not change. You are able to edit the new version directly.
Click on the **Play** button to set a new version as the deployed version: the version of the code used when actions of the repo are executed.
### Use a repository linked to Git
Each repository can be linked to an external Git repository in which case versions are synced with Git commits. This allows you to update & test actions continuously without affecting the version deployed in production.
#### Linking a repository to Git
To link your repository on the platform to Git, you must click on the gear icon to edit an existing repository or create a new a one. Either of these actions will open the repository configuration window.
Click on _Connect to Git_ to expand the window.
Here the first thing you need to enter is the Repository SSH URL.
It is generally found on repository cloning options of common Git solutions. On GitHUB, for instance, it can be found on Code > Clone > SSH.
Next click on _FETCH_ to get the distant branches and select the branch you desire to use.
Finally, copy the Public SSH key given at the bottom and paste it in the list of SSH keys associated to your account on your Git solution. For GitHUB, you can follow [this tutorial](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account).
#### Committing and pushing
After changing an object in a repo, your repo will display the uncommitted changes as shown in the image below. You can commit and push them by clicking on the arrow pointing upwards indicated in the image.
:::info
If there is a conflict when you push, your changes will not be pushed to the branch you configured, instead they will be pushed to a new branch with same name of the current branch and a suffix of _\_conflictN_, where N is the number of the conflict. For example, if your branch is named _current-branch_ and your push causes a conflict, the new branch will be named _current-branch\_conflict1_. If there is a conflict again when you push, the new branch will be named _current-branch\_conflict2_.
:::
#### Pulling and merging
Pull distant changes by clicking on the arrow pointing downwards indicated in the image, a popup window will prompt you to confirm your decision. All uncommitted changes will be lost and both branches will be merged together.
## Organising and sharing code across actions
An action is a single `.py` file with one entry function (`customfunc(event)` by default), added through drag & drop or the advanced IDE. There is **no multi-file package structure inside a single action**.
To reuse code across actions, use one of the documented paths:
- **Reference a Git repository** in the _Python Requirements_ field:
```
git+https://github.com/OWNER/REPO.git@
git+https://github.com/OWNER/REPO.git@latest
```
- **Version your whole actions repository through Git**: [link it over SSH](#linking-a-repository-to-git), fetch a branch, and paste the public SSH key on the Git side. This is the intended mechanism for sharing code between actions.
:::info
There are **no global Python variables** shared between actions: each job runs in a fresh container that shuts down when the job finishes. Reusable values are passed through **Environments** (environment variables), which can be assigned to several actions and workflows at once.
:::
## Triggering a Notebook from an action
Triggering a Notebook from a Custom action is **not supported**. Notebooks are Jupyter instances with their own manual Start/Stop lifecycle, and the documented flow is the reverse one: converting a notebook into an action.
**Recommended pattern:** convert the notebook code into a Custom action, which can then be orchestrated in a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) or through an API call.
## Managing actions programmatically (CI/CD)
There is **no dedicated REST endpoint** for creating, updating or deleting actions. The official approach is [Git synchronisation](#use-a-repository-linked-to-git): link the actions repository to a Git repository over SSH, then commit and push from the platform, or pull changes made outside of it.
For CI/CD, this means that changing an action is a **commit into the linked Git repository**, followed by a pull from the platform. [Workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) can be run manually or through an API call, so triggering is available even though the CRUD of actions goes through Git.
## Build behaviour
- **What triggers a build.** Worker builds are triggered by **dependency changes**: updating dependencies redeploys the environment. For Git dependencies, click **Force Build** to reinstall.
- **No image cache.** Pre-built workers and image caching for actions are not available today (unlike the preconfigured Notebook images: Base, Data Science, PySpark, TensorFlow).
- **Autosave and versions.** Editing an action is always in autosave mode. To avoid affecting production, create a new [version](#versioning-repositories-of-actions): it becomes the editable _latest_, while the _deployed_ version stays frozen.
:::info
To keep rebuilds to a minimum, factor shared dependencies into a reusable **Environment** instead of redeclaring them per action.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-load.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Load raw source data into your warehouse with the Load action
## Objective
:::info
This article is about actions that use Data Platform's proprietary Python data processing engine. To use Apache Spark clusters, see [Load PySpark action](https://docs.dataplatform.ovh.net/dpe-actions-load-pyspark.md).
:::
The _Load_ action **extracts raw data from sources and inserts it into primary tables** in your data warehouse. It is usually the first step in any data processing system, necessary before running any other processes on data coming from connected sources.
:::info
In other terms, if you've [set-up a connector through the Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md), the Project has the information to connect to the source as well as its metadata, but won't contain the actual data until it is actually loaded using the _Load_ action.
:::
## Configure a Load action
To create a new _Load_ action, navigate to the _Action_ tab of the Data Processing Engine (DPE), click on **New Action** and select the _Load_ action.
### Choose a source and destination
To set-up the _Load_ action, start by selecting a **source** from the list that you have already set-up in the [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) tab of the Connectors.
Then a **destination** table to send your data to. Similarly to the sources, the tables have to be configured ahead of time in the [Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) tab of the Lakehouse Manager. Make sure you've [built](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#build-table) your data model otherwise the tables won't be accessible by other components in the platform.
### Map attributes
Once the source & destination have been selected, the system will **automatically map** the destination attributes of your destination table with the different columns that it has identified in the raw data source.
:::info
On the picture below, you can see that the _date_ attribute has a a value of '1' beside the notepad icon. This indicates the number of **data validation rules, also called blueprint rules**, defined in the Lakehouse Manager, that will be automatically applied when loading the data. [Blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md) allow users to quickly filter out invalid data like nulls / empty cells or reformat values such as date or currencies.
:::
Note that you might want to **insert raw text** directly inside your destination table. This might come in handy for instance when you want to count the number of data points or tag a certain source with a label that will be later used in an aggregate table. To do that, select _\< txt >_ which will set the source attribute to a raw text input field.
### Configure the action's preferences
Once you're satisfied with the mapping of your attributes, navigate to the [preferences tab to configure the action settings](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
When you are done editing your action, click on **Create** to validate the action.
## Advanced Mode
If you need to access the JSON configuration file of the action, you can activate the Advanced mode by clicking on **Advanced** at the top of the page.
[Use the Advanced mode of a Load action](https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Advanced mode for the Load action
## Objective
Note the overall schema is very close to the Aggregate action though with some limitations on certain options. Keep in mind that Load actions are designed to simply load the data from raw sources to Data Platform while Aggregate actions are made to run operations on tables contained inside the Data Platform data warehouse.
In broad terms, the Load action is a simplified version of the Aggregate action, aimed at providing an easy way to load your data. For full documentation of Aggregate action, please refer to the [Aggregate action's advanced documentation](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md).
The JSON configuration file for an Load action is composed of higher level objects which are mandatory. This includes:
- [**Sources**](#source): Defined the source tables used to load the data from.
- [**Schema**](#schema): Defines the individual mapping between the destination table and the source tables.
- [**Destination**](#destination): Defines in which table we will be loading the output data from the Load action.
Other optional parameters can be added as well, we will cover that in specific section [**Other optional parameters**](#other-optional-parameters) which include for instance the segmentation, perimeter or transformation rules.
## Source
**Format**: The JSON contains a string value which points to the source and optional additional parameters. The source path looks like that: `dwh/database_name/table_name`, where `database_name` is the name of the source location (source or coming from Lakehouse Manager) and `table_name` the name of the source object or the object in the schema of Lakehouse Manager.
**JSON key**: `params.load_from[0].source` or `params.load_from.source`
Example:
```json
"load_from" : [
{
"source" : "dwh/default_dataset/prim_calendar",
"force_utf8" : true
}
]
load_from" : {
"source" : "dwh/default_dataset/prim_calendar",
"force_utf8" : true
}
```
:::tip
For SQL-like sources : You can alias tables by adding a space after the table\_name then the character(s) for the table alias. For instance: "source": "dwh/mysql\_source/lifetime\_insights fb\_i". This will alias the `lifetime_insights` table by `fb_i`. So you can reuse this alias in joins and schema.
:::
## Schema
The schema determines the renaming between the name of the fields as present in the source, to the naming that these fields must have in the destination.
**Format**: the schema object contains a series of JSON objects, in which each key is the name of an attribute of the destination, and the value contains the parameters associated with the attribute.
**JSON key**: `params.schema`
Example:
```json
{
"schema": {
"month": {
"table": "ref_month",
"operation": "SELECT",
"type": "default",
"attribute": "month"
},
"vegetables": {
"type": "sql",
"sql": "CONCAT(\"HELLO : \", prim_calendar.vegetables)"
},
"season": {
"type": "replace",
"value": "summer"
}
}
```
- `schema.keys()` the destination attributes
- `type` the mapping mode of each attribute
There are 3 different types of mapping:
- type:`default` : to map one attribute to another without any transformation
- - `table` the original source object of the attribute
- - `attribute` source attribute to map from
- - `operation` represents the SQL operation types mentioned in SQL operation section. (only for sql-like sources).
- type:`replace` : will put a hard-coded value in all destination rows.
- - `value`: this attribute will be filled with the value for all destination rows about to be inserted/updated
- `type`:"sql" : the following sql formula will be applied (only for sql-like sources).
- - `sql` the value is an SQL statement.
## Destination
Destination represents the location where action data will be injected.
**Format**: Its path looks like this: `dwh/database_name/table_name`. Where database\_name is the name of the destination database (typically `default_dataset`) and `table_name` the name of the destination table in the Lakehouse Manager schema.
**JSON key**: `params.load_to[source]`
Example:
```json
"load_to": {
"source": "dwh/default_dataset/tmp_calendar",
"force_utf8": true
}
```
Note that you can fill in multiple destinations at once:
```json
"load_to": [
{"source": "dwh/default_dataset/prim_facebook"},
{"source": "dwh/default_dataset/tmp_facebook"},
]
```
## Other optional parameters
### Extract\_chunk\_size
`extract_chunk_size` is an integer value which represents the maximum allowed size at once for extracting data.
**JSON key**: `params.load_from[0].extract_chunk_size`
Example:
```json
{
"params": {
"load_from": [ { "extract_chunk_size": 5000 , ... } ]
}
}
```
This will allow the worker to extract data with batches of 5000 lines.
### Load\_batch\_size
`load_batch_size` is an integer value of the maximum allowed size at once for loading data into database
**JSON key**: `params.load_batch_size`
Example:
```json
"load_batch_size": 5000
```
This will allow the worker to load the data with batches of 5000 lines
### Source\_rules
`source_rules` represents the rule for simple data preparation. This setting will override the existing rules synchronized from DWH. They will be applied to attributes from the source. For instance let's assume we would like to apply transformation rules to the attribute `date_source`.
**Format**: A list contains JSON objects, where each represents a rule applied to an attribute.
**JSON key**: `load_from.source_rules`
Example:
```json
"source_rules": [
{
"action": "date_replace",
"action_values": [
"%d/%m/%Y %H:%i:%s",
"%Y-%m-%d %H:%M:%S"
],
"attribute_name": "date_source",
"name": "correct date format"
}
]
```
For detailed configuration documentation, please refer to the article on [blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md)
### Rules
Rules are similar to source\_rules except that they will be applied to the destination attribute instead of the extraction attribute. This allows you to map different destination attributes to the same source attribute but assign it different rules.
**Format**: A list contains JSON objects, which each represents a rule applied to an attribute.
**JSON key**: `rules`
Example:
```json
"rules": [
{
"action": "date_replace",
"action_values": [
"%d/%m/%Y %H:%i:%s",
"%Y-%m-%d %H:%M:%S"
],
"attribute_name": "date_dest",
"name": "correct date format"
}
]
```
### Handling JSON/XML semi structured mapping
in order to handle the mapping of semi-structured data,
you have to indicate which is the node to loop on in order to structure the data in a normalized way.
for instance if your source looks like this:
```json
{
"data": [
{ "id":1,"name":"marco" },
{ "id":2,"name":"polo" },
]
}
```
you have to add in your load configuration in the "params" section :
```json
{
"params":{
"dataXPATH":"data",
...
}
}
```
### More options on structured CSV/XLSX files
Sometimes excel and csv files doesn' begin at the very first row.
there are parameters to set in order to
#### encoding
specify the character encoding of the CSV file (default auto-detect).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"encoding":"utf-8",
...
}]
}
}
```
#### header
indicates if the file has a header as first row (default: true).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"header":true,
...
}]
}
}
```
#### header\_offset
indicates how many rows in your file source the DPE needs to skip before the file starts (default 0).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"header_offset":1,
...
}]
}
}
```
#### offset
indicates how many rows are empty between your header and the data rows in your source file (default 0).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"offset":1,
...
}]
}
}
```
#### limit
define the maximum number of rows to read from the file (default None).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"limit":10,
...
}]
}
}
```
#### separator
set the character used to separate fields in the CSV file (default auto-detect).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"separator":",",
...
}]
}
}
```
#### quotechar
define the character used to quote fields containing special characters (default '"').
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"quotechar":"\"",
...
}]
}
}
```
#### escapechar
specify the character used to escape the quote character within fields (default '').
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"escapechar":"\\",
...
}]
}
}
```
#### lineterminator
determine the character sequence that indicates the end of a line (default "\n").
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"lineterminator":"\n",
...
}]
}
}
```
#### skipinitialspace
ignore whitespace immediately following the separator (default false).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"skipinitialspace":true,
...
}]
}
}
```
#### doublequote
Controls whether two consecutive quote characters are interpreted as one (default true).
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"doublequote":true,
...
}]
}
}
```
#### worksheet
:::info
only for .xls and .xlsx source files
:::
Indicates which worksheet should be loaded.
you have to add in your load configuration in the "params.load\_from\[0]" section :
```json
{
"params":{
"load_from":[{
"worksheet":"your_worksheet_name",
...
}]
}
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-load-pyspark.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Load PySpark Action
## Objective
The _Load PySpark_ action **extracts raw data from sources and inserts it into primary tables** in your data warehouse using [Apache Spark™](https://spark.apache.org/). It is usually the first step in any data processing system, necessary before running any other processes on data coming from connected sources.
:::info
In other terms, if you've [set-up a connector through the Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md), the Project has the information to connect to the source as well as its metadata, but won't contain the actual data until it is actually loaded using the _Load PySpark_ action.
:::
:::warning
The _Load PySpark_
action is only compatible with the following source connectors: [MySQL](https://docs.dataplatform.ovh.net/connectors-sources-mysql.md)
, PostgreSQL, Snowflake, [Trino](https://docs.dataplatform.ovh.net/connectors-sources-trino.md)
, AWS S31
, [Microsoft Azure Blob Storage](https://docs.dataplatform.ovh.net/connectors-sources-blob-storage.md)
, [Data Platform Buckets](https://docs.dataplatform.ovh.net/connectors-sources-dataplatform-buckets.md)
, [File Upload](https://docs.dataplatform.ovh.net/connectors-sources-upload-files.md)
, and S3.
:::
## Configure a Load PySpark action
To create a new _Load PySpark_ action, navigate to the _Actions_ tab of the Data Processing Engine (DPE), click on **New Action** and select the _Load PySpark_ action.
### Choose a source and destination
To set-up the _Load PySpark_ action, start by selecting a **source** from the list that you have already set-up in the [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) tab of the Connectors.
Then a **destination** table to send your data to. Similarly to the sources, the tables have to be configured ahead of time in the [Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) tab of the Lakehouse Manager. Make sure you've [built](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md) your data model otherwise the tables won't be accessible by other components in the platform.
### Map attributes
Once the source & destination have been selected, the system will **automatically map** the destination attributes of your destination table with the different columns that it has identified in the raw data source.
:::info
On the picture below, you can see that the _date_ attribute has a a value of '1' beside the notepad icon. This indicates the number of **data validation rules, also called blueprint rules**, defined in the Connectors, that will be automatically applied when loading the data. [Blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md) allow users to quickly filter out invalid data like nulls / empty cells or reformat values such as date or currencies.
:::
Note that you might want to **insert raw text** directly inside your destination table. This might come in handy for instance when you want to count the number of data points or tag a certain source with a label that will be later used in an aggregate table. To do that, select _\< txt >_ which will set the source attribute to a raw text input field.
### Configure the action's preferences
Once you're satisfied with the mapping of your attributes, navigate to the [preferences tab to configure the action settings](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
When you are done editing your action, click on **Create** to validate the action.
## Advanced Mode
If you need to access the JSON configuration file of the action, you can activate the Advanced mode by clicking on **Advanced** at the top of the page.
[Use the Advanced mode of a Load action](https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Transform data between tables with the Aggregate action
## Objective
:::info
This article is about actions that use Data Platform's proprietary Python data processing engine. To use Apache Spark clusters, see [Aggregate PySpark action](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md).
:::
The _Aggregate_ action allows you to transfer data from any pre-existing table into any other table, including the same table. The Aggregate action's scope - which typically allows SQL handling as well as joining multiple tables and grouping results - is wide, making it one of the most used action in data processing workflows.
## Configure an Aggregate action
An Aggregate action is composed of a list of one or more _source tables_ from which the values will be extracted and placed into a single _destination table_. Each destination attribute needs to be mapped to a source attribute.
When multiple _source tables_ are selected, it is necessary to specify a join condition, just like it's done in SQL.
:::warning
Note that leaving a destination attribute unmapped in the Aggregate action configuration **will trigger an error** when the action is launched. If you'd rather leave the destination field empty, make sure to simply remove it from the list of mapped attributes.
:::
When destination attributes are removed from the list of mapped attributes, the following behavior shall be expected:
- If the Aggregate action is **inserting new rows, it will simply fill the destination attribute column with NULL values**.
- If the Aggregate action is **updating existing rows (i.e. inserting over existing data), it will leave the previous value intact regardless of whether it was NULL or not**.
Use case examples for both single and multiple _source tables_ are described below.
### Transfer raw data into a primary table
Create a new Aggregate action:
Select the source (left) and destination table. Matching attributes are automatically assigned where possible. It is possible to manually select which attributes are mapped between the source and destination.
If needed, you can change the mapping to either SQL or plain text by clicking on the **MAP** button and selecting either _SQL_ or _txt_.
In the attributes section, it is also possible to change the compute mode of each attribute. In the example above, _SELECT_ has been chosen but other options are available.
Press _Create_ at the top right to create the action.
### Join a referential to a primary table
To specify a join, click on **Add a source** located just below the source table, you will be prompted to select a table to join with. Select the type of join from the dropdown menu, and input the join conditions in the UI. Multiple source tables can be joined.
Attributes from all the joined tables can then be selected in the bottom attributes section either from the dropdown menus or using SQL.
### Configure the action's preferences
Once you're satisfied with the mapping of your attributes, use the [preferences tab to configure the action settings](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
## Use the Advanced Mode
If you need to access the JSON configuration file of the action, you can activate the Advanced mode by clicking on **Advanced** at the top of the page.
[Use the Advanced mode of an Aggregate action](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Advanced mode for the Aggregate action
## Objective
Switching to advanced mode on the header of your action editor (DPE > Actions > New Action > Aggregate) in the DPE lets you **go further in customizing the Aggregate action**, by displaying and working directly in a JSON configuration file. The JSON files offer more flexibility and access to all options for Aggregate actions.
The JSON configuration file for an Aggregate action is composed of higher level objects which are mandatory. This includes:
- [**Sources**](#sources): Defined the source tables used to load the data from.
- [**Grouping**](#grouping): Define by which dimension the data will be grouped by based on the primary keys of the destination table.
- [**Schema**](#schema): Defines the individual mapping between the destination table and the source tables.
- [**Join**](#join): Defines the tentative joining conditions in the case where there are multiple sources.
- [**Destination**](#destination): Defines in which table we will be loading the output data from the Aggregate action.
Other optional parameters can be added as well, we will cover that in specific section [**Other optional parameters**](#other-optional-parameters) which include for instance the segmentation, perimeter or transformation rules.
## Sources
**Format**: The JSON contains a string value which points to the source and optional additional parameters. The source path looks like that: `dwh/database_name/table_name`, where `database_name` is the name of the source location (source or coming from Lakehouse Manager) and `table_name` the name of the source object or the object in the schema of Lakehouse Manager.
**JSON key**: `params.load_from[0].source` or `params.load_from.source`
Example:
```json
"load_from" : [
{
"source" : "dwh/default_dataset/prim_calendar",
"force_utf8" : true
}
]
load_from" : {
"source" : "dwh/default_dataset/prim_calendar",
"force_utf8" : true
}
```
:::tip
You can alias tables by adding a space after the table\_name then the character(s) for the table alias. For instance: "source": "dwh/source\_facebook/lifetime\_insights fb\_i". This will alias the `lifetime_insights` table by `fb_i`. So you can reuse this alias in joins and schema.
:::
## Grouping
Scaling determines how granular the data will be aggregated. This helps in particular for the constitution of data to insert into aggregation tables.
**Format**: This is a JSON list of attributes or SQL rules to determine the grouping granularity.
**JSON key**: `params.scale`
Example:
```json
{
"load_from": {
"source": "dwh/default_dataset/prim_facebook"
},
"scale": ["date","page_id"],
…
```
This will aggregate the number of `fb_fans` by `date` and `page_id`
:::info
Implementing a scale will generate a GROUP BY operation in the SQL extraction query. In this example: GROUP BY `page_id`, date.
:::
## Schema
The schema determines the renaming between the name of the fields as present in the source, to the naming that these fields must have in the destination.
**Format**: the schema object contains a series of JSON objects, in which each key is the name of an attribute of the destination, and the value contains the parameters associated with the attribute.
**JSON key**: `params.schema`
Example:
```json
{
"schema": {
"month": {
"table": "ref_month",
"operation": "SELECT",
"type": "default",
"attribute": "month"
},
"vegetables": {
"type": "sql",
"sql": "CONCAT(\"HELLO : \", prim_calendar.legumes)"
},
"season": {
"type": "replace",
"value": "hello"
}
}
```
There are 3 different types of mapping:
- type:`default` : to map one attribute to another without any transformation
- - `table` the original source object of the attribute
- - `attribute` source attribute to map from
- - `operation` represents the SQL operation types mentioned in SQL operation section. (only for sql-like sources).
- type:`replace` : will put a hard-coded value in all destination rows.
- - `value`: this attribute will be filled with the value for all destination rows about to be inserted/updated
- `type`:"sql" : the following sql formula will be applied (only for sql-like sources).
- - `sql` the value is an SQL statement.
## Join
Joins are only possible between tables present in the same database or in the same database instance (for example, only tables of `default_dataset`).
**Format**: A list of JSON objects, each representing a JOIN statement. The order of the objects is the order of JOIN action sequence.
**JSON key**: `params.joins`
Example:
```json
"joins": [
{
"table": "ref_month",
"type": "LEFT",
"condition": "prim_calendar.month_number = ref_month.month_number"
},
{
"table": "ref_saison",
"type": "LEFT",
"condition": "prim_calendar.month_number = ref_season.season_number"
}
]
```
- `table`: the table to join the source
- `type`: join type mentioned in join section
- `condition`: the join condition, which follows the ON keyword in SQL statement
## Destination
Destination represents the location where action data will be injected.
**Format**: Its path looks like this: `dwh/database_name/table_name`. Where database\_name is the name of the destination database (typically `default_dataset`) and `table_name` the name of the destination table in the Lakehouse Manager schema.
**JSON key**: `params.load_to[source]`
Example:
```json
"load_to": {
"source": "dwh/default_dataset/tmp_calendar",
"force_utf8": true
}
```
Note that you can fill in multiple destinations at once:
```json
"load_to": [
{"source": "dwh/default_dataset/prim_facebook"},
{"source": "dwh/default_dataset/tmp_facebook"},
]
```
## Other optional parameters
### Extract\_chunk\_size
`extract_chunk_size` is an integer value which represents the maximum allowed size at once for extracting data.
Example:
```json
"extract_chunk_size": 5000
```
This will allow the worker to extract data with batches of 5000 lines.
**JSON key**: `params.load_from[0].extract_chunk_size`
#### Source\_rules
`source_rules` represents the rule for simple data preparation. This setting will override the existing rules synchronized from DWH. They will be applied to attributes from the source. For instance let's assumer we would like to apply transformation rules to the attribute `date_source`.
**Format**: A list contains JSON objects, where each represents a rule applied to an attribute.
**JSON key**: `params.load_from[source_rules]`
Example:
```json
"source_rules": [
{
"action": "date_replace",
"action_values": [
"%d/%m/%Y %H:%i:%s",
"%Y-%m-%d %H:%M:%S"
],
"attribute_name": "date_source",
"name": "correct date format"
}
]
```
For detailed configuration documentation, please refer to the article on [blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md)
#### Rules
Rules are similar to source\_rules except that they will be applied to the destination attribute instead of the extraction attribute. This allows you to map different destination attributes to the same source attribute but assign it different rules.
**Format**: A list contains JSON objects, which each represents a rule applied to an attribute.
**JSON key**: `params.rules`
Example:
```json
"rules": [
{
"action": "date_replace",
"action_values": [
"%d/%m/%Y %H:%i:%s",
"%Y-%m-%d %H:%M:%S"
],
"attribute_name": "date_dest",
"name": "correct date format"
}
]
```
#### SQL conditions
In order to filter with complexe SQL conditions, you can add them through the "condition" option.
**JSON key**: `params.load_from[0].condition`
**Format**: A string that contains the "WHERE" clause of the extraction of the SQL query that will be generated on the source. (excluding the 'WHERE').
Example:
```json
{
"params":
{
"load_from": [{
"condition": " (attribute1 > 1 OR attribute2 BETWEEN 2019 and 2021) ",
...
}]
}
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Aggregate PySpark Action
## Objective
The _Aggregate PySpark_ action allows you to transfer data from any pre-existing table into any other table (including the same table), using [Apache Spark™](https://spark.apache.org/).
The Aggregate PySpark action's scope - which typically allows SQL handling as well as joining multiple tables and grouping results - is wide, making it one of the most used action in data processing workflows.
- [Configure an Aggregate PySpark action](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md#configure-an-aggregate-pyspark-action)
- [Transfer raw data into a primary table](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md#transfer-raw-data-into-a-primary-table)
- [Join a referential to a primary table](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md#join-a-referential-to-a-primary-table)
- [Configure the action's preferences](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md#configure-the-actions-preferences)
- [Use the Advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md#use-the-advanced-mode)
## Configure an Aggregate PySpark action
An Aggregate PySpark action is composed of a list of one or more _source tables_ from which the values will be extracted and placed into a single _destination table_. Each destination attribute needs to be mapped to a source attribute.
When multiple _source tables_ are selected, it is necessary to specify a join condition, just like it's done in SQL.
:::warning
Note that leaving a destination attribute unmapped in the Aggregate PySpark action configuration **will trigger an error** when the action is launched. If you'd rather leave the destination field empty, make sure to simply remove it from the list of mapped attributes.
:::
When destination attributes are removed from the list of mapped attributes, the following behavior shall be expected:
- If the Aggregate PySpark action is **inserting new rows, it will simply fill the destination attribute column with NULL values**.
- If the Aggregate PySpark action is **updating existing rows (i.e. inserting over existing data), it will leave the previous value intact regardless of whether it was NULL or not**.
Use case examples for both single and multiple _source tables_ are described below.
### Transfer raw data into a primary table
Create a new Aggregate PySpark action:
Select the source (left) and destination table. Matching attributes are automatically assigned where possible. It is possible to manually select which attributes are mapped between the source and destination.
If needed, you can change the mapping to either SQL or plain text by clicking on the **MAP** button and selecting either _SQL_ or _txt_.
In the attributes section, it is also possible to change the compute mode of each attribute. In the example above, _SELECT_ has been chosen but other options are available.
Press _Create_ at the top right to create the action.
### Join a referential to a primary table
To specify a join, click on **Add a source** located just below the source table, you will be prompted to select a table to join with. Select the type of join from the dropdown menu, and input the join conditions in the UI. Multiple source tables can be joined.
Attributes from all the joined tables can then be selected in the bottom attributes section either from the dropdown menus or using SQL.
### Configure the action's preferences
Once you're satisfied with the mapping of your attributes, use the [preferences tab to configure the action settings](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
## Use the Advanced Mode
If you need to access the JSON configuration file of the action, you can activate the Advanced mode by clicking on **Advanced** at the top of the page.
[Use the Advanced mode of an Aggregate action](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-sql.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Run SQL statements with the SQL action
## Objective
The _SQL action_ lets you write a SQL query and run it against your datasets and tables as a Data Processing Engine action. Like any other action, it can be run on demand, scheduled with a cron trigger, or orchestrated inside a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md).
Queries run on the underlying **Trino** engine, so tables are addressed in the fully qualified `catalog.schema.table` form. The Tables panel (see below) builds those names for you, so you rarely have to type them by hand.
By default a SQL action runs on 2 DPU; you can adjust the allocation from the resource selector at the top of the configuration screen.
## Configure a SQL action
The configuration screen has two panels: the **Tables** browser on the left to help you find and reference assets, and the **SQL query** editor on the right where you write and validate your statement.
### Browse tables and copy names
The **Tables** panel lists the datasets in your project, each expandable to its tables and their attributes. Use the search box to quickly locate an asset, then click it to copy its name to the clipboard, shaped so it drops straight into your query:
| Asset clicked | What gets copied |
| ------------- | ----------------------------------------------------------------------------------------------- |
| **Dataset** | `catalog.schema`: the catalog and schema that Trino needs to resolve tables in that dataset. |
| **Table** | The fully qualified `catalog.schema.table`, ready to use directly in a `FROM` or `JOIN` clause. |
| **Attribute** | The attribute (column) name only. |
This saves you from remembering how each dataset maps to its Trino catalog and schema, and helps the engine resolve your query on the first try.
### Write and validate the query
Type your query in the **SQL query** editor. If you need a starting point, open the **Query examples** drawer at the bottom of the page and click any example to copy it into the editor.
Once your query is ready, click **Validate**. This runs the query as a dry run and, on success, shows a **validation summary** at the bottom of the editor: a beautified version of `EXPLAIN` that tells you what the query will do, with the key details for that query type.
:::info
**Validate** does not gate creation. The **Create** button is always available, but validating first is best practice: it confirms the query is well-formed and resolves against your tables before you commit the action.
:::
When you're satisfied, click **Create** at the top right to create the action.
## Supported query types
The SQL action supports a broad range of statements. The engine recognizes the type of each query and behaves accordingly:
- **Read:** return rows without changing anything.
- **Metadata:** inspect schema or query plans.
- **DML:** modify the rows in a table.
- **DDL:** change the structure of a table.
Select a tab below to see an example query for each type:
**Read**
```sql
SELECT *
FROM catalog.schema.orders
WHERE status = 'OPEN';
```
**Metadata**
```sql
SHOW COLUMNS FROM catalog.schema.orders;
```
**DML**
```sql
UPDATE catalog.schema.orders
SET status = 'CLOSED'
WHERE id = 42;
```
**DDL**
```sql
ALTER TABLE catalog.schema.orders
ADD COLUMN note VARCHAR;
```
:::warning
Access-control statements such as `GRANT` and `REVOKE` are not supported and will be rejected during validation.
:::
## Create a table from a query
When you run a `SELECT`, the action offers the option to **create a new table** from the query result.
If you create and run an action with a `CREATE TABLE AS SELECT` (CTAS), or any other `CREATE` statement, the table is created on the target dataset, and the corresponding logical table is registered and built in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) as well. The new table is then available across the platform like any other Lakehouse Manager table.
## Validation at run time
Validation does not only happen while you author the query. **Every time the action runs**, the query is re-validated with `EXPLAIN` before any statement is executed.
This catches cases where the query no longer fits the table. For example, if a column referenced by the query was renamed or removed because the table's schema changed since the action was created, the run-time validation fails and the action is stopped before it executes anything, so a stale query can't run against an incompatible table and corrupt your data.
:::info
Because validation runs before execution on every run, a SQL action that suddenly fails to validate is often a signal that an upstream table has changed. Check the table's schema before updating the query.
:::
## Use the Advanced mode
If you need to access the JSON configuration file of the action, activate the Advanced mode by clicking on **Advanced** at the top of the page.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-custom.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Run custom Python scripts with the Custom action
## Objective
:::info
This article is about actions that use the Data Platform's Python data processing engine. To use Apache Spark clusters, see [Custom PySpark action](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md).
:::
A _Custom action_ allows you to execute custom Python scripts in a scalable cloud cluster environment.
Using our [Software Development Kit (SDK)](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) to easily interact with the different components of the platform, _Custom actions_ can be used to implement a variety of use-cases such as:
- Execute a manipulation algorithm or ETL job on your data warehouse
- Execute a simple data analysis or machine learning algorithm
- Extract data from data sources not available on the Data Platform marketplace without having to create connectors for it
- Extract real time data (like MQTT, Kafka, etc..)
:::info
Custom actions benefit from the whole Data Processing Engine's feature-set, typically the power of the [segmentation to parallelize the execution](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md) of your algorithm or the orchestration within [workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) triggered immediately or on a scheduled basis.
:::
## Configure a Custom action
In the Data Processing Engine of your Project, go in the Actions tab and click on the **New Action** button. Choose the action type _Custom_.
Drag and drop your _.py_ script onto the "Drag and drop" section.\
Alternatively, select the **Start with a boilerplate** option to get started directly on the Platform's Python interface with example code snippets.
The name of the function to be executed **must be entered manually** at the top of the screen in the Information panel (here and by default: "customfunc").
You will be able to edit your source file directly in the editing interface or drop a new file if needed. When you are developing your own custom actions you can use any functions provided in the [Software Development Kit (SDK)](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md). to easily interact with other components of the platform.\
To read more about all the available SDK functions check out the article below:
[Discover all SDK methods](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md)
:::info
Everything your script writes to standard output or standard error, including plain `print()` calls, appears in the job logs alongside the messages emitted through the `logging` module.
:::
## Use the helper panel
The Custom action editor includes a helper panel, so you can find guidance without leaving your script:
- **Scenarios**: ready-to-use action scripts organized by category, to copy and adapt to your use case.
- **SDK guides**: documentation for the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) methods you can call from your script.
- **Data**: browse your project's data directly from the editor, to check names and structures while you write your code.
- **FAQ**: answers to common Custom action questions.
:::info
The helper content is the same live catalog that powers the [Data Platform Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md) in notebooks, so it is always up to date.
:::
## Manage dependencies
### Setting language version
You can choose the Python version of your custom action among the following:
- Python 3.11
- Python 3.9 _(default option)_
:::info
[Workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) cannot be executed with multiple versions at once.
:::
:::info
We are regularly updating the available versions to provide you with a best-practice development framework. Your existing work is not migrated to a new version as long as its language version is still supported.
:::
### Installing Python packages
You might need to install specific packages not included by default. You can add them in the "Python Requirements" field respecting the format used in a basic requirements file for "pip" (Python package manager) then press "ENTER" on your keyboard.
This is what it should looks like once you pressed "ENTER":
:::info
When working with a Custom Action in an [Always-up](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#execution-modes) execution environment, updating dependencies triggers a redeployment of the environment to put the changes into effect.
:::
### Installing packages from a Git repository
You can install Python packages directly from a GitHub or GitLab repository using the `git+` prefix in your requirements:
```
git+https://github.com/{OWNER}/{REPO}.git
```
To pin a specific version, add a tag or commit hash:
```
git+https://github.com/{OWNER}/{REPO}.git@
```
#### Auto-install the latest release
To always install the most recent published release without manually tracking version tags, use `@latest`:
```
git+https://github.com/{OWNER}/{REPO}.git@latest
```
The platform detects the `git+` prefix and `@latest` suffix, then automatically resolves and substitutes the latest release tag before installing.
:::info
After adding or modifying a Git dependency, click **Force Build** to reinstall. You no longer need to manually update the tag or commit hash each time a new version of your module is published, but note that the latest release is not picked up automatically at runtime, a manual **Force Build** is always required.
:::
### Default list of dependencies
:::warning
Data Platform blocks the minors of the versions allowing bug fixes to be installed. If you need a more recent version of a library you can override it manually by adding the same package with the new version in the "Requirements" field.
:::
Here is the list of all the packages and their version (as you could find them in a requirements file for pip) shipped with the Data Processing Engine workers:
[Discover all default Python packages](https://docs.dataplatform.ovh.net/dpe-actions-custom-default-packages.md)
## Examples of sample scripts
### Example of the extraction of a file followed by loading it into the default dataset
```python
import logging
import sys
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
def customfunc(event):
logger = logging.getLogger(__name__)
try:
logger.info("Begin function")
# Connect to the source connector
connector = connect("dwh/dropbox_test/consommations.csv")
# Upload raw file from the source connector
connection_str = get_raw(connector)
# Connect to the source
source = connect(connection_str)
# Connect to the destination connector
destination = connect("dwh/default_dataset/consommations")
# Extract dataframe from source and bulk insert into the destination
for df in extract(source):
stats, error = bulk_insert(destination, "consommations", df)
logger.info(stats)
logger.info(error)
del connector, source, destination
logger.info("END function")
except Exception as e:
raise Exception("err:{} L:{}".format(e, sys.exc_info()[2].tb_lineno))
```
### Example of a data transfer between two datasets
```python
import logging
import sys
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
def customfunc(event):
logger = logging.getLogger(__name__)
try:
logger.info("Begin function")
# Connection to a source datastore
connector = connect("dwh/default_dataset/consommations")
# Data extraction from the source by a SELECT
lines = connector.select("consommations", {"filter_attribute": "2018-01-01"})
del connector
# Treatment of each line of the data
for line in lines:
line["new_insight"] = (line["factor1"] + line["factor2"] * 2) / 100
# Connection to the destination datastore
connector = connect("dwh/analytics_dataset/agr_consommations")
# Bulk insert into the destination
stats, err = bulk_insert(connector, "agr_consommations", lines)
logger.info(stats)
logger.info(err)
del connector
logger.info("END function")
except Exception as e:
raise Exception("err:{} L:{}".format(e, sys.exc_info()[2].tb_lineno))
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-custom-default-packages.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Packages preloaded in Data Platform action containers
## Objective
All actions & workflows when launched are built and run in **containers which come with preloaded packages**. That way, especially in the case of custom actions, you don't have to always manually specify which librairies you need to run your code.
Currently Data Platform jobs are only compatible with Python 3+, there is therefore only 2 default packages for you to manage:
- **Python Libraries**: If you're about to write your own custom action, no need to give you a lecture on Python libraries... in the background, we're simply running a _pip install_ command.
- **Data Platform Modules**: In order to keep a light weight image of the containers all Data Platform connectors are not loaded automatically. To use a specific connector using the SDK, please specify the required modules from our internal supported list.
## Default Python Librairies
Below is the list of all the Python libraries and corresponding version number installed by default in the Data Platform DPE containers:
```text
nameko==2.12.0
pyopenssl==19.1.0
minio==5.0.8
bson==0.5.8
requests==2.22.0
timedelta==2019.4.13
coverage==4.5.4
pycryptodome==3.9.4
kubernetes==10.0.1
sqlalchemy==1.3.8
mysqlclient==2.0.1
prometheus_client==0.7.1
google-cloud-storage==1.36.2
pandas==1.0.0
```
:::warning
**Note** that we don't allow users to override the default libraries installed with older versions to prevent incompatibility issues.
:::
However, **you can absolutely install any other libraries on top of the list provided** above by adding them in "Requirements" field of your custom action configuration whether in the UI or in the JSON file. Simply use the same format as when using a _pip install_ command to specify the version's library.
[Explore all Python libraries on Pypi's official website](https://pypi.org/)
## Data Platform Modules
When you create a new custom action using the Data Platform Data Processing Engine you will certainly be connecting to external data sources and use a variety of file formats. **Data Platform possesses a pre-existing list of modules to handle both data formats & connector types inside your code**.
The list of modules preinstalled by default includes primarily support for generic file format & other component of the platform (data warehouse & data store): mySQL, S31
, CSV, XML, XLS, XLSX, JSON. Parquet file formats are not installed by default because of the size of the module (approximately 400 MB) which would add latency for the build of the workers.
To install Data Platform modules for your custom actions you can either do it via the Custom action configuration GUI under the _Requirements_ section. For instance, if you use the SDK functions to connect to an external Dropbox source but have not installed the Dropbox module, your code will throw an error unless you configure your custom action as below:
Or you can directly input the list of the modules in the JSON configuration of the action under the key _modules_:
```json
{
"modules": [
{
"type": "action",
"id": "aggregate",
"version": "latest"
},
{ ... }
],
```
:::info
**Note** that by default in the UI dropdown, all modules will be installed to their latest version. If you are trying to install a specific version please specify it in directly in the JSON configuration of the action (see syntax above) in Advanced mode.
:::
Below is the **exhaustive list of additional available Data Platform modules** that can be installed in DPE workers (which corresponds to the list you can find in the Lakehouse Manager's source's marketplace):
| Module | Description | Type | Technical Name |
| --------------------- | ------------------------------------------------------------------------ | --------- | ----------------- |
| Dropbox | Connector to Dropbox file storage system | protocol | dropbox |
| Dropzone | Connector to Data Platform' dropzone protocol | protocol | dropzone |
| FTP | Module to support FTP protocols | protocol | ftp |
| Google Drive | Connector to Google Drive file storage system | protocol | google-drive |
| HTTP | Module to support HTTP protocols | protocol | http |
| HTTP REST | Module to support HTTP REST protocols | protocol | httprest |
| Microsoft One Drive | Connector to One Drive file storage system | protocol | onedrive |
| AWS S3 | Connector to AWS S3 file storage system | protocol | s3 |
| SFTP | Module to support SFTP protocols | protocol | sftp |
| App Figures | Connector to [appfigures](https://appfigures.com/) analytics & ASO tools | connector | appfigures |
| GCP BigQuery | Connector to Google's big data analytics platform | connector | bigquery |
| Data Platform Project | Connector to another Data Platform' Project | connector | dataplant |
| Facebook | Connector to Facebook analytics | connector | facebook |
| Google Analytics | Connector to Google Analytics | connector | ganalytics |
| Apache Impala | Connector to Apache Impala distributed SQL query engine | connector | impala |
| Instagram | Connector to Instagram analytics | connector | instagram |
| Intercom | Connector to Intercom analytics | connector | intercom |
| LinkedIn | Connector to LinkedIn analytics | connector | linkedin |
| Mailchimp | Connector to Mailchimp analytics | connector | mailchimp |
| Parquet | Module to support Parquet file formats | connector | parquet |
| PostgreSQL | Connector to PostgreSQL database engine | connector | postgresql |
| SQLServer | Connector to SQLServer database engine | connector | sqlserver |
| Twitter | Connector to Twitter analytics | connector | twitter |
| Weibo | Connector to Weibo analytics | connector | weibo |
| Youtube | Connector to Youtube account analytics | connector | youtube |
| Youtube Analytics | Connector to Youtube Analytics tool | connector | youtube-analytics |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Custom PySpark Action
## Objective
A _Custom PySpark action_ allows you to execute custom PySpark scripts in a scalable cloud cluster environment using [Apache Spark™](https://spark.apache.org/).
Using our [Software Development Kit (SDK)](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) to easily interact with the different components of the platform, _Custom PySpark actions_ can be used to implement a variety of use-cases such as:
- Execute a manipulation algorithm or ETL job on your data warehouse
- Execute a simple data analysis or machine learning algorithm
- Extract data from data sources not available on the Data Platform marketplace without having to create connectors for it
- Extract real time data (like MQTT, Kafka, etc..)
:::info
Custom PySpark actions can be orchestrated within [workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) and triggered immediately or on a scheduled basis. At the moment, a single workflow **cannot** contain both PySpark actions and normal actions.
:::
## Configure a Custom PySpark action
In the Data Processing Engine of your Project, go in the Actions tab and click on the **New Action** button. Choose the action type _Custom PySpark_.
Drag and drop your _.py_ PySpark script onto the "Drag and drop" section.\
Alternatively, select the **Start with a boilerplate** option to get started directly on the Platform's Python interface with example code snippets.
You will be able to edit your source file directly in the editing interface (or drop a new file if needed). Check out the PySpark documentation below:
[PySpark Documentation Portal](http://spark.apache.org/docs/latest/api/python/)
Note you can also use any function provided in the [Software Development Kit (SDK)](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) to easily interact with other components of the platform. To read more about all the available SDK functions check out the article below:
[Discover all SDK methods](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md)
## Use the helper panel
The Custom PySpark action editor includes a helper panel, so you can find guidance without leaving your script:
- **Scenarios**: ready-to-use action scripts organized by category, to copy and adapt to your use case.
- **SDK guides**: documentation for the [SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) methods you can call from your script.
- **Data**: browse your project's data directly from the editor, to check names and structures while you write your code.
- **FAQ**: answers to common action questions.
:::info
The helper content is the same live catalog that powers the [Data Platform Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md) in notebooks, so it is always up to date.
:::
## Manage parallelization
Scale your job by adding more parallel worker instances in the action's [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
Set the number of **executors** and the [DPU](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#pyspark-actions) size of each. The Spark **driver** has its own DPU size, set independently of the executors. The Resources panel shows the total DPU of the execution.
Specify how to split your job's workload on those different workers inside your PySpark script.
:::warning
Contrary to actions that use the Platform's proprietary data processing engine, PySpark actions don't have settings for [segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#segmentation) and [perimeter](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#perimeter) in the action's preferences. These must be handled inside your PySpark code.
:::
## Manage dependencies
### Installing Python packages
You might need to install specific packages not included by default. You can add them in the "Python Requirements" field respecting the format used in a basic requirements file for "pip" (Python package manager) then press "ENTER" on your keyboard.
This is what it should looks like once you pressed "ENTER":
### Installing packages from a Git repository
You can install Python packages directly from a GitHub or GitLab repository using the `git+` prefix in your requirements:
```
git+https://github.com/{OWNER}/{REPO}.git
```
To pin a specific version, add a tag or commit hash:
```
git+https://github.com/{OWNER}/{REPO}.git@
```
#### Auto-install the latest release
To always install the most recent published release without manually tracking version tags, use `@latest`:
```
git+https://github.com/{OWNER}/{REPO}.git@latest
```
The platform detects the `git+` prefix and `@latest` suffix, then automatically resolves and substitutes the latest release tag before installing.
:::info
After adding or modifying a Git dependency, click **Force Build** to reinstall. You no longer need to manually update the tag or commit hash each time a new version of your module is published, but note that the latest release is not picked up automatically at runtime, a manual **Force Build** is always required.
:::
### Default list of dependencies
:::warning
Data Platform blocks the minors of the versions allowing bug fixes to be installed. If you need a more recent version of a library you can override it manually by adding the same package with the new version in the "Requirements" field.
:::
Here is the list of all the packages and their version (as you could find them in a requirements file for pip) shipped with the Data Processing Engine workers:
[Discover all default Python packages](https://docs.dataplatform.ovh.net/dpe-actions-custom-default-packages.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-file-transfer.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# File Transfer Action
## Objective
The _File Transfer_ action allows you to **copy files and folders between buckets, or from a configured source in your Connectors to a destination bucket.** This action is designed to efficiently duplicate data, always preserving the source's full path structure within the destination.
## Configure a File Transfer action
To create a new _File Transfer_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"File Transfer"**.
### Choose a Source for Transfer
First, you need to define where the files will be copied _from_.
_Figure: Selecting the source for your file transfer._
You have two main options for your source:
- **Connectors Source:** Select one of the sources you have already configured in the [Sources tab](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) of the Connectors. This is ideal for bringing data from external systems (e.g., databases, other object storage providers) into your project buckets.
- **Project Bucket:** Select an existing [Project bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) directly from the list. This is useful for copying data between buckets within your OVHcloud environment.
Once you've selected your source, you will need to specify the **Source File Path**. This path indicates which files or folders within your chosen source should be copied.
- **To transfer specific files:** Enter the full path to the file, e.g., `folderA/subfolderB/my_file.csv`.
- **To transfer an entire folder and its contents:** Use a wildcard `*` at the end of the folder path.
- **Example Path:** `uploads/nyc_taxi/*`
- This example path indicates that all files and sub-folders directly within the `nyc_taxi` folder (which itself is located inside the `uploads` folder) will be included in the transfer.
- If you want to transfer the _entire content_ of the root of your source, you would specify `*`.
:::info
**Important:** The File Transfer action always preserves the **full source path** under the destination. For example, if you copy the folder `uploads/nyc_taxi/` (using `uploads/nyc_taxi/*`) from your source to a destination bucket with a destination path `/my_new_data/`, the `nyc_taxi` folder and its contents will appear as `/my_new_data/uploads/nyc_taxi/` in the destination.
:::
### Specify Destination Bucket
Next, you will define the destination where your files will be copied _to_.
_Figure: Specifying the destination bucket and path._
1. **Choose Destination Bucket:** Select the target [Project bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) from the provided list.
2. **Enter Destination Path:** In the field below the destination bucket selection, you can specify a path within the destination bucket.
- If you enter a forward slash (`/`), the system will provide a list of available paths and folders within that destination bucket, allowing you to easily select an existing location.
- You can also type a new path, and the system will create the necessary folders if they don't already exist.
3. **Run the Action:** Once both your source and destination are configured, click the "Run" button to initiate the file transfer.
## Verify Your Transfer
After the _File Transfer_ action completes successfully, you can visually confirm the changes by navigating to the **Buckets** section within the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). Here, you will see the copied files and folders appear in your destination bucket, reflecting the structure preserved from the source.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-load-to-bucket.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Load to Bucket Action
## Objective
The _Load to Bucket_ action **extracts raw data from sources and puts it into a file** in your Project bucket. You can choose the file format.
:::info
**Note:** This action supports the functionality to have customised date in the folder name. You can find the tutorial [here](https://docs.dataplatform.ovh.net/tutorials-dated-export-folders.md).
:::
## Configure a Load to Bucket action
To create a new action, navigate to the **Action tab** of the Data Processing Engine (DPE), click on **New Action** and select the _Load to Bucket_ action.
### Choose a source and a destination
To set-up the action, start by selecting a source from the list that you have already set-up in the [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) tab of the Connectors.
Then, choose the _destination bucket_ from the list of [Project buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
You will then be prompted to choose the file format and the destination folder. If you leave the folder name empty, it will automatically be named as the source name.
:::warning
Note that **the following sources are NOT supported**
in this action: Azure Blob Storage, Data Store, Dropbox, FTP, Google Drive, HTTP Files, HTTP REST, One Drive, S31
-compatible-storage, Amazon S3 and SFTP.
:::
:::info
If you choose a **PostgreSQL source** your action will be have optimised performance similarly to the [PostgreSQL to Parquet action](https://docs.dataplatform.ovh.net/dpe-actions-postgresql-to-parquet.md).
:::
### Destination file
Your data will be put into the destination folder in a file named _part-1_ with the format you specified. If your file is too big, it will be split into multiple files named _part-1_, _part-2_, _part-3_ and so on.
The order of your rows will be preserved in this process. Hence, the first rows will be in the _part-1_ file and the following rows will be in the _part-2_ file.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/dpe-actions-export-table-to-source.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Export Table to Source Action
## Objective
The _Export Table to Source_ action allows you to **export one or more tables from your Lakehouse Manager directly to a compatible Connectors source.** This action facilitates moving processed or raw data from your managed Lakehouse environment into external databases or other systems supported by Connectors connectors.
## Configure an Export Table to Source action
To create a new _Export Table to Source_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"Export Table to Source"**.
### Choose Tables for Export
First, you need to select the tables from your Lakehouse Manager that you wish to export.
1. A list of available tables from your [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) will be displayed.
2. You can select **one or multiple tables** from this list for export. Simply click on the table names to highlight them for selection.
### Select Destination Source
Next, you will define the external source where the selected tables will be exported.
1. A dropdown or list will appear, showing available compatible sources from your [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md).
2. **Compatibility Note:** Currently, this action supports Connectors sources of **type database and protocols**.
3. **Important:** If no destinations are shown, it means you have not yet connected any compatible database or protocol sources in your Connectors. Please ensure you have configured at least one such source before attempting to use this action.
4. Select the desired destination source from the list.
5. click confirm and run.
:::warning
**Important Note on Export Table to Source:** The _Export Table to Source_ action might **fail** if the destination database you are exporting to has a configuration requiring primary keys for tables being created or inserted. For example, some database systems or specific table settings (like `REQUIRE PRIMARY KEY` in certain MySQL/MariaDB setups or stricter NoSQL configurations) might enforce this.
:::
:::info
By default, tables within the Lakehouse Manager **do not inherently have primary keys defined**. If your destination database requires a primary key and the incoming table schema does not provide one, the export operation will fail. To successfully export to such databases, you may need to adjust the destination database's configuration to temporarily or permanently relax the primary key requirement.
:::
## Verify Your Export
After the _Export Table to Source_ action completes successfully, you can verify the export by connecting to your chosen destination database or system (e.g., using a SQL client for a database) and checking for the presence of the newly exported tables and their data. You can even check the connectors source to check for the update.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-csv-to-postgresql.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# CSV to PostgreSQL Action
## Objective
The _CSV to PostgreSQL_ action allows you to **ingest data from a CSV file (sourced from any compatible connection in your Connectors) into a table within a PostgreSQL database.** By default, this action appends data to an existing table, with an explicit option to entirely overwrite its contents.
## Configure a CSV to PostgreSQL action
To create a new _CSV to PostgreSQL_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"CSV to PostgreSQL"**.
### Choose a CSV File for Transfer
First, you need to select the specific CSV file you wish to transfer. This list will comprise CSV files accessible from **all compatible sources** configured in your Connectors.
1. You will be presented with a unified list of available CSV files. These files are drawn from all [Connectors sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) that contain CSV data.
2. Browse or search through this list to locate the CSV file you want to transfer.
3. Select the desired CSV file.
4. **Choose Delimiter (Optional):** You can specify the delimiter used in your CSV file. By default, the system will attempt to **auto-detect** the delimiter. Alternatively, you can explicitly select one of the following common delimiters: `,` (comma), `;` (semicolon), or `|` (pipe).
### Select Destination PostgreSQL Table and Options
Next, you will define the target PostgreSQL table and how the data should be loaded. You will see a consolidated list of all tables from all connected PostgreSQL sources.
1. **Select Destination Table:** You will see a comprehensive list of all PostgreSQL tables available across all your connected PostgreSQL [Connectors sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md).
- For easier identification, the **name of the PostgreSQL source** will be mentioned below each table name.
- **Limitation:** Currently, this action supports pushing data only into **existing tables** in your PostgreSQL database.
- **To load into a new table:** You must first create an empty table in your PostgreSQL database (using your preferred PostgreSQL client or tools) and then select it from this list.
2. **Loading Option:** By default, the action will **append** the new data from your CSV file to the selected PostgreSQL table.
- You will also have an explicit option to **Overwrite** the table. If selected, this will completely delete all existing data from the chosen PostgreSQL table before inserting the new data from your CSV file. Use this with extreme caution, as it will erase all previous data in that table.
3. **Run the Action:** Once your CSV file, destination PostgreSQL table, and loading option are configured, click the "Run" button to initiate the transfer.
## Verify Your Transfer
After the _CSV to PostgreSQL_ action completes successfully, you can verify the transfer by connecting to the respective PostgreSQL database (e.g., using a SQL client) and querying the destination table to confirm the presence and correctness of the loaded CSV data.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-mysql-to-parquet.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# MySQL to Parquet Action
## Objective
The _MySQL to Parquet_ action provides a streamlined way to **transfer a table from a MySQL source, configured in your Connectors, directly into a Project bucket.** The data from your MySQL table will be stored in a highly efficient Parquet file format within your chosen object storage bucket.
## Configure a MySQL to Parquet action
To create a new _MySQL to Parquet_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"MySQL to Parquet"**.
### Choose a MySQL Table
First, you need to select the specific MySQL table you wish to transfer.
1. A list of available tables from your connected MySQL sources in the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) will be displayed.
2. **Important:** If this list is empty, it means you do not have any MySQL sources connected or configured in your Connectors. Please ensure you have added and connected your MySQL database as a source before attempting to use this action.
3. Select the desired MySQL table from this list.
### Select Destination Bucket and Path
Next, you will define the destination bucket and the folder path where your Parquet file(s) will be stored.
1. **Choose Destination Bucket:** Select an available [Project bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) from the list where your Parquet file(s) will be stored.
2. **Specify Folder Path:** In the field below the destination bucket selection, you will specify the desired folder path within the bucket.
- **Example:** If you want your data to be stored in a folder named `my_mysql_data` inside a `raw_data` directory, you would enter `raw_data/my_mysql_data/`.
- The system will automatically create any necessary folders if they don't already exist in the specified path.
- **File Naming Convention:** The exported data will always be stored in one or more files named `part-1.parquet`, `part-2.parquet`, and so on, depending on the size of your table.
- **Default Behavior:** If you leave the folder path empty, the action will, by default, create a folder named after the MySQL table you selected (e.g., if your table is named `users`, it will create a `users/` folder at the root of the bucket). Within this default folder, the data will be stored as `part-1.parquet`, `part-2.parquet`, etc.
3. **Run the Action:** Once both your MySQL table and destination are configured, click the "Run" button to initiate the transfer.
## Verify Your Transfer
After the _MySQL to Parquet_ action completes successfully, you can visually confirm the transfer by navigating to the **Buckets** section within the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). Here, you will find your new Parquet file(s) at the specified path within your destination bucket.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-postgresql-to-csv.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PostgreSQL to CSV Action
## Objective
The _PostgreSQL to CSV_ action allows you to **transfer a table from a PostgreSQL source, configured in your Connectors, directly into a Project bucket.** The data from your PostgreSQL table will be stored in a comma-separated values (CSV) file format within your chosen object storage bucket.
## Configure a PostgreSQL to CSV action
To create a new _PostgreSQL to CSV_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"PostgreSQL to CSV"**.
### Choose a PostgreSQL Table
First, you need to select the specific PostgreSQL table you wish to transfer.
1. A list of available tables from your connected PostgreSQL sources in the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) will be displayed.
2. **Important:** If this list is empty, it means you do not have any PostgreSQL sources connected or configured in your Connectors. Please ensure you have added and connected your PostgreSQL database as a source before attempting to use this action.
3. Select the desired PostgreSQL table from this list.
### Select Destination Bucket and Path
Next, you will define the destination bucket and the folder path where your CSV file(s) will be stored.
1. **Choose Destination Bucket:** Select an available [Project bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) from the list where your CSV file(s) will be stored.
2. **Specify Folder Path:** In the field below the destination bucket selection, you will specify the desired folder path within the bucket.
- **Example:** If you want your data to be stored in a folder named `my_postgres_data` inside an `analytics` directory, you would enter `analytics/my_postgres_data/`.
- The system will automatically create any necessary folders if they don't already exist in the specified path.
- **File Naming Convention:** The exported data will always be stored in one or more files named `part-1.csv`, `part-2.csv`, and so on, depending on the size of your table.
- **Default Behavior:** If you leave the folder path empty, the action will, by default, create a folder named after the PostgreSQL table you selected (e.g., if your table is named `products`, it will create a `products/` folder at the root of the bucket). Within this default folder, the data will be stored as `part-1.csv`, `part-2.csv`, etc.
3. **Run the Action:** Once both your PostgreSQL table and destination are configured, click the "Run" button to initiate the transfer.
## Verify Your Transfer
After the _PostgreSQL to CSV_ action completes successfully, you can visually confirm the transfer by navigating to the **Buckets** section within the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). Here, you will find your new CSV file(s) at the specified path within your destination bucket.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-postgresql-to-parquet.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PostgreSQL to Parquet Action
## Objective
The _PostgreSQL to Parquet_ action extracts data from a PostgreSQL database and puts it into a [Parquet file](https://parquet.apache.org/) in your Project bucket. You can choose the file format.
:::info
**Note:** This action supports the functionality to have customised date in the folder name. You can find the tutorial [here](https://docs.dataplatform.ovh.net/tutorials-dated-export-folders.md).
:::
## Configure a PostgreSQL to Parquet action
To create a new action, navigate to the **Action tab** of the Data Processing Engine (DPE), click on **New Action** and select the _PostgreSQL to Parquet_ action.
### Choose a source and a destination
To set-up the action, start by selecting a source from the list that you have already set-up in the [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) tab of the Lakehouse Manager.
Then, choose the _destination bucket_ from the list of [Project buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
### Destination file
You will then be prompted to choose the destination folder. If you leave the folder name empty, it will automatically be named as the source name.
Your data will be put into the destination folder in a file named _part-1_. If your file is too big, it will be split into multiple files named _part-1_, _part-2_, _part-3_ and so on.
The order of your rows will be preserved in this process. Hence, the first rows will be in the _part-1_ file and the following rows will be in the _part-2_ file.
:::info
Note that this action has its **transfer speed optimised** for the PostgreSQL source.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-diff.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Convert cumulative values to daily deltas with the Diff action
## Objective
When you have tables which contain dated attributes describing total values (total daily orders, total monthly revenue, etc.) at a given time, the diff action lets you compute the difference of these values between consecutive time increments.
For example, let's consider the _total\_fans_ attribute in the table below calculated each day for a certain social media profile page. Assuming the _total\_fans_ describes the total number of fans since the creation of this profile, the **diff** action will let you compute the number of new fans acquired or lost between each day.
Please note that first dates of each ids won't have their first incremental value computed, as the previous date is not known.
There is an optional advanced parameter if you have the date of creation so that the first incremental date will be computed as equal to the total attribute.
## Action Configuration
### Action's configuration interface
When configuring the **diff** action from the interface, you will be asked to specify:
- the **table name** which contains the total value
- the **timeline attribute** which will be used to get the time increments between 2 total values
- the **measure** to compute, there can be several of them for a given table, for each specify the **cummulative value attribute** and the **incremental value attribute**
The primary keys of the table are automatically assigned.
### Action's example JSON configuration
Like any other DPE actions, you can also configure the action in "Advanced Mode" by editing its JSON configuration, a snippet of which is shown below.
```json
{
"table": "dwh/default_dataset/posts",
"attribute_timeline": "date",
"attribute_total": ["total_impressions"],
"attribute_diff": ["incr_impressions"],
"table_attribute_created_at": "ref_posts",
"attribute_created_at": "post_createdat",
}
```
### Action's Parameters
Below is a short description of all available attributes in the JSON configuration.
| Parameter | Description | Example | Required |
| :-------------------------------: | ----------------------------------------------------- | ----------------------- | -------- |
| **table** | Name of the table on which to run the diff action | Dwh/agr\_site\_date | Yes |
| **attribute\_timeline** | Name of the attribute of timeline | date | Yes |
| **attribute\_total** | List of attribute names with totals. | \["total\_impressions"] | Yes |
| **attribute\_diff** | List of attribute names with incremental values. | \["incr\_impressions"] | Yes |
| **table\_attribute\_created\_at** | Optional table containing date of creation of ids | ref\_posts | No |
| **attribute\_created\_at** | Optional attribute containing date of creation of ids | post\_createdat | No |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-delete.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Empty tables with the Delete action
## Objective
This action is used to empty one, or multiple, tables in the database. Using the perimeter in the preferences section, you can limit the deletion to a specific subset of data. The perimeter can be preformed on:
- Based on predefined set (A table attribute)
- Date perimeter of the workflow
- Based on attribute of the Project (An attribute of the Project)
:::info
_Pro-tip_
The perimeter type _Based on predefined set_ can have multiple values, but only one attribute can be selected. Whilst unorthodox, you can use segmentation to act as an additional attribute based on a table attribute.
:::
## Setup example
```json
{
"params": {
"tables": [
"dwh/default_dataset/table_1",
"dwh/analytics_dataset/table_2"
]
}
}
```
## Parameters
| Parameter | Description | Example | Required |
| :--------: | ----------------- | ---------------------------------------------------------------- | -------- |
| **tables** | Tables to delete. | `["dwh/default_dataset/table1", "dwh/analytics_dataset/table2"]` | Yes |
\*This parameter should be used carefully, as if the action terminates early, it will result in partial deletion of data, and if multiple tables are selected, some tables may not be deleted at all.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-delete-diff.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Delete Diff Action
## Objective
This action is used to delete obsolete entries that existed at one time but no longer exist. Compatible with all Data Platform data engines.
## Usecase example
An erroneous data entry was made to a source file. The DPE automatically inserted the entry into the primary and aggregate tables. Once the incorrect entry was discovered, it was deleted from the source, however, it still remains in the primary and aggregate tables.
## How to use it
When you need to delete items that no longer exist, we recommend creating a temporary table that will serve as a work zone, containing only the data you wish to keep. This will allow you to set up a perimeter on the date field which allows you to act only for a given duration. If you have to act on all the data, then the easiest way is not to use delete diff, but to simply empty the tables beforehand with flush tmp and to fill it with the new data via the source.
Create a delete diff action for each table you wish to remove the records from, using the temporary table as the reference table. Select the attribute you wish to preform the comparison on. Optionally, set a date perimeter via the preferences section.
## Advanced mode example
```json
{
"diffToCheck" : [
{
"tablecheck": "dwh/default_dataset/testfile",
"tabletodelete": "dwh/default_dataset/testfile2",
"uniqFields": "attr1,attr2"
}
]
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-delete-bucket.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Delete Bucket Action
## Objective
The _Delete Bucket_ action allows you to **remove data from a bucket, including both folders and files.** You can choose to delete the entire bucket contents or target specific paths for deletion.
## Configure a Delete Bucket action
To create a new action, navigate to the **Action tab** of the Data Processing Engine (DPE), click on **New Action** and select the _Delete Bucket_ action.
### Choose items for deletion
Upon selecting the _Delete Bucket_ action, you will be presented with a list of your [Project buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) on the left side of the screen. This panel allows you to browse and select the data you wish to delete.
1. **Select the target bucket:** Locate the bucket from which you want to delete data.
2. **Expand bucket contents (optional):** To see the folders and files inside a bucket, click on the **name of the bucket**. This will expand the view, showing its contents.
3. **Choose items for deletion:**
- To delete an **entire bucket** and all its contents, click the **right arrow icon** (▶) next to the bucket's name.
- To delete a **specific folder** (and all its contents), navigate into the bucket and click the **right arrow icon** (▶) next to the desired folder's name.
- To delete an **individual file**, navigate into the bucket and folder where the file resides, then click the **right arrow icon** (▶) next to the file's name.
4. **Review selected items:** As you click the right arrow, the selected buckets, folders, or files will be moved to the right column. This column indicates all items that are currently marked for deletion.
Carefully review the items in the right column to ensure you are deleting only what is intended. Once satisfied with your selection, proceed with the final steps as prompted by the DPE interface to confirm the deletion. **Be aware that deletion actions are permanent and cannot be undone.**
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-rebuild.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Rebuild a table from scratch with the Rebuild action
## Objective
The _Rebuild_ action **entirely rebuilds an existing table** by creating a new physical table with the same schema and inserting all of its data into it.
It is mainly used to change field properties that cannot be modified once a table is built, most notably adding a **Required** or **Required + Identifier** constraint to an attribute (see [attribute constraints](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints)).
## Why rebuild a table
Rebuilding a table has several benefits:
- **Change field properties** such as _Required_, which [cannot be done once a table is built](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints).
- **Restructure partitioned data.** If the table uses partitions, all data is rewritten using those partitions. Note that when partitions are added to an existing table, only new data is stored under those partitions. The Rebuild action lets you properly restructure _all_ of the data.
- **Clean metadata and storage.** All metadata and history is removed to keep a single clean version of the table. This can reduce storage and improve performance, as Iceberg optimizes the way it stores data.
:::warning
The Rebuild action removes **all of the table's history**. Once it runs, you can no longer use the [Time Travel](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md) feature to roll back to a previous version. It is a fresh start.
:::
## Configure a Rebuild action
To create a new _Rebuild_ action, navigate to the _Actions_ tab of the Data Processing Engine (DPE), click on **New Action**, and select the _Rebuild_ action. Then select the table to rebuild and adjust its field configuration (required flag, identifier flag, and default value).
### Default values
When changing an attribute's property to **Required** or **Required + Identifier**, you must set a **default value**. This value replaces any NULL values found for that attribute during the rebuild, ensuring the new required constraint is satisfied.
### Keep backup table
The **Keep backup table** option keeps a backup of the previous table, _with its history_. The backup table is named with the prefix `_back_{hash}` (for example, `my_table_back_d26433d4`). You can find this name in the execution logs.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-deduplicate.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Remove duplicate rows with the Deduplicate action
## Objective
The _Deduplicate action_ removes duplicate rows from a table, **in place**, keeping exactly one row per **deduplication key**. The typical use case is a table that was loaded in _append_ mode when _upsert_ was intended: deduplicating on the table's identifier fields produces exactly the table that upsert would have produced. Like any other action, it can be run on demand, scheduled with a cron trigger, or orchestrated inside a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md).
The action works on tables stored in **lakehouse (Iceberg)** and **PostgreSQL** datasets.
:::warning
Deduplicate modifies the target table itself. On lakehouse tables the previous state remains available as a snapshot, so a run can be undone with time travel. On PostgreSQL tables the removal is **permanent**: enable the backup option before deduplicating anything you are not ready to lose.
:::
## Configure a Deduplicate action
The configuration screen has two panels: **Table configuration** on the left, where you pick the table, the deduplication key and the row to keep, and **Backup & scan** on the right, where you protect the run and preview its effect before creating the action.
### Choose the table
Select the target table from the **table selector**. Each entry displays the table name together with the dataset it belongs to. The action name is prefilled from the selected table, for example _Deduplicate - orders_.
### Choose the deduplication key
The deduplication key is the set of columns that defines a unique row: after the run, the table contains exactly one row per distinct combination of these columns.
When the table declares **identifier fields**, they are displayed above the key selector and the key is prefilled with them. This is the recommended key: it deduplicates the table exactly the way an _upsert_ load would have deduplicated it on ingestion.
You can select any set of existing columns instead. If your selection differs from the table's declared identifier fields, a warning reminds you that rows which are not duplicates by the table's own definition may be removed. If the table declares no identifier field, pick the column or columns that define a unique row for your data.
:::info
Rows whose key columns are empty group together, following standard SQL `GROUP BY` semantics: two rows with NULL in the same key column count as duplicates of each other.
:::
### Choose the row to keep
When several rows share the same key, the **Row to keep** setting decides which one survives:
- **Any remaining row** (default): the survivor is arbitrary. This is the right choice when duplicated rows are identical copies. If duplicated rows differ outside the key, the kept row is arbitrary.
- **Most recent / oldest entry**: deterministic. Choose **Most recent** or **Oldest**, then select the **Order by column**: a date, timestamp or numeric column that reflects recency, such as an update timestamp, a version number, or an auto-increment ID. Rows with an empty ordering column rank last: a row with a value always wins over a row without one.
### Keep a backup
Turn on **Keep a backup of this table** to create a full copy of the table before anything is removed. The copy is named `
_backup_` and registered in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) like any other table.
The backup is created only when the run actually has duplicates to remove, and its row count is verified before the target table is touched: if the backup cannot be created or verified, the run stops and the table is left untouched. The backup table is never deleted automatically, even when a run fails; delete it yourself once you no longer need it.
On lakehouse tables, a snapshot of the current state is kept in any case, so the backup is an extra safety net. On PostgreSQL tables, the backup is the only way to recover removed rows.
### Scan for duplicates
Before creating the action, click **Scan** to get a read-only estimate of what the current configuration would remove. The scan never modifies the table. It requires a selected table and at least one key column.
The result card shows:
- how many rows would be removed, and the survivor rule that would apply;
- how many keys have more than one row, and the worst per-key copy count;
- whether the previous state will remain available (lakehouse snapshot) or the removal is permanent (PostgreSQL).
If the scan finds no duplicates for the key, the card confirms there is nothing to remove. If more than half of the table would be removed, an extra warning asks you to double-check that the key really defines a unique row: removing most of a table usually means the key is wrong, for example a lone date column on a table with many rows per day.
The scan result is discarded whenever you change the table, the key or the survivor rule: run it again to get a fresh estimate. The same statistics are recomputed at the start of every run, so the run confirms what the scan previewed.
## How deduplication runs
The mechanism depends on the engine of the dataset that holds the table:
| Engine | Mechanism | Undo |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Lakehouse (Iceberg)** | The surviving rows are written to a temporary table, then the target table is emptied and refilled from it. If anything fails mid-run, the table is automatically rolled back to its pre-run snapshot. | The pre-run state remains available as a snapshot: restore it with [time travel](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md). |
| **PostgreSQL** | A single atomic `DELETE` statement removes the duplicate rows. | None: the removal is permanent. Use the backup option. |
Every run starts by scanning the table. When the scan finds no duplicates for the key, the run ends immediately and the table is not touched.
The action removes rows only: it never modifies the table's schema, partitioning or identifier fields.
:::info
On lakehouse tables, a transient `
_dedup_tmp` table may appear in the dataset while the action runs. It is dropped automatically at the end of the run, and a leftover from a hard failure is cleaned up at the start of the next run.
:::
:::warning
Do not write to the target table while the action is running: concurrent writes are unsupported. On lakehouse tables the conflicting commit fails and the automatic rollback restores the pre-run state.
:::
## Check the outcome of a run
Each run writes a summary to the action's execution logs: the number of rows before and after, the number of rows removed, the number of keys that had duplicates, the key that was used, and how to undo the run. When the backup option is enabled, the summary also names the backup table. For example:
```text
Removed 42 duplicate rows across 17 keys from 'orders' (1042 rows before, 1000 after),
key: (order_id). Previous state: snapshot 4632019173921515424 (restore via time travel
within the snapshot retention period, 14 days by default, see
/lakehouse-manager-time-travel).
Backup kept: orders_backup_a1b2c3d4.
```
## Undo a deduplication
- **Lakehouse tables**: the pre-run snapshot id is printed in the run's logs. Restore the table to that snapshot with [time travel](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md), within the snapshot retention period (14 days by default).
- **PostgreSQL tables**: there are no snapshots, so the removal itself cannot be undone. If the run was created with the backup option, the `
_backup_` copy holds the exact pre-run content of the table.
## Examples
Select a tab below to see a typical configuration, shown as the JSON you would see in [Advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-deduplicate.md#use-the-advanced-mode):
**Fix an append**
A load ran twice in _append_ mode, so every row of `orders` exists twice. The rows are identical copies, so any survivor works. Deduplicating on the table's identifier field `order_id` produces the table the upsert would have produced:
```json
{
"table": "dwh/sales/orders",
"keys": ["order_id"],
"survivor": { "mode": "any" }
}
```
**Keep the latest version**
The `customers` table accumulated several versions of each customer over time. Keep only the most recent row per `customer_id`, based on the `updated_at` column, with a backup of the table just in case:
```json
{
"table": "dwh/crm/customers",
"keys": ["customer_id"],
"survivor": { "mode": "latest", "order_by": "updated_at" },
"backup": true
}
```
**Composite key on PostgreSQL**
A PostgreSQL table of sensor measurements should have one row per sensor and per measurement time; when several rows exist, the first ingested one is the trustworthy one. Removal is permanent on PostgreSQL, so the backup is enabled:
```json
{
"table": "dwh/telemetry/measurements",
"keys": ["sensor_id", "measured_at"],
"survivor": { "mode": "earliest", "order_by": "ingested_at" },
"backup": true
}
```
## Use the Advanced mode
If you need to access the JSON configuration file of the action, activate the Advanced mode by clicking on **Advanced** at the top of the page.
The action takes the following parameters:
| Parameter | Type | Required | Description |
| ------------------- | ---------------- | -------------------------- | --------------------------------------------------------------------------------------------------- |
| `table` | string | yes | Target table, as `dwh//
`. The graphical editor builds this path for you. |
| `keys` | array of strings | yes | The deduplication key: one or more existing columns of the table. |
| `survivor` | object | no | Which row survives per key. Defaults to `{"mode": "any"}`. |
| `survivor.mode` | string | no | `any` (arbitrary survivor), `latest` or `earliest` (deterministic, ordered by `survivor.order_by`). |
| `survivor.order_by` | string | with `latest` / `earliest` | The ordering column: a date, timestamp or numeric column. Must be omitted when `mode` is `any`. |
| `backup` | boolean | no | Create and register a backup copy of the table before deduplicating. Defaults to `false`. |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-maintenance.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Optimize table storage with maintenance actions
## Objective
Users want the data to load almost instantly in their final consumer endpoint whether an API or a visual dashboard application. For this purpose, Data Platform works with caches to pre-compute the result of queries or pre-load the general information about the data model.
However, this also means that cache & metadata need to be managed and updated regularly during your processing procedures. Within the marketplace of actions in the Data Platform Store, you will therefore find 3 types of **maintenance actions**:
- **Update Metadata**: which updates the metadata of the data model.
- **Flush Cache: Analytics Manager**: which updates the cache of the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) to display the latest data from the data model.
- **Flush Cache: App Services**: which updates the cache of the [deployed API](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) in your project to display the latest result of each queries.
All three are best run at the end of a workflow that affects the data model. We highly recommend to simply append them by default at the end of each workflow.
## Automated Maintenance
While it can be useful to precisely choose when these actions are run during a workflow for optimisation purposes, we understand that it can be cumbersome to manage. We therefore suggest 2 simple ways you can append this maintenance routine to your processing procedures.
### Auto Flush-all Workflow
When you create a new Project, we advise you create a workflow _\_flushall_ which contains 4 actions that run sequentially in 4 stages: Update metadata, Flush Cache: Analytics Manager, Flush Cache: App Services and then Purge Buckets.
:::warning
The **Purge Buckets** action should be deactivated by default (or not included if you foresee no immediate use) as it empties all the data currently stored in the object store. This action is to be used with caution as it will **permanently** delete all archived information.
:::
### Auto Flush-all Options
Another way to run this maintenance routine is simply to activate the "Automatically flush all caches" option for your workflows, located in the preferences tab of your workflow. This will run the 3 actions: Update metadata, Flush Cache: Analytics Manager, Flush Cache: App Services.
## Update Metadata
The Update Metadata action is used to **update the metadata of the data model** stored either in the data warehouse or the datastore. It is strongly advised to run it at the end of the execution of a workflow which usually affects the data model. The metadata of the data model is used by other components to navigate the data model such as the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md), or the [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md).
When creating a new Update Metadata action, no configuration is required.
:::info
Note that **the action can be quite intensive and run for a long time** if the size of your data warehouse is large. We therefore recommend running them once per workflow not systematically after each action, to optimize your execution runtime.
:::
## Flush Cache: Analytics Manager
This action is used to flush the [cache](https://developer.mozilla.org/en-US/docs/Web/API/Cache) of [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md). It is highly recommended to launch it at the end of a workflow. This guarantees that the Analytics Manager uses the latest metadata from [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
The action takes an optional `params.timeout` parameter, see [Setting an action timeout](https://docs.dataplatform.ovh.net/dpe-actions-maintenance.md#setting-an-action-timeout). Its JSON configuration looks as follows:
```json
{
"display_name": "Flush Cache: Analytics Manager action",
"action": "flush-cache-qb",
"params": {
"timeout": "2h"
}
}
```
## Flush Cache: App Services
This action is used to **call the [Project's APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) in order to flush its [cache](https://developer.mozilla.org/en-US/docs/Web/API/Cache)**. It is highly recommended to launch it at the end of a workflow. This guarantees that the API displays the latest values of pre-configured queries or attributes.
The action takes an optional `params.timeout` parameter, see [Setting an action timeout](https://docs.dataplatform.ovh.net/dpe-actions-maintenance.md#setting-an-action-timeout). Its JSON configuration looks as follows:
```json
{
"display_name": "Flush Cache: App Services action",
"action": "flush-cache-front",
"params": {
"timeout": "2h"
}
}
```
## Setting an action timeout
Both flush cache actions accept a timeout:
| Parameter Name | Description | Examples | Required |
| :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------- | :------- |
| params.timeout | Maximum duration of the action, if the action takes longer to execute, it will cut off before the end of its execution. This can lead to partial data insertions or other problems. To be used with great caution for specific cases only. | Format using Xd Xh Xm Xs | No |
In order to add an action timeout, go to the _preferences_ and enable the timeout parameter, adding the desired time, or alternatively, edit the JSON configuration of the action.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Send OpenLineage Events
## Objective
The **Send OpenLineage Events** action forwards data lineage events from your Data Platform project to an external lineage solution connected via the [OpenLineage consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md). It reads lineage events recorded by the platform and pushes them to your lineage backend using the [OpenLineage](https://openlineage.io/) standard.
:::info
Before using this action, you must configure an [OpenLineage consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md) in the Connectors to define where events should be sent.
:::
## Requirements
1. You have an external lineage solution with an OpenLineage-compatible HTTP API (e.g. [Marquez](https://marquezproject.ai/)).
2. You have configured an [OpenLineage consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md) with the endpoint URL and auth token of your lineage backend.
## Create a Send OpenLineage Events action
1. Navigate to the **Actions** tab in the Data Processing Engine.
2. Click **New action**.
3. Select **Send OpenLineage Events** from the action store.
This action requires no additional configuration. It automatically reads the endpoint and authentication token from your [OpenLineage Consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md) settings and uses them to forward events.
### How it works
Once triggered, the action processes all events stored in the `queued/` folder of the **DWH Bucket** and sends them to the configured OpenLineage endpoint.
### File handling
- **Successful sends**:
- If `keep events` is set to `true` in the [OpenLineage Consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md), successfully sent files are moved to the `success/` folder in the DWH Bucket.
- Additionally, a summary log file is created under `openlineage/events/success/log/`.
- **Errors**:\
In case of failure, the events are moved to the `error/` folder, and a detailed log file is generated under `error/logs/` containing more information about the error.
## Execution modes
You can run the _Send OpenLineage Events_ action in two ways depending on how frequently you want your lineage solution to be updated.
### Cron job
Schedule the action to run at a fixed interval using a cron expression. This is the recommended mode for most use cases where near-real-time lineage is not required.
To configure a cron schedule, open the action's **Settings** tab and set the trigger to **Cron**. Enter your desired cron expression (e.g. `0 * * * *` for every hour).
:::info
See [Time-Based Trigger configuration](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#time-based-trigger) for more details on scheduling jobs.
:::
### Always-up
Set the action to run continuously as a long-running process. In this mode the action stays active and forwards lineage events as soon as they are produced, keeping your lineage solution in near real time sync with Data Platform.
To enable always-up mode, open the action's **Settings** tab and set the trigger to **Always up**.
:::warning
Always-up actions consume compute resources continuously. Use this mode only when near real-time lineage is a requirement.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-workflows.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Orchestrate ETL pipelines with Data Platform workflows
## Objective
Workflows are basically your orchestrated ETL and data processing pipelines on the Platform. A workflow is an organizational structure in the Data Processing Engine which represents **a sequence of [actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md)** which can be:
- Run manually or through an API call
- Scheduled to run at a given time (CRON)
When **creating workflows**, you can organize actions into stages to customize the processing of your data, as stages are run sequentially and actions in parallel.
Each time a workflow is launched, you can **monitor the execution of a workflow** by reviewing the logs.
## Create a workflow
The Platform offers a simple point-and-click interface to orchestrate your actions into scheduled workflows. The creation of the workflow has 2 main steps:
- **Stages**: Configure the workflow's successive stages, actions in the respective stages can be either added upon the creation of the workflow or later on.
- **Preferences**: Users can define various parameters and information about the workflow
### Stages
Within a workflow, actions are organized in **sequential** stages. Within stages, actions are run in **parallel** (meaning they can run [in parallel on multiple instances](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally))
If a workflow contains actions with multiple language versions, it will be executed with the **latest language version** it contains. You can override this behavior from the [workflow's preferences](https://docs.dataplatform.ovh.net/dpe-workflows.md#preferences)
:::warning
At the moment, a single workflow **cannot** contain both [PySpark actions](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md) and normal actions.
:::
:::info
A workflow must contain **at least one action** before it can be started. Empty workflows cannot be run.
:::
### Preferences
The preferences tab allows the user to give the workflow a name, a description and tags; and to define several [execution parameters](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md).
:::info
**What does the "Auto Flushall" do?** The platform stores the output of the queries in cache (in other words they are "pre-computed") for better performance when using an application. By running the "Flush all" the queries are re-run and the data in the app refreshed. It is therefore activated by default & usually adds an additional 5-10s of processing time to your workflow. For complex successive data processing workflows, you might want to deactivate it and run it only once at the end as a separate workflow.
:::
### Advanced mode
You can also go under the hood by using our integrated IDE in the Advanced mode: the **advanced** tab is available to further customize your workflow using the JSON config files.
[Learn how to configure a workflow using the Advanced mode](https://docs.dataplatform.ovh.net/dpe-workflows-advanced-mode.md)
## Manage workflows
In the workflows screen, you can quickly see which workflows are being run, have successfully ran or have encountered errors during their execution.
By hovering on the _More_ icon (i.e. "...") on each line of a workflow, you can also easily:
- **Delete** a workflow
- **Edit** an existing workflow
- **Run** a workflow
- **Inspect the logs** of a workflow
When the number of workflows increases, you can organise the view using _Folders_ or look for a specific workflow using the search bar on the top right of the screen. To add workflows into folders, simply drag & drop a workflow inside a given folder.
## Monitor a workflow
### Logs
To monitor a workflow's progress, it is possible to open its logs by clicking on the **Console icon**. This will open the [Control Center Logs Explorer](https://docs.dataplatform.ovh.net/control-center-logs-explorer.md) in a new window.
Note that in the [preferences](https://docs.dataplatform.ovh.net/dpe-workflows.md#preferences) of a workflow, you can define a level of information available in the logs (debug, info, notice, warning, error, critical).
### Alerts
Just like for actions, you can monitor the execution of a workflow by setting alerts on some key execution metrics (like job success, runtime, memory used, etc.). This will create an [alert](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md) in the [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-workflows-advanced-mode.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure a workflow in the Advanced mode
## Objective
Workflows can also be configured directly using a JSON configuration. To access the JSON configuration for a new workflow, click on the _Expert_ tab of the "New Workflow" or "Edit Workflow" window.
## Discover the JSON configuration
Users can type directly in the JSON editor to configure a Workflow. The JSON configuration includes more options than the ones available via the GUI.
### Workflow parameters reference
In alphabetical order, the parameters are the following:
| Parameter | Type | Description |
| :------------- | :---------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| display\_name | String | Allows to set a name of the workflow |
| description | String | Allows to set a description to the workflow |
| workflow | Table | List of stages with their associated actions |
| logLevel | String | Log level value defines the minimum log level for this workflow. It can take one of the following value _debug_, _info_, _notice_, _warning_, _error_ or _critical_ |
| auto\_flushall | Boolean | Condition to append or not the Flush All routine at the end of the workflow. This is a maintenance routine which refreshes all caches in the data flow (QB, API and APP) |
| dateMin | String | Absolute lower limit of the date perimeter |
| dateMax | String | Absolute higher limit of the date perimeter |
| days | JSON Object | Minimum and maximum values for the relative dates of the perimeter value |
### Example JSON configuration
Below is the JSON configuration of a typical workflow. It has 2 stages with respectively 2 actions in "Stage 1" and 2 action in "Stage 2".
```json
{
"params": {
"log_level": "info"
},
"auto_flushall": true,
"display_name": "flushall",
"description": "routine maintenance workflow",
"stages": [
{
"stageName": "Stage 1",
"actions": [
{"action_conf": "5c9bba68555fb219eb000020"},
{"action_conf": "5c9bba68555fb2190000ll99"}
]
},
{
"stageName": "Stage 2",
"actions": [
{"action_conf": "5c9bba68555hghjgjhgjhgjh"}
]
}
]
}
```
### Date perimeter parameters
To define the date perimeter you do so using an **absolute** date limit (e.g. set in time) or a **relative** date limit (e.g. calculated off the curent date).
To define an **absolute** date perimeter use the _dateMin_ & _dateMax_ parameters and enter the dates as string using the format YYYY-MM-DD. For instance, this would let you recompute a workflow between specific dates in the past.
```json
{
...
"dateMin": "2019-04-25",
"dateMax": "2019-04-25"
}
```
To define a **relative** date perimeter use the _days_ parameters and enter distance from the current date in days of the lower & upper limits. For instance, this would be used in a daily workflow that would need to only affect data from "the day before yesterday" and "today". The number corresponds to the number of days to subtract from today's date.
```json
{
...
"days": {
"min": "2",
"max": "0"
}
}
```
### Stage parameters
The parameters of stages can also be modified directly from the user interface.
| Parameter | Type | Description |
| :---------------- | :------ | :----------------------------------------------------------------------------------------------------- |
| stageName | String | Allows to name each stages |
| disable (Stages) | Boolean | Allows to enable / disable specific stages within a workflow |
| action\_conf | String | Corresponds to the action unique ID which you can find when accessing individual action's information. |
| disable (Actions) | Boolean | Allows to enable / disable specific actions within a stage |
```json
{
"display_name": "flushall",
…
"workflow": [
{
"disable": false,
"stageName": "update_metas_lite",
"actions": [
{
"action_conf": "update_metas_lite",
"disable": false
}
]
…
]
}
```
## Override configurations
### Action configurations
Overriding action configurations lets you reuse the same action in several workflows with different execution parameters without affecting the actions' configurations themselves. For instance lets assume that you have set the perimeter of an action, with a specific set of values: `"perimeterValues":["1","2"]`
You can reuse this same action with different perimeter values in the same workflow as follow:
```json
{
"logLevel": "info",
"auto_flushall": true,
"display_name": "flushall",
"workflow": [
{
"stageName": "Stage 1",
"actions": [
{"action_conf": "5c9bba68555fb219eb000020", "perimeterValues": ["1","2"]},
{"action_conf": "5c9bba68555fb219eb000020", "perimeterValues": ["8"]}
]
}
]
}
```
:::info
This specific example is meaningless but shows you how you can easily define different configurations for the same actions in a single workflow.
:::
### Configuration priorities
Configurations can be overriden at the 3 different levels: workflow, stages or actions. The most granular level of configuration applies to all sub-objects. For instance configurations set at the stage level will be overriden by any configurations set at the action level. Let's use the following example to illustrate the priority order:
```json
{
"logLevel": "info",
"perimeterValues": ["0"],
"auto_flushall": true,
"display_name": "flushall",
"workflow": [
{
"stageName": "Stage 1",
"perimeterValues": ["1"],
"actions": [
{"action_conf": "abc", "perimeterValues": ["2"]},
{"action_conf": "efg"}
]
},
{
"stageName": "Stage 2",
"actions": [
{"action_conf": "hij"}
]
}
]
}
```
Following the JSON configuration above, the perimeter values for the different actions run in the workflow will be as follow:
| Action ID | perimeterValue | Explanation |
| :-------- | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| abc | 2 | The action _abc_ has perimeterValues set at the action level which overrides any other perimeterValues. |
| efg | 1 | The action _efg_ has no perimeterValues set at the action level which means it uses the perimeterValues set for the stage it is contained in, in this case _Stage 1_. |
| hij | 0 | The action _hij_ & _Stage 2_ both don't have any perimeterValues set, so the perimeterValues which will be applied is the one of the workflow. |
## Workflow timeout
If an action within a workflow takes more time than expected, it is possible to set a timeout option to the workflow gets interrupted. The timeout duration is set as the value of the _workflow\_timeout_ parameter in a human readable format such as `"worfklow_timeout":"1h 10m"`
Values are cumulative from one time unit to another. Supported time units include:
| Time Unit | Description |
| :-------- | :---------- |
| w | weeks |
| d | days |
| h | hours |
| m | minutes |
| s | seconds |
Below a JSON configuration example using the workflow timeout option:
```json
{
"logLevel": "info",
"auto_flushall": true,
"worfklow_timeout":"59m 50s",
"display_name": "flushall",
"workflow": [
{
"stageName": "Stage 1",
"actions": [
{"action_conf": "5c9bba68555fb219eb000020"},
{"action_conf": "5c9bba68555fb2190000ll99"}
]
}
]
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-notebooks.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Run Jupyter notebooks on Data Platform
## Objective
Notebooks are a built-in integration of [Jupyter Notebooks](https://jupyter.org/) on Data Platform. Coming with pre-loaded packages for data science, they allow you to perform non-real-time interactive statistical analysis and data visualization. Notebooks are highly connected to the rest of your Project, making it possible to read/write data in the Lakehouse Manager thanks to [our SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) or convert your notebook code into a [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) action.
## Create a Notebook
To create a notebook instance, head to the **Notebook** section of the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md).
To get started, click on **New notebook** in the top-right.
You have to choose a [container image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html) for your Jupyter environment, i.e. the set of packages and dependencies you want to have for your new notebook. Data Platform features two ready-to-run images which contain the most popular libraries for data analysis and data science.
- **Base Notebook:** Provides a clean Python 3.13 environment with no pre-installed scientific packages, ideal for a minimal setup where you install your own specific packages.
- **PySpark Notebook:** Designed for data science and big data tasks, including Python 3.13, popular libraries (pandas, numpy, scikit-learn, etc.), Apache Spark, and additional utilities.
- **Data Science Notebook:** Includes Python 3.13 with popular data analysis, visualization, and machine learning libraries pre-installed (pandas, numpy, matplotlib, scikit-learn, etc.), ideal for general data science workflows.
- **Tensorflow Notebook:** Builds on the Data Science Notebook by adding TensorFlow and deep learning dependencies, ready for AI and neural network model development.
Choose the notebook that best fits your project’s needs. Once you've made your selection, your notebook will be ready to use.
:::info
If you are new to creating notebooks, we recommend starting with a **Base notebook**. This provides a clean, lightweight Python installation with minimal dependencies, allowing you to install only the specific packages you need.
:::
You can then configure your notebook further before it is created. Once you press **Create**, the server for the notebook will be initialized with its specific environment and dedicated computing resources.
1. **Notebook Name Field** – Enter the name of the new notebook.
2. **Add Dependencies** – Click this button to add additional dependencies for the notebook.
3. **Instance Size Selection** – Choose the required instance size (DPU) for the notebook.
4. **Create Button** – Click this to create the new notebook after filling in the details.
## Notebook Dependencies
Your notebook's image already comes with some default packages, which you can check in the image description in the marketplace.
You can specify additional dependencies by clicking on the **Add** button in the _Dependencies_ panel.
Simply choose the type of dependency and enter the name of the package to install in the server.
You can add several packages at once by pressing _Enter_ every time, like tags.
### Types of Dependencies
There are two types of dependencies:
- **Python dependencies:** You can install any package, such as pandas or numpy.
- **Data Platform dependencies:** You can choose from a list of available packages managed by the Data Platform.
You can only use one type of dependency per notebook instance. Mixing both types within the same notebook is not allowed.
:::warning
**Important:** If you attempt to add or modify dependencies while a notebook is running, these changes will not take effect immediately. You must restart the notebook for the new dependencies to be applied. To restart, click the 'Stop' button to shut down the notebook instance, and then click 'Start' to relaunch it with the updated dependencies. Any unsaved work may be lost during this restart process, so be sure to save your work beforehand.
:::
:::info
If you need to install a dependency while your notebook is running, you can do so directly from the terminal within the JupyterLab environment. However, such installations are temporary and will not persist if the notebook is stopped and restarted. For permanent dependency additions, always use the "Add Dependencies" panel and restart your notebook.
:::
## Open a Notebook
Notebook instances are **automatically turned on** when you open them.
The notebook instance will consume [all assigned DPU](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units) for the duration it is turned on.
You can manually turn off a notebook by clicking on the _Stop_ button when it is active.
To open a notebook, **double-click** on it or press the button shown below. This will open the notebook in a new JupyterLab window.
Starting up a notebook might take anywhere between **3-5 minutes**, and up to **10 minutes** for the first time.
The [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/) interface allows you to work on your notebooks with all the features provided by Jupyter, such as having multiple _.ipynb_ files open at the same time, keyboard shortcuts, download and export options, and much more.
Every notebook also ships with the [Data Platform Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md): a side panel to browse your datasets and buckets, run read-only SQL queries, create ready-to-use notebooks wired to your own data, and [export a notebook as a custom action](#export-a-notebook-as-a-custom-action). The [marimo](https://marimo.io/) notebook environment is preinstalled in all images as well.
:::info
Use the Data Platform SDK to easily interact with the rest of your Project, typically to **import/process/export data** from and to your [Lakehouse Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
:::
## Export a notebook as a custom action
Once your notebook code is ready to run as a job, the [Data Platform Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md#export-a-notebook-as-a-custom-action) can turn it into a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) without leaving JupyterLab: the action is created from your notebook's code, and the **pip installs** found in your notebook are automatically converted into the action's dependencies. See the extension page for the full flow.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform Extension
## Objective
Every Data Platform notebook ships with the **Data Platform extension**: a companion panel inside JupyterLab that brings the platform to your notebook. Browse the **documentation**, explore your **datasets and buckets**, run **read-only SQL**, open **ready-to-use notebooks** wired to your own data, and **export a notebook as a custom action** when it is ready to run as a job. Nothing to install, no credentials to configure: it works the moment your [notebook](https://docs.dataplatform.ovh.net/dpe-notebooks.md) starts.
:::info
Browsing, previewing and querying are **read-only**: they never modify your data. The only things the extension creates are the notebooks you generate from the gallery, saved in your home folder, and the [custom actions you export](#export-a-notebook-as-a-custom-action).
:::
## Open the extension
There are two ways to open it, depending on how you like to work:
- **Side panel:** click the **Data Platform icon** in the left sidebar of JupyterLab. The panel opens next to your notebook, so you can explore data and write code side by side. Use the **expand button** in its top bar to switch to a full-width tab.
- **Full view:** click the **Data Platform** tile in the JupyterLab launcher (under _Other_).
The top bar is always available: a **global search** field, a **refresh** button that reloads the platform data (new tables show up immediately), and the **language** switch.
:::info
The extension is available in **English, French and Japanese**: use the **EN / FR / JP** switch in the top bar to change the language of the whole panel at any time.
:::
## Documentation
The **Documentation** tab shows the live platform catalog: complete action scenarios, SDK snippets, a PySpark section and a FAQ. It is the same content that powers the helpers of the platform console, always up to date.
Every snippet has a **copy button** and a link to the full documentation page. Combined with the table pin (see [Data](#data-browse-your-datasets) below), the snippets are filled with your real dataset and table names: what you copy runs as-is in your notebook.
## Data: browse your datasets
The **Data** tab lists the datasets of your project, the same ones you see in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md). Open a dataset to browse its tables: the search stays instant even with thousands of tables.
Expand a table to get everything you usually open the console for:
| Action | What you get |
| --------------------- | ----------------------------------------------------------------------------------------------- |
| **Schema** | Column names and types, in a compact scrollable grid |
| **Count rows** | The live row count of the table |
| **Preview** | The first 20 rows, with a **Copy for notebook** button that gives you the matching pandas code |
| **Copy as** | Ready-to-paste snippets: `connect()` call, SQL SELECT, filtered `select()`, pandas DataFrame |
| **Open in SQL** | Jumps to the [SQL tab](#sql-query-without-leaving-jupyter) with a query prefilled on this table |
| **Analyze (gallery)** | Jumps to the [gallery](#notebook-gallery) with this table preselected |
| **Use in snippets** | Pins the table: every documentation and gallery snippet is filled with its real name |
## SQL: query without leaving Jupyter
The **SQL** tab is a lightweight editor to query a dataset in read-only SQL. Pick the dataset, type a SELECT, press **Run** (or Cmd/Ctrl+Enter): the results show up right below, with the execution time.
A few guardrails keep it safe and fast:
- only **SELECT** queries are accepted: nothing can be modified from here;
- a row limit is added automatically if you forget one;
- **Copy for notebook** on any result gives you the exact SDK code to reproduce the query in a notebook cell.
## Buckets: explore your files
The **Buckets** tab is a file explorer over the buckets of your project: navigate folders, see file sizes. Every file comes with a copy-ready snippet to read it from your notebook.
## Notebook gallery
The **Gallery** is a set of ready-to-use notebooks, organized in packs. When you click **Create & open**, a pop-up lets you choose:
- **wire it to a real table of your project**: the notebook opens already connected to your data, ready to run top to bottom;
- **keep the placeholders**: you get a generic template to adapt later.
The notebook is created in your home folder and opens immediately as a JupyterLab tab.
| Pack | Notebooks |
| ------------------------------ | ------------------------------------------------------------------------------------- |
| **Getting started** | Data Platform 101: the guided tour of the platform on your project's real data |
| **Explore & assess** | Explore a table, Data quality report, Cross-dataset analytics |
| **Pipelines** | ETL table to table, Ingest an API into the DWH, Load bucket files into the DWH |
| **ML & Spark** | ML quickstart, Segmentation, Anomaly detection, Time-series forecast, PySpark starter |
| **AI (OVHcloud AI Endpoints)** | LLMs on your data, Talk to your data (SQL agent) |
The gallery adapts to your notebook image: the ML notebooks appear on **Data Science** and **TensorFlow** notebooks, and the PySpark starter on **PySpark** notebooks.
:::info
The **AI pack** uses [OVHcloud AI Endpoints](https://www.ovhcloud.com/en/public-cloud/ai-endpoints/) with your own API key. The rows you select are sent to the AI Endpoints API, so mind data sensitivity. See also [Jupyter AI Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-jupyter-ai-extension.md) for the chat assistant.
:::
## Export a notebook as a custom action
Once your notebook code is ready to run as a job, the extension can turn it into a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) without leaving JupyterLab, and without copying code around.
Start the export from the extension, then fill in the export form:
1. Select the **notebook** to export.
2. Pick the **repository** of the Data Processing Engine the action is exported to.
3. Choose whether to **create a new custom action** or **replace an existing action** with this export.
4. Give the action a **name**, and pick its type: **Custom Python** or **Custom PySpark**.
5. Click **Preview**.
The preview sums up what will be created: target repository and version, action name and type, entry function, code size, and the dependencies detected from the notebook's **pip installs**, automatically converted into action dependencies. Below the summary you can review the generated code: non-code cells are dropped, and the code is wrapped in a `def customfunc(event):` entry function to match the action format.
Happy with the result? Click **Create action**. The action is then available in the [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) screen like any other custom action: configure it, run it, or schedule it in a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md).
## Search everything
The **search bar** at the top looks across the whole extension at once: documentation snippets, datasets, tables you have browsed, and buckets. Results are grouped by type; clicking a table opens it directly with its details expanded.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-notebooks-jupyter-ai-extension.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Integrate OVH AI Endpoints with Jupyter AI
## Objective
The [Jupyter AI extension](https://jupyter-ai.readthedocs.io/en/v2/#) brings the power of generative AI directly into your notebook environment, allowing you to interact with large language models, generate code, explain concepts, and perform various AI-powered tasks without leaving JupyterLab.
**Data Platform Notebooks** come with the Jupyter AI extension **pre-installed**, allowing you to start leveraging AI immediately. By integrating your **OVH AI Endpoints**, you can leverage custom models and services directly within your notebooks.
This guide will walk you through the process of configuring and using the Jupyter AI extension with your OVH AI Endpoints on Data Platform.
## Requirements
Before you begin, ensure you have:
- An active Data Platform Notebook instance (Jupyter AI is pre-installed).
- An AI Endpoints API key for the OVH AI service you wish to integrate.
## Configure Jupyter AI with OVH AI Endpoints
Once your Data Platform notebook is open, you can configure the pre-installed Jupyter AI extension to connect to your OVH AI Endpoints.
:::info
**Important Notes:**
- This configuration needs to be performed **once per notebook instance**. It will be retained automatically for that specific notebook upon restart, but it is not synced across different notebook instances.
- Jupyter AI often configures models using environment variables. While you'll set these through the UI here, understanding this helps with advanced configurations or troubleshooting.
:::
1. **Access Jupyter AI Panel:** In your JupyterLab interface, locate and click the "Chat" icon on the left sidebar.
2. **Start Configuration:** Click on _⚙️ Start Here_ within the chat panel to begin the configuration process.
3. **Define Completion Model:**
- **3.1 Completion Model:** Select `OpenAI (General Interface)`. This will reveal further configuration settings for the model ID, base URL, and API key.
- **3.2 Model ID:** Enter the ID of your desired model from the [AI Endpoints Catalog](https://endpoints.ai.cloud.ovh.net/catalog). For example, let's use `Qwen2.5-Coder-32B-Instruct`.
- **3.3 Base API URL:** Enter the base URL for your OVH AI Endpoints, such as `https://oai.endpoints.kepler.ai.cloud.ovh.net/v1/`.
- **3.4 API Keys:** Enter your OVH AI Endpoints API Key in the designated field.
4. **Save and Return:** Click _Save Changes_, and then click the back arrow to return to the Jupyter AI Chat panel. Your Jupyter AI extension is now configured and ready for use.
## Using Jupyter AI in your Notebook
Once your OVH AI Endpoint is configured with Jupyter AI, you can interact with it directly within your notebook environment using the Jupyter AI Chat Panel.
### Jupyter AI Chat Panel
The **Jupyter AI chat panel**, powered by _Jupyternaut_, enables conversational interaction directly within JupyterLab. This is your primary interface for interacting with your configured OVH AI Endpoint. You can use it to get programming help, generate notebooks, and much more with ease.
You can:
- Ask questions about your data or code using the chat box.
- Generate new Jupyter notebooks from text prompts using the `/generate` command.
- Fix errors in selected notebook cells using the `/fix` command.
- Export chat history to a Markdown file using the `/export` command.
- Teach Jupyternaut about files on your system using the `/learn` command.
- Use `@file` to include selected file contents in your question for more accurate responses, providing context to the AI directly from your files.
For more detailed information on all capabilities, explore the [Jupyter AI documentation](https://jupyter-ai.readthedocs.io/).
## Example Workflow: Generating a Machine Learning Notebook from Kaggle Hub Data
Let's demonstrate a powerful feature: using the `/generate` command to create a new notebook that fetches data, performs data exploration, and builds a basic machine learning model. For this example, we'll use a commonly available version of the Kaggle Titanic dataset, accessible via a direct URL.
1. **Ensure your Jupyter AI is configured** as described in the "Configure Jupyter AI with OVH AI Endpoints" section, using your chosen model (e.g., `Qwen2.5-Coder-32B-Instruct`).
2. **Open the AI Chat Panel** (if not already open) by clicking the "Chat" icon on the left sidebar.
3. **Type the `/generate` command** followed by your detailed prompt into the chat input field:
```text
/generate Create a Python notebook to perform binary classification on the Titanic dataset. The notebook should:
1. Download the 'train.csv' dataset from 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv' using pandas.
2. Load the downloaded data into a pandas DataFrame.
3. Display basic information (first 5 rows, data types, descriptive statistics).
4. Handle missing values (e.g., fill 'Age' with the mean, 'Embarked' with the mode, and drop the 'Cabin' column).
5. Encode categorical features ('Sex', 'Embarked').
6. Split the data into training and testing sets (80/20 split), with 'Survived' as the target variable.
7. Train a RandomForestClassifier model.
8. Evaluate the model using accuracy and a classification report.
```
4. **Press Enter** to send your prompt. Jupyter AI will process your request using the configured OVH AI Endpoint. After a short time, it will generate a new `.ipynb` file in your current directory (e.g., `Titanic Binary Classification Notebook.ipynb`), pre-populated with Python code cells performing the requested steps.
5. **Open the generated notebook** and run the cells. You can then modify, expand, or debug the code as needed, using the Jupyter AI chat panel for further assistance. This significantly accelerates your data science workflow.
## Troubleshooting
- **`API Key not found` or `Authentication Error`:** Double-check the API Key entered during Jupyter AI configuration. Ensure it is correct and has the necessary permissions for your OVH AI Endpoint.
- **`Could not connect to the model` or `Bad URL`:** Verify the **Base API URL** provided during Jupyter AI configuration. Ensure the URL is correct and your OVH AI Endpoint is running and accessible.
- **No AI panel/button in JupyterLab:** If you don't see the AI panel or button, try refreshing your browser tab or restarting the Jupyter server. While Jupyter AI is pre-installed, sometimes a fresh load helps.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-environments.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Reuse execution configurations with environments
## Objective
Environments allow you to save and re-use quickly a set of [configurations and execution preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) in actions and workflows.
_Figure 1: The environment panel_
Environments are not permanent. At any time you can [change a jobs environment](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) even if one had already been applied.
:::info
If you make changes to any of the environments parameters, the changes will be reflected in all the actions and workflows to which the environment is assigned.
:::
- [Create an environment](https://docs.dataplatform.ovh.net/dpe-environments.md#create-an-environment)
## Create an environment
When you first access the **Environment** tab, you will be greeted with a blank canvas for you to fill your custom environments.
Click on **New environment** and a reduced version of the [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) section will appear.
From here, select your preferred _resources_, _options_ and _environment variables_. Don't forget to give your environment a name and a description.
[How to configure the execution preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Processing Engine jobs
## What is a job?
A job is a single execution of an action or a workflow. Every job carries its own execution environment: the [resources](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) allocated to it, its [segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md) and [perimeter](https://docs.dataplatform.ovh.net/dpe-jobs-perimeter.md) settings, and its environment variables. Those come from the [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) of the action or workflow, or from a shared [environment](https://docs.dataplatform.ovh.net/dpe-environments.md).
The Data Processing Engine is job-based. When a run is triggered, the platform builds a container matching the environment defined, executes it, then shuts it down, so resources are consumed only for the duration of the run.
The Jobs screen shows three states.
**Running** jobs stream their logs live, and the number that can run at once is capped by the project [quotas](https://docs.dataplatform.ovh.net/dpe-settings.md#quotas).
**Queued** jobs are runs ordered beyond that cap. Queuing is deactivated by default and released on request through support. Queued jobs start in the order shown on screen, top to bottom, as each running job finishes.
**Finished** jobs keep 30 days of history, each with per-run CPU and RAM statistics, and a result of successful, failed, or manually interrupted. A job that exceeds its timeout is flagged as such, to distinguish it from other failures. Runs of PySpark actions, and of the workflows containing them, open directly in the [Spark History Server](https://docs.dataplatform.ovh.net/dpe-spark-history-server.md).
Segmentation and perimeter carry vocabulary of their own. The two references below define the terms as they appear in the interface and in advanced mode.
[### Resources→
Size a job vertically, or spread it across workers.
](/dpe-jobs-resources)[### Segmentation→
Split one job into parallel tasks by date, value or account.
](/dpe-jobs-segmentation)[### Perimeter→
Filter the data a job processes before it runs.
](/dpe-jobs-perimeter)[### Execution preferences→
Timeouts, service account, write strategy, triggers and execution modes.
](/dpe-jobs-preferences)
## Segmentation and perimeter vocabulary
:::details Segmentation and perimeter vocabulary
**Segments**:
Subset of a data source. Segments for row / column sources can be defined based on a fixed number of lines, an attribute of the source (a date or a value) or a user account (for account based sources).
**Segmentation / To segment**:
Action to split a DPE action into multiple tasks that handle one or more segments.
**Segmentation values**:
Referenced values used to divide an action into smaller tasks.
**Segmentation type**:
Technical name in advanced mode: _segmentationValues_. Determines the process according to which the list of segmentation values are retrieved.
**Source Attribute / (Segmentation Variable)**:
Technical name in advanced mode: _segmentationVarName_. Determine which column in the source is used to apply the filter to the segmentation values
**Perimeter**: Filter the source of a DPE action.
**Perimeter type**: Technical name in advanced mode: _perimeterValues_. Determines how the list of perimeter values is retrieved.
**Perimeter values**: List of values to apply the scope.
**Source Attribute / (Perimeter Variable)**: Technical name in advanced mode: _perimeterVarName_. Determines which column in the source is used to apply the filter to the perimeter values.
**Bucket Size**:
Technical name in advanced mode: _segmentationChunkSize_. The number of segmentation values that each task handles.
:::
## Bucket size in detail
:::details Bucket size in detail
At any time it is possible to **group some tasks in buckets (chunk) to manage several segments of the data source**. When the list of segments (values, date or social network accounts) is consistent, it avoids generating too many tasks.
The bucket size sets how many segments each task handles:
- If you set the bucket size to 2, each task will handle 2 segments
- If you set the bucket size to 10, each task will handle 10 segments
- Etc.
Below is an illustration of the impact of setting the bucket size to 30 compared to a bucket size of 1. Assume we’re running the segmentation based on the date of the workflow which runs from Jan 1st of 2018 to Dec 31st of 2019.
| w/out Segmentation Chunk Size | with Segmentation Chunk Size |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ChunkSize = 1 | ChunkSize = 30 |
| | |
In the first case the DPE action will be split in 730 tasks (365 x 2), one for each day for 2 years. In the second case, the bucket size is set to 30, in which case the DPE action will only be split in 25 tasks (365 x 2 / 30) as each task handles 30 days.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure job execution preferences
## Objective
Each [action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) or [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) in the Data Processing Engine has several configuration settings which can be managed via the **Preferences** tab.
When an _action_ is executed, it will use those preferences to run.
When a _workflow_ is executed, its preferences will **override** the preferences of all the actions it contains. If a certain preference option is deactivated at workflow level (such as segmentation or perimeter) it will fall back to each action's preferences during the workflow's execution.
Those preferences can be templated to be re-used in one click every time by saving them in an [environment](https://docs.dataplatform.ovh.net/dpe-environments.md).
## Environments
Environments on the Platform are a set of pre-defined preferences that you can quickly assign to your actions or workflows, so you don't need to configure it every time.
[↪ Learn more about environments](https://docs.dataplatform.ovh.net/dpe-environments.md)
## Timeout options
:::info
Timeout options are not templated in an [environment](https://docs.dataplatform.ovh.net/dpe-environments.md), and are to be configured in the preferences in each action/workflow
:::
### Timeout
Timeout is an execution runtime duration after which your job is stopped and reported with the `TIMEOUT` status. Timeout duration includes the resource provisioning time, the build time, and the actual code execution duration.
The maximum timeout duration that you can set is 24 hours, except for the following types of jobs where you can set it at _null_, meaning infinite (no timeout will be enforced):
- A [Load](https://docs.dataplatform.ovh.net/dpe-actions-load.md) action having a streaming-type [connector](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) as a source
- A [Custom](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) action
If a job execution times out, it is flagged in the [jobs](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md) list with the `TIMEOUT` status, which differentiates it from other sources of failure.
### Stop workflow on failure
If the action is executed as part of a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md), and fails throwing a _CRITICAL_ error (which will be displayed in the logs), activating this option will interrupt the workflow's execution and the next stages or actions will not be run.
## Execution options
### Service account
Each action, workflow or environment has a [service account](https://docs.dataplatform.ovh.net/iam-service-accounts.md) associated to it, which **the action/workflow impersonates as it is executed**. This service account acts as the identity of the running job, and controls the access level of the job through the [service account's roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md).
:::info
Even if you launch a job manually, its execution will be authenticated by the associated service account.
:::
### Perimeter
:::info
This parameter is not available for [Custom PySpark actions](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md).
:::
When executing a data processing job, when you reach large volumes of data to process, you may need to filter the execution of an action. The _perimeter_ parameter allows you to **define a field on which to filter the data processing scope**.
There are several perimeter modes available depending on the data source and the type of action you are performing.
The scope serves in particular to:
- Apply a processing filter on a column;
- Make the execution of actions faster;
- Relieve the requested workload of certain data sources;
- In a workflow that is scheduled daily, do not process the data from the entire history, but only from the last X days.
[How to configure the perimeter](https://docs.dataplatform.ovh.net/dpe-jobs-perimeter.md)
### Logs level
The logs level defines the list of logs that will be stored during the action's or workflow's execution. The different levels range from _debug_ to _critical_. All the logs above the one selected (included) will be stored.
For PySpark actions, you set the log level of your own libraries and the Spark log level separately.
### Automatically flush all caches
When this parameter is activated, the components' caches will automatically be flushed after each execution of the action/workflow.
Learn more about caches in [this article](https://docs.dataplatform.ovh.net/dpe-actions-maintenance.md#auto-flush-all-options).
### Write strategy
:::info
This option is only available for the [Load](https://docs.dataplatform.ovh.net/dpe-actions-load.md), [Aggregate](https://docs.dataplatform.ovh.net/dpe-actions-aggregate.md), [Load PySpark](https://docs.dataplatform.ovh.net/dpe-actions-load-pyspark.md), and [Aggregate PySpark](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md) actions.
:::
The write strategy defines how the action writes new data into the destination table when rows match existing ones. Matching is based on the table's [identifier fields](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints).
There are three strategies:
- **Upsert:** the action matches new data against existing data using all identifier fields. If a new row has the same identifier fields as an existing one, the existing row is **updated**. Otherwise, the new row is added.
- **Insert when not matched:** if a new row matches an existing row (using identifier fields), nothing is done. Otherwise, the new row is added.
- **Insert all:** all new rows are inserted into the table, even if some match existing rows on their identifier fields. This may create duplicates.
:::info
**Load and Aggregate actions default to Insert all.** Select **Upsert** (or **Insert when not matched**) manually in the action preferences if you want matching behavior.
:::
#### Example
Consider the following `users` table:
| Firstname | Lastname | Age |
| --------- | -------- | --- |
| Alice | Smith | 42 |
| Bob | Duncan | 27 |
We run the action with this new data, using **Firstname + Lastname** as identifier fields:
| Firstname | Lastname | Age |
| --------- | -------- | --- |
| Alice | Smith | 43 |
| Charles | Jackson | 33 |
The result depends on the write strategy.
**Upsert:** Alice matches an existing row and is updated; Charles is new and added:
| Firstname | Lastname | Age |
| --------- | -------- | --- |
| Alice | Smith | 43 |
| Bob | Duncan | 27 |
| Charles | Jackson | 33 |
**Insert when not matched:** Alice matches an existing row, so nothing is done; Charles is new and added:
| Firstname | Lastname | Age |
| --------- | -------- | --- |
| Alice | Smith | 42 |
| Bob | Duncan | 27 |
| Charles | Jackson | 33 |
**Insert all:** both new rows are inserted as-is, leaving a duplicate Alice:
| Firstname | Lastname | Age |
| --------- | -------- | --- |
| Alice | Smith | 42 |
| Bob | Duncan | 27 |
| Alice | Smith | 43 |
| Charles | Jackson | 33 |
## Environment variables
:::info
This is only available for [Custom](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) actions, workflows, and environments.
:::
Environment variables are key-value pairs that you can template and inject into your custom actions, to avoid hardcoding the values in your custom script. They are saved in the `PARAMS` dictionary object of the `forepaas.core.settings` package (available in the SDK).
A code-snippet for a typical use case is available at [this page](https://docs.dataplatform.ovh.net/developers-python-sdk-use-environment-variables.md).
:::warning
Environment variables do not go through a KMS lifecycle and as such should not be used to store and inject secrets and credentials.
:::
## Triggers
:::info
Triggers are not templated in an [environment](https://docs.dataplatform.ovh.net/dpe-environments.md), and are to be configured in the preferences in each action/workflow.
:::
With triggers, you can set your jobs to run automatically. Typical usages include launching the job by accessing an API endpoint, or by scheduling time-based triggers (e.g. to run your jobs every day at midnight).
### API endpoint trigger
This endpoint lets you trigger the job via API.
It is always available to you through the trigger named _Launch API Endpoint_. Click on it to see the endpoint details.
In order to trigger your job, you will have to authenticate first with the _Authentication_ endpoint and then trigger the job with the _Launch Job_ endpoint.
1. You have to input your [API and Secret keys](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) in trigger details and then copy the _Authentication_ endpoint text. Once you login, you will get a token.
2. Input this token in the _Launch job_ endpoint and then access it to launch your job.
### Time-based trigger
Use the _CRON_ triggers to schedule executions of your action or workflow every pre-defined time period.
You can either use the _Complete mode_ to set the configuration visually or use the _Advanced mode_ for custom CRON expressions. The Advanced mode configuration follows the CRON syntax (described in detail [here](https://www.netiq.com/documentation/cloud-manager-2-5/ncm-reference/data/bexyssf.html)).
You can deactivate a cron trigger to pause scheduled runs, or activate it again when you want scheduling to resume, without losing the trigger configuration.
## Resources
### Execution modes
There are two execution modes available in the Preferences page:
- **Serverless**: computing resources deployed on the fly.
- **Always-up**: dedicated instances always running to swiftly execute your jobs whenever you want. For PySpark actions, this mode is in beta.
#### **Serverless**
By default, every job you run on the Platform is executed with the Serverless mode. An execution environment is deployed when you start the job execution and it is undeployed when this execution is done.
You will be [billed](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units) for:
- The computing resources used to perform the task.
:::info
In the Serverless mode, your execution environment is up only during the task execution because, once the task is done, it is automatically killed.
:::
#### **Always-up**
:::info
For PySpark actions, the Always-up mode is in beta: instabilities may occur.
:::
When you activate the Always-up mode, a dedicated computing environment is deployed for the respective action/workflow/environment. It will stay deployed until you set the Execution mode back to Serverless so you won't need to wait for its deployment every time you run the respective job.
You will be [billed](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units) for:
- The computing resources used in the time during which the execution environment was up (this includes the time taken to perform the jobs).
:::warning
With the Always-up mode, your execution environment will NOT be automatically undeployed once job execution is done. You have to set the Execution Mode back to Serverless to kill it.
:::
If you edit the configuration of an action/workflow set in Always-up mode, it will be **updated in a full blue-green way**: the deployment will wait until all current pending tasks are complete before gracefully shutting down for the update.
:::info
It is recommended to use the Always-up execution mode on [environments](https://docs.dataplatform.ovh.net/dpe-environments.md) rather than individual actions/workflows, in order to re-use the resources for multiple workloads.
:::
### Resource scaling
You can find out all about how to scale your computing power in the dedicated article below.
[Learn how to scale your jobs resources](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md)
## Parallelization options
### Segmentation
:::info
This parameter is not available for [Custom PySpark actions](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md).
:::
The **segmentation options allow you to define how a job will be broken down in multiple tasks** based on specific criteria. Once split, the workload of the job can then be [distributed across multiple workers](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally) in parallel (parallel computing).
Among most common usage, segmentation allows you to:
- **Apply a specific treatment filter** to a column;
- **Accelerate action run time** by parallelizing the processing on multiple workers in parallel and relieving the workload on data source connectors;
- In a workflow planned daily for instance, **not to ingest all of the historical data** each time you run the load action but filter on only the last day(s).
[How to configure segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md)
### Concurrent executions
This option allows the action/workflow to be executed multiple times at the same time (instead of returning an error if an execution is triggered while one is already happening).
In [Serverless mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#serverless), **each** concurrent execution will use the allocated computing power for the job.\
In [Always-up mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up), multiple concurrent executions are [distributed across the available workers](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally) in parallel (parallel computing).
Multiple concurrent executions can only be triggered [via API](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#api-endpoint-trigger), not by GUI.
## Development language
:::info
This is only available in workflows (and [Custom actions](https://docs.dataplatform.ovh.net/dpe-actions-custom.md)' configuration page).
:::
This option lets you specify which language version to use for the overall workflow. By default, a workflow will use the latest of all the versions it contains.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-jobs-resources.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Scale Data Processing Engine resources
## Objective
The Platform is designed to scale easily as the requirements of your data Project evolve.
In the Data Processing Engine, each individual action or workflow can be **scaled both horizontally and vertically**, meaning you can parallelize processing tasks and add more computing power to each indefinitely.
## Scale your jobs horizontally
You can **parallelize the workload** of your action or workflow by selecting the number of parallel instances in which your job will be ran. Typically, multiple instances will be used for the following processes:
- Using [segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md) for a job
- Running workflows with multiple actions in the same [stage](https://docs.dataplatform.ovh.net/dpe-workflows.md#stages)
- Running multiple [concurrent](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#concurrent-executions)
occurrences of the same [Always-up action/workflow](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up)
:::info
Failure to allocate multiple instances for the above processes will simply have the broken-down tasks executed back-to-back on the single instance.
:::
To change the amount of parallel instances allocated to an action or a workflow, go to its **Preferences** tab.
Change the number of instances in the Resources panel.
## Scale your jobs vertically
You can allocate **more computing power** to an action or a workflow by increasing the number of Data Platform Units (DPU) of each computing instance. The DPU is a unit of processing capability, representing access to a certain amount of CPU and memory, as such:
- **DPU**: - corresponds to approximately _1 vCPU_ and _4 GB of RAM_, based on hardware availability.
:::info
By default, newly created actions or workflows are allocated 2 DPU.
:::
To change the amount of DPU allocated to an action or a workflow, go to its **Preferences** tab.
Set the DPU size of [each parallel instance](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally) in the Resources panel. The panel shows the total DPU of the execution as you change the values.
The final total amount of DPU allocated to a DPE job is the number of parallel instances times the DPU size of each.
### PySpark actions
For [PySpark actions](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md), the Resources panel sizes the Spark executors and the Spark driver separately:
- **Executors**: the number of executors and the DPU size of each. Executors run the distributed part of your job.
- **Driver**: a single instance with its own DPU size. The driver coordinates the executors and runs the non-distributed part of your script, so size it independently of the executors.
The total DPU of the execution is the DPU of the executors plus the DPU of the driver, as shown in the panel.
:::info
The number of drivers is fixed at 1; only its size can be changed.
:::
:::info
If you are using the [Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md), you still are allowed to update your resources. The update will come into effect shortly after you change your settings and your workflows/actions will not be interrupted in the process.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Split large jobs with segmentation
## Objective
When executing an [action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) in the Data Processing Engine, it is important to think about the amount of data that will be processed. For large volumes, there is a risk to reach the maximum compute power of a worker. To be able to process these actions, you can split the execution of actions into a set of smaller tasks. This broken-down workload can then be shared [between multiple workers](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally) concurrently.
The segmentation options allow you to **specify how to break down the data workload**. The execution will be split into **multiple tasks**, each task managing a smaller set of values at a time.
:::warning
The segmentation options should be configured your segmentation carefully. Depending on your segmentation choices, your action could be much faster, or slower.
:::
There are several segmentation modes available inside the Platform depending on the data source and the type of action you are using:
1. [Based on the date perimeter of the workflow](#1-based-on-the-date-perimeter-of-the-workflow)
2. [Based on an attribute of a Project table](#2-based-on-an-attribute-of-a-project-table)
3. [Based on a predefined set](#3-based-on-a-predefined-set)
4. [Based on a fix number of lines](#4-based-on-a-fixed-number-of-lines)
5. [Based on the source's accounts](#5-based-on-the-sources-accounts)
6. [Based on files](#6-based-on-files)
7. [Automatic segmentation](#7-automatic-segmentation)
## 1. Based on the date perimeter of the workflow
If your source data contains a date attribute, it is possible to split the action in separate processes which will individually address a specific date or a group of dates (bucket). In this case, each task will apply a filter on the date that it needs to extract.
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle; **API**: Facebook, Twitter, Google Analytics, YouTube Private, YouTube Public |
### Bucket Size
If your source contains a significant number of dates, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time. This is particularly common when dealing with datetime type attributes that can record time information up to the milli-second unit.
To avoid generating too many tasks, you can make each task manage several dates at once by using the “bucket size” option. Simply set a size for your bucket to decide how many dates at a time you want your task to manage. The default “bucket size” value is 1, that is:
- If you do not set the bucket size, each task will handle 1 date
- If you set the bucket size to 2, each task will handle 2 dates
- If you set the bucket size to 10, each task will handle 10 dates
- And so on...
If you’re still unclear about what the “bucket size” does, find out more in the [bucket size reference](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#bucket-size-in-detail).
### Configuration
To configure this segmentation option select _Based on the date perimeter of the workflow_ in the "Segmentation Type" dropdown, and select the attribute.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "workflow_dates",
"var_name": "id2",
"chunksize": 1
}
```
[↪ Get further with this type of segmentation with our tutorial series](https://docs.dataplatform.ovh.net/tutorials-segmentation-workflow-dates.md)
## 2. Based on an attribute of a Project table
This option lets you segment an action using a list of values contained in the column of a [given table (usually a Reference Table) from your data model](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md). For instance, let’s consider the following case where we’re trying to set the segmentation options for an action which loads the data from the Source Table “orders” to the Destination Table “aggr\_date”:
| Reference Table | Source Table | Destination Table |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| | | |
Assuming, you want to aggregate “income” data per the date of orders “date\_order”. Let's say that the “tmp\_date” table serves as the reference table of all dates during which there was ever an order made, e.g. it contains a list of dates from 2019-01-01 to 2019-01-03.
By activating this segmentation mode, it will automatically generate a pre-stage to your main stage which contains your action, which will list the unique list of dates available in order to split your action in separate tasks as describe in the diagram below:
By doing so, the data from your Source Table “orders” will therefore be loaded to the Destination Table “aggr\_date” on day at a time.
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle |
### Bucket Size
If your source contains a significant number of unique values within the attribute used for segmentation, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.
To avoid generating too many tasks, you can make each task manage several values of your segmentation attribute at once by using the “bucket size” option. Simply set a size for your bucket to decide how many values at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:
- If you do not set the bucket size, each task will handle 1 value
- If you set the bucket size to 2, each task will handle 2 values
- If you set the bucket size to 10, each task will handle 10 values
- And so on...
If you’re still unclear about what the “bucket size” does, find out more in the [bucket size reference](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#bucket-size-in-detail).
### Configuration
To configure this segmentation option select _Based on an attribute of a Project table_ in the "Segmentation Type" dropdown.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "dwh_attributes",
"var_name": "id",
"attribute": "name",
"object": "s3.deldiff/delete_diff.csv",
"attribute_type": "map",
"chuncksize": 2
}
```
[↪ Get further with this type of segmentation with our tutorial series](https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md)
## 3. Based on a predefined set
Use this mode in order to use a set of hard-coded values as your segmentation criteria. For this you must choose the column of the source table on which the segmentation will be applied & manually input the set of values. For each value entered, a task will be generated (when the bucket size is set to 1).
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle |
### Bucket Size
If your source contains a significant number of unique values within the attribute used for segmentation, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.
To avoid generating too many tasks, you can make each task manage several values of your segmentation attribute at once by using the “bucket size” option. Simply set a size for your bucket to decide how many values at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:
- If you do not set the bucket size, each task will handle 1 value
- If you set the bucket size to 2, each task will handle 2 values
- If you set the bucket size to 10, each task will handle 10 values
- And so on...
If you’re still unclear about what the “bucket size” does, find out more in the [bucket size reference](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#bucket-size-in-detail).
### Configuration
To configure this segmentation option select _Based on predefined set_ in the "Segmentation Type" dropdown.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "predefined_set",
"var_name": "id",
"chunksize": 1,
"values": [
"1"
]
}
```
[↪ Get further with this type of segmentation with our tutorial series](https://docs.dataplatform.ovh.net/tutorials-segmentation-predefined-values.md)
## 4. Based on a fixed number of lines
This segmentation mode allows you to split the loading of the data from a source simple using a fixed number of rows defined by the bucket size variable.
Each task will handle a specific number rows equal to the "bucket size" .
- If the bucket size is set to 1000, each task will extract 1000 rows.
- If the bucket size is set to 50000, each task will extract 50000 rows.
- Etc …
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Load | **Files**: CSV Files, Excel Files; **Datastore**: Big Query, AWS S31; **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle, SQLServer |
### Configuration
To configure this segmentation option select _Based on a fix number of lines_ in the "Segmentation Type" dropdown.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "lines",
"chunksize": 1000
}
```
:::warning
This segmentation type can be used with database sources such as SQLServer, PostgreSQL, etc. However, in these cases, **make sure that the Project is granted access to all the attributes of the table** otherwise your SELECT queries might get rejected for lack of authorization when attempting to segment an action.
:::
[↪ Get further with this type of segmentation with our tutorial series](https://docs.dataplatform.ovh.net/tutorials-segmentation-source-chunks.md)
## 5. Based on the source's accounts
For [sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) like social media which are user account specific, it is possible to configure multiple accounts for the same source within the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md). You can then leverage these accounts to split a DPE action using that source in several tasks, where each task will retrieve the data from one or more accounts.
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| ------------ | --------------------------------------------------------------------------------------- |
| Load | **API**: Facebook, Twitter, Google Analytics, YouTube Private, YouTube Public, LinkedIn |
### Bucket Size
If your source contains a significant number of accounts, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.
To avoid generating too many tasks, you can make each task manage several accounts at once by using the “bucket size” option. Simply set a size for your bucket to decide how many accounts at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:
- If you do not set the bucket size, each task will handle 1 account
- If you set the bucket size to 2, each task will handle 2 accounts
- If you set the bucket size to 10, each task will handle 10 accounts
- And so on...
If you’re still unclear about what the “bucket size” does, find out more in the [bucket size reference](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#bucket-size-in-detail).
### Configuration
To configure this segmentation option select _Based on the date perimeter of the workflow_ in the "Segmentation Type" dropdown.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "accounts",
"chunksize": 1000
}
```
[↪ Get further with this type of segmentation with our tutorial series](https://docs.dataplatform.ovh.net/tutorials-segmentation-source-accounts.md)
## 6. Based on files
For sources with the same structure in a distant source folder, this segmentation mode allows the DPE to automatically find all files in a folder, then create tasks with a specific number of files. It will replace the source file in “load\_from.source” actions for each files.
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| ------------ | -------------------------------------------------- |
| Load | S3 (file upload, datastore, AWS, etc..); FTP; SFTP |
### Bucket Size
If your source contains a significant number of files , it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.
To avoid generating too many tasks, you can make each task manage several files at once by using the “bucket size” option. Simply set a size for your bucket to decide how many accounts at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:
- If you do not set the bucket size, each task will handle 1 file
- If you set the bucket size to 2, each task will handle 2 files
- If you set the bucket size to 10, each task will handle 10 files
- And so on...
If you’re still unclear about what the “bucket size” does, find out more in the [bucket size reference](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#bucket-size-in-detail).
### Configuration
To configure this segmentation option select _Based on files_ in the "Segmentation Type" dropdown.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "files",
"values": ["a.*"],
"chunksize": 10
}
```
[↪ Get further with this type of segmentation with our tutorial series](https://docs.dataplatform.ovh.net/tutorials-segmentation-files.md)
## 7. Automatic Segmentation
With this option, you do not need to worry about configuring your segmentation settings, it will be automatically configured for you. Unfortunately this option isn't available with all [Data Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) or all [Action types](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md).
Below a list of supported Data Sources for the supported Actions as well as an explanation on how many workers to set for the [Jobs](https://docs.dataplatform.ovh.net/control-center-jobs-central.md#monitor-job-executions).
**Supported Data Sources with the Table to files Action**
- PostgreSQL: maximum number of workers is not limited.
**Supported Data Sources with the Load Action**
- [Apache Kafka](https://docs.dataplatform.ovh.net/connectors-sources-kafka.md): maximum of one worker per topic partition.
:::warning
Note that **you need to increase the number of instances for the automatic segmentation to take effect**. You can increase instances until they match the maximum number of workers allowed. For instance, if you have 3 partition's in the Kafka topic of your Load Action, you can set up to 3 workers for that action. If you set less than the 3 instances, for instance 2, one of the workers will take care of 2 partitions. This is set in the [Resources configuration](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally) and by [scaling the computing power horizontally](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#scale-your-jobs-horizontally) you will increase the number of instances.
:::
Check-out additional information about our _Segmentation_ and _Perimeter_ options for DPE action in the **Additional Remarks** articles of the DPE Action settings.
[↪ Segmentation and perimeter vocabulary](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#segmentation-and-perimeter-vocabulary)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/dpe-jobs-perimeter.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Filter job execution with the data perimeter
## Objective
When executing a data processing job, when you reach large volumes of data to process, you may need to filter the execution of an action. The _perimeter_ allows you to **define a field on which to filter the scope of the data to be processed** by the action.
## 1. Based on an attribute of a Project table
If you want to filter an action against a list of data present in the column of [a table from the Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md), for example:
| Reference Table | Source Table | Destination Table |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| | | |
You want to aggregate the data of the "income" attribute contained in the source tab "orders" in the destination table "aggr\_date" grouped by "date\_order". The tmp\_date table contains a list of dates to aggregate from 2019-01-01 to 2019-01-03.
By activating this perimeter mode, this will create a pre-stage automatically which aims to constitute the list of perimeter values contained in the table tmp\_date, attribute date, and will filter the action between all the values found.
### Compatibility
This perimeter option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle |
### Configuration
To configure this perimeter option select _Based on an attribute of a Project table_ in the "Perimeter Type" dropdown.
If you are using the advanced mode interface, you will need to add the following fields to your action JSON configuration:
```json
"perimeter": {
"active": true,
"type": "dwh_attributes",
"values": "dwh/default_dataset/tmp_dates/date",
"var_name": "date_order",
"attribute": "date",
"attribute_type": "map"
}
```
## 2. Based on a predefined set of values
This mode lets you enter a list of values to filter a source/table attribute on.
### Compatibility
This perimeter option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle |
### Configuration
To configure this perimeter option select _Based on predefined set_ in the "Perimeter Type" dropdown.
If you are using the advanced mode interface, you will need to add the following fields to your action JSON configuration:
```json
"perimeter": {
"active": true,
"type": "predefined_set",
"var_name": "value_2",
"values": [
"10",
"15"
]
}
```
## 3. Based on the date perimeter of the workflow
If your source data has a date attribute, it is possible to filter the processing of your action against the dates set in your workflow.
### Compatibility
This perimeter option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle; **API**: Facebook, Twitter, Google Analytics, YouTube Private, YouTube Public, LinkedIn |
### Configuration
To configure this perimeter option select _Based on the date perimeter of the workflow_ in the "Perimeter Type" dropdown.
You must switch to advanced mode to configure this option. You simply need to add the following fields to your action JSON configuration:
```json
"perimeter":{
"active":true,
"type":"workflow/dates",
"var_name":"date"
}
```
Check-out more detail about our _Segmentation_ and _Perimeter_ options for DPE action in the **Additional Remarks** articles of the DPE Action settings.
[Segmentation and perimeter vocabulary](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#segmentation-and-perimeter-vocabulary)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-spark-history-server.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Spark History Server
## Objective
The Spark History Server lets you inspect the execution history of your Spark jobs directly from the Data Processing Engine, without setting up any external tooling.
It applies to **PySpark-based actions** ([Load PySpark](https://docs.dataplatform.ovh.net/dpe-actions-load-pyspark.md), [Aggregate PySpark](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-pyspark.md), [Custom PySpark](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md)) and the **workflows** that run them. It does not cover [notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md).
Once enabled, every PySpark job records its event logs. You start the server when you need to read them, browse your recent runs, open each run's Spark UI (live or completed), analyze it with DataFlint, and apply retention rules to keep storage under control.
:::info
The server is **billed per [DPU](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) for the time it is running**. It runs only on demand and shuts down automatically after a period of inactivity, so you don't pay for idle time.
:::
## Start and stop the server
The History Server runs only when you need it. Use the **Start** button at the top right of the page to launch it; its status (_Stopped_ / _Running_), uptime, and time before auto-stop are shown next to it.
The server stops on its own after the inactivity period set in the [Preferences](#server-resources-and-auto-stop), so it never stays up longer than needed. You can also stop it manually at any time.
## Overview
The **Overview** tab summarizes your environment and lists your recent Spark runs.
### Statistics
The Statistics panel gives you a snapshot of your environment:
- **Applications**: total number of applications and how many are currently in progress.
- **Storage**: total size of stored event logs and the amount pending cleanup.
- **Activity**: time of the last run and the average Spark run time.
### Recent applications
The Recent applications table lists your most recent Spark runs with their **action or workflow** name, **job number**, **Spark app ID**, **status**, **end of run**, **Spark run time**, and **size**.
From each row you get one-click access to:
- **Spark UI**: the standard Spark interface to explore stages, tasks, and execution details.
- **DataFlint**: an enhanced analysis view with performance insights and optimization tips, powered by the open-source [DataFlint](https://github.com/dataflint/spark) project.
Opening **Spark UI** gives you the full standard Spark interface for the run, where you can explore jobs, stages, and tasks:
**DataFlint** layers performance insights and optimization tips on top of the same run:
:::info
You can also reach a specific run from the [Jobs](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md) page: select a Spark job (a PySpark action or workflow) to open it in the Spark History Server.
:::
## Follow running jobs
Running jobs appear at the top of the table, marked as live. As soon as a job's interface is ready, you can open its live Spark UI and DataFlint view without waiting for the run to finish.
:::info
Live interfaces are only available for running jobs when [**Live Spark UI**](#live-spark-ui) is enabled in the Preferences.
:::
## Preferences
The **Preferences** tab controls what is captured, the size of the server, when it stops, and how long logs are kept.
### Event logs
**Enabled by default.** Records the detailed activity of every PySpark job for analysis in the Spark History Server, including the performance insights and optimization tips powered by [DataFlint](https://github.com/dataflint/spark).
When disabled, the analysis is not available for the next jobs. The history of existing runs is kept.
:::info
Event logs are stored in your project's **DPE S31 bucket**. The [Retention](#retention) rules below control how much of that storage they use.
:::
### Live Spark UI
**Disabled by default.** Exposes the live Spark UI of PySpark jobs so you can watch them run without waiting for the event logs. The setting applies to the next runs.
### Server resources and auto-stop
- **Server resources**: the compute power allocated to the server, from **1 to 4 DPU**. The change is applied at the next start of the server.
- **Auto-stop**: automatically stops the server after the selected period of inactivity. Options: **Never**, **After 2 hours**, or **After 8 hours**. The default is **24 hours**.
### Retention
Event logs are cleaned up automatically every day. An application is removed once it exceeds any of the enabled limits below; the most recent applications are always kept. Set a limit to **0** (or _Unlimited_) to disable it. Use **Clean up now** to run the cleanup immediately.
- **Retention (days)**: how long applications are kept. Options: **7** (default), **30**, **90**, **365**, or _Unlimited_.
- **Maximum applications kept**: options **100** (default), **10**, **50**, **100**, **500**, or _Unlimited_.
- **Maximum total size**: options **1**, **5**, **10**, or **50 GB**, or _Unlimited_.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/dpe-settings.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage Data Processing Engine repositories and quotas
## Objective
The settings tab allows you to manage your Data Processing Engine [repositories](https://docs.dataplatform.ovh.net/dpe-settings.md#repositories) and [quotas](https://docs.dataplatform.ovh.net/dpe-settings.md#quotas).
## Repositories
The _Repositories_ panel gives a broad overview of each of your action repositories, allowing not only a quick overview of each one, but also the means to edit each repository such as changing the linked Git account.
## Quotas
The quotas sections allows you to request more resources for your Data Processing Engine, as needed.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Analytics Manager
## What is the Analytics Manager?
The Analytics Manager is where project data becomes answers. It queries the [tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) held in the Lakehouse Manager, through a no-code builder or a SQL editor, and turns the results into charts and dashboards.
A query is both the unit of work and the unit of reuse. Queries are saved, named and versioned, so one definition backs a chart, a dashboard tile and an API endpoint instead of being rewritten in each. Queries run on cloud-native engines, and read either the tables in the lakehouse or, where a [source is marked as directly queryable](https://docs.dataplatform.ovh.net/connectors-sources-configure.md#make-your-data-source-directly-queryable), the external system itself.
[Dashboards](https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md) assemble saved queries on a page for end users, and can be published beyond the people who built them. [History](https://docs.dataplatform.ovh.net/analytics-manager-history.md) records every execution with its performance metrics, which is where a slow dashboard gets diagnosed. [Settings](https://docs.dataplatform.ovh.net/analytics-manager-settings.md) holds the repositories and the query cache.
The Analytics Manager is also the seam between roles. Data engineers land and model the tables, analysts write the queries, and application developers consume them through [Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services.md), which serves any saved query as a live endpoint under IAM authentication.
[### Queries→
Save, name and version the queries the rest of the project reuses.
](/analytics-manager-queries)[### Use the visual builder→
Build a query without writing SQL.
](/analytics-manager-queries-visual-builder)[### Use the SQL editor→
Write SQL directly, including against directly queryable sources.
](/analytics-manager-queries-sql-editor)[### Visualize a query as a chart→
Turn a result set into a chart.
](/analytics-manager-queries-charts)[### Dashboards→
Assemble saved queries on one page.
](/analytics-manager-dashboards)[### Add content to a dashboard→
Lay out tiles, filters and text.
](/analytics-manager-dashboards-edit)[### Publish a dashboard→
Share a dashboard beyond the people who built it.
](/analytics-manager-dashboards-publish)[### History→
Every execution with its performance metrics.
](/analytics-manager-history)[### Settings→
Repositories and the query cache.
](/analytics-manager-settings)[### Resources→
Compute allocated to query execution.
](/analytics-manager-resources)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-queries.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create versioned SQL queries in the Analytics Manager
## Objective
The core feature of the Analytics Manager is to provide versioned queries, which can be **run on the fly or used in** [**dashboards**](https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md) **or** [**front-end applications**](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md). Queries thus constitute the first step of structuring your data and displaying it to the outside world.
You can create, view, organize or execute queries and manage which [versions](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#manage-existing-queries)
to deploy. Queries are stored and organized into _repositories_.
You can drill down on **execution performance metrics and KPI** for your queries, to easily optimize your analytics system. Further, **create API endpoints** to expose the data to the Platform apps or external applications in one click.
## Create queries
In the Analytics Manager, there are several interfaces to create and execute queries.
- The [visual builder](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md) lets you discover insights through an array of data visualizations in a simple point-and-click environment.
- The [SQL editor](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md) lets you write ANSI SQL queries to explore your data freely.
Click on **New query** to create a new query. It will be using the _visual builder_ interface by default.
[Learn how to do a query with the no-code builder](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md)
## Manage existing queries
From the queries tab you can:
- Create, delete or edit (1) queries
- Duplicate (1) queries
- Organize (2) your queries in folders
- Manage the [version](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#versioning-queries) (3) of the repository
### Versioning queries
When you are trying to iterate on existing queries, it is best to create several **versions** of the query. This lets you incrementally introduce variations to experiment and test your work, while **keeping the version in production always intact**.
This versioning happens through **repositories**, which regroup both queries and [dashboards](https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md). Each repository can have multiple versions of the code (queries + dashboards) they contain:
- For queries it is the query parameters (or SQL), as well as default visualization options
- For dashboards it is just their layout, filters and other minor overrides (like the color palette)
Repositories all have a _deployed version_, which is the version served when one of the queries is called in any downstream process (API, apps, etc.), or opened via a dashboard.
They also have an _active version_, which is the version currently being edited through the editor panel.
- If your repository only has one version, the active version is the same as the deployed version, which means that you are editing your queries in production.
- If your repository has two versions or more, the deployed version cannot be edited. It can be viewed in read-only mode.
:::info
It is no longer possible to version an individual query directly. All versions are now at the level of the repository.
:::
:::info
Repositories allow compartmentalizing your queries into different sections, but more than this, they can be linked to a Git repository _(coming soon!)_
:::
## Discover the design philosophy behind queries
Since the release of its first version, the component behind the Analytics Manager which orchestrates the query execution has kept a simple design philosophy: **where flexibility meets automation**.
### Need of flexible data models
The more complex a BI Project becomes, the more important it is to understand the data schema and each of its attributes.
A data lake / data warehouse in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) may change over time, which could be because of:
- Addition of new data sources (resulting in new business KPIs)
- Changes and optimizations made to the data schemas
- Addition, deletion, or modification of aggregate tables or repositories
The Lakehouse Manager (in charge of the schema structure containing the attributes, thus defines where and how to execute the queries, so that the people in charge of developing restitution interfaces (front-end) and APIs do not have to worry about it.
### Automated multi-base query engine
To meet the above needs, the Analytics Manager leverages an agnostic multi-base engine, which can request your data model in SQL no matter the underlying dataset or schema.
When you create a query using the visual point-and-click interface, you don't have to specify the table names.The Platform will **automatically choose the most suitable table** by performing those steps:
1. Check all the attributes needed to perform your query (and translate the virtual attributes into required attributes as well to append to this required list).
2. Check all candidate tables that possess all required attributes (excluding the non-queryable tables).
3. Select the table that has fewer rows.
:::info
While the Platform provides you the above flexibility, you can always override the automatically-chosen tables and force the table to query.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use the visual builder
## Objective
The visual builder offers a **generic way to query stored information** using a specific proprietary framework built on top of open-source query engines.
Queries are created by populating the 4 following fields with **attributes** from your [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md):
- [Data](#configure-the-field-data): the measures that you want to aggregate
- [Scale](#configure-the-field-scale): the dimensions by which you want to group your data
- [Filter](#configure-the-field-filter): to filter the results of your query
- [Order](#configure-the-field-order): to sort the results of your query
In addition, you can set a _limit_ to the number of rows returned by the interface by using the widget next to the Run button. This limit will not be saved in the query for downstream processes (APIs and apps).
[Learn how to configure a query](#configure-the-field-data)
### How does it work?
Let's assume for instance that the people in charge of developing visualization dashboards want to display a turnover curve (attribute: _income_) by date (attribute: _date_).
1. The visual builder interface can be used to specify the data to display (_**data**_: _income_), and the metric with which to group this data by (_**scale**_: _date_).
2. At this point the builder automatically determines which table is the most appropriate for the query - the **smallest to contain all required attributes**. In the example, "agr\_shop\_date", which is a table with income data aggregated by dates, will be selected and queried.
:::info
The visual builder does not ask you to specify a table. This ensures that your Project can scale easily in time as the storage system you use for your data varies (nature, amount of information stored, data warehouse schema, etc.)
:::
3. Finally the results are sent back to the user as a [light-weight, DBMS-agnostic JSON object](#discover-the-response-format).
## Configure the field Data
In the field _Data_ enter the **attributes that you want to display**. It is possible to aggregate the data to display, using [simple aggregation functions](#aggregation-functions).
You can add as many attributes as you want as long as they exist in the same table, otherwise an error will be returned. The list of available attributes in the interface is automatically filtered to only display the other available attributes.
You [don't need to specify the table](#choice-of-table). Use the widget to the left of the screen to get metadata about the tables and attributes you want to query.
### Aggregation functions
- **SELECT**: The _SELECT_ function simply displays the value of the selected attribute. It will select the first value of the list if several values are returned.
- **SUM**: The _SUM_ function is used to calculate the total sum of the values of an attribute.
- **COUNT**: The _COUNT_ function is used to count the number of occurrences returned by the query.
- **COUNT DISTINCT**: The _COUNT DISTINCT_ function is used to count the unique different values of an attribute.
- **MIN & MAX**: The _MIN_ & _MAX_ functions are used to display the minimum and maximum values of an attribute.
- **SELECT DISTINCT**: The _SELECT DISTINCT_ function is used to display the unique different values of an attribute.
- **AVG**: The _AVG_ function is used to calculate the arithmetic average of an attribute.
### Choice of table
The visual builder does not ask you to specify a table. By default,the Platform determines the table that is the most fit for the query: it is the table containing all attributes in the data, scale, filter and order fields which has the least amount of rows.
If you want the query to be forced on a specific table, you can always override the system-automatically-chosen tables and select your own by clicking the **table icon** next to _Data_.
:::info
Forcing the queried table can come in handy when certain attribute names (such as "date") are common to a large number of tables. However, it is not a good practice as the Analytics Manager is designed to fetch the data using an optimal path to reduce runtime.
:::
## Configure the field Scale
In the field _Scale_, enter the attributes (usually dimensions) that you want to **group your [Data](#configure-the-field-data) attributes (usually measures) by**.
:::info
An example of use case is if the query needs to return the revenue amount per month or the revenue amount per store.
:::
You can add as many attributes as you want as long as they exist in the same table, otherwise an error will be returned. The list of available attributes in the interface is automatically filtered to only display the other available attributes.
## Configure the field Filter
This parameter is used to apply filters to the values that the query is expected to return.
:::info
Filters are particularly useful to prevent nulls or unlabeled values to be displayed or taken into account.
:::
Enter the reference values for **string-datatype** filters as is, without the _" "_ around the string. Enter the reference values for **numeric-datatype** filters as is too.
:::info
Use the widget panel _"See table details"_ to the left of the screen to check the type of the attributes to use as filters.
:::
Below is the list of all filter options available:
- _Equal_ - Value is equal to reference.
- _Not equal_ - Value is not equal to reference.
- _In_ - Value contained in reference list (press Enter to add a value to the list).
- _Not In_ - Value not contained in reference list (press Enter to add a value to the list).
- _Greater than_ (>) - Value is greater than a reference.
- _Greater than equal_ (>=) - Value is greater than or equal to a reference value.
- _Lower than_ (\<) - Value is lower than a reference value.
- _Lower than equal_ (\<=) - Value is lower than or equal to a reference value.
- _Between_ - Value is between 2 values.
- _Not between_ - Value is not between 2 values.
- _Null_ - Value is null.
- _Not null_ - Value is not null.
- _Like_ - Substring of value is contained in reference value (formatted using [SQL LIKE syntax](https://sql.sh/cours/where/like)).
- _Not like_ - Substring of value is not contained in another reference value (formatted using [SQL LIKE syntax](https://sql.sh/cours/where/like)).
## Configure the field Order
This parameter is used to order the results returned by the query based on the order of values of one or more fields, in ascending or descending order.
:::info
Attributes in the [Scale](#configure-the-field-scale) field are not added to the _Order_ field by default, in order to optimize runtimes.
:::
## Discover the response format
When a query is executed through the visual mode, the results are returned using the following format.
```json
[
{
"scales": {
"company": "Snapchat"
},
"data": {
"sales": {
"avg": [
{
"company": "Snapchat",
"value": 494.96303030302937
}
]
}
}
},
{
"scales": {
"company": "Pied Piper"
},
"data": {
"sales": {
"avg": [
{
"company": "Pied Piper",
"value": 492.7505523255817
}
]
}
}
},
{
"scales": {
"company": "Google"
},
"data": {
"sales": {
"avg": [
{
"company": "Google",
"value": 501.16340892465297
}
]
}
}
},
{
"scales": {
"company": "Facebook"
},
"data": {
"sales": {
"avg": [
{
"company": "Facebook",
"value": 512.5209205642168
}
]
}
}
}
]
```
To view the raw response sent by the Analytics Manager when your query is executed, visualize the results as a table. Then click on the **gear icon** and activate _Display raw response_.
Finally, it is possible to export the query results in various different formats (such as a .csv, copying tab-separated values, etc.) by clicking on the **Download** button next to the Run button.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use the SQL editor
## Objective
The Analytics Manager SQL editor lets you use [SQL](https://en.wikipedia.org/wiki/SQL) to write query statements on either your databases, or [directly on sources](#query-data-sources-directly). SQL queries can join data located on multiple different [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) or [sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md).
To use the SQL editor, click on **SQL** in the header of a query.
:::warning
After you switch a query to the SQL mode, it is impossible to switch it back to the visual mode.
:::
At the moment, SQL editor can only return results in table format. It cannot be used for charts.
## Write queries in the SQL editor
The SQL editor only supports statements written in ANSI-compliant SQL syntax, that start with a _SELECT_ clause.
As you type, the editor suggests SQL keywords, ready-made snippets for common statement shapes, and every Trino function available to your queries, each with a short description of what it does. Pick a suggestion to insert it at the cursor.
### Query stored databases
To query data stored in your Project's tables, use standard SQL statements to refer to the attributes and tables you want to select:
To reference **a table**, simply use its name as entered in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
```sql
SELECT * FROM my_table
```
To reference **an attribute**, simply use its name as entered in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md).
```sql
SELECT my_attribute FROM my_table
```
You can use the widget on the left of the screen to find and copy the exact name of tables and attributes.
### Query data sources directly
The Analytics Manager can execute queries on your connected data sources [with Direct Query mode activated](https://docs.dataplatform.ovh.net/connectors-sources-configure.md#make-your-data-source-directly-queryable), without importing the data on Data Platform.
To reference **a table** from a source, use the _technical name_ of the source as entered in the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) preceded by `db_{projectID}_` and followed by a dot, followed by the _schema_ for this data source (usually the schema name or database name from your connector), followed by a dot, followed by the _name of the table/object_ at it is in the source.
```sql
SELECT * FROM db_lcibesgk_my_mongodb_source.database_name.my_table
```
To reference **an attribute**, simply use the name it has in the source object.
```sql
SELECT my_attribute FROM db_lcibesgk_my_mongodb_source.database_name.my_table
```
:::info
If the source object's name doesn't also exist in the [Lakehouse Manager Tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md), **you can simply reference it by name** (as if it was stored in the Platform database). However, if there is a table with the same name in the platform, it is necessary to write the _source technical name_ and the _schema_ as explained.
:::
## Discover the response format
When a query is executed through the SQL mode, the results are returned using the following compact format.
```json
[
[
"Google",
501.16340892465297
],
[
"Facebook",
512.5209205642168
],
[
"Snapchat",
494.96303030302937
],
[
"Pied Piper",
492.7505523255817
]
]
```
To view the raw response sent by the Analytics Manager when your query is executed, click on the **gear icon** and activate _Display raw response_.
## Working with large result sets
Queries execute **asynchronously**: that is how Trino works natively, and the Analytics Manager relies on it. A slow or large query is therefore not a browser-timeout problem.
What can hurt is the **volume of rows returned to the browser**: when a result set is very large, the page tries to load it all into memory at once and may become unresponsive. This is unrelated to the complexity of the query itself. A statement with many `CASE WHEN` clauses runs fine; it is the number of rows displayed that matters.
:::info
**Best practice:** narrow the result set before returning it. Add filters, aggregate, or use `LIMIT`. This keeps the editor responsive.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-queries-charts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Visualize a query as a chart
## Objective
By default, all queries that are executed are displayed in the interface in a table format. However, the Analytics Manager is able to **render it in a wide variety of chart types** to help you visualize trends and insights in your data.
:::warning
At the moment, only queries executed through the [visual mode](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md) can be viewed as charts. Queries executed through the [SQL mode](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md) can only return results in table format.
:::
:::info
All queries that are executed through the [visual mode](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md) return the results as a DBMS-agnostic JSON object.
:::
Choose the type of chart you want to use in the bottom part of the interface.
The visual aspects of all charts can be configured by clicking on the **gear icon**.
Each chart can be individually rendered using a specific color scheme set in the chart settings. However, a chart rendered in a dashboard will use the dashboard's color palette.
## Use a table
Table is the default visualization mode for queries. It allows to get a complete view of the data results at once and easily compare figures. Create one by clicking the **table icon**.
## Use a line chart
Line charts are great to visualize trends over time in your data. Create one by clicking the **line icon**.
### Chart settings
#### Labels and legends
- **Show values**: display the raw value for each data point on the graph
- **Show % of total**: display the percentage share of each data point's value compared to the line's total
- **Data labels position**: the position of each data point's label
- **Hide legend**: hide the legend from the chart
- **Legend position**: position of the legend on the chart
#### Axis
- **Regroup by**: choose which attributes from the query to regroup the X-axis by (e.g. regroup values first by year, then by month -- or vice-versa). If multiple attributes are selected, it will use combinations of X values
- **X-axis/Y-axis scale**: choose an option among:
- Category: the axis will assume its values are categorical
- Time: the axis will assume it is a time series (requires _Datetime_-type attributes)
- Linear: the axis will be linear (requires _Numeric_-type attributes). You can override the automatically-detected minimum and maximum.
- Logarithmic: the axis will be logarithmic (requires _Numeric_-type attributes). You can override the automatically-detected minimum and maximum.
- **Show grid values**: show the grid labels for this axis
## Use an area chart
Area charts are perfect to compare trends of multiple related metrics over time. Create one by clicking the **area icon**.
### Chart settings
#### General
- **Display stacked**: if multiple area charts are plotted, display them stacked instead of overlapping
#### Labels and legends
- The options are identical to that of the [line chart](#use-a-line-chart).
#### Axis
- The options are identical to that of the [line chart](#use-a-line-chart).
## Use a bar chart
Use bar charts to visualize categorical or nominal data easily. Create one by clicking the **bar icon**.
### Chart settings
#### General
- **Display stacked**: if multiple bar charts are plotted, display them stacked instead of next to each other.
- **Orientation**: display the bars oriented either vertically or horizontally
#### Labels and legends
- The options are identical to that of the [line chart](#use-a-line-chart).
#### Axis
- The options are identical to that of the [line chart](#use-a-line-chart).
## Use a scatterplot
Use scatterplots to visualize the correlation between two metrics, each represented on an axis. Create one by clicking the **scatterplot icon**.
Each row in your result table will be plotted as a dot at the intersection of the two dimensions which have been assigned to each axis.
### Chart settings
#### General
- **Display trend line**: plots a linear regression for each of the scatterplots displayed
#### Labels and legends
- The options are identical to that of the [line chart](#use-a-line-chart).
#### Axis
- **X-axis / Y-axis attribute**: choose the query's attribute to be represented on the X and Y axes.
- **Display axis name**: display the name of the attribute represented on this axis
- The other axis options are identical to that of the [line chart](#use-a-line-chart).
## Use a pie chart
Pie charts work best to show relative proportions and percentages of categorical data, especially when the number of categories doesn't exceed 5-6. Create one by clicking the **pie icon**.
It is only possible to display one single [Data](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#configure-the-field-data) attribute on a pie chart. If you query has multiple _Data_ attributes, only the first one will be plotted.
If your query has multiple [Scale](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#configure-the-field-scale) attributes, each slice of the pie chart will represent one combination of _Scale_ attributes' values.
### Chart settings
#### General
- **Display as a donut**: displays the chart as a donut chart instead of a whole pie chart
#### Labels and legends
- The options are identical to that of the [line chart](#use-a-line-chart).
## Use a metric
Metrics displayed as single big numbers are handy if you want to display multiple KPIs in a dashboard. Create one by clicking the **number icon**.
Only the first cell (first row of first column) **among the [Data](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#configure-the-field-data) attributes** of your results table will be displayed as the big metric.\
If your query returns more than one row or one column, those will be ignored unless you want to display the metric's evolutions or dimensions (see chart settings below).
### Chart settings
#### General
- **Compare**: displays the evolution of the metric vs. the second row of data (of first _Data_ attribute still), i.e. the cell right below the metric in the results table
- **Display as a percentage**: displays this evolution as a percentage rather than the raw number
- **Switch colors for positive/negative evolutions**: if activated, positive evolutions will now be displayed in red and negative evolutions will be displayed in green
- **Display dimensions**: this will display the corresponding values of the first [Scale](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#configure-the-field-scale) attribute, if present in your query. If your query has multiple _Scale_ attributes, the others will be ignored
## Use a gauge chart
Visualize KPIs as a gauge chart if you want to see how far they are from reaching a specified target. Create one by clicking the **gauge icon**.
The metric displayed has identical requirements to that of the [Metric](#use-a-metric) chart type: only the first cell (first row of first column) **among the [Data](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#configure-the-field-data) attributes** of your results table will be displayed as the big metric.
This value must be a numeric for the gauge chart to work.
### Chart settings
#### General
- **Target**: enter the target for the gauge chart
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build dashboards in the Analytics Manager
## Objective
Dashboards let you arrange charts on a single page to visualize and monitor your data in real-time.
You can combine various data sources and display them in a variety of formats, including charts and tables. Add interactive filters to update all charts at once, click on specific parts of charts to further drill down on them, and add notes to tell stories about your data visualization.
Share your dashboard with your teammates by publishing it, and manage versions to make sure everyone always get the validated KPIs.
:::info
Dashboards come with a limited set of customization options, making it a perfect tool for Data Analysts. The Platform also makes it possible for developers to build and deploy fully custom applications in [Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services.md).
:::
## Manage dashboards
Dashboards are managed in the _Dashboards_ tab of the Analytics Manager.
All your dashboards are stored in repositories, the [same as for queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#versioning-queries). Each chart in a dashboard corresponds to a [query](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) from the _Queries_ tab. In a dashboard, you can **only display queries stored in the same repository** as the dashboard.
## Add content to a dashboard
Dashboards are made of tiles which contain charts. Each chart is powered by a [query](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) saved in the _Queries_ tab.
It is possible to customize your dashboard layout and appearance. You can rearrange tiles by dragging and dropping them, resize them by dragging the edges, and change the color palette to match your branding.
[Add content to a dashboard](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-edit.md)
## Publish a dashboard
Each dashboard has a deployed version which is hosted on a separate URL, allowing you to share a dashboard with other people without giving them the possibility to edit the dashboard.
[Publish a dashboard](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-publish.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-dashboards-edit.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Add content to a dashboard
## Objective
Dashboards are made up of the following elements:
- A title and description
- Filters
- Chart tiles
You can add as many tiles and filters as you want to a dashboard.
## Arrange tiles
Tiles are the main constituting elements of dashboards. Each tile contains a chart saved as a [query](https://docs.dataplatform.ovh.net/analytics-manager-queries.md), along with a title and a story.
Tiles can be moved around by dragging and dropping them. They are naturally positioned in successive rows. There cannot be more than 8 charts on the same row.
### Resize a tile
Tiles can be resized horizontally by drag-dropping the little handles on the right and left.
Tiles can be resized vertically by drag-dropping the main handle under each row. Note that this resizes the whole row.
### Change a tile's visualization settings
Tiles are linked to a saved query which gives:
- the result of the query
- a [default visualization type](https://docs.dataplatform.ovh.net/analytics-manager-queries-charts.md) and its settings
However, it is sometimes useful to bind two different tiles to the same query, using two different visualization modes.
The visualization type can be overridden in each tile by clicking on the corresponding chart type in the tile's settings.
This will use the visualization options (such as axis or label settings, etc.) [set at the query level](https://docs.dataplatform.ovh.net/analytics-manager-queries-charts.md#visualize-a-query-as-a-chart). You can edit them here by clicking on the **gears** icon.
### Add data stories
Stories allow you to add a summary, conclusion, special observations to note, or simply comments about a chart.
Future users can hover over the **story** icon to read it. Click on the icon to edit the story.
## Create filters
You will probably experience a situation where you need to create multiple versions of the same dashboard, but with only one variable changed.
For example, you may have a retail report which you need to show for multiple store IDs. Rather than creating dozens of copies of the dashboard, you can add filters to allow users to filter the queries in the tiles within a single dashboard.
### Add and use a filter
Adding a variable to filter on makes it available in the [live dashboard](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-publish.md). Users accessing the live dashboard will be able to change the variable of the filter (but not remove or add a new variable to filter on).
:::info
Filters can take up various formats: dropdown menus, date pickers, etc... This is managed in the [advanced filters options](#edit-advanced-filter-options).
:::
The default value of the filter in the live dashboard is the **value saved in the edition mode** of the dashboard.
Filters set on a dashboard are applied on top of the [filters set at the query level](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md#configure-the-field-filter).
For example, a chart which is filtered on `country is in [Japan, USA]` at the query level, and which is added to a dashboard containing the filter `country is not Japan` will be displayed filtered on the country USA only.
:::warning
Exception: Filters set on the **same attribute and using same operator but with a different value** in the query and the dashboard are the only situation where the dashboard filter overrides and replaces the query filter.
:::
### Force tiles to ignore a filter
By default, new filters are applied on all queries (i.e. charts) of the dashboard, except on queries which are written in custom SQL - which cannot be filtered as of now.
It is possible to force a tile to ignore a filter, either from the filter [advanced options](#edit-advanced-filter-options) or from the tile itself.
:::info
Filters can be detected as _irrelevant_ for a tile's query, meaning that filtering on any value for this filter will return _no data_ for the query. They are flagged as such in the interface and it is recommended to ignore the filter at the tile's level.
:::
### Edit advanced filter options
The advanced filter options allow you to:
- Set the type of filter: dropdown menu, date picker, etc.
- Manage the tiles which [ignore](#force-tiles-to-ignore-a-filter) this filter.
- Check the page for the [attribute](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md) of the filter
Click on the **gear icon** in a filter's header to open its advanced options.
By default, the type of a filter is automatically inferred when it is created.
- **Categorical**: choose among the possible values for the attribute. The possible values for each filter are rendered by a query saved automatically in a subfolder in the _Queries_ tab.
- **Date**: use a datepicker to select a date range. This type is automatically set when the attribute appears as a _Date_ in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
- **Date & Time**: use a timepicker to select a date and/or time range. This type is automatically set when the attribute appears as a _Datetime_ in the Lakehouse Manager.
- **Boolean**: choose between True/False. This type is automatically set when the attribute appears as a _Boolean_ in the Lakehouse Manager.
## Drill down on the dashboard
**Click on a part of a chart** to reveal a range of options letting you explore your data further, such as dynamically filtering all the other charts on a value.
- **Filter out**: sets a filter on the whole dashboard to _exclude_ the value that was clicked on
- **Filter on this**: sets a filter on the whole dashboard to _show only_ the value that was clicked on
- **Show these values**: opens up a new query listing all values (as a SELECT) constituting the part of the graph that was clicked on
- **Breakdown further**: opens up a new query showing the data point that was clicked on, to let you try out more filters and break down options.
## Override visual aspects of the dashboard
### Color palette
The colors of the dashboard are templated in the color palette of the dashboard, in the Preferences page.
## Share dashboard
Dashboards are designed to be shared with end-users. In addition to the edition mode, reachable by default from the _Dashboards_ tab, they also all possess a **live version**, rendered when the dashboard [is published](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-publish.md).
### Share a link
Click on the **link** icon to copy the live dashboard link to the clipboard.
### Send a dashboard by email
:::info
This feature is coming soon!
:::
[Discover how to publish a dashboard](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-publish.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-dashboards-publish.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Publish a dashboard
## Objective
Dashboards are designed to be shared with the rest of your team. This is why each dashboard can be opened as a **live published version**.
## Publish a dashboard live
Each dashboard created in the Analytics Manager can be opened in either:
- the **editing interface**: users can edit the [whole configuration and preferences of the dashboard](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-edit.md), such as adding charts, filters, stories, etc.
- the **live interface**: users can view the dashboard and interact with the filters to refresh the charts, but they cannot edit it
_Example of a live dashboard_
_Example of the edition interface of the same dashboard_
Each mode **has its respective URL**. The editing mode URL depends on the [version](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#versioning-queries) currently being edited. The live mode URL always stays the same, referencing the _shared\_id_ of the dashboard which stays the same across versions.
:::info
All URLs require to be [authenticated in the Project IAM and granted access to the dashboard](#grant-access-to-the-dashboard) to open them.
:::
### How to publish a version?
A live version of a dashboard is published automatically based on the [deployed version of the repository](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#versioning-queries) containing the dashboard and its queries (i.e. charts).
#### If your dashboard repository has only one version
The dashboard is published **automatically at each modification**.
This means that all modifications made by editors will be visible in real-time in the live dashboard.
:::info
Updates may take up to a few minutes to be visible in the live dashboard.
:::
#### If your dashboard repository has two or more versions
The dashboard is **published automatically when a new version of the repository is deployed**. The live dashboard always corresponds to the version in the deployed repository.
This means that the editors can work on one version of the dashboard, while the viewers accessing the live dashboard will not see the modifications until the new version is deployed by the editor.
## Grant access to the dashboard
Multiple viewers can be granted access to the live dashboard easily, without granting them the right to edit it.
:::info
First, note that both dashboard editors and viewers, will have to be registered in a Project's IAM in order to access it, even if they are external to the Platform. They will only access the data/dashboards associated to their [roles and groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md).
:::
The rest of this page aims to explain how to give read access to a live dashboard for your end-users. There are **two usual modes** for access control to a live dashboard.
### Basic access control
By default, you can easily give end-users **full read access** to dashboards (the charts, tables and data they contain) - without giving any direct querying rights on the underlying databases.
This is due to the _Basic_ access control mode which is set by default in the dashboard's preferences.
With this mode, you give individual access to users or groups to the live dashboard as a whole. This means that **all data contained in the charts and filters will be visible to people who are granted access**, only from the live dashboard.
The _Grant access_ section shows all the users/groups from the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md) that have either one of these roles: _**Admin**_, or _**Dashboard Viewer**_ set with a condition on the dashboard name. Those users can open and view the live dashboard.
You can give access to new or existing users/groups from the _Grant access_ tab. Click on **New user/group** and choose between _New user/group_ and _Existing user/group_.
:::warning
Please note that you must have `write` permissions on [IAM users](https://docs.dataplatform.ovh.net/iam-users.md) / [IAM groups](https://docs.dataplatform.ovh.net/iam-groups.md) to use this feature.
:::
Selecting existing users will give them the [IAM role](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) _Dashboard Viewer_, with a [condition](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#add-a-condition-when-binding-a-role) set on this specific dashboard. They will get access to the whole dashboard's data **only from the live dashboard**.
Adding a new user will add them in the Identity Access Manager with the role _Dashboard Viewer_ (with a condition set on this specific dashboard) and automatically generate credentials for them (see details below). They will get access to the whole dashboard's data only from the live dashboard.
Since those users are usually external BI users and not the Platform users, they are likely to not need the Platform account. Instead,the Platform automatically generates a simple login and password for them through the **Project IAM's own authentication method** (called [**Project**](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project)).
With these credentials, automatically shared with them by email, they will be able to authenticate to the live dashboard's URL and view the dashboard.
### Fine-grained access control
:::info
This feature is coming soon!
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-history.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Review query execution history
## Objective
The history tab allows you to see the list of the latest query executions such as:
- queries ran from the Analytics Manager interface
- queries passed on from an [API](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) (or an [application](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md))
- queries executed on the fly in the [Lakehouse Manager Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md)
A _SQL_ column is displayed to show the **actual query string executed by the Analytics Manager** behind the scenes. This query is automatically converted from the original query written by the user (either in [SQL](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md#write-queries-in-the-sql-editor) or using the [visual builder](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md)) which usually has a much simpler syntax. Typically, the Analytics Manager will automatically:
- append engine/database/schema information when necessary to route the query to the correct [data source](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md)
- replace the [virtual attributes](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md#virtual-attributes) with their corresponding SQL definition
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-settings.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage Analytics Manager repositories and cache
## Objective
The settings tab allows you to manage your Analytics Manager [repositories](https://docs.dataplatform.ovh.net/analytics-manager-settings.md#repositories) and [cache](https://docs.dataplatform.ovh.net/analytics-manager-settings.md#flush-cache).
## Repositories
The _Repositories_ panel gives a broad overview of each of your query repositories, allowing not only a quick overview of each one, but also the means to edit each repository such as changing the linked Git account _(coming soon!)_.
## Flush Cache
The Analytics Manager has a cache on executed queries.
The _flush cache_ button provides you the simplicity of getting rid of unnecessary blocked memory because of various queries results or cache metadata and lets you work with improved speed and performance.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/analytics-manager-resources.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage the Analytics Manager query engine
## Objective
The resources tab allows you to manage the **query engine** of your Project, i.e. the technology powering SQL analytics in the Project.
It executes queries on one or several [tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md), and is able to distribute the query processing on multiple worker instances.
You can activate one query engine per Project, in order to use the advanced features of the Analytics Manager.
## Manage the query engine
### No query engine
By default, newly created Projects have **no query engine activated**. You can activate it manually on the resources page.
Without a query engine activated, you can still do the following:
- Create and run queries using the [visual mode](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md), on a single [dataset](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md)
- Deploy and use [applications in Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md)
- [Connect](https://docs.dataplatform.ovh.net/developers-python-sdk-reference.md#the-connect-function) to your data from a [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) action.
- Connect to the [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) directly from outside the Project - _coming soon!_
However, it is impossible to do the following without a query engine:
- Create and run queries using the [SQL mode](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md)
- Query (and join) data spread on multiple [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md), or even on [directly queryable sources](https://docs.dataplatform.ovh.net/connectors-sources-configure.md#make-your-data-source-directly-queryable)
- Create [dashboards](https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md)
- View the query [history](https://docs.dataplatform.ovh.net/analytics-manager-history.md) and detailed query execution metrics
- Implement fine-grained [permissions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) on the tables and access to the data they contain - _coming soon!_
## Manage instances
You can manage the number of parallel worker instances within a Trino cluster. There will also always be a coordinator, which distributes queries in the cluster.
If you only have one instance in your cluster, the coordinator will also be a worker.
You can also scale your query engine vertically and allocate **more computing power** to it by increasing the number of [Data Platform Units](https://docs.dataplatform.ovh.net/manage-pricing.md) (DPU) of each instance. The DPU is a unit of general processing capability, representing access to approximately _1 vCPU_ and _4 GB of RAM_, based on hardware availability.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-app-services.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Application Services
## Where data leaves the project
Application Services is where a project's data is exposed to the outside world: as [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) that other systems call, and as [apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) that people open in a browser. The two share one lifecycle, one resource model and one identity layer, so they present as a single service.
That lifecycle is build, then deploy. Editing an API or an app changes its source, and nothing reaches production until an **artifact** is built, which is a container image prepared automatically for that component, and then deployed, which runs a container from that image.
_Figure 1: building an artifact on Data Platform_
Deployment is **blue/green**. Two identical production environments exist, only one is live at a time, and the live one serves all traffic. Switching between them keeps downtime and rollback risk low.
_Figure 2: diagram of a blue/green deployment_
Once a component is deployed, nodes can be added or removed, each node an instance of the artifact. That is how an API or an app scales horizontally.
Every app is linked to a deployed API, and every API can serve any query saved in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) under IAM authentication, so data leaves the platform through one authenticated path rather than through direct database access.
[### APIs→
Serve project data over authenticated HTTP endpoints.
](/landing-page-app-services-apis)[### Apps→
Deploy a web application on top of those endpoints.
](/landing-page-app-services-apps)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Application APIs
## What is a Data Platform API?
An API in a Data Platform project is a deployed service that exposes project data over HTTP, with authentication handled by the project's [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md). Applications call the API rather than reaching a database, so credentials and query logic stay inside the platform.
APIs are built in two ways: from the point-and-click templates shipped with the platform, or from code written against the Front API SDK in NodeJS. Either way the result is versioned, built and deployed through the [Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services.md) lifecycle, and served by nodes that can be scaled out.
:::info
Any query saved in the [Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) can be exposed as an endpoint, so an analyst's saved query becomes an application's data source with no code in between. Because the query syntax abstracts the underlying store, analytics are served without exposing any database directly.
:::
Every [app](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) in the project is linked to a deployed API: the app knows which API to call to run its queries.
[Extensions](https://docs.dataplatform.ovh.net/app-services-apis-extensions-list.md) are modules attached to an API rather than code written into it: caching in memory or Redis, IAM user filtering, Analytics Manager query passthrough, key-value web storage. Adding one requires reloading and rebuilding the API before it takes effect.
[### Create and deploy an API→
From a template or from your own NodeJS code.
](/app-services-apis-deploy)[### API editor interface→
Version, build and deploy from the low-code editor.
](/app-services-apis-interface)[### API extensions list→
The extensions available to a deployed API.
](/app-services-apis-extensions-list)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apis-deploy.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create and deploy an API
## Objective
From your Project's home page, click on **+** inside the API module.
You will be able to choose between **an existing API template** from the internal _Platform Store_, or to **import your own code** by linking a Git repository.
Choose a name for your API, and enable the option **auto build**. This will download the source code of your API and automatically build it for you. You can also activate auto-deploy right away, or check out down below how to deploy your API manually.
From there, give it a couple of seconds as the creation of the API may take a moment.
:::info
the Platform uses a **[blue-green](https://en.wikipedia.org/wiki/Blue-green_deployment) deployment methodology** ensuring 100% availability of your data. You can therefore seamlessly create & deploy new versions of your API without risking any downtime!
:::
On the home page of your API, you'll find three parts:
- **Monitoring**: status of your API once it has been deployed
- **Active**: status of your "active" version, that is to say, the one you are currently working on
- **Versions**: list of all the versions (also called tags) available in your API
You can see more details about the version you just created using the arrow on the right of it.
Check the activity log by clicking on "Open Console". Since you have activated the auto-build when creating the API, it has already started.
When it is done, you should see a last log "EndBuild success".
To start the deployment, please click on "Deploy".
Deployment starts, and may take a few minutes.
When the deployment is complete, the "Open" button appears.
When you open your API, you now land on this screen.
To access your API, simply install an application and connect it to the API. You can also use its endpoints live.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apis-interface.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# API management interface
## Objective
Once an API has been created, you will be able to access its management interface. The _Overview_ tab allows you to easily manage the whole life cycle of your API.
The screen is divided into 3 parts:
- The top block (green) shows some **indicators related to the resources used by the API**. These indicators make it possible, for example, to know if it is necessary to increase the number of instances.
- The left block (yellow) indicates **the status of the active version** (in production). Some shortcuts are proposed by buttons according to the state (New / Built / Deployed) of the version.
- The right block (blue) lists the **different versions of the API**. The active version is displayed in blue. Different actions are possible for each version:
- Synchronize / Download - action depending on the type of code source (ZIP file or GIT repository)
- Duplicate version
- Build the artifact of this version
- Deploy / Stop version
- Open the building console
- Remove version - requires that the version has not been deployed
## Configuring extensions
When creating an API through the Marketplace, the latter is created with a set of default extensions that can be viewed in the _Extensions_ tab.
[Go to the extensions list](https://docs.dataplatform.ovh.net/app-services-apis-extensions-list.md)
This screen lets you visualize the different libraries organized by categories with the possibility to:
- Remove extensions which are already installed
- Add new extensions which are not yet installed on the API
- Configure the package of an extension
- Access additional information about libraries
- Import an open source or proprietary extension, specific to a Project or an organization
:::warning
Adding an extension requires you to reload your API (on the Overview page) in order to be able to build it.
:::
## Settings
Three types of API settings are configurable in the _Settings_ tab.
- **Global**: Refers to global API information such as its name, description, and main image.
- **Environment Variables**: Defines the variables that the API needs in order to work. Those necessary for the smooth functioning of the API and its various extensions are filled in automatically. Only the variables specific to a business need are presented here.
- **Domain Name System**: Allows you to add one or more URLs allowing access to the API, in addition to the one generated by default. These URLs have the domain "eu.dataplatform.ovh.net" unless you have added another domain at the organization level.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apis-extensions-list.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# API extensions list
## Objective
:::warning
Adding an extension requires you to reload your API (on the Overview page) in order to be able to build it.
:::
| **ID** | **Name** | **Description** |
| :------------------: | ----------------------- | -------------------------------------------------------------------------------------------------------- |
| **sdk** | SDK | Development kit essential for the good functioning of other modules. |
| **cache-controller** | Cache Controller | Expose the cache flush feature of this API. |
| **cache-memory** | Memory Cache | Enables the cache in the node's RAM. |
| **cache-redis** | Redis Cache | Enable the cache in Redis. |
| **iam** | Identity Access Manager | Enables the user filtering feature through the Data Platform authentication system. |
| **qb** | Analytics Manager | Allows Analytics Manager query requests to be passed through the API. |
| **webstorage** | Web Storage | Provides simple key-value storage and enables the SDK front web backup feature (similar to localstorage) |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform applications
## What is an app?
An app is a deployed web application served from the project, built to put dashboards and data in front of end users without running any front-end infrastructure of your own. Apps are created either from a template in the Platform Store or by importing existing code from a Git repository.
Building one follows a fixed order: pick a template or import code, choose extensions including the visualization libraries, build the dashboards, arrange navigation and menus, then optionally adjust style, value formatting and translations. Versioning and deployment come last, and a change reaches production only through a build and a deploy.
Each app is linked to a deployed [API](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md), which is how it reads project data, and each app authenticates its users against the project's [IAM](https://docs.dataplatform.ovh.net/iam-application-authentication.md), so an end user sees only the data their roles allow.
The pages below cover each part of an app: dashboards and their charts, the extension catalogue, style, menus, translations, value formatting, Git integration and custom domains.
Creating an app, building it, deploying a version and configuring its settings are covered together in [Create and deploy an app](https://docs.dataplatform.ovh.net/app-services-apps-deploy.md).
[### Create and deploy an app→
From a template or a Git import, through build, deploy, versions and settings.
](/app-services-apps-deploy)[### Dashboards→
Build the pages an end user sees, and configure their charts.
](/app-services-apps-dashboards)[### Extensions→
Add visualization libraries and other capability.
](/app-services-apps-extensions)[### Style→
Colours, fonts and branding.
](/app-services-apps-style)[### Menus→
Arrange the app's navigation.
](/app-services-apps-menus)[### Translations→
Localize labels for each audience.
](/app-services-apps-translations)[### Formatter→
Control how values are displayed.
](/app-services-apps-formatter)[### Git integration→
Link an external Git repository to the app.
](/app-services-apps-git-integration)[### Custom domain→
Serve the app on your own domain name.
](/app-services-apps-custom-domain)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-deploy.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create and deploy an app
## Objective
Creating an app, building an artifact from it, deploying a version and pointing a domain at it are one workflow. This page covers that workflow and the settings that go with it.
What each part of a finished app does, dashboards, extensions, style, menus and the rest, is covered from the [Apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) page.
## Create an application
Generally speaking, the steps to create and deploy an application are as follows:
1. Choosing a template from the Marketplace or importing an existing code
2. Choosing extensions, including data visualization libraries
3. Building dashboards
4. Adapting organization/navigation
5. Adapting design and style (optional)
6. Formatting data and translating labels (optional)
7. Managing versions and deployment
Initiate your application from **Application Services** by clicking **Create an application**.
You will be able to choose between **an existing app template** from the internal _Platform Store_, or to **import your own code** by linking a Git repository.
Fill in the different fields as shown in the screenshot below and press **Confirm**.
Click on **Confirm**, your app will automatically build and deploy. The whole process should take a couple of minutes.
From the APP manager component, you can then edit your app and manage its versions and deployments.
## Manage an app through the interface
### The Preview tab
In the management interface of an application, the _Preview_ tab allows to monitor the life cycle of an application in an easy way.
The screen is divided into 3 parts:
- The top block (green) shows some **indicators related to the resources used by the application**. These indicators are useful, for example, to know if it is necessary to increase the number of instances.
- The left block (yellow) indicates **the status of the active version** (in production). Some shortcuts are possible according to the state (New / Constructed / Deployed) of the version.
- The right block (blue) displays **different versions of the application**. The active version is displayed in blue. Different actions are possible for each version:
- Synchronize / Download - action depending on the type of code source (ZIP file or GIT repository)
- Duplicate version
- Build this version
- Deploy / Stop version
- Open the building console
- Remove version - requires the version not to be deployed
### Manage versions and deployments
You will arrive on the "Dashboard" of your application.
Here are three elements:
- **Summary**: status of your application once it has been deployed
- **Active**: status of your "active" version, that is to say, the one you are currently working on
- **Tags**: list of all the versions available in your application
You can see more details about the version you just created using the arrow on the right of it.
Since you activated auto-build, it has already started. You can check the activity log by clicking on "Open Console".
When it is done, you should see a last log "EndBuild success".
To start the deployment, please click on "Deploy 1.0.0".
Deployment starts and may take some time.
When the deployment is complete, the "Open" button appears. If you open your application, you should see this screen.
You can now login with your "DataPlatform" account or with a "Project" account created directly within your IAM.
## Application settings
The settings of a deployed application hold its global information (name, description and main image), its environment variables, its Git repository link and its domain names.
Environment variables are the values an application needs at runtime. Those required by the application itself and by its extensions are filled in automatically, so only the variables specific to a business need appear here.
An application is always reachable on a generated URL under the `eu.dataplatform.ovh.net` domain, unless another domain is set at organization level. Additional URLs can be added on top of that one.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-dashboards.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build application dashboards with graphical components
## Objective
The Platform allows you to create custom analytics applications using a set of graphical components made available. The **management and organization of the restitution** is centralized in the _Dashboard_ tab of an application.
This tab lets you list, add, and edit dashboards that can contain various types of content:
- Graphical representations from different extensions installed in the application
- Dynamic parameter that can be linked to graphics to make the data displayed interactive
- Static texts
- Blocks with tabs, to go from one graphic representation to another for the same data for example
- Blocks that can also contain different types of content with or without display condition
- Pop-ins activated at the click of the user
- Code integration specific to the current Project
:::warning
The Dashboard tab will only be available once the application is deployed.
:::
## Create your first dashboard (in point & click)
Before you begin, you must first have an application already deployed.
To do this, you can start from the ReactJS - Dashlab application, for example, and use automatic build and deployment.
In this example, the screenshots were made from the same application theme.
### Main screen and dashboard creation
To get started, go to the Overview screen of your application.
From this screen, select "Dashboard"
From this screen, you will be able to create new dashboards, delete them, edit them, etc.
For the moment we will focus on creating a new dashboard.
1. Select "New Dashboard" on your screen.
2. Give it a name, for example: "My first dashboard".
The url of this dashboard (visible in the url once the application in production), will be filled automatically.
You can edit it, just kept in mind that it must be unique, two dashboards cannot have the same url.
The description field is optional and will allow you to better document your applications.
We will not (intentionally) go through the "advanced" mode, it is an option to modify the default behavior of the dashboard. This default mode is perfect for a first time, so we don't have to change it.
## Edit your new dashboard
### Add a link to your dashboard
Once the dashboard is created, you can enter the editor mode by hovering over the new block and clicking on "Edit".
You have a "Full Screen" mode available at the top right of the page, all screenshots will be made from this mode for the next steps of this tutorial.
Your dashboard is empty! and more importantly, if you were not already on the page, there is no link to get to that dashboard.
So first, we need to create a link in the menu to go to the dashboard.
To do this, use the "+" button, in the left column. Select "Basic UI" and then "Link".
You will then arrive on a link editor.
You can enter a text (which will be displayed on the screen), in our example: "My first dashboard".
In the url field, if you did not change it in the previous step, select: "my-first-dashboard".
Also select "Navigation" to switch to active, this allows the link to change color, when it is the active page.
The rest of the fields are optional but let you customize further your link. In this example we will just add "fa fa-dashboard" in "Icon", this corresponds to an icon of the library [Font Awesome 4](https://fontawesome.com/v4.7.0/icons/).
Once validated, your link will appear in the menu on the left!
### Add a chart to your dashboard
Your page is still empty: your first thought could be to add a graphic, and you would be right... but in order to make everything "prettier" we will start by adding a "panel".
This will allow us to have our charts on a white background, instead of the "gray" background of the application.
To do this, click on the "+" at the top of the screen.
Select "Container" and then "Panel".
Then expand your "panel" via the arrows available when hovering your mouse over it.
Select the "+" of this panel, then "Chart" to add your first graph inside this "panel".
## Configuring a chart
:::details Chart editor tabs
### Type
You will then come to a chart editor.
You first need to select a chart type.
For our example, we will select "Morris - Donut"
:::info
Morris is an open source library, the donut it provides is pretty good, but the customization of it is low.
:::
### Options
A new tab has just appeared! This is the "Options" tab. This tab is not always present, some libraries do not always offer (yet) click customization.
In this tab, we can customize our donut's colors. The default template being colorful enough, modify the colors if you wish, but it is not necessary.
### Queries
In this tab, we will be able to link our graph to a query made in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
If you've ever tried to do this, click on "Select a Query".
If not, choose "Custom Query" and create a query directly via the interface.
To display a "nice donut", the ideal would be a single "Data", and a single "Scale".
In my example, I want to reduce a turnover.
### Infos
In this section, we will be able to give a name and description to our chart (optional).
Enter for example: "My first chart", in the field name.
You will also find "Linked Dynamic Parameter".
This configuration will allow you soon to connect your graph to a datepicker or a selectbox, so as to dynamically filter the data displayed by the graph.
### Advanced mode
This advanced mode allows you to directly edit the configuration JSON file of your chart.
All previously used screens are related to this JSON, and we do not yet have an interface to perform all possible configurations via this JSON.
Validate, and your graph will be displayed!
:::
[Next article: Style](https://docs.dataplatform.ovh.net/app-services-apps-style.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-style.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Customize your application style with CSS and LESS
## Objective
Within an application, the Platform allows to modify the style and design. The definition of the graphic chart to apply to the application is centralized in the _Style_ tab. These changes in CSS / LESS can be applied in different files as in the image below:
Changes to the application's style require rebuilding the artifact as well as redeploying it.
[↪ Next article: Extensions](https://docs.dataplatform.ovh.net/app-services-apps-extensions.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-extensions.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Add features to your application with extensions
## Objective
Extensions are a list of packages, accessible through the extensions tab, which can provide your application with additional features, such as a third-party chart software, translation services, or dynamic parameters.
## Manage your extensions
The _Extensions_ tab of the APP Manager lets you view and manage the different supported libraries:
- **Add** extensions that you need for your Project
- **Configure** Global settings for the extension, where applicable
- **Delete** extensions that are already installed if they are not necessary, or if you need a new version of the library.
- **Get more information** concerning extensions linked to external libraries.
- **Import** an open-source or proprietary Project-specific or organization-specific extension
:::warning
Adding an extension requires you to reload your app (on the Overview page) in order to be able to build it.
:::
## Customize the extensions
Each extension within the Data Platform marketplace has its own specific configuration in a JSON file that you can edit and customize.
[Hands-on tutorial on configuring extensions](https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md)
## Available extensions
Four types of extensions are available: system extensions, platform utilities, dynamic parameters and charts libraries.
:::details System extensions
These extensions are the core part of the Data Platform SDK to build applications. They should always be installed.
| Identifier | Name | Description |
| :-------------------- | :------------------ | :-------------------------------------------------------------------------------------------------- |
| **api** | Api | Make queries to the API of your front-end application |
| **chart** | Graphique | Load a chart |
| **core-ui** | Core UI | This component includes graphic components necessary to the app creation |
| **dashboarding** | Dashboarding | Build a dashboard |
| **dynamic-parameter** | Paramètre dynamique | A core component to use radio buttons, datepicker, selectbox... |
| **formatter** | Formatter | Allows you to format data extracted from your project |
| **menu** | Menu | Build navigation menus |
| **query-builder** | Query builder | Make queries to the Analytics Manager via an API |
| **sdk** | SDK | Development kit essential for the good operation of the other extensions |
| **store** | Store | Add a data store to your application. Essential to the development of modern JavaScript application |
| **toaster** | Toaster | Display messages as toast at the bottom of the screen |
| **translate** | Translate | This is the translation system for the app |
| **xhr** | Xhr | Make queries to various APIs |
| **webpack** | Webpack builder | An application builder used to make all extensions work together |
:::
:::details Platform utilities
These extensions manage other Platform services but they are not 100% required to create an application.
| Identifier | Name | Description |
| :--------------------------- | :---------------------- | :--------------------------------------- |
| **client-authority-manager** | Identity Access Manager | Plug the Platform authentication manager |
:::
:::details Dynamic parameters
These are all the possible dynamic parameters you can use to dynamically filter values in applications.
Find more information about dynamic parameters here:
- [configure dynamic parameters](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters.md)
- [create custom dynamic parameters](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-create.md)
| Identifier | Name | Description |
| :------------------------------- | :------------------- | :----------------------------------- |
| **dynamic-parameter-datepicker** | Date range selection | Select a date range |
| **dynamic-parameter-selectbox** | Select Box | Choose value(s) in a list box |
| **dynamic-parameter-checkbox** | Checkbox | Choose value(s) as check boxes |
| **dynamic-parameter-radio** | Radio | Choose value though radio buttons |
| **dynamic-parameter-toggle** | Toggle | Choose value though an On/Off toggle |
:::
:::details Charts libraries
These extensions helps integrate chart libraries to render data visualization.
Find more information about chart libraries here:
- [configure charts](https://docs.dataplatform.ovh.net/sdk/app-charts.md)
- [create custom charts](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md)
| Identifier | Name | Description |
| :--------------- | :----------- | :---------------------------------------------------------------------- |
| **chart-table** | Chart table | Display as a table |
| **chartist** | Chartist JS | Graphic responsive and easy-to-use library |
| **chartjs** | ChartJs | JavaScript Graphic library, flexible for designers and developers |
| **echarts** | Echarts | A very advanced graphical library that has all the tools you will need. |
| **google-chart** | Google Chart | Google graphical tools are powerful, easy to use and free |
| **jvectormap** | JVectorMap | Add interactive SVG cards to your dashboards |
| **morris** | Morris Chart | Morris Chart |
| **recharts** | ReCharts | ReCharts |
:::
Ready to move on? Discover the next tab of the APP component:
[Next article: Menu](https://docs.dataplatform.ovh.net/app-services-apps-menus.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-menus.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure application menus and navigation
## Objective
The Platform makes it possible and simple to build complex analytics applications - with multiple pages, redirects, multiple dashboards and performance indicators - via Drag & Drop options and ease of use.
A menu allows you to structure the application and its basic chapters. The management of this navigation is centralized in the Menus tab of an application. This tab is used to define one or more navigation menus, as well as the containers and blocks that compose them.
A container corresponds to a subsection of a menu. A container can be used to display a logo, a set of tabs, a disconnect button, and so on.
A block is an element of the container, such as:
- A redirection link to an application dashboard or an external URL
- Static text
- An image, to display a logo for example
- A block displaying the name of the active user
- A dynamic parameter that can be linked to graphs in the dashboards to make the displayed data interactive.
- A custom block, available in the Marketplace or developed specifically for this application.
## Interface
To access the menu interface, either select the menu tab from the _App-editor sidebar_, or , if you are actively editing your app, the main toolbar (top-middle) contains a menu icon that will also open the interface.
_Figure 1 : Menu editor interface_
From this interface, you can add new menus and menu elements, edit existing ones, and remove redundant components.
The different menus of the application are represented below each other like this:
_Figure 2 : Menu example_
Each menu is divided into light gray dotted containers which themselves contain different dark gray dotted elements. Each of these containers are editable, movable and deletable, just like the elements that compose it.
_Figure 3 : Editing a container_
:::warning
**Warning**: Adding a menu does not automatically add it to the application interface. The theme of the application must have a space dedicated to the reception of each menu. According to the theme, the name of a menu can have a lot of importance on its visual rendering.
:::
[Next articles: Translations](https://docs.dataplatform.ovh.net/app-services-apps-translations.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-translations.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Translate your application into multiple languages
## Objective
Applications created on the Platform are multilingual by nature. The fields to translate as well as the languages available for the application can be found in the Translations tab.
This tab allows you to:
- Add languages
- List all the expressions requiring a translation
- Add or edit the translation of these expressions in all added languages
The themes offered in the Marketplace have a number of translations to allow the application to work properly. It will therefore only be necessary to add translations when the expressions used are not included in the theme, such as the name of the data fields. Similarly, if an application is only built from the SDK, the majority of translations will have to be added.
## Interface
Figure 1 illustrates the translation editor interface, which allows you to:
- Add a language
- Add a translation key
- Save translations
_Figure 1 : Main actions_
When adding a new translation key, a new line will be created allowing you to enter the translations of these keys in each language (see Figure 2).
Adding a language only requires the language code (fr, en, es, etc.). This will create a new column where the translation can be entered (see Figure 2). Each translation key can then be modified directly or deleted.
_Figure 2: List of translations for an app_
## App Integration
### In charts
Many predefined charts provide a simple translating system.
#### Translate an axis
```json
"site" : "Site"
```
site will be replaced by "Site" in all charts
#### Translate a value
To translate an axis value, you need to use a combination of axis name and value
```json
"site-2" : "New York"
```
Instead of "2", all the charts will display "New York" for an axis "site"
[Next article: Formatter](https://docs.dataplatform.ovh.net/app-services-apps-formatter.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-formatter.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Format displayed data with the application formatter
## Objective
Data displayed in the different dashboards of an application are often very different, coming from various sources.
This sometimes implies that some data, that should be displayed together, do not have the same order of magnitude or the same unit. To overcome this, it is possible to manage, field by field:
- The unit
- The multiplying factor
- Decimal or thousands separators
- How many decimal places the value is shown too
Furthermore, for scales, it is possible to indicate the temporality of the data. All of this is configurable in the _Formatter_ tab of an application.
## Interface
The Formatter tab is divided into 3 parts corresponding to the 2 types of data used (_Measures and Scales_) and an essential element to the visualization of encrypted data: _Separators_.
_Figure 1 : Editing a measure_
In this tab, a measure is the name of one of the fields available in the Lakehouse Manager. These fields each represent a value. The editing panel shown in Figure 1 allows you to modify:
- **Unit**: The unit of measure for this data (euros, hours, kilometers, etc.)
- **Round**: The desired number of digits after the decimal point
- **Multiplier**: The multiplier lets you dynamically modify the data. This multiplier is particularly relevant when the data is a percentage in the form of a number between 0 and 1, which must therefore be multiplied by 100.
_Figure 2 : Editing a scale_
Unlike measures that represent a value, scales are fields that allow data grouping and segmentation. The most common scales are time dimensions, such as dates, months or years. The editing panel shown in Figure 2 allows you to edit:
- Type of scale (e.g. : temporal)
- Sub-type, giving a bit more detail of the type. Indeed, temporal data can be a given day, month, year, etc.
- Display format
_Figure 3 : Editing a separator_
Finally, there are two types of separators:
- Decimals
- Thousands
These separators can be any character and will be inserted to make the value more readable.
[↪ Learn more about formatters](https://docs.dataplatform.ovh.net/sdk/app-formatter.md)
[Application settings](https://docs.dataplatform.ovh.net/app-services-apps-deploy.md#application-settings)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-git-integration.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Link your app with a GIT repository
## Objective
:::info
Good to know: this tutorial also works for your APIs.
:::
## Pre-requisites
### Access rights definition
In this tutorial, we will explain how to give the read / write privileges on the Git repository for your application.
### Repository configuration
If you already know Git, and if you have a repository ready to use, go straight to the next step.
Otherwise, here is a list of repository providers:
- [Github](https://github.com)
- [Gitlab](https://gitlab.com)
- [Bitbucket](https://bitbucket.org)
Creating a first repository is quite simple, just be careful not to create a public repository if you want to store a confidential application.
## What to do
Once you've set up your repository, go to your app interface in order to link them.
Go to your APP Manager and click on Edit on the app you want to link.
On the left-hand side menu, go to your Settings. Now you have to fill in the following fields:
**Repository**: this is your repository's address. This is provided by your repository provider, and you can often find it on its homepage. This address can be either an HTTP URL, or a SSH address (depending on the connection mode you chose)
**SSH**: if you wish to use an SSH address, you have to copy the "Public Key" displayed on your screen in your remote repositories and accounts, in addition to the address entered above. The SSH key is automatically generated by your APP Manager, you don't need to generate one yourself.
**HTTPS**: this is the HTTPS address of your repository. Clone's URLS https\:// are available on all repositories, public or private. In the case of private repository, each interaction with the repository requires you to enter a user name and a password,
**Branch**: when you have completed the previous configurations, you can synchronize the remote branches via this button, then select one before saving.
Once the configuration is over, you can perform 2 new actions from the Overview screen:
**Push that version**: This lets you create a new branch on your Git repository, from this version. This is the first step to initialize your deposit.
**Synchronize**: This allows you to download the content of the previously selected branch, and automatically create a new version within your Data Platform environment.
### Setting up a repository with SSH authentication
When configuring your Application, if you select SSH as the connection mode, the required SSH key is automatically provided by your Application Manager.
This key must be copied / pasted into your Git repository. According to the providers, the screens to perform this action may change.
Let's take Github as an example, which is one of the most common providers.
To add SSH keys to a repository, you need to go to its "Settings".
Select "Deploy Keys", then click on "Add deploy key".\
Here you can paste your SSH key (provided by your APP Manager), give it a name, for example: Data Platform - MyFirstProject.\
Remember to check "Allow write Access" in order to be able to use synchronization in both directions. This option is necessary for the "Push the version" function that we discussed in the previous point.
### Setting up a repository with HTTPS authentication
#### Private authentication
Regarding the creation of an HTTPS authentication, you just need to fill in the fields "username" and password of your account (who has access to the repository). At each synchronization, the password will be asked again, we do not store it in your Project.
#### Public authentication
Regarding the public authentication, it is not necessary to fill in information.
Your application is now synced with the repository.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/app-services-apps-custom-domain.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure a custom domain for your application
## Objective
**Why would you want to create your own domain?** Now, let's imagine that you have a project with a new application that is accessible from the following URL:
```yaml
https://my-project.eu.dataplatform.ovh.net/my-app/
```
It is easy to change your application's subdomains by changing its global settings but it'll always be followed by the domain `eu.dataplatform.ovh.net` such as for instance:
```yaml
https://my-app-on-my-project.eu.dataplatform.ovh.net/
```
**What if you would like to get rid of the `eu.dataplatform.ovh.net` domain entirely?** You would like to completely white label the final product you are building. Your application URL could look like anything you fancy as long as its available, such as:
```yaml
https://my-app.example.com/
```
Let's check-out how to do this with the following steps:
1. [Generate a private Key and .csr file](https://docs.dataplatform.ovh.net/app-services-apps-custom-domain.md#generate-a-private-key-and-a-csr-file)
2. [Obtaining .pem file using your private key](https://docs.dataplatform.ovh.net/app-services-apps-custom-domain.md#obtaining-a-crt--pem-file)
3. [Register your domain name at organization level](https://docs.dataplatform.ovh.net/app-services-apps-custom-domain.md#register-domain-for-your-organization)
4. [Register your new application domain name](https://docs.dataplatform.ovh.net/app-services-apps-custom-domain.md#register-domain-for-your-application)
## Requirements
To attempt following the steps indicated in this article, we highly recommend that you already know about:
- How to install & use [OpenSSL](https://www.openssl.org/) tools
- How to generate a valid [SSL Certificate](https://en.wikipedia.org/wiki/Transport_Layer_Security)
- How to add a [hostname](https://en.wikipedia.org/wiki/Hostname) to your domain
:::warning
Please note that the following steps also assumes that you **own an existing Domain**. If that's not the case, make sure to first create your own domain so you can use it to for your application.
:::
If you do not have an existing domain, check out online to learn how to do so, there are many available options such as the [Google Domains](https://domains.google/) tool.
If you fulfill all the above criteria, we'll need to go through one last step before setting up our application's custom domain, that is _generate a private key and CSR file_. See the following section to do that.
## Generate a Private Key and a CSR file
### Required Information
To generate a CSR (Certificate Signed Request), you will need to provide following informations:
| Parameter | Description |
| :--------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **C** (Country Name) | Use the two-letter code without punctuation for country. _Example: US or CA_ |
| **ST** (State or Province) | Spell out the state completely; do not abbreviate the state or province name. _Example: California_ |
| **L** (Locality or City) | The Locality field is the city or town name, for example: Berkeley. Do not abbreviate. _Example: Saint Louis, not St. Louis_ |
| **O** (Company) | If the company or department has an &, @, or any other symbol using the shift key in its name, the symbol must be spelled out or omitted, in order to enroll. _Example: XY & Z Corporation would be XYZ Corporation or XY and Z Corporation_ |
| **OU** (Organizational Unit) | The Organizational Unit (OU) field is the name of the department or organization unit making the request. To skip the OU field, press Enter on the keyboard. |
| **CN** (Common Name) | The Common Name is the Host + Domain Name. It looks like "[www.company.com](http://www.company.com)" or "company.com". |
| **emailAddress** | E-mail address of a real contact for the domain name |
Open a simple text editor and create a new **my-app.txt** file with the following structure:
```txt
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=FR
ST=Hauts-de-Seine
L=Neuilly-sur-Seine
O=MyCompany
OU=MyCompany
emailAddress=contact@example.com
CN=my-app.example.com
```
:::warning
Obviously replace the parameters as described in the table above with your own information.
:::
### Creating files using OpenSSL
In your personal command line environment execute the following bash commands:
```bash
openssl req -new -sha256 -nodes -out my-app.csr -newkey rsa:2048 -keyout my-app.key -config my-app.txt
```
That will create 2 separate files:
- **my-app.csr** that you will be need to provide to SSL Certification Authority
- **my-app.key** that is your personal private key which you should keep securely and not communicate
:::warning
If you are new to creating custom domain, make sure to keep the application key secret! Read-more about [OpenSSL Essentials in this article from Digital Ocean](https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs#private-keys).
:::
## Obtaining a CRT & PEM file
Now that we have the `.csr` file, we will be able to use it to obtain a Certificate (CRT) & Privacy Enhanced Mail (PEM) file. To do that, you should simply provide the my-app.csr file to your favorite SSL Certification Authority which will provide you a valid `.crt` & `.pem` file.
Before using the `.pem`file let's make sure that it's valid. To do that you can run the following commands which should all yield the same output.
```bash
openssl rsa -noout -modulus -in my-app.key | openssl md5
openssl req -noout -modulus -in my-app.csr | openssl md5
openssl x509 -noout -modulus -in my-app.pem | openssl md5
```
## Register the new domain to your organization & application
To use the new domain we'll first need to register it at the **Organization** level so every project or applications inside the project can leverage it. We will then need to head to the **applications** themselves to set-it up individually.
### Register domain for your organization
Head over to the _Organization_ information, access the _Settings_ section and navigate to the _Infrastructure_ tab.
Click on **Add a Certificate**. In the new certification window, enter your information:
- _Name_: a descriptive name for your own usage (has no incidence on the actual certificate name)
- _Domain name_: the hostname associates with the SSL certificate that you generated
- _Server certificate_: copy the content of the `.pem`file provided by your SSL Certification Authority
- _Key_: copy the content of the `.key`file that you created using OpenSSL
Confirm the creation of your name certificate and then click on **Add a Domain**. In the new domain window, make sure to give the same domain as the one for your certificate.
That's it! You're good to go. Your SSL certificate is now set up for your organization. **All your projects and applications deployed within can use this new domain.** Let's head to the application settings to set-up this custom DNS.
### Register domain for your application
Navigate to the project and edit the application that you would like to change the DNS for. Head to the _Settings_ tab and go the _Domain Name System_ section.
Following our example, create a DNS entry for `my-app.example.com` as a CNAME to `my-project.eu.dataplatform.ovh.net`, where:
- `my-app.example.com`is the hostname you provided an SSL certificate for
- `my-project.eu.dataplatform.ovh.net` is the project's main URL
Your application is now available at your own URL, for example `https://my-app.example.com/`.
## Struggling to set it up by yourself?
That's fine, tell us how we can help. Get in touch via the [OVHcloud Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
[Send your questions on the support portal](https://help.ovhcloud.com/csm?id=csm_get_help)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-control-center.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Control Center
## What is the Control Center?
The Control Center is the observability surface of a Data Platform project. It collects metrics across data pipelines, applications and deployed instances, drills down into logs, raises alerts when something goes wrong, and is where components get more resources.
It opens at two levels. From a project home page it reports on that project. From the header buttons it reports across every project in the organization the user [has permissions on](https://docs.dataplatform.ovh.net/iam-users.md#manage-organization-iam-users), each returning its own metrics. Most metrics can be filtered by timeframe, project and cloud provider.
[Monitoring](https://docs.dataplatform.ovh.net/control-center-monitoring.md) covers CPU, memory, storage and network use per component. [Alerting](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md) turns those metrics into email notifications with configurable trigger conditions. [Jobs Central](https://docs.dataplatform.ovh.net/control-center-jobs-central.md) tracks job executions from one place. [Logs Explorer](https://docs.dataplatform.ovh.net/control-center-logs-explorer.md) streams current activity or reads back past logs. [Auditing](https://docs.dataplatform.ovh.net/control-center-auditing.md) records who changed what in the project.
Scaling also happens here, from the monitoring panel of the component concerned. Project [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md), [apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) and [Data Processing Engine jobs](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) scale both horizontally and vertically. The [Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-resources.md) and the Application Service component scale vertically.
[Allocate more resources to a component](https://docs.dataplatform.ovh.net/control-center-monitoring.md#allocate-more-resources-to-a-component-in-the-control-center)
[### Monitoring→
CPU, memory, storage and network per component, and where to scale it.
](/control-center-monitoring)[### Alerting→
Notify people by email when a job fails or a deployment degrades.
](/landing-page-control-center-alerting)[### Jobs Central→
Track every job execution from one place.
](/control-center-jobs-central)[### Logs Explorer→
Stream live logs or read back a past timeframe.
](/control-center-logs-explorer)[### Auditing→
Who changed what in the project, and when.
](/control-center-auditing)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-monitoring.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Monitor Project infrastructure metrics
## Objective
Monitoring panels are **dashboards built to monitor key infrastructure metrics** about all the elements in a Project. Giving a bird's-eye view of your entire data Project's health status, from your data processing job executions to the RAM usage of your APIs and apps, monitoring panels can even be opened at the level of the [organization](https://docs.dataplatform.ovh.net/manage-organizations-settings.md) to keep an eye on all your Projects at once.
By default, Projects come with a few default monitoring panels, to watch the basic metrics of your data system, and react to failures by allocating more resources.
## The Platform default monitoring panels
A few monitoring panels are available by default when you open the Control Center: _Infrastructure Monitoring_ and _Overview_ panels. Those panels are non-editable and give you insight on the active services in your data Project.
### _Infrastructure Monitoring_ panels
Those panels give you full visibility on all the infrastructure elements deployed in [each component of your Project](https://docs.dataplatform.ovh.net/manage-projects.md#what-services-are-included-in-a-project), and allow you to edit the [allocated resources](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units) of jobs and deployments.
They generally list the following things:
- [Deployed instances](#deployed-instances)
- If applicable, [serverless executions](#serverless-executions)
#### Deployed instances
Displayed with a server icon, Infrastructure Monitoring panels list **all instances that are currently deployed**, along with their metrics.
If the monitoring panel is opened at the level of the organization, each different Project will be color coded (cyan, grey blue, navy blue, pink red, and so on). Each deployed instance visible on the monitoring panel will bear the colors of its Project.
Each deployed instance shows [CPU](https://en.wikipedia.org/wiki/CPU_time) usage _(in % of total allocated)_, [RAM](https://en.wikipedia.org/wiki/Random-access_memory) usage _(in % of total allocated)_ and **Network** _(total in and out traffic since the birth of the service - in GB of data moved)_. Some instances, mostly APIs, also show [requests-per-second (RPS)](https://en.wikipedia.org/wiki/Web_server#requests_per_second) and [Latency](https://en.wikipedia.org/wiki/Latency_\(engineering\)) _(in s)_ metrics.
Those values are displayed both at the level of the deployment, as well as broken down per worker in it - in which case only the instant current value is displayed. The **age** of each worker (time elapsed since it was created or last hard-rebooted), as well as the number of automatic **restarts** over its age is also shown.
It is possible to check the execution logs of each worker in the deployment, and to reboot it if needed.
Finally, [always-up deployments](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up) show the [actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) / [workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) that use them.
#### Serverless executions
When it is applicable to the component, the corresponding monitoring panel displays the [serverless](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#serverless) jobs with their execution metrics (status, runtime, etc). Executed objects are ranked by descending value of latest computing power consumption, making this view typically useful to watch at a glance the jobs that run fine and those that need increased attention or computing power.
#### Allocate more resources to a component in the Control Center
For the components that can be scaled horizontally, you can **parallelize their workload** on several containers by selecting the number of parallel instances on which they will run.
For the components that can be scaled vertically, you can allocate **more computing power** by increasing the number of [Data Platform Units](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units) (DPU) of each instance. The DPU is a unit of processing capability, representing access to a certain amount of CPU and memory, as such:
- **DPU**: general processing - corresponds to approximately _1 vCPU_ and _4 GB of RAM_, based on hardware availability.
Increase the vertical/horizontal resources for the desired component by clicking on the **gears** button.
### _Overview_ panels
Those panels give you a high-level overview of the organization-level products you are using for your Projects: [Projects](https://docs.dataplatform.ovh.net/manage-projects.md) and [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md).
They are integrated with the rest of the platform, and allow you to edit configurations such as the [allocated resources](https://docs.dataplatform.ovh.net/manage-pricing.md) of instances.
## Custom monitoring panels
This feature will soon be available!
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Control Center alerts
## What is an alert?
An alert watches one target and notifies people when it misbehaves. Targets come in two kinds: the execution of jobs, such as [Data Processing Engine workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md), and long-running deployments, such as [apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) and [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md).
:::info
Alerting has been reworked and is available again on the platform. This is a **beta release**, so some issues may still surface. Please reach out on the Data Platform channel of the [Discord server](https://discord.com/channels/850031577277792286/1163465539981672559) if you run into anything unexpected.
:::
:::warning
In this release, alert notifications are delivered by **email only**. The previous integrations with Slack, AWS SNS, Opsgenie and custom webhooks are no longer available.
:::
The behaviour is the same for every alert. The platform evaluates the trigger condition against the target every minute. Once the condition has held for the duration configured on that alert, the alert fires, which logs a triggering and sends an email to every linked subscriber. It stays firing until the condition is no longer met, at which point it resolves itself and returns to watching.
An alert therefore has three statuses:
- `watching`: the default state of a created alert. It is monitoring its target, and the triggers have not held long enough to fire.
- `active`: the alert is firing. The trigger conditions have been met for the configured waiting duration.
- `error`: the alert status could not be retrieved.
Creating an alert takes two steps: create it with a target and its triggers, then link the [subscribers](https://docs.dataplatform.ovh.net/control-center-alerting-subscribers.md) who receive the email. Pick the type when you create it, since a job execution alert and a health check alert watch different things and cannot be converted into one another.
:::info
After you create an alert, it may take a couple of minutes before it is completely operational.
:::
[### Configure a job execution alert→
Alert on the status or metrics of a job run.
](/control-center-alerting-job-execution)[### Configure a health check alert→
Alert on the uptime or resource use of a deployment.
](/control-center-alerting-health-checks)[### Add subscribers to an alert→
Who receives the email, and the two gotchas when adding people.
](/control-center-alerting-subscribers)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-alerting-job-execution.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure a job execution alert
## Objective
Alerts can be set to monitor **job execution metrics** on the Platform, such as the status of the job execution or the average CPU usage during it.
A job execution alert is configured by specifying a [target](#configure-the-target-of-the-alert) as well as [trigger conditions](#configure-the-trigger-conditions-of-the-alert) and [triggering options](#configure-the-triggering-options-of-the-alert).
## Configure the target of the alert
When creating a new job execution alert, it is necessary to choose the type of job to monitor. The following types are available:
- [Data Processing Engine actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) job executions
- [Data Processing Engine workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) job executions
- [Lakehouse Manager physical build](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#build-table) jobs
If not done already, you also need to choose the Project in which to create the alert.
Except for alerts set on _Lakehouse Manager physical build_ jobs, it is possible to either target one specific object (workflow, action, etc..) or set the alert to monitor all of them at once.
If you choose to monitor all of them at once, the alert will be triggered **every time** a job execution of the aforementioned type matches the [trigger conditions](#configure-the-trigger-conditions-of-the-alert).\
For example in the screenshot below, the alert will be triggered every time any [Data Processing Engine workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) job execution fails.
If you choose to monitor one specific object, the alert will only be triggered if the job execution of that object meets the trigger conditions.\
For example in the screenshot below, the alert will only be triggered if the execution of workflow "chicago ml" lasts for more than 10 minutes (600s).
Finally, it is possible to set the alert to be triggered only if the _trigger condition_ is met during a [CRON-scheduled](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#time-based-trigger) execution of the job. This is useful to manually test your actions/workflows/pipelines while developing without receiving an alert every time.
## Configure the trigger conditions of the alert
A trigger condition on an alert is a logical statement made of a _metric_, an _operator_, and a _value_. Multiple conditions can be added to the same alert: in that case, they will be combined with an _AND_ condition. At each evaluation loop - occurring every minute - the Platform runs the expression defined in each alert trigger. The alert's status is thus changed to _active_ if **all trigger conditions are and stay true** for longer than the [waiting period of the alert](#configure-the-triggering-options-of-the-alert) (configurable downstream in the process).
To add a new trigger condition, press **Add** in the corresponding panel.
Triggers can be set on the following metrics for job execution:
- **Status**: note that failure and timeout are two different values: an execution that times out will not trigger an alert set on _status = failure_
- **Maximum CPU usage** (in %)
- **Average CPU usage** (in %)
- **Maximum RAM usage** (in %)
- **Average RAM usage** (in %)
- **Runtime** (in s)
- **Log detection** (_coming soon!_)
## Configure the triggering options of the alert
The triggering options are used to specify how exactly is the alert triggered, based on the trigger set above, and how are notifications sent to subscribers.
### Specify the waiting period for the alert
The _waiting period_ is the **duration during which the alert's [triggers](https://docs.dataplatform.ovh.net/control-center-alerting-health-checks.md#configure-the-trigger-conditions-of-the-alert) must stay true** for the alert to be effectively triggered.
This optional parameter causes the Platform to wait for a certain duration between first meeting the trigger condition of the alert (like a failed execution) and counting the alert as _active_.
If this duration is 0 (default value), the alert will immediately be triggered to an _active_ status as soon as an execution validating the trigger conditions finishes.\
If it is higher than 0, the alert will stay in _watching_ status for the whole duration after the first execution. If all executions occurring in this timeframe validate the trigger condition, the alert will be triggered to an _active_ status
:::info
Since the Platform only evaluates trigger conditions every minute, it may take a few more seconds (up to one minute) of delay on top of the waiting duration before the system effectively acknowledges the alert has being triggered.
:::
:::info
Let's take the example of a workflow, which is scheduled to run every 15 minutes. You set an alert on it with a waiting duration of 1 hour. On the first execution that matches the trigger condition, the alert won't be activated but a silent countdown of 60 minutes will begin. If all executions within the 1-hour timeframe validate the condition, the alert will be triggered. In other words, you essentially allow the execution to match the trigger conditions 3 times in a row, before transitioning the alert to **active** at the 4th execution if it still matches the conditions.
:::
### Specify the notification frequency
For each alert, it is possible to either:
- notify the subscribers **only once at the moment of triggering** (when the alert transitions to _active_ state). Subscribers won't be notified again until the alert is resolved and triggered again.
- notify the subscribers regularly **as long as the alert is active**. You can set the frequency of notification per alert.
It is also possible to activate resolution notifications. This will send a notification to all subscribers when the alert is resolved - which happens automatically as soon as the evaluated trigger condition is no longer verified.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-alerting-health-checks.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure a health check alert
## Objective
Alerts can be set to monitor **metrics of deployed instances** on the Platform, such as the status of APIs or the CPU usage of a query engine. They are called _health check alerts_ on the Platform.
A health check alert is configured by specifying a [target](#configure-the-target-of-the-alert) as well as [trigger conditions](#configure-the-trigger-conditions-of-the-alert) and [triggering options](#configure-the-triggering-options-of-the-alert).
## Configure the target of the alert
When creating a new health check alert, it is necessary to choose the type of deployment to monitor. The following types are available:
- [Always-up deployments in the Data Processing Engine](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up)
- [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md)
- [Applications](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md)
If not done already, you also need to choose the Project in which to create the alert.
If you choose to monitor one specific object (app, API, etc..), the alert will only be triggered if that specific instance meets the trigger conditions.\
For example in the screenshot below, the alert will only be triggered if API "public" uses more than 99% of its RAM for more than 1 minute.
## Configure the trigger conditions of the alert
A trigger condition on an alert is a logical statement made of a _metric_, an _operator_, and a _value_. Multiple conditions can be added to the same alert: in that case, they will be combined with an _AND_ condition. At each evaluation loop - occurring every minute - the Platform runs the expression defined in each alert trigger. The alert's status is thus changed to _active_ if **all trigger conditions are and stay true** for longer than the [waiting period of the alert](#specify-the-waiting-period-for-the-alert) (configurable downstream in the process).
To add a new trigger condition, press **Add** in the corresponding panel.
Triggers can be set on the following metrics for deployed instances:
- **Status**: an alert will be raised if the monitored target is unavailable (for value _DOWN_).
- Note that status _DOWN_ will not raise an alert if the monitored target is stopped/rebooted manually by a user
- **CPU usage** (in %): an alert will be raised if the CPU used by the monitored target goes above/below a certain threshold
- **RAM usage** (in %): an alert will be raised if the memory used by the monitored target goes above/below a certain threshold
- **Log detection** (_coming soon!_)
## Configure the triggering options of the alert
The triggering options are used to specify how exactly is the alert triggered, based on the trigger set above, and how are notifications sent to subscribers.
### Specify the waiting period for the alert
The _waiting period_ is the **duration during which the alert's [triggers](#configure-the-trigger-conditions-of-the-alert) must stay true** for the alert to be effectively triggered.
This optional parameter causes the Platform to wait for a certain duration between first meeting the trigger condition of the alert (like an instance with a high CPU usage) and counting the alert as _active_.
If this duration is 0 (default value), the alert will immediately be triggered to an _active_ status as soon as the trigger conditions are all true.\
If it is higher than 0, the alert will stay in _watching_ status until enough evaluation loops that keep validating the trigger condition are performed by the Platform to reach this waiting duration, after which it will be triggered to an _active_ status
:::info
Let's take the example of an alert on CPU usage > 99% for an API, where such duration is set to 5 minutes. The first time that the CPU goes above the threshold of 99%, an invisible countdown of 5 minutes will begin. If the CPU stays above 99% for more than 5 minutes, then the alert will transition to **active**.
:::
:::info
Since the Platform only evaluates trigger conditions every minute, it may take a few more seconds (up to one minute) of delay on top of the waiting duration before the system effectively acknowledges the alert has being triggered.
:::
### Specify the notification frequency
For each alert, it is possible to either:
- notify the subscribers **only once at the moment of triggering** (when the alert transitions to _active_ state). Subscribers won't be notified again until the alert is resolved and triggered again.
- notify the subscribers regularly **as long as the alert is active**. You can set the frequency of notification per alert.
It is also possible to activate resolution notifications. This will send a notification to all subscribers when the alert is resolved - which happens automatically as soon as the evaluated trigger condition is no longer verified.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-alerting-subscribers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Add subscribers to an alert
## Objective
Subscribers are the people who receive the email an [alert](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md) sends when its trigger conditions are met. Subscribers are set per alert, and the same list applies whether the alert watches a [job execution](https://docs.dataplatform.ovh.net/control-center-alerting-job-execution.md) or a [health check](https://docs.dataplatform.ovh.net/control-center-alerting-health-checks.md).
Subscribers are the users who receive the email notification sent when the trigger conditions (and triggering options) of the alert are met.
In the Subscribers tab of the alert, find the panel _Users_. This table lists all the users in your organization that are currently subscribed to the alert, meaning they will be notified of the alert on their [account](https://docs.dataplatform.ovh.net/manage-organizations-profile.md)'s email address.
To link a new user, click on **Add users** in the top right.
You can either add existing users from your organization, or whole new users. If you add a new user, they will automatically be added in your organization as simple users with no permission. Don't forget to [grant them access to the Projects](https://docs.dataplatform.ovh.net/iam-users.md#manage-organization-iam-users) if you want them to access the alert's page on the Platform after receiving alert emails.
:::warning
New subscribed users won't be able to receive notifications until they verify their email address.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-jobs-central.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Monitor all job executions in Jobs Central
## Objective
The Jobs Central is the watchtower from where all job executions on the platform - past, present, queued and scheduled - can be monitored.
On the platform, a 'job' (or 'job execution') is the unitary execution of code, which can be either launch manually from UI or API, or scheduled based on triggers. Many elements on the platform, such as [Data Processing Engine workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) and [Connectors metadata extraction](https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md), are executed in the form of jobs. The [computing power allocated to this job](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units) is usually set by the user. Some other processes on the platform, like the physical build of your data model, are also jobs fully managed by the platform
If the page is opened at the organization level, all jobs across all Projects will be visible.
## Monitor job executions
The _Job executions_ page lists the executions of the following types of jobs:
- [Data Processing Engine actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) job executions
- [Data Processing Engine workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) job executions
- [Connectors metadata extract](https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md) jobs
- [Lakehouse Manager physical build](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md) jobs
If needed, this page can be filtered on only the executions that have triggered by a CRON schedule.
## Monitor scheduled job triggers
The _Scheduled triggers_ page lists all the CRON-triggers that have been set on [Data Processing Engine actions and workflows](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#triggers).
It allows you to have an overview of the recurrent computing activity on your Projects.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-logs-explorer.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Query and export job logs with the Logs Explorer
## Objective
The Logs Explorer is a comprehensive tool to **query and export logs from job executions and live deployments** on the Data Platform.
Like all subcomponents in the Control Center, the Logs Explorer can be opened at the level of the organization. However, logs exploration will only be possible inside a specific Project. If the Logs Explorer is opened at the level of one Project, then this field cannot be edited.
## Get logs
### Get logs of past job executions
The Logs Explorer allows you to check the logs of:
- [Data Processing Engine actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) job executions
- [Data Processing Engine workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md) job executions
- [Connectors metadata extract](https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md) jobs
- [Lakehouse Manager physical build](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md) jobs
Choose _Jobs_ as the **Resource** on the left column.
Choose the **type of jobs** to explore.
By default, the Logs Explorer will display the logs for all jobs matching this type. You can drill-down on a specific object in the bottom left list - ranked by number of executions.
You can drill-down further by filtering on a specific **execution ID** or a specific time-range above the display.
:::warning
Make sure that the execution that you want to display is contained in the time-range that you are filtered on.
:::
:::info
Note that logs will be queried by successive batches of at most 250 logs. Scroll down in the console to load more logs.
:::
### Get logs of deployments
The Logs Explorer also allows you to check the logs of:
- [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md)
- [Applications](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md)
- [Notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md)
- [Always-up deployments in the Data Processing Engine](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up)
- The _App Service_ component: an infrastructure service to control APIs and apps
Choose _Deployments_ as the **Resource** on the left column.
Choose the **type of deployment** to explore.
By default, the Logs Explorer will display the logs for all deployments matching this type. You can drill-down on a specific object in the bottom left list - ranked by number of active workers. By default, the logs of all workers for that deployed object are aggregated altogether in the display.
You can drill-down further by filtering on a specific **worker ID** or a specific time-range above the display.
:::info
Note that logs will be queried by successive batches of at most 250 logs. Scroll down in the console to load more logs.
:::
### Use the streaming mode
By default, the Logs Explorer must be refreshed manually to display new logs.
Activate the **streaming mode** to receive the latest logs in real time.
## Filter logs
### Search in logs
To filter down your log result, it is possible to either search for an exact substring or to match to a [regular expression (Regex)](https://en.wikipedia.org/wiki/Regular_expression) using the search bar above the console. Searching something in the searchbar will **run a new log query on the whole timeframe** and return the logs matching the search filters.
By default, simply typing text in the searchbar will search for all logs containing this substring, ignoring lowercase/capital letters.
Alternatively, it is possible to start your search with any of the following operators:
- `=` followed by text: search for all logs containing this substring (case sensitive)
- `!=` followed by text: search for all logs not containing this substring (case sensitive)
- `~` followed by a regex: search for all logs matching the regular expression
- `!~` followed by a regex: search for all logs that don't match the regular expression
### Severity
Logs have different level of severity - ranged from _INFO_ to _CRITICAL_ - to filter on.
:::info
If you don't see all the levels that you should be seeing, make sure your jobs are [configured to export the correct level](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#logs-level) of logs.
:::
### Timestamp
As mentioned above, logs are all associated to a specific timestamp, so they can be filtered on a time-range. For jobs execution logs, it is also possible to filter on one specific execution.
## Share logs
Logs can be shared in one of two ways: by sharing the link to the Logs Explorer or by exporting a text file containing the logs.
### Share an Explorer link
Click on the **link** icon to get a link for the page with the current filter configuration.
### Export a log file
Click on the **export** icon to download a .log file containing the logs currently displayed in the UI.
:::info
Share those log files to the Data Platform Support team whenever you need help debugging a Project on Data Platform.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/control-center-auditing.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Audit activity on Data Platform for compliance
## Objective
**Auditing** logs, stores and exposes all significant activity happening on Data Platform, so you can meet compliance obligations, enforce internal governance, and provide transparency to your stakeholders.
Every create, update, delete and administrative action performed on a resource is recorded as an audit event. Events are retained and can be filtered, browsed and reviewed by authorized users.
:::info
**Initial release.** Auditing is being rolled out progressively. Additional services and audit event types will be added in upcoming releases. The lists on this page will be kept up to date as coverage expands.
:::
## Access levels
Auditing is exposed at **two levels**:
| Level | Where | What it contains |
| :--------------- | :------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Project** | Control Center → Auditing (inside a Project) | Events scoped to a single Project: Connectors, Lakehouse Manager, Data Processing Engine, Analytics Manager and project IAM activity |
| **Organization** | Organization Settings → Auditing | Events scoped to the organization itself. Currently covers organization IAM and member management. Coverage for billing, quotas and similar organization-level activity is coming in a future release |
:::warning
Organization-level audit events **do not** include activity that happened inside a Project. To review project activity, open the corresponding Project and use its own Auditing page.
:::
## Project-level auditing
In the [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md) of any Project, open the **Auditing** tab to see every significant event that happened inside that Project.
Each event is displayed with the following columns, in order:
| Column | Description |
| :----------- | :-------------------------------------------------------------------------------------------- |
| **Time** | When the action occurred |
| **User** | User or service account that performed the action |
| **Action** | The operation performed (create, update, delete, etc.) |
| **Resource** | The specific resource acted upon (e.g. a table, workflow, dashboard) |
| **Service** | The service the event comes from (Connectors, Lakehouse Manager, DPE, Analytics Manager, IAM) |
You can [search](#search) the events by text, [filter](#filters) them by service, resource type and a custom time window, [export](#export) them as a file, open the [Security events](#security-events) tab, and switch to the [organization-level](#organization-level-auditing) view at any time from the same page.
### Event details
Click the **info icon** at the end of any row to open the event detail panel. From there, two views are available:
- **Event details**: the full payload of the event: actor, action, target resource, timestamps and any metadata recorded with the event.
- **Comparison**: a side-by-side diff of the resource _before_ and _after_ the action, so you can see exactly what changed.
## Organization-level auditing
From **Organization Settings**, open the **Auditing** tab to see organization-level activity.
Currently tracked at the organization level:
- Organization IAM (roles, policies, authentication providers)
- Member management (invitations, removals, role assignments)
Coming in upcoming releases:
- Billing activity
- Quotas
- Other organization-level settings
The organization-level view is scoped strictly to organization-level resources. For Project-internal activity, open the Project and use its own Auditing page.
## Security events
Both auditing pages have a **Security events** tab next to the audit trail. It lists the security events recorded at that scope. Click the **info icon** at the end of a row to open the event detail panel.
## Export
Click the **export** button above the event list to download the events currently listed as a file. The export is available on the audit trail and on the **Security events** tab, at Project and organization level. If an export fails, the error message tells you why.
## Roles
Auditing access is gated by two dedicated roles. The same two role names exist at **both levels** (organization and Project):
| Role | Capabilities |
| :--------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| **Audit Trail Admin** | Full management of auditing: view all events and configure audit settings at the scope of the role (organization or Project) |
| **Audit Trail Viewer** | Read-only access: view audit events at the scope of the role (organization or Project) |
A user needs the organization-level role to see the organization Auditing page, and the project-level role to see a Project's Auditing page. Assign them the same way as any other role from the [Users, Roles & Groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md) page.
## Search
Every audit view has a search field above the event list. Type any text to keep only the events that contain it, in any of their fields: a user name, a resource name, an action, or a value recorded in the event payload. The search combines with the [filters](#filters) below, so you can narrow to a service and a time window first, then search within the result.
## Filters
### Services (project level)
At the Project level, events can be filtered by service:
- **Connectors**
- **Lakehouse Manager**
- **Data Processing Engine**
- **Analytics Manager**
- **Identity Access Manager**
### Resources
Once a service is selected, the resource filter exposes the resource types available for that service. The full list of audited resource types is:
| Category | Resource types |
| :----------------------------------------- | :----------------------------------------------------------------------------------------------- |
| Lakehouse Manager & Connectors | Source, Table, Attribute, Rule, Dataset, Virtual Attribute, View, Policy Tag, Policy Tag Binding |
| Data Processing Engine & Analytics Manager | Query, Dashboard, Repository, Workflow, Action, Environment, Notebook, Application |
| Identity Access Manager | Role, Service Account, User, Group |
### Time window
Every audit view lets you scope events to a specific time window. You can pick one of the preset ranges or define your own:
- **Today**
- **Yesterday**
- **Last 1 hour**
- **Last 6 hours**
- **Last 24 hours**
- **Last 7 days**
- **Last 30 days**
- **Custom**: pick exact **From** and **To** dates and times on the calendar, with a selectable timezone.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-iam.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Identity Access Manager
## What is the Identity Access Manager?
The Identity Access Manager decides who can do what. It exists as two almost entirely independent modules: one for each [organization](https://docs.dataplatform.ovh.net/manage-organizations.md), one for each [project](https://docs.dataplatform.ovh.net/manage-projects.md).
[Organization IAM](https://docs.dataplatform.ovh.net/iam-organization.md) governs the organization itself: the [authentication settings](https://docs.dataplatform.ovh.net/iam-organization-authentication.md) for members signing in to Data Platform, single sign-on through an external provider, enforced multi-factor authentication, join requests, and the access rights members hold over organization-level resources and settings.
[Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md) governs one project, and reaches further. It covers the team members working inside the project and the end users of the applications it deploys. Access is role-based and granular: [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) are granted to users, to service accounts or to groups, and evaluated against project resources, down to individual table attributes where [policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) are involved. Project IAM also owns the login page of every deployed application and the authentication providers offered on it.
Three providers exist by default on every application: Data Platform IDs, Project IDs and API/Secret keys. Additional SSO or MFA providers are configured per project, and adding one means editing each application's authentication configuration before users see it in the login menu.
Authenticating against the platform programmatically, rather than through a login page, is covered by the [Authentication API](https://docs.dataplatform.ovh.net/developers-api-authentication.md).
Service accounts exist only in Project IAM. Every Data Processing Engine job runs as one, which is how a pipeline's permissions stay separate from those of the person who wrote it.
[### Organization IAM→
Members and their rights over organization-level resources.
](/iam-organization)[### Authentication settings→
SSO, enforced MFA and join requests for the organization.
](/iam-organization-authentication)[### MFA setup guide→
Enrol a second factor on your own account and recover it if you lose the device.
](/iam-organization-mfa-setup)[### Project IAM→
Access inside one project, for teams and for end users.
](/landing-page-iam-project)[### Users, roles and groups→
The principals in a project and the permissions they hold.
](/landing-page-iam-users)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-organization.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage organization authentication and access
## Objective
The Organization IAM is where you can manage all authentication requirements for members of your organization as well as their level of access to the organization's resources.
In the Organization IAM you will be able to:
- Set-up **authentication providers** to enable single-sign-on to your Data Platform Organization via external solutions.
- Enforce multi-factor authentication (MFA)
- Manage Organization members **access rights to the Organization-level resources and settings**.
You can **access this service from the sidebar**, as shown in the picture below:
## Resources managed by the Organization IAM
When using the Organization IAM, you will create roles and groups with permissions to access the following resources:
- [Billing Settings](https://docs.dataplatform.ovh.net/manage-pricing.md)
- [Control Center organization-level resources (alerts, logs, panels, consumers, etc)](https://docs.dataplatform.ovh.net/landing-page-control-center.md)
- [Organization IAM users, roles and groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md)
- [Organization's Projects deployed services and admins](#link-between-project-iam-and-organization-iam)
### Link between Project IAM and Organization IAM
While the Organization IAM provides manages access to projects within the organization, it does not allow management of access-rights of the resources within a project. In other words, the Organization IAM provides IT admins with a high-level access-rights management of the projects.
Below are the project-related permissions in the Organization IAM:
- _Projects | Project | Admin_: special permission that makes user become admin of the Project's IAM.
- _Projects | Project | List_: ability to view a list of existing projects in the organization and their basic information.
- _Projects | Project | Get_: ability to view basic information about a project
- _Projects | Project | Create_: ability to create a project.
- _Projects | Project | Update_: ability to update services inside a project.
- _Projects | Project | Delete_: ability to delete a project.
:::warning
Note that the **Project Admin permission is the only permission to overlap between both IAMs**. A user that has the _Project Admin_ permission in the Organization IAM has the _Admin_ permission in the underlying Project IAM(s).
:::
### Other Organization IAM permissions
Beyond the Project permissions, the access-rights to the following resources are managed in the Organization IAM via read, write and delete permissions:
- Organisation IAM
- _Users_
- _Roles_
- _Groups_
- _Authentication Settings_
- Control Center (organization-level)
- _Monitoring_
- _Alert_
- _Job_
- _Logs_
The only exception to this rule are the [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) whose permissions work slightly differently:
- **Create**: Create a dataset
- **Read**: See the dataset(s) of the organization
- **Update**: Change the amount of DPU allocated to the dataset(s)
- **Delete**: Delete the dataset(s)
## Authentication Settings
In the **Authentication Settings** tab, you can set-up & configure additional login requirements for all the users in the organization. To enforce security you can for instance delegate authentication to your third-party enterprise Single-Sign-On provider and require all users to verify their credentials on it to access the platform. Additionally, you can require all users to use multi-factor authentication to use projects or even allow users from a given email domain to request to join the organization.
[Authentication Settings](https://docs.dataplatform.ovh.net/iam-organization-authentication.md)
## Users, Groups and Roles
The Organisation IAM uses the same role-based access-rights control system of the [Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md): permissions are granted by giving users roles and adding them to groups.
[↪ Learn more about Users, Groups and Roles](https://docs.dataplatform.ovh.net/landing-page-iam-users.md)
### Accept or deny user requests to join your organization
It is possible for people with the **same email domain as any existing member** in an organization to request to join this organization when they first sign up. _Admins_ of the organization will have to manually accept the person into the organization after they requested to join.
To manage requests, click on the _Users_ tab. At the top of the list of users in your organization, you will see all the people that have requested to join.
You can accept or deny their request from here. If you accept, they will be added in your organization as simple _users_ with no permission.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-organization-authentication.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure organization authentication requirements
## Objective
In the Authentication Settings tab you can set global authentication and login requirements and configurations for users of the organization.
## Setting up a Single-Sign-On provider
### Setup
You can configure an external Single-Sign-On provider that can be used instead of their Data Platform account to validate the credentials of all users upon login.
To do so, first click on **+ Add Authentication provider** under the _Enterprise Authentication_ section.
Next, choose a [compatible authentication provider](https://docs.dataplatform.ovh.net/iam-auth-providers.md#list-of-authentication-providers) from our list.
After confirming your choice, you will have to configure your provider.
Once your provider is configured, it is operational ready-to-use.
### Enforcing SSO-login for all users
If your security needs require all your organization users to login using your enterprise SSO, you can enforce this login method by activating the _disable all other login methods_ option.
:::warning
Note that if you do not configure your provider properly and then disable all other login options, you will lock all unauthenticated users out of the platform. Therefore, be sure to properly configure your provider before disabling the other login methods.
:::
## Enforcing Multi-Factor Authentication
You can also require all users to setup and use Multi-Factor-Authentication (MFA) work on projects by checking the _Enforce MFA for all Data Platform accounts_ option.
Activating this option will require all organization users to verify a phone number of their choice before working on any project on Data Platform.
## Allow users to request to join organization
To avoid having to add many users that share the same email domain (e.g. your company's email domain), you can add a domain as _allowed to request to join_. Do this by typing the desired domain in the box indicated below and clicking on ENTER.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-organization-mfa-setup.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Multi-Factor Authentication (MFA) Setup Guide
## Objective
This guide will walk you through enabling Multi-Factor Authentication (MFA) on your account. MFA enhances your account's security by requiring a second form of verification in addition to your password. You can configure MFA using an authentication app or via SMS as a backup method.
## Steps to Enable MFA
### 1. Access Your Security Settings
- Log in to your account on the platform.
- Navigate to your profile settings by clicking on your user avatar in the top right corner.
- Select the **Security** tab to view your security preferences.
### 2. Enable Two-Factor Authentication (2FA)
- Under the **Two Factor Authentication** section, you will see two available methods for enabling MFA:
- **Authenticator App** (recommended for better security)
- **SMS Authentication** (can be used as a backup option)
You can choose either method to enable MFA, but we recommend setting up both for maximum security.
#### Option 1: Authenticator App
Follow these steps to enable MFA using an authenticator app:
1. **Download an Authenticator App**\
Download and install an app like Google Authenticator, Authy, or Microsoft Authenticator from your phone’s app store.
2. **Scan the QR Code**
- Open your authenticator app and select the option to add a new account.
- Scan the QR code displayed on your portal.
- Alternatively, you can manually enter the 2FA key shown below the QR code.
3. **Enter the Verification Code**\
After scanning the QR code, the authenticator app will generate a 6-digit code. Enter this code in the text field on the portal.
4. **Activate 2FA**\
Click **Activate** to complete the setup. You will receive a confirmation message indicating that MFA has been successfully enabled.
#### Option 2: SMS Backup Authentication
After enabling 2FA with the authenticator app, it is recommended to set up SMS as a backup method in case you lose access to your authenticator app or device:
:::warning
**Warning**: If you have not registered your phone number in the **Preferences** tab, the **Enable SMS Authentication** button will be disabled. To enable SMS-based authentication as a backup method, please ensure that your phone number is correctly registered in your account preferences.
:::
1. **Enter Your Phone Number**
- Under the **SMS Backup Authentication** section, enter your phone number. The system will display the country code and number fields for verification.
- Click **Enable SMS Authentication**.
2. **Download Backup Codes** (Recommended)
Once MFA is activated, **you can only download your backup codes during the initial setup**. This is a security measure to ensure that the codes are not easily accessible after the setup is complete, minimizing the risk of unauthorized access to your account.
**Important:** Make sure to store these backup codes in a secure location because **if you lose access to both your authenticator app and phone, these codes are the only way to regain access to your account**. You will not be able to download them again later.
### 3. Final Confirmation
After successfully enabling MFA via an authenticator app and optionally via SMS, your account is now secured with two-factor authentication. You can proceed without SMS authentication if you prefer, but we strongly recommend setting it up to avoid losing access to your account.
## Important Notes
- **Backup Codes Can Only Be Downloaded Once**: For security reasons, the backup codes can only be downloaded during the initial setup. This ensures that your account remains secure by preventing anyone from obtaining the codes after you've set up MFA.
- **Choosing Between Authenticator App and SMS**: While you can choose between using an authenticator app or SMS as your primary 2FA method, the app-based method is considered more secure. We highly recommend configuring both to have a backup in case you lose access to one method.
## Conclusion
You have now successfully enabled multi-factor authentication on your account. MFA ensures an added layer of protection to your account, minimizing the risk of unauthorized access. Remember to store your backup codes in a secure location and keep your phone and authenticator app handy for future logins.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-iam-project.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Project IAM
## What does Project IAM cover?
Project IAM manages access inside one project: the team members who build it, and the end users who consume the applications it deploys.
Three things live here.
**Applications** lists every deployed app and API in the project, and is where each one's login page is defined: its design, its access URL, and which authentication modes are offered on it.
**Auth providers** configures the directories users authenticate against, from SAML, JWT and OpenID single sign-on through to MFA providers, on top of the three providers every project carries by default.
**Users, roles and groups** is the authorization side: which principals exist, what they are allowed to do, and how those grants are bundled.
Multi-factor authentication is managed with the auth providers, and applies to the Project ID method only. For an external provider, use that provider's own MFA rather than the platform's.
[### Application authentication→
Auth modes and the sign-in page of a deployed application.
](/iam-application-authentication)[### Auth providers→
Add SSO and MFA providers on top of the three defaults.
](/iam-auth-providers)[### Set up custom mappings→
Map external claims onto project roles and groups.
](/iam-auth-providers-custom-mapping)[### Set up Azure AD→
Sign in through Microsoft Entra ID.
](/iam-auth-providers-azure-ad)[### Set up Okta→
Sign in through Okta.
](/iam-auth-providers-okta)[### Set up Auth0→
Sign in through Auth0.
](/iam-auth-providers-auth0)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-application-authentication.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage authentication for deployed applications
## Objective
When an application is deployed on the platform, it is automatically created in the Identity Access Manager as a separate entity for which you can manage the **authentication protocols**. Applications include:
- **Your analytics applications**: individual applications that can be deployed from the [APP Manager](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md), as well as consumers that you can connect in the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md). This typically includes user-friendly interfaces to expose the data such as dashboards, reporting, or alerting tools.
- **Identity Access Manager**: accessible as an independent URL, `https://project-name.eu.dataplatform.ovh.net/iam`, the Identity Access Manager external application is used when you need users (say the administrator of a given application) to manage its user base without having to access the Platform.
For each deployed application, the platform lets you configure the application information, the authentication modes that are visible on the sign-in page, and the style of that sign-in page.
## Application information
- **ID**: the ID of the application in the IAM
- **Name**: the name of the application shown by default in its login page
- **Version**: version of the application currently deployed (not required, only for custom use cases)
- **Description**: description of the application
- **Tags**: tags of the application
- **URL**: Base URL of the application.
- **"Share URL" button**: Send by email the URL of the application to the selected users or groups of users.
:::info
Please note that you can edit the template in the Identity Access Manager _Settings > Email > Application invitation > Edit_
:::
- **Display URL**: Title of the application used in the email sent by the "Share URL" button
## Authentication Mode
The authentication page is an important part of your Platform application: it is the page on which users will log in in order to reach the content of the app.
The **Authentication Mode** list allows you to manage which authentication methods are displayed on the sign-in page of the application. The [Auth. Provider](https://docs.dataplatform.ovh.net/iam-auth-providers.md) tab in your IAM lets you configure which authentication methods can be used in your Project, and those methods can then be displayed or hidden to visually customize your login page.
:::warning
This is a pure visibility configuration and not a way to avoid users from logging into the application through certain authentication modes.
:::
### Display an authentication provider on an application login page
Click on **Applications** and edit the desired application.
Locate the panel **Authentication Mode**
Add your new provider if it doesn't appear automatically. Make sure the authentication providers of your choice are not set as _hidden_. Those who are will not be visible from the application's login page.
:::warning
It is not possible to **deactivate** an authentication provider for a specific application. For instance, a user who is already logged in through Azure AD will still be able to access the application even if Azure AD is set as hidden on the application login page.
:::
### Link to a specific authentication provider
You can generate URL links to reach a specific authentication page for your application, even if you have several authentication providers active on your Project. You will need:
- your app URL
- your authentication method's ID
You can get your app URL in the IAM settings of your application:
You can get your authentication provider's ID by going to its settings page in the IAM and looking at the URL:
To reach a specific authentication page for your application, use the URL link:
:::info
**your-app-URL**#/?auth\_mode\_id=**your-auth-method-ID**
:::
:::warning
This link is currently only valid for applications created from [Application Services](https://docs.dataplatform.ovh.net/landing-page-app-services.md), not for the 'Project' application.
:::
:::info
Shortcuts:
:::
- Another ID for the [_the platform account_](https://docs.dataplatform.ovh.net/iam-auth-providers.md#data-platform) authentication method is `king`
- Another ID for the [_Project_](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) authentication method is `password`
## Customize the sign-in page
The visual style of your sign-in page includes the following elements:
- **Welcome title**: edit the welcome message on the sign-in page. if empty, it will show the application name instead.
- **Welcome message**: the message under the welcome title
- **Color Theme**: default primary CSS color for the sign-in page
- **Background image**: adds a background image to the sign-in page.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-auth-providers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure authentication providers for your Project
## Objective
The Identity Access Manager (IAM) lets you configure the authentication methods used to login to **your Project and your applications**.
You can set any authentication provider as active or inactive. Setting an authentication provider as inactive means that it can no longer be used to authenticate to a Project or an application.
## Default Data Platform authentication providers
Data Platform offers 3 default authentication providers in every Project: [**Data Platform**](#data-platform), [**Project**](#project), and [**API key**](#api-key).
### Data Platform
This is the main authentication provider for the Data Platform. It lets you use your [Data Platform account](https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md) that you created when you first joined the platform.
:::info
It is hosted in the Cloud Management Platform, which serves all organizations and Projects that you are part of.
:::
### Project
This is a simple sign-in-by-email method provided by the [Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md) for all applications deployed on this Project. It provides an easy method to invite end-users to your applications without having them create an account on Data Platform.
:::info
It is hosted in the Project itself, which means all user information is stored within the Project itself and not shared with other Projects.
:::
### API key
API keys are usually used to let users connect using [API and secret keys](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md). Their main purpose is for advanced or custom configuration using [Data Platform APIs](https://docs.dataplatform.ovh.net/landing-page-developers-apis.md).
:::warning
This authentication method must be active for external users or applications to connect to the Project via API.
:::
## Add an authentication provider
Click on the _+New auth. provider_ button.
### List of authentication providers
Data Platform supports 7 external [single sign-on](https://en.wikipedia.org/wiki/Single_sign-on) providers:
| Provider | Type | Description |
| :----------- | :-------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Azure AD** | OAuth 2.0 | Active Directory (AD) is a directory service that Microsoft developed for the Windows domain networks. |
| **Okta** | OAuth 2.0 | Okta provides cloud software that helps companies manage and secure user authentication into applications. |
| **Auth0** | OAuth 2.0 | Auth0 is a cloud-based identity authentication platform for application developers. |
| **Google** | OAuth 2.0 | Google SSO is the authentication protocol of Google. |
| **Open AM** | OAuth 2.0 | OpenAM is an open-source access management, entitlements and federation server platform supported by the Open Identity Platform Community. |
| **Open ID** | OAuth 2.0 | OpenID is an open standard and decentralized authentication protocol promoted by the non-profit OpenID Foundation. |
| **SAML 2.0** | SAML 2.0 | Security Assertion Markup Language 2.0 (SAML 2.0) is a version of the SAML standard for exchanging authentication and authorization data between security domains. |
You can then configure your new authentication provider. The configuration depends on the provider.
- [↪ Learn how to plug in your Azure AD](https://docs.dataplatform.ovh.net/iam-auth-providers-azure-ad.md)
- [↪ Learn how to plug in your Okta](https://docs.dataplatform.ovh.net/iam-auth-providers-okta.md)
- [↪ Learn how to plug in your Auth0](https://docs.dataplatform.ovh.net/iam-auth-providers-auth0.md)
### Automatically map roles/groups from your authentication provider
Automatically map groups or roles on Data Platform using permissions that were already set up in your authentication provider.
[↪ Configure custom mapping of groups/roles](https://docs.dataplatform.ovh.net/iam-auth-providers-custom-mapping.md)
### Troubleshoot authentications from your external authentication provider
:::info
This feature is only available for [external authentication providers](https://docs.dataplatform.ovh.net/iam-auth-providers.md#list-of-authentication-providers), and for the [Data Platform authentication provider](#data-platform).
:::
It is possible to troubleshoot the external authentication provider configuration by logging the requests to the external server and the responses received from the server. By viewing what the identity provider sends back for each login request and how it is interpreted by Data Platform, this allows you to detect errors in a configuration such as the user information or the [mapping](https://docs.dataplatform.ovh.net/iam-auth-providers.md#automatically-map-rolesgroups-from-your-authentication-provider).
Login logs are available in the _Log history_ tab of an existing authentication provider.
Opening a specific login will display:
- the raw response received from the external authentication server
- the configuration used by Data Platform to map user information to data sent by the external server
- the result that is interpreted using the two aforementioned payloads: this is the final user information in the IAM as computed from the external authentication server.
## Authentication Policies
This section describes how to configure the authentication policies for account suspension due to multiple failed attempts. The settings are designed to enhance the security of user accounts by defining thresholds and actions to take when those thresholds are exceeded.
| **Setting** | **Description** | **Options/Default** |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------- |
| **Enable Account Suspension** | This toggle enables or disables the account suspension feature. | Enabled (On) |
| **Login Attempts** | Specifies the number of failed login attempts allowed before the account is suspended. | Infinite (∞) |
| **Password Renewal Attempts** | Specifies the number of failed password renewal attempts allowed before the account is suspended. | Infinite (∞) |
| **Interval Time for Password Attempts** | Defines the time window for considering the number of failed password attempts. | 1 Hour (default), adjustable |
| **Suspension Duration** | Defines the duration for which the account remains suspended after reaching the failed attempts threshold. | 5 Minutes (default), adjustable |
| **Notify Admin on Suspension** | This toggle enables or disables email notifications to the administrator when an account is suspended. | Disabled (Off) by default |
## Enable multi-factor authentication (MFA)
Multi-factor authentication (MFA) is a feature which forces users to go through a second method of authentication in addition to their usual method. The available authentication methods are:
- a code sent by email
- a code sent by SMS
- a code retrieved on a two-factor app (e.g. Google Authenticator, Microsoft Authenticator, etc..)
:::warning
MFA enforcement at project-level is only available for the [Project](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) authentication method. In other words, if the user is logged in through their [Data Platform account (Data Platform ID)](#data-platform) (or any other registered authentication provider in the project) they will bypass the MFA requirement in the project (as the MFA enforcement should be delegated to the third-party system in this case).
:::
:::info
It is possible for an admin of an organization to enforce MFA on the [Data Platform ID](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) authentication method for a whole organization, by turning on the option in the [organization settings](https://docs.dataplatform.ovh.net/manage-organizations-settings.md#settings). For single sign-on from other authentication providers, the MFA enforcement should be delegated to the third-party system.
:::
To enforce MFA in your Project for the [Project](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) authentication method, open the _Project_ authentication provider, find the panel "Multi factor authentication (MFA)" and turn it on.
Choose the methods you will allow your users to authenticate with.
Methods have the following requirements:
- **Email**: none
- **SMS**: a phone number must be registered on the [user's page](https://docs.dataplatform.ovh.net/iam-users.md) for this option to be enabled
- **External two-factor apps**: a 2FA app code **must be generated** by an administrator from the [user's IAM page](https://docs.dataplatform.ovh.net/iam-users.md) for this option to work. Users will NOT be able to self-generate a code from the app they are logging to.
After you save, any user logging in via the _Project_ authentication method will always need to authenticate through a second method.
:::warning
If the _Project_ authentication method is used for one or more [Data Platform apps](https://docs.dataplatform.ovh.net/iam-application-authentication.md), you must update the _CAM/IAM_ extension of all your apps or your users will no longer be able to login.
:::
### Migrate an app to IAM 1.4.0
#### 1 - Update the version of _client-authority-manager_ module
- Open `./config/forepaas.json` file (root level of app)
- Edit "client-authority-manager" : "XXX" to "client-authority-manager" : "1.4.0"
- **Reload** modules in the application's Overview, above **Versions**. If you work locally, use\
command \`npx git+[https://gitlab.forepaas.com/open-sources/fppm.git](https://gitlab.forepaas.com/open-sources/fppm.git) install -f\`\`
- Check version update by opening `./config/forepaas/client-authority-manager/forepaas.json` and looking at `version`
#### 2 - Add new translation to i18n configuration
- Edit `./config/i18n/en.json`
- Add these lines:
```json
"authentication.mfa.authenticator": "Use authenticator app",
"authentication.mfa.authenticator.code_label": "Enter the code from the two-factor app on your mobile device. If you’ve lost your device, please contact your admin to regenerate an authentication code",
"authentication.mfa.authenticator.send": "Activate",
"authentication.mfa.code": "XXXXXX",
"authentication.mfa.description": "Two-factor authentication is required to reach this page.",
"authentication.mfa.description2": "Please select one of the methods below to authenticate.",
"authentication.mfa.email": "Send me an email",
"authentication.mfa.email.code_label": "Enter the code you received by email",
"authentication.mfa.email.resend": "Resend email",
"authentication.mfa.email.send": "Send email",
"authentication.mfa.email.send.success": "An email containing an authentication code has been sent to you",
"authentication.mfa.no-phone": "You don’t have a registered phone number. Please ask your admin to add your phone number to your user info to be able to use this method.",
"authentication.mfa.send.authenticator": "An email containing a QR code has been sent to you. Please scan it with your external two-factor app to activate MFA for Data Platform.",
"authentication.mfa.sms": "Send me an SMS",
"authentication.mfa.sms.code_label": "Enter the code you received by SMS",
"authentication.mfa.sms.resend": "Resend SMS",
"authentication.mfa.sms.send": "Send SMS",
"authentication.mfa.sms.send.success": "An SMS containing an authentication code has been sent to you"
```
- Edit `./config/i18n/fr.json`
- Add these lines
```json
"authentication.mfa.authenticator": "Utiliser une app Authenticator",
"authentication.mfa.authenticator.code_label": "Saisissez le code de l'application à deux facteurs sur votre appareil mobile. Si vous avez perdu votre appareil, veuillez contacter votre administrateur pour régénérer un code d'authentification",
"authentication.mfa.authenticator.send": "Activer",
"authentication.mfa.code": "XXXXXX",
"authentication.mfa.description": "Une authentification à deux facteurs est requise pour accéder à cette page.",
"authentication.mfa.description2": "Veuillez sélectionner l'une des méthodes ci-dessous pour vous authentifier.",
"authentication.mfa.email": "M'envoyer un email",
"authentication.mfa.email.code_label": "Entrez le code que vous avez reçu par email",
"authentication.mfa.email.resend": "Réenvoyer email",
"authentication.mfa.email.send": "Envoyer email",
"authentication.mfa.email.send.success": "Un email contenant un code d’authentification vient de vous être envoyé.",
"authentication.mfa.no-phone": "Vous n'avez pas de numéro de téléphone enregistré. Veuillez demander à votre administrateur d'ajouter votre numéro de téléphone à vos informations utilisateur pour pouvoir utiliser cette méthode.",
"authentication.mfa.send.authenticator": "Un email contenant un code QR vous a été envoyé. Veuillez le scanner avec votre application externe pour activer l’authentification a multiple facteurs pour Data Platform.",
"authentication.mfa.sms": "M'envoyer un SMS",
"authentication.mfa.sms.code_label": "Entrez le code que vous avez reçu par SMS",
"authentication.mfa.sms.resend": "Renvoyer SMS",
"authentication.mfa.sms.send": "Envoyer SMS",
"authentication.mfa.sms.send.success": "Un SMS contenant un code d’authentification vient de vous être envoyé."
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-auth-providers-custom-mapping.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Set up custom mapping for authentication provider
## Objective
Custom mappings allow you to automatically assign existing [Data Platform groups](https://docs.dataplatform.ovh.net/iam-groups.md) or [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) (or auto-create them if they don't exist) to users signing in through that authentication provider.
:::info
In the **Log History** section, [JSONPath formatting](https://jsonpath.com/) can be used to select from the response object returned after a successful authentication, and configure the custom mappings below.
:::
Click on **Add** in the Custom Mapping panel to set up a new mapping.
## Normal mode
With the normal mode, you can assign **existing** [groups](https://docs.dataplatform.ovh.net/iam-groups.md) or [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) to users signing in from the authentication provider.
To do this, choose between _Role_ and _Group_, and select the role/group from the selection.
By default, it will assign the role/group to all new users imported from the authentication provider, but you can set a condition on the binding.
Conditions are **based on fields coming from the authentication providers response**.\
An example of condition is:
```json
{
"user.jobTitle": {
"eq": "Admin"
}
}
```
This condition will look the field _user.jobTitle_ in the authentication response for any new user signing in from the authentication provider, and check if the value is equal to _Admin_. If true, it will bind the role/group to that user, otherwise it won't.
Conditional operators are:
- `eq`: Equal to the value
- `ne`: Not equal to the value
- `in`: In the set of value
- `nin`: Not in the set of value
- `lt`, `lte`, `gt`, `gte`: Lower than, Lower or equal to, Greater than, Greater or equal to
- `regexp`: Will try to match the auth response field with the value (regex)
## JSON mode
With the JSON mode, you can dynamically [assign](#assign-existing-rolesgroups) or [create](#dynamically-find-or-create-rolesgroups) groups or roles for users signing in from the authentication provider. Switch to the JSON mode in the header of the window to activate it.
### Assign existing roles/groups
The binding of **existing** groups/roles with conditions is carried out using the _if_ keyword.
```json
{
"group": "_identifier_of_the_group",
"if": {
"organization.name": {
"eq": "Example Corp"
}
}
}
```
### Dynamically find or create roles/groups
You can dynamically define bound resources using the option `bind`. It takes in 5 possible parameters:
- `upsert`: false/true
- if _true_: It will autogenerate groups/roles if they don't already existing in the Identity Access Manager
- if _false_: It will only bind to existing groups/roles
- `input`: It's a JSON path accessor to get the list of "resources to bind"
- `split`: In some cases, you could have such a field in the answer: "group1,group2,group3". By inputting "," it will automatically split the list using the commas as separator
- `match`: [see below](#use-the-match-parameter)
- `replace`: [see below](#use-the-replace-parameter)
#### Use the _match_ parameter
You can take only some groups matching a **specific regex pattern**.
For example, if the authentication provider has the following groups: _PRIVATE\_ADMIN_, _IAM\_APP\_GROUP1_, _IAM\_APP\_GROUP2_, then the following configuration will only create the 2 groups starting by "IAM\_APP".
```json
{
"input": "$.memberships[*].displayName",
"match": 'IAM_APP_.*'
}
```
With that configuration I will only get the 2 group starting by IAM\_APP
#### Use the _replace_ parameter
The `replace` parameter can only be used if there is a "match" pattern set. It allows you to **rename the groups created**, using [regex capturing groups](https://javascript.info/regexp-groups):
```json
{
"input": "$.memberships[*].displayName",
"match": 'IAM_APP_(.*)',
"replace": '$1'
}
```
In the previous example, it will create groups called "GROUP1" and "GROUP2" in the Identity Access Manager instead of the full name.
#### Three examples
For instance, you can bind organizations' name to IAM groups:
```json
{
"bind": {
"upsert": false, // if set to true, it will create the field automatically
"input": "$.organizations[*].name", // JSON path accessor to input elements
"split": null // Optional, if set, it will try to split every value with the value in delimiter
"match": ".*", // Optional, it will match the value found in input with that regexp
"replace": "$0" // Optional, it will replace the input with that pattern before setting it as name for group/role.
}
}
```
This configuration will automatically create new groups from user's memberships:
```json
{
"bind": {
"upsert": true,
"input": "$.memberships[*].displayName"
}
}
```
This example configuration:
- looks for `$.user.isMemberOf` field
- splits values using comma (`,`)
- for each value:
- checks if it matches a regular expression starting with `APP_MY_DASHBOARD_` and capturing the end
- replaces by `APP_` and the captured value
- as `upsert` is false, checks whether that group does exist in IAM and bind users to that group
```json
{
"bind": {
"upsert": false,
"split": ",",
"input": "$.user.isMemberOf",
"match": "APP_MY_DASHBOARD_([\\w-]*)",
"replace": "APP_$1"
}
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-auth-providers-azure-ad.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Set up SSO authentication with Azure AD
## Objective
This guide explains how to use Azure Active Directory to log in to your Data Platform applications.
Open the IAM and click on **Auth. Provider**.
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/iam-auth-providers.md#add-an-authentication-provider) _Azure AD_ authentication provider. Else, just edit it.
The configuration is done in two parts. First you need to [configure the rights on Microsoft Azure](#configuration-on-azure) and then [configure the authentication provider in the Data Platform Identity Access Manager](#configuration-on-the-data-platform-identity-access-manager).
## Configuration on Azure
You need to register Data Platform on the Azure console. Go there: [https://portal.azure.com](https://portal.azure.com).
### Step 1. Retrieve the Tenant ID
:::info
Note that the interface of Azure Active Directory may change over time, and screenshots could be outdated. Please refer to the [Azure AD documentation portal](https://docs.microsoft.com/en-us/azure/active-directory/) if you are unable to find some of the features listed below.
:::
The _Tenant ID_ is the ID of your Azure company environment. To find it, select **Azure Active Directory** in the list of Azure Services. Then, in the sidebar click on "Properties".
:::info
Note that this ID will be required later on during the IAM configuration.
:::
### Step 2. Register the Identity Access Manager
You then need to register an application inside of Microsoft Azure AD. This works with any app you would like to connect to Azure but in this case, you are going to register the Data Platform IAM directly.
In the sidebar, select **App Registrations**. Then click on _New registration_.
#### _Name_
The name is purely descriptive: it will be useful to find your configuration within your Azure AD.
#### _Supported account types_
This parameter specifies which Azure AD directories you'll be able to use to authenticate on the Data Platform IAM. It is recommended to select _Accounts in this organizational directory only_ in order that only your AD can log in to your Data Platform IAM.
#### _Redirect URL_
Select _Web_.
In the URL field, copy and paste the **Reply URL** field in the new authentication provider you added on Data Platform (it is the same for the entire Data Platform IAM).
Once this is done, click on **Register**.
### Step 3. Retrieve the Client ID
The _Client ID_ is the user ID of your newly added application (i.e. the Data Platform IAM). You can find it on its Overview page: look for _Application (client) ID_.
:::info
Note that this Client ID will be required later on during the IAM configuration.
:::
### Step 4. Retrieve the Client Secret
On the page of your IAM application, click on **Certificates & secrets** in the sidebar. Click on _New client secret_.
Fill in the description and expiration date of your choice and click on **Add**.
Once the client secret is created, copy and save its **Value**.
:::info
Please note that your Client Secret will be required later on during your IAM configuration.
:::
### Step 5. Enable basic permissions from Azure AD
On the page of your IAM application, click on **API permissions** in the sidebar. Click on _New permission_ and select _Microsoft Graph_.
Select _Delegated permissions_.
Two permissions must be added in order for the entire feature-set of Azure AD authentication provider on Data Platform to work.
- **profile**: this permission is needed if you need to get user information from Azure AD using the [UserInfo endpoint](#fill-in-azure-configuration).
- **Group.Read.All**: this permission is needed if you need to get the list of groups from your Azure AD for a [custom mapping](#custom-mappings)
Find them in the list and add them both.
Finally, click on **Grant admin consent for your-app** and confirm.
When all permissions have been correctly added, your screen should look like this.
### Summary
If you have properly followed all the previous steps, you should have this list of information.
```
Tenant ID (Step 1): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Client ID (Step 3): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Client Secret (Step 4): XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
```
:::info
If you want to further customize the configuration of your new application from the Azure portal, we recommend you check out [Azure AD's documentation](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/what-is-application-management).
:::
## Configuration on the Data Platform Identity Access Manager
You now have to configure your new authentication provider in your Data Platform Identity Access Manager.
### Fill in the basic information
- **Name / Description**: This is the internal name and description for your new authentication method
- **Directory**: This is a unified tag for all accounts that will be created from this directory. For example, you can input _Azure_
- **Tags**: This offers more tagging options. They are completely optional
- **Reply URL**: This is the Data Platform URL where your authentication provider redirects to
- **Active**: This specifies whether this authentication provider can be used to login or not
- **Debug Mode**: When this is turned on, this enables you to see the authentication logs in the [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md).
### Fill in Azure configuration
Now locate the **Azure Configuration** panel.
Fill in the information collected on Azure in the previous section, as such:
- **Tenant ID**: _Tenant ID_ (retrieved in [step 1](#step-1-retrieve-the-tenant-id) of the configuration on Azure)
- **Client ID**: _Client ID_ (retrieved in [step 3](#step-3-retrieve-the-client-id) of the configuration on Azure)
- **Client Secret**: _Client Secret Value_ (retrieved in [step 4](#step-4-retrieve-the-client-secret) of the configuration on Azure)
- **Scope**: This represents how much information Data Platform will be able to access from your Azure AD. This field is optional: if left blank it will use the value `openid email profile`
- **Enable user info endpoint on AzureAD**: This endpoint will allow the IAM to do an extra `GET https://graph.microsoft.com/v1.0/users/me` call on Azure AD Graph API, the result of which will add a sub-object "user" in addition to the standard OpenID field in the response. This allows you to get additional information on users, like the job title.
### Bindings
The **Bindings** panel allows you to customize the mapping of the fields between Azure AD and Data Platform.
Email, login, first and last name are mapped by default.
Bindings can be overriden if you need to save additional information from Azure AD into your Data Platform Identity Access Manager, like the job title.
The easiest way to configure bindings is to retrieve a response from a login of an user by following these steps:
_1_ - Do a sample login to any Data Platform application using your newly configured Azure AD authentication provider.\
_2_ - Open the **Log History** page of the IAM authentication provider (here Azure AD).
_3_ - Open the latest log: the original response from Azure AD will look like this:
```json
{
"aud": "3a0f2280-36e3-457b-a72a-d74fca8a0eae",
"iss": "https://sts.windows.net/058c4703-07ad-4c14-87e5-e08241c8cb8f/",
"iat": 1628084238,
"nbf": 1628084238,
"exp": 1628088138,
"amr": [
"pwd"
],
"email": "test@example.com",
"family_name": "Smith",
"given_name": "John",
"idp": "https://sts.windows.net/82b3359c-9391-4d7c-a0f2-816751a97641/",
"ipaddr": "62.23.151.245",
"name": "John Smith",
"oid": "db6723ba-d536-4c72-bfb4-c089d88fe7d8",
"rh": "0.ARAAA0eMBa0HFEyH5eCCQcjLj4AiDzrjNntFpyrXT8qKDq4QAOw.",
"sub": "3H2vG1r81UmzHRIp74IHcYjONOSMZf4gQ12M_ELMorY",
"tid": "058c4703-07ad-4c14-87e5-e08241c8cb8f",
"unique_name": "test@example.com",
"uti": "GO1KbxWFg0ChEVKPUDqhAA",
"ver": "1.0",
"memberships": [
{
"@odata.type": "#microsoft.graph.group",
"id": "7d232d98-c942-4405-9380-a644b6db1eb7",
...
"description": "TestGroup",
"displayName": "My Group",
},...
]
}
```
_4_ - On top of the default email, login, first and last name, any additional property from Azure AD can be saved as a custom `configuration.XXX` field on Data Platform by using the "JSON" mode.
For example, you can add a _fullname_ property on Data Platform from the `name` field from Azure AD:
```json
{
"login": "email",
"email": "email",
"pronoun": "",
"firstname": "given_name",
"lastname": "family_name",
"configuration.fullname": "name"
}
```
:::info
Test the result of your bindings by checking the logs of further test connections in Log History: the _Computed Result_ section of each login will show what properties have been saved in Data Platform.
:::
### Custom mappings
The **Custom mappings** panel allows you to automatically assign all users logged in through this authentication provider to a role or a group.
[Learn how to set up custom mappings.](https://docs.dataplatform.ovh.net/iam-auth-providers-custom-mapping.md)
### Set live
You are all set configuring your Azure AD! If you haven't done it already, click on **Create** - else make sure to **Save**.
In the Authentication Providers list, you now see your new method. Make sure you set it as _active_.
Make sure this new authentication method [is visible](https://docs.dataplatform.ovh.net/iam-application-authentication.md#display-an-authentication-provider-on-an-application-login-page) for the applications of your choice. You should now see an Azure AD option on your application's login page. By clicking on it, you will be redirected to the Azure login screen, or directly into the app if you are already logged into your account.
[Make Azure AD visible on your apps' login page](https://docs.dataplatform.ovh.net/iam-application-authentication.md#display-an-authentication-provider-on-an-application-login-page)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-auth-providers-okta.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Set up SSO authentication with Okta
## Objective
This guide explains how to use [Okta](https://www.okta.com/) to log in to your Data Platform applications.
Open the IAM and click on **Auth. Provider**.
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/iam-auth-providers.md#add-an-authentication-provider) _Okta_ authentication provider on Data Platform. Else, just edit it.
The configuration is done in two parts. First you need to [configure the rights on Okta](#configuration-on-okta) and then [configure the authentication provider in the Data Platform Identity Access Manager](#configuration-on-the-data-platform-identity-access-manager).
## Configuration on Okta
You need to register Data Platform as an application on your Okta environment. Open your Okta environment.
### Step 1. Register the Identity Access Manager on Okta
:::info
Note that the interface of Okta may change over time, and screenshots could be outdated. Please refer to the [Okta documentation portal](https://help.okta.com/en/prod/Content/index.htm) if you are unable to find some of the features listed below.
:::
You need to register an application inside of Okta. This works with any app you would like to connect to Okta but in this case, you are going to register the Data Platform IAM directly.
In the sidebar, select **Applications**. Then click on _Create App Integration_.
Select _OIDC (OpenID Connect)_ as the method and _Web Application_ as the application type.
Press **Next**.
#### _Callback URL_
Give the name of your choice to this new application and make sure that the Grant Type is _Authorization Code_.
Scroll down to the **Sign-in Redirect URIs** section.
In the URL field, copy and paste the **Reply URL** field in the new authentication provider you added on Data Platform (it is the same for the entire Data Platform IAM).
Save the new application integration.
### Step 2. Retrieve the Domain
The Domain is the equivalent of your tenant id on Okta. You can find it in the General Settings of the application: look for the **Okta domain** section.
:::info
Note that this Domain will be required later on during the IAM configuration.
:::
### Step 3. Retrieve the Client ID
The Client ID is specific to your application. You can find it in the Client Credentials of the application: look for the **Client ID** section.
:::info
Note that this Client ID will be required later on during the IAM configuration.
:::
### Step 4. Retrieve the Client Secret
The Client Secret is specific to your application. You can find it in the Client Credentials of the application: look for the **Client secret** section.
:::info
Note that this Client Secret will be required later on during the IAM configuration.
:::
### Summary
If you have properly followed all the previous steps, you should have this list of information.
```
Okta domain (Step 2): xxxxxxxxxx.okta.com
Client ID (Step 3): XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Client secret (Step 4): XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
```
:::info
If you want to further customize the configuration of your new application from the Okta portal, we recommend you check out [Okta's documentation](https://developer.okta.com/).
:::
## Configuration on the Data Platform Identity Access Manager
You now have to configure your new authentication provider in your Data Platform Identity Access Manager.
### Fill in the basic information
- **Name / Description**: This is the internal name and description for your new authentication method
- **Directory**: This is a unified tag for all accounts that will be created from this directory. For example, you can input _Okta_
- **Tags**: This offers more tagging options. They are completely optional
- **Reply URL**: This is the Data Platform URL where your authentication provider redirects to
- **Active**: This specifies whether this authentication provider can be used to login or not
### Fill in Okta configuration
Now locate the **Okta Configuration** panel.
Fill in the information collected on Okta in the previous section, as such:
- **Domain**: _Okta domain_ (retrieved in [step 2](#step-2-retrieve-the-domain) of the configuration on Okta)
- **Client ID**: _Client ID_ (retrieved in [step 3](#step-3-retrieve-the-client-id) of the configuration on Okta)
- **Client Secret**: _Client secret_ (retrieved in [step 4](#step-4-retrieve-the-client-secret) of the configuration on Okta)
- **Scope**: This represents how much information Data Platform will be able to access from your Okta. This field is optional: the default value is `openid email profile`
### Fill in the bindings rules
The **Bindings** panel allows you to customize the mapping of the fields between Okta and Data Platform.
Email, login, first and last name are mapped by default.
Bindings can be overriden if you need to save additional information from Okta into your Data Platform Identity Access Manager, like the job title.
The easiest way to configure bindings is to retrieve a response from a login of an user by following these steps:
_1_ - Do a sample login to any Data Platform application using your newly configured Okta authentication provider.\
_2_ - Open the **Log History** page of the IAM authentication provider (here Okta).
_3_ - Open the latest log: the original response from Okta will look like this:
```json
{
"sub": "00u4au2tpeFfNw3uR696",
"name": "John Smith",
"email": "jsmith@example.com",
"ver": 1,
"iss": "https://example.okta.com",
"aud": "4j63bjk34b6jk3b",
"iat": 1634310728,
"exp": 1634314328,
"jti": "ID.TRhY73q_LejPMKVxrt936nLTLaayJJgPMni78ww0ouQ",
"amr": [
"mfa",
"pwd",
"otp"
],
"idp": "00o4au2o4QD29JO5Q696",
"preferred_username": "jsmith@example.com",
"auth_time": 1634310281,
"at_hash": "OExX1fo9d8vq-z2-vZtkIg",
"user": {
"sub": "00u4au2tpeEfNw3uR696",
"name": "John Smith",
"locale": "en-US",
"email": "jsmith@example.com",
"preferred_username": "jsmith@example.com",
"given_name": "John",
"family_name": "Smith",
"zoneinfo": "America/Los_Angeles",
"updated_at": 1634310506,
"email_verified": true
}
}
```
_4_ - On top of the default email, login, first and last name, any additional property from Okta can be saved as a custom `configuration.XXX` field on Data Platform by using the "JSON" mode.
For example, you can add a _fullname_ property on Data Platform from the `name` field from Okta:
```json
{
"login": "email",
"email": "email",
"pronoun": "",
"firstname": "user.given_name",
"lastname": "user.family_name",
"configuration.fullname": "name"
}
```
:::info
Test the result of your bindings by checking the logs of further test connections in Log History: the _Computed Result_ section of each login will show what properties have been saved in Data Platform.
:::
### Custom mappings
The **Custom mappings** panel allows you to automatically assign all users logged in through this authentication provider to a role or a group.
[Learn how to set up custom mappings.](https://docs.dataplatform.ovh.net/iam-auth-providers-custom-mapping.md)
### Set live
You are all set configuring your Okta connector! If you haven't done it already, click on **Create** - else make sure to **Save**.
In the Authentication Providers list, you now see your new method. Make sure you set it as _active_.
Make sure this new authentication method [is visible](https://docs.dataplatform.ovh.net/iam-application-authentication.md#display-an-authentication-provider-on-an-application-login-page) for the applications of your choice. You should now see an Okta option on your application's login page. By clicking on it, you will be redirected to the Okta login screen, or directly into the app if you are already logged into your account.
[Make Okta visible on your apps' login page](https://docs.dataplatform.ovh.net/iam-application-authentication.md#display-an-authentication-provider-on-an-application-login-page)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-auth-providers-auth0.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Set up SSO authentication with Auth0
## Objective
This guide explains how to use [Auth0](https://auth0.com/) to log in to your Data Platform applications.
Open the IAM and click on **Auth. Provider**.
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/iam-auth-providers.md#add-an-authentication-provider) _Auth0_ authentication provider on Data Platform. Else, just edit it.
The configuration is done in two parts. First you need to [configure the rights on Auth0](#configuration-on-auth0) and then [configure the authentication provider in the Data Platform Identity Access Manager](#configuration-on-the-platform-identity-access-manager).
## Configuration on Auth0
You need to register Data Platform as an application on your Auth0 environment. Go there: [https://manage.auth0.com/dashboard](https://manage.auth0.com/dashboard).
### Step 1. Register the Identity Access Manager on Auth0
:::info
Note that the interface of Auth0 may change over time, and screenshots could be outdated. Please refer to the [Auth0 documentation portal](https://auth0.com/docs) if you are unable to find some of the features listed below.
:::
You need to register an application inside of Auth0. This works with any app you would like to connect to Auth0 but in this case, you are going to register the Data Platform IAM directly.
In the sidebar, select **Applications**. Then click on _Create application_.
Select _Regular Web Application_.
Open the **Settings** of the newly created application.
#### _Callback URL_
Scroll down to the **Allowed Callback URLs** section in the Settings page of the application.
In the URL field, copy and paste the **Reply URL** field in the new authentication provider you added on Data Platform (it is the same for the entire Data Platform IAM).
### Step 2. Retrieve the Domain
The Domain is the equivalent of your tenant id on Auth0. You can find it in the Settings of the application: look for the **Domain** section.
:::info
Note that this Domain will be required later on during the IAM configuration.
:::
### Step 3. Retrieve the Client ID
The Client ID is specific to your application. You can find it in the Settings of the application: look for the **Client ID** section.
:::info
Note that this Client ID will be required later on during the IAM configuration.
:::
### Step 4. Retrieve the Client Secret
The Client Secret is specific to your application. You can find it in the Settings of the application: look for the **Client Secret** section.
:::info
Note that this Client Secret will be required later on during the IAM configuration.
:::
### Summary
If you have properly followed all the previous steps, you should have this list of information.
```
Domain (Step 2): xxxxxxxxxx.auth0.com
Client ID (Step 3): XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Client Secret (Step 4): XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
```
:::info
If you want to further customize the configuration of your new application from the Auth0 portal, we recommend you check out [Auth0's documentation](https://auth0.com/docs/get-started/create-apps/regular-web-apps).
:::
## Configuration on the Platform Identity Access Manager
You now have to configure your new authentication provider in your Data Platform Identity Access Manager.
### Fill in the basic information
- **Name / Description**: This is the internal name and description for your new authentication method
- **Directory**: This is a unified tag for all accounts that will be created from this directory. For example, you can input _Auth0_
- **Tags**: This offers more tagging options. They are completely optional
- **Reply URL**: This is the Data Platform URL where your authentication provider redirects to
- **Active**: This specifies whether this authentication provider can be used to login or not
### Fill in Auth0 configuration
Now locate the **Auth0 Configuration** panel.
Fill in the information collected on Auth0 in the previous section, as such:
- **Domain**: _Domain_ (retrieved in [step 2](#step-2-retrieve-the-domain) of the configuration on Auth0)
- **Client ID**: _Client ID_ (retrieved in [step 3](#step-3-retrieve-the-client-id) of the configuration on Auth0)
- **Client Secret**: _Client Secret_ (retrieved in [step 4](#step-4-retrieve-the-client-secret) of the configuration on Auth0)
- **Scope**: This represents how much information Data Platform will be able to access from your Auth0. This field is optional: the default value is `openid email profile`
### Fill in the bindings rules
The **Bindings** panel allows you to customize the mapping of the fields between Auth0 and Data Platform.
Email, login, first and last name are mapped by default.
Bindings can be overriden if you need to save additional information from Auth0 into your Data Platform Identity Access Manager, like the job title.
The easiest way to configure bindings is to retrieve a response from a login of an user by following these steps:
_1_ - Do a sample login to any Data Platform application using your newly configured Auth0 authentication provider.\
_2_ - Open the **Log History** page of the IAM authentication provider (here Auth0).
_3_ - Open the latest log: the original response from Auth0 will look like this:
```json
{
"given_name": "John",
"family_name": "Smith",
"nickname": "jsmith",
"name": "John jsmith",
"picture": "https://lh3.googleusercontent.com/a/AATXAJwHYHTaalrHB9puNg1bjfCuKIuZPqVFMUcma5e5=s96-c",
"locale": "fr",
"updated_at": "2021-10-01T15:14:35.524Z",
"email": "jsmith@example.com",
"email_verified": true,
"iss": "https://dev-m4zopi2a.us.auth0.com/",
"sub": "google-oauth2|112554878744",
"aud": "****",
"iat": 1633101540,
"exp": 1633137540,
"user": {
"sub": "google-oauth2|112554878744",
"given_name": "John",
"family_name": "Smith",
"nickname": "jsmith",
"name": "John Smith",
"picture": "https://lh3.googleusercontent.com/a/AATXAJwHYHTaalrHB9puNg1bjfCuKIuZPqVFMUcma5e5=s96-c",
"locale": "fr",
"updated_at": "2021-10-01T15:14:35.524Z",
"email": "jsmith@example.com",
"email_verified": true
}
}
```
_4_ - On top of the default email, login, first and last name, any additional property from Auth0 can be saved as a custom `configuration.XXX` field on Data Platform by using the "JSON" mode.
For example, you can add a _fullname_ property on Data Platform from the `name` field from Auth0:
```json
{
"login": "email",
"email": "email",
"pronoun": "",
"firstname": "given_name",
"lastname": "family_name",
"configuration.fullname": "name"
}
```
:::info
Test the result of your bindings by checking the logs of further test connections in Log History: the _Computed Result_ section of each login will show what properties have been saved in Data Platform.
:::
### Custom mappings
The **Custom mappings** panel allows you to automatically assign all users logged in through this authentication provider to a role or a group.
[Learn how to set up custom mappings.](https://docs.dataplatform.ovh.net/iam-auth-providers-custom-mapping.md)
### Set live
You are all set configuring your Auth0 connector! If you haven't done it already, click on **Create** - else make sure to **Save**.
In the Authentication Providers list, you now see your new method. Make sure you set it as _active_.
Make sure this new authentication method [is visible](https://docs.dataplatform.ovh.net/iam-application-authentication.md#display-an-authentication-provider-on-an-application-login-page) for the applications of your choice. You should now see an Auth0 option on your application's login page. By clicking on it, you will be redirected to the Auth0 login screen, or directly into the app if you are already logged into your account.
[Make Auth0 visible on your apps' login page](https://docs.dataplatform.ovh.net/iam-application-authentication.md#display-an-authentication-provider-on-an-application-login-page)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-iam-users.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Users, roles and groups
## How authorization works
This is the authorization half of [Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md). Users, service accounts and groups are the principals; roles, with the conditions attached to them, are what gets granted.
A **role** is a set of permissions on resources or data. Roles are the only thing that grants access, and they are granted to principals, never attached to resources.
**Users** are the principals with a person behind them. They come in two kinds that share one mechanism: team members who need access to project resources, and end users who reach a deployed dashboard or app with a specific data access level.
**Service accounts** are principals without a person: servers, scripts, API consumers, and every [Data Processing Engine job](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md). They exist only in Project IAM, not at organization level.
**Groups** collect users and service accounts so a set of roles is granted once instead of per principal. A group usually maps to a team.
Roles can carry conditions, evaluated in CEL against the request, which is how access is narrowed to one bucket, one dataset or one attribute rather than a whole service.
[### Roles and conditions→
Permission sets, and the CEL conditions that narrow them.
](/iam-roles-conditions)[### Users→
Team members and application end users.
](/iam-users)[### Service accounts→
Non-human principals, including every DPE job.
](/iam-service-accounts)[### Groups→
Grant a set of roles once, to a whole team.
](/iam-groups)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-users.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage Project users in the Identity Access Manager
## Objective
When a person is registered in a Project either as a **team member** (i.e. connected using their Data Platform account and part of your organization) or as an **application end-user** (i.e. connected via the embedded authentication service of the Project), they will be found in the _Users_ tab of the Identity Access Manager.
On Data Platform, each user is identified by their **email address**, which must be unique to them.
## Organization IAM users vs. Project IAM users
The users in the Organization IAM correspond to all your team members who have access to the platform and who can potentially access the organization's resources. These users are managed in the same way as the Project IAM users, with the sole difference that there is no distinction between team member users and end-users. All users in the Organization IAM are team members with access to Data Platform.
On the other hands, users in the Project IAM can be either team members with access to the services/resources within a project ([Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md), [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md), etc) or end-users who only have access to the applications and dashboards. End-users are not a part of the organization, hence they do not exist in the Organization IAM and do not have access to the platform itself.
As a rule of thumb:
- If you want to add a team member to work with you on Data Platform: **add them in the Organization IAM**. Then make sure to grant permissions either at organization-level or project-level through roles and groups.
- If you want to grant access to an app or a dashboard to a end-user, **add them in the Project IAM**.
### Manage Organization IAM users
All team members that need to work on Data Platform **MUST be added** in the Organization's IAM.
To add someone in the Organization's IAM, head to the _Users_ tab of the Organization IAM, and click on **Invite users**.
:::info
Make sure to grant permissions to the new user by binding them a role or adding them to a group.
:::
- [↪ Add users to a group](https://docs.dataplatform.ovh.net/iam-groups.md#assign-users-or-service-accounts-to-a-group)
- [↪ Assign a role to a user](https://docs.dataplatform.ovh.net/iam-organization.md#resources-managed-by-the-organization-iam)
### Manage Project IAM users
The _Users_ tab of the Project IAM lists all users that can get access to your Project. These are either:
- [Team members](#team-members) from your organization
- [End-users](#application-end-users) that can open your applications
#### Team members
Your team members are the people with whom you work on data Projects in the Project. They must [have a Data Platform account](https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md) and [be part of your organization](https://docs.dataplatform.ovh.net/manage-organizations.md) to open and edit the Project (i.e. they must exist in the users list of the Organization IAM).
To add a new team member in your Project:
1. Click on **+ New User**
2. Type their email
3. Choose the _Team Member_ option
Click on _Next_ and follow the instructions to add the new team member. This will automatically [add the user in the Organization IAM](#manage-organization-iam-users) first.
:::info
Members that are _Admin_ in an organization will also have full _Admin_ rights in the Identity Access Managers of all Projects in this organization. Members that are added as _User_ in an organization will need to have their permissions added manually, either by placing them in a [group](https://docs.dataplatform.ovh.net/iam-groups.md) or by assigning them a [role](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) directly. Furthermore, if a user is added as _Project Admin_ in the Organization IAM, they will be added as _Admin_ in the _Project IAM_.
:::
- [↪ Add users to a group](https://docs.dataplatform.ovh.net/iam-groups.md#assign-users-or-service-accounts-to-a-group)
- [↪ Assign a role to a user](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#bind-a-role-to-a-user-service-account-or-group)
#### Application end-users
The end-users that you are building the application for don't need a Data Platform account to connect. Your Project comes with an [internal authentication service](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) that uses an email and password.
To add an end-user in your Project, click on **New User**.
You must enter an email for the end-user, as it will be their default authentication method.
Once the end-user is created, they receive an email with a single-use link to set their password. No password is generated for you to pass on. To reset a password later, use the reset option on the user: after you confirm, the end-user receives a single-use reset link by email.
:::info
All information entered in the Identity Access Manager, including phone number and email address, is securely stored in the Project and cannot be accessed by Data Platform. It is for your own internal usage and will never be used for marketing purposes.
:::
:::info
By default, new end-users don't have any access right. You will need to either place them in [groups](https://docs.dataplatform.ovh.net/iam-groups.md) or assign them [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) directly.
:::
- [↪ Add users to a group](https://docs.dataplatform.ovh.net/iam-groups.md#assign-users-or-service-accounts-to-a-group)
- [↪ Assign a role to a user](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#bind-a-role-to-a-user-service-account-or-group)
## Generate an API & secret key for a user
Whenever you need to trigger specific jobs (such as launching workflows) from outside of Data Platform, you will need to generate an API and a secret key to authenticate.
[↪ Generate API & secret keys for users](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-service-accounts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create service accounts for API and automated access
## Objective
A service account is a special kind of account used by API consumers, remote server access, scripts, or other automated entities, rather than by a physical person.
On Data Platform, service accounts are generally used for two reasons:
- **Externally**, to authenticate a remote entity that needs to access the Project through [API and secret keys](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md)
- **Internally**, to authenticate Project jobs. For example, a service account can be attached to a [Data Processing Engine workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md), so that when executed, the job can authenticate as the service account. The service account's [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) control which resources the job can access.
They will be found in the _Service Accounts_ tab of the Identity Access Manager. On Data Platform, each service account is identified by their **technical name** which must be unique to them.
:::info
Note that Service Accounts **exist ONLY in the Project IAM**.
:::
## Manage your Project service accounts
A **service account can be granted Project IAM** [**roles**](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) that let any associated entity (whether external entities or internal jobs) access resources. The service account is used as the identity of the associated entity, and the service account's roles control which resources the entity can access.
:::info
This makes service accounts fit to restrict the access level of a CRON-triggered Data Processing Engine job or a remote API that needs to access data on Data Platform, for example.
:::
There are two types of service accounts:
- **Default Data Platform-originated service accounts**: they are generated at the [creation of the Project](https://docs.dataplatform.ovh.net/manage-projects-create.md) so that basic services can run immediately, like the _Default Project Service Account_
- **User-managed service accounts**: they are created by Data Platform users
### Default service account
The _Default Project Service Account_ is a Data Platform-originated service account that is generated automatically at the creation of the Project. It is used by default by all jobs in the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) for authentication and access control. This service account cannot be deleted.
Any [user](https://docs.dataplatform.ovh.net/iam-users.md) from a Project can always impersonate this service account in the Data Processing Engine.
:::warning
It is impossible to restrict the impersonation of this service account by users that are authenticated in the Project's IAM. If you want to avoid a possible privilege escalation, you need to **[lower the role permissions](https://docs.dataplatform.ovh.net/iam-service-accounts.md#use-a-service-account-internally-for-jobs) given to this default service account**.
:::
By default, at the creation of the Project, the _Default Project Service Account_ has **Admin access** in the Project.
### Create a new service account
To add a service account in your Project, click on **New service account**.
Enter a name and an email address for the service account.
:::warning
The email address will be used to send over the [API and secret keys](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) necessary for the authentication of the remote entity.
:::
:::info
By default, new service accounts don't have any access right. You will need to either place them in [groups](https://docs.dataplatform.ovh.net/iam-groups.md) or assign them [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) directly.
:::
[Add service accounts to a group](https://docs.dataplatform.ovh.net/iam-groups.md#assign-users-or-service-accounts-to-a-group)
[Assign a role to a service account](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#bind-a-role-to-a-user-service-account-or-group)
## Use a service account internally for jobs
All jobs in [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) use service accounts for authentication and access control of the workload. As such, any user able to execute this job is essentially able to impersonate the service account (and get its access) through the executed workload.
:::info
The service account of a job currently cannot be changed. This feature is coming soon!
:::
By default, jobs are configured to use the [Default Project Service Account](https://docs.dataplatform.ovh.net/iam-service-accounts.md#default-service-account). It is impossible to restrict impersonation of this service account by users, so to avoid a possible privilege escalation (in the case some of your users should not be able to access everything) you should make sure to remove unnecessary permissions on this service account.
To do so, open the Default Project Service Account page, remove all groups and roles bound to this service account, add the basic roles/groups needed for the service account, and save. The base permissions for a Data Processing Engine job to be able to run without errors are the following:
- `Data Manager | * | Read`
- `Data Processing Engine | * | Read`
- `Data Processing Engine | * | Write` is optional and needed only if the update of metadata (such as number of rows in table) is wanted at the end of the job execution
- `Data Store | Bucket | Read`, with a [condition](https://docs.dataplatform.ovh.net/tutorials-bucket-role-conditions.md) on bucket _dpe_
## Use a service account externally
Whenever you need to trigger specific jobs (such as launching workflows) from outside of Data Platform, you will need to generate an API and a secret key to authenticate.
[Generate API & secret keys for service accounts](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-roles-conditions.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Roles and conditions
## Objective
Data Platform uses a [role-based access control](https://en.wikipedia.org/wiki/Role-based_access_control) approach to manage rights in Projects. **Roles** are a set of permissions that can be attributed either to a user, a service account (only in Project IAM) or a group, in order to grant them access to resources or data in your Project/Organization. **Conditions** allow to be more fine-grained and grant access only if specified conditions are met - such as if the request is made during the work hours, or if the requested resource has a certain tag.
Each user or service account has the total sum of all the permissions given to them 1) individually through roles, and 2) given to the groups they belong to.
:::info
Note that Roles **exist both in the Project and Organization IAMs**. The screenshots in this article have been taken in the Project IAM, but the steps are the same in the Organization IAM.
:::
## Roles
There are two types of roles in IAM:
- Pre-made roles: which provide access for popular combinations of services and resources in a Project or Organization. These are managed by Data Platform.
- Custom roles: which provide fine-grained access based on a user-specified list of permissions.
### Pre-made roles
Data Platform comes with many pre-made IAM roles designed to help you grant the typical permissions for any Project or Organisation easily.
For instance, here is the list of pre-made roles in the Project IAM, starting with the most popular ones:
- _Admin_: gives full admin access to the entire Project
- _Application User_: gives access to query the [custom APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) of the Project, typically from an [application](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md). This is typically the role that you would grant to your end-users that need to access your apps, controlling data access using [a role condition](#add-a-role-condition-on-specific-data-values)
- _Analytics Consumer_: this is typically given to users who will run analytics using an [external consumer](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md). The role gives read access to data sources and tables, and the ability to create and run queries on them.
- _Dashboard Editor_: gives permissions to create queries and dashboards in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
- _Dashboard Viewer_: gives permissions to access the read-only version of dashboards in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
- _Read-only_: gives read-access to all resources on the Project
- _AM Admin_: gives full admin access to all resources in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
- _API & App Editor_: gives full admin access to all [custom APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) and [applications](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md)
- _Control Center Editor_: gives full admin access to all resources in the [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md)
- _LM & AM Editor_: gives full admin access to all resources in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) (except Buckets) and the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
- _DPE Editor_: gives full admin access to all resources in the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md)
- _DataStore Editor_: gives full admin access to all resources in the [Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md)
- _Identity Access Manager Editor_: gives full admin access to all resources in the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md)
:::info
Note that the list of default roles is different for the Organisation IAM since the permissions and typical use cases are different.
:::
### Create a role
#### From the Identity Access Manager interface
In the Identity Access Manager, open the **Roles** tab. Click on **New Role** at the top-right of the screen.
Roles are defined by the atomic _permissions_ they contain, which are rights on specific resources.\
After you fill out the picture, name, description and tags of your role, click on **Add Permission** to add a new permission to the role.
Permissions are defined by specifying the following elements:
- the **service** and the **resource**, i.e the Project resource given access to by the permission (e.g in Data Processing Engine: actions, workflows, etc.)
- the **action**, i.e. which access is given to this service and resource (e.g. create, read, delete, etc.)
:::info
The asterisk (\*\*\*\*\*) value means all services/resources/actions are selected
:::
:::info
[Conditions](#conditions) on a property of the resource or user agent can be set during the binding of the role with a user or a group.
:::
You can add as many permissions as you want to a role. For example, the role below combines full access to the Workflows in the DPE with a read-only permission on every resource of a second service.
Finally, click on **Create** to save your role.
#### Using the API
```sh
# {permission} should be replaced by one or multiple permissions.
# A permission has the following format: "service:resource:action"
# e.g.:
# "iam:user:read": will give read-only access to the users list of a Project
curl --request POST \
--url '/roles' \
--header 'content-type: application/json' \
--data '{
"display_name": "My role",
"permissions":[{permission}]
}'
```
### Bind a role to a user, service account or group
#### From the Identity Access Manager interface
The process is identical for users, service account, and groups. To bind a role to any of them, simply go to their respective tab in the Identity Access Manager.
Select the user/service account/group you want to modify, and click on the **Edit** ✏️ button at the end of the row.
Click on **Add** in the Role Conditions box.
Choose the role you want to bind in the dropdown menu. You can also set a condition on the role binding (see below).
Click on **Create**, then on **Save**.
#### Using the API
```sh
# {type} should be replaced by "users", "service_accounts" or "groups"
# {id} should be replaced by the id of the user, service account or group you want to edit
# {roleId and anotherRoleId} should be the id of the roles to add
## REMEMBER that it will replace all the roles of the edited object. It won't keep other roles already there, so be careful not to remove existing roles.
curl --request PUT \
--url '/{type}/{id}' \
--header 'content-type: application/json' \
--data '{
"roles": [{
"role":"{roleId}"
},{
"role":"{anotherRoleId}"
}]
}'
```
## Conditions
Conditions are **filters that you can set on the permissions granted by a role** when you are [binding it](#bind-a-role-to-a-user-service-account-or-group) to a user / service account / group.
By default, roles are bound with no condition. Binding a role without setting a condition will give the full access with no restriction described by the permissions of this role. Adding a condition will filter the access provided by the permissions only to more specific resources, or for users validating specific properties.
:::info
For example, if you bind a role with permission `cc:alert:write` without any condition to a user, they will be able to edit the whole list of Control Center alerts.
:::
However, if you add a condition such as `Name.contains("dev-")`, this role only gives the user editing-access to the alerts whose name contain _"dev-"_, and they won't be able to edit the other alerts.
2 models of conditions currently co-exist: [legacy JSON conditions](#legacy-json-conditions) and [new CEL conditions](#new-cel-conditions). It is impossible to create a legacy JSON condition on a role that now support new CEL conditions. See down below the compatibility info for each model.
### New CEL conditions
:::info
CEL conditions are currently unavailable for roles that contain permissions for the service _API_.
:::
CEL conditions are expressed using [Common Expression Language (CEL)](https://github.com/google/cel-spec/blob/master/doc/langdef.md). They are made of **logic statements** that can be combined using boolean operators (AND, OR), each logic statement being constituted of the following elements:
- one attribute
- one operator (which can be a function applied on the attribute)
- a value
In general, CEL conditions can either be applied on the **resources** described by the role's permissions, on the **IAM principal** to which the role is granted (_coming soon!_), or on a combination of both. On top of this, conditions can also be applied on elements of the **permissions** granted by a role, in order to fully fine-grain the conditions to parts of the role only. See down the reference for all attributes and operators.
#### Condition attributes
The following conditions (i.e. condition's attributes) are currently available in the Identity Access Manager:
- **Conditions on the role's permissions**
- Service (_CEL name:_ `Service`): apply a condition on the Data Platform service containing the resource described by the permission (e.g. "Data Processing Engine", "Analytics Manager", etc.)
- Resource (_CEL name:_ `Resource`): apply a condition on the Data Platform resource described by the permission (e.g. "Workflows", "Dashboards", etc.)
- Action (_CEL name:_ `Action`): apply a condition on the access granted to the resource by the permission (e.g. "Read", "Write", etc.)
- **Conditions on the resources**
- ID (_CEL name:_ `Id`): apply a condition on the ID of the resource
- Technical Name (_CEL name:_ `Name`): apply a condition on the technical name of the
resource
- Path (_CEL name:_ `Path`): apply a condition on the path of the resource
- **Conditions on policy tags (Advanced Data Access Control)**
- Policy Tag value (_CEL name:_ `PolicyTags.`): apply a condition on the value of a policy tag assigned to the resource. Replace `` with the tag key name (e.g., `PolicyTags.sensitivity == "high"`). Available when the role contains permissions for the service `adac`.
- Policy Tag existence (_CEL name:_ `has(PolicyTags.)`): check whether a policy tag key exists on the resource (e.g., `has(PolicyTags.sensitivity)`). Returns true if the tag key is present, regardless of its value.
:::info
When using ADAC conditions, the `Resource` attribute accepts the values `dataset`, `table`, and `attribute`. The `Service` attribute value for Advanced Data Access Control is `adac`. See [Policy Tags, CEL Conditions](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#granting-access-with-cel-conditions) for detailed patterns and examples.
:::
- **Conditions on the user/service account**: coming soon!
:::warning
**WARNING**: conditions on a _read_ permission are currently only enforced when getting an object from API - they are **not enforced** in the graphical interface when listing and opening objects. For example, with a role containing read-access to _IAM groups_ bound with a condition on the _name of one IAM group_, a user will still be able to list, open and view all IAM groups in the GUI, but will only be able to call (GET) that one specific group by API.
:::
#### Condition operators
The following conditions (i.e. condition's operators) are currently available in the Identity Access Manager.
:::info
Some operators are only available for specific attributes.
:::
- Equals (CEL modelling: `attribute == ""`)
- Not equals (CEL name: `attribute != ""`)
- In (CEL name: `attribute in []`)
- Not in (CEL name: `!(attribute in [])`)
- Starts with (CEL name: `attribute.startsWith("")`)
- Not starts with (CEL name: `!(attribute.startsWith(""))`)
- Ends with (CEL name: `attribute.endsWith("")`)
- Not ends with (CEL name: `!(attribute.endsWith(""))`)
- Matches (CEL name: `attribute.matches("")`)
- Not matches (CEL name: `!(attribute.matches(""))`)
- Contains (CEL name: `attribute.contains("")`)
- Not contains (CEL name: `!(attribute.contains(""))`)
#### Examples
Here are some examples in Common Expression Language (CEL) of conditions:
1. Restrict access to only the resources named "demo":
```text
Name == "demo"
```
2. Restrict access to only the workflows contained in the folder "my\_folder", and full access to the other resources allowed by the role (provided the role initially gives access to workflows and other resources):
```text
Resource != "workflow"
||
(
Resource == "workflow"
&&
Path.contains("/my_folder")
)
```
3. Restrict access to only the bucket named "my\_bucket" (related service name is _datastore_) and to the rest of the resources in other services as allowed by the role (provided the role initially gives at least access to buckets and to some resources in other services - note that this condition will block access to resources in _datastore_ that are not buckets altogether):
```text
Service != "datastore"
||
(
Service == "datastore"
&&
Resource == "bucket"
&&
Name == "my-bucket"
)
```
4. Enforce policy tags at all three levels, dataset, table, and attribute (related service name is _adac_). This is the recommended pattern for full access control:
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "attribute")
)
```
5. Grant access to tables tagged with `sensitivity: high`, with pass-through at dataset and attribute levels (no tag enforcement at those levels):
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
6. Grant access to all tables that do not have the `sensitivity` policy tag:
```text
Service == "adac" && (
Resource == "dataset"
|| (!has(PolicyTags.sensitivity) && Resource == "table")
|| Resource == "attribute"
)
```
:::warning
Through conditions, you cannot give access to resources/actions that were not initially permitted by the role. Conditions can only filter a role further.
:::
#### Add a condition when binding a role
There are 2 ways to add a condition on a role binding.
- The **visual builder** gives a point-and-click interface to specify conditions using a field, an operator and a value (e.g. _Resource name equals my\_bucket_).
- The **custom CEL editor** is a condition writer in [Common Expression Language (CEL)](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to create similar conditions.
[Tutorial: how to build a role condition on a specific bucket](https://docs.dataplatform.ovh.net/tutorials-bucket-role-conditions.md)
### Legacy JSON conditions
The **legacy JSON editor** is a condition writer using a JSON format. JSON conditions are currently deprecated and will be migrated automatically to new [CEL conditions](#new-cel-conditions).
#### Add a role condition on specific data value(s)
:::info
This section relates to a legacy JSON condition. JSON conditions are currently deprecated and will be migrated automatically to new [CEL conditions](#new-cel-conditions).
:::
To restrict access to specific data values in your application or your API, you must use the _**filter**_ parameter on an _**API**_-related permission.
First, create a role having access to `API * *`:
Then, [bind](#bind-a-role-to-a-user-service-account-or-group) this role to the group/user of your choice, and add a **role condition**.
Set the condition with the _filter_ parameter to the binding. The _filter_ parameter takes in a dictionary of attributes names and list of values for each attribute.
```json
{
"filter": {
"station_id": [
40010
]
}
}
```
You can enter several values in the _filter_, to filter on multiple attributes at once (AND operator) or on multiple values of an attribute (OR operator).
#### Add a role condition on a specific bucket
:::info
This section relates to a legacy JSON condition. JSON conditions are currently deprecated and will be migrated automatically to new [CEL conditions](#new-cel-conditions).
:::
To restrict access to specific buckets in your [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md), you must use the _**bucket\_name**_ parameter on an _**Data store**_-related permission.
:::info
Data Store is the original name of the buckets on Data Platform.
:::
First, create a role having access to `datastore * *`.
Then, [bind](#bind-a-role-to-a-user-service-account-or-group) this role to the group/user of your choice, and add a **role condition** with the _bucket\_name_ parameter to the binding.
```json
{
"bucket_name": "mybucket"
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-groups.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Assign roles to multiple users with IAM groups
## Objective
Groups enable you to assign the same [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) to multiple [users](https://docs.dataplatform.ovh.net/iam-users.md) or [service accounts](https://docs.dataplatform.ovh.net/iam-service-accounts.md) at once.
By default, some groups are pre-generated by Data Platform for you to use directly. These are called **default groups** and they **cannot be deleted**.
For example, the default group _Data Engineers_ in the [Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md) gives its members editor roles for the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md), the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) and the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::info
Note that Groups **exist both in the Project and Organization IAMs**. The screenshots in this article have been taken in the Project IAM, but the steps are the same in the Organization IAM.
:::
## Create a group
To create a group, head to the _Groups_ tab of the Identity Access Manager. Click on **New group**.
Give your group a name. You can also add a description and tags to help you manage your groups more easily.
### Assign roles to a group
You can either give additional roles to the group right now, or at any time after it's been created.
[Learn how to assign a role to a group](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#bind-a-role-to-a-user-service-account-or-group)
### Finish creating your group
You can either [assign users](https://docs.dataplatform.ovh.net/iam-groups.md#assign-users-or-service-accounts-to-a-group) to the group right now, or at any time after it's been created.
To finish creating your group, press **Create**.
## Assign users or service accounts to a group
To add users or service accounts into a group, open the group and head to the corresponding tab.
You can see the list of users / service accounts who are currently part of the group. **Users, like service accounts, can be in multiple groups**: they wind up having all the roles associated to any group they are part of.
To add users / service accounts into the group, click on **Add** then select all the Project users / service accounts that you want to add to the group.
:::info
You can also check which groups a specific user is part of by heading to the [user page](https://docs.dataplatform.ovh.net/iam-users.md) in the Identity Access Manager.
:::
You can remove users / service accounts from a group by clicking on the **trash** icon.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-manage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage your platform
## What sits outside a project
An **organization** is the boundary for identity and billing. It holds the team members who can reach Data Platform, the certificates and domains shared across projects, the quotas that cap what projects may consume, and the billing accounts the consumption is charged to. Every project belongs to exactly one organization.
A **project** is created from the organization, in a chosen cloud region, and is deleted from there too. Its subdomain and configuration identifiers are what external tools and code use to address it, so they turn up in almost every API call and connection string in this documentation.
**Pricing** explains what is metered and what is not: the compute unit, the components that hold it, storage and query volumes, and where to watch cost before it becomes a surprise.
:::info
Permissions for these actions are not granted here. Creating and deleting projects, inviting people and changing organization settings are all governed by the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
:::
[### Navigate the platform→
Homepage, sidebar and header, and how to move between projects.
](/manage-navigation)[### Understanding Data Platform pricing→
What is metered, what is not, and where to watch cost.
](/manage-pricing)[### Organizations and user profiles→
The identity and billing boundary every project belongs to.
](/manage-organizations)[### Projects→
What a project contains and how it is isolated.
](/manage-projects)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-navigation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Navigate the platform
## Objective
Upon logging-in, you will arrive at the Platform Home (or Homepage). From any page in the platform, including the Home, you have access to the two main ways to quickly move around: the Sidebar and the Header.
- The **Homepage** is the central hub of the platform where you can see all your [Projects](https://docs.dataplatform.ovh.net/manage-projects.md) and easily go anywhere in the platform.
- The **Sidebar** allows you to navigate per service: select which service you want to go to and then the Project you want to open it in. You can also find your [Favorites](https://docs.dataplatform.ovh.net/manage-navigation.md#favorites) in the Sidebar.
- The **Header** allows you to navigate per Project: select which Project you want to go to and then the service you want to open within the Project.
## Homepage
As long as you are part of an Organisation that has Projects, everytime you log-in to the Platform you will be welcomed into the platform at the Homepage (shown below).
This page is the central hub that shows all the your Projects within an organisation and allows you to easily navigate to any service in the platform.
:::info
The Homepage is specific to a single organisation and if you are a part of multiple organisations, you will have one homepage per organisation. You can swap organisations in the [Header](https://docs.dataplatform.ovh.net/manage-navigation.md#header) and upon logging-in you will arrive at the homepage of the last organisation you accessed.
:::
### Widgets
Widgets are snippets of data about your organisation and Projects shown to you here so you can be sure to catch critical information such as compute and storage consumption without having to look for it. They are **specific to your user account** and not your organisation so any change you make to your widgets will not affect your teammates' widgets.
#### Manage Widgets
Via the _Manage Widgets_ menu you can add new wigets or hide the widget section entirely if you do not wish to see them.
### Back to recent work
Services that you accessed recently appear in this section along with their project. The list is updated everytime you access a new service.
### Browse all services
Clicking in this link will redirect you to the **All Services** page which details the entirety of Platforms' service portfolio and what each service can do for you.
### Your Projects list
All Projects of the organisation that you have access to will be displayed here in the form of a Project Card:
Each card contains:
- Project name
- Project icon
- DPU consumption
- Amount of data stored
- Cloud provider icon and deployment site name
- Creation date
- Number of users with access to the Project
#### Change a Project's icon
The Project icon can be changed by clicking on it which will display the options:
:::info
Bear in mind that changing the icon will **change it for all other user** as well, not only for you.
:::
#### Add Project Homepage to the Sidebar's favorites
By clicking on the top-right corner menu that appears upon hovering you over the card, you can add your [Project Homepage](https://docs.dataplatform.ovh.net/manage-projects-home.md) to the [Sidebar's Favorites section](https://docs.dataplatform.ovh.net/manage-navigation.md#favorites).
#### Delete a Project
By clicking on the top-right corner menu that appears upon hovering you over the card, you can also [delete your Project](https://docs.dataplatform.ovh.net/manage-projects-delete.md).
### New Project button
You can also easily create a Project from the Organisation Button by clicking on this button which will open the [Project Creation page](https://docs.dataplatform.ovh.net/manage-projects-create.md).
## Sidebar
The sidebar offers a quick way to move around the platform. It is accessible from all platform pages and opens by default whenever you open the Homepage. It is particularly useful to quickly navigate to a service you need or to access your favorites.
It opens automatically in all Platforms' pages with the exception of when you are within a service. You can [open or close it in the header](https://docs.dataplatform.ovh.net/manage-navigation.md#open-or-close-sidebar) and clicking outside it will close it automatically.
### Platform Home button
At the top there is the _Platform Home_ button. Clicking on it will send you back to the Homepage.
### Favorites
The _Favorites_ section can be used to easily access your recurrent services and Projects.
You can add two types of favorites:
- Service in Project
- Project Home (service name displayed as _Home_)
Clicking on _service in Project_ favorites will bring you back to that service in the specific Project you configured while clicking in the _Project home_ favorite will open the respective [Project's homepage](https://docs.dataplatform.ovh.net/manage-projects-home.md).
:::info
Clicking on the Project icon of a favorite will bring you to the [Project's homepage](https://docs.dataplatform.ovh.net/manage-projects-home.md) regardless of the type of favorite you added.
:::
#### Adding Favorites
You can add _service in Project_ by clicking on the pin icon that appears when hovering over a Project in the [Open Service in Project menu](https://docs.dataplatform.ovh.net/manage-navigation.md#open-service-in-project-menu).
:::info
The pin icon appears filled if you have the service/project combination in your favorites or unfilled if you do not (it is unfilled in the screenshot).
:::
Project Home favorites are [added directly via the Homepage](https://docs.dataplatform.ovh.net/manage-navigation.md#add-project-homepage-to-the-sidebars-favorites).
### All services
The entirety of the Platform service portfolio is listed here so you can quickly access them in any Project you want. Hovering over a service will display the [Open Service in Project menu](https://docs.dataplatform.ovh.net/manage-navigation.md#open-service-in-project-menu) while clicking on the service **once** will keep the aforementioned menu open for that service.
:::info
The [Billing Explorer](https://docs.dataplatform.ovh.net/manage-pricing.md) and Storage Engines services are tied only to an [Organisation](https://docs.dataplatform.ovh.net/manage-organizations.md) and not to a specific Project, therefore you do not have to select Projects when opening them.
:::
#### Open Service in Project menu
This menu allows you to **select in which Project you will open your service**. There are three options (as shown in the screenshot above):
1. Open service in a Project that already includes it.
2. Open service in a Project that doesn't include it (_Other Projects_ in the screenshot).
3. Open service in a new Project (_New Project_ button in the screenshot).
Choosing the first option (_Projects including this service_) will simply redirect you to that service's page within the Project you selected.
The second option (_Other Projects_) will activate the service in the selected Project and then you will be redirected (may take a few seconds).
The third option (_New Project_) will open the [Project creation page](https://docs.dataplatform.ovh.net/manage-projects-create.md) so you can create a new Project with that service.
## Header
The header is always displayed no matter where you are in the platform. With it you can:
- Open or close the sidebar
- See where you are in the platform.
- Quickly navigate to other Projects and organisations.
- Change organisation and move to another organisation's Homepage.
- Access your profile settings.
- Access your [organisation settings](https://docs.dataplatform.ovh.net/manage-organizations-settings.md) from any page on the Platform.
- Change language.
- Log-out.
### Open or close Sidebar
This button allows you to open or close the Sidebar no matter in which page you are.
### Organisation, Project and service panels
These panels can be displayed in icon or panel form depending on where you are in the platform. Hovering over an icon will make it turn into a panel.
From left to right, there is the:
- Organisation panel
- _Used to change organisation._
- Project panel
- _Used to change Project within an organisation._
- Service panel
- _Used to change service within a Project._
### Other settings
In this part of the header you can:
- Change the platform language
- Access your [Organisation Settings](https://docs.dataplatform.ovh.net/manage-organizations-settings.md)
- Check general useful information (components' status, roadmap, support, etc)
- Acess your [Profile Settings](https://docs.dataplatform.ovh.net/manage-organizations-profile.md)
### Maintenance banner
When the Data Platform is undergoing scheduled maintenance, a banner is displayed in the header for the entire duration of the maintenance window. The banner is visible to all users in every organisation and Project.
While the banner is displayed, the platform enters a **read-only mode**:
- **Allowed:** running existing jobs, workflows, queries, deployments and viewing resources.
- **Blocked:** creating, updating or deleting any resource (connectors, tables, actions, workflows, dashboards, IAM entities, etc.).
The banner is removed automatically once maintenance ends, and normal write operations resume immediately. No action is required on your side.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-pricing.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Understanding Data Platform pricing
## Objective
Data Platform is billed pay-as-you-go. There are no up-front costs, no per-user licence and no data transfer charges: you pay for the compute you run, by the minute, and for the data you store and query, by volume.
This page explains what generates a cost, the units those costs are counted in, and where to find the current prices. It does not list prices: those live on the OVHcloud public cloud price list, which is the single source of truth for them.
[See the Data Platform prices](https://www.ovhcloud.com/en-ie/public-cloud/prices/#data-platform)
## What you pay for
The total cost of a Data Platform project is the sum of a small number of components.
**Storage.** The data held in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md), tables and [buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) alike, charged on the volume stored over the month.
**Data processing.** The compute consumed by [Data Processing Engine jobs](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md): loads, aggregations, SQL, custom Python and PySpark. Charged on compute allocated for the duration of each run.
**Analytics queries.** The queries run from the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) and through external [consumers](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md), charged on the volume of data scanned.
**Application Services.** The compute behind deployed [APIs and apps](https://docs.dataplatform.ovh.net/landing-page-app-services.md), charged for the time each deployment is up rather than per request.
**Support.** OVHcloud support levels above the standard one are charged as a percentage of the total OVHcloud bill. See the [OVHcloud support levels](https://www.ovhcloud.com/en-ie/support-levels/) for what each includes.
## What is not billed
Data Platform does not charge:
- data transfer, in either direction, for moving data into or out of your account
- data ingest fees
- service API calls
## Billing units
Compute is expressed in **Data Platform Units**.
:::info
**1 DPU is 1 vCPU and 4 GB of RAM.** Compute cost is therefore DPU multiplied by the time those DPU were allocated, usually expressed in DPU hours.
:::
The other units follow the resource they measure: storage in GB or TB held over the month, and analytics queries in TB scanned.
Knowing which components hold DPU, and for how long, is most of cost control:
- A [job](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md) in serverless mode holds its DPU only while it runs.
- A job in always-up mode holds its DPU continuously, until the execution mode is set back to serverless.
- A [notebook](https://docs.dataplatform.ovh.net/dpe-notebooks.md) holds all its assigned DPU for as long as it is turned on, whether or not code is running.
- A deployed API or app holds its DPU while deployed.
## Worked cost examples
Three examples of what a project costs in practice, at small, mid-market and enterprise scale. They show which components each workload consumes and in what volumes, so the shape of a bill is visible before you build one.
:::warning
**Figures are as of 1 September 2026** and are illustrative, not a quote. Rates change: the [price list](https://www.ovhcloud.com/en-ie/public-cloud/prices/#data-platform) is the only current source.
:::
:::details See three worked cost examples
The three following examples provide insight into the total cost in operating OVHcloud Data Platform to load, process, query data, and expose dashboards.
#### Cost example: Smaller Company
This hypothetical customer is a small retail business leveraging Data Platform data warehouse capabilities for BI. They ingest 2 GB of data every working day, totaling 40 GB (80M lines) monthly. The process requires 2 DPU for 2 hours daily to load data, plus an additional 2 DPU for 2 hours daily to aggregate tables. Consequently, their total monthly DPU usage reaches 160 hours (2 DPU x 4 hours/day x 20 working days). Their total storage currently stands at 500 GB, and their analytics team scans an average of 2 TB of stored data monthly.
| Services | Volume / month | Cost / month |
| :-------------------------- | :--------------------------------------- | :----------- |
| Storage | 500 GB | 10.50 € |
| Data Processing | 160 h of DPU (2 DPU x 4h x 20 work days) | 12.96 € |
| Lakehouse/Analytics Queries | 2 TB | 10.96 € |
| **Total cost excl. VAT** | | **34.42 €** |
#### Cost example: Mid-Market
This hypothetical customer is a medium-sized Tech company using Data Platform to centralize and aggregate data analytics from their main Information System tools. They ingest a continuous stream of 720 GB of data per month (1.4 billion lines). Their Data Engineering team performs advanced data transformation using PySpark. Overall, data loading and transformation consume an average of 20 DPUs for 6 hours per working day each month, resulting in 2,400 hours of DPU usage for data processing (20 DPU x 6 hours x 20 working days). Their total lakehouse storage is 25 TB, and their data analysts run queries scanning a total of 50 TB of stored data. Additionally, they use 4 DPUs 24/7 to run deployed Application Services, totaling 2,920 hours of additional DPU usage (4 DPU x 730 hours).
This client decided to subscribe to [OVHcloud Business Support](https://www.ovhcloud.com/en-ie/support-levels/business/) to access our technical expert team 24/7 with less than 30 min response time.
| Service | Volume / month | Cost / month |
| :-------------------------- | :------------------------------------------------ | :------------- |
| Storage | 25 TB | 525.00 € |
| Data Processing | 2,400 h of DPU (20 DPU x 6h / day x 20 work days) | 192.00 € |
| Lakehouse/Analytics Queries | 50 TB scanned | 273.50 € |
| App Services | 2,920 h of DPU (4 DPU x 730 h on a month) | 233.60 € |
| 24/7 Business Support | 10% of total OVHcloud bill | 122.41 € |
| **Total cost excl. VAT** | | **1,346.51 €** |
#### Cost example: Enterprise
This hypothetical customer is a health insurance company operating in multiple countries, using the solution to cover the analytics needs of its subsidiaries, aggregated at a group level. They utilize an average of 40 DPUs for approximately 10 hours each, 6 days a week, for advanced data exploration and processing. They store 100 TB of data in their Data Lakehouse. Their data analyst team scans about 500 TB of data for their queries, both locally in the Data Lakehouse and on their remote on-prem storage via direct queries. Additionally, they use 4 DPUs to host APIs and application services deployed 5 days a week (400 hours a month), totaling 1,600 hours of additional DPU usage (4 DPU x 400 hours).
This client decided to subscribe to [OVHcloud Business Support](https://www.ovhcloud.com/en-ie/support-levels/business/) to access our technical experts team 24/7 with less than 30 min response time.
| Service | Volume / month | Cost / month |
| :-------------------------- | :------------------------------------------------------------ | :------------- |
| Storage | 100 TB | 2,100.00 € |
| Data Processing | 9,600 h of DPU (40 DPU x 10h / day x 6 days / week x 4 weeks) | 768.00 € |
| Lakehouse/Analytics Queries | 500 TB | 2,735.00 € |
| App Services | 1,600 h of DPU (4 DPU x 400 h) | 172.80 € |
| 24/7 Business Support | 10% of total OVHcloud bill | 577.58 € |
| **Total cost excl. VAT** | | **6,353.38 €** |
:::
## Your usage and your invoices
OVHcloud produces a monthly usage statement for every customer with at least one active service. From a project or an organization, open the configuration wheel at the top right of the screen, then **Billing information**.
The statement itemizes the month's usage in credits consumed and currency spent, and carries an estimate of the next OVHcloud invoice alongside the history of previous ones, split per service.
:::info
For each Data Platform service the statement shows its region, its type, its volume of usage and the related cost.
:::
Invoices themselves, for Data Platform and every other OVHcloud service, are accessed and managed from the [OVHcloud Control Panel
](https://www.ovh.com/manager/#/). See the [OVHcloud article on managing bills](https://help.ovhcloud.com/guides/account-and-service-management/managing-billing-payments-and-services/invoice-management)
for what can be done there.
Live resource consumption, as opposed to its cost, is visible in the [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md): CPU, memory, storage and network use per component, at project or organization level.
## Subscriptions and billing accounts
Subscription plans, committed resources and the ownership of a billing account are managed at OVHcloud account level rather than inside a project, from the [OVHcloud Control Panel
](https://www.ovh.com/manager/#/).
## Keeping costs down
- [Build a dashboard on your billing data](https://docs.dataplatform.ovh.net/tutorials-billing-analysis.md) so consumption is visible before it appears on an invoice. The [FOCUS and Superset tutorial](https://docs.dataplatform.ovh.net/tutorials-finops-focus-superset.md) takes the same data further for FinOps reporting.
- [Set quotas](https://docs.dataplatform.ovh.net/manage-organizations-settings.md#quotas) at organization level as a safeguard against unintended consumption.
- Prefer a shared [environment](https://docs.dataplatform.ovh.net/dpe-environments.md) in always-up mode over always-up on individual actions, so the same resources serve several workloads.
- Archive cold data to [OVHcloud Object Storage](https://www.ovhcloud.com/en-ie/public-cloud/object-storage/), in the Standard, Infrequent Access or Cold Archive class as the access pattern requires.
- For a structural review, OVHcloud [Professional Services](https://www.ovhcloud.com/en-ie/professional-services/) and the data consultancies in the [OVHcloud Partner Program](https://partner.ovhcloud.com/en-ie/directory/) assess data architecture and cost together.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-organizations.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Organizations and user profiles
## Objective
Organizations are the compartmentalized teams that link individual users (or members) and [Projects](https://docs.dataplatform.ovh.net/manage-projects.md) as well as their services together. You can have as many teammates in your organization as you want and everyone in the organization must have [user profile](https://docs.dataplatform.ovh.net/manage-organizations-profile.md) to use the platform.
Your organization can work on multiple [Projects](https://docs.dataplatform.ovh.net/manage-projects.md) at the same time and you can create as many as you want.
:::info
To join an organization, make sure that you already have a [User Profile](https://docs.dataplatform.ovh.net/manage-organizations-profile.md).
:::
## Create an organization
When you connect to the Data Platform from the OVHcloud Control Panel, an organization is automatically created. This organization will have the same name as the corresponding Public Cloud Project.
**Note:** Organizations cannot be created through any other method. Every organization must be linked to a Public Cloud Project.
## Join an existing organization
To join an organization that already exists (if your team already has one, for example), you need to ask someone already on there to invite you.
You will receive an email with a clickable link inviting you to join their organization.
## Manage your organization settings
You can manage your organization by opening the drop-down menu at the top right and by clicking on **Manage organization settings**.
From there, you will be able to configure:
- Your Projects
- Other parameters: Name, description, SSL certificates, domain names, etc.
[Learn how to configure an organization](https://docs.dataplatform.ovh.net/manage-organizations-settings.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create an account on Data Platform
## Objective
:::warning
ForePaaS Legacy Platform no longer accepts new signups. If your company would like to get started on the platform, please use [OVHcloud Data Platform](https://eu.dataplatform.ovh.net/).
:::
Connecting to OVHcloud Data Platform can be done in either one of two ways:
- connecting by using an OVHcloud account
- connecting by using a local Data Platform account (invitation required)
## Signup using an OVHcloud account
:::info
This is the recommended method to connect to Data Platform.
:::
You can connect to Data Platform using an OVHcloud account by pressing the blue button on the signup/signin page.
You will then have the option to either use an existing account or to create a new one there.
## Signup using a local Data Platform account
:::warning
To use this option, it is **required to [have been invited](https://docs.dataplatform.ovh.net/iam-users.md#manage-organization-iam-users) in an existing Data Platform Organization** by an admin from your company beforehand. Consequently, at least one person in your organization needs to have signed up [using an OVHcloud account](https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md#signup-using-an-ovhcloud-account).
:::
If you are joining people from your organization that are already signed-up on the platform, you can fill out the form to signup to the platform using a email & password combination.
:::warning
To use this option, it is **required to [have been invited](https://docs.dataplatform.ovh.net/iam-users.md#manage-organization-iam-users) in an existing Data Platform Organization** by an admin from your company beforehand. Consequently, at least one person in your organization needs to have signed up [using an OVHcloud account](https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md#signup-using-an-ovhcloud-account).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-organizations-settings.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage your organization settings
## Objective
To open the settings of an organization, navigate to the top right of the screen and hover over the **gear ⚙️ icon**. Click on **Manage organization settings**.
## Overview
Landing on your organization dashboard, you have access in the blink of an eye to the main information of your organization on the platform:
- Number of members and admins (**Now moved to the organization IAM**)
- Projects activity
- Resource consumption
You can also see the main metrics usage of your Project(s).
## Members (DEPRECATED)
This page has been deprecated and organization members (also known as team-members) are now managed at the [Organization IAM Users](https://docs.dataplatform.ovh.net/iam-organization.md) tab.
:::warning
Note that these are your organization's members, which are different from the Project's users you'll find in the Identity Access Manager (IAM).
:::
[Manage organization members](https://docs.dataplatform.ovh.net/iam-users.md#manage-organization-iam-users)
[Manage Project users in the IAM](https://docs.dataplatform.ovh.net/landing-page-iam-users.md)
## Projects
The Projects tab gives you an overview and direct access to all the [Projects](https://docs.dataplatform.ovh.net/manage-projects.md) created inside the organization you're currently in.
## Quotas
:::info
Quotas are now live and available!
:::
Quotas are automated restrictions applied to all organizations, serving as safeguards against unintended over-consumption. These are standard impositions by all cloud providers. They can be compared to weekly withdrawal or payment limits set by a bank.
The **Organization Quotas** section, located on the left side of the page, displays the total number of projects and DPUs available to your organization. These quotas are automatically set by the Data Platform as safeguards against accidental over-consumption. As an administrator, you have the ability to increase or decrease the number of projects and DPUs to prevent over-consumption. By default, each organization starts with 5 projects and 20 DPUs, with each project typically requiring 2 DPUs. If you require higher quotas, please submit a request to the Data Platform support team via this page. You may also consider upgrading your subscription plan.
On the right side of the screen, the **Distribution of DPUs** section allows you to distribute the total number of DPUs you have across each project within your organization. At the bottom of this section, you will see the total number of DPUs remaining to be allocated.
Furthermore, if any projects have pre-allocated resources for critical actions or workflows, this will be explicitly displayed underneath the respective project's DPU allocation, for example:
### DPU Pre-allocation for Actions and Workflows
Within the Data Processing Engine (DPE), certain actions and workflows can be critical and require consistent DPU allocation to ensure uninterrupted operation. To address this, we introduce the **pre-allocation** feature for actions and workflows.
When pre-allocation is enabled for a serverless action or workflow, a chosen number of DPUs will be permanently allocated to it. This guarantees that during runtime, the action or workflow will always have the necessary resources, preventing failures or performance issues due to resource scarcity.
This differs from the `always-up` mode. While `always-up` also ensures continuous resource availability, the key distinction is that in this mode, resources are continuously consumed, even when the action or workflow is not actively running. In contrast, pre-allocation for serverless actions/workflows only reserves the DPUs for when the action is actually executed, making it a more efficient approach for ensuring critical operations without constant consumption.
If there are any projects with pre-allocated resources, that too will be seen under the distribution of DPUs sections
## Settings
The Settings tab is the place where you'll manage:
**Basic information** about your organization: name, description, multi-factor authentication requirements, and plan cancellation. This is also where you can manage the option to let users with the same email domain as people in your organization to request to join it.
When _Force multi-factor authentication_ is enabled, organization's members will need to go through a second authentication method whenever they try to sign in to the platform using their account login/password.
**Infrastructure** options: domain name and certificates. You can customize domain names for your applications deployed on the platform, and create SSH certificates to access your secure domain names.
Adding your certificate, you'll need a server certificate (.crt file) or key (.key).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-organizations-profile.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# User Profile Guide
## Objective
Creating your own user profile is the first necessary step in order to use the platform. You can then create or join an [Organization](https://docs.dataplatform.ovh.net/manage-organizations.md)
An [Organization](https://docs.dataplatform.ovh.net/manage-organizations.md) is a workspace shared with your teammates to work on Projects. Once you are part of an [Organization](https://docs.dataplatform.ovh.net/manage-organizations.md), you can create [Projects](https://docs.dataplatform.ovh.net/manage-projects.md) which are dedicated environments that assemble multiple services together and can answer one or more use cases.
:::info
Charges are at the organization level through the purchase of a subscription plan.
:::
## Create an account
Join your team on the platform by creating yourself an account on the platform. It is completely free!
[Create your profile!](https://docs.dataplatform.ovh.net/manage-organizations-signin-signup.md)
## Create or join an organization
An organization represents a company or a specific business unit. It is where you can add all your teammates to work on multiple Projects.
[↪ Learn more about organizations!](https://docs.dataplatform.ovh.net/manage-organizations.md)
## Edit your account
At any time, you have the possibility to view and edit your user profile from the platform by clicking on **Profile** at the top right.
The user profile section of the platform is split into three main sub-pages: **Preferences**, **Security**, and **Organizations**. This guide explains each section in detail to help you manage your account and settings effectively.
## 1. Preferences
In the **Preferences** section, you can update your contact information, manage communication preferences, and view your organization memberships.
### 1.1 Contact Information
You can update your personal details, such as:
- **First Name** and **Last Name**
- **Phone Number** (with country code selection)
- **Salutation** (Mr, Ms, None)
- **Email Address**
- **Profile Picture** (option to upload)
Additionally, there is a dropdown to choose your role in the organization.
### 1.2 Information Preferences
In this section, you can opt to receive promotional content about the platform's products and services. You can also open the cookies preferences to control how your data is used.
### 1.3 My Organizations
This section lists all the organizations you are a part of, with the following details:
- **Organization Name**
- **Role** within the organization (e.g., Admin, Member)
- **Join Date**
## 2. Security
The **Security** section helps you manage the security of your account. This includes options to update your password, manage linked accounts, review recent sessions, and enable multi-factor authentication (MFA).
### 2.1 Update Password
You can change your account password here. It is recommended to choose a strong password to enhance account security.
Steps to update:
1. Enter your current password.
2. Input a new password and confirm it.
3. Save the changes.
### 2.2 Linked Accounts
You can link third-party accounts (such as Google or GitHub) to your platform account for easier login options. You can also remove linked accounts from here.
### 2.3 Recent Sessions
This section shows the details of recent login sessions, including:
- Date and time of login
- Device used
- Location (IP address)
If you notice any suspicious activity, you can terminate active sessions from this section.
### 2.4 Multi-Factor Authentication (MFA)
MFA adds an additional layer of security by requiring a second form of verification (e.g., a code sent to your mobile phone or an authenticator app). You can enable or disable MFA here.
For detailed instructions on how to configure and manage Multi-Factor Authentication (MFA) for your account, please refer to the dedicated MFA setup documentation. The document provides a step-by-step guide on enabling MFA, setting up an authenticator app or SMS-based authentication, and securely handling backup codes.
You can find the complete MFA documentation at the following location: [MFA Setup and Configuration Guide](https://docs.dataplatform.ovh.net/iam-organization-mfa-setup.md).
It is highly recommended that all users review the MFA documentation to ensure their accounts are adequately protected.
## 3. Organizations
In the **Organizations** section, you can manage your organization memberships, view available organizations, and accept invitations.
### 3.1 Invitations to Join Organizations
If you have been invited to join an organization, it will be listed here. You can accept or decline invitations as needed.
### 3.2 Organizations Available to You
This section lists the organizations that are publicly available for you to join. You can search for specific organizations using the search bar, and click **Request to Join** for the desired organization.
### 3.3 Managing Organization Memberships
Once you are part of an organization, you can manage your role within the organization, view other members, and leave the organization if necessary. Additionally, you can create new projects within the organization’s workspace.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-organizations-delete.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Delete an organization
## Objective
Deleting an organization ensures that all resources associated in this organization are **permanently deleted**. It is not possible to recover an organization that has been deleted.
To delete an organization there is a checklist to be followed. In the chance that you miss out on something in the checklist, you will not be allowed to delete the organization.\
All following 5 steps must be completed for an organization to be deleted:
1. [Delete all projects](https://docs.dataplatform.ovh.net/manage-projects-delete.md) within the organization
2. [Delete all datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md#dataset-management) from the organization
3. Make sure there is no pending balance on [billing](https://docs.dataplatform.ovh.net/manage-pricing.md#subscriptions-and-billing-accounts)
4. [Remove all users](https://docs.dataplatform.ovh.net/iam-users.md#manage-organization-iam-users) from the organization
5. Delete the organization _(see documentation below)_
## Delete the organization from the user interface
:::info
Make sure you have followed all afore-mentioned steps before going further.
:::
The option to delete an organization can be accessed by going to the organization settings. Click on the **organization settings** as shown in the image below.
Click on settings and you will land on the page which has the option to delete the organization. The button to delete will only be visible if you have followed all the steps in the checklist mentioned above.
If the checklist was not followed, you will be shown a **leave organization** button instead of the **delete organization**.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-projects.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Projects and services
## Objective
On the Platform, a **Project** is a cloud environment with dedicated resources that contains all the components required to manage your data's lifecycle, from ingestion to restitution. The physical data, in turn, is stored in a dataset linked to the Project.
## What can I do in a Project
Projects are the hub in which you access all the platform services as well as your data. The tasks that are necessary to build a data application into production are all automated, such as:
- getting an entire team to work on the same environment, with different access rights to the data or the modules
- building a logical database schema, without being linked to a specific database
- automating the data collection, ETL-ing and visualization
- monitoring the cloud infrastructure and its performances
:::info
In general, one Project can answer one or a range of specific use cases. You can use several Projects in order to work on different environments (staging, production...) for the same use case.
:::
[↪ Learn more about how to export a Project configuration.](https://docs.dataplatform.ovh.net/tutorials-export-import-config.md)
## What services are included in a Project
A Project is a flexible environment composed of multiple services, meaning you can add services in your Project as you wish. These services are separated into categories and only the _Admin tools_ services are mandatory for your project to run, all the others you may add according to your needs.
- **Data Analytics**: collection, data warehousing and processing, and querying.
- The [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md)
- The [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)
- The [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md)
- The [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
- **Application Services** : API gateways and data visualization.
- The [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md)
- The [APPs](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md)
- **Admin Tools**: resources management and access control.
- The [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md)
- The [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md)
[How to create a Project?](https://docs.dataplatform.ovh.net/manage-projects-create.md)
## How to allocate resources in a Project?
Projects were designed from scratch to scale easily as the requirements of your data Project evolve. You can independently **scale horizontally and vertically** in any component of your choice in just a few clicks, for example in the [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) of a [Data Processing Engine workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md). Deployed instances, such as [APIs](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) and [applications](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) can also be scaled independently.
Each component in your Project will consume [resources](https://docs.dataplatform.ovh.net/manage-pricing.md) when it is actively running (either through a job or a deployment).
:::info
Even when it is idle, your Project uses a handful of DPUs by default. You can manage this amount in [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md).
:::
[↪ Learn more about resource consumption](https://docs.dataplatform.ovh.net/manage-pricing.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-projects-home.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Navigate your Project from the Project home
## Objective
The Project home is the central hub of your Project. It displays all the services in the form of [Service Cards](#sections-and-service-cards) that are arranged into [Sections](#sections-and-service-cards).
In this page you can:
- Access all services in your Project.
- Add services to your Project.
- Use shortcuts to quickly go to useful pages within each service.
- View aggregate metrics of each service in one place.
- Customise the layout of the Project Home to your liking.
## Sections and Service Cards
Your service cards are grouped into sections by use case. Cards come in three sizes:
- **Small:** displays only the card's name and icon.
- **Medium:** displays the card's name, icon, two shortcut buttons and a couple of metrics or a table.
- **Large:** display card's name, icon, two shortcut buttons, a couple of metrics and one to two tables
The size of the card can be changed by clicking on the three-dots-icon that appears upon hovering over the card (card settings).
In the card settings you can:
- Change the size of the card
- Move the card
- Open the service
- Export the service configuration
## Moving cards (edit layout)
Clicking on _Move card_ in the card settings will enter the **Layout Edit Mode** which allows you to move all sections and cards.
:::info
Note that you **cannot** create custom sections or move cards from section to section.
:::
After arranging the layout of the cards you must _Confirm_ you changes or discard them by clicking on the buttons below.
## Export the service configuration
You can also easily download the service configuration files in the card settings. These files can later be [imported into the same service on any Project](https://docs.dataplatform.ovh.net/tutorials-export-import-config.md).
## Add a service
With the _Add service_ button at the bottom of the screen you can quickly activate new services in Project as it evolves. Clicking on this button will show you the service options available as well as which ones are already active in the current Project.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-projects-create.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a Project
## Objective
Creating a new Project only takes a few minutes! It is possible to choose the cloud provider and region for your data (cloud or on-premise) as well as which Project template you want to start with.
:::info
Data Platform was engineered to easily scale Projects as they grow. At any time after your Project is created, you will be able to expand its capabilities and storage volume throughout the Project thanks to horizontal and vertical scalability.
:::
## Choose a template
Start by picking the template that suits your use case the most. If no template fits your needs perfectly, you can add services to the Project later. To make sure all your needs are covered we suggest that you **pick the End-to-end template**.
## Choose a region for your Project
This is the location of the data center where:
- your Project's configuration will be stored
- your object store will be stored
- your data processing jobs, querying, APIs and applications will be running
Once you are happy with your choice, enter the name of your Project. **This name cannot be modified later on** and will by default correspond to the Domain Name System (DNS) of your Project.
:::info
You can also **customise domain and subdomain** in the advanced options of this step.
:::
## Overview and confirmation
Finally, you'll get to this ultimate screen summing up all the information related to your Project.
By clicking on **Confirm**, the creation of your Project will start.
A collection of cloud micro-services are automatically set up in just a few minutes: web-app firewalls, security groups, CDNs, DNS configuration, SSL/TLS certificates, API endpoints, capacity planning, load balancer, auto-scaling, Kubernetes, cache, ETL, data warehouse engine, and more…
You will receive an email once the creation is finished, meaning the Project is ready to be used.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-projects-config-variables.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Find your Project configuration variables
## Objective
Some configuration variables for your project are presented on this page.
## Project subdomain
Your project's subdomain is used in the base URL for the APIs of all services of the platform. It is the **stringified version of your project's name**.
You can find it in [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md) **> Applications > Project**, at the base of the field _URL_.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/manage-projects-delete.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Delete a Project
## Objective
:::info
You must have **project permission** `delete` in order to delete projects. These permissions are set at user level in the [Organization IAM](https://docs.dataplatform.ovh.net/iam-organization.md#resources-managed-by-the-organization-iam).
:::
Deleting a Project **completely and permanently erases its data**. Use only when necessary!
## Choose the Project to delete
You can delete a Project via the [Project Card menu](https://docs.dataplatform.ovh.net/manage-navigation.md#your-projects-list) in the [Homepage](https://docs.dataplatform.ovh.net/manage-navigation.md).
## Confirm decision by typing Project name
After you click on _Delete Project_ you will be prompted to confirm your decision by typing your Project name.
## Click on link on email received
You will receive an email with a link to confirm the deletion of your Project. Just click on the confirmation link to delete the Project.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-developers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform for developers
## What you can do programmatically
Everything a Data Platform project does through its interface can also be done in code. This section is written for people building on the platform rather than operating it.
The [Data Platform APIs](https://docs.dataplatform.ovh.net/landing-page-developers-apis.md) are the programmatic alternative to the interface: REST endpoints for buckets and for the Identity Access Manager, available to any user and authenticated with project credentials.
Three SDKs cover three jobs:
| SDK | Language | Main context of use |
| :---------------------------------------------------------------------------------------- | :------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Custom Actions](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) | Python | [Custom actions](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) and [integrated Jupyter notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md) |
| Front APP | ReactJS | Custom dashboards inside a deployed app |
| Front API | NodeJS | Customizing the API an app calls |
The Custom Actions SDK is the one most projects use: it runs inside Custom actions, Custom PySpark actions and notebooks, and it is how a job reads and writes datasets, tables and buckets.
:::warning
Both Front SDKs are being replaced. Their current documentation stays on the legacy site until the new ones ship. See [the Front SDKs signpost](https://docs.dataplatform.ovh.net/developers-front-sdks.md).
:::
The stack behind all of this: [Docker](https://docs.docker.com/) runs every process and deployed component, [Python 3](https://www.python.org/downloads/) backs the data processing and data science components, and [NodeJS](https://nodejs.org/en/) backs the API and app components, which matters if you intend to develop against them locally.
[### Data Platform APIs→
REST endpoints for buckets and for IAM.
](/landing-page-developers-apis)[### Custom Actions SDK (Python)→
Read and write project data from an action or a notebook.
](/landing-page-developers-python-sdk)[### Front App and API SDKs→
Both SDKs are being replaced. Current docs stay on the legacy site.
](/developers-front-sdks)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-developers-apis.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform APIs
## What the APIs cover
The Data Platform APIs are the programmatic alternative to the interface, for building and managing a [project](https://docs.dataplatform.ovh.net/manage-projects.md) in code rather than by hand.
:::warning
**The complete Data Platform API is in development.** What this section documents today is [authentication](https://docs.dataplatform.ovh.net/developers-api-authentication.md): how to obtain a token by password or by API key credentials, and how to invalidate it. Endpoint references for the rest of the platform land here as they ship.
:::
Until then, two other routes cover most programmatic work. The [Custom Actions SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md)
reads and writes datasets, tables and buckets from inside a project's own compute. Project buckets speak S31
, so any S3-compatible client works against them directly, as the [bucket access tutorial](https://docs.dataplatform.ovh.net/tutorials-bucket-s3-access.md)
shows.
The conventions below apply to the authentication endpoints and to the API as it grows.
[### Authentication→
Obtain a token by password or API key credentials, and invalidate it.
](/developers-api-authentication)
## General formats
### Response format
- Data format: JSON see [http://www.json.org](http://www.json.org)
- Character set: UTF-8
- All non-empty responses are sent with the header: `Content-Type: application/json; charset=utf-8`
### HTTP response codes
| Code | Message | Signification |
| :--- | :-------------: | :-------------------------------------------------------------------------- |
| 200 | OK | Default response for successful GET or POST requests with non-empty content |
| 400 | Bad Request | Invalid value for GET or POST parameters |
| 401 | Unauthorized | Invalid Credentials or Rights |
| 403 | Forbidden | Authentication error |
| 404 | Not Found | Action or request not found |
| 500 | Internal Server | A server-side error occurred |
### Error handling
In case of client-side (HTTP status code 4xx) or server-side (HTTP status code 5xx) error, the response body may contain a JSON object describing the error:
```json
{
"error": "Invalid parameter", // code 400, 403, 500
"warning": "Action not found" // code 404
}
```
### Timestamp
All data endpoints return the timestamp associated with the data provided.
```json
{
"timestamp": 1435929960
}
```
### Duration
All data endpoints return the duration attribute. It represents the effective time taken by the system to execute a request in seconds.
```json
{
"duration": 0.056
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/developers-api-authentication.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Authenticate against the Data Platform IAM API
## Objective
This page contains the API reference for the [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md). IAM APIs will always look like this:
```text
https://{project_url}/iam
```
## Login endpoint
### Using a login/password method
This authentication mode allows you to [connect with a login and password](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) defined in your [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
:::info
In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
Example request :
```bash
curl -X POST \
https://my-project.eu.dataplatform.ovh.net/iam/login \
-H 'Content-Type: application/json' \
-d '{
"login":"LOGIN",
"password":"****",
"app_id":"APPID",
"auth_mode":"forepaas_id_resource_owner"
}'
```
Example response :
```json
{
"uid": "John Doe", // The unique id of the logged-in user
"user": "M. John Doe", // The full name of the logged-in user
"firstname": "John", // The first name of the logged-in user
"lastname": "Doe", // The last name of the logged-in user
"civility": "M. ", // The civility of the logged-in user
"login": "john.doe", // The login of the logged-in user
"password_renew": false, // flag to know if the user has to change their password
"password_expiration": "2020-02-28T17:17:10.274Z", // password expiration date
"token": "f54e34a56b897"
}
```
| Name | Type | Description |
| :--------- | :----: | ----------------------------------------------------------------------------------------: |
| login | String | The login of the user, as received in the registration email. |
| password | String | The password of the user, as received in the registration email. |
| app\_id | String | The application identifier (set to 'forepaas' to login to the project's IAM in general) |
| auth\_mode | String | The authentication mode to used. The value must be set to “forepaas\_id\_resource\_owner” |
### API Key Credentials
This authentication mode allows you to [connect with an API and secret key](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) defined in your [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
:::info
In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
Example request:
```bash
curl -X POST \
https://my-project.eu.dataplatform.ovh.net/iam/login \
-H 'Content-Type: application/json' \
-d '{
"apikey":"APIKEY",
"secretkey":"****",
"app_id":"APPID",
"auth_mode":"apikey"
}'
```
Example response:
```json
{
"uid": "John Doe", // The unique id of the logged-in user
"user": "M. John Doe", // The full name of the logged-in user
"firstname": "John", // The first name of the logged-in user
"lastname": "Doe", // The last name of the logged-in user
"civility": "M. ", // The civility of the logged-in user
"login": "john.doe", // The login of the logged-in user
"password_renew": false, // flag to know if the user has to change their password
"password_expiration": "2020-02-28T17:17:10.274Z", // password expiration date
"token": "f54e34a56b897"
}
```
| Name | Type | Description |
| :--------- | :-------------------: | --------------------------------------------------------------------------------------: |
| apikey | String \[A-Za-z0- 9-] | The api key provided by the new device registration. |
| secretkey | String \[A-Za-z0- 9-] | The secret key provided by the new device registration. |
| app\_id | String | The application identifier (set to 'forepaas' to login to the project's IAM in general) |
| auth\_mode | String | The authentication mode to used. The value must be set to “apikey” |
## Logout endpoint
In order to delete a token, you can call the logout endpoint.
:::info
In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
:::
Example request:
```bash
curl -X DELETE \
https://my-project.eu.dataplatform.ovh.net/iam/logout?app_id=*****&token=***** \
-H 'Content-Type: application/json'
```
Example response:
```json
{
"success": true,
"status": 200,
"duration": 0.02
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Custom Actions SDK
## What the SDK is for
The Custom Actions SDK is the Python library used to interact with a project from inside the project's own compute. It is available in [Custom actions](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), [Custom PySpark actions](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md) and the [integrated Jupyter notebooks](https://docs.dataplatform.ovh.net/dpe-notebooks.md), so code can be written and tested in a notebook and then pasted into an action unchanged. Where a function is not available in one of those contexts, its own page says so.
## Quick starts
Four short walkthroughs, one per target: read a dataset, read a bucket, read a registered source, and the PySpark variants of the same reads. Each is a complete action you can paste in and run, and the natural first stop if you have not used the SDK before.
## Connect to project data
Its entry point is `connect()`, which takes a connection string and returns a connector for whatever that string names: a dataset, a table, a bucket, or a registered source. The object returned differs by target, which is why the connector guides cover each one separately. `bulk_insert()` is the counterpart for writing, taking a pandas dataframe into a destination table in batches.
PySpark is handled through Spark-compatible methods on the same connector object, so a PySpark action reads tables as Spark dataframes rather than pandas ones. The SDK reference lists every signature.
## Advanced
These guides cover what makes an action production-ready: environment variables instead of hardcoded values, segmentation so one action becomes parallel tasks, overriding a Load action, importing custom Python modules from Git, and emitting lineage events.
:::info
Code samples still import from `forepaas.*`. Those module names are unchanged on the platform today, and the samples are updated when the SDK rename ships.
:::
[### Quick start: dataset→
A first script against a project table.
](/developers-python-sdk-quick-start-dataset)[### Quick start: bucket→
A first script against a project bucket.
](/developers-python-sdk-quick-start-bucket)[### Quick start: source→
A first script against a registered source.
](/developers-python-sdk-quick-start-source)[### Quick start: PySpark→
A first PySpark script.
](/developers-python-sdk-quick-start-spark)[### Connectors and connection strings→
Every connection string option, and what each returns.
](/developers-python-sdk-connectors)[### SDK reference: connect and bulk_insert→
The two entry-point functions, with the bulk_insert signature.
](/developers-python-sdk-reference)[### Track data lineage→
Emit lineage from custom code and notebooks.
](/developers-python-sdk-lineage)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Custom Action with Lakehouse Manager tables
## Objective
This short example shows how to extract data from a table in Lakehouse Manager, transform it and then insert it or update the Lakehouse Manager.
The code is written in [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) context and it uses the _stations\_rides_ table from the [Getting Started](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) Tutorial. If you did that tutorial, you can just copy and paste the code below to test it, otherwise you need to adapt it to your tables and data sources.
:::info
Do not forget to build the table you use in the DM and then load it with a [DPE Load Action](https://docs.dataplatform.ovh.net/dpe-actions-load.md). You need to load your table before using the code below, otherwise it will not work.
:::
```python
import sys
import pandas as pd
import logging
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
logger = logging.getLogger(__name__)
def customfunc(event):
try:
logger.notice("Begin function")
# make connection to the default dataset
cn = connect("dwh/default_dataset/")
# option 1 : extract data from the table with no SQL required
df = cn.select("stations_rides")
# option 2 : extract data with custom SQL
df = cn.query("SELECT station_id, date, rides, station_name FROM stations_rides")
# perform your custom transform in the dataframe
df.loc[df["station_name"] == 'Harlem-Lake', "rides"] = 0
# reinsert your dataframe in the destination table
stats = bulk_insert(cn, "stations_rides", df)
# show insertion statistics (if DBMS compatible)
logger.info(stats)
# delete rows where station name is "Davis"
cn.delete("stations_rides", {"station_name":"Davis"})
# update rows set rides to 0 where station_id=40040
cn.update("stations_rides", {"rides":0}, {"station_id":40040})
# when finished, disconnect cn
del cn
logger.notice("END function")
except Exception as err:
raise Exception("err:{} L:{}".format(err,sys.exc_info()[2].tb_lineno))
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-bucket.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Custom Action with Data Platform Buckets
## Objective
Sometimes you need to handle a complex file format beyond our [Load Action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) capabilities.
In this case, we advise you to store and manipulate files with the [Data Platform Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) in your Project.
:::info
In the Data Platform Python SDK, the **Datastore connector** is used to interact with the Data Platform Buckets. You may think of the Datastore simply as a bucket container.
:::
The following sample is written in [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) context. You may adapt it as needed.
```python
import sys
import pandas as pd
from logging import getLogger
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
logger = getLogger(__name__)
def extract_func(event):
try:
# we get data from a bucket and we will archive them in another bucket
bucket_source_name = "your_source_bucket_name_here"
bucket_archives_name = "your_source_bucket_name_here"
# create a connector to handle bucket
bucket_connector = connect("data_store/{}".format(bucket_source_name))
# list files from bucket
files = bucket_connector.list()
# retrieve a file from Data Store bucket to temporary local folder
bucket_filepath = "stations_rides.csv"
local_filepath = "/tmp/stations_rides.csv"
bucket_connector.fget(bucket_filepath, local_filepath)
# read then transform the file as you need
# here the date column format is simply adjusted for compatibility reasons
df = pd.read_csv(local_filepath, sep=';')
df['date'] = pd.to_datetime(df['date'])
# load the dataframe into a project table named 'raw_file'
cn = connect("dwh/default_dataset/")
bulk_insert(cn, "stations_rides_artur", df)
del cn
# option 1 : copy the file into the archives bucket
bucket_archive_filepath = "archives/stations_rides.csv"
bucket_connector.fcopy_to(bucket_archives_name, bucket_archive_filepath, bucket_filepath)
# option 2 : put a file into the archives
bucket_archives = connect("data_store/{}".format(bucket_archives_name))
bucket_archives.fput(bucket_archive_filepath, local_filepath)
del bucket_archives
# delete file from source bucket
bucket_connector.delete(bucket_filepath)
# disconnect from datastore
del bucket_connector
except Exception as err:
raise Exception("err:{} L:{}".format(err,sys.exc_info()[2].tb_lineno))
```
:::tip
This also works with any **Object Store** that you may define as **S31 compatible source** in the Connectors.
:::
## Another example
Below is an example code which uploads an image to a bucket from a simple URL.
```python
from forepaas.dwh import connect
data_store = connect('data_store')
# Get bucket and upload image from URL to path forepaas/test.jpg.
# And finally get the image from the bucket
bucket_test = data_store.get_bucket('test')
lists = bucket_test.list(recursive=True)
bucket_test.put_request("https://i.stack.imgur.com/r8jTK.jpg", path='forepaas/test.jpg')
data = bucket.get('hello/test.jpg')
# Create a bucket if it does not already exists
if data_store.bucket_exists('test-exists') is False:
data_store.create_bucket('test-exists')
# Connect directly to the bucket test and remove the file
bucket_test2 = connect('data_store/test')
bucket_test2.delete('hello/test.jpg')
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-source.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Custom Action with Connectors source directly
## Objective
This example will show you how to get a file directly from a [Connectors Source](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md), treat it and put the treated data into a [Lakehouse Manager Table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
The code is written in the [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) context and it uses the _chicago\_files_ source from the [Getting Started Tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). If you did that tutorial, you can just copy and paste the code below to test it, otherwise you need to adapt it to your tables and data sources.
:::tip
This works with any `Source` protocol such as [FTP](https://docs.dataplatform.ovh.net/connectors-sources-ftp.md), [Dropbox](https://docs.dataplatform.ovh.net/connectors-sources-dropbox.md), etc.
:::
```python
import sys
from forepaas.dwh.connect import connect
from forepaas.dwh import bulk_insert
import logging
logger = logging.getLogger(__name__)
def customfunc(event):
try:
# here we are connecting to a source named 'chicago_files'
source_address = "dwh/chicago_files_artur/"
# specify unsupported filename from list of files in source
filename_w_extension = "stations_rides.csv"
# connect to to file to get the address
source_file_connector = connect(source_address + filename_w_extension)
# connect to file directly
file_address = source_file_connector.get()
file_connector = connect(file_address)
# Extract and treat the file so it is usable
df = file_connector.extract(return_type='dataframe')
# Treat the data
# - - - -
# connect to the Lakehouse Manager
dm_connector = connect("dwh/default_dataset/")
# insert into an existing destination table
stats = bulk_insert(dm_connector, "chicago_calendar_full", df)
logger.info(stats)
# disconnect from datastore and remote source
del source_connector
del dm_connector
except Exception as err:
raise Exception(f"err:{err} L:{sys.exc_info()[2].tb_lineno}")
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-spark.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PySpark Use Cases
## Objective
This guide walks through five PySpark use cases with the Data Platform Python SDK: reading from a bucket, querying databases with SQL, extracting a dataframe, and writing to a bucket or to object storage.
## Use Case 1: read from Bucket and write to Dataset
This sample show how to retrieve data from [Data Platform Bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) _bucket\_test_ and insert it into the _default\_dataset_ dataset from the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
Please refer to [Data Platform Buckets Connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-bucket.md) and [Lakehouse Manager Dataset Connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md) for more details on the connection strings.
```python
from logging import getLogger
from forepaas.dwh import connect, update_metas
from pyspark import SparkContext
from pyspark.sql import SQLContext
logger = getLogger(__name__)
cn_source = connect("dwh/bucket_test/chicago_calendar_full.csv")
cn_default = connect("dwh/default_dataset/")
# Spark compatible connector extract_dataframe function returns Spark DataFrame
spark_df = cn_source.extract_dataframe()
logger.notice(f"CSV Columns: {list(spark_df.columns)}")
# insert_dataframe uses Spark DataFrame as well
cn_default.insert_dataframe("chicago_calendar_full_copy", spark_df)
# At the end you can run update_metas() it will update the metas for all tables, so from lakehouse manager, you will see the correct number of rows
update_metas()
```
## Use Case 2: SQL query the Databases
This sample highlights the querying of databases through the _get\_spark\_options()_ and _get\_spark\_context()_ methods of the [Data Platform Connector object](https://docs.dataplatform.ovh.net/developers-python-sdk-reference.md#the-connect-function).
```python
from logging import getLogger
from forepaas.dwh import connect
from pyspark import SparkContext
from pyspark.sql import SQLContext
logger = getLogger(__name__)
# get_spark_context() and get_spark_options() are available for database connectors (snowflake, postgresql, mysql)
cn_default = connect("dwh/default_dataset/")
sc_default = cn_default.get_spark_context()
so_default = cn_default.get_spark_options()
# Depends on Snowflake or PostgreSQL
sql_driver = "net.snowflake.spark.snowflake" # "jdbc" or "net.snowflake.spark.snowflake"
spark_default = SQLContext(sc_default)
sql = "select * from chicago_calendar_full"
spark_df = spark_default.read.format(sql_driver).options(**so_default).option("query", sql).load()
logger.notice(f"SQL Columns: {list(spark_df.columns)}")
```
:::info
Note that this use case is only working with MySQL, PostgreSQL and Snowflake
:::
## Use Case 3: extracting a PySpark Dataframe (with options)
This sample shows how to quickly extract Spark DataFrames with the _extract\_dataframe()_ method and also how to do it manually with the _get\_spark\_url()_, _get\_spark\_context()_ and _get\_spark\_session()_ methods.
```python
from logging import getLogger
from forepaas.dwh import connect
from pyspark import SparkContext
from pyspark.sql import SQLContext
logger = getLogger(__name__)
# Get all tables from default_dataset
cn_default = connect("dwh/default_dataset/")
# If needed you can print all tables
# logger.info(cn_default.list())
cn_source = connect("dwh/bucket_test/chicago_calendar_full.csv")
# Manual override extract_dataframe file options
spark_df = cn_source.extract_dataframe(options)
logger.notice(f"CSV1 Columns: {list(spark_df.columns)}")
# Manual read from file
# get_spark_url(), get_spark_context() and get_spark_session() are available for s3 / buckets connectors
spark_session = cn_source.get_spark_session()
# getting stations_rides.csv under bucket buc_test
url = cn_source.get_spark_url("", "stations_rides.csv", bucket="buc_test")
logger.notice(f"SparkURL: {url}")
# Use format(file_suffix) for other files, check spark documentation for more information
options= {"encoding": "utf-8", "sep": ";", "header": True}
spark_df = spark_session.read.format("csv").options(**options).load(url)
logger.notice(f"CSV2 Columns: {list(spark_df.columns)}")
```
## Use Case 4: write to another bucket
This sample shows how to use the _get\_spark\_url()_ method to write from one [Data Platform Bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) to another.
```python
from logging import getLogger
from forepaas.dwh import connect
from pyspark import SparkContext
from pyspark.sql import SQLContext
logger = getLogger(__name__)
cn_source = connect("dwh/bucket_test/stations_rides.csv")
spark_df = cn_source.extract_dataframe()
url_dst = cn_source.get_spark_url("", "stations_rides_copy.csv", bucket="test2")
logger.notice(f"SparkURL Dest: {url_dst}")
spark_df.write.format("csv").options(**options).save(url_dst)
url_dst = cn_source.get_spark_url("", "stations_rides_copy.parquet", bucket="test3")
logger.notice(f"SparkURL Dest: {url_dst}")
spark_df.write.format("parquet").save(url_dst)
```
## Use Case 5: write to object storage using insert\_dataframe
The `insert_dataframe()`
function simplifies operations in the previous use case.
It is available for compatible object-storage-type PySpark connectors (currently Data Platform Buckets, S31
and [Azure Blob Storage](https://docs.dataplatform.ovh.net/connectors-sources-blob-storage.md)
)
```python
from logging import getLogger
from forepaas.dwh import connect
logger = getLogger(__name__)
cn_source = connect("dwh/bucket_test/chicago_calendar_full.csv")
spark_df = cn_source.extract_dataframe()
# cn_dest: Data Platform Buckets, S3, Azure Blob Storage
cn_dest = connect("dwh/dest_bucket/")
# Insert using custom type
params={"type": "csv"}
# Insert using default type inferred from file name and default platform options
# Raises an exception if file suffix not in ["csv", "json", "parquet"]
# Destination path will be destination where you will find a .csv file under the configured path of the source
cn_dest.insert_dataframe("destination", spark_df)
# Insert using custom type
params = {"type": "parquet"}
# Reads from params.type, if not provided and no suffix in file name, an exception will be raised
cn_dest.insert_dataframe("destination", spark_df, params)
# Insert to specified absolute path
params = {"path": "output/destination", "type": "csv"}
# Destination file will be output/destination.csv
cn_dest.insert_dataframe("", spark_df, params)
# Insert with custom options
params = {"write_options": {"sep": ",", "header": False}, "type": "csv"}
cn_dest.insert_dataframe("destination", spark_df, params)
# Current default platform options:
# CSV: {"encoding": "utf-8", "sep": ";", "header": True}
```
:::info
When saving a PySpark DataFrame to a file system like S3, PySpark creates a folder instead of a single file because it processes data in a distributed manner. Inside this folder, multiple part-\*.csv files are generated, each representing a partition of the DataFrame, with the number of files depending on the DataFrame's partitions. An empty \_SUCCESS file is also created to indicate a successful write operation.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-connectors.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connectors and Connection Strings
## Objective
In Data Platform's Project, all available **data** are defined and configured in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
In the Python SDK, all those **data sources** are named by a unique **connection string** and accessible through **connectors** returned by the `connect()` function. The returned **Connector object** as well as its available methods depend on the **data source's type**.
A simple example to list tables from the default dataset:
```python
from forepaas.dwh import connect
# connect to default dataset
cn_prim = connect('dwh/default_dataset/')
# list databases tables
tables = cn_prim.list()
print(tables)
```
:::info
**Connection strings** are used in all Project's components to reference **data sources**, for instance in Data Processing Engine [Load](https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md) and [Aggregate](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md) actions or when [applying segmentation](https://docs.dataplatform.ovh.net/tutorials-segmentation-table-attribute.md#understanding-advanced-parameters).
:::
## Buckets
[Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) are Data Platforms' built-in object store. Below, the connection string and its respective Connector object returned by the `connect function`.
| connection string | Connector | description |
| :-------------------------------- | :-------: | :---------------------------- |
| `connect("data_store")` | DataStore | Connects to the buckets |
| `connect("data_store/my-bucket")` | Bucket | Connects to a specific bucket |
Read more about [DataStore and Bucket connectors](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-bucket.md).
## Storage Engines
Projects are linked to **Storage Engines** as main databases to store and query data at the heart of your Project.
A Project can be defined with multiple [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md), but by default you will receive 1 dataset: **default\_dataset**. You can also create custom internal and external datasets.
| connection string | Connector | description |
| :-------------------------------- | :-------: | :----------------------------------------------------------------- |
| `connect("dwh/default_dataset/")` | Dataset | Connects to the default dataset - **default\_dataset** |
| `connect("dwh/exampleDataset/")` | Dataset | Connects to the custom dataset created by you - **exampleDataset** |
Read more about [Database connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md).
## Query Engine (Trino)
| connection string | Connector | description |
| :-------------------------------------------- | :-------: | :------------------------------------------------- |
| `connect("query_engine")` | Trino | Connects to the query engine (Trino) |
| `connect(f"query_engine/{catalog}/{schema}")` | Trino | Connects directly to a **schema** in a **catalog** |
## Sources
You may also connect to external data sources declared in [Connectors's Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md). Each external source has an associated **connection string** based on the initial name given to the source.
For instance, a new data source named `My first data` will have the connection string equal to `dwh/my_first_data`.\
The **technical name** is displayed on the main Connectors Source's page.
External sources may be split in a few types:
- **Databases connectors** have same methods as storage engines, see [Database connectores](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md#connector-methods)\
(i.e. Snowflake, PostgreSQL, Oracle, Microsoft SQL Server, Amazon Redshift, MongoDB...)
- **Protocol connectors** when accessing file storage or object store (Data Platform Buckets, Data Platform File Upload, SFTP, Amazon S31, ...)\
Those connectors have same methods as [Bucket connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-bucket.md#bucket-connector-methods)
- **Stream** (i.e. Kafka, MQIT, Amazon Kinesis) or **External APIs** (i.e. Google Analytics, Facebook, Twitter, Mailchimp..)\
Those connectors are mainly designed to be used from [Connectors Analyzer](https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md) and [Data Processing Engine Load Actions](https://docs.dataplatform.ovh.net/dpe-actions-load.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Lakehouse Manager Dataset connector
## Objective
:::info
Using Dataset connector, you may query, insert, update or delete content programmatically.
:::
## Connect to the Lakehouse Manager
In order to create a Connector and use it to interact with a [Lakehouse Manager table](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md), you can connect to the **Default Dataset** or to a **Custom Dataset**.
Below, you will see the connection strings used for each type:
```python
from forepaas.dwh import connect
cn_default = connect("dwh/default_dataset/")
cn_custom = connect("dwh/custom_dataset_name/")
```
After that you can use the `cn_default.list()` method to see the tables available in your Lakehouse Manager and then `cn_default.select(...)` or `cn_default.query(...)` to get the data from the table you want.
See the next section of this article for additional details.
:::warning
Note that **tables need to be loaded in the Lakehouse Manager** before using the `Connector.list()` method and other Connector object methods. In other words, you have to create a table first in the Lakehouse Manager in order to use a table in the SDK.
:::
## Connector methods
### list()
Get the list of tables in the dataset.
**Output**
| Type | Description | Example |
| :--: | :------------------ | :-------------------------------------------- |
| list | list of table names | `["chicago_calendar_full", "stations_rides"]` |
### query(sql, limit=-1, return\_type='dataframe')
Execute a SQL query on a compatible `source`, returns a dataframe (default), an iterable cursor, or a list of dict.
**Input Parameters**
| Name | Type | Description | Example |
| :----------- | :--: | :--------------------------------------- | :------------------------------ |
| sql | str | SQL query to execute | `select * from stations_rides` |
| limit | int | maximum number of results (-1: no limit) | -1 |
| return\_type | str | type of return | 'dataframe', 'cursor' or 'dict' |
**Output**
| Type | Description | Example |
| :---: | :---------------------------------------------------------- | :------ |
| mixed | results in the choosen type (dataframe, cursor, list\[dict] | |
### select(table\_name, conds=\{}, limit=-1, return\_type='dataframe' )
Extract data from a table with simple filters, returns a dataframe (default), an iterable cursor, or a list of dict.
**Input Parameters**
| Name | Type | Description | Example |
| :----------- | :----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------- |
| table\_name | str | source table name | `stations_rides` |
| conds | dict | Dict of values to use as filters. `Keys` are attributes to filter with `values` as the filtering values. `List` produces a IN operator, otherwise produce a `=` operator | None / `{"date":["01/11/2013", "01/12/2013"], "station_name":"Harlem-Lake"}` |
| limit | int | maximum number of results. (-1: no limit) | -1 |
| return\_type | string | type of return | 'dataframe', 'cursor' or 'dict' |
**Output**
| Type | Description | Example |
| :---: | :---------------------------------------------------------- | :------ |
| mixed | results in the choosen type (dataframe, cursor, list\[dict] | |
### update(table\_name, set, conds=None, ignore=False)
Performs an `UPDATE` SQL query through simple parameters and return the number of affected rows.
**Input Parameters**
| Name | Type | Description | Example |
| :---------- | :-----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------- |
| table\_name | str | name of the source table | `'stations_rides'` |
| set | dict | Dict of values to update. Key as the field name to update, values as the value to update. | `{“rides”:0}` |
| conds | dict | Dict of values to use as filters. `Keys` are attributes to filter with `values` as the filtering values. `List` produces a IN operator, otherwise produce a `=` operator | None / `{"date":["01/11/2013", "01/12/2013"], "station_name":"Harlem-Lake"}` |
| ignore | boolean | If `True`, performs an `UPDATE IGNORE` query. Otherwise `UPDATE` query. | `True`, `False` |
**Outputs**
| Type | Description | Example |
| :--: | :------------------------------------------ | :------ |
| int | Number of updated rows (if DBMS compatible) | 42 |
### delete(table\_name, conds)
Performs a deletion based on a specific query with filtering conditions. Returns the number of deleted rows.\
If you don’t need any filter, use the `truncate` method described below.
**Input Parameters**
| Name | Type | Description | Example |
| :---------- | :--: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------- |
| table\_name | str | name of the source table | `'stations_rides'` |
| conds | dict | Dict of values to use as filters. `Keys` are attributes to filter with `values` as the filtering values. `List` produces a IN operator, otherwise produce a `=` operator | `{"station_name":"Harlem-Lake"}` |
**Outputs**
| Type | Description | Example |
| :--: | :------------------------------------------ | :------ |
| int | Number of deleted rows (if DBMS compatible) | 42 |
### truncate(table\_name)
Truncates (clears) all rows from a table.
**Input Parameters**
| Name | Type | Description | Example |
| :---------- | :--: | :----------------------- | :----------------- |
| table\_name | str | name of the source table | `'stations_rides'` |
**Outputs**
| Type | Description | Example |
| :-----: | :---------------------------------------------------------------------------------------------------- | :-------------- |
| boolean | True if truncate succeeded. False if table does not exist. Raise Exception if anything else occurred. | `True`, `False` |
### insert(table\_name, rows, odku=0, returnWarnings=False)
Executes an insert query, adds `on duplicate key update` (ODKU) operation optionally, and returns statistics about the insertion of the batch.\
If you activate the ODKU option, the query will automatically performs updates on existing rows instead of returning errors, regarding the existing primary key values in each row that you are inserting.
:::warning
Please note that you **MUST** provide, for each row that you want to insert, at least the primary key of the destination table.
:::
:::info
We encourage you to use the module **bulk\_insert** instead of this function, as documented in the Connector methods section of this page, because it has better management of data chunk and insertion.
:::
**Input Parameters**
| Name | Type | Description | Example |
| :------------- | :-------: | :---------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------- |
| table\_name | str | name of the source table | `'stations_rides'` |
| rows | dataframe | Dataframe to insert or update. | `pd.DataFrame([ {"station_id": 1, "station_name":"First Station"}, {"station_id":2, "name":"Second Station"}])` |
| odku | boolean | `False` = Don’t perform an ODKU; `True` = Performs an ODKU | `True`, `False` |
| returnWarnings | boolean | If True, return warnings aggregated and counted for each type of warning strings returned | True / False |
**Outputs**
| Type | Description | Example |
| :--: | :----------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| dict | Statistics about the inserted batch. | `{"inserts":13, "skipped":10, "updates":3, "records": 30, "affected":4, "warnings": {"integer truncated":4}}` |
### drop\_table(table\_name)
drops the table from the query engine
**Input Parameters**
| Name | Type | Description | Example |
| :---------- | :--: | :----------------------- | :----------------- |
| table\_name | str | name of the source table | `'stations_rides'` |
**Outputs**
| Type | Description | Example |
| :-----: | :------------------------------------------------------------------------------------------------ | :-------------- |
| boolean | True if drop succeeded. False if table does not exist. Raise Exception if anything else occurred. | `True`, `False` |
### get\_table\_schema(table\_name, catalog\_name, schema\_name, strict=True)
Return a table schema as a list of `forepaas.dwh.attributes.AttributeSchema` object
**Input Parameters**
| Name | Type | Description | Example |
| :------------ | :--: | :--------------------------------------------------------------------------------------------------- | :------------------ |
| table\_name | str | name of the source table | `'stations_rides'` |
| catalog\_name | str | name of the catalog. If `None`, the default connector's catalog is used. | `'default_dataset'` |
| schema\_name | str | name of the schema. If `None` uniqueness of the table will be checked across all catalog's schemas. | `'default_schema'` |
| strict | bool | if true, will raise an error if a column has an unsupported type. Otherwise will just log a warning. | True / False |
**Output Parameters**
| Type | Description |
| :--: | :---------------------------------------------------------------------------------------------- |
| list | AttributeSchema object can have 4 attributes: `name`, `data_type`, `nullable` & `default_value` |
**Example Output**
```python
[
AttributeSchema(name='id', data_type=DataType.INT, nullable=False),
AttributeSchema(name='name', data_type=DataType.VARCHAR, nullable=True),
AttributeSchema(name='email', data_type=DataType.VARCHAR, nullable=False, default_value=''),
AttributeSchema(name='created_at', data_type=DataType.TIMESTAMP, nullable=False)
]
```
## LogicalObject Methods
In order to use the `LogicalObject` methods you will need to first import it using:
```python
from forepaas.dwh.logical import LogicalObject
logical = LogicalObject()
logical.list()
```
### build(objects)
Launches a logical build in the Lakehouse Manager.
**Input Parameters**
| Name | Type | Description | Example |
| :------ | :----: | :---------------------------------- | :--------------------------------- |
| objects | `list` | A list of logical objects to build. | `['user_data', 'product_metrics']` |
**Outputs**
| Type | Description |
| :---: | :----------------------------------- |
| `int` | The status code of the build result. |
**Output Example**
```python
200 # Build successful
```
### create\_from\_physical(table, dataset='default\_dataset', \*\*kwargs)
Creates a logical object based on an existing physical table.
**Note**: Currently works only with Trino and PostgreSQL catalogs.
**Input Parameters**
| Name | Type | Description | Example |
| :------ | :----: | :----------------------------------------------------------------------------------------- | :----------------------------------------------------- |
| table | `str` | The name of the physical table. | `'raw_customer_data'` |
| dataset | `str` | The dataset name. Works only with Trino/PostgreSQL datasets. Default: `'default_dataset'`. | `'my_trino_dataset'` |
| kwargs | `dict` | Named arguments passed as JSON content to the API for additional configuration. | `{'owner': 'data_team', 'tags': ['customers', 'raw']}` |
**Outputs**
| Type | Description |
| :-------------: | :-------------------------- |
| `LogicalObject` | The created logical object. |
### get(name)
Retrieves a logical object from the Lakehouse Manager.
**Input Parameters**
| Name | Type | Description | Example |
| :--- | :---: | :------------------------------------------ | :----------------------- |
| name | `str` | The name of the logical object to retrieve. | `'sales_dashboard_view'` |
**Outputs**
| Type | Description |
| :-------------: | :---------------------------- |
| `LogicalObject` | The retrieved logical object. |
### list(dataset\_name='default\_dataset')
Lists all logical objects in the Lakehouse Manager.
**Input Parameters**
| Name | Type | Description | Example |
| :------------ | :---: | :------------------------------------------------------------------------- | :------------------ |
| dataset\_name | `str` | The dataset to filter logical objects by. Defaults to `"default_dataset"`. | `'default_dataset'` |
**Outputs**
| Type | Description |
| :-------------------: | :------------------------- |
| `list[LogicalObject]` | A list of logical objects. |
### remove(name)
Removes a logical object from the Lakehouse Manager.
**Input Parameters**
| Name | Type | Description | Example |
| :--- | :---: | :---------------------------------------- | :----------------- |
| name | `str` | The name of the logical object to remove. | `'old_temp_table'` |
**Outputs**
| Type | Description |
| :------: | :----------------------------------- |
| `object` | The status of the removal operation. |
**Output Example**
```python
{'success': True, 'info': None, '_id': '690325793a04befefec45bf1', 'uid': None}
```
## Complementary notes
Due to lacks of metrics collected by PostgreSQL or Snowflake libraries, the statistics returned by Data Platform connectors are calculated based on limited factors.
- When no error happens, _inserts_ and _skipped_ rows are calculated based on total rows before and after inserting, as well as the _records_ (length of data to insert) provided. For example, if we try to insert 3 rows into a table of 5 rows, and the result is 7 rows in total, we consider 2 rows are _inserted_ and 1 row _skipped_, while _records_ is 3.
- If error happens in a batch of data, the whole batch will be marked as warnings.
- Note for Snowflake: statistics returned by `insert_many()` and `insert_dataframe()` wont have _warnings_ calculated, all rows fail to insert or skipped will be marked as _skipped_
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-connect-bucket.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform Buckets connector
## Objective
:::info
In the current Data Platform SDK, the **Datastore connector** is used to interact with the Data Platform Buckets. You may think of the Datastore simply as a bucket container.
:::
## Connect to the Datastore
In order to interact with the Datastore you have to connect to it first, as shown in the code below:
```python
from forepaas.dwh import connect
cn_datastore = connect('data_store')
```
After that you can use the `cn_datastore.list()` method to see the buckets available in your Datastore and then connect to the bucket of your choice to interact with it.
You can connect directly to a specific bucket in the Datastore as shown in the code below:
```python
from forepaas.dwh import connect
bucket_name = "name"
cn_bucket = connect('data_store/' + bucket_name)
```
The datastore connector will return a Data Store Connector object and connecting directly to a bucket will return a Bucket Connector object.
See the next section of this article for additional details on the methods of each connector.
## Datastore Connector methods
### datastore.list(return\_type = 'array')
Lists all buckets in the Datastore.
**Input Parameters**
| Name | Type | Description | Example |
| :----------- | :--: | :-------------------------------------------------- | :------ |
| return\_type | str | Determine the type you want to get `array` or `str` | |
**Output**
| Type | Description | Example |
| :--------------: | :---------------------------- | :------ |
| `str` or `array` | List of buckets in Data Store | |
### datastore.get\_buckets()
Gets all buckets from the Datastore
**Output**
| Type | Description | Example |
| :------------: | :------------------------- | :------ |
| `list[bucket]` | list of `bucket` instances | |
### datastore.get\_bucket(name)
Gets a bucket instance from its name.
**Input Parameters**
| Name | Type | Description | Example |
| :--- | :--: | :---------- | :------ |
| name | str | Bucket name | |
**Output**
| Type | Description | Example |
| :------: | :------------------------------ | :------ |
| `bucket` | Bucket instance to handle files | |
### datastore.create\_bucket(name)
Adds a bucket in the Data Store.
**Input Parameters**
| Name | Type | Description | Example |
| :--- | :--: | :---------- | :------ |
| name | str | Bucket name | |
**Output**
| Type | Description | Example |
| :-------: | :------------------- | :------ |
| `boolean` | Success of operation | |
### datastore.remove\_bucket(name)
Removes a bucket from the Data Store.
**Input Parameters**
| Name | Type | Description | Example |
| :--- | :--: | :---------- | :------ |
| name | str | Bucket name | |
**Output**
| Type | Description | Example |
| :-------: | :------------------- | :------ |
| `boolean` | Success of operation | |
### datastore.bucket\_exists(name)
Finds out if a bucket exists or not.
**Input Parameters**
| Name | Type | Description | Example |
| :--- | :--: | :---------- | :------ |
| name | str | Bucket name | |
**Output**
| Type | Description | Example |
| :-------: | :------------------- | :------ |
| `boolean` | Success of operation | |
## Bucket Connector methods
### bucket.list(bool metadata=True, bool recursive=True, \*\*kwargs)
Lists files from Data Store's bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---------- | ---- | ------------------------------------------------ | ------- |
| metadata | bool | (optional) Get metadata for all files listed | True |
| recursive | bool | (optional) List recursively through folders | True |
| \*\*kwargs | | Additional arguments passed to list\_objects\_v2 | |
**Output**
| Type | Description | Example |
| ------------- | -------------------- | ------- |
| list\[Object] | List of bucket files | |
**Short Example**
```python
from forepaas.dwh.connect import connect
import logging
bucket_name = "name"
bucket = connect('data_store/' + bucket_name)
files = bucket.list()
logger.info(f"Bucket contents: {files}")
```
### bucket.list\_filename(return\_type='array', contains='', recursive=True, \*\*kwargs)
Lists filenames from Data Store's bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ------------ | ---- | ------------------------------------------------ | ------- |
| return\_type | str | Return type format: 'array' or 'str' | 'array' |
| contains | str | Filter filenames containing this value | '2023' |
| recursive | bool | (optional) List recursively through folders | True |
| \*\*kwargs | | Additional arguments passed to list\_objects\_v2 | |
**Output**
| Type | Description | Example |
| ------------- | --------------------------- | -------------- |
| str or \[str] | List of filenames in bucket | \['file1.csv'] |
**Short Example**
```python
filenames = bucket.list_filename()
logger.info(f"Filenames: {filenames}")
```
### bucket.get(file\_name, \*\*kwargs)
Gets raw file content from a Data Store bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---------- | ---- | ---------------------------- | --------------- |
| file\_name | str | Name of the file to retrieve | "data/file.csv" |
**Output**
| Type | Description |
| ----------------------------- | ---------------------------- |
| urllib3.response.HTTPResponse | HTTP response with file data |
**Short Example**
```python
file_data = bucket.get('uploads/file.csv')
logger.info(f"Stream: {file_data.stream(1024)}")
```
### bucket.fget(object\_name, file\_path, \*\*kwargs)
Gets an object from Datastore's bucket to local path.
**Input Parameters**
| Name | Type | Description | Example |
| ------------ | ---- | ----------------------------------- | ---------------------- |
| object\_name | str | Name of the object in the bucket | "data.csv" |
| file\_path | str | Local path where file will be saved | "./downloads/data.csv" |
**Output**
| Type | Description |
| ------ | ----------------------- |
| Object | Object stat information |
**Short Example**
```python
bucket.fget("uploads/file.csv", "/tmp/file.csv")
logger.info("File downloaded to /tmp/file.csv")
```
### bucket.put(object\_name, data, int length, \*\*kwargs)
Puts an object to Data Store's bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ------------ | ------------ | ------------------------- | -------------- |
| object\_name | str | Name to assign the object | "uploaded.csv" |
| data | io.RawIOBase | Data stream | stream |
| length | int | Length of the data | 2048 |
**Output**
| Type | Description |
| ---- | ----------------------- |
| str | Object ETag from server |
**Short Example**
```python
import io
data = io.BytesIO(b"name,age\nJohn,30")
etag = bucket.put("people.csv", data, data.getbuffer().nbytes)
logger.info(f"Uploaded with ETag: {etag}")
```
### bucket.fput(object\_name, file\_path, \*\*kwargs)
Puts a file to Data Store's bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ------------ | ---- | -------------------------------- | -------------- |
| object\_name | str | Name of object to be created | "backup.csv" |
| file\_path | str | Path to the file on local system | "./backup.csv" |
**Output**
| Type | Description |
| ---- | ----------------------- |
| str | Object ETag from server |
**Short Example**
```python
bucket.fput("people.csv", "/tmp/people.csv")
logger.info("Uploaded /tmp/people.csv")
```
### bucket.put\_request(url, path, data=\{}, method='GET', headers=\{}, \*\*kwargs)
Gets an object from an HTTP request and upload it to Datastore's bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ------- | ---- | -------------------------------------- | ---------------- |
| url | str | Source URL to download the object from | "https\://..." |
| path | str | Path to store the file in the bucket | "raw/data.csv" |
| data | dict | Request body data (if any) | \{} |
| method | str | HTTP method to use | 'GET' |
| headers | dict | Custom headers for the request | \{'Auth': '...'} |
**Short Example**
```python
bucket.put_request(
url="https://example.com/file.csv",
path="remote/file.csv"
)
logger.info("File fetched from URL and uploaded to bucket.")
```
### bucket.delete(path, \*\*kwargs)
Deletes multiple/single file in Data Store's bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---- | ----------- | ---------------------------- | --------------- |
| path | str or list | Path(s) of file(s) to delete | "data/file.csv" |
**Short Example**
```python
bucket.delete("people.csv")
logger.info("File deleted from bucket.")
```
### bucket.fcopy\_to(new\_bucket, object\_name, object\_source, \*\*kwargs)
Copies file from bucket to a new bucket in Data Store.
**Input Parameters**
| Name | Type | Description | Example |
| -------------- | ---- | ---------------------------------------- | ------------------ |
| new\_bucket | str | Target bucket name | "archive" |
| object\_name | str | New name for the copied object | "file\_backup.csv" |
| object\_source | str | Original object's name in current bucket | "file.csv" |
**Short Example**
```python
bucket.fcopy_to("archive", "people_backup.csv", "people.csv")
logger.info("File copied to archive bucket.")
```
### bucket.exists(filename, \*\*kwargs)
Checks whether a file exists in the bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---------- | ---- | -------------------------------------- | --------------- |
| filename | str | Name or path of the file to check | "logs/2024.csv" |
| `**kwargs` | | Additional options for internal checks | |
**Output**
| Type | Description |
| ---- | ----------------------- |
| bool | Whether the file exists |
**Short Example**
```python
if bucket.exists("people.csv"):
logger.info("File exists.")
```
### bucket.get\_content(file\_name, \*\*kwargs)
Retrieves the full content of a file from the bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---------- | ---- | ---------------------------------- | ------------ |
| file\_name | str | Name of the file in the bucket | "people.csv" |
| `**kwargs` | | Additional options (e.g., version) | |
**Output**
| Type | Description |
| ----------- | ----------------------- |
| bytes / str | Raw content of the file |
**Short Example**
```python
content = bucket.get_content("people.csv")
logger.info(f"File content: {content.decode()}")
```
### bucket.get\_path(path)
Returns the full qualified path (URL or reference) of an object in the bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---- | ---- | ----------------------------- | --------------------- |
| path | str | Path or object name in bucket | "reports/summary.csv" |
**Output**
| Type | Description |
| ---- | ----------------------- |
| str | Full path to the object |
**Short Example**
```python
full_path = bucket.get_path("people.csv")
logger.info(f"Full path: {full_path}")
```
### bucket.remove\_path(path)
Removes a specific path from the bucket.
**Input Parameters**
| Name | Type | Description | Example |
| ---- | ---- | -------------------------------- | ------------------ |
| path | str | Path to the object to be removed | "uploads/file.csv" |
**Output**
| Type | Description |
| ---- | ---------------------- |
| bool | Success of the removal |
**Short Example**
```python
bucket.remove_path("people.csv")
logger.info("Removed specific path from bucket.")
```
## Deprecated Methods
:::info
The `bucket.stat()` method is no longer supported and has been deprecated. Use `exists()` or `get_content()` as alternatives depending on your use case.
:::
## Additional methods
The Data Platform Datastore is built on [Minio](https://min.io/) technology. Please refer to the [Minio Technical Documentation](https://docs.min.io/docs/python-client-api-reference.html) for more information on the advanced settings of the SDK functions.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-connect-spark.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connect PySpark with the Data Platform Python SDK
## Objective
:::info
In a [Custom Action PySpark](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md), you can use the Connector object's Spark-compatible methods. These methods are available **only in Custom PySpark Actions** and they vary according what type of source you are connected to.
:::
Do not hesitate to check out the [sample code](https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-spark.md) for common use cases.
## The connect module with PySpark
When you create a [Custom PySpark action](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md), two things are done by Data Platform:
- _SparkContext_ is created before calling the script.
- _SparkContext.stop()_ is called after the script.
So you can simply access Spark's functionalities with:
```python
SparkContext.getOrCreate()
```
To access the data stored within Data Platform, you have to first instantiate a [Connector object with the connect function](https://docs.dataplatform.ovh.net/developers-python-sdk-reference.md#the-connect-function). Then you can use **Spark-compatible methods in the Connector object** to interact with the connected data.
For instance, the method for getting a Spark Dataframe object is **connnector.extract\_dataframe()**. The sample code below connects directly to a source and then uses that method to get the Spark object:
```python
from forepaas.dwh import connect
cn_source = connect("dwh/file_upload_source/chicago_calendar_full.csv")
# Spark compatible connector extract_dataframe function returns Spark DataFrame
spark_df = cn_source.extract_dataframe()
```
:::info
Note that not all Data Platform connectors are compatible with PySpark. Some compatible connectors are: Snowflake, PostgreSQL, MySQL, Amazon S31
, File-Upload and Data Platform Bucket but this list is evolving constantly.
:::
## Spark-compatible Connector Methods
### get\_spark\_options()
Returns Spark options to connect to a Database.
_Not available with Protocol sources (Buckets, File Upload, Amazon S3, etc)._
**Output**
| Type | Description | Example |
| :--: | :------------------------------------- | :--------- |
| Dict | SparkOptions to connect to a Database. | See below. |
```python
{
"sfAccount": ...,
"sfURL": ...,
"sfUser": ...,
"sfSchema": ...,
"sfDatabase": ...,
"sfTimezone": ...,
"sfWarehouse": ...,
"preactions": ...,
}
```
### get\_spark\_context()
Returns the current SparkContext.
**Output**
| Type | Description | Example |
| :------------------: | :-------------------- | :------ |
| pyspark.SparkContext | Current SparkContext. | - |
### get\_spark\_session()
Returns current spark session, configured with different settings when using Data Platform bucket / file-upload.
_Available only with Protocol sources (Buckets, File Upload, Amazon S3, etc)._
**Output**
| Type | Description | Example |
| :----------------------: | :--------------------- | :------ |
| pyspark.sql.SparkSession | Current Spark session. | - |
### get\_spark\_url(path, filename, bucket=None)
Returns file url to given path of object in object store. path needs to be absolute path in object store, and if bucket is not set, it uses the one from the user config
_Available only with Protocol sources (Buckets, File Upload, Amazon S3, etc)._
**Input Parameters**
| Name | Type | Description | Example |
| :---------------- | :--: | :------------------------------------------- | :------ |
| path | str | Absolute path within the bucket to the file. | - |
| filename | str | Name of the file with extension. | - |
| bucket (optional) | str | Bucket in the object store. | - |
**Output**
| Type | Description | Example |
| :--: | :-------------- | :-------------------------------------------- |
| str | File Spark URL. | `s3a://bucket_name/path/chicago_calendar.csv` |
### insert\_dataframe(table, dataframe)
Inserts a Spark DataFrame into the connected source.
_Available with Databases and some Protocol sources (Buckets, Amazon S3, Azure Blob Storage)._
For more details, check the [use case 4 and 5](https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-spark.md).
**Input Parameters**
| Name | Type | Description | Example |
| :-------- | :-------------------: | :---------------------------------------- | :------------ |
| table | str | Table name inside the database. | `my_s3_table` |
| dataframe | pyspark.sql.DataFrame | DataFrame containing data to be inserted. | - |
### extract\_dataframe(params=\{})
Extract SparkDataframe from file. File extraction options can be set via params or in table.parameters
**Input Parameters**
| Name | Type | Description | Example |
| :----- | :--: | :--------------------------------------------------------------------------------------------------------------- | :--------------- |
| params | Dict | Dict that contain a SQL query structure to execute. An empty dictionary by default, results in getting all rows. | Empty dict: `{}` |
Detailed example:
```python
params = {
"scale": ["attribute_1", "attribute_2"],
"joins": [{
"type": "INNER",
"table": "table_1",
"condition": "table_1.attribute = table_2.attribute"
}],
"schema": {
"attribute_dest": "attribute_source"
}
}
```
**Output**
| Type | Description | Example |
| :-------------------: | :------------------------------- | :------ |
| pyspark.sql.DataFrame | DataFrame with the sources data. | - |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# SDK reference: connect and bulk_insert
## Objective
The two functions every script in the [Custom Actions SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) starts from: `connect()` to reach a dataset, table, bucket or source, and `bulk_insert()` to write a dataframe back.
## The connect function
The `connect()` function is, most of the time, the main entry point to interact with the [Project](https://docs.dataplatform.ovh.net/manage-projects.md).
It takes a **connection string** as parameter and returns a **Connector object**.\
For example:
```python
from forepaas.dwh import connect
connection_string = "dwh/default_dataset/"
connector_default = connect(connection_string)
```
Change the connection string to connect to different sources within the Data Platform environment.\
As a consequence, the object, and available methods, returned by `connect(...)` will be different.
Our [Connectors and Connection Strings article](https://docs.dataplatform.ovh.net/developers-python-sdk-connectors.md) explains each option available in more detail and below you will find the method list for two common connectors:
- [Lakehouse Manager Dataset Connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md)
- [Data Platform Buckets Connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-bucket.md)
## The bulk\_insert function
The `bulk_insert()` function is a very important part of the SDK. You can use it to insert data back into a table. As shown in the code below, it takes a [Dataset Connector](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md), a table name and a pandas DataFrame as parameters.
```python
import pandas as pd
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
# make connection to the default dataset
cn = connect("dwh/default_dataset/")
# extract data from the table with no SQL required
df = cn.select("stations_rides")
# perform your custom transform in the dataframe
df.loc[df["station_name"] == 'Harlem-Lake', "rides"] = 0
# reinsert your dataframe in the destination table
stats = bulk_insert(cn, "stations_rides", df)
```
### bulk\_insert(connector, table, data, source\_default\_schema=\{}, batch\_size=None)
**Input Parameters**
| Name | Type | Description | Example |
| :-------------- | :--------------: | :---------------------------------------------------------------------------------------------------- | :------------------------------------- |
| connector | Connector | Connector instance from the connect() function | `cn = connect("dwh/default_dataset/")` |
| table | str | name of the destination table where the data is going to be loaded in | - |
| dataframe | pandas.DataFrame | Dataframe of data to insert, must to have the same column names and types that the target table does. | - |
| default\_schema | dict | - | - |
| batch\_size | int | insertion batch size | - |
**Output**
| Type | Description | Example |
| :-----------------: | :--------------------------------------- | :------ |
| Tuple(stats, error) | tuple of the statistics of the insertion | - |
## PySpark Support
Data Platform offers extensive support to handle data with PySpark.
All is done through Spark-compatible methods in the Connector object.\
Check out our [PySpark article](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-spark.md) for further details.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-use-environment-variables.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use environment variables in a Custom action
## Objective
In a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), you might want to use environment variables. You can set your own in the preferences or set them in the JSON _Advanced Mode_ of the action, in the `params` JSON object, and re-use them in your custom action.
```python
import sys
from logging import getLogger
from forepaas.core.settings import PARAMS
logger = logging.getLogger(__name__)
def my_custom(event):
try:
# Show the parameter value
logger.info("Environment variable value is: ".format(PARAMS["env variable key"]))
except Exception as err:
raise Exception("err:{} L:{}".format(err,sys.exc_info()[2].tb_lineno))
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-segmentation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use segmentation (and perimeter) in a Custom action
## Objective
In a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), you can fully benefit from the segmentation system and even go further:
- parallelizing the calculation of your complex algorithm
- parallelizing huge custom data loading
- etc..
You can set the **segmentation type and values** as you would do in other compatible action types, through the interface, or through advanced JSON mode.
For more information, please refer to the [segmentation documentation page](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md)
Here, you will learn **how to get the segmentation configuration and values** in your custom Python script, in order to apply it as you want.\
Let's dig into an example where you set a segmentation over a set of values, and apply a filter on the source.
```python
import sys
from logging import getLogger
from forepaas.core.settings import PARAMS
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert
logger = logging.getLogger(__name__)
def my_custom(event):
try:
# get the segmentation and perimeter settings
segmentation = PARAMS.get("segmentation",{})
segmentationValues = PARAMS.get("segmentation",{}).get("values")
perimeter = PARAMS.get("perimeter",{})
perimeterValues = PARAMS.get("perimeter",{}).get("values")
# set a filter dict regarding the segmentation and perimeter configuration
filters = {}
if segmentation["type"] in ["workflow_dates", "dwh_attributes", "predefined_set"] and segmentationValues is not None:
filters[segmentation["var_name"]] = segmentationValues
if perimeter["type"] in ["workflow_dates", "dwh_attributes", "predefined_set"] and perimeterValues is not None:
filters[perimeter["var_name"]] = perimeterValues
cn = connect("dwh/default_dataset/")
# extract with filters
df = cn.select(table_name, filters)
# perform some changes on the df optionaly
# df = df
# insert into destination
bulk_insert(cn, table_destination, df)
# close connection
del cn
except Exception as err:
raise Exception("err:{} L:{}".format(err,sys.exc_info()[2].tb_lineno))
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-load-override.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Override a Load action
## Objective
In a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), you can re-use existing action types, overriding some parameters or creating "hooks".
Here is an example of re-using a [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md), adding a mapping value equals to the date contained in the source filename.
:::info
**Some requirements to follow:**
1. select the `aggregate` module in the required modules
2. your JSON `params` must follow the [required format for the load action](https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md)
:::
**Tip** : you may first create your action configuration through a regular Load action, then copy / paste the "params" JSON object into your custom action JSON `params` object.
```python
import sys,os
from logging import getLogger
from forepaas.core.settings import PARAMS
from forepaas.actions.aggregate.fpaggregate import aggregate
logger = getLogger(__name__)
def override_aggregate(event):
try:
# Set filename. could be get through segmentation or parameters
filename = "sales_2020-01-01.csv"
# Override source filename
path = "/".join(PARAMS["load_from"][0]["source"].split("/")[:-1]) + "/" + filename
PARAMS["load_from"][0]["source"] = path
# override source_date in schema
source_date = filename.replace("sales_","").replace(".csv","")
PARAMS["schema"]["source_date"] = {"type":"replace", "value": str(source_date)}
logger.info(f"{filename} {source_date}")
# executing the aggregate (=load) action, it will use the overriden PARAMS.
aggregate(event)
except Exception as err:
message="error on file:{}: {} L:{}".format(filename,err,sys.exc_info()[2].tb_lineno)
raise Exception(message)
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-git.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Use custom Python modules from a Git repository
## Objective
In some cases, you will want to use your own and already existing Python code.
In order to import your custom Python modules and use it in a [Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), you may refer to an external Git repository that will be pulled before each build of your job.
Here are the steps in order to use an external Git repository as a Python module:
1. Adapt your module so that its compatible with `pip install`
- see [setuptools documentation](https://packaging.python.org/guides/distributing-packages-using-setuptools/)
2. Your Python module must be hosted on an accessible host, if repository is private, you need to get an `access token`.
For instance, in [GitHub](https://github.com), you must generate a **personal access token**
(Other providers may have similar procedures)
- [how to set tokens](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token)
- [set your personal token in GitHub](https://github.com/settings/tokens)
3. Check that downloading of the package works well with your token
- Zip Archive: `https://@github.com///archive/.zip`
- Git: `git+https://@github.com//.git`
- Git with a tag name or commit hash: `git+https://@github.com//.git@`
- Git with automatic latest release: `git+https://@github.com//.git@latest`
4. Check that `pip install` works with your URL on your local computer
```bash
pip install
```
5. In your project, in a custom action, add the dependencies in the **Advanced Mode** in `forepaas.json`:
- with a HTTPS Zip archive
```json
{
"dependencies": {
"python": {
"https://@github.com///archive/.zip": null
}
}
}
```
- with a Git+HTTPS repository
```json
{
"dependencies": {
"python": {
"git+https://@github.com//.git": null
}
}
}
```
- with a Git+HTTPS repository, including a `tag` name or a `commit hash`
```json
{
"dependencies": {
"python": {
"git+https://@github.com//.git@": null
}
}
}
```
- with a Git+HTTPS repository, automatically using the latest release
```json
{
"dependencies": {
"python": {
"git+https://@github.com//.git@latest": null
}
}
}
```
:::info
**Auto-resolving the latest release:** Using `@latest` at the end of a `git+` dependency URL instructs the platform to automatically fetch and substitute the most recent release tag before installing. This removes the need to update each action's dependency manually when a new version of your module is published, click **Force Build** to pick up the new release. Note that the latest release is not detected automatically at runtime, a manual **Force Build** is always required.
:::
:::warning
Pinning a specific `tag` or `commit hash` gives you precise control over which version is used at execution time and is recommended for production workloads.
:::
6. In your custom action python code, you can now import and use it!
```python
import mypackage
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Track data lineage in a Custom action
## Objective
Data Platform records lineage events automatically for **Load** and **Aggregate** actions, Python and PySpark alike (including schema and column-level lineage). For **Custom actions** (Python and PySpark) and **notebooks**, lineage is opt-in: you decide which datasets to declare as inputs and outputs.
:::info
**Want to visualize lineage?** Explore it directly in the built-in [Lineage](https://docs.dataplatform.ovh.net/lakehouse-manager-lineage.md) view of the Lakehouse Manager, or forward all lineage events to your own OpenLineage-compatible solution (e.g. [Marquez](https://marquezproject.ai/)): set up the [OpenLineage consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md) in Connectors, then use the [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) DPE action to push events on a schedule or continuously.
:::
## Quick start with `lineage_run`
The `lineage_run` context manager is the recommended way to track lineage. It handles the full lifecycle automatically:
- Generates a `run_id` (UUID v4)
- Emits a `START` event on entry
- Emits a `COMPLETE` event on successful exit
- Emits a `FAIL` event on exception (then re-raises your error)
- Lineage errors never crash your action. All emit calls are wrapped in `try/except`
```python
from forepaas.dwh import connect
from forepaas.dwh.lineage import lineage_run
def my_custom(event):
with lineage_run("custom_titanic_transform",
inputs=["default_dataset/titanic"],
outputs=["default_dataset/titanic_survivors"]):
connector = connect("dwh/default_dataset/")
connector.query("""
CREATE TABLE IF NOT EXISTS titanic_survivors AS
SELECT passengerid, name, sex, age, pclass, fare, embarked
FROM titanic
WHERE survived = 1
""")
```
:::info
Pass table names as plain strings (`database/table`). The platform automatically attaches the correct namespace.
:::
## Examples
### 1. Simple SQL transformation
```python
from forepaas.dwh import connect
from forepaas.dwh.lineage import lineage_run
def my_custom(event):
with lineage_run("custom_titanic_transform",
inputs=["default_dataset/titanic"],
outputs=["default_dataset/titanic_survivors"]):
connector = connect("dwh/default_dataset/")
connector.query("""
CREATE TABLE IF NOT EXISTS titanic_survivors AS
SELECT passengerid, name, sex, age, pclass, fare, embarked
FROM titanic
WHERE survived = 1
""")
```
### 2. `bulk_insert` with new columns
```python
from forepaas.dwh import connect, bulk_insert
from forepaas.dwh.lineage import lineage_run
def my_custom(event):
with lineage_run("custom_titanic_newcolumns",
inputs=["default_dataset/titanic"],
outputs=["default_dataset/titanic"]):
connector = connect("dwh/default_dataset/")
df = connector.query("SELECT * FROM titanic")
df["newsurvived"] = df["survived"].apply(lambda x: "Yes" if x == 1 else "No")
df["newclass"] = df["pclass"].apply(lambda x: f"Class {x}")
bulk_insert(connector, "titanic", df)
```
### 3. Automatic schema detection with `connector=`
When you pass a connector, `lineage_run` automatically calls `connector.get_table_schema()` for each input and output, and enriches the `COMPLETE` event with schema facets (column names and types). The `START` event is emitted with simple inputs/outputs (no schema). If `get_table_schema` fails for a table, that table is kept as-is without crashing.
```python
from forepaas.dwh import connect, bulk_insert
from forepaas.dwh.lineage import lineage_run
def my_custom(event):
connector = connect("dwh/default_dataset/")
with lineage_run("custom_titanic_newcolumns",
inputs=["default_dataset/titanic"],
outputs=["default_dataset/titanic"],
connector=connector):
df = connector.query("SELECT * FROM titanic")
df["newsurvived"] = df["survived"].apply(lambda x: "Yes" if x == 1 else "No")
df["newclass"] = df["pclass"].apply(lambda x: f"Class {x}")
bulk_insert(connector, "titanic", df)
```
### 4. Multiple connectors (cross-database)
When inputs and outputs span different databases, pass a dict mapping each database prefix to its connector. Each table is routed to the correct connector for schema detection. Tables with no matching prefix are kept as-is.
```python
from forepaas.dwh import connect, bulk_insert
from forepaas.dwh.lineage import lineage_run
def my_custom(event):
cn_default = connect("dwh/default_dataset/")
cn_analytics = connect("dwh/analytics_dataset/")
with lineage_run("enrich_orders",
inputs=["default_dataset/raw_orders", "default_dataset/customers"],
outputs=["analytics_dataset/enriched_orders"],
connector={
"default_dataset": cn_default,
"analytics_dataset": cn_analytics,
}):
orders = cn_default.query("SELECT * FROM raw_orders")
customers = cn_default.query("SELECT * FROM customers")
enriched = orders.merge(customers, on="customer_id", how="left")
bulk_insert(cn_analytics, "enriched_orders", enriched)
```
### 5. Manual schema with `schema_facet`
If you want to declare schemas explicitly (without a connector), use the `schema_facet` helper. It builds the correct OpenLineage dict with `_producer` and `_schemaURL` automatically.
```python
from forepaas.dwh import connect
from forepaas.dwh.lineage import lineage_run, schema_facet
def my_custom(event):
with lineage_run("custom_titanic_transform",
inputs=[schema_facet("default_dataset/titanic", [
("passengerid", "Integer"),
("survived", "Integer"),
("name", "String"),
("sex", "String"),
("age", "Number"),
])],
outputs=[schema_facet("default_dataset/titanic_survivors", [
("passengerid", "Integer"),
("name", "String"),
("sex", "String"),
("age", "Number"),
("pclass", "Integer"),
("fare", "Number"),
("embarked", "String"),
])]):
connector = connect("dwh/default_dataset/")
connector.query("""
CREATE TABLE IF NOT EXISTS titanic_survivors AS
SELECT passengerid, name, sex, age, pclass, fare, embarked
FROM titanic
WHERE survived = 1
""")
```
### 6. Join with column lineage
For joins or complex transformations, use `column_lineage_facet` to declare which output columns come from which input tables and columns.
```python
from forepaas.dwh import connect
from forepaas.dwh.lineage import lineage_run, column_lineage_facet
def my_custom(event):
connector = connect("dwh/default_dataset/")
with lineage_run("test_with_join",
inputs=["default_dataset/titanic",
"default_dataset/chicago_calendar_full"],
outputs=[column_lineage_facet("default_dataset/titanic_enriched", {
"passengerid": {
"source": "default_dataset/titanic",
"field": "passengerid",
},
"name": {
"source": "default_dataset/titanic",
"field": "name",
},
"humidity": {
"source": "default_dataset/chicago_calendar_full",
"field": "humidity",
"operation": "JOIN",
},
"temperature": {
"source": "default_dataset/chicago_calendar_full",
"field": "temperature",
"operation": "JOIN",
},
})],
connector=connector):
connector.query("""
CREATE TABLE IF NOT EXISTS titanic_enriched AS
SELECT t.passengerid, t.name, c.humidity, c.temperature
FROM titanic t
INNER JOIN chicago_calendar_full c
ON t.passengerid = c.passengerid
""")
```
:::info
When both `connector=` and a `column_lineage_facet` are used together, the connector auto-adds schema facets to assets that don't already have one. Assets with existing facets (like column lineage) are preserved as-is.
:::
### 7. Access the run ID
The context manager yields a `LineageRun` object with a `run_id` attribute.
```python
from forepaas.dwh.lineage import lineage_run
def my_custom(event):
with lineage_run("my_job",
inputs=["default_dataset/source"],
outputs=["default_dataset/target"]) as run:
print(f"Run ID: {run.run_id}")
# ... processing ...
```
## API reference
### `lineage_run(job_name, ...)`
```python
from forepaas.dwh.lineage import lineage_run
```
**Parameters**
| Name | Type | Required | Description |
| :----------- | :-------------------: | :------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `job_name` | `str` | Yes | A stable identifier for the job |
| `inputs` | `list` | No | Datasets read by the job (strings or dicts) |
| `outputs` | `list` | No | Datasets written by the job (strings or dicts) |
| `run_id` | `str` | No | UUID for this run. Auto-generated if omitted |
| `job_facets` | `dict` | No | Additional OpenLineage job facets |
| `connector` | `connector` or `dict` | No | Connector for auto schema enrichment. Pass a single connector or a dict mapping database prefix to connector (see [Example 4](#4-multiple-connectors-cross-database)) |
Yields a `LineageRun` object with a `run_id` attribute.
### `schema_facet(name, fields)`
```python
from forepaas.dwh.lineage import schema_facet
```
Builds a dataset dict with a `SchemaDatasetFacet`. Adds `_producer` and `_schemaURL` automatically.
**Parameters**
| Name | Type | Required | Description |
| :------- | :-----------: | :------: | :--------------------------------------------- |
| `name` | `str` | Yes | Dataset name (e.g. `"default_dataset/orders"`) |
| `fields` | `list[tuple]` | Yes | List of `(column_name, column_type)` tuples |
Returns a dict suitable for use in `inputs` or `outputs`.
**Example**
```python
schema_facet("default_dataset/orders", [
("order_id", "Integer"),
("amount", "Number"),
])
# Returns:
# {
# "name": "default_dataset/orders",
# "facets": {
# "schema": {
# "_producer": "https://gitlab.forepaas.com/...",
# "_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json",
# "fields": [
# {"name": "order_id", "type": "Integer"},
# {"name": "amount", "type": "Number"}
# ]
# }
# }
# }
```
### `column_lineage_facet(name, mappings)`
```python
from forepaas.dwh.lineage import column_lineage_facet
```
Builds a dataset dict with a `ColumnLineageDatasetFacet`. Adds `_producer` and `_schemaURL` automatically. The namespace is injected from the platform configuration.
**Parameters**
| Name | Type | Required | Description |
| :--------- | :----: | :------: | :--------------------------------------------------- |
| `name` | `str` | Yes | Dataset name (e.g. `"analytics_dataset/enriched"`) |
| `mappings` | `dict` | Yes | Dict mapping output column names to input field info |
Each value in `mappings` is a dict with:
- `source` (required): source dataset name
- `field` (required): source column name
- `operation` (optional): transformation description (e.g. `"SUM"`, `"JOIN"`)
Returns a dict suitable for use in `outputs`.
**Example**
```python
column_lineage_facet("analytics_dataset/enriched", {
"order_id": {"source": "raw_orders", "field": "order_id"},
"total": {"source": "raw_orders", "field": "amount", "operation": "SUM"},
})
```
### `emit_lineage(event_type, job_name, ...)`
```python
from forepaas.dwh.lineage import emit_lineage, EventType
```
Low-level function that sends a single OpenLineage `RunEvent`. Use [`lineage_run`](#lineage_runjob_name-) instead for most use cases.
**Parameters**
| Name | Type | Required | Description |
| :----------- | :------------------: | :------: | :--------------------------------------------------- |
| `event_type` | `EventType` or `str` | Yes | One of `START`, `COMPLETE`, `ABORT`, `FAIL`, `OTHER` |
| `job_name` | `str` | Yes | A stable identifier for the job |
| `run_id` | `str` | No | UUID for this run. Auto-generated if omitted |
| `inputs` | `list` | No | Datasets read by the job |
| `outputs` | `list` | No | Datasets written by the job |
| `job_facets` | `dict` | No | Additional OpenLineage job facets |
| `event_time` | `str` | No | ISO-8601 timestamp. Defaults to `datetime.utcnow()` |
### Input/output format
Each entry in `inputs` or `outputs` can be:
- A **string**: the dataset name (namespace is added automatically):
```python
inputs = ["default_dataset/raw_orders"]
```
- A **dict**: when you need to attach facets or override the namespace:
```python
inputs = [{"name": "default_dataset/raw_orders", "namespace": "custom_ns"}]
```
- A **helper**: [`schema_facet(..)`](#schema_facetname-fields) or [`column_lineage_facet(..)`](#column_lineage_facetname-mappings) which return dicts.
All formats can be mixed in the same list.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/developers-front-sdks.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Front App and API SDKs
## Objective
The Front APP SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. The rewrite has no delivery date yet. Until it ships, both SDKs are documented in full in this section.
:::warning
**Rework pending, no delivery date.** The documentation below covers the current releases, remains accurate, and is maintained. Code written against these SDKs keeps working.
:::
- [Front App](https://docs.dataplatform.ovh.net/sdk/app.md), ReactJS, for building custom dashboards and components inside a deployed app.
- [Front API](https://docs.dataplatform.ovh.net/sdk/api.md), NodeJS, for customizing the API an app calls.
For platform work that does not touch the front end, the [Custom Actions SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) in Python is documented in full in this section, and the [Data Platform APIs](https://docs.dataplatform.ovh.net/landing-page-developers-apis.md) cover buckets and IAM.
:::info
Building an app right now? The [Apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) pages cover the low-code path, which needs no SDK, and the [Develop your own app](https://docs.dataplatform.ovh.net/tutorials-app-development.md) tutorial covers importing your own code from Git.
:::
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/app.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Application Development SDK
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Data Platform' flexible platform provides the ability to access the source code of all parts of your app and **use ReactJS to customize them**. You can develop your own styles, charts and components to create an app that matches to your needs and vision.
:::warning
We strongly suggest that you also take a look at [our Getting Further guide](https://docs.dataplatform.ovh.net/tutorials-app-development.md). It goes through specific use cases and practical examples.
:::
***
## Applications Customization Guides
Learn how to:
- Leverage [style templates](https://docs.dataplatform.ovh.net/sdk/app-charts-template.md) to improve **productivity and maintenability** with existing charts.
- Add [simple custom components](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md) that suits your needs.
- Improve your UX with [custom Dynamic Parameters](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-create.md).
- Make [custom charts](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md) for unique data vizualization.
***
## Technical Specifications
You might also want to check out the technical description of Data Platform apps' main default building blocks:
- [Default Dynamic Parameters technical specification](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters.md)
- [Formatters technical specification](https://docs.dataplatform.ovh.net/sdk/app-formatter.md)
- [Containers technical specification](https://docs.dataplatform.ovh.net/sdk/app-container-panel.md)
- [Basic UI items technical specification](https://docs.dataplatform.ovh.net/sdk/app-basic-ui-button.md)
***
## Frequently Asked Questions
Cant find what you're looking for in the above sections? Check out our FAQ for the App Manager.
[Read our FAQ on Application Development](https://docs.dataplatform.ovh.net/sdk/app-faq.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Dynamic Parameters
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Dynamic parameters enables your end-users to select values which are passed on to Charts to be filtered on and refreshed.
Dynamic Parameters values are applied to Chart requests at the [Front API level](https://docs.dataplatform.ovh.net/sdk/api.md) via [Analytics Manager Transformers](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md). You may extend default Transformers to your own needs.
For specific display needs, it is also possible to create custom Dynamic Parameters.
***
## List of available Dynamic Parameters
| name | description |
| :----------------------------------------------------------------------------------------------------- | :---------------------------------------------- |
| [selectbox](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-selectbox.md) | Select from a list, one or more items |
| [checkbox](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-checkbox.md) | Select from checkboxes one or more items |
| [radio](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-radio.md) | Selects an item through a list of radio buttons |
| [datepicker](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-datepicker.md) | Select a date range |
| [datepicker.day](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-daypicker.md) | Select a specific day |
| [datepicker.months](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-monthspicker.md) | Select a month range |
| [datepicker.month](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-monthpicker.md) | Select a specific month |
| [toggle](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-toggle.md) | Return a boolean |
### Configuration
```json
{
"id": "my-id",
"component": "selectbox",
"type": "filter"
}
```
| fields | description |
| :---------- | :------------------------------------------------------------------------------------------------------------------------------ |
| `id` | Used when saving its state and to connect it to Charts. |
| `component` | Specify the dynamic parameter to use, either those listed above, or your own. |
| `type` | Specify how `requests` are affected, acting either as a `filter` or as a `scale`. Custom transformers may handle your own types |
:::warning
**Tip** When using **Architect** it will auto-generate an `id` with a non-meaningful name. We strongly recommend to rename it as soon as you have created it, as this `id` will be used in `chart.dynamic-parameters`.
:::
***
## Adding Dynamic Parameters in Dashboard
You can add a dynamic parameter :
- from the online dashboard editor and its dialog boxes
- in **Advanced Mode** to edit JSON configuration
- by editing directly configuration files in `./config/`
- Via the `menu` tab in the main menu selection
***
## Using a Dynamic Parameter in another React Component
You may use a dynamic parameter directly from code. Use the same JSON configuration as described above, and then call it as any React component.
Example:
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
const myDynP = {
id: 'city',
type: 'filter',
component: 'checkbox',
reference: 'city',
dictionary: 'city',
default: ['Paris', 'San Francisco']
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
***
## Creating your own Dynamic Parameter
You can write your own component to customize the appearance of your dynamic parameter.
?>➡️ Check out the available tutorial [**here**](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-create.md)!
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-checkbox.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Checkbox
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Checkboxes allow you to choose one or more items from a defined list.
```json
{
"id": "year",
"type": "filter",
"component": "checkbox",
"reference": "year",
"default": ["2020"]
"dictionary": "year",
"items": [
{
"value": "2020",
"label": "Year 2020"
},
{
"value": "2019",
"label": "Year 2019"
},
{
"value": "2018",
"label": "Year 2018"
}
]
}
```
| fields | description |
| :----------- | :------------------------------------------------------------------------------------------------------------------ |
| `reference` | Specifies the name of the field to modify in the queries in the case of a transform of type `filter` |
| `default` | When you first display, it will take this value. |
| `dictionary` | Load items automatically and dynamically from your DataPlant |
| `items` | Manually define the elements to display, `label` will be displayed, `value` will be sent in the associated queries. |
:::warning
Checkboxes choices comes either from a list of `items` **or** from a `dictionary`.
:::
## Code example
Example:
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
import moment from 'moment'
const myDynP = {
id: 'city',
type: 'filter',
component: 'checkbox',
reference: 'city',
dictionary: 'city'
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-datepicker.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Datepicker
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The Datepicker allows you to select a **date range** to filter the data in a chart.
## Configuration
Here is a configuration example followed by further explanations.
```json
{
"id": "datepicker",
"type": "filter",
"component": "datepicker",
"reference": "range_date",
"format": "YYYY-MM-DD",
"startAt": "subtract(1,'year').startOf('day')",
"endAt": "endOf('day')",
"minDate": "startOf('day')",
"maxDate": "endOf('day')",
"request": {
"data": {
"fields": {
"date": ["max", "min"]
}
}
},
"ranges": [
{
"label": "Past week",
"start": "subtract(1, 'week').startOf('week')",
"end": "subtract(1, 'week').endOf('week')"
},
{
"label": "Current week",
"start": "startOf('week')",
"end": "endOf('day')"
}
]
}
```
:::warning
Dates are defined with the [momentjs library](https://momentjs.com/) (see below for more information).
:::
| field | description |
| :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `reference` | field to `filter` when linked to a chart (`range_` **must** be added the field name) |
| `format` | _(optionnal)_ Display format inside the datepicker. If not given, will use the one in `./config/formatter.json`. |
| `startAt` | _(optionnal)_ Default start date of the datepicker |
| `endAt` | _(optionnal)_ Default end date for the datepicker |
| `minDate` | _(optionnal)_ Minimum date selectable in the datepicker |
| `maxDate` | _(optionnal)_ Maximum date selectable in the datepicker |
| `request` | _(optionnal)_ dynamically get `min` and `max` dates from underlying data |
| `ranges` | Allows you to display selectable ranges directly via a shortcut bar. With parameters `start` and `end` to select the beginning and end of the range proposed, as well as `label` to be displayed. |
***
## Configuring minimum and maximum dates available
The following rules explains how to control the datepicker edges :
- in case a `request` is provided
- `datepicker` will look for the first field in the query, looking for `min` and `max` compute mode
- if only `max` is given, `min` takes same value as `max`
- if `request` is not set :
- `max` is set to `now()`
- `min` is set to now()
- `maxDate`, `startAt`, `endAt` are defined relative to `max`
- `minDate` is defined relative to `min`
- if `minDate` is not provided, it will be the beginning of previous year relative to `max`
- in all case, you may specifiy a precise date by `set({'year': 2020, 'month': 0, 'date': 1})`
> _To wrap it up_, to define selectable dates :
- by default it goes :
- from beginning of previous year
- until today
- you may define them using `minDate` and `maxDate` properties relative from today
- you may define a `request` to select `min` and/or `max` date
:::warning
When getting a `min` from `request`, `minDate` is relative to `min` and not anymore from `max`
:::
***
## Adding it in another React component
Example:
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
import moment from 'moment'
const myDynP = {
id: 'datepicker',
type: 'filter',
component: 'datepicker',
reference: 'range_date',
startAt: 'subtract(1,"year")',
endAt: 'endOf("day")',
minDate: moment().subtract(3, 'year'),
maxDate: moment().endOf('day'),
format: 'YYYY-MM-DD',
request: {
data: {
fields: {
date: ['max']
}
}
},
ranges: [{
start: moment().subtract(1, 'week').startOf('week'),
end: moment().subtract(1, 'week').endOf('week'),
label: 'Past week'
}, {
start: moment().startOf('week'),
end: moment().endOf('day'),
label: 'Current week'
}]
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
***
## moment.js Library
The datepicker uses the library [momentjs](https://momentjs.com/).
This allows you to easily select a desired period.
In code you can directly supply an object in momentjs format for each parameter.
However, in JSON format, momentjs is used through a "string" format.
For instance:
```js
moment().subtract(1, 'week').endOf('week')
```
becomes
```json
"subtract(1, 'week').endOf('week')"
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-daypicker.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Daypicker
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The Daypicker allows you to select a date to filter the data of a graph.
> Please refer to [datepicker](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-datepicker.md) for a full reference.
> The only difference is value of `component`.
## Configuration
```json
{
"id": "daypicker",
"type": "filter",
"component": "datepicker.day",
"reference": "range_date",
"startAt": "startOf('day')",
"endAt": "endOf('day')",
"minDate": "subtract(3, 'year').startOf('month')",
"maxDate": "endOf('day')",
"disable_reference_date": false,
"request": {
"data": {
"fields": {
"date": ["max"]
}
}
},
"diamonds": null
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-monthpicker.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Monthpicker
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The monthpicker allows you to select a month to filter data from a chart
> Please refer to [datepicker](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-datepicker.md) for a full reference. The only difference is value of `component`.
## Configuration
```json
{
"id": "monthpicker",
"type": "filter",
"component": "datepicker.month",
"reference": "range_date",
"startAt": "startOf('month')",
"endAt": "endOf('day')",
"minDate": "subtract(3, 'year').startOf('month')",
"maxDate": "endOf('day')",
"disable_reference_date": false,
"request": {
"data": {
"fields": {
"date": ["max"]
}
}
},
"diamonds": null
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-monthspicker.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Monthspicker
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The Monthspicker allows you to select **a range of months** to filter the data in a chart.
> Please refer to [datepicker](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-datepicker.md) for a full reference. The only difference is value of `component`.
## Configuration
```json
{
"id": "monthspicker",
"type": "filter",
"component": "datepicker.months",
"reference": "range_date",
"startAt": "startOf('month')",
"endAt": "endOf('day')",
"minDate": "subtract(3, 'year').startOf('month')",
"maxDate": "endOf('day')",
"disable_reference_date": false,
"request": {
"data": {
"fields": {
"date": ["max"]
}
}
},
"diamonds": null,
"ranges": [{
"label": "Past month",
"start": "subtract(1, 'month').startOf('month')",
"end": "subtract(1, 'month').endOf('month')"
}, {
"label": "Current month",
"start": "startOf('month')",
"end": "endOf('day')"
}]
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-radio.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Radio
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The radio buttons allow you to choose **one** item from a defined list.
> Please refer to [checkbox](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-checkbox.md) for a full reference.\
> The differences are :
- `component` : **radio**
- `default` : that is a `string` and not an `array`
```json
{
"id": "year",
"type": "filter",
"component": "radio",
"reference": "year",
"dictionary": "year",
"default": "2020",
"items": [
{
"value": "2020",
"label": "Year 2020"
},
{
"value": "2019",
"label": "Year 2019"
},
{
"value": "2018",
"label": "Year 2018"
}
]
}
```
- **items** : Allows to manually define the elements to display, the label will be displayed on the screen, the value will be displayed in the associated queries.
- **dictionary** : load items automatically and dynamically from your DataPlant.
- **default** : When you first see your radio buttons, this value will be selected.
- **reference** : Specifies the name of the field to modify in the queries in the case of a transform of type "filter".
## Code example
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
import moment from 'moment'
const myDynP = {
id: 'city',
type: 'filter',
component: 'radio',
reference: 'city',
dictionary: 'city'
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-selectbox.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Selectbox
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Selectboxes allow you to choose one or more items (depending on the desired configuration) from a defined list.
```json
{
"id": "the-year",
"type": "filter",
"component": "selectbox",
"customclass": "my-selectbox"
"reference": "year",
"placeholder": "Select a year",
"sortBy": "value",
"notEmpty": true,
"multi": false,
"default": [
"2019"
],
"dictionary": "year",
"tableName": null,
"order": null,
"parents": [],
"items": [
{
"value": "2020",
"label": "Year 2020"
},
{
"value": "2019",
"label": "Year 2019"
},
{
"value": "2018",
"label": "Year 2018"
}
]
}
```
| field | description |
| :------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------- |
| `customclass` | Adds a custom CSS class around the selectbox. |
| `reference` | Specifies the name of the field to modify in the queries in the case of a transform of type `filter`. |
| `placeholder` | Sets the message displayed when no item is selected. |
| `sortBy` | Allows you to order your results by value or label (possible value `none`, `label`, `value`, `-label`, `-value`). |
| `notEmpty` | On the first load, if notEmpty is used and no default value is set, the first element will automatically be selected. |
| `multi` | If this value is used, several values can be selected in the selectbox. |
| `default` | When you first display your selectbox, it will take this value. |
| `dictionary` | Load items automatically and dynamically from your DataPlant. See below for further details. |
| `tableName` | (Optional) When using `dictionary` it forces the Analytics Manager to use a given table. This is not the recommended usage. |
| `order` | (Optional) Only when `type` is `scale`, see below for further details |
| `parents` | Array to list the `id` of Dynamic Parameters that will filter the contents of this one |
| `items` | allows to manually define the elements to display in the selectbox, the label will be displayed on the screen, the value will be sent in the query. |
### dictionary
The `dictionary` attribute may be:
- a `string` that refers to a **field name**
```json
"dictionary": "year"
```
- an `object` with `id` for the **field name** and a `filter` to specify some criteria
```json
{
...
"dictionary": {
"id": "year",
"filter": {
"year": {
"gte": ["2015"]
}
}
}
...
}
```
Then, the Analytics Manager will request all values for this **field** requesting it from the table with that field and minimum number of rows.
If the **field name** is a primary key (i.e. **station\_id**) and has been associated in the Lakehouse Manager to an **Object's Label**, the **Analytics Manager** will automatically retrieve _label_ to display and _value_ to pass as _reference_ when filtering _charts_.
### order
When dynamic parameter `type` is `scale`, the `value` will be added to `scale.fields` request.
The `order` is an object that details for each `value` how it should be ordered.
For instance:
```json
{
"type": "dynamic-parameter",
"dynamic-parameter": {
"type": "scale",
"id": "select_scale",
"component": "selectbox",
"default": "yearmonth",
"order": {
"yearmonth": "asc",
"date": "asc"
},
"items": [
{
"label": "by month",
"value": "yearmonth"
},
{
"label": "by date",
"value": "date"
}
]
},
"label": "Scales"
}
```
## Code example
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
import moment from 'moment'
const myDynP = {
id: 'city',
type: 'filter',
component: 'selectbox',
reference: 'city',
dictionary: 'city'
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-list-toggle.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Toggle
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Toggle allows to return a Boolean.
```json
{
"id": "my-toggle",
"type": "filter",
"component": "toggle",
"default": false
}
```
| field | description |
| :-------- | :----------------------------------------------------------------------------------------- |
| `default` | When you first display your toggle, it will be initialized with this value (true / false). |
## Code example
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
import moment from 'moment'
const myDynP = {
id: 'my-toggle',
type: 'filter',
component: 'toggle'
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-formatter.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Formatters
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Formatters are used to format the data of your application so they display correctly, with units and separators for example.
This configuration will go into the `./config/formatter.json` configuration file loaded from `./config/global.json`.
***
## Formatters list
### `separator`
**Description**: Allows you to choose the separators of thousands and decimals.
| Option | Description | Type | Values |
| :--------- | :----------------- | :------- | :----- |
| `thousand` | Thousand separator | _string_ | _-_ |
| `decimal` | Decimal separator | _string_ | _-_ |
**Example**:
```json
{
"separator": {
"thousand": ",",
"decimal": "."
}
}
```
### `measures`
**Description**: Allows you to format a number with a suffix.
| Option | Description | Type | Values |
| :------ | :----------------------------------------------------------------------------------------------------------- | :------- | :----- |
| `unit ` | Unit suffixed to the value of the data | _string_ | - |
| `round` | The rounding of the data (ignored if `short` is not empty) | _number_ | - |
| `mult` | Multiplication of the data by the provided value | _number_ | - |
| `pows` | List of unit prefixes to applied at every threshold (1, 103, 106, 109, ...) | _array_ | - |
| `short` | Number of digits displayed, will be used by "pows" to determine which prefix to apply | _number_ | - |
**Example**:
In the following example, **turnover** is the data to be formatted. The configuration can be read as follows:
> The unit displayed will be €; rounded to 2 digits and multiplied by 1000.\
> When the value reaches **10**, the value displayed will be **10 k€**.
```json
{
"measures": {
"turnover": {
"unit": "€",
"round": 2,
"mult": 1000,
"pows": [
" ",
" k",
" M",
" G"
],
"short": 3
}
}
}
```
:::warning
Measure format will be **automatically applied by all Charts** on field **turnover** included in an Analytics Manager request.\
You may also use it through code in your own components :
:::
```jsx
import { FpMeasure } from 'forepaas/formatter'
function formatValue(field, value) {
let measure = new FpMeasure(field);
return measure.setValue(value).toString();
}
console.log(formatValue("turnover", 100));
```
### `dimensions`
**Description**: Allows you to indicate the nature of the data used for axes.
| Option | Description | Type | Values |
| :-------- | :------------------ | :--------------- | :-------------------------------------------------------------------------------------------------------------------------------- |
| `type ` | Data type | _null_; _string_ | - _null_ - _temporal_ |
| `subtype` | Data subtype | _string_ | - _year_ - _month_ - _quarter_ - _semester_ - _date_ - _datetime_ - _month-year_ - _week-year_ |
| `format` | Formatting to apply | _string_ | Multiples |
**Example:**
```json
{
"dimensions": {
"sample_year": {
"type": "temporal",
"subtype": "year",
"format": "YYYY"
},
"sample_month": {
"type": "temporal",
"subtype": "month",
"format": "MM"
},
"sample_date": {
"type": "temporal",
"subtype": "date",
"format": "DD/MM/YYYY"
}
}
}
```
:::warning
Dimension format will be automatically used by Chart components for `scale` rendering in general and X-axis specifically.
:::
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Charts in Application SDK
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Charts in the context of the ForePaaS SDK gather the main aspects of rendering data in an application context:
- **Observe**: update when variables changes, in ForePaaS it is called _dynamic parameter_.
- **Request**: building a request and getting results.
- **Render**: render those results.
***
## Observe
The chart may be linked to `dynamic-parameters`.
In the example below, the _chart_ will be updated in case of a different date range selected in a _datepicker_ or a different _city_ in a _selectbox_.
The `not-nullable-dynamic-parameter` indicates that it must specify a value for this parameter to render.
```json
{
"chart": {
"dynamic-parameters": [
"datepicker",
"city"
],
"not-nullable-dynamic-parameters": [
""datepicker"
],
"component": "echarts"
}
}
```
:::warning
The `dynamic-parameters` list the `id` of previously created `dynamic-parameter`
:::
There are no other settings to be done.
This chart will _observe_ all changes done in _dynamic parameters_ and will automatically make new requests and render as needed.
## Request
The `request` uses the same format as the [Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md).
```json
{
"chart": {
"component": "echarts",
"request": {
"data": {
"fields": {
"quantite": [
"sum"
]
}
},
"scale": {
"fields": [
"year",
"month",
"city"
],
"axis": {
"x": [
"year",
"month"
],
"y": [
"city"
]
}
},
"evol": {
"scale": "year"
}
}
}
}
```
## Render
Here, parameters of interest are `component` and `options`.
```json
"chart": {
"component": "echarts",
"options": {
"title": {
"text": "Nombre de visites mensuelles"
}
},
"request": {},
"dynamic-parameters": [],
"not-nullable-dynamic-parameters": []
}
```
- `component`: the Charts module will look for a registered component, in this case _echarts_, and will pass over all informations needed to render itself. You can also
create your own [Custom Chart Component](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md).
- `options`: additionnal parameters, specific to each component, that are passed to this component to customize behaviour and rendering.
## Putting it together
```json
{
"component": "echarts",
"dynamic-parameters": [
"datepicker"
],
"options": {
"title": {
"text": "Monthly visitors"
}
},
"request": {
"data": {
"fields": {
"vists": [
"sum"
]
}
},
"scale": {
"fields": [
"year",
"month",
"category"
],
"axis": {
"x": [
"year",
"month"
],
"y": [
"category"
]
}
}
}
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts-chart-html.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Chart HTML
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The Chart HTML allows you to format a query result into an HTML fragment. **This is typically used to display one value such as a KPI**.
***
## Chart HTML configuration example
In this example :
- it declares the `html` component
- a `request` gets the `sum` of `rides` on the period defined by the `datepicker` dynamic parameter, and get the value for the previous year
- `options` describes how the component is rendered
```json
{
"component": "html",
"request": {
"data": {
"fields": {
"rides": ["sum"]
}
},
"evol": {
"scale": "year"
}
},
"dynamic-parameter": [
"datepicker"
],
"options": {
"template-name": "block", // 'block', 'evol' or 'own'
"template": "
{scope.value} rides
", // a custom template
"class": "", // a class name to be used by template
"icon": "", // may be used by templates
"text": "" // may be used by templates
}
}
```
***
## Default templates
The following templates are already defined and may be set as `template-name`.
### block
```html
{scope.value||0}{scope.options.text}
```
### evol
```html
{scope.value||0}{scope.evol}%{scope.options.text}
```
***
## Custom Template
It is possible to define a custom HTML fragment using `options.template`.
In that fragment, everything between `{}` will be interpreted as Javascript with a `scope` variable defined.
### The scope variable
The `scope` variable is an object with the following fields :
- `value`: the first field of the result, formatted with defined rules
- `evol`: whether `evol` is defined in the request, the evolution in percentage
- `evolClass`: whether `evol` is defined in the request, an helper for styling, either `evol positive` or `evol negative`
- `options`: the complete `chart.options` to be used in templates
- `query_params`: the complete query executed by the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) with expanded Dynamic parameters
- `response`: the complete response from the Analytics Manager with all results
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts-chart-table.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Chart Table
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
## Setting up
- from App Manager, using [Extensions](https://docs.dataplatform.ovh.net/app-services-apps-extensions.md)
- using ForePaaS Package Manager on your local host: `fppm install chart-table`
***
## Table JSON configuration
Example:
```json
{
"component": "table",
"options": {
"i18n": {
"fr": {
"sum": "",
"visits": "",
"evol-year-2": "(Y-2)"
}
},
"searcheable": true,
"theme": "standard",
"headerHeight": 40,
"rowHeight": 40,
"padding": 5,
"columns": [
{
"type": "scale",
"scale": "date",
"align": "right",
"sortable": false
},
{
"type": "data",
"data": "visits",
"computeMode": "sum",
"evol": 0
}
]
},
"request": {
"data": {
"fields": {
"visit": [
"sum"
]
}
},
"scale": {
"fields": [
"city",
"date"
],
"axis": {
"x": [
"city"
],
"y": [
"date"
]
}
}
}
}
```
## request
The `chart-table` component takes `axis` into account:
- `axis.y`: allows display in _rows_
- `axis.x`: creates as many _columns_ as found values
## options
- `i18n`: using same format as global `i18n`, it allows to specify values for this table only.
- `searcheable`: `true` / `false` – Allows to activate or not a sear bar.
- `theme`: `standard` is default theme, it is possible to customize it (see below).
- `headerHeight`: allows to define in pixel the height of the header
- `rowHeight`: define rows height in pixels
- `padding`: define padding inside table cells
- `columns`: allows for precise configuration of columns.
If not provided, it will be created from the request.
## options.columns
An array of JSON object specifying each column :
- `type`: `scale` / `data` - set the type of column to display
- `"type":"scale"`
- `scale`: define name of the scale to use
- `"type":"data"`
- `data`: name of the `data.fields` to be specified
- `computeMode`: specify which computeMode (e.g. `select`, `sum`, `count`) present for that field.
- `evol`: Specify which evolution to use
- `align`: `left` / `center` / `right`
- `sortable`: `true` / `false`
## theme
It is possible to customize entirely theme used to display the Chart Table.
In `options`, specify for instance :
```json
"theme": "mytheme"
```
From this example, as a copy of the `standard` theme, it allows to entirely define a customized theme:
```css
.fp-table.fp-table-mytheme {
height:100%;
position: relative;
.fp-row {
.fp-cell {
display: inline-block;
white-space: nowrap;
overflow: hidden;
text-overflow:ellipsis;
position: relative;
i {
position: absolute;
right:5px;
top:50%;
background:white;
transform:translateY(-50%);
z-index: 2;
}
}
}
.fp-row:nth-child(even) {
background:rgba(0, 0, 0, .02);
border-bottom:1px solid #e5e8eb;
}
.fp-head {
position:absolute;
top:0px;
left:0;
right:0;
.fp-cell-is-sortable {
cursor:pointer;
}
.fp-row {
height:100%;
border-bottom:2px solid #eee;
}
}
.fp-body {
position:absolute;
bottom:0;
left:0;
right:0;
overflow: auto;
.fp-cell-sort {
background:rgba(0,0,0,0.03);
}
}
.fp-content {
position: absolute;
top:0;
left:0;
right:0;
bottom:0;
background:white;
border-radius:3px;
}
}
.fp-table.fp-table-search.fp-table-mytheme {
.fp-content {
top:40px;
}
.fp-search {
z-index:2;
position: absolute;
right:0;
height:40px;
top:0px;
width:150px;
border:#ddd solid;
border-width: 1px 1px 1px 1px;
background:white;
i {
position: absolute;
z-index: 2;
left:3px;
top:3px;
font-size:25px;
}
input {
position: absolute;
width:100%;
left:0;
right:0;
border:0;
outline:none;
padding-left:30px;
top:0;
bottom:0;
}
}
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Recharts
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
[ReCharts](https://recharts.org/en-US/) is a graphic library for ReactJS.\
It is possible to easily add and configure ReCharts in your application.
- [Recharts configuration](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#recharts-configuration)
- [Platform additions](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#platform-additions)
- [fpFormat](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#fpformat)
- [colors](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#colors)
- [i18n](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#i18n)
- [series](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#series)
- [tickStyle](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#tickstyle)
- [tooltip](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#tooltip)
- [Examples](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#examples)
- [Sample 1: Bar Chart](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#sample-1-bar-chart)
- [Sample 2: Bar/Line Chart](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#sample-2-barline-chart)
- [Sample 3: Pie Chart](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#sample-3-pie-chart)
- [Streamline using templates](https://docs.dataplatform.ovh.net/sdk/app-charts-chart-recharts.md#streamline-using-templates)
***
## Recharts Configuration
When configuring a ReCharts with JSON, 2 kind of attributes are available:
- **Natives**: all existing [ReCharts Options](http://recharts.org/en-US/api)
- **Platform**: offers further customization and integration with the platform features
The Recharts `data` field is automatically filled from the results of the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) `request` provided in the Chart definition.
***
## Platform additions
### fpFormat
Use `fpFormat`, to customize what is displayed on an axis.
For instance :
```json
"options": {
"yAxis": [
{
"fpFormat": {
"round": 2,
"unit": "%"
}
}
]
}
```
It uses the same definition as [measure's formatter](https://docs.dataplatform.ovh.net/sdk/app-formatter.md#measures).
It is also possible to re-use an already defined formatter by just specifying its name.
```json
"options": {
"yAxis": [
{
"fpFormat": "pct"
}
]
}
```
### colors
Use `colors` to provide an array of colors to be used by the chart.
```json
"options": {
"colors": ["#FF6F90", "#FACD30", "#9EDF10"]
}
```
### i18n
Use `i18n` to provide translations dedicated to that chart.\
Those translations are used to create **labels** in **legends** and **tooltips** built upon [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) `request`:
- `field` name used in `data.fields` and `scale.fields`
- `compute mode` used in `data.fields` such as `select`, `sum`, `count`, `avg`...
Code below replace _sum_ by _Sum of_ and _rides_ by _Rides_ inside chart tooltips for engligh language.
```json
"options": {
"i18n": {
"en": {
"sum": "Sum of",
"rides": "Rides"
}
}
}
```
### series
Use `series` to define options to be applied to data series from the `request`.
It is either an `Array of Object` or an `Object` (keys are fields definition and values are options).
Main options in `series` are :
| Name | Type | Description |
| :------ | :----: | :---------------------------------------------- |
| `_type` | string | Diplay to be used (i.e. bar, line, area, etc.) |
| `order` | number | Order to be used in **Legend** and **Tooltips** |
In the following example, `yAxis` should be defined as described in [Recharts yAxis](https://recharts.org/en-US/api/YAxis) asn an _Array_ with 2 _Objects_, each of them define an `yAxisId` used below.
```json
"options": {
"series": {
"cumm_progress_actual_select_0": {
"stroke": "rgb(97, 125, 233)",
"color": "rgb(97, 125, 233))",
"name": "Actual (Cum.)",
"yAxisId": "right",
"order": 1,
"label": true
},
"cumm_progress_planned_select_0": {
"color": "rgba(62,69,80,0.15)",
"stroke": "rgba(62,69,80,0.15)",
"name": "Plan (Cum.)",
"yAxisId": "right",
"order": 0,
"label": true
},
"incr_progress_actual_select_0": {
"fill": "rgb(97, 125, 233)",
"color": "rgb(97, 125, 233)",
"_type": "bar",
"name": "Actual (Inc.)",
"yAxisId": "left",
"maxBarSize": 6,
"radius": [5, 5, 5, 5],
"order": 3
},
"incr_progress_planned_select_0": {
"fill": "rgba(62,69,80,0.15)",
"color": "rgba(62,69,80,0.15)",
"_type": "bar",
"name": "Plan (Inc.)",
"yAxisId": "left",
"maxBarSize": 6,
"radius": [5, 5, 5, 5],
"order": 2
}
}
}
```
### tickStyle
Use `tickStyle` inside of [xAxis](https://recharts.org/en-US/api/XAxis) or [yAxis](https://recharts.org/en-US/api/yAxis) to define style applied to **tick** (i.e. label at axis levels).
Main options in `tickStyle` are :
| Name | Type | Description |
| :----------- | :----: | :---------------------------------------------------- |
| `x` | number | Horizontal offset |
| `y` | number | Vertical offset |
| `textAnchor` | string | From where it is displayed (`start`, `middle`, `end`) |
| `fontSize` | number | Font size |
| `fill` | string | Text color |
```json
"yAxis": [
{
"yAxisId": "left",
"tickStyle": {
"x": -50,
"y": -10,
"textAnchor": "start",
"fontSize": 11
}
}
]
```
### tooltip
Use `tooltip` to customize tooltip styles.\
This configuration might be defined once for all in `./config/templates.json`.
Main options in `tooltip` are :
| Name | Type | Description |
| :-------------- | :----: | :------------------------------- |
| `style` | object | tooltip box |
| `style-label` | object | tooltip title |
| `style-content` | object | container around labels and data |
| `style-name` | object | fields name |
| `style-value` | object | values |
```json
"tooltip": {
"style": {
"padding": "10px 20px",
"minWidth": "200px",
"backgroundColor": "white",
"borderRadius": 10,
"boxShadow": "0 10px 30px 0 rgba(151,167,183,0.3)",
"textTransform": "capitalize"
},
"style-label": {
"fontSize": 13
},
"style-content": {
"display": "flex",
"justifyContent": "space-between"
},
"style-name": {
"fontSize": 13
},
"style-value": {
"fontSize": 13,
"fontWeight": 600
}
}
```
***
## Examples
### Sample 1 (Bar Chart)
Here is a simple bar chart displaying the total number of rides per week day, with a custom bar color and styled x-axis ticks:
```json
{
"type": "chart",
"chart": {
"component": "recharts.bar",
"request": {
"data": {
"fields": {
"rides": ["sum"]
}
},
"order": {
"week_day": "asc"
},
"scale": {
"fields": ["week_day"]
},
"dictionaries": ["week_day"]
},
"dynamic-parameters": ["filter_date"],
"options": {
"barSize": 12,
"xAxis": {
"tickLine": false,
"tickStyle": {
"fontSize": 11
}
},
"legend": {
"verticalAlign": "top",
"align": "right"
},
"series": {
"rides_sum": {
"name": "Rides",
"fill": "rgb(97, 125, 233)",
"color": "rgb(97, 125, 233)",
"radius": [5, 5, 0, 0]
}
}
}
}
}
```
The request sums the `rides` measure, scaled by `week_day`, and the `series` block styles the resulting bars. Any other option from the [Recharts Bar API](https://recharts.org/en-US/api/Bar) can be added to the series object in the same way.
### Sample 2 (Bar/Line Chart)
```json
"options": {
"barGap": 3,
"barSize": 6,
"i18n": {
"en": {
"select": ""
}
},
"margin": {
"left": 50,
"right": 60,
"top": 40
},
"yAxis": [
{
"yAxisId": "left",
"domain": ["auto", "dataMax => (dataMax * 3)"],
"fpFormat": {
"round": 2,
"unit": "%"
},
"tickLine": false,
"axisLine": false,
"mirror": true,
"width": 200,
"tickStyle": {
"x": -50,
"y": -10,
"textAnchor": "start",
"fontSize": 11
}
},
{
"yAxisId": "right",
"orientation": "right",
"fpFormat": {
"round": 2,
"unit": "%"
},
"tickLine": false,
"axisLine": false,
"mirror": true,
"tickStyle": {
"x": 30,
"y": -10,
"textAnchor": "start",
"fontSize": 11
}
}
],
"legend": {
"verticalAlign": "top",
"align": "right",
"height": 60,
"iconSize": 9
},
"series": {
"cumm_progress_actual_select_0": {
"stroke": "rgb(97, 125, 233)",
"color": "rgb(97, 125, 233)",
"name": "Actual (Cum.)",
"yAxisId": "right",
"order": 1,
"label": true
},
"cumm_progress_planned_select_0": {
"yAxisId": "right",
"order": 0,
"name": "Plan (Cum.)",
"label": true,
"color": "rgba(62,69,80,0.15)",
"stroke": "rgba(62,69,80,0.15)"
},
"incr_progress_actual_select_0": {
"_type": "bar",
"yAxisId": "left",
"order": 3,
"name": "Actual (Inc.)",
"fill": "rgb(97, 125, 233)",
"color": "rgb(97, 125, 233)",
"maxBarSize": 6,
"radius": [5, 5, 5, 5]
},
"incr_progress_planned_select_0": {
"_type": "bar",
"yAxisId": "left",
"order": 2,
"name": "Plan (Inc.)",
"fill": "rgba(62,69,80,0.15)",
"color": "rgba(62,69,80,0.15)",
"maxBarSize": 6,
"radius": [5, 5, 5, 5]
}
}
}
```
### Sample 3 (Pie Chart)
Here is a simple pie chart definition:
```json
{
"type": "chart",
"chart": {
"component": "recharts.pie",
"request": {
"data": {
"fields": {
"rides": ["sum"]
}
},
"order": {
"week_day": "asc"
},
"scale": {
"fields": ["week_day"]
}
},
"dynamic-parameters": ["filter_date"]
}
}
```
That displays:
By doing some additions, we will improve this layout:
- adding `chart.request.dictionaries` to translate `week_day` using label defined in _Lakehouse Manager_
- using \`chart.options.series\* to define some [Recharts Pie configuration](https://recharts.org/en-US/api/Pie)
- using some additional platform options:
- `radiusRatio` in a `serue` object: to control at which distance labels are drawn (default is 2.5)
- `chart.options.labels` set to `false` will hide labels
- whether `chart.options.labels.minPercent` is set, only values over will be displayed
```json
{
"type": "chart",
"chart": {
"component": "recharts.pie",
"request": {
"data": {
"fields": {
"rides": ["sum"]
}
},
"order": {
"week_day": "asc"
},
"scale": {
"fields": ["week_day"]
},
"dictionaries": ["week_day"]
},
"dynamic-parameters": ["filter_date"],
"options": {
"series": {
"interval": 0,
"label": true,
"labelLine": true,
"innerRadius": 10,
"outerRadius": 60,
"radiusRatio": 1.5,
"endAngle": 360,
"paddingAngle": 0,
"cx": "50%",
"cy": "50%"
}
}
}
}
```
:::info
The former Query Builder evol feature, which produced automatic time-period comparisons such as a year-over-year dual pie, is not available in the Analytics Manager. To compare two periods, use two chart components, each with its own request.
:::
***
## Streamline using templates
It is highty advised to use [Templates](https://docs.dataplatform.ovh.net/sdk/app-charts-template.md) to define **Chart styles** rather than copying same definition over and over.
To do so, just add a `recharts` object in `./config/templates.json` such as for example :
```json
{
"recharts": {
"cartesianGrid": {
"stroke": "#f5f5f5",
"vertical": false
},
"tooltip": {
"style": {
"padding": "10px 20px",
"minWidth": "200px",
"backgroundColor": "white",
"borderRadius": 10,
"boxShadow": "0 10px 30px 0 rgba(151,167,183,0.3)",
"textTransform": "capitalize"
},
"style-label": {
"fontSize": 13
},
"style-content": {
"display": "flex",
"justifyContent": "space-between"
},
"style-name": {
"fontSize": 13
},
"style-value": {
"fontSize": 13,
"fontWeight": 600
}
},
"xAxis": {
"axisLine": false,
"tickLine": false
},
"yAxis": {
"tickLine": false,
"axisLine": false,
"mirror": true,
"tickStyle": {
"x": -8,
"y": -10,
"textAnchor": "start"
}
}
}
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts-jvectormap.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# JVectorMap
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Jvectormap is a library that allows you to use maps on your dashboards.
In our standard implementation, we currently provide allow displaying a map, and with a scale on city name, display a number by city.
Official library documentation : [https://jvectormap.com/](https://jvectormap.com/)
:::info
If you do not understand the objects referred to in this article, please check the [Charts article](https://docs.dataplatform.ovh.net/sdk/app-charts.md) for more details.
:::
- [Sample code in JSON](https://docs.dataplatform.ovh.net/sdk/app-charts-jvectormap.md#sample-code-in-json)
- [Request](https://docs.dataplatform.ovh.net/sdk/app-charts-jvectormap.md#request)
- [Response](https://docs.dataplatform.ovh.net/sdk/app-charts-jvectormap.md#response)
- [Options](https://docs.dataplatform.ovh.net/sdk/app-charts-jvectormap.md#options)
- [Going further](https://docs.dataplatform.ovh.net/sdk/app-charts-jvectormap.md#going-further)
***
## Sample code in JSON
The sample code below should be pasted in an Application dashboard to use the JVectorMap library.
```json
{
"type": "chart",
"col": 0,
"row": 1,
"sizeX": 12,
"sizeY": 20,
"title": "",
"chart": {
"dynamic-parameters": [
"datepicker"
],
"component": "jvectormap",
"options": {
"map": "fr_regions_mill"
},
"request": {
"data": {
"fields": {
"qte": [
"sum"
],
"ca": [
"sum"
]
}
},
"scale": {
"fields": [
"city"
]
}
}
}
}
```
***
## Request
There are two conditions that need to be respected in order to have a correct JVectorMap request.
### First condition
The request must contain at least one data, it can be multiple like in the previous sample
```json
"data": {
"fields": {
"qte": [
"sum"
],
"ca": [
"sum"
]
}
},
```
That data will appear on each point of the map (marker)
### Second condition
The request must contain at least one scale, it will be use to compute marker
The value generated by the scale will be send through the google geocoding api : [https://www.npmjs.com/package/geocoding](https://www.npmjs.com/package/geocoding)
If the value generate valid lat, lng coordinates, the marker will apear with the data associated
```json
"scale": {
"fields": [
"city"
]
}
```
***
## Response
A correct answer with my previous example look like that
```json
{
"scales": {
"city": "Bordeaux"
},
"data": {
"qte": {
"sum": [
{
"city": "Bordeaux",
"value": 18073
}
]
},
"ca": {
"sum": [
{
"city": "Bordeaux",
"value": 190987.45
}
]
}
}
},
{ .....
```
That answer will display a marker on Bordeaux (looking the position of the city with google geocoding) and with 2 value : sum qte, and sum ca
***
## Options
options.map : Allow the user to select a map
List of all maps
| Name | JSON Value |
| :-----------------------: | :---------------------: |
| Worlds, Continents | continents\_mill |
| World, Country | world\_mill |
| World, Country | world\_mill |
| Africa, Country | africa\_mill |
| Asia, Country | asia\_mill |
| Europe, Country | europe\_mill |
| North America, Country | north\_america\_mill |
| Oceania, Country | oceania\_mill |
| South America, Country | south\_america\_mill |
| Argentina | ar\_mill |
| Australia | au\_mill |
| Austria | at\_mill |
| Belgium | be\_mill |
| Canada | ca\_lcc |
| China | cn\_mill |
| Colombia | co\_mill |
| Denmark | dk\_mill |
| France, Departments | fr\_mill |
| France, Regions | fr\_regions\_mill |
| France, Regions 2016 | fr\_regions\_2016\_mill |
| Germany | de\_mill |
| India | in\_mill |
| Italy, Provinces | it\_mill |
| Italy, Regions | it\_regions\_mill |
| Netherlands | nl\_mill |
| New Zeland | nz\_mill |
| Norway | no\_mill |
| Poland | pl\_mill |
| Portugal | pt\_mill |
| Russia, Federal districts | ru\_fd\_mill |
| Russia | ru\_mill |
| South Africa | za\_mill |
| South Korea | kr\_mill |
| Spain | es\_mill |
| Sweden | se\_mill |
| Switzerland | ch\_mill |
| Thailand | th\_mill |
| United Kingdom, Regions | uk\_regions\_mill |
| United Kingdom | uk\_countries\_mill |
| USA | us\_aea |
| Venezuela | ve\_mill |
| Venezuela | ve\_mill |
| USA, Chicago | us-il-chicago\_mill |
| USA, New York City | us-ny-newyork\_mill |
***
## Going further
Of course that library give you a sample of usage.
But jvectormap offer much more feature
Looking the code in "forepaas/jvectormap/FpJvectormap.jsx" and the documentation to create your own chart component you can customize it.
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts-template.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Chart templates
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Style templates can be applied to your favorite chart libraries **to automatically setup the configuration for every chart you want to look the same**. You can use style templates in a simple way, either from _Architect_ (ForePaaS' dashboard creation tools) or locally on your machine.
***
## Why use style templates?
Here is a chart configuration structure. Please [see this documentation page for more details on charts](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md).
```json
{
"type": "chart",
"col": 0,
"row": 1,
"sizeX": 12,
"sizeY": 12,
"title": "",
"chart": {
"component": "my-component",
"options": {},
"request": {}
}
}
```
You can pass props to your charts using the `options` object.\
For example, if you want to define a bar chart with Recharts, with a `barGap` of 3, a` barSize` of 6 and a specific margin, you can pass the following options:
```json
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
```
Things can get complicated if you want all your bar charts to have the same configuration controllable outside of the individual chart widgets, without having to repeat the setup for every chart.
ForePaaS offers a simple solution to do that: **create style templates for your configurations**.
***
## Step 1: Create Templates
Everything will happen in the `config/templates.json` file. This is where you define all your templates.
A template is defined like this:
```json
"templateName": {
"component": "", // Name of the component where the options will be applied
"options": {
...yourOptions
}
}
```
To use the example above, this would translate as follows:
```json
"rechartsBar": {
"component": "recharts.bar",
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
}
```
### Defining the template name
The `templateName` can be used in 2 ways:
#### 1. _Using the name of an existing component_
The options will apply to all components of this type.
In this case, the `component` property is omitted, otherwise the options will not be applied. This works for either a component created by ForePaas or a spec component [(see documentation)](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md).
**Example**: these options will be applied by default as soon as you use a `recharts.bar` chart
```json
"recharts.bar": {
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
}
```
#### 2. _Defining a template name_
The options will apply as soon as you use this template in the definition of your chart (more details in _Step 2: Using templates_).
> Component names are especially used when you want to define a theme for your application, while template names will be used in more specific cases (for example to make a modification to the theme on a particular chart).
### Creating a second template based on a first template
ForePaaS provides a simple way to create templates that not only leverage most of the options from an initial one and but also receive modifications done to the latter automatically.
Simply add the `extends` element to your template:
```json
"rechartsBarWithBiggerSize": {
"extends": "rechartsBar",
"options": {
"barSize": 10,
"barCategoryGap": 5
}
}
```
?>[🌠 Learn more about the extends element here!](https://docs.dataplatform.ovh.net/sdk/app-charts-template-extends.md)
***
## Step 2: Using Templates
To use your templates, you have 2 possibilities:
- Architect (via the platform)
- Locally (via config .json files)
### Via Architect
You will find the templates you previously created while selecting the type of chart in the category 'Custom'.
You just have to select it from the chart, indicate the request and validate it at the top right.
### Locally
Whether locally or via the platform editor, **you just need to enter the name of your template** as the value of the `component` of your chart:
If you wish to modify one of the options, you can modify them in the `option` object. These will add and / or replace the options defined in the template (exactly as the `extend` seen above).
## Bonus: Addition of configurable options via Architect
These options are those that you can configure from _Architect_, the chart creation or editing interface.
When you define your charts and options via your code, it is easy to rewrite an option of a template (such as a color for example) by indicating it in the options of the chart.
But it's not that easy if you (or your users) are using _Architect_. Indeed, your new templates are now available from this UI but apart from the charts defined by ForePaaS and their pre-configured options, you cannot modify or rewrite the options defined in your template without going through the json configuration in the advanced mode (which is not the most intuitive for your users).
There are ways to add options in this interface in order to simplify creating or editing a chart for your users.
To sum it up, using a config in .json you will be able to manage a form accessible via the chart creation/edition UI which will add the relative keys & values in the `options` object of your chart.
The type of the option (object, text, number, table, ...) is defined first, then the type of the input.
**Example**:
```json
{
"xAxis": {
"label": "xAxis",
"type": "object",
"options": {
"hide": {
"type": "toggle",
"label": "Hide"
},
"type": {
"label": "Type",
"type": "select-box",
"options": [
{
"label": "number",
"value": "number"
},
{
"label": "category",
"value": "category"
}
]
}
}
}
}
```
Will generate the following object (because the type is `object`) in the `options`:
```json
"xAxis": {
"hide": true / false, // Toggle will give a boolean
"type": "number" / category // according to the value selected in the select-box
}
```
By adding this configuration in the `optionsArchitect` property, you will find these options in the `options` tab from your chart configuration.
Here is an example configuration:
```json
{
"rechartsBar": {
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
},
"component": "recharts.bar"
},
"optionsArchitect": {
"barSize": {
"type": "number",
"label": "Bar Size"
},
"barGap": {
"label": "Bar Gap",
"type": "number"
},
"barCategoryGap": {
"label": "Bar Category Gap",
"type": "number"
}
}
}
```
Which gives the following result:
> **Tip:** Do not hesitate to have a look at how the options of the libraries already defined by the platform are defined: see file `forepaas.json` -> key`options` for more examples.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/app-charts-template-extends.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a second template based on a first template using *extends*
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
The platform provides a simple way to create templates that not only leverage most of the options from an initial one and but also receive modifications done to the latter automatically.
Simply add the `extends` element to your template:
```json
"rechartsBarWithBiggerSize": {
"extends": "rechartsBar",
"options": {
"barSize": 10,
"barCategoryGap": 5
}
}
```
Using `extends`, **the options of your second template will merge with the options of the first**, so when you use the second template, the options will be as follows:
```json
"options": {
"barGap": 3,
"barSize": 10,
"barCategoryGap": 5
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
```
This will also work if you only want to retrieve the options from `rechartsBar` for example, but for another component. Simply specify which component in the following example:
```json
"rechartsBarWithBiggerSize": {
"component": "recharts.line",
"extends": "rechartsBar",
"options": {
"barSize": 10,
"barCategoryGap": 5
}
}
```
The merged options above will apply to recharts.line as soon as you use the `rechartsBarWithBiggerSize` component.
:::warning
There must be **at least one component** (keyword `component` or a component used as a template) defined if you use the `extends`. The example below will not work because neither `rechartsBar` nor` rechartsBarWithBiggerSize` have any component assigned:
:::
```json
{
"rechartsBar": {
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
},
"rechartsBarWithBiggerSize": {
"extends": "rechartsBar",
"options": {
"barSize": 10,
"barCategoryGap": 5
}
}
}
```
Instead, this is the correct configuration:
```json
{
"rechartsBar": {
"component": "recharts.bar", // define component that will be used here
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
},
"rechartsBarWithBiggerSize": {
"extends": "rechartsBar",
"options": {
"barSize": 10,
"barCategoryGap": 5
}
}
}
```
or
```json
{
"recharts.bar": { // recharts.bar is a component so it will work
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
},
"rechartsBarWithBiggerSize": {
"extends": "recharts.bar",
"options": {
"barSize": 10,
"barCategoryGap": 5
}
}
}
```
> **_Note_** : If you are using a chart family, you don't need to specify the `extends`, by default the children gets the same options as the parent.
For example:
```json
{
"recharts": {
"options": {
"barGap": 3,
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
},
"recharts.bar": {
"options": {
"barGap": 1
}
}
}
```
Here, `recharts.bar` gets the options from `recharts` and modifies the `barGap` property. The options for `recharts.bar` will then be:
```json
{
"recharts.bar": {
"options": {
"barGap": 1, // On garde la définition de recharts.bar
"barSize": 6,
"margin": {
"left": 20,
"right": 20,
"top": 40,
"bottom": 35
}
}
}
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-container-panel.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Panel
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Panels delimit an area on your application where you can group a title, graphics, etc. together. They should be used in order to bring clarity to an application.
## Add a Panel
Select Container -> Panel.
In the following example, the panel (here the white background frame) contains all the graphs related to the sales of the company.
---
url: https://docs.dataplatform.ovh.net/sdk/app-container-tabs.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Tabs
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Tabs represent panels grouped together. Each tab contains a single element (graphic, dynamic parameters, etc.).
## Add Tabs
Select Container > Tabs.
## Add elements
Then add the items by clicking on the +.
## Edit names of tabs
You can edit the tab name at any time by clicking on the small pencil on the right side of the tab.
## Result
---
url: https://docs.dataplatform.ovh.net/sdk/app-container-switch.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Switch
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Switches are used to set conditions on the display of elements (graphics, button, etc.).\
If you add multiple items, only the one whose condition is filled first is displayed.
> **Example:**
- Here, only **the chart** will be displayed :
- 1st case : true - Chart
- 2nd case : true - Button
> - Here, only **the button** will be displayed :
- 1st case : false - Chart
- 2nd case : true - Button
## Adding a Switch
Select Container > Tabs.
## Configuring a Switch
### Simple configuration
This configuration allows you to configure the switch in a simple and intuitive way.
To do this, simply enter the display condition of an element and select the element type.
Then, if you wish, link this switch to a dynamic parameter by selecting it via the drop-down menu. This allows you to use the dynamic parameter data in the conditions.
In the example below, the switch is linked to the _siege\_social_ parameter and the Chart element is displayed if "Acme" is selected as the _siege\_social_
> You can add as many cases as you want.
### Advanced configuration
This configuration allows you to configure the Switch via a JSON document. Below is the JSON corresponding to the Switch created with the simplified configuration:
```json
{
"type": "switch",
"dynamic-parameters": [
"siege_social"
],
"items": [
{
"case": "\"this.siege_social.indexOf('Acme')!==-1\"",
"type": "chart",
"chart": {
"dynamic-parameters": [
"datepicker",
"siege_social"
],
"component": "jvectormap",
"options": {
"map": "fr_regions_mill"
},
"request": {
"data": {
"fields": {
"qte": [
"sum"
],
"ca": [
"sum"
]
}
},
"scale": {
"fields": [
"ville"
]
}
},
"id": "chart-9370143"
}
}
]
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-basic-ui-button.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Button
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Allows you to add a clickable button with triggers an action in the interface of your dashboard.
## Add a Button element
Hover over the section of the dashboard that you willing to edit. Click on the "+" button to get the list of elements you can add, select "Basic UI" and then "Button".
## Configure a Button element
### Simple configuration
The interface allows you to configure the button in a simple and intuitive way. You can easily edit the following parameters for the button:
- **Text**: The button text as displayed on the interface for instance "Open Page" or "Next"
- **On Click**: The action that the click will trigger, this is usually a trigger to open a new dashboard page.
- **Class**: The CSS class to which the button belongs in case you are applying shared attributes such as for styling.
- **Style**: Specific styling attributes to apply to the button like the background or text color (see example below).
### Advanced configuration
Below is the JSON configuration for a **Button** as created above :
```json
{
"type": "button",
"style": {
"backgroundColor": "red",
"alignContent": "middle"
},
"text": "testing",
"onClick": "console.info(\"HelloWorld\")",
"customclass": "test"
}
```
## Result
---
url: https://docs.dataplatform.ovh.net/sdk/app-basic-ui-html.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# HTML
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Allows you to create any element in HTML (paragraph, title, image, etc.).
## Add HTML
Select Basic UI -> Html.
## Configure HTML
### Simple configuration
This configuration allows you to configure the HTML in a simple and intuitive way.
### Advanced configuration
This configuration allows you to configure the HTML via a JSON document. Below is the JSON corresponding to the Switch created with the simplified configuration:
## Result
---
url: https://docs.dataplatform.ovh.net/sdk/app-basic-ui-image.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Image
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Allows you to add an image to your dashboard.
## Add Image
Select Basic UI -> Image.
## Configure Image
### Simple configuration
This configuration allows you to configure the image in a simple and intuitive way.
### Advanced configuration
Below is the equivalent JSON configuration :
```json
{
"type": "image",
"style": {},
"alt": "my image",
"onClick": "console.info('click')",
"width": "50",
"height": "42"
}
```
## Result
---
url: https://docs.dataplatform.ovh.net/sdk/app-basic-ui-link.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Link
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Allows you to add a link to another dashboard.
## Add Link
Select Basic UI -> Link.
## Configure Link
### Simplified configuration
This configuration allows you to configure the link in a simple and intuitive way.
### Advanced configuration
Below is the JSON configuration for an **Image** as created above :
```json
{
"type": "link",
"style": {
"color": "blue"
},
"text": "My link",
"url": "/table",
"icon": "fa fa-table"
}
```
## Result
---
url: https://docs.dataplatform.ovh.net/sdk/app-basic-ui-select-lang.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Select Lang
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Allows you to add a language selector for your dashboard.\
Available languages are defined in `./config/global.json` [at the key `i18n`](https://docs.dataplatform.ovh.net/tutorials-app-development-config.md#global-configuration-file-globaljson).
## Add Select Lang
Select Basic UI -> Select lang.
## Configure Select Lang
### Simplified configuration
This configuration allows you to configure the selector in a simple and intuitive way.
### Advanced configuration
Below is the JSON configuration for **Select Lang** as created above :
```json
{
"type": "select-lang",
"style": {},
"showLabel": false
}
```
## Result
---
url: https://docs.dataplatform.ovh.net/sdk/app-basic-ui-username.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Username
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Allows you to retrieve and display the name of the person logged on the dashboard. You will then be able to display a message such as: "Welcome Mr. Smith".
## Add Username
Select Basic UI -> Username.
## Configure Username
### Simple configuration
This configuration allows you to configure the component in a simple and intuitive way.
### Advanced configuration
This configuration allows you to configure the component via a JSON document. Below is the JSON corresponding to the Switch created with the simplified configuration:
```json
{
"type": "username",
"style": {
"color": "blue"
},
"format": "User is named :"
}
```
---
url: https://docs.dataplatform.ovh.net/sdk/app-custom-component.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create custom components
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Data Platform lets you create your own fully customized components. This is done in two steps. First, you [create a custom component](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md#step-1-create-the-component), then you [add it to your app](https://docs.dataplatform.ovh.net/sdk/app-custom-component.md#step-2-add-it-to-your-configuration).
> For more specific guides, you can also check out the pages about creating your own:
- [custom charts](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md) 📊
- [custom dynamic parameters](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md) 🎛
***
## Step 1: Create the component
### Create the main component file
```jsx
// ./src/components/Hello/Hello.jsx
import React, { Component } from 'react'
import PropTypes from 'prop-types'
/**
* Renders an hello-world
*/
class Hello extends Component {
state = {}
/**
* render
* @return {ReactElement} markup
*/
render () {
return (
Welcome {this.props.name}
Welcome to your newly created Application
)
}
}
// You can add props, you will be able to set value from Architect or from JSON
Hello.propTypes = {
name: PropTypes.string
}
export default Hello
```
### Create a style file
```less
// ./src/components/Hello/Hello.less
.hello {
.logo {
margin-top: 20px;
width: 200px;
}
text-align: center;
}
```
### Create and load files
Create an index
```js
// ./src/components/Hello/index.js
import Hello from './Hello.jsx'
import './Hello.less'
export default Hello
```
Add it to the components index
```js
// ./src/components/index.js
import Hello from './Hello'
import FpSdk from 'forepaas/sdk'
import DashboardTitle from './DashboardTitle'
import Username from './Username'
import Toaster from './Toaster'
export default {
components: {
Hello,
DashboardTitle,
Username,
Toaster
},
camelCaseToDash (myStr) {
return myStr.replace(/([a-z])([A-Z])/g, '$1-$2').toLowerCase()
},
init () {
for (let component in this.components) {
FpSdk.modules[this.camelCaseToDash(component)] = this.components[component]
}
}
}
```
At that moment, your component is ready to use, it will allow you to display the HTML set inside the render function of the React component, and it will need a props `name` to display the welcome message.
***
## Step 2: Add it to your configuration
### From JSON
You can add the component to two different parts of the application.
##### Inside a Dashboard
Open a dashboard file `./config/dashboards/overview.json`
Create a new item inside the root `items` or anywhere inside a panel.
In the following sample, the `hello` component has been added inside a panel.\
You can set `props` directly inside the JSON (`name`: user in the following sample)
```json
{
"name": "Overview",
"width": 36,
"height": 36,
"description": "",
"items": [
{
"type": "panel",
"sizeX": 36,
"sizeY": 36,
"row": 0,
"col": 0,
"items": [
{
"sizeX": 36,
"sizeY": 34,
"row": 0,
"col": 0,
"type": "hello",
"name": "user"
}
]
}
],
"url": "/overview",
"path": "",
"tags": []
}
```
#### Inside a menu
Open a menu file `./config/menus/header.json`
Create a new item inside a container.
```json
{
"id": "header",
"class": "header",
"containers": [
{
"id": "mycontainer",
"items": [
{
"type": "hello",
"name": "user"
}
]
}
...
]
}
```
The component should now appear in your menu.
### From Architect
:::warning
First, you need to upload your app with the new code, rebuild it, and deploy it.\
:::
Then, go in **Dashboard**, and select a dashboard to edit. Select a **+ button** (in menu or dashboard) then click on **custom**.
You will have to add it in JSON:
```json
{
"type": "hello",
"name": "user"
}
```
Click **Confirm**.
Congratulations! 🎉 You should now see the component inside your dashboard.
***
Ready to go even further?
?>➡️ Learn how to create [**custom charts**](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md)!
?>➡️ Learn how to create [**custom dynamic parameters**](https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md)!
---
url: https://docs.dataplatform.ovh.net/sdk/app-custom-chart.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create custom charts
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Data Platform makes it possible for you to **create your own chart templates**.
As a reminder, here is the configuration example of a chart:
```json
{
"type": "chart",
"col": 0,
"row": 1,
"sizeX": 12,
"sizeY": 12,
"title": "",
"chart": {
"component": "my-component",
"options": {
"title": "My component"
},
"request": {
"data": {
"fields": {
"quantity": [
"sum"
]
}
},
"filter": {},
"scale": {
"fields": [
"city"
]
}
},
"dynamic-parameters": [
"dp-datepicker",
"dp-city"
],
"not-nullable-dynamic-parameters": [
"dp-datepicker"
]
}
}
```
> This format is explained in detail in our [Getting Further on App Development](https://docs.dataplatform.ovh.net/tutorials-app-development-components.md) and lists the different templates provided by Data Platform (echart, table, jvectormap ....)
***
## Step 1: Creating a "Hello World" component
```jsx
import React from 'react'
import PropTypes from 'prop-types'
class ChartMyComponent extends React.Component {
render () {
return (
Hello World
)
}
}
ChartMyComponent.propTypes = {
chart: PropTypes.object
}
export default ChartMyComponent
```
This declares a ReactJS component that will simply display "Hello World". For this example, save it in `src/MyComponent.jsx`.\
Finally, you must import it into your Project by adding the highlighted line in the file `src/index.jsx`:
```jsx
FpSdk.start()
.then(() => {
var FpAppTemplate = FpSdk.modules.sdk.templates.default
FpSdk.modules['chart-my-component'] = require('./MyComponent.jsx')
render(, document.getElementById('root'))
})
```
Once done, you should see your chart display "Hello World".
:::warning
If you don't, it is perhaps because your request does not return results.
:::
***
## Step 2: Display data in your chart
Your template is called by the Data Platform SDK. You can access it via the `props` of your ReactJS component which holds a chart object with the following information:
- `this.props.chart.request`: copy of the query made by your chart
- `this.props.chart.options`: list of options that you have associated with your chart
- `this.props.chart.data`: result of the query sent to the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
In this example, HelloWorld is replaced with a simple table:
```jsx
import React from 'react'
import PropTypes from 'prop-types'
class ChartMyComponent extends React.Component {
render () {
return (
)
}
}
ChartMyComponent.propTypes = {
chart: PropTypes.object
}
export default ChartMyComponent
```
Thanks to this code, you now have a table with one row per result.
:::info
[See the Analytics Manager for a sample result.](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
:::
Now, let's go further and convert the JSON that appears into a real data row.
```jsx
import React from 'react'
import PropTypes from 'prop-types'
class ChartMyComponent extends React.Component {
render () {
return (
)
}
}
ChartMyComponent.propTypes = {
chart: PropTypes.object
}
export default ChartMyComponent
```
:::warning
In this example, there are several **hard coded** lines.\
As a result, if you change the query for the associated JSON, the Chart will not work.
:::
```jsx
{result.scales.city}
{result.data.quantity.sum[0].value}
```
Now, let's create a function that will dynamically search the **scales** and **data** of your graph:
```jsx
import React from 'react'
import PropTypes from 'prop-types'
class ChartMyComponent extends React.Component {
renderLine (result, i) {
let values = []
// Here we give a sample of loop that parses automatically a query result
// You have 3 levels:
// - data : It's the data you wan't to analyse, like turnover, quantity, etc....
// - computeMode : It's a compute mode like sum, count, avg ...
// - evol : You will always have 0 for 'now', but if you set an evol parameter like 'year', the previous year will be in 1
// Here you just need to convert QueryBuilder values to an easier format (an array)
for (let field in result.data) {
for (let computeMode in result.data[field]) {
for (let evol in result.data[field][computeMode]) {
values.push(result.data[field][computeMode][evol].value)
}
}
}
return (
{/* Here we get the scales from the result (in this case 'city' */}
{Object.values(result.scales).map((scale, j) => {
return
{scale}
})}
{/* Here we display our values */}
{values.map((d, j) => {
return
)
}
}
ChartMyComponent.propTypes = {
chart: PropTypes.object
}
export default ChartMyComponent
```
Finally, you can put the header line of your table dynamically from the query:
```jsx
import React from 'react'
import PropTypes from 'prop-types'
class ChartMyComponent extends React.Component {
renderHeader () {
let values = []
// Here I loop over the request to create the column header
let request = this.props.chart.request
for (let field in request.data.fields) {
for (let computeMode of request.data.fields[field]) {
values.push(field + '-' + computeMode)
if (request.evol && request.evol.scale) {
values.push(field + '-' + computeMode + '-1')
}
}
}
return (
)
}
renderLine (result, i) {
let values = []
// Here I give a sample of loop for parsing automatically a query result
// You have 3 level :
// - data : It's the data you wan't to analyse, like ca, qte, etc....
// - computeMode : Its a compute mode like sum, count, avg ...
// - evol : You will always have 0 for 'now', but if you set an evol parameter like 'year', the last year will be in 1
// Here we just need to convert QueryBuilder values to an easier format (an array)
for (let field in result.data) {
for (let computeMode in result.data[field]) {
for (let evol in result.data[field][computeMode]) {
values.push(result.data[field][computeMode][evol].value)
}
}
}
return (
{/* Here we get the scales from the result (like in my case the 'siege_social' */}
{Object.values(result.scales).map((scale, j) => {
return
{scale}
})}
{/* Here we display our values */}
{values.map((d, j) => {
return
)
}
}
ChartMyComponent.propTypes = {
chart: PropTypes.object
}
export default ChartMyComponent
```
Here is the visual result of your code:
***
## Step 3: Use a graphics library such as D3.js, Chart.js, or other
The first two steps have shown you how to display a simple table.
Now let's see how to plug the result with a library such as D3.js.
For this example, the previous result is connected with: [http://bl.ocks.org/phuonghuynh/54a2f97950feadb45b07](http://bl.ocks.org/phuonghuynh/54a2f97950feadb45b07)
First of all, let's add the dependencies requested by the plugin to index.html (this one does not seem compatible with Webpack, so simply add them to the \ tag)
```html
Data Platform dashboard
```
Then, copy and paste the example provided on the link in the `componentDidMount` method of our ReactJS component.
There are some modifications to make:
- Remove the `$(document).ready` from jQuery, because it will not be used by ReactJS
- Use the property `container` to pass to d3.js the **DOM node**, which will allow it to draw in the div present in the `render` function.
```jsx
import React from 'react'
import PropTypes from 'prop-types'
class ChartMyComponent extends React.Component {
componentDidMount () {
new window.d3.svg.BubbleChart({
supportResponsive: true,
container: this.refs.chart,
size: 600,
innerRadius: 600 / 3.5,
radiusMin: 50,
data: {
items: [
{ text: 'Java', count: '236' },
{ text: '.Net', count: '382' },
{ text: 'Php', count: '170' },
{ text: 'Ruby', count: '123' },
{ text: 'D', count: '12' },
{ text: 'Python', count: '170' },
{ text: 'C/C++', count: '382' },
{ text: 'Pascal', count: '10' },
{ text: 'Something', count: '170' }
],
eval: function (item) { return item.count },
classed: function (item) { return item.text.split(' ').join('') }
},
plugins: [
{
name: 'central-click',
options: {
text: '(See more detail)',
style: {
'font-size': '12px',
'font-style': 'italic',
'font-family': 'Source Sans Pro, sans-serif',
'text-anchor': 'middle',
'fill': 'white'
},
attr: { dy: '65px' },
centralClick: function () {
alert('Here is more details!!')
}
}
},
{
name: 'lines',
options: {
format: [
{// Line #0
textField: 'count',
classed: { count: true },
style: {
'font-size': '28px',
'font-family': 'Source Sans Pro, sans-serif',
'text-anchor': 'middle',
fill: 'white'
},
attr: {
dy: '0px',
x: function (d) { return d.cx },
y: function (d) { return d.cy }
}
},
{// Line #1
textField: 'text',
classed: { text: true },
style: {
'font-size': '14px',
'font-family': 'Source Sans Pro, sans-serif',
'text-anchor': 'middle',
fill: 'white'
},
attr: {
dy: '20px',
x: function (d) { return d.cx },
y: function (d) { return d.cy }
}
}
],
centralFormat: [
{// Line #0
style: { 'font-size': '50px' },
attr: {}
},
{// Line #1
style: { 'font-size': '30px' },
attr: { dy: '40px' }
}
]
}
}]
})
}
render () {
return
}
}
ChartMyComponent.propTypes = {
chart: PropTypes.object
}
export default ChartMyComponent
```
So far, it displays the same graph as the example found online.
To display your own data, convert the `this.props.chart.data` object to the format requested by the chart.
Add a `getItems` method:
```jsx
getItems () {
return this.props.chart.data.results.map(item => {
return {
text: item.scales.city,
count: item.data.quantity.sum[0].value
}
})
}
```
That you will call in the chart configuration:
```jsx
radiusMin: 50,
data: {
items: this.getItems(),
eval: function (item) { return item.count },
classed: function (item) { return item.text.split(' ').join('') }
},
plugins: [
```
This is the final result:
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters-create.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a custom Dynamic Parameter
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Creating your dynamic parameter makes it possible to use your favorite library or to implement all the details necessary for your use case.
> Check out our [Getting Further on App development](https://docs.dataplatform.ovh.net/tutorials-app-development-custom-components.md#generate-a-component) for specific use cases.
To get started, here is a hello-world for this dynamic parameter.
We will copy it to /src/FpDynamicParameterCustom.jsx
```jsx
import React from 'react'
import {connect} from 'react-redux'
import {set} from 'forepaas/store/querystring/action'
// Our system uses Redux to store information in a store named "querystring".
// Here we connect this component to the relevant store.
@connect((state) => ({
querystring: state.querystring
}))
class FpDynamicParameterCustom extends React.Component {
constructor (props) {
super(props)
// We will always store our current value in this.state.model
this.state = {
model: ''
}
// At the creation of composant, we have to check if a previous value is already on store
this.initFromModel()
}
// Initialize the value with the current state in store
initFromModel () {
if (this.props.id) {
var model = this.props.querystring[this.props.id]
if (model) {
this.state.model = model
}
}
}
// Update the store with "model" value
updateModel (model) {
if (this.props.id) {
this.props.dispatch(set(this.props.id, model))
}
}
// For now we will only render a "hello world string"
render () {
return (
Hello world
)
}
}
export default FpDynamicParameterCustom
```
Then in the file /src/index.js, add these lines to declare it. They should be located before the line `FpSdk.start()`
```jsx
import FpDynamicParameterCustom from './FpDynamicParameterCustom.jsx'
FpSdk.modules['dynamic-parameter-custom'] = FpDynamicParameterCustom
```
Once this is done, the dynamic parameter is working: it's called `custom`.
Add it to your **Dashboard**, via this configuration JSON snippet inside a _dynamic parameter_.
```json
{
"id": "custom",
"type": "filter",
"component": "custom"
}
```
Or directly as code like in this example:
```jsx
import React from 'react'
import FpDynamicParameter from 'forepaas/dynamic-parameter'
import moment from 'moment'
const myDynP = {
id: 'custom',
type: 'filter',
component: 'custom'
}
class MyComponent extends React.Component {
render () {
return (
)
}
}
export default MyComponent
```
Your component will display "Hello World".
In this example, simply put an input text, which will be used to filter on the `city` field.
For this purpose, add to the configuration the `reference` field, which in the App SDK defines the field on which dynamic parameter will act.
```json
{
"id": "custom",
"type": "filter",
"component": "custom",
"reference": "city"
}
```
Then modify the component, adding:
- the `handleChange` function
- an input instead of the "hello world"
```jsx
handleChange (event) {
this.setState({model: event.target.value})
}
render () {
return (
)
}
```
At each modification of the input, we will update the component state. But as mentioned earlier, this has no effect on the store.\
You must add one line in the `handleChange` function.
```jsx
handleChange (event) {
this.setState({model: event.target.value})
this.updateModel(event.target.value)
}
```
Your dynamic parameter is ready!
- If you change its content, URL will change too (with its value).
- If you connect it to a chart, it will be filtered automatically.
---
url: https://docs.dataplatform.ovh.net/sdk/app-faq.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Troubleshooting & FAQ
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
This page provides an accessible reference to solutions for common issues you may have when developing your application on Data Platform as well as some **Frequently Asked Questions**.
- [Troubleshooting](https://docs.dataplatform.ovh.net/sdk/app-faq.md#troubleshooting)
- [Incorrect query](https://docs.dataplatform.ovh.net/sdk/app-faq.md#incorrect-query)
- [FAQ](https://docs.dataplatform.ovh.net/sdk/app-faq.md#faq)
- [How to use a specific NodeJS version?](https://docs.dataplatform.ovh.net/sdk/app-faq.md#how-to-use-a-specific-nodejs-version)
- [How to format a value using an existing formatter?](https://docs.dataplatform.ovh.net/sdk/app-faq.md#how-to-format-a-value-using-an-existing-formatter)
- [How to use registered translations?](https://docs.dataplatform.ovh.net/sdk/app-faq.md#how-to-use-registered-translations)
- [How to make a query and get its results outside a Chart?](https://docs.dataplatform.ovh.net/sdk/app-faq.md#how-to-make-a-query-and-get-its-results-outside-a-chart)
- [How to deploy a Simple Web App?](https://docs.dataplatform.ovh.net/sdk/app-faq.md#how-to-deploy-a-simple-web-app)
If you do not find an answer to your question, please reach out to our [Support team](https://docs.dataplatform.ovh.net/sdk/app-faq.md#cannot-find-an-answer-to-your-question-%F0%9F%86%98)!
***
## Troubleshooting
### Incorrect Query
#### Occurrence
This error occurs when you use attempt to access a _query_ that isn't available.
##### Common causes & solutions
**Undeployed API**
If the API associated to your APP isn't deployed, your APP won't be able to retrieve the query in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
Check if your API is deployed and deploy it if it isn't.
**Typo in your query**
Check if the fields in your query are written correctly and if they exist in the Analytics Manager.
***
## FAQ
### How to use a specific NodeJS version?
It is possible to specify the NodeJS version to use when building App by specifying in `./forepaas.json`:
```json
"node": 14,
```
***
### How to format a value using an existing formatter?
```jsx
// Leverage "turnover" as declared in ./config/formatter.json to format a given value
import { FpMeasure } from 'forepaas/formatter'
function formatValue(field, value) {
let measure = new FpMeasure(field);
return measure.setValue(value).toString();
}
console.log(formatValue("turnover", 100));
```
***
### How to use registered translations?
```jsx
import FpTranslate from 'forepaas/translate'
console.log(FpTranslate("key_to_be_translated"})
```
***
### How to make a query and get its results outside a Chart?
```jsx
import FpQueryBuilder from 'forepaas/query-builder'
const query = {
data: {
fields: {
date: ["min","max"]
}
}
}
let request = new FpQueryBuilder.FpQuery(query)
request.compute().then((response) => {
// display array of results from the query
console.log(response.results)
// displays the first item in results
console.log(response.results[0])
// get 'data' from 1st item and displays values
let item= response.results[0].data
console.log(item.date.min[0].value, item.date.max[0].value)
})
```
***
### How to deploy a Simple Web App?
Sometimes it is needed to deploy a simple web app or using a different SDK.
App's code may be provided either through a ZIP file or a Git repository.
It must provide at least:
- `forepaas.json`: minimal version described below
- `package.json`: minimal version described below
- `production/`: folder that will be the root of the website after deployment, the `package.json` file may contain a `build` command to make that folder
**forepaas.json**
```json
{
"type": "app",
"language": "js",
"framework": "custom",
"modules": {}
}
```
**package.json**
```json
{
"name": "app_name",
"version": "1.0.0",
"main": "index.js",
"scripts": {
"build": "echo no build to do"
}
}
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Front API SDK
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
In Data Platform, the Front API is the **Project's gateway for remote requests**. It is built and deployed using [API Manager](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) and has two main purposes:
- Access from the [Front App](https://docs.dataplatform.ovh.net/sdk/app.md) with all tight integration built-in
- Access from other remote systems that need to request your Project
In addition, vanilla Front APIs deployed by default have the following features:
- Enforce authentication and data ACL as declared in the [Identity Application Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md)
- Pass on requests using [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) format
- Access to a `WebStorage` to be used by Front App
- adding [available extensions](https://docs.dataplatform.ovh.net/app-services-apis-extensions-list.md), export CSV/XLSX
***
## Customize your Project's API using the SDK
### When is customization needed?
The aforementioned features are sufficient for most use cases, but you can extend it much further. The Front API SDK is written in NodeJS. After an [API has been deployed](https://docs.dataplatform.ovh.net/app-services-apis-deploy.md), it is easy to download it and start hacking.
Typical cases:
- needs to provide an endpoint with a specific request and response format
- needs to implement _transformers_ to dynamically change QB requests (adding fields, changing filters or scales on the fly)
- implement advanced permissions restrictions on data fields
- merge several QB requests to one result
- request external API on behalf of Front App
- ... and whatever you may do in NodeJS!
:::warning
Customizing a Front API requires some knowledge of JavaScript, NodeJS and Web Service development, and should be done by advanced users.
:::
### Requirements
- [NodeJS](https://nodejs.org/en/): Front API is implemented in NodeJS
- [Docker](https://www.docker.com/): Front API may be started with Docker
- [Git](https://git-scm.com/): we strongly advise to version your code with Git and use it to deploy on Data Platform
### What can you customize?
Check out the following articles to discover how to customize an API using the SDK.
- [Discover the structure of an API](https://docs.dataplatform.ovh.net/sdk/api-api-files.md)
- [Launch in a local environment](https://docs.dataplatform.ovh.net/sdk/api-api-launch.md)
- [Manage modules](https://docs.dataplatform.ovh.net/sdk/api-configure-module.md)
- [Handle queries](https://docs.dataplatform.ovh.net/sdk/api-queries.md)
- [Add query transformers](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md)
- [Add query endpoints](https://docs.dataplatform.ovh.net/sdk/api-qb-endpoint.md)
- [Add query data ACL](https://docs.dataplatform.ovh.net/sdk/api-qb-acl.md)
- [Create custom API endpoints](https://docs.dataplatform.ovh.net/sdk/api-custom-endpoints.md)
- [Expose a bucket file](https://docs.dataplatform.ovh.net/sdk/api-download-file-datastore.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-api-files.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Structure of an API on Data Platform
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Here is the content of the source code of an API. Note that the SDK is available in NodeJS.
```
├── .gitignore
├── Dockerfile.dev
├── LICENSE.md
├── LICENSE.txt
├── app.js
├── dev.sh
├── forepaas
| |-- cache-controller
| |-- cache-redis
| |-- cam
| |-- logger-console
| |-- qb
| |-- reportings
| |-- sdk
| `-- webstorage
├── forepaas.json
├── package.json
├── run.sh
└── yarn.lock
```
- [Core files of the API](https://docs.dataplatform.ovh.net/sdk/api-api-files.md#the-core-files-of-the-api)
- [Launcher utilities](https://docs.dataplatform.ovh.net/sdk/api-api-files.md#launcher-utilities)
- [Git](https://docs.dataplatform.ovh.net/sdk/api-api-files.md#git)
- [License](https://docs.dataplatform.ovh.net/sdk/api-api-files.md#license)
### The core files of the API
```
├── app.js
├── forepaas.json
├── forepaas
├── package.json
└── yarn.lock
```
- `app.js`: central point of the API and will serve to launch all the rest. Here you can directly access the ExpressJS instance (via the app variable)
- `forepaas.json`: describes the list of extensions that make up your API
- `forepaas/`: contains the extensions installed in your API
- `package.json` : contains the NodeJS dependencies associated with your API
- `yarn.lock` : we use [yarn](https://yarnpkg.com/lang/en/) as package manager, it generates a file `yarn.lock` to **lock dependencies associated with your API**.
### Launcher utilities
- Docker
- `Dockerfile.dev`: a **Docker** description of the context in which the API runs.
- `dev.sh`: a script to launch the docker environment quickly.
- NodeJS
- `run.sh` : a script to launch your API directly
### Git
- `.gitignore`: defines the Git files and folder that will not be _commited_ (in case you use git).
### License
These two files mention the Data Platform license associated with this SDK.
- `LICENSE.md`
- `LICENSE.txt`
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-api-launch.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Launch an API in a local environment
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
While **OVHcloud Data Platform** is great as a tool for hosting deployed APIs during both development and production, it’s not always the natural place to edit code. Most software engineers prefer to edit their code on their computer.
Below is how you can build and deploy an API from a local development environment.
- [0) Requirements](https://docs.dataplatform.ovh.net/sdk/api-api-launch.md#requirements)
- [1) Download and install locally](https://docs.dataplatform.ovh.net/sdk/api-api-launch.md#download-and-install-locally)
- [2) Launch the API](https://docs.dataplatform.ovh.net/sdk/api-api-launch.md#launch-the-api)
- [3) Manage authentication](https://docs.dataplatform.ovh.net/sdk/api-api-launch.md#manage-authentication)
- [4) Launch everything together](https://docs.dataplatform.ovh.net/sdk/api-api-launch.md#launch-everything-together)
***
## Requirements
### NodeJS
Currently we use v10 in production, but we will soon update it to 14.
In local, it will work with both versions, we recommend using v14 right now, but to keep a look on the functions used while the update is not done in production.
Install it from [here](https://nodejs.org/en/).
### Redis (optional)
By default the API use Redis as cache in production
It can be disable for dev, but you can also use a local redis
#### Install redis
If you want to have it locally, we recommand to install and use [Docker](https://docs.docker.com/desktop/).
Then it is possible to:
```sh
# start a redis locally
docker run --name redis -d -p 6379:6379 redis
# stop it
docker stop redis
# restart it
docker start redis
# remove it (need to be stop before, or to add the flag -f in the command)
docker rm redis
```
#### or Disable cache
- Open "forepaas.json"
- Replace these occurence
```json
"cache": {
"module": "cache-redis"
},
```
by
```json
"_cache": {
"module": "cache-redis"
},
```
- Remove the cache redis main occurence
```json
"cache-redis": {
"version": "2.0.0"
},
```
***
## Download and install locally
- From the GUI, go to the API Overview
[https://eu.dataplatform.ovh.net/#/api/\\\{ProjectID\\}/\\\{API\_NAME\\}](https://eu.dataplatform.ovh.net/#/api/\\\{ProjectID\\}/\\\{API_NAME\\})
- Select the tag you want to copy, then click on "Download that version"
- Extract the zip in a folder of your choice
- Install node dependencies (we use the package manager yarn by default, we recommend you to use the same to have same condition than production)
```sh
yarn
```
***
## Launch the API
By default you should find a "run.sh" script, already here, giving a sample of quick run
```sh
KONG_INTERNAL_URI=false \
FOREPAAS_APPLICATION_CREDENTIALS=./creds.json \
FOREPAAS_QB_RESOURCE_ID=http://qb-service:9000 \
REDIS_URI=redis://127.0.0.1 \
ENV=development \
yarn run start
```
- **KONG\_INTERNAL\_URI**: it will disable a production mode for URL computing, we plan to remove that variable in future
- **FOREPAAS\_APPLICATION\_CREDENTIALS**: used to link to a local `creds.json` file, it will be described below
- **FOREPAAS\_QB\_RESOURCE\_ID**: get legacy ACL resources
- **REDIS\_URI**: if using a local redis, it should override the URL here, as explained in previous section (it should already be the correct value)
- **ENV**: set to `development` for local development
- `yarn run start`: it will launch the API, use `yarn run watch` to reload as soon as code is updated in local environement
If you launch the API, it should start well, but you will not be identify to the platform and your API will not be able to call the Project API. For this, you need to get authenticated.
***
## Manage authentication
By default, in production on **OVHcloud Data Platform**, the API is self-authenticated to the other Project components. But when you are in a local development environment, you need to authenticate though the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
The best way to do it is by [generating API and secret key for a user (or a service account)](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md). You will also need your [Project subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md).
Once you have the keys and project subdomain, you can create a file called "creds.json" at the root of the API. This file should never be committed (set in .gitignore by default).
```json
{
"dataplant_url": "https://{DATAPLANT_SUBDOMAIN}.eu.dataplatform.ovh.net",
"api_key": "610aa8766608131b00000095610aa87f6608131b00000097",
"secret_key": "8559d0f805dbb981XXXXXXXXXXXXXXXXXXXXXXXXXXXXX8bf9af358a4"
}
```
***
## Launch everything together
Launch your API by calling the "run.sh" file.
```sh
sh run.sh
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-configure-module.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage modules
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
APIs work primarily with **modules** that can be interconnected. Those modules are configured in the `forepaas.json` file at the root of your [Front API](https://docs.dataplatform.ovh.net/sdk/api-api-files.md), along with the rest of the API configuration:
```json
//forepaas.json
{
"id": "frontapi", // unique ID: used if you choose to publish as a "template" in the store
"type": "api", // forepaas.json is used for 'app' and 'api', here it is 'api'
"language": "node", // Language used for the API
"name": "FrontAPI",
"version": "0.0.1",
"modules": {...}, // Detailed below in the "Configure modules" section
"dependencies": {...} //
```
> This file is fully configurable via the **OVHcloud Data Platform**, on the [Extension configuration of your API](https://docs.dataplatform.ovh.net/app-services-apis-extensions-list.md).
> The file `forepaas.json` is automatically modified each time you install or uninstall a module from the platform.
The modules' configuration is done via two separate fields:
- the `modules` key: it contains the list of modules to be installed
- the `dependencies` key
***
## Configure modules
Here is an example of a configuration :
```json
"modules": {
"sdk": {
"version": "2.4.0"
},
"iam": {
"version": "1.0.0"
},
"cache-redis": {
"version": "2.0.0"
},
"qb": {
"version": "2.4.0",
"dependencies": {
"authentication": {
"module": "iam"
},
"cache": {
"module": "cache-redis"
}
}
}
}
```
For a better understanding, here is a short description of each module:
- `sdk`: the heart of the Front API. It is essential, but replaceable if you want to modify the standard behavior of **OVHcloud Data Platform** APIs
- `iam`: a connector to the Project's [Identity Access Manager](https://docs.dataplatform.ovh.net/developers-api-authentication.md). It allows you to manage access to your API.
- `cache-redis`: a cache connector to a _Redis_ database
- `qb`: a connector to the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md), to query and use your data in your applications
All modules act independently. However, `qb` module relies upon _cache_ and _authentication_. In this sample, it is configured to use `iam` module to manage _authentication_ and `cache-redis` for _cache_.
:::warning
⚠️ Without the `iam` module, your API does not have any access control!
:::
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-queries.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Handle queries
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
One of the core function of the API is to handle queries to your Data Platform data system from the outside world, or from an [application](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md).
You can [add transformers](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md) to your queries to modify them on the fly, add new [query endpoints](https://docs.dataplatform.ovh.net/sdk/api-qb-endpoint.md), and [manage the data ACL](https://docs.dataplatform.ovh.net/sdk/api-qb-acl.md) for the queries passed on to Data Platform.
[Pass on external queries](https://docs.dataplatform.ovh.net/sdk/api-queries.md#pass-on-external-queries-queries)
[Add query transformers in your API](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md)
[Add query endpoints in your API](https://docs.dataplatform.ovh.net/sdk/api-qb-endpoint.md)
[Add query data ACL in your API](https://docs.dataplatform.ovh.net/sdk/api-qb-acl.md)
***
## Pass on external queries
### Configure a web service for query execution
Queries can be executed from an external appliance / service through configuring a Web Service. The return formats may be JSON, CSV or XLSX.
#### Get credentials
By default, in production, the API is self-authenticated to the other Project components. But when you are calling it from the outside world, you need to authenticate though the [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
The best way to do it is by [getting an authentication token using API and secret key for a user (or a service account)](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md).
#### Run queries via the web service
**JSON output**
```
curl -X POST \
https://my-project.eu.dataplatform.ovh.net/myapi-api/qb/query?app_id=your_app_id \
-H 'Content-Type: application/json' \
-H 'authorization: Bearer your_token' \
-d '{
"data": {
"fields": {
"date": ["min", "max"]
}
}
}'
```
**CSV output**
```sh
curl -X POST \
https://my-project.eu.dataplatform.ovh.net/myapi-api/qb/query.csv?app_id=your_app_id \
-H 'Content-Type: application/json' \
-H 'authorization: Bearer your_token' \
-d '{
"data": {
"fields": {
"date": ["min", "max"]
}
}
}'
```
> Note that CSV formatting may be controlled with `options` in the query, as for instance:
```json
"options":{
"collapse": "false",
"scales": {
"x": [
{"field": "field_name_1", "type":"scale"},
/*...*/
{"field": "field_name_n", "type":"scale"}
],
"y": [{"type": "data"}]
}
}
```
**XLSX output**
```sh
curl -X POST \
https://my-project.eu.dataplatform.ovh.net/myapi-api/qb/query.xlsx?app_id=your_app_id \
-H 'Content-Type: application/json' \
-H 'authorization: Bearer your_token' \
-d '{
"data": {
"fields": {
"date": ["min", "max"]
}
}
}'
```
:::info
Note that using `options` it is possible to better control the output.
:::
### Front API basic endpoints reference
Here are the list of basic endpoints that comes with the default Front API.
## _GET_ /refresh
Flush the cache
**Responses**
| Code | Description |
| ---- | :------------------------------ |
| 200 | A simple success `true` message |
## _GET_ /qb/dataset
Retrieves the dataset from the Analytics Manager
**Responses**
| Code | Description |
| ---- | :------------------------------------- |
| 200 | The dataset from the Analytics Manager |
## _POST_ /qb/query
Make a query from an Analytics Manager JSON request
**Responses**
| Code | Description |
| ---- | :---------------------------------- |
| 200 | A result from the Analytics Manager |
## _GET_ /webstorage/`{key}`
Retrieves a value for a key in the webstorage
**Parameters**
| Name | Located in | Description | Required | Schema |
| :--- | :--------- | :---------- | :------- | :----- |
| key | path | | Yes | string |
**Responses**
| Code | Description |
| ---- | ------------- |
| 200 | The content |
| 404 | Key not found |
## _PUT_ /webstorage/`{key}`
Puts a value from the body of the request to a key in the webstorage
**Parameters**
| Name | Located in | Description | Required | Schema |
| :--- | :--------- | :---------- | :------- | :----- |
| key | path | | Yes | string |
**Responses**
| Code | Description |
| ---- | --------------- |
| 200 | The new content |
## _DELETE_ /webstorage/`{key}`
Removes a value for a specific key in webstorage
**Parameters**
| Name | Located in | Description | Required | Schema |
| :--- | :--------- | :---------- | :------- | :----- |
| key | path | | Yes | string |
**Responses**
| Code | Description |
| ---- | -------------------- |
| 200 | The previous content |
| 404 | Key not found |
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create Transformers for Analytics Manager queries
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
Transformers intercept and change queries right before they are sent to the Analytics Manager.
In the default flow, **Default Transformers** are called by the API's query module (`/forepaas/qb`), in order to inject the values of the **Dynamic Parameters** to the filters and scales of the query.
***
## Typical use case
For a better understanding of how transformers work and can be created, let's use an example that is often found in data Projects.
Let's say that most of the displayed data is filtered on the same `date` attribute. However, let's imagine some tables do not have a `date` attribute, but instead a `previous_date` attribute.\
With default behavior, queries made to this table would return a **No table found for this query** error.
To solve this, a custom transformer is needed.
- [Create a custom transformer](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md#Create-a-custom-transformer)
- [Configure the Front App](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md#Configure-the-Front-App)
- [Customize a transformer](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md#Customize-Transformer)
- [Check the transformation](https://docs.dataplatform.ovh.net/sdk/api-qb-transformers.md#Check-the-transformation)
***
## Create a custom transformer
Just like custom endpoints, creating a new transformer requires adding files in a specific directory: `/src/qb/transformers`.
There are 3 types of transformers available for each part of a query :
- `filter`: for which you can create a `filter.js` file
- `data`: for which you can create a `data.js` file
- `scale`: for which you can create a `scale.js` file
In the case described above, create `./src/qb/transformers/filter.js`, copying the default one from `./forepaas/qb/transformers/filter.js`
**./src/qb/transformers/filter.js**
```js
module.exports = {
default:(query,{reference,value})=>{
if(value === null) return Promise.resolve(query)
if(!Array.isArray(value)) value = (value||"").split(';')
query.filter = query.filter || {}
if (reference.search(/^range_/) !== -1) {
let rangeref = reference.replace(/^range_/, '')
query.filter[rangeref] = query.filter[rangeref] || {}
if(Array.isArray(value)) value = value[0]
if(value) query.filter[rangeref].between = value.split(',')
return Promise.resolve(query)
}
query.filter[reference] = query.filter[reference] || {}
query.filter[reference].in = query.filter[reference].in || []
query.filter[reference].in = query.filter[reference].in.concat(value)
return Promise.resolve(query)
}
}
```
Transformers are objects with at least one `default` method.\
This method takes the following parameters:
- `query`: the full query modified by the function
- `reference`: as defined in the dynamic parameter configuration in the **Front App**
- `value`: value(s) passed by the **Front App**
This function is called for each Dynamic Parameter passed with the Query.
At this point, the Front API keeps the same behavior, going through this custom transformer rather than the default one.
***
## Configure the Front App
With that in mind let's have a look at a regular request.\
Here is the configuration of a chart:
```json
{
...
"chart": {
"requestId": "xyz123-xyz123-xyz123-xyz123-xyz123",
"dynamic-parameters": [
"datepicker"
]
},
...
}
```
The generated HTTPS request is a _POST_ call on the endpoint `/qb/query/xyz123-xyz123-xyz123-xyz123-xyz123` with the following body:
```json
{
"dynamic_parameters": [
{
"type": "filter",
"transform": "default",
"value": [
"1582502400,1582588799"
],
"params": [],
"reference": "range_date"
}
]
}
```
Here only 2 things need to change in order to get the needed feature:
- `default`: needs to be changed to a custom value, for instance `changeRef`
- `params`: needs to include `previous_date` so our custom transformer may use it.
The `dynamic-parameters` configuration allows you to do both those changes.
Going back to the chart's configuration, the transformer type will be set using a `|` and the `params` using a `:`.
```json
{
...
"chart": {
"requestId": "xyz123-xyz123-xyz123-xyz123-xyz123",
"dynamic-parameters": [
"datepicker|changeRef:range_previous_date"
]
},
...
}
```
And the _POST_ body becomes :
```json
{
"dynamic_parameters": [
{
"type": "filter",
"transform": "changeRef",
"value": [
"1582502400,1582588799"
],
"params": ["rnge_previous_date"],
"reference": "range_date"
}
]
}
```
It is now necessary to handle it in the `filter` transformer.
***
## Customize transformer
A new method `changeRef` needs to be implemented in the object taking the same input as `default`, plus the `params`array described above.
```js
module.exports = {
...
changeRef: (query, { reference, value, params }) => {}
}
```
In that case, it is suited to factorize `default` method, so `changeRef` may be implemented by calling it with different rules for `reference`.
Therefore, the full **/src/qb/transformers/filter.js** ends up looking like this:
```js
const defaultTransformer = (query, { reference, value }) => {
if (value === null) return Promise.resolve(query)
if (!Array.isArray(value)) value = (value || "").split(';')
query.filter = query.filter || {}
if (reference.search(/^range_/) !== -1) {
let rangeref = reference.replace(/^range_/, '')
query.filter[rangeref] = query.filter[rangeref] || {}
if (Array.isArray(value)) value = value[0]
if (value) query.filter[rangeref].between = value.split(',')
return Promise.resolve(query)
}
query.filter[reference] = query.filter[reference] || {}
query.filter[reference].in = query.filter[reference].in || []
query.filter[reference].in = query.filter[reference].in.concat(value)
return Promise.resolve(query)
}
module.exports = {
default: defaultTransformer,
changeRef: (query, { reference, value, params }) => {
return defaultTransformer(query, {
reference: params[0] || reference,
value
})
}
}
```
The `changeRef` method calls the `defaultTransformer` function, changing `reference` in the process.
***
## Check the transformation
The last step is to make sure that query uses the `changeRef` option. In the end, the query sent by the chart will have the following body:
```json
{
"filter": {
"previous_date": {
"between": ["1582502400","1582588799"]
}
}
}
```
> 💡 Using these techniques on `filter`, `scale` or `data` coupled with [Front App's Dynamic parameters](https://docs.dataplatform.ovh.net/sdk/app-dynamic_parameters.md) allows you to thouroughly customize your apps!
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-qb-endpoint.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a custom endpoint for Analytics Manager queries
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
It is possible to easily extend the Analytics Manager query flow by adding new `endpoints`.\
Those endpoints can then be called from any `chart` in the **Front App** to be displayed.
Here are some use examples:
- [merge 2 queries in one result](https://docs.dataplatform.ovh.net/sdk/api-qb-endpoint.md#merge-two-queries)
- use Machine Learning to compute and display forecast
- do some computation on results before returning them
***
## Create a new endpoint
> Simply add a file in `./src/qb/endpoints/`.\
> This folder does not exist by default, it must be created from the root of the Front API.
For instance, adding the following file provides an answer from a _GET_ request on `qb/hello_world` :
**./src/qb/endpoints/hello\_world.js**
```js
module.exports = {
method: "GET",
request: function(req, res, next) {
res.send('hello')
}
}
```
***
## Merge two queries
In this example, consider a dataset containing the **turnover** of Data Platform and 4 other companies. The objective is to compare :
- the evolution of the Data Platform turnover
- the evolution of the sum of all turnover (Data Platform + 4 others).
The Analytics Manager query format needs 2 different queries :
```json
// Data Platform Turnover
{
"data":{
"fields":{
"turnover":["sum"]
}
},
"scale":{
"fields":["date"]
},
"filter":{
"company": ["Data Platform"]
}
}
// Global Turnover
{
"data":{
"fields":{
"turnover": ["sum"]
}
},
"scale":{
"fields":["date"]
}
}
```
In order to build a chart with 2 curves in the Front App, the custom endpoint should return the appropriate result set.
```js
const QueryBuilder = require('../../../forepaas/qb/services/QueryBuilder')
const Merge = require('../../../forepaas/qb/services/Merge')
const Formatter = require('../../../forepaas/qb/services/Formatter')
module.exports = {
method: 'GET',
request: (req, res, next) => {
// define 2 queries
let queries = {
// query 1 : forepaas turnover
forepaas: {
data: {
fields: {
turnover: ["sum"]
}
},
scale: {
fields: ["date"]
},
filter: {
company: ["Data Platform"]
}
},
// query 2: global turnover
global:{
data: {
fields: {
turnover: ["sum"]
}
},
scale: {
fields: ["date"]
}
}
};
// execute queries and merge results
QueryBuilder.queries(req, queries).then(responses => {
let response = new Merge(responses, { renameWithId:true }).request();
return Formatter(res, 'json', response);
}).catch(next)
}
}
```
In this example :
1. both queries are made through the `QueryBuilder.queries()` helper function.
2. responses consist of 2 distinct results : the Front App does not handle it as is.
3. the module `Merge` offers basic merge functions to accumulate queries.\
The `renameWithId` option provides two _forepaas:turnover_ and _global:turnover_ fields in return.
:::info
**To be noticed**\
Caching and activation of authentication is done automatically.
:::
***
## Improve to be dynamic
In most cases, custom endpoints are not simple **GET** requests that always compute the same request. Usually, the request(s) sent to the Analytics Manager are dynamically generated using the body of the request.\
In order to do so, the request's method needs to change to **POST** and queries passed as variable.
These changes result in the following code:
```js
const QueryBuilder = require('../../../forepaas/qb/services/QueryBuilder')
const Merge = require('../../../forepaas/qb/services/Merge')
const Formatter = require('../../../forepaas/qb/services/Formatter')
module.exports = {
method: 'POST',
request: (req, res, next) => {
// define 2 queries from req.body
let queries = {
forepaas:{
...req.body,
filter: {
company: ["Data Platform"]
}
},
global: req.body
};
// execute queries and merge results
QueryBuilder.queries(req,queries).then(responses=>{
let response = new Merge(responses, { renameWithId:true }).request()
return Formatter(res, 'json', response)
}).catch(next)
}
}
```
***
## Call it from the Front App
In the JSON of a `Chart`, instead of using the `request` parameter, all you have to do is use the `endpoint` parameter, as in this example:
```json
{
"component": "echarts",
"endpoint": {
"method": "GET",
"query": "qb/hello_world",
"params": {
"data":{
"fields":{
"turnover": ["sum"]
}
},
"scale":{
"fields":["date"]
}
}
}
}
```
:::warning
For an endpoint in **POST**, dynamic parameter values will be passed in `req.body`.
:::
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-qb-acl.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Customize Analytics Manager ACL rules in Front API
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
:::warning
The extension `qb` must be installed.
:::
Create a file: /src/qb/services/ACL.js from this template:
```js
module.exports = async (req,query,dataset)=> {
return query
}
```
For the moment no rules will be applied.
From this function you will be able to :
- Disallow the user to make the query
OR
- Update the query content by adding field inside the query (filter for example)
A complete example :
```js
// This function is called each time you launch a query from the Front API,
// and allows you to intercept it.
// thus you can easily forbid a query or modify it (in order to apply a filter)
module.exports = async (req,query,dataset)=> \{
// req.authentication.session contains the object provided by IAM
// or your custom authentication
// If you wish to deny certain requests for non-admins.
if(query.data.fields.quantity && !req.authentication.session.isAdmin) \{
let err = new Error('QuantityNotAuthorized')
err.status = 401
throw err
\}
// If you need to apply a filter for non-admin
if(!req.authentication.session.isAdmin) \{
query.filter = query.filter || \{\}
query.filter.company = \{
in:["Data Platform"]
\}
\}
return query
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-custom-endpoints.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create custom endpoints
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
It is possible to create completely new **endpoints** in your API.
- [Add new routes](#add-new-routes)
- [Upload files through APIs](#upload-files-through-apis)
***
## Add new routes
**./src/index.js**
```js
const IAM = require('../forepaas/iam/IAM')
/*
We define an AuthInterceptor, that checks for user authentication, to be used in the new routes.
Without adding it, the new routes will be publicly available.
In the /qb/* endpoints the same mechanism is already done.
*/
let AuthInterceptor = function (req, _res, next) {
new IAM().checkSession(req, (err, session) => {
if (err) {
return next(err);
}
req.authentication = {
session
}
next();
})
}
/*
The SDK loads src/index.js, if present, and execute the exported function passing the "Express app".
That gives an opportunity to add routes.
*/
module.exports = (app) => {
// this route checks for user authentication
app.get('/hello/user', [AuthInterceptor], (req, res, next) => {
res.ok({"hello": "user !"});
});
// this route will be public
app.get('/hello/everyone', (req, res, next) => {
res.ok({"hello": "world !"});
});
}
```
> **Notes:**
- The SDK uses [Express](https://expressjs.com/)
- If the `./src/index.js` file exists, it is executed as a function passing the Express `app`
- This allows you to declare new routes using Express methods
- It is needed to handle Authentication in those implementations
***
## Upload files through APIs
You can use the same mechanism and a library such as [Multer](https://www.npmjs.com/package/multer).
For instance, to process `my-budget.csv`:
```js
const IAM = require('../forepaas/iam/IAM')
const upload = multer()
module.exports = (app) => {
app.get('/send/budget', [AuthInterceptor, upload.single('budget')], (req, res, next) => {
console.log(req.file)
res.ok()
})
}
```
It is then possible to process the file from the Front API, store it in the [Lakehouse Manager buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) and launch a [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) workflow or action!
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/sdk/api-download-file-datastore.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Expose a file from your buckets
:::warning
**Rework pending, no delivery date.** The Front App SDK (ReactJS) and the Front API SDK (NodeJS) are both being replaced. This page documents the current release, remains accurate, and is maintained until the replacement ships. Code written against it keeps working.
:::
A custom endpoint can be added to your API to serve a file from a [Lakehouse Manager bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) (object store).
- [1) Get credentials](#get-credentials)
- [2) Create a custom module to expose the file](#create-a-custom-module-to-expose-the-file)
- [3) Launch the API](#launch-the-api)
***
## Get credentials
The person/service authenticating to access the file through this new API endpoint must have at least **read access** on the buckets in the Lakehouse Manager.
Create an [Identity Access Manager user](https://docs.dataplatform.ovh.net/iam-users.md) (or service account) with a role containing the permission _Datastore bucket read_. Then [generate API and secret key for this user](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md): those keys will be used when configuring the API endpoint.
***
## Create a custom module to expose the file
In your api, you can add / edit this code
```json
// forepaas.json
// In modules key, add the configuration of your new module
"s3-sample": {
"version": "local",
"dependencies": {
"authentication": {
"module": "iam"
}
}
},
```
In the folder forepaas, create a subfolder named "s3-sample"
Inside, you will have to create 2 main files
```json
// forepaas/s3-sample/forepaas.json
{
"id": "s3-sample",
"type": "package",
"name": "S3 Saple",
"version": "1.0.0",
"npmDependencies": {
"aws-sdk": "2.966.0"
}
}
```
```js
// forepaas/s3-sample/factory.js
const express = require('express')
const router = express.Router()
const AWS = require('aws-sdk')
const util = require('util')
// Modify those values for your environment
let accessKey = 'YOUR_ACCESS_KEY'
let secretKey = 'YOUR_SECRET_KEY'
let endpoint = 'https://YOUR_PROJECT_SUBDOMAIN.eu.dataplatform.ovh.net'
let bucket = 'project-YOUR_PROJECTID-YOUR_BUCKET'
let region = 'forepaas'
// If no authentication is bind, it will use that default function
let defaultDependencies = {
authentication: {
checkSession:(req,cb) => cb()
}
}
let dependencies = {}
let factory = new AbstractFactory(dependencies)
let createSuper = factory.create
factory.create = function(moduleName) {
createSuper.call(moduleName)
this.moduleName = 's3-sample'
factory.setDependencies(defaultDependencies)
let dependencies = this.getDependencies()
// Use authentication if binded in forepaas.json
router.use(async (req,res,next)=>{
try {
req.session = await util.promisify(dependencies.authentication.checkSession)(req)
next()
} catch (err) {
next(err)
}
})
// Set our custom endpoint
// Will be available on GET /s3-sample/custom-endpoint
router.get('/custom-endpoint',async (req,res,next)=>{
try {
// Use s3 connector, to get a file in datstore
const dataStoreClient = new AWS.S3({
accessKeyId: accessKey,
secretAccessKey: secretKey,
endpoint: `${endpoint}/datastore`,
region: region,
signatureVersion: 'v4',
s3ForcePathStyle: true
})
dataStoreClient.getObject({
Bucket: bucket,
Key: 'my-file.csv'
})
.createReadStream()
.on('error', (err) => {
next(err)
})
.pipe(res)
} catch (err) {
next(err)
}
})
// Initialise the router for enable endpoint
this.initRouter(router)
return this
}
module.exports = factory
```
***
## Launch the API
You can now run the API, locally or on Data Platform.
Get your endpoint as such:
```
GET API_URL/s3-sample/custom-endpoint
```
If you have enabled authentication (from the main _forepaas.json_ file), you will have to add a queryString parameter with a [valid token from the Identity Access Manager](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md#generate-an-authentication-token)
```
GET API_URL/s3-sample/custom-endpoint?token=1234
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/index.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# ForePaaS Legacy Platform
:::warning
The ForePaaS Legacy Platform no longer accepts new signups. If your company would like to get started on the platform, please use [OVHcloud Data Platform](https://www.ovhcloud.com/fr/public-cloud/data-platform/) and its [documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
OVHcloud Data Platform evolved from the ForePaaS platform. Most concepts carried over under new names, and a few services were replaced or retired. This section groups the documentation that only applies to the ForePaaS Legacy Platform, for customers still operating on it.
## Terminology mapping
| ForePaaS Legacy Platform | OVHcloud Data Platform |
| :-------------------------------------- | :---------------------------------------------------------------------------------------------------------------- |
| Dataplant | [Project](https://docs.dataplatform.ovh.net/manage-projects.md) |
| Data Manager | [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) |
| Query Builder | [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) |
| Storage Engines (PostgreSQL, Snowflake) | Built-in [Trino/Iceberg lakehouse storage](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) |
| PRIM, MART and ML datasets | A single [default dataset](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md), plus custom datasets |
| Machine Learning Manager | Not available on OVHcloud Data Platform |
| ForePaaS Store | [Connector catalog](https://docs.dataplatform.ovh.net/landing-page-connectors.md) and DPE action catalog |
## What you will find in this section
- [Storage Engines](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md): the PostgreSQL and Snowflake warehouse engines of the legacy platform, and [how to link them](https://docs.dataplatform.ovh.net/legacy/lakehouse-manager-settings.md) to a Dataplant.
- [Machine Learning Manager](https://docs.dataplatform.ovh.net/legacy/ml.md): model training, pipelines and notebooks, including the [getting started tutorial](https://docs.dataplatform.ovh.net/legacy/getting-started-ml.md).
- [ForePaaS APIs](https://docs.dataplatform.ovh.net/legacy/api-reference.md): the legacy API references (Query Builder, Data Manager, IAM, Buckets, ForePaaS Store).
- [Account setup](https://docs.dataplatform.ovh.net/legacy/account-setup.md): legacy account and organization management.
- [Query Builder to Analytics Manager migration guide](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md): what changed when queries moved to the Analytics Manager.
- [ForePaaS Store](https://docs.dataplatform.ovh.net/legacy/project-marketplace.md): the legacy collaborative marketplace.
Every page in this section carries a notice linking to the closest equivalent in the current documentation.
---
url: https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Storage engines
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Lakehouse Manager documentation](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
:::
Storage engines are responsible for **”physically” storing the data** of your Project. They are the technological system with which your Project services - such as [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) or [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) - communicate when they need to read or write data.
If you use storage services such as [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md), then you Project is linked to at least one storage engine. It is possible to use multiple storage engines in the same Project: you can link a different one for [every dataset](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) in [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md). It is also possible to use the same storage engine for multiple Projects.
> You must have **storage engine permissions** in order to manage (i.e. create, link to Project, delete) storage engines. These permissions are set at user level in your [organization settings](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-member-permissions-in-your-organization). Even if a user has no permission on storage engines, they are still able to use it and access the data in it from inside the [Project](https://docs.dataplatform.ovh.net/manage-projects.md) - it's only the management of engines that is restricted.
Every storage engine has a technical name that is sometimes used alternatively in API calls. This technical name always starts with _i\__, for example: `i_0941a7f635f1`. This ID can typically be found in the [SQL-string converted by the Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-history.md) from the original user query and used to effectively run the query on the engines.
Storage engines consume [DPU](https://docs.dataplatform.ovh.net/manage-pricing.md) (the platform Units) when they are active. You can choose the size (in DPU) of your engine when you [create it](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md#create-a-storage-engine) or at any time later on, by [editing it](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md#edit-a-storage-engine).
- [Create a storage engine](#create-a-storage-engine)
- [Choose the type of engine](#choose-the-type-of-engine)
- [Choose the cloud provider and location](#choose-the-cloud-provider-and-location)
- [Choose the size of your engine](#choose-the-size-of-your-engine)
- [Choose high availability status](#choose-high-availability-status)
- [Include in your default configuration](#include-in-your-default-configuration)
- [Link a storage engine to a Project](#link-a-storage-engine-to-a-Project)
- [Edit a storage engine](#edit-a-storage-engine)
***
## Create a storage engine
You can create a storage engine instance from the **Storage Engines** service.
> You must have **storage engine permission** `create` in order to create storage engine instances. These permissions are set at user level in your [organization settings](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-member-permissions-in-your-organization).
In both cases, you will need to fill in the following information:
#### Choose the type of engine
the Platform currently lets you choose among two engines.
- **PostgreSQL**: [PostgreSQL](https://www.postgresql.org/) is an open-source relational database powering leading business and mobile applications. PostgreSQL is also very popular as an affordable data warehousing solution with a very good price/performance ratio. PostgreSQL is currently available on _Google Cloud Platform_, _Microsoft Azure_, _Amazon Web Services_ and _OVHcloud_.
- **Snowflake**: [Snowflake](https://www.snowflake.com/) offers a cloud-based data storage and analytics service. Its architecture has been built for the cloud and leverages all the scalability and performances available. Snowflake is currently only available on _Google Cloud Platform_, _Microsoft Azure_ and _Amazon Web Services_.
#### Choose the cloud provider and location
You can choose the cloud provider and location of your storage engine among the following:
- _Google Cloud Platform_ (1 location available): Snowflake and PostgreSQL available
- _Microsoft Azure_ (1 location available): Snowflake and PostgreSQL available
- _Amazon Web Services_ (1 location available): Snowflake and PostgreSQL available
- _OVHcloud_ (1 location available): PostgreSQL available
It is recommended to use the same cloud provider and location for your storage engine as the one chosen for your Projects. This is because by default, storage engine and Projects communicate on private network for better performances and security.
:::warning
If you link a storage engine to a Project that is in another cloud provider or location, it will **permanently open your storage engine instance publically**. The connection will then be encrypted and established on public network.
:::
#### Choose the size of your engine
Storage engines consume [DPU](https://docs.dataplatform.ovh.net/manage-pricing.md) (the platform Units) when they are active. You can choose the size (in DPU) of your engine which will determine the computing power allocated to read and write data in your databases.
For PostgreSQL, the smallest possible size is 2 DPU with _Google Cloud Platform_ and _OVHcloud_, or 4 DPU with _Microsoft Azure_ and _Amazon Web Services_. For Snowflake, the smallest possible size is 16 DPU anywhere.
#### Choose high availability status
High availability is a common paradigm in infrastructure management where **instances are replicated several times in order to avoid interruption of service** in case one of them goes down. Since the storage engines host the databases which are the core of all your Projects, avoiding downtime on them is an essential part of a setting up a production-grade data system.
- **On PostgreSQL**: high-availability is _off_ by default and can be switched on. When it is switched on, the number of instances deployed is doubled (i.e you are invoiced twice as much): if one the instances goes down, the other one stays active for you to use.
- **On Snowflake**: high-availability is _always on_ by default and cannot be switched off. It is an internal feature when using Snowflake and replication is handled internally on their platform to ensure no interruption of service on your end. You are only paying for one storage engine instance.
> Creating a new storage engine can take up to 5-10 minutes, depending on hardware availability. Please make sure your engine has been correctly created and is marked as active ✅ before linking it to a new Project.
***
## Link a storage engine to a Project
#### Sandbox
Each Project features a free **Sandbox** instance. This is a storage engine which is located in the data center of your Project, and shared with other users of this data center. By shared, we mean that the performances are shared. Your data is always protected.
:::warning
The Sandbox instance is intended for proof-of-concepts (POCs), development or test purposes; and is not designed to be used in a production environment as the performances can be inconsistent.
:::
:::info
For the moment, Sandbox is based on the [Snowflake](https://www.snowflake.com/) technology. Essentially, this means that if you choose to use a Sandbox storage engine, any SQL expression used in your Project must be Snowflake-compatible.
:::
#### Dedicated storage engine
As an alternative to the free Sandbox instance, you can either **choose storage engines that you already own** in your organization, or **[create](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md#create-a-storage-engine) a new one**. Any engine can be used with any Project - if the engine is in a different region or cloud provider, the connection will be encrypted and established on public network.
***
## Edit a storage engine
> You must have **storage engine permission** `update` in order to edit storage engine instances. These permissions are set at user level in your [organization settings](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-member-permissions-in-your-organization).
You can edit a storage engine's characteristics from the **Storage Engines** service, by clicking on the **pen** 🖊️ icon of a storage engine.
You can edit the **name, description, tags** and **size** of your storage engine.
***
## Delete a storage engine
> You must have **storage engine permission** `delete` in order to delete storage engine instances. These permissions are set at user level in your [organization settings](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-member-permissions-in-your-organization).
You can delete a storage engine instance from the **Storage Engines** service, by clicking on the **Delete** 🗑️ icon of a storage engine.
This option is only available if the storage engine isn't linked to any [project](https://docs.dataplatform.ovh.net/manage-projects.md) in your organization. [Deleting a project](https://docs.dataplatform.ovh.net/manage-projects-delete.md) unlinks all storage engines from it.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/lakehouse-manager-settings.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Settings
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Lakehouse Manager documentation](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
:::
The Lakehouse Manager settings let you manage the [storage engine(s)](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md) linked to your Project, i.e. the system storing all the structured data you bring in your Project.
- [Link / unlink a storage engine](#link-unlink-a-storage-engine)
***
## Link / unlink a storage engine
A storage engine is said to be **linked** to a Project if the Project is able to connect to it (through an automated generation of credentials).
The list of storage engines currently linked to a Project are displayed in the "Linked storage engines" table:
The possible status for a linked storage engine are:
- _linking_: the storage engine instance is currently being linked to the Project
- _unlinking_: the storage engine instance is currently being unlinked from the Project
- _failed_: the storage engine instance has failed to link to the Project. Please delink it and link it again
- _updating_: the storage engine instance is linked but is currently being updated (for ex: horizontal or vertical resizing). Availability may be reduced
- _terminated_: the storage engine instance is being deleted
- green checkmark: the storage engine instance is successfully linked to the Project and fully available
### Link a storage engine
You can only link storage engines that are part of your [organization](https://docs.dataplatform.ovh.net/manage-organizations.md). To link a new storage engine, click on **Link storage engine**
You will see the list of non-linked storage engines that are in your organization. Select one or more, and press **Link** to link them to the Project.
> The linking process can take up to a minute.
You can also [create a new storage engine](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md#create-a-storage-engine) by pressing **Add new**. They will be added to the list of storage engines of your organization.
### Unlink a storage engine
!> You can only unlink a storage engine if no [table](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) is using it.
To unlink an engine, press the **unlink** icon next to the engine.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/account-setup.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Profile and organizations
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [Organizations and user profiles](https://docs.dataplatform.ovh.net/manage-organizations.md).
:::
Creating your own user profile is the first necessary step in order to use ForePaaS. It can be done in under 5 minutes and is **completely free**!
You can then create or join an **organization**: it is a workspace shared with your teammates to work on projects. Once you are part of an organization, you can create [Dataplants](https://docs.dataplatform.ovh.net/manage-projects.md) which are dedicated environments where you can carry out specific data projects. Each Dataplant usually represents one project.
> 💡 ForePaaS charges at the organization level through the purchase of a subscription plan.
***
## Create a ForePaaS account
Join your team on ForePaaS by creating yourself an account on the platform. It is completely free!
[Create your ForePaaS profile!](https://docs.dataplatform.ovh.net/legacy/account-setup-create-account.md)
***
## Create or join an organization
An organization represents a company or a specific business unit. It is where you can add all your teammates to work on multiple data projects.
[Learn more about organizations!](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-create.md)
***
## Activate a subscription for your organization
**ForePaaS doesn't charge per user or Dataplant, but for an organization's subscription plan and consumption**. You can purchase a subscription for your organization by yourself through your [Billing Account](https://docs.dataplatform.ovh.net/manage-pricing.md).
[Get started with a subscription!](https://docs.dataplatform.ovh.net/manage-pricing.md)
***
## Create your first Dataplant
A Dataplant is like a mini-factory to build AI projects and applications. It contains all the necessary components from raw extraction to restitution. Dataplants individually have a set of dedicated resources that can be allocated freely depending on the needs of your project.
[Create your first Dataplant!](https://docs.dataplatform.ovh.net/index.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/account-setup-create-account.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a ForePaaS account
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [Organizations and user profiles](https://docs.dataplatform.ovh.net/manage-organizations.md).
:::
!> We have temporarily restricted signups on the ForePaaS Platform. Please reach out to our support team if you have any question.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/account-setup-edit-account.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Edit your ForePaaS account
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [Organizations and user profiles](https://docs.dataplatform.ovh.net/manage-organizations.md).
:::
At any time, you have the possibility to view and edit your user profile from the platform by clicking on [Profile](https://hq.forepaas.io/#/user) at the top right.
### Preferences
The **Preferences** tab lets you edit basic user information on your profile.
You can also see the list of organizations you are part of, as well as organizations where you might have potential teammates based on your email domain. You can request to join those organizations by clicking on the corresponding **Request to join** button. _Admins_ of this organization will receive an email and will have to [manually approve your application](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#accept-or-deny-requests-to-join-your-forepaas-organization).
### Security
The **Security** tab lets you change your password, manage multi-factor authentication and track recent logins.
It also enables you to manage _linked accounts_, i.e. the single-sign-on (SSO) accounts which you use to log into ForePaaS.
---
url: https://docs.dataplatform.ovh.net/legacy/account-setup-organization-create.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create or join an organization
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [Organizations and user profiles](https://docs.dataplatform.ovh.net/manage-organizations.md).
:::
Organizations are the compartmentalized teams that link individual members, Dataplants and plans together. You can have as many teammates in your organization as you want.
Your organization can work on multiple data projects at the same time. Each project is carried out inside a [Dataplant](https://docs.dataplatform.ovh.net/manage-projects.md): you can create as many as you want.
> To create or join an organization, make sure that you already have a [ForePaaS user account](https://docs.dataplatform.ovh.net/legacy/account-setup.md)
***
## Create an organization
Go to [hq.forepaas.io](https://hq.forepaas.io).
### If you are not in an organization
If you are not part of an organization already, you will be able to create a new one:
### If you are already in an organization
If you are already in an organization, click on the drop-down menu located at the top left and then on **New organization**.
Fill in the name and press **Done**.
***
## Join an existing organization
To join an organization that already exists (if your team already has one, for example), you need to ask someone already on there to invite you.
[Learn how to invite members](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md)
You will receive an email with a clickable link inviting you to join their organization.
***
## Manage your organization settings
You can manage your organization by opening the drop-down menu at the top right and by clicking on **Organization**.
From there, you will be able to configure:
- Your organization members and permissions
- Your Dataplants
- Your datasets
- Your billing settings
- Other parameters: Name, description, SSL certificates, domain names, etc.
[Learn how to configure an organization](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-settings.md)
---
url: https://docs.dataplatform.ovh.net/legacy/account-setup-organization-settings.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Edit organization settings
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [Organizations and user profiles](https://docs.dataplatform.ovh.net/manage-organizations.md).
:::
To open the settings of an organization, navigate to the top right of the screen and hover over the **badge 💳 icon**. Click on **Organization**.
***
## Overview
Landing on your organization dashboard, you have access in the blink of an eye to the main information of your organization on ForePaaS:
- Number of members and admins
- Dataplants activity
- Resource consumption
You can also see the main metrics usage of your Dataplant(s).
***
## Members
Manage your organization's members, add a new member or edit access rights.
:::warning
These are your organization's members, which are different from the Dataplant's users you'll find in the Identity Access Manager (IAM).
:::
[Manage organization members](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md)
[Manage Dataplant users in the IAM](https://docs.dataplatform.ovh.net/landing-page-iam-users.md)
***
## Dataplants
The Dataplants tab gives you an overview and direct access to all the [Dataplants](https://docs.dataplatform.ovh.net/manage-projects.md) created inside the organization you're currently in.
***
## Storage
The Storage tab gives you an overview and direct access to all the **storage engines** that have been added in your organization.
[Learn more about storage engines](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md)
***
## Quotas
:::info
This feature is coming soon!
:::
Quotas are restrictions that are set automatically for all organizations. They constitute safeguards against accidental over-consumption and are imposed by all cloud providers.\
Think of them as an equivalent of the quotas your bank can impose on the maximum withdrawal or payment amount per week.
> You cannot modify quotas by yourself. To do so, submit your request on the quotas page. A ForePaaS expert will get back to you shortly!
***
## Plan
The Plan tab is where you can **manage your subscription plan**. It shows all the features included in your current plan, as well as the option to upgrade it.
- In _Billing_, you can get the details of your next invoice with a breakdown of the month-to-date balance per cost category.
- In _Receipts_, you can download past invoices as PDFs.
[Learn more about billing and pricing](https://docs.dataplatform.ovh.net/manage-pricing.md)
***
## Settings
The Settings tab is the place where you'll manage:
- **basic information** about your organization: name, description, multi-factor authentication requirements, and plan cancellation. This is also where you can manage the option to let users with the same email domain as people in your organization to request to join it.
- **infrastructure** options: domain name and certificates. You can customize domain names for your applications deployed on ForePaaS, and create SSH certificates to access your secure domain names.
Adding your certificate, you'll need a server certificate (.crt file) or key (.key).
---
url: https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Add members to an organization
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [Organizations and user profiles](https://docs.dataplatform.ovh.net/manage-organizations.md).
:::
On the ForePaaS Platform, you can have as many people as you want in your organization. Those members can have full or restricted access to the projects (i.e. [Dataplants](https://docs.dataplatform.ovh.net/manage-projects.md) and [storage engines](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md)) in the organization.
:::info
To manage access control to the components or the applications inside a specific Dataplant, you need to use this Dataplant's [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md).
:::
- [How to invite a member in your ForePaaS organization](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#invite-members-in-your-organization)
- [How to accept/deny requests to join your ForePaaS organization](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#accept-or-deny-requests-to-join-your-forepaas-organization)
- [How to manage member permissions](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-member-permissions-in-your-organization)
***
## Invite members in your organization
### Organization member list
Go directly to [ForePaaS](https://hq.forepaas.io) and access the organization for which you want to add members by clicking on the drop-down menu at the top left.
Then open your organization by clicking on **Organization** at the top-right.
Click on the _Members_ tab. You should be able to **see the list of members within the organization** as well as yourself.
Click on _+ Add Member_ button to invite a collaborator.
### Fill in member information
Fill in the new user's **email address** regardless of whether this person already has a ForePaaS account:
- If the new collaborator _doesn't have a ForePaaS account_, they will receive an invitation to sign-up and log-in to the platform. They will be automatically added to the organization after signing-up and validating the email.
- If the new collaborator _has a ForePaaS account_, they will simply be prompted to join the organization
> 💡 You can add multiple members with the same permissions in one go by adding multiple addresses in the **email(s)** field
You will need to choose the user's **role**. There are 2 organization roles:
- **Admin**: Admins have **all access & control rights** on all elements of an organization. They can therefore edit, manage or delete Dataplants, organization members or even change the subscription plan of an organization. They will also de-facto have access to all the services of the Dataplants deployed in the organization i.e. the DPE, Data Manager, etc.
- **User**: Users have by default **no access or control rights**. By default, users will not even see Dataplants contained in the organization. They will only be able to access the organization and its details such as its users, plan or quotas. You can assign specific rights to users using [access control rules](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-user-permissions-in-your-organization).
***
## Accept or deny requests to join your ForePaaS organization
It is possible for people with the **same email domain as any existing member** in an organization to request to join this organization when they first sign up. _Admins_ of the organization will have to manually accept the person into the organization after they requested to join.
> This feature can be deactivated in your [organization's settings](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-settings.md) if your [pricing plan](https://docs.dataplatform.ovh.net/manage-pricing.md) is Premium or above.
To manage requests, click on the _Members_ tab. At the top of the list of members in your organization, you will see all the people that have requested to join your organization.
You can accept or deny their request from here. If you accept, they will be added in your organization as simple _users_ with no permission. You can then assign specific rights to them using [access control rules](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#manage-user-permissions-in-your-organization).
***
## Manage member permissions in your organization
By default, _Admins_ have full access to the organization settings as well as its projects. For simple _Users_, there are 3 main categories of rules that you can manage at the organizational level:
- [Dataplants access rights](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#dataplants) - Allow users to get specific control rights to a [Dataplant](https://docs.dataplatform.ovh.net/manage-projects.md) or its infrastructure.
- **Storage config. rights** - Allow users to update the default configuration of your [storage engines](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md)
- [Storage Engines rights](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-invite-user.md#storage-engines) - Allow users to get specific control rights to [storage engines](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md) in your organization
### Creating rules
To create a rule, simply navigate to the category of interest and click _+ Add new rules_. Inside the rule editor window, follow the instructions.
For instance in the example below, the rule is granting all access & control (Action set to \*\*\*\*\*, i.e. all) to the Dataplant _demonstrator_ (Attribute set to _demonstrator_) to the user.
:::warning
Rules are additive. In other words, they "stack" on top of each other. To give access to 2 Dataplants, you can create 2 separate rules which individually grant access to these Dataplants.
:::
:::warning
The order of rules matters. If two rules overlap (at least one service is affected by both rules), the rule that is processed last wins.\
:::
_Example:_ If a rule grants access to all Dataplants (say rule 1) and one blocks access to one of them (say rule 2):
- If rule 1 is listed after rule 2, then rule 1 will be processed last and the user will have access to all Dataplants.
- If rule 1 is listed before rule 2, then rule 2 will be processed last and the user will have access to all Dataplants but the one restricted by rule 2.
### Dataplants
These permissions are the most important and allow you to define the rights of your user to access your [Dataplants](https://docs.dataplatform.ovh.net/manage-projects.md) and your different Custom Domains (if you have any). It is possible to modify these rights per Dataplant.
- **Read**: See the Dataplant(s) of the organization
- **Create**: Create a Dataplant
- **Update**: Change the amount of DPU allocated to the Dataplant(s)
- **Delete**: Delete the Dataplant(s)
- **Deploy**: Deploy an application or a worker on the Dataplant(s)
- **Logs**: Access to the monitoring and logs of the different components (Data Processing Engine, Query Builder, etc.) present in the Dataplant(s).
> These permissions are high-level read/write access to Dataplants and their infrastructure. To manage access control to the applications or the components inside a specific Dataplant, such as data processing pipelines or buckets, you need to use this Dataplant's [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md).
### Storage Engines
These permissions allow you to define the rights of your user to edit your [storage engines](https://docs.dataplatform.ovh.net/legacy/project-storage-engine.md).
- **Create**: Create a storage engine
- **Read**: See the storage engine(s) of the organization
- **Update**: Change the amount of DPU allocated to the storage engine(s)
- **Delete**: Delete the storage engine(s)
### Validation
Once you're satisfied, simply click on **Save** and the user(s) will receive an email invitation to join the organization. It might happen the user does not receive the email or forgets to join, if that's the case you have the option to resent the invitation at any time by clicking on the paper plane icon:
> Note that members who have not accepted your invitation to join an organization will be marked with a **status** ❌. All members who have successfully accepted your invitation and joined will be marked as ✅.
***
Check out the following article to learn how to manage the other aspects of your organization such as its subscription plan, billing information, or quotas.
[Set up your organization](https://docs.dataplatform.ovh.net/legacy/account-setup-organization-settings.md)
---
url: https://docs.dataplatform.ovh.net/legacy/ml.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Machine Learning Manager
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The Machine Learning Manager lets you **train and deploy AI models into production**. It offers different ways to get started with machine learning. You can either start by experimenting on [notebooks](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md), train prediction models from scratch in [pipelines](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md), or even deploy an externally trained [model](https://docs.dataplatform.ovh.net/legacy/ml-models.md) file.
***
## Component overview
The Machine Learning Manager enables you to **put models into production for AI-powered applications**. Like all other components on ForePaaS, it is based on the containers and micro-services technology. This means that individual ML jobs including training, scoring and predictions can be scaled to adapt to a large volume of processed data and high-frequency data throughput.
Once it is created, any model can either be used:
- in the **Data Processing Engine (DPE)** to write predictions in your ForePaaS data model.
- as an **external API**.
Inside the Machine Learning Manager you will be able to perform the following actions:
- [Create notebooks](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md) and use them to explore your data
- [Set up pipelines to train new ML models](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md) from scratch
- [Compare & deploy models](https://docs.dataplatform.ovh.net/legacy/ml-models.md) as well as uploading and sharing existing models leveraging a public/private marketplace
***
## Quickstart tutorial
Getting started with machine learning can be a little tedious for first-timers. Fortunately, we have prepared a simple step-by-step tutorial with sample data to learn how to use our ML Manager!
[Check-out the ML 🤖 tutorial!](https://docs.dataplatform.ovh.net/legacy/getting-started-ml.md)
***
## Three ways to get started in the Machine Learning Manager
### Notebooks
Discover our integration of Jupyter notebooks which lets you deep-dive in your data without having to bother for the computing performances.
[Create Jupyter notebooks on ForePaaS](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md)
### Pipelines
Use pipelines to design and deploy a machine learning model from scratch. From the data preparation to the validation, everything is made simple for you to focus on the business results!
[Train models with pipelines](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md)
### Models
If you already have trained models on hand 🦾 and want to use ForePaaS' technology to maintain them in a versioned production environment, check out the link below:
[Deploy 🚀 trained models with ForePaaS](https://docs.dataplatform.ovh.net/legacy/ml-models.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Notebooks
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Notebooks are a built-in integration of [Jupyter Notebooks](https://jupyter.org/) on ForePaaS. Coming with pre-loaded packages for data science, they allow you to perform non-real-time interactive statistical analysis and data visualization. Notebooks are highly connected to the rest of your Project, making it possible to read/write data in the Data Manager thanks to [our SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) or convert your notebook code into a [Data Processing Engine action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) or a [Machine Learning Manager pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md).
- [Create and delete notebooks](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#create-and-delete-notebooks)
- [Open a notebook](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#open-a-notebook)
- [Edit the settings of a notebook](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#edit-the-settings-of-a-notebook)
- [Export a notebook into ForePaaS](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#export-a-notebook-into-forepaas)
***
## Create and delete notebooks
You can manage your **notebook instances** in the corresponding tab in the Machine Learning Manager. A notebook instance is a Jupyter environment with specific resources and dependencies.
A notebook instance can contain several notebook documents, i.e. _.ipynb_ files, which you can manage directly from the JupyterLab interface.
From the Notebooks home page, you can **create, work on, delete and edit the settings** of all your notebooks.
:::info
On the ForePaaS Platform, the word _notebook_ is used to talk about notebook instances.
:::
[Learn how to create a notebook](https://docs.dataplatform.ovh.net/legacy/ml-notebooks-create.md)
You can delete a notebook by clicking on the 🗑️ icon. Deleted notebooks cannot be recovered.
***
## Open a notebook
Notebook instances are **automatically turned on** when you open them.
The notebook instance will consume [all assigned DPU](https://docs.dataplatform.ovh.net/manage-pricing.md) for the duration it is turned on.
You can manually turn off a notebook by clicking on the _Stop_ button when it is active.
To open a notebook, **double click** on it, or press the button shown below. This will open the notebook in a new JupyterLab window.
The [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/) interface allows you to work on your notebooks with all the features provided by Jupyter, such as having multiple _.ipynb_ files open at the same time, keyboard shortcuts, download and export options, and much more.
> Use the [ForePaaS SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md) to easily interact with the rest of your Project, typically to **import/process/export data** from and to your [Data Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
***
## Edit the settings of a notebook
To edit the settings of a notebook, click on the **gears** icon.
You can edit the following options:
- [Edit dependencies](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#edit-dependencies)
- [Edit resources](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#edit-resources)
### Edit dependencies
You can edit the packages and dependencies of your notebook environment, including the Python version. The notebook will have to be restarted for any modification to be applied.
All notebooks have an [instance image](https://docs.dataplatform.ovh.net/legacy/ml-notebooks-create.md), which comes with some default packages and **Python 3.9**. You can specify additional dependencies by clicking on the **Add** button in the _Dependencies_ panel.
Simply choose the type of dependency and enter the name of the package to install in the server, then press Enter.
You can add several packages at once by pressing _Enter_ every time, like tags.
### Edit resources
This is where you can edit the [resources](https://docs.dataplatform.ovh.net/manage-pricing.md) you want your notebook to use. The notebook will have to be restarted for any modification to be applied.
***
## Export a notebook into ForePaaS
You can also export the entirety of the notebook's content to an object in ForePaaS. This will export all the code in your cells as it is and will turn any markdown cells into comments.
:::info
Notebook exporting is only available with Python version 3.9.7. Please select that version on the notebook's settings, otherwise you won't be able to export your notebook.
:::
There are three options regarding where to export your notebook to:
- [Custom Action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md), in the Data Processing Engine.
- [Custom Estimator](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md), in the Machine Learning Manager.
- [Custom Scoring Function](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#custom-scoring-function), in the Machine Learning Manager.
To export your notebook, open it and click on _File_ and then on _ForePaaS export_.
This will open an export widget allowing you to choose between the three export options. Choose your preferred option and click on Next.
Choose the repository and the repository version where to export the notebook. If are not familiar with repositories, simply leave it at the _Default_ repository with version _v1_.
Then, you will be required to decide if you want to export to an existing object or create a new one. If you create a new one, you also have to choose the name and the framework to be used.
Finally, click on confirm to export the notebook.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-notebooks-create.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a notebook
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
To create a notebook instance, head to the **Notebook** section of the Machine Learning Manager.
Click on **New notebook** in the top-right.
You have to choose a [container image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html) for your Jupyter environment, i.e. the set of packages and dependencies you want to have for your new notebook. ForePaaS features several ready-to-run images which contain the most popular libraries for data analysis and data science.
> If you are new at creating notebooks, we recommend starting off with a _Data Science notebook_. It contains all the basic Python libraries for data exploration such as pandas, scipy, scikit-learn, matplotlib and much more!
You can then configure your notebook further before it is created (and it is still possible to change those settings later on). Once you press **Create**, the server for the notebook will be initialized - with its specific environment and dedicated computing resources.
The two main aspects to configure are:
- [Additional dependencies](https://docs.dataplatform.ovh.net/legacy/ml-notebooks-create.md#notebook-dependencies)
- [Resources](https://docs.dataplatform.ovh.net/legacy/ml-notebooks-create.md#notebook-resources)
***
## Notebook dependencies
Your notebook's image already comes with some default packages, which you can individually check in the image description in the marketplace.
You can specify additional dependencies by clicking on the **Add** button in the _Dependencies_ panel.
Simply choose the type of dependency and enter the name of the package to install in the server.
You can add several packages at once by pressing _Enter_ every time, like tags.
***
## Notebook resources
Check out the dedicated sections to learn how to configure your new notebook's resources. It is suggested to configure them before confirming the creation of the notebook.
[Configure a notebook's resources](https://docs.dataplatform.ovh.net/legacy/ml-notebooks.md#edit-resources)
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Pipelines
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Pipelines are the feature that enables you to **create a machine learning model from scratch and deploy it to make predictions**. It is a production-grade end-to-end environment in which a model evolves through its [life-cycle](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure.md#manage-execution-options) in a traceable way.
> Pipelines can store several trained models and their versions, but only one can be deployed to make predictions.
- [Create a pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md#create-a-pipeline)
- [Manage pipelines](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md#manage-pipelines)
- [Configure a pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure.md)
- [Execute a pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md)
***
## Create a pipeline
To create a pipeline, head to **Pipelines** in the ML component.
If you haven't already started a _machine learning Project_, start by [initializing a repository](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md#pipeline-repositories). Else simply click on **New Pipeline** at the top-right.
Wait a few seconds for your pipeline to be built. After being built for the first time, pipelines are in auto-save mode. You can exit and come back to the configuration at any time.
[Configure a pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure.md)
***
## Manage pipelines
### Pipeline repositories
Pipelines are stored in ForePaaS repositories. You can initialize a new repository by pressing the ➕ sign in the Pipelines tab.
You can edit the name and description of the pipeline, as well as connect it to an existing Git repository.
### Edit pipelines
Pipelines can be configured from the repository list by clicking on the **pen 🖊️ icon**, or by double-clicking the pipeline.
Pipelines can be renamed when configuring them, by double-clicking on their name at the top-right.
Pipelines can be deleted from the repository list by clicking on the **trash 🗑️ icon**.
:::warning
Deleting a pipeline does not delete its saved models. Models that were generated by a deleted pipeline are still accessible on [the Models tab](https://docs.dataplatform.ovh.net/legacy/ml-models.md).
:::
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure a pipeline
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Each pipeline can be configured from the data input to the deployment options in order to create accurate production-grade prediction models.
***
## The five configuration steps
There are five core steps to setting up a pipeline on ForePaaS.
1
[## Dataset preparation
Specify everything related to the train and test datasets of your model.
](/legacy/ml-pipelines-configure-dataset)
2
[## Training procedure
Set up the estimator at the basis of your model as well as the scoring and validation options needed to fine-tune it.
](/legacy/ml-pipelines-configure-training)
3
[## Hyper-parameter tuning
Fine-tune the hyper-parameters of your estimator using our intuitive and insightful studio.
](/legacy/ml-pipelines-configure-tuning)
4
[## Model selection
Compare models that have been trained since the creation of your pipeline and decide which to deploy.
](/legacy/ml-pipelines-configure-validation)
5
[## Deployment settings
Specify the various settings for your production-grade inference service
](/legacy/ml-pipelines-configure-deployment)
***
## Manage execution options
Once a pipeline is set up, you can manage various execution options to ensure a reliable life-cycle of your models in a production environment. Learn more about the execution of a pipeline and the life-cycle of a model below:
[Manage a pipeline's execution](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data preparation
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Data preparation is the first step to any machine learning process. It is the step during which you will **specify your data source and the rules according to which it is turned into ML datasets**.\
ML datasets are modeled and stored as [tables that are automatically created in the Data Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) and fed with data from the source specified at the entry of your pipeline. Specifying the rows and columns of these tables is the gist of the data preparation step.
:::warning
In order to better orchestrate data versioning, data can only be added to an ML dataset. You have the option to manually reset your ML datasets in your [pipeline preferences](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#reset-datasets).
:::
Data preparation is usually the first step you need to parametrize since the rest of the pipeline should be designed according to this choice. To enter data preparation, click on **Dataset** on your pipeline's main page.
***
## Dataset generation
You can connect data from multiple sources - either within or outside of ForePaaS - at the entry point of your pipeline. This data must then be split into a train and a test dataset for the model to be fitted and evaluated.
[Manage source and train-test split](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md)
***
## Feature engineering
Good data preparation means selecting the correct features for your model. Learn how to add variables to your X and Y sets, and how to execute basic checks on the data content.
[Manage your features](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-features.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Dataset generation
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The dataset generation consists in **choosing an data input** and then **splitting that data into a train and a test dataset**. When doing production-level AI, this split occurs continually over time as new [Dataset Generation jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#dataset-generation-jobs) are launched (manually or through triggers).
***
## Choose a data input
ForePaaS supports several input types:
- [Data Manager Tables](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#tables)
- [Buckets](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#pictures-from-buckets)
> We are working hard to add more integrations to ML pipelines. If we are missing something critical for your Project, please reach out and drop us a line on [our Roadmap Portal](https://hq.forepaas.io/#/features)! 💡
### Tables
The _Tables_ input type lets you choose from the **tables in your Data Manager**. You can choose from raw, prim and mart tables.
### Pictures from buckets
The _Pictures from buckets_ input type lets you choose from the [buckets in your Data Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). This makes it the perfect choice to work with unstructured data, especially pictures.
:::info
This input type currently only accepts files with EXIF data and files that are considered as an image by [the pillow library](https://pillow.readthedocs.io/en/5.1.x/handbook/image-file-formats.html).
:::
To enable unstructured data training, the images must first be uploaded to the [Data Manager's buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) in **a folder named after each class**. Using the [Cats and Dogs](https://www.tensorflow.org/datasets/catalog/cats_vs_dogs) image classification dataset as an example, the images will be uploaded into two folders in a bucket. One named `cats` (so that the path is _cats/my\_images.jpg_) the other `dogs` (so that the path is _dogs/my\_images.jpg_).
***
## Train-test split
After choosing an origin for your data, you need to make it usable by machine learning algorithms by splitting it into a training and a testing set. The nature of the sets depends on whether you are working with structured _(in which case it will be tables of data)_ or unstructured data _(in which case it will be filepaths)_.
:::info
**Training set**: Sample of data used to fit (i.e. train) the model.\
**Testing set**: Independent sample of data used to provide an unbiased assessment of a fully trained model. Training and testing sets must remain disjoint over time.
:::
> On ForePaaS, training and testing sets are modeled and stored as [ML tables in the Data Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
In a pipeline, you can choose to use either [the same source](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#use-the-same-source) or [two separate sources](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#use-different-sources) for your training and testing datasets.
### Use the same source
Under Training, select the desired data source.
Then activate the **link 🔗 icon** in between Training and Testing to use the same source for your training and testing sets.
The next step is to decide how to split the data between Training and Testing. Use the slider to choose the share of data points amount (rows for Tables, files for Buckets, etc.) that each set should have **at the end** of the [Dataset Generation job](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#dataset-generation-jobs).
You can also control the method used to execute the split in the Data Set panel with the **Split Type** option. _Random_ will assign each entering data point randomly in either the training or testing set so that the desired train-test ratio is respected. Specifying a **seed** for the pseudo-randomization will enable you to reproduce randomness.
:::warning
Remember that data points cannot be individually removed from a testing ML dataset even if they are removed from your source. You have the option to manually reset your ML datasets in your [pipeline preferences](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#reset-datasets).
:::
### Use different sources
Under Training, select the desired data source.
Make sure to turn off the **link 🔗 icon** in between Training and Testing to use the same source for your training and testing sets.
Under Testing, select the desired data source.
> With different sources you don't need to specify a train-test-split method since the split is already done! Each entering data point will be assigned accordingly either to the training or the testing set.
:::warning
Remember that data points cannot be individually removed from a testing ML dataset even if they are removed from your input. You have the option to manually reset your ML datasets in your [pipeline preferences](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#reset-datasets).
:::
After specifying the rows of your datasets, you need to specify their columns.
[Select your model's features](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-features.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-features.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Feature engineering
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
ForePaaS enables you to fully specify which variables to select for your model.
***
## Add a value-to-predict
The value you want to predict (i.e. the label) is saved in the _Y dataset_. To add a value-to-predict, click on **Add a source** next to Y.
Choose the source based on the **input type** you specified when [generating your dataset](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#choose-a-data-input) and save.
#### When the input type is _Tables_
You will be able to select one attribute from the training table.
#### When the input type is _Pictures from buckets_
You will be able to select [one or several folders](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#pictures-from-buckets) from the training bucket.
***
## Add features
The features, or variables, that are used to make a prediction are saved in the _X dataset_. To add features to your structured data model, click on **Add a source** next to X.
Choose as many variables as you want from the source based on the input type you specified when [generating your dataset](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#choose-a-data-input) and save.
> Shortcut tip: you have the option to _add all variables_ in one go: this will add all the variables from the source **except** the one you chose as the value-to-predict.
***
## Pre-processing
Advanced feature engineering should be carried out in the [Data Processing Engine (DPE)](https://docs.dataplatform.ovh.net/landing-page-dpe.md) on your source itself.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Training procedure
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The training procedure is at the core of every machine learning model. It defines the mathematical algorithm (i.e. estimator) at its basis as well as the set of rules used to **fit and evaluate it**.
Training is usually the second step in configuring a pipeline. Coupled with the third step, [hyper-parameters tuning](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md), your pipeline will be able to produce trained models for you to compare and deploy.
Training jobs [can be accelerated by a lot using GPUs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#use-gpu-for-your-processing).
***
## Train with a standard estimator
Give your pipeline configuration a quick start by using one of the standard estimators from a curated list from the most popular ML libraries!
[Use a standard estimator](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-standard-estimator.md)
***
## Train with a custom estimator
Can't find what you want on the ForePaaS Store? Carry out your training procedure exactly the way you want it by importing your own custom estimator.
[Use a custom estimator](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md)
***
## Scoring and validation
Scoring and validation are necessary aspects of the training step in order to evaluate models in an unbiased way. Learn how to configure it right.
[Configure scoring and validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-standard-estimator.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Standard estimators
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Pipelines let you choose from the most popular machine learning algorithms in the industry. ForePaaS currently supports the following frameworks:
- [Scikit Learn](https://scikit-learn.org/stable/index.html)
- Keras _(coming soon!)_
> If you cannot find the algorithm of your choice among the standard estimators, you can also upload your own script using the **custom estimator** option.\
> [Learn more about custom estimators.](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md)
***
## Add a standard estimator
Selecting _Standard estimator_ will open the store catalog where you can choose the algorithm you want to use for your model.
Depending on your use case, you need to choose between a [classifier](https://en.wikipedia.org/wiki/Statistical_classification) and a [regressor](https://en.wikipedia.org/wiki/Regression_analysis). You can know what category is each algorithm by looking at their tags.
***
## Change your pipeline's estimator
You can change the estimator at the core of your pipeline by clicking the **Change** button on the Training page.
:::warning
Changing the estimator of your pipeline will completely erase the configuration for the [Training](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training.md) and [Tuning](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md) steps, including all previous hyper-parameter combinations generated in the studio. It will **not** delete saved models generated by the previous estimator.
:::
***
## Manage libraries and dependencies
The Dependencies panel lets you manage all packages requirements for the execution of your pipeline.
[Libraries and dependencies](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md)
***
## Configure scoring and validation
After choosing the base algorithm at the core of your model, you need to be able to evaluate it in an unbiased fashion. Learn how to set validation and scoring functions:
[Configure scoring and validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Custom estimators
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
If you cannot find what you want among the standard estimators, or if you already have a training script on hand, ForePaaS allows you to upload your own estimator to be fitted on your pipeline's data. This can be done either by opening the Training configuration page for the first time, or by [changing the algorithm](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-standard-estimator.md#change-your-pipeline39s-estimator) if there is already one.
It is necessary to choose a framework for your pipeline. ForePaaS currently supports the following frameworks, details of which can be found on their respective pages:
- [Scikit Learn](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-scikit-learn.md)
- [Keras](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-keras.md)
- [PyTorch](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-pytorch.md)
***
## Common features from SDK
Some features provided by the Machine Learning Manager have a common behavior for any custom estimator written in Python, and require you to **input additional elements of code in your training script**. The following configurations can be integrated:
- [Use a validation configuration from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-configuration)
- [Use a validation score function from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-score-function)
- [Use hyper-parameters specified in the Tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-hyper-parameters)
### Use a validation configuration
You can connect the [validation settings](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md) configured during the Training step.
If using simple validation, you can get the share allocated to the validation set by including:
```py
from forepaas.core.settings import ML_CONFIG
# Get ratio of the % size of the validation set
ratio = ML_CONFIG["train"]["ratio"]
```
If using cross-validation, you can get the number of folds chosen by including:
```py
from forepaas.core.settings import ML_CONFIG
# Get cv as either the number of folds, or False if cross-validation is turned off.
cv = ML_CONFIG["train"]["execution_options"]["cross_validation"]
```
An example for Scikit Learn can be found [here](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-scikit-learn.md#sample-code).
### Use a validation score function
You can connect the [scoring functions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md) added during the Training step in order to use them for the validation score.
You can get the default ⭐ scoring function and turn it into an sklearn scorer by including:
```py
from forepaas.ml import get_train_scoring_function
from sklearn.metrics import make_scorer
scoring_function = get_train_scoring_function()
```
An example for Scikit Learn can be found [here](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-scikit-learn.md#sample-code).
### Use hyper-parameters
You can connect the [hyper-parameter list](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md) specified in the Tuning step.
To get the dictionary of hyper-parameters and their lists of values, include:
```py
from forepaas.ml import get_hyper_parameters
hyper_parameters = get_hyper_parameters()
```
An example for Scikit Learn can be found [here](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-scikit-learn.md#sample-code).
[Import a custom Scikit Learn script](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-scikit-learn.md)
[Import a custom Keras script](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-keras.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-scikit-learn.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Import a custom Scikit Learn script
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The Scikit Learn framework allows you to upload a Python _.py_ file containing an estimator **compatible with the Scikit Learn library**. The supported libraries typically include (but aren't restricted to): [Scikit Learn](https://scikit-learn.org/), [XGBoost](https://xgboost.readthedocs.io/en/latest/), [lightgbm](https://lightgbm.readthedocs.io/en/latest/), ...
When you select this framework, some packages are imported by default in the pipeline's environment:
- scikit-learn
- the libraries [that come out of the box with ForePaaS' SDK](https://forepaas-sdk.readthedocs.io/en/latest/index.html)
Any other library used by your custom script has to be specified in the **Python libraries** field of the [Dependencies panel](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md) (along with its version, if you wish to not use the latest one).
> 💡 You have the possibility to load a _requirements.txt_ file in order to [add all necessary libraries in one go](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md#import-a-requirementstxt-file).
By default, new pipelines will be using **Python 3.9.9**.
### Upload a custom estimator
To upload a custom estimator, either drag your _.py_ file into the Training Script box or click the box to open the file explorer.
When a [Training job is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs), this _.py_ file is the file that will be executed. It must contain a function that has the following two requirements:
- It must have `event` as its first argument
- It must return a fitted estimator based on the training dataset
This function's name must be written down in the _function name_ box.
Below is a sample code of a basic custom Scikit Learn estimator:
```py
from sklearn.ensemble import RandomForestClassifier
from forepaas.ml import get_train_dataset
def my_random_forest(event):
# Retrieve train dataset.
x, y = get_train_dataset()
# Fit estimator.
clf = RandomForestClassifier(n_estimators=200, max_depth=9)
clf.fit(x, y)
# Return fitted estimator.
return clf
```
> With the above code, you would have to specify _my\_random\_forest_ as the **function name**.
While the previous code will not be able to use any of the configurations made in the Training and Tuning steps, you have the possibility to connect to other parts of your pipeline configuration from your estimator thanks to the [ForePaaS SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md). This allows you to use all the features provided by ForePaaS pipelines with your custom estimator.
The following configurations can be integrated:
- [Use a validation configuration from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-configuration)
- [Use a validation score function from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-score-function)
- [Use hyper-parameters specified in the Tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-hyper-parameters)
!>
Since only the custom _.py_ script is executed during Training jobs, failing to integrate the aforementioned configurations into the Python script will result in any respective interface-originated modification to be ignored by the pipeline.
### Sample code
This example combines all three aforementioned connectors:
```py
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from forepaas.core.settings import ML_CONFIG
from forepaas.ml import get_train_dataset, get_hyper_parameters, get_train_scoring_function
def my_random_forest(event):
# Retrieve train dataset. No need to split it since it will be done by cross validation
x, y = get_train_dataset()
# Retrieve validation settings.
cv = ML_CONFIG["train"]["execution_options"]["cross_validation"]
if not cv:
cv = get_shuffle_split(x, y)
# Retrieve scoring function used for the validation score, and make it compatible with scikit's gridSearchCV
scoring_function = get_train_scoring_function()
scoring = make_scorer(scoring_function["func"])
# Retrieve hyper parameters.
hyper_parameters = get_hyper_parameters()
# Start grid search with cross validation
clf = RandomForestClassifier()
grid_search = GridSearchCV(clf, hyper_parameters, scoring=scoring, cv=cv, return_train_score=True)
grid_search.fit(x, y)
return grid_search
def get_shuffle_split(x, y):
from sklearn.model_selection import ShuffleSplit
ratio = ML_CONFIG["train"]["ratio"]
rs = ShuffleSplit(n_splits=1, test_size=ratio)
return rs.split(x)
```
> With the above code, you would have to specify _my\_random\_forest_ as the **function name**.
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-keras.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Import a custom Keras script
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The Keras framework allows you to upload a Python _.py_ file containing a Keras sequential model **compatible with the Keras library**.
When you select this framework, some packages are imported by default in the pipeline's environment:
- Keras
- Tensorflow
- the libraries [that come out of the box with ForePaaS' SDK](https://forepaas-sdk.readthedocs.io/en/latest/index.html)
Any other library used by your custom script has to be specified in the **Python libraries** field of the [Dependencies panel](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md) (along with its version, if you wish to not use the latest one).
> 💡 You have the possibility to load a _requirements.txt_ file in order to [add all necessary libraries in one go](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md#import-a-requirementstxt-file).
By default, new pipelines will be using **Python 3.9.9**.
- [Upload a custom Keras model](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-keras.md#upload-a-custom-model)
- [Use Keras for unstructured data training](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-keras.md#use-keras-for-unstructured-data-training)
***
## Upload a custom model
To upload a custom model, either drag your _.py_ file into the Training Script box or click the box to open the file explorer.
When a [Training job is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs), this _.py_ file is the file that will be executed. It must contain a function that has the following two requirements:
- It must have `event` as its first argument
- It must return a fitted and compiled sequential model based on the training dataset
This function's name must be written down in the _function name_ box.
Below is a sample code of a basic custom Keras model using _my\_keras\_model_ as the **function name**:
```py
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from forepaas.ml import get_train_dataset
def my_keras_model(event):
# Retrieve train dataset.
x, y = get_train_dataset()
model = Sequential()
model.add(Dense(60,input_dim=60,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
model.compile(loss="binary_crossentropy",
optimizer='rmsprop',
metrics=["accuracy"])
model.fit(x, y, batch_size=5, epochs=1, verbose=1)
return model
```
:::warning
Due to the way the imports of libraries is handled by the platform, it is required to prefix imports from the `keras` library by `tensorflow`.
:::
While the previous code will not be able to use any of the configurations made in the Training and Tuning steps, you have the possibility to connect to other parts of your pipeline configuration from your model thanks to the [ForePaaS SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md). This allows you to use all the features provided by ForePaaS pipelines with your custom estimator.
The following configurations can be integrated:
- [Use a validation configuration from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-configuration)
- [Use a validation score function from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-score-function)
- [Use hyper-parameters specified in the Tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-hyper-parameters)
!>
Since only the custom _.py_ script is executed during Training jobs, failing to integrate the aforementioned configurations into the Python script will result in any respective interface-originated modification to be ignored by the pipeline.
***
## Use Keras for unstructured data training
The Machine Learning Manager supports images for classification in training and predictions of Keras models.
### Data preparation
To enable unstructured data training, the images must first be uploaded to the Data Store **in a folder named after each class**.
[Learn how to carry out unstructured data preparation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#pictures-from-buckets)
### Training
During training, the SDK methods `get_train_dataset` and `get_test_dataset` **behave differently than with structured data**. They both return two one-column DataFrames containing paths to the training images and associated labels (as .txt files). The column names for the images and labels are `input` and `output` respectively.
:::warning
Labels are returned as integers based on the order they appear in the bucket starting from 0. For most cases, this will be alphabetical.
:::
As such it is necessary to load and process the images and labels, as shown below in the sample code:
```py
def example_process_data():
x_df, y_df = get_train_dataset()
x, y = list(),list()
for f, l in zip(x_df["input"], y_df["output"]):
img = load_img(f, target_size=(28, 28))
img = img_to_array(img) / 255
x.append(img)
with open(l, "r") as f:
_label = f.read()
y.append(int(_label))
x = np.array(x)
y = np.array(y)
return x, y
```
:::warning
When an unstructured-data model is [used for predictions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-consumer-to-make-predictions) later on, you are supposed to feed it the actual unstructured objects (images, etc.). The platform automatically 'numpyfies' them and applies standardization/normalization on them. As such, **you must normalize the data entering the input layer of your neural network script too**.
:::
The rest of the Keras model can then be added with `event` as the only argument to the method:
```py
def my_model(event):
# Retrieve train dataset from earlier defined function.
x,y = example_process_data()
model = Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu',
input_shape=(28, 28, 3)))
model.add(Dense(1,activation="sigmoid"))
model.compile(loss="binary_crossentropy",
optimizer=optimizers.RMSprop(lr=1e-5),
metrics=["acc"])
model.fit(x, y, batch_size=5, epochs=1, verbose=1)
return model
```
> 💡 Multiple methods can be included in the custom _.py_, only the one specified in the `function name` field via the UI will be executed.
Neural network training can be **accelerated by using GPUs** in your ForePaaS pipeline. Check out the link below to learn more.
[Use GPU for your ML processing](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#use-gpu-for-your-processing)
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-pytorch.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Import a custom PyTorch script
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The PyTorch framework allows you to upload a Python _.py_ file containing a PyTorch model **compatible with the PyTorch library**.
When you select this framework, some packages are imported by default in the pipeline's environment:
- Torch and Torchvision
- Pillow
- the libraries [that come out of the box with ForePaaS' SDK](https://forepaas-sdk.readthedocs.io/en/latest/index.html)
Any other library used by your custom script has to be specified in the **Python libraries** field of the [Dependencies panel](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md) (along with its version, if you wish to not use the latest one).
> 💡 You have the possibility to load a _requirements.txt_ file in order to [add all necessary libraries in one go](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md#import-a-requirementstxt-file).
By default, new pipelines will be using **Python 3.9.9**.
- [Upload a custom PyTorch model](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-pytorch.md#upload-a-custom-model)
- [Use PyTorch for unstructured data training](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-pytorch.md#use-PyTorch-for-unstructured-data-training)
***
## Upload a custom model
To upload a custom model, either drag your _.py_ file into the Training Script box or click the box to open the file explorer.
When a [Training job is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs), this _.py_ file is the file that will be executed. It must contain a function that has the following two requirements:
- It must have `event` as its first argument
- It must return a fitted and pytorch model based on the training dataset
This function's name must be written down in the _function name_ box.
Additionally, and unique to the ForePaaS PyTorch framework, the PyTorch **base classes for models and dataset have to be included in the script**, and cannot be nested within other functions or classes:
- `torch.utils.data.Dataset`
- `nn.Module`
The names of the child class have limitations, the only restrictions are that the parent class must be those previously mentioned. Both should be included to avoid failures.
Finally, the expected dimensions of the input layer must be provided alongside the function name to run, in the overview page:
Below is sample code of a custom PyTorch model used with the [Getting Started tutorial dataset](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md), an important note is the structure of the code:
```py
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from logging import getLogger
logger = getLogger(__name__)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
batch_size = 64
epochs = 5
# define structured dataset class
class TabularDataset(Dataset):
def __init__(self, data, input_cols=None, output_col=None):
self.n = data.shape[0]
if output_col:
self.y = data[output_col].astype(np.float32).values.reshape(-1, 1)
else:
self.y = np.zeros((self.n, 1))
if input_cols:
self.x = data[input_cols].astype(np.float32).values
else:
input_cols = [col for col in data.columns if col != output_col]
self.x = data[input_cols].astype(np.float32).values
def __len__(self):
return self.n
def __getitem__(self, idx):
return [self.y[idx], self.x[idx]]
# define model class
class Model(nn.Module):
def __init__(self, in_features=13, h1=32, h2=8, out_features=1):
# how many layers?
super().__init__()
self.fc1 = nn.Linear(in_features, h1)
self.fc2 = nn.Linear(h1, h2)
self.out = nn.Linear(h2, out_features)
def forward(self, x):
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.out(x)
return x
def pytorch_model_train(event):
# forepaas get data
x_df, y_df = get_train_dataset()
data = pd.concat([x_df,y_df], axis=1)
output_feature = "is_overcrowded"
# get data
train_data = TabularDataset(data=data,input_cols=None,output_col=output_feature)
train_loader = DataLoader(dataset = train_data, batch_size=batch_size, shuffle=True )
logger.info(len(train_data))
model = Model().to(device)
model.train()
# some additional params
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
for epoch in range(epochs):
epoch_loss = 0
epoch_accuracy = 0
for y, x in train_loader:
x = x.to(device)
y = y.to(device)
# Forward Pass
preds = model(x)
loss = criterion(preds, y)
# Backward Pass and Optimization
optimizer.zero_grad()
loss.backward()
optimizer.step()
acc = ((preds.argmax(dim=1) == y).float().mean())
epoch_accuracy += acc/len(train_loader)
epoch_loss += loss/len(train_loader)
logger.info('Epoch : {}, train accuracy : {}, train loss : {}'.format(epoch+1, epoch_accuracy,epoch_loss))
return model
```
> With the above code, you would have to specify _pytorch\_model\_train_ as the **function name** and _13,1_ as the **input\_shape**.
While the previous code will not be able to use any of the configurations made in the Training and Tuning steps, you have the possibility to connect to other parts of your pipeline configuration from your model thanks to the [ForePaaS SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md). This allows you to use all the features provided by ForePaaS pipelines with your custom estimator.
The following configurations can be integrated:
- [Use a validation configuration from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-configuration)
- [Use a validation score function from the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-a-validation-score-function)
- [Use hyper-parameters specified in the Tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#use-hyper-parameters)
!>
Since only the custom _.py_ script is executed during Training jobs, failing to integrate the aforementioned configurations into the Python script will result in any respective interface-originated modification to be ignored by the pipeline.
***
## Use PyTorch for unstructured data training
The Machine Learning Manager supports images for classification in training and predictions of PyTorch models.
### Data preparation
To enable unstructured data training, the images must first be uploaded to the Data Store **in a folder named after each class**.
[Learn how to carry out unstructured data preparation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#pictures-from-buckets)
### Training
During training, the SDK methods `get_train_dataset` and `get_test_dataset` **behave differently than with structured data**. They both return two one-column DataFrames containing paths to the training images and associated labels (as .txt files). The column names for the images and labels are `input` and `output` respectively.
:::warning
Labels are returned as integers based on the order they appear in the bucket starting from 0. For most cases, this will be alphabetical.
:::
As such it is necessary to load and process the images and labels within the pytorch dataset class, as shown below in the sample code:
```py
train_transforms = transforms.Compose([
transforms.Resize(150),
transforms.RandomResizedCrop(150),
transforms.ToTensor(),
])
class dataset(torch.utils.data.Dataset):
def __init__(self,image_list,label_list,transform=None):
self.file_list_df = image_list
self.label_list_df = label_list
self.transform = transform
def __len__(self):
self.filelength = len(self.file_list_df)
return self.filelength
def __getitem__(self,idx):
file_list = self.file_list_df["input"].values.tolist()
img_path = file_list[idx]
img = Image.open(img_path)
resized_img = img.resize((vertical_px, horizontal_px))
img = resized_img.convert(mode='L')
image_2_npArray_2_tensor = transforms.ToTensor()(img)
label_list = self.label_list_df["output"].values.tolist()
with open(label_list[idx],'r') as file:
label = int(file.read())
return image_2_npArray_2_tensor,torch.tensor(label)
...
def pytorch_train(event):
# forepaas get data
x_df, y_df = get_train_dataset()
train_feature,val_feature,train_label,val_label= train_test_split(x_df, y_df, test_size=0.2)
# get data
train_data = dataset(train_feature,train_label)
val_data = dataset(val_feature,val_label, transform=test_transforms)
...
```
:::warning
When an unstructured-data model is [used for predictions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-consumer-to-make-predictions) later on, you are supposed to feed it the actual unstructured objects (images, etc.). The platform automatically 'numpyfies' them and applies standardization/normalization on them. As such, **you must normalize the data entering the input layer of your neural network script too**.
:::
The rest of the PyTorch model can then be added with `event` as the only argument to the method:
> 💡 Multiple methods can be included in the custom _.py_, only the one specified in the `function name` field via the UI will be executed.
Neural network training can be **accelerated by using GPUs** in your ForePaaS pipeline. Check out the link below to learn more.
[Use GPU for your ML processing](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#use-gpu-for-your-processing)
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage libraries and dependencies
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The Dependencies panel lets you manage the language version and all the packages requirements for the execution of your pipeline. There are two kinds of dependencies:
- **ForePaaS modules**: they are prepared by ForePaaS and already contain all the libraries required to run a standard estimator.
- **Python libraries**: they are the libraries used in your estimator script that aren't already contained in a ForePaaS module. They only need to be added when you are using a [custom estimator](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md).
When using a standard estimator, **all dependencies are automatically configured to reflect your choice**. No need to scratch your head handling the library requirements, we are taking care of this for you!
On top of that, if you are using a [custom estimator](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md), you can also choose the Python version with which to run the pipeline. You can choose amongst the following:
- Python 3.11
- Python 3.9 _(default option)_
:::info
We are regularly updating the available versions to provide you with a best-practice development framework. Your existing work is not migrated to a new version as long as its language version is still supported.
:::
***
## Import libraries
You can either specify your libraries [one by one](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md#import-a-single-library), or import them [all in one go](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-dependencies.md#import-a-requirementstxt-file) using a _requirements.txt_ file.
### Import a single library
To specify a single library, spell the package name in the **Python libraries** field.
You can also specify the version as such: _library:version_.
### Import a requirements.txt file
You can bulk-import all the libraries needed for your Project using a text file formatted like the output of a `pip freeze`[command](https://pip.pypa.io/en/stable/reference/pip_freeze/).
Click on _Load .txt_ to upload the text file.
Here is the expected requirements format:
```txt
docutils==0.11
Jinja2==2.7.2
MarkupSafe==0.19
Pygments==1.6
Sphinx==1.2.2
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Scoring and validation
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
To evaluate the performances of the models you train in your pipelines, it is advised to **score them in an unbiased fashion**. The score result is always dependent on the dataset on which the model is scored.
:::info
Scoring a model on a given dataset consists of using the model to make predictions for each data point in the set, and then quantifying the discrepancy between the predictions and the true values using a basis scoring function.
:::
You can score a model on the dataset of your choice. However it is usually not advisable to do it on data that was used to train the model because you want to evaluate the ability of your model to generalize to new data. **ForePaaS pipelines let you score models on two different datasets**: the [testing dataset](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#train-test-split) and the [validation dataset](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#validation).
- [Specify a validation set](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#validation)
- [Simple validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#simple-validation)
- [Cross-validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#cross-validation)
- [Choose one or several scoring functions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#scoring)
- [Standard scoring function](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#standard-scoring-function)
- [Custom scoring function](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#custom-scoring-function)
***
## Validation
:::info
Validation is a method in machine learning where a subset is extracted from the initial training set and is used to score all trained models in order to [select the best one](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md). The extracted subset is called the _validation set_. This set provides an unbiased evaluation of models which are trained on the remainder of the initial training set. It also ensures that the testing dataset remains isolated in order to perform the final model assessment, i.e. quantifying the generalization error.\
[Learn more about validation.](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets)
:::
In a pipeline, you can either perform:
- [Simple validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#simple-validation)
- [Cross-validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#cross-validation)
### Simple validation
Simple validation consists of extracting a subset of the initial training set to be used as the validation set. To use it, select _Simple validation_ as the validation type:
Select the size of the validation set as a **share of the initial training set**, either by moving the slider around or entering the desired percentage in the validation field:
> Choosing _0_ as the validation size will disable validation in your pipeline.
When a training job [is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs), the pipeline will **automatically score the produced model on the validation set**. The resulting score is called the _validation score_ in the rest of the pipeline. There is only one scoring function used to compute the validation score per training job: it is the scoring function entered in the Visual Options panel of the [Tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md).
### Cross-validation
[Cross-validation](https://en.wikipedia.org/wiki/Cross-validation_\(statistics\)) consists of splitting the initial training set in a given number of subsets, called the _number of folds_. A [simple validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#simple-validation) is then performed that many times, successively using each subset as a simple validation set. This method ensures that no data point in the initial training set is left unused.
:::warning
Because training a model is repeated as many times as the chosen number of folds, **using cross-validation can be very resource-consuming**.
:::
To use it, select _k-fold cross validation_ as the validation type:
Enter the chosen number of folds in **Fold**:
When a training job [is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs), the pipeline will **automatically score the produced model on each successive fold**. The resulting scores are then **averaged** to produce the _validation score_ displayed in the rest of the pipeline.
There is only one scoring function used to compute the validation score per training job: it is the scoring function entered in the Visual Options panel of the [Tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md).
***
## Scoring
Whenever you choose an estimator at the beginning of the Training step, a default scoring function is automatically added to the list. This default score depends on whether the estimator is a [classifier](https://en.wikipedia.org/wiki/Statistical_classification) or a [regressor](https://en.wikipedia.org/wiki/Regression_analysis).
When a testing job [is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#testing-jobs), **all scores in the list are computed on the _testing set_ simultaneously**.
Each scoring function is associated with a **threshold**: it is the minimum score that models have to reach on both the validation and the testing set in order to be saved.
- During a training job: if the chosen score doesn't reach its threshold on the validation set, the model will not be saved.
- During a testing job: if **any** score doesn't reach its respective threshold, the model will not be saved.
A scoring function can be set as primary by clicking on the **star** ⭐ icon next to it, meaning it will be the score displayed by default in the rest of the pipeline.
When adding a new scoring function, you can either choose a [standard scoring function](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#standard-scoring-function) among a curated catalog of the most popular scores, or upload your own [custom scoring script](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#custom-scoring-function).
### Standard scoring function
To add a standard scoring function, click on **Add**.
Choose the desired standard estimator in the store and press **Select**.
A default threshold is automatically configured for your new score. To change it, click on the **edit** ✏️ icon on the row of the new score.
You can then enter the new chosen threshold as a percentage.
### Custom scoring function
To add your own custom scoring script, click on **Add**.
Choose _Custom score_ in the store and press **Select**.
Enter the name and threshold, then upload your custom Python scoring script in the dedicated box. This _.py_ script must contain a function that takes in two arrays - representing respectively the predictions and true values - and return a numeric
value representing the score.
Write the name of the aforementioned function in **Code function name**:
#### Custom script template
Here is a template of a custom scoring script:
```py
import logging
logger = logging.getLogger(__name__)
def customfunc(y, y_pred, **kwargs):
logger.info("BEGIN function customfunc")
# Function goes in here
print(y, y_pred)
logger.info("END function customfunc")
return 0.8
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Hyper-parameter tuning
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Hyper-parameter tuning is a pipeline configuration step closely linked to the [training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training.md) where you can fine-tune the estimator implemented in a pipeline.
:::info
Hyper-parameters are the core variables of any machine learning estimator that are not learned using the training data. They have to be hard-coded while configuring a model and are needed for the training jobs. For more details about hyper-parameters, check out the associated [Wikipedia page]().
:::
***
## Set specific hyper-parameter values
The tuning studio lists all the hyper-parameters associated to the estimator chosen during the [training configuration](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training.md).
Each of them has a default value already entered. To modify the value set for a specific hyper-parameter, edit it in the _Value_ column.
When your pipeline [is executed](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md), training jobs will use the resulting hyper-parameter configuration for the ML estimator. The scoring function used for the validation method can be set in the Visual Options panel.
***
## Explore different values
If you don't know how to configure it, ForePaaS also lets you explore the impact of different hyper-parameters values on the score of the resulting model in the studio.
Select the _Multiple_ value for the **Mode** column and enter all the values that you want to explore for any hyper-parameter.
This will create a grid of combinations that will all be trained and scored (using the function specified in the Visual Options panel) on the validation set every time the pipeline [is executed](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#Training_jobs).
:::warning
Exploring a grid of hyper-parameters can quickly be **very time and resource-consuming** since all generated model combinations need to be trained and evaluated - especially when you have [cross-validation](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md) activated.
:::
When a pipeline [is executed](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#Training-jobs) with multiple hyper-parameter combinations, while all resulting validation scores are displayed in the studio **only the model with the highest validation score is saved for deployment**. This saved model will then be listed in the [model selection step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md).
[Select a model to deploy](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md)
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Model selection
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Model selection is the step in which you can **compare all trained models in the pipeline, especially over time**, and choose which one you want to deploy to production. The word **_model_** always refers to the combo of two elements: an _estimator and its parameters_ fitted on _training data_.
Think of this step as a control center where you can come back at any time to monitor how well your deployed model is doing over time. If needed, you can switch it for another more accurate or efficient model.
:::warning
A pipeline can store several trained models (they are all displayed on the Model Selection page) but only one can be deployed for predictions.
:::
Here, you can:
- [Manage model versions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md#manage-versions)
- [Deploy a model](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md#deploy-model)
- [Compare scores](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md#scores)
- [Set up automatic deployment of models](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation.md#set-up-automatic-deployment-of-models)
***
## Manage trained models
The main focus of this page is the **Generated models board**. It lists all models that have been fitted since the creation of the pipeline.
> Remember that not all combinations from the [hyper-parameter tuning step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-tuning.md) are saved but only the one with the best validation score.
### Manage versions
For the moment, models' versions are identical to the version of the pipeline whose configuration was used to generate it. This ensures you have traceability over how every one of your predictions was made.
All model versions that were saved over time are listed in the table. You can choose to download a specific model version by clicking the 📥 icon.
### Deploy model
The model at the top with the green atom ⚛️ icon is the model that is currently deployed in your pipeline. This means that it is the model used to make inferences, using the specifications (i.e. consumers) listed in the [Deployment options step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md).
You can choose to manually deploy another model of your choice by clicking on the **Play ▶️ button** at the end of its line.
### Scores
The Generated models list also display both the **validation score** (evaluated on the [Validation set](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md)) and the **testing score** (evaluated on the [Testing set](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#train-test-split)). By default, the testing score isn't calculated in order to minimize the resources used. You can choose to compute it manually by clicking on the **Refresh 🔄 button** in the _Testing Score_ column.
> There is a cup 🏆 symbol next to the model with the best validation score to help you spot it more easily!
The default **scoring function** used in both columns is the _Highlight score_: it is the scoring function [you highlighted ⭐ in the Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md). To display the results with another scoring function, click on the dropdown menu at the top-right of the table and select a function among those you added in the Training step.
Finally, you can also compare the scores of several functions side by side for a specific model on its **details page**. Simply click on the **Stats 📊 icon** at the end of the model's row to open it.
***
## Set up automatic deployment of models
A key feature of the Model Selection step is the ability to configure the auto-deployment of models based on certain criteria.
[Learn how to auto-deploy models](#en/product/ml/pipelines/configure/validation/auto-deploy.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation-auto-deploy.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Automatic deployment of models
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
This feature automates everything you can manually do at the Model Selection step: deploying your favorite model among all versions, based on scores and runtimes.
It is activated in the Deployment Option panel, by turning on **Automatic Deployment**.
Once this is activated, the pipeline will check against the _Preferred Deployment Criteria_ every time a model is generated or scored, and automatically deploy a new model if applicable.
> If Automatic Deployment isn't activated, the preferred deployment criteria is used when [manually playing a pipeline, for the Deployment job](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#deployment-jobs).
You need to specify the _Preferred Deployment Criteria_ used to select the model to deploy. ForePaaS has a curated list of criteria from which to choose:
- [Best score over period of time](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation-auto-deploy.md#best-score-over-period-of-time)
- [Best prediction time over period of time](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation-auto-deploy.md#best-prediction-time-over-period-of-time)
### Best score over period of time
**Parameter**: _Time period_
The model with the best **testing score** over the past \[_time period_] will be automatically deployed.
### Best prediction time over period of time
🚧 Coming soon!
**Parameter**: _Time period_
The model with the best **prediction time** over the past \[_time period_] will be automatically deployed.
***
## Need more options?
ForePaaS is actively adding & updating automation options. If we are missing something critical for your Project, please reach out and drop us a line on [our Roadmap Portal](https://hq.forepaas.io/#/features) 💡!
You can also ask for help via the [OVHcloud Help Centre](https://help.ovhcloud.com/csm/fr-home?id=csm_index) or the [support guide](https://docs.dataplatform.ovh.net/index.md).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Deployment settings
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Setting up the deployment is the last step needed to configure a pipeline: this is where you decide how to operationally use the models it produces.
All the configuration is done through what is called **consumers**. They are entities plugged into the output of your pipeline in order to access its deployed model. They can then be fed with data in order to produce predictions. A single pipeline can have several consumers which makes it possible to use its deployed model in multiple ways at once.
Thanks to consumers, you can call your deployed model both **externally** and **internally** to your Project.
- External calls use an endpoint from one of your Project's APIs, requiring an authentication
- Internal calls use the Data Processing Engine (DPE) to load and write data in your Data Manager
- [Create a consumer](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#create-a-consumer)
- [Use a consumer to make predictions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-consumer-to-make-predictions)
***
## Create a consumer
To create a new consumer, click on _Add_ in the consumers panel.
You then need to choose an **input** and an **output**. The input represents the way to pass on data to the consumer while the output represents the way predictions are returned.
### **Input options**
ForePaaS lets you choose from the following options for the input of the consumer:
| Input Type | Description |
| :--------------: | --------------------------------------------------- |
| **API Endpoint** | The endpoint to send data points to. |
| **Table** | The Data Manager table used to provide data points. |
| **Bucket** | The Data Store bucket used to provide data files. |
#### API Endpoint input
You must have a deployed [Project API](https://docs.dataplatform.ovh.net/landing-page-app-services-apis.md) to be able to create a prediction endpoint. When creating the consumer, specify the Project API you want to use.
Then, specify the name of the endpoint to create.
:::warning
You cannot use an endpoint that already exists, even in another pipeline
:::
#### Table input
When creating the consumer, specify one of your Data Manager tables.
> The attributes of your Data Manager table don't need to have the same names as your model's features: the mapping can be customized in the [generated Predict Action](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-dpe-predict-action). However, all of your model's features need to exist in one form or another in the input table.
#### Bucket input
When creating the consumer, specify one of your Data Store buckets. The consequently [generated Predict Action](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-dpe-predict-action) will try to make a prediction for **all** the files that are stored in the bucket, provided your model has been trained on unstructured data.
### **Output options**
ForePaaS lets you choose from the following options for the output of the consumer:
| Input Type | Description |
| :--------------: | -------------------------------------------------------- |
| **API Endpoint** | The endpoint to receive predictions from. |
| **Table** | The Data Manager table in which predictions are written. |
#### API Endpoint output
This option is automatically selected when you choose an _API endpoint_ as the input. The endpoint is the same as the input.
#### Table output
This option is automatically selected when you choose either _Bucket_ or _Table_ as the input. You can choose either the same table as the input or another one.
> The attribute of your Data Manager table doesn't need to have the same name as your model's value-to-predict: the mapping can be customized in the [generated Predict Action](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-dpe-predict-action).
***
## Use a consumer to make predictions
Once a consumer has been created, it can be used to make predictions.
- Using _API Endpoint_ as either the input or the output of a consumer creates an [API endpoint which can be called](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-an-api-endpoint)
- Using _Table_ as either the input or the output of a consumer automatically generates a Data Processing Engine [Predict Action which can be executed](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-dpe-predict-action)
- Using _Bucket_ as the input of a consumer automatically generates a [DPE Predict Action which can be executed](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md#use-a-dpe-predict-action)
### Use an API endpoint
#### For structured data
Open the API control panel by clicking the **eye 👁️ icon**.
To make a prediction from an API endpoint, you have to [generate an authentication token](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md#generating-authentication-token) first. You can easily generate this token by entering your [API & secret keys](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) and running the command specified in the API control panel.
To launch a prediction job, you can run the sample command provided in the API control panel. You will need to enter two parameters in the request: the **authentication token** you just generated, as well as the **data on which to make predictions**. Each data point is represented by a dictionary containing the model's features and their values. Multiple data points can be predicted at once by including several dictionaries in the array.
> The control panel is pre-populated with a template data payload to show you how variable names and values must be entered.
#### For unstructured data
When an unstructured-data model is used for predictions, you are supposed to feed it the actual unstructured objects (images, etc.). The platform automatically 'numpyfies', standardizes and normalizes them.
To get predictions on an [unstructured data model](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#pictures-from-buckets), make a POST request to the endpoint containing the files with the content-type set to `multipart/form-data`.
### Use a DPE Predict action
The Predict action associated with a consumer is by default generated in the Data Processing Engine, in the _Machine Learning_ repository of the Actions menu.
[Learn how to configure and run Predict actions](https://docs.dataplatform.ovh.net/index.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Execute a pipeline
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
After a pipeline is configured, it can be executed in order to realize different tasks called **jobs**. These jobs are ran using ForePaaS technology to ensure scalability to large volume or frequency of processed data.
The execution of a pipeline is the backbone of having a model evolve through its natural life-cycle. Following the configuration you specified, this is how you can incorporate additional data over time, re-train models, deploy new ones and make predictions.
***
## Pipeline jobs
There are **four** kind of jobs that can be executed either manually or through [triggers](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-triggers.md).
- [Dataset generation jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#dataset-generation-jobs)
- [Training jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs)
- [Testing jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#testing-jobs)
- [Deployment jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#deployment-jobs)
To manually execute pipeline jobs, simply press the **Play ▶️ button** at the top-right.
You can then specify which jobs you want to run and how.
### Dataset generation jobs
Dataset generation jobs are used to create (or add to) the training, testing and validation datasets using your data inputs.
Training and testing sets are modeled and stored as [ML tables in the Data Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md).
When a dataset generation job is launched, the training and validation sets are flushed of their previous data. Then any data point in the data input that **isn't already in the testing set** is split between the training and testing sets following the rules you set in the [Dataset step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-dataset-input.md#train-test-split). The validation set is generated following the rules you set in the [Training step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-validation.md#validation).
> The testing set is never erased naturally, unless you manually reset your datasets from the [pipeline preferences](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#reset-datasets).
### Training jobs
Training jobs are used to train models based on an algorithm and a combination of hyper-parameters, and score them on the validation set.
Training jobs can be accelerated with graphics processing units (GPU) using [FPU-G](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#use-gpu-for-your-processing).
### Testing jobs
Testing jobs are used to evaluate models on the testing set using all added scoring functions.
Testing jobs can be accelerated with graphics processing units (GPU) by using [FPU-G](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#use-gpu-for-your-processing).
### Deployment jobs
Deployment jobs are used to deploy inference APIs which are a model's interface to make predictions. The model deployed is based on the criteria specified in the [Model Selection step](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation-auto-deploy.md).
***
## Set up triggers
Instead of having to manually press the Play ▶️ button, you can choose to configure triggers to **execute pipelines at specific moments or events**.
[Pipeline triggers](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-triggers.md)
***
## Manage pipeline preferences
In the Preferences tab, you can set up fundamental options for the execution of your pipelines such as **allocated resources, timeout and logs settings and even reset datasets** if needed.
[Pipeline preferences](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md)
***
## Scale your pipelines
Each pipeline can be scaled independently in the Preferences, for you to specify the resources allocated to training, scoring and dataset generation jobs.
[How to scale pipelines](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Pipeline Preferences
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Each ForePaaS pipeline possesses several execution settings which can be managed via the Preferences tab of the pipeline edition menu.
Here are the main settings:
- [Pipeline information](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#pipeline-information)
- [Timeout options](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#timeout-options)
- [Environment options](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#environment-options)
- [Resources](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#resources)
- [Reset datasets](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-preferences.md#reset-datasets)
***
## Pipeline information
In the Informations box, you can set the name, description and tags for your pipeline.
***
## Timeout options
The Execution Parameters box lets you set timeout and failure options.
**Timeout** is the runtime after which a job is interrupted if it hasn't finished. You can set a different runtime for all four possible jobs.
:::warning
If a job fails while running a pipeline (i.e. several consecutive jobs), **the whole pipeline is interrupted and the next jobs will not be run**.
:::
***
## Environment options
### Set an environment
The boxes on the right of the screen are the ones that relate to an environment, just like in a [DPE action](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md). By default, there is no environment applied and you can customize all settings within the Preferences page.\
If you want to harmonize the set of settings to all the pipelines you build, you can **specify them at the environment-level**. Go to Environment in the sidebar to create an environment, and apply it to each pipeline by specifying it in the Related Environment box in the pipeline's Preferences page.
### Resources
You can specify the resources dedicated to each kind of [machine learning jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#pipeline-jobs) in the Resources panel.
[How to scale pipelines](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md)
***
## Reset datasets
By default, pipelines are engineered so that the training and testing datasets never spill into each other over time, as ForePaaS manages the life-cycle of the data for you. However, you might want to occasionally reset your ML datasets, if corrupt or incorrect data leaked into your Testing set or if the structure of the training data changes radically.
You can decide to manually reset training, testing and validation datasets of a specific pipeline in its Preferences page by pressing this button:
> This **will not** reset the rest of your pipeline configuration such as the features choice, estimator and hyper-parameter tuning, or the generated models.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Scale pipelines
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The ForePaaS Platform is designed to scale easily as the requirements of your data Project evolve.
In the Machine Learning Manager, each individual pipeline can be **scaled vertically**, meaning you can add more computing power to each job indefinitely. _Dataset generation_ jobs can also be **scaled horizontally**, by splitting the workload on multiple parallel instances.
:::info
For the moment, it is not possible to scale _training_/_testing_ jobs horizontally, i.e. to do distributed training. Our engineering team is working hard to implement this feature soon. If you have a particular need for this, please drop us a note on our [roadmap portal](https://hq.forepaas.io/#/features)!
:::
- [Scale your jobs vertically](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#scale-your-jobs-vertically)
- [Use GPU for your processing](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-resources.md#use-gpu-for-your-processing)
***
## Scale your jobs vertically
You can allocate **more computing power** to each kind of [machine learning jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#pipeline-jobs) independently in the Resources panel of the Preferences tab, by increasing the number of [ForePaaS Units](https://docs.dataplatform.ovh.net/manage-pricing.md) (FPU) of each computing instance. The FPU is a unit of processing capability, representing access to a certain amount of CPU and memory, as such:
- **FPU-S**: general processing - corresponds to approximately _0.5 CPU_ and _2 GB of RAM_, based on hardware availability.
- **FPU-G**: GPU-accelerated - represents approximately _1 GPU_ with at least _5 vCPU_ and _40 GB of RAM_, based on hardware availability.
:::info
By default, newly created pipelines are allocated 4 FPU-S for each type of job.
:::
To change the amount of FPU allocated to a pipeline job, go to its **Preferences** tab.
Use the corresponding slider to set the desired FPU size allocated for the execution.
***
## Use GPU for your processing
You can configure your pipeline to use graphics processing units (GPU) when they are executed in order to accelerate the machine learning and data processing scripts by a lot. To do this, change the allocated resource type to _FPU-G_ in the pipeline's **preferences**.
:::info
Note that jobs using FPU-G will usually have a longer build phase, up to a few minutes.
:::
Currently, FPU-G are available for [training](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs) and [testing](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#testing-jobs) jobs.
> FPU-G are not available on all cloud providers at the moment. If you are unable to find them in your Machine Learning Manager, please reach out to the support team to get more information on the roadmap.
[Learn more about FPU-G consumption](https://docs.dataplatform.ovh.net/manage-pricing.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-triggers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Triggers
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Triggers allow you to schedule the automatic execution of your pipelines, such as generating the datasets or training models.
- [Add a trigger to a pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-triggers.md#add-a-trigger-to-a-pipeline)
- [Trigger on a time schedule](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-triggers.md#trigger-on-a-time-schedule)
***
## Add a trigger to a pipeline
To create a trigger on a pipeline, go to the home page of the pipeline and press **Add**.
You can choose a name and description for your trigger in the **Info** section.
In the **Jobs** section, select the [pipeline jobs](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#pipeline-jobs) that you want to trigger.
Finally, in the **Trigger** section, select the condition for the trigger. When that condition is true, the jobs that you have previously selected will automatically be launched.
### Trigger on a time schedule
You can execute your pipeline on a time schedule by selecting _Cron_ as the trigger type.
:::info
Our engineering team is working hard to implement more trigger options soon. If you have a particular need, please drop us a note on our [roadmap portal](https://hq.forepaas.io/#/features)!
:::
---
url: https://docs.dataplatform.ovh.net/legacy/ml-models.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Models
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The **Models page** is where you will find the full list of models that exist in all the ML Projects (i.e. repositories) of your Project. It is by default filtered on _deployed models only_ but you can also choose to display _all models_.
On the ForePaaS Platform, we call **_model_** the combo of two elements: an _estimator and its parameters_ fitted on _training data_. There are two types of models you can find on ForePaaS:
- Models created through a ForePaaS pipeline: **ForePaaS models**
- Models either imported externally or that belonged to an erased pipeline: **standalone models**
***
## Import an external model
If you already have a model that you trained outside of ForePaaS but you'd like to use our deployment technology to use it in production, you have the possibility to do so. Check out the link below to learn how.
[Import 📥 a trained model](https://docs.dataplatform.ovh.net/legacy/ml-models-import.md)
> To design and train your own model on ForePaaS, you need to [go through a pipeline](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md). Either go to the **Pipelines tab** in the sidebar or use the shortcut '**Create pipeline**' at the top-right of the Models page.
***
## Manage your models
The Models page is a control center from where you can pilot all your models.
If you want to modify your model, simply click the **Edit 🖋 icon**. If your model was made through a ForePaaS pipeline, this will let you edit the pipeline. If your model is a standalone model, this will open its configuration page.
You can also download a model file onto your computer by clicking the **Download 📥 icon**.
Finally, you can erase a model by clicking the **Trash 🗑 icon**.
:::warning
Erasing a **ForePaaS model** does not erase its mother pipeline. This means you can still run the pipeline to generate a new version of the model.\
An erased **standalone model** cannot be recovered unless you import it again.
:::
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-models-import.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Import a model
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
You can very easily import a trained model to ForePaaS by clicking on **Upload Model** on the Models page.
You can either **upload a private model** from your computer or a git repository, or **import a shared community model** that has been posted to the Store by another user.
To use a shared community model, simply search for the desired model within the Store and import it.
If you want to use your own private model instead, simply select one of the _Custom_ options in the store.
> If you can't find your model language among the Custom options in the Store, it is not supported on ForePaaS yet.\
> We are working hard to regularly add more integrations to the platform! Drop us a message on [the Roadmap page](https://hq.forepaas.io/#/features) so we know how to prioritize your needs for future releases 😀.
Specify the name and file for your model, and click on **Import**.
Now you need to configure your model deployment. This is done through several steps:
- [Configuration](https://docs.dataplatform.ovh.net/legacy/ml-models-import.md#configuration)
- [Manage versions](https://docs.dataplatform.ovh.net/legacy/ml-models-import.md#manage-versions)
- [Specify variables](https://docs.dataplatform.ovh.net/legacy/ml-models-import.md#specify-variables)
- [Deployment settings](https://docs.dataplatform.ovh.net/legacy/ml-models-import.md#deployment-settings)
> Shared models are already configured by the author: you can deploy them in one click of a button!
***
## Configuration
Setting up your model is done in the **Configuration tab**.
### Manage versions
All model versions are listed in the table so you can track changes and compare them instantly.
> For the moment, it is not possible to get scoring on imported models - please [drop us a note on the roadmap portal](https://hq.forepaas.io/#/features) if you would like to see this in the future.
To update the model version, simply press the **Refresh** button at the top of the table.
You will then be prompted to select the newest version of your model from your files.
### Specify variables
Depending on the language/software you have used to build your model, you might need to specify its features. This step is required for the **inference API to correctly map input data** when you will be making predictions.
First, specify the name of your value-to-predict by clicking on **Add a variable** next to Y.
Then, specify the variables in your X dataset by clicking on the same button next to X.
You have two options to specify your X variables:
- enter them manually one-by-one
- add them all in one go by uploading a text file. [Click here](https://docs.dataplatform.ovh.net/legacy/ml-models-import-bulk-add-x.md) to see code boilerplates.
:::warning
The variables' names must be the same as the names you will use when requesting a prediction.
:::
***
## Deployment settings
This is where you decide how to use your deployed models to make predictions. All the configuration is done through what is called **consumers**. They are entities plugged into your deployed model which are fed data in order to make predictions.
Deployment settings are identical for standalone models and for pipelines: check out the article below for a detailed documentation.
[Manage deployment settings](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md)
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/ml-models-import-bulk-add-x.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Code boilerplates to specify multiple variables in a model
:::warning
This page describes the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
Supported file formats:
- [.txt](https://docs.dataplatform.ovh.net/legacy/ml-models-import-bulk-add-x.md#txt-file)
- [.json](https://docs.dataplatform.ovh.net/legacy/ml-models-import-bulk-add-x.md#json-file)
***
## .txt file
```txt
variable_name_1
variable_name_2
variable_name_3
```
> You can obtain such a _.txt_ file from your pandas DataFrame by running the NumPy command:\
> **np.savetxt("file\_name.txt", `your_X_dataframe.columns.values`, fmt="%s")**
***
## .json file
```json
{
"columns": [
"variable_name_1",
"variable_name_2",
"variable_name_3"
]
}
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/getting-started-ml.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build your AI model from scratch on ForePaaS
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
If you’ve been hearing about how Artificial Intelligence (AI) can help you make better business decisions and want to add more intelligence and machine learning to the data you have on hand, you're in the right place!
In this tutorial, you'll learn the basics about the ForePaas Platform Artificial Intelligence features. You'll create, train and deploy your own AI model **in less than 30 minutes**.
⛳️ The goal is to get you up to speed on how to use the AI features of the ForePaas Platform so that you can start operationalizing AI asap.
📥 No need to bring your own data or model, we provide them throughout the tutorial to accelerate the process.
:::info
If you have never used the ForePaaS Platform before, we suggest you have a look at the first 3 parts of our Getting Started guide to learn how your [Connectors](https://docs.dataplatform.ovh.net/getting-started-connect-sources.md), [Lakehouse Manager](https://docs.dataplatform.ovh.net/getting-started-organize-data.md) and [Data Processing Engine](https://docs.dataplatform.ovh.net/getting-started-prepare-data.md) work.
:::
***
## AI in production in five easy steps
Generally speaking, the creation and deployment of your machine learning model will be completed in 5 major steps:
1
[## Data preparation
Specify your train and test datasets.
](/legacy/getting-started-ml-dataset)
2
[## Training procedure
Choose the algorithm at the core of your model.
](/legacy/getting-started-ml-training)
3
[## Hyper-parameter tuning
Fine-tune the hyper-parameters of your estimator using our insightful studio.
](/legacy/getting-started-ml-tuning)
4
[## Model selection
Compare trained models and pick the one that suits your needs.
](/legacy/getting-started-ml-validation)
5
[## Deployment settings
Specify the settings for your production-grade inference service
](/legacy/getting-started-ml-deployment)
?>💡 **Good news!**\
The Machine Learning Manager, as in the rest of the platform, does a lot of the work for you automatically, from the configuration to the deployment and scalability. This will help you focus on the logic of your AI model rather than the technical administration. ⭐
---
url: https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Prepare your data for Machine Learning
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
You work for the [Chicago Transit Authority](https://www.transitchicago.com/about/) and want to set up an AI system to **predict whether your metro stations are going to be overcrowded** based on temporal and environmental factors in the city.
_Photo by Skyler Gerald on Unsplash_
You will go through the following steps, to prepare the data for your AI model:
- [Build your first pipeline](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#build-your-first-pipeline)
- [Prepare your datasets](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#prepare-your-datasets)
- [Define the source tables](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#define-the-source-tables)
- [Define the features](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#define-the-features)
- [Pre-processing](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#pre-processing)
***
## Pre-requisites
Before going through this tutorial:
- Download [this zip](https://hq.forepaas.io/getting-started/ml-data-csv.zip) containing the two data files necessary for this guide: _ml\_dataset.csv_ and _ml\_sample.csv_.
- Load the files into your [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md):
- Load them into the **Sources tab** of your Connectors.
- Extract their metadata in the **Analyzer**.
- Build the data models in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)
- Add both tables (respectively naming them _ml\_dataset_ and _ml\_sample_) to your **Tables tab**.
- Don't forget to enable the corresponding Load actions.
- Make sure _date_ and _station\_id_ are set as primary keys for both.
- Save and Build your model by clicking on the **Play** icon
- Process the physical data in your [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) (DPE):
- Open your DPE and head to the **Workflows tab**.
- Create a workflow containing both newly created Load actions.
- Run the workflow and wait a few minutes for the job to be completed.
- Head back to the Lakehouse Manager, in the list view of the **Tables tab**, and make sure that both tables are filled with data (# of rows is greater than 0).
You're all set! The sample data model you just created will be used by the machine learning model you are about to build.
Ready? Go! 🏃
***
## Build your first pipeline
From your project's home page, open the **Machine Learning Manager**.
Feel free to read through the welcome page, and when you are ready click on **New Project**.
All your AI files will be stored within a ForePaaS repository (called _Default_) to ensure production-grade version control. You have the option to connect your own Git repository. For more details on how to do this, please check out our [Product Guide](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md#pipeline-repositories).
The ForePaas Platform gives you the option to import your own trained ML model and even use Jupyter Notebooks. Since you’re going to build your own model, click on **Pipeline**, then **Get Started**.
You just created your first ML pipeline on the ForePaaS Platform. Now it's time to configure it! 🔩
Double-click on the pipeline's name in the header to rename it to _Chicago ML Pipeline_.
As you can see, a ForePaaS pipeline is made of **five configuration steps** from the data preprocessing to the deployment options. Each guide in this **Getting Started for Machine Learning** series will tackle one of them. You'll be ready to go live in no time!
***
## Prepare your datasets
### Define the source tables
Open the **Dataset** panel.
Choose **Tables** as the Input Type, to use the data from the data model you just created in the Lakehouse Manager.
The data used in an AI Project is typically segregated into two subsets: one to train your model, the other to test it. To select your training data, under **Training**, select _ml\_dataset_.
Since you want this table to be the source for your testing data too, simply make sure the **link 🔗** icon between Training and Testing is active. You can toggle this on and off by clicking the link icon.
To select how much training vs. testing data to use, move the slide bar to _85%_ of the data for **Training** and _15%_ of the data for **Testing**.
As you can see in the initial panel under _Tables_, the split is made randomly on the rows of the input table. You can control the reproducibility of the split with the **Seed** in order to get the exact same results as this tutorial. Write _27062020_ as the seed.
> Note that pipelines autosave 💾! You can exit and come back at any time.
### Define the features
The next step in AI modeling is to choose the X and Y variables. Y is the value to predict, and X is the set of features that influence this prediction. In this case, you want to use the available data to predict whether subway stations in the Chicago metropolitan area will be overcrowded.
Let’s start by adding the Y value, the “overcrowded” outcome. Click on **Add a source** next to Y.
Then select _is\_overcrowded from ml\_dataset_. The possible values of Y will be 1 if the stations are overcrowded or 0 if they’re not. Add a description to reflect these outcomes then press **Save**.
Now let's add the X features. Click on **Add a source** next to X.
Click on _Add all variables_. Don't worry about duplicates: this option does not add the Y value-to-predict to your X! ✨
### Pre-processing
In the steps above, you added the X variables in bulk. This is great because you don't have to add each variable one by one, manually. However, since all of the variables were added you should go through each variable and verify if you really need all of them or not. 👀
For example, some of the X variables in your _ml\_dataset_ are redundant and could thus negatively impact your prediction. To remove a variable, click on the **trash 🗑 icon** on the right-hand side of the variable row. Remove the following variables from your X set:
| variable | Reason to remove |
| :---------------: | --------------------------------- |
| **date** | Redundant with holidays variables |
| **latitude** | Redundant with _station\_id_ |
| **longitude** | Redundant with _station\_id_ |
| **station\_name** | Redundant with _station\_id_ |
Your datasets are successfully configured and ready to be used for some cool 🤖 machine learning.\
Let's move on to the next step: choosing and configuring a training procedure.
[Step no 2: Training](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-training.md)
---
url: https://docs.dataplatform.ovh.net/legacy/getting-started-ml-training.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Pick a training procedure
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
Moving on to the next step of building your own pipeline, it is now time to configure the training procedure at the core of your model.
Here's what you'll learn during this tutorial:
- [Choose an algorithm](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-training.md#choose-an-algorithm)
- [Configure scoring and validation](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-training.md#configure-scoring-and-validation)
***
## Choose an algorithm
The ForePaaS Platform makes it easy for you to choose industry-standard AI algorithms, or to use your own. The ForePaaS Platform comes with a long list of AI algorithms. The goal is to save you the trouble of learning about technical algorithms and instead focus on your business needs.
On the following screen, select **Standard Estimators**.
Let's use a [random forest of decision trees](https://en.wikipedia.org/wiki/Random_forest) for this problem. [Decision trees](https://en.wikipedia.org/wiki/Decision_tree_learning) are one of the most mainstream algorithms used to predict a categorical label (such as True/False or 1/0) based on a set of numerical variables. They are also simple to understand, implement and interpret. This makes them a good fit for your Chicago traffic example.
Start by **filtering on the Classifier category** in the store, then choose **Random Forest** (the **classifier** version - not Random Forest Regressor) by Scikit Learn, and press **Select**.
The **Algorithm** and **Dependencies** panels have been automatically configured to reflect the choice you just made: no need to scratch your head handling the library requirements, we are taking care of this for you!
:::warning
Be careful not to erase pre-imported libraries and modules in the Dependencies box, else your pipeline will not run.
:::
***
## Configure scoring and validation
Let’s go through a few concepts before we continue. These are important 💡 to help you build the perfect predictive model.
You chose the base algorithm but it still requires fine-tuning. For this, you'll eventually need to be able to measure the performance of your model. Since you are still in the training phase, you cannot touch the testing set. Best practices say you should instead define a **validation set** 🕵️♂️.\
The validation set is a subset of the previously-defined training set that is used to score models while you are fine-tuning them. Conversely, the testing set is used to measure how well your final model is doing on new, real-world data.
> In other words, your pipeline will split your initial real-world data into three datasets as follows:\
> **Training set**: used to train (or 'fit') the model.\
> **Validation set**: used to provide an unbiased evaluation of a model fit during the training phase.\
> **Testing set**: used to provide an unbiased evaluation of a fully trained model.
But, again, nothing to worry about! The ForePaaS Platform makes it easy to create your validation dataset and to score your predictive model.
The ForePaaS Platform gives you the choice of 2 validation types to score your model: cross-validation or simple validation - let's stick to the latter for now. Use the slide bar to make sure the Validation set represents **20%** of the original Training Set:
> Note that the size of the testing set cannot be edited any longer as this should be done in [the first step](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#define-the-source-tables).
There are many ways to score your validation dataset, and the ForePaaS Platform allows you to pick as many scoring functions as you want. To keep things simple, the ForePaaS Platform has picked by default the [_accuracy_](https://en.wikipedia.org/wiki/Accuracy_and_precision).
The platform also automatically configured a **threshold** at 0%. This value represents the minimum score that any model in the pipeline needs to reach (on both the validation and testing set) in order to be saved. Models for which any of the added scores doesn't reach its respective threshold will be discarded.
Make sure your screen eventually looks like this before moving on to the next step:
Everything is almost ready for your pipeline to run! The next step will be to fine-tune 🕹 the hyper-parameters of the random forest algorithm you just selected.
[Step no. 3: Hyper-parameters tuning](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-tuning.md)
---
url: https://docs.dataplatform.ovh.net/legacy/getting-started-ml-tuning.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Fine tune the hyperparameters of your model
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
In this tutorial, you will set the hyperparameters for your Random Forest algorithm and finally launch your first training and scoring jobs.
:::info
Hyperparameters are core variables of a machine learning algorithm that are set by the data scientist prior to the learning process. Think of it as setting the temperature and cooking time in an oven before baking. In a similar way, setting hyperparameters refers to a specific configuration of the underlying mathematical model with which the training is done.\
[Learn more about hyper-parameters on Wikipedia.](https://en.wikipedia.org/wiki/Hyperparameter_optimization)
:::
You'll go through the following:
- [Make a first search of parameters](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-tuning.md#make-a-first-search-of-parameters)
***
## Make a first search of parameters
You just arrived in the hyperparameter tuning studio. It looks empty for the moment 🙇♂️ but you are going to fill it with precious insights very soon!
The two upper panels, Estimator and Options, give you a recap of the training algorithm and the scoring function you chose on the previous step. You can configure the color code for the scores in your pipeline by changing the score gradient.
Let's focus on the hyperparameters list. It has been populated with all the parameters that constitute a random forest algorithm.
The hyperparameter tuning studio lets you generate multiple configurations that are trained and scored. This gives you insight about the optimal combination of hyperparameters so you can either settle down with one of the combinations, or be more accurate in a second iteration of the search. By default, **each iteration saves only the best model in terms of score, for you to use it in the next steps**.
The first two parameters in the list are **bootstrap** and **criterion**: leave them to _Single_ mode with the default values _True_ and _Gini_.
> 💡 All parameters have a default value defined by the algorithm.
The third parameter in the list is **max\_depth**. It represents the maximum possible [depth of each decision tree](https://en.wikipedia.org/wiki/Tree-depth) in the forest.
> 💡 You can get a description of each hyperparameter by hovering over its name.
Let's explore a couple of values for this parameter. Set the **Mode** to _Multiple_, and in the **Value** field remove the _None_ value tag and enter _5_, _7_, _9_, _11_ and _13_ instead.
Now find the parameter **n\_estimators** in the list. It defines the number of decision trees to be used.\
The score usually increases with the number of trees, but so does the training time and risk of overfitting. Set the **Mode** to _Multiple_, and in the **Value** field remove the _100_ value tag and instead enter _50_, _200_, and _400_.
Finally, make sure the configuration for all other parameters is as follows:
| Hyperparameter | Mode | Value |
| :-----------------------------: | ------ | ------ |
| **max\_features** | Single | 'Auto' |
| **max\_leaf\_node** | Single | None |
| **max\_samples**. | Single | None |
| **min\_impurity\_decrease** | Single | 0 |
| **min\_samples\_leaf** | Single | 1 |
| **min\_samples\_split** | Single | 2 |
| **min\_weight\_fraction\_leaf** | Single | 0 |
| **random\_state** | Single | None |
As you can see on the top-right just above the parameters list, you will generate 15 combinations.
You are now ready to run your pipeline for the first time. Press **Play**. Activate only _Dataset Generation_ and _Training_, and run it.
You deserved a hot drink. ☕️ In the meantime, the ForePaaS Platform is deploying cloud instances to generate datasets, train your models and score your combinations on the validation set.
After 5 minutes, all jobs should be done and you can move on to the next step: deploying your winner model.
[Step no. 4: Model selection](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-validation.md)
---
url: https://docs.dataplatform.ovh.net/legacy/getting-started-ml-validation.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Confirm your choice of model and deploy it
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
The Model Selection step is the **watchtower where you'll be able to control all models created by your pipeline over time**. Pipelines in the ForePaaS Platform are designed to live and evolve as new data is fed into your models, and as your models are trained, validated and tested. Keeping track of the model versions and score drift is essential for a production AI application.
You'll go through the following:
- [Manually deploy a model](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-validation.md#manually-deploy-a-model)
- [Set automatic management options](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-validation.md#set-automatic-management-options)
- [Configure auto-deployment](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-validation.md#configure-auto-deployment)
- [Configure triggers](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-validation.md#configure-triggers)
***
## Manually deploy a model
The _Generated Models_ table lists **all the models that have been trained and saved** since the pipeline was created. For the moment, you can only see one model. It is the best model that was returned by the hyper-parameter exploration in the previous step!
In order to limit the amount of resources used, the ForePaaS Platform doesn't automatically test your models with the testing dataset. You need to choose when to test your model. To do so, you can either press the pipeline's Play button and select the third option _'Testing'_, or manually click on the **refresh 🔄icon** for a specific model.
Now, it is time to deploy a model 🚀. To do this, click on the **Play ▶️ icon** which is located on the right-hand side of the line of the model you wish to deploy.
It's that simple! Your model has been deployed, as indicated by the green atom icon next to its name, and is now available for predictions.
***
## Set automatic management options
### Configure auto-deployment
As you have seen above, it is really easy to deploy a model. But, it’s still manual work! So, the ForePaaS Platform makes it even easier, by allowing you to auto-deploy your models. Once in production, if you are regularly gathering new data, this will save you the task of having to open the pipeline and manually launch the deployment.
The default [criterion for deployment](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-validation-auto-deploy.md) is the _best score over a period of time_. Leave it as it is. Simply note that you might also want to optimize the training or prediction time rather than only the score - this tab lets you do that too.
Set the time period to **14 days**.
Turn on **Automatic deployment**.
Your pipeline will now automatically deploy the best model every 14 days. Naturally, this feature should be combined with an automatic re-train / re-score of the model, else the deployed model cannot possibly change.
### Configure triggers
The remaining automation options, such as re-training, can be set through **triggers**. To do so you’ll need to go back to your pipeline main page. Click on the **X** button at the top-left.
> Note that pipelines autosave 💾! You can always exit and come back at any time.
As you can see, there are no triggered events yet. Click on **Add**.
Name your trigger _Auto re-train_, give the description of your choice and click on **Jobs**.
You will now need to set up your pipeline so that it refreshes the training and the testing datasets with the latest gathered new data, and that it builds a new model to be trained. To do so, turn on the _Dataset Generation_ and _Training_ sliders. Then click on **Trigger**.
Let’s have the pipeline do this every 14 days. Select _Daily_ and choose _Every 14 days_. Click on **Create** to close the window.
Your pipeline will now not only incorporate all new data for training and scoring every 14 days but it will also train a new model and automatically deploy the one with the best score.
Great job setting up an automated AI pipeline! Let's conclude this series and configure a few deployment options to make some predictions.
[Step no. 5: Deployment options](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-deployment.md)
---
url: https://docs.dataplatform.ovh.net/legacy/getting-started-ml-deployment.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Set model deployment options
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
In the Deployment step, you can configure all output options for your pipeline. In other words, this is the step where you truly put your model into production in order to operationalize the flow of predictions.
First, open the Deployment page.
In this last machine learning tutorial, you'll go through the following:
- [Manage consumers for your model](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-deployment.md#manage-consumers)
- [Make your first prediction](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-deployment.md#make-your-first-prediction)
***
## Manage consumers
The ForePaaS Platform gives you multiple options on how to use your model to make predictions: write in your Lakehouse Manager, make API calls, etc. For the moment let's keep it simple 😌 and fill a ForePaaS table.
In the Consumers panel, click on **Add**.
Name it _Internal consumer_ and select _Table_ as the input (_Table_ is automatically chosen as the output too).
Select _ml\_sample_ as the table for both the input and the output, then click on **Create**.
***
## Make your first prediction
Finally the time has come!
Let’s go over the business context from the beginning of this tutorial: You’ve built an AI system to predict whether subway stations will be overcrowded and on which day. In other words, your model will make a prediction for each station id and date combination.
The best way to represent this is with an output table. Each row of the output table will feature a station id and a date combination, and show the value of 1 if the station is overcrowded, and 0 if it's not. You will keep other information from your dataset in that table, such as holidays, weather conditions and station coordinates in case you want to do other analyses or correlations. You will also keep the true outcome value for each row, in order to compare it with your predictions and evaluate how well your predictive model is working.
Your ML model has been incorporated in your Data Processing Engine when you created the consumer, in the form of a new Predict Action. Exit the Machine Learning Manager and head to the Data Processing Engine, in the Actions tab. Select the **Machine Learning** repository: there is a new Predict Action there.
Open the action. The lower panel is where you select the input and output for your model: edit the output on the right by clicking on _Select an attribute_.
The _predictions_ column in the table _ml\_sample_ is currently empty and ready to be filled with predictions. To do this, select the attribute _predictions_.
You also need to specify which primary keys the action should use to write predictions. To do this, turn on **Direct Loading**.
Direct loading is used to copy your input data straight into your output table where the predictions will be, without going through your model. typically row indexes. This is where you can specify the primary keys to copy from the input table: select _date_ in the left column.
Select _date_ in the right column.
Repeat with _station\_id_, which is the second primary key of _ml\_sample_.
You're all set! The Predict action will automatically write the prediction in the correct row identified by the _date_ and _station\_id_ primary keys.
Run your prediction by pressing the **Play** button.
Wait a couple of minutes for the sample data to call your deployed ML instance and gather predictions. You'll know it's finished when the _Jobs_ tab in your DPE is clear.
Run a [query in the Analytics Manager](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) on the table _ml\_sample_ to check the values that have been predicted by your model!
***
** Congrats! 🎉🎊
**
**You've completed the ForePaaS Getting Started Machine Learning 🤖 tutorial**. Impressive work! You have successfully trained and deployed to production a first machine learning model. You also ran some quick predictions.
> We know you might have faced some difficulties 🧐 during the tutorials. We're committed to improving the product and do better... but for that we need your feedback! Please take 3 min ⌛to fill in [this short survey](https://forms.office.com/Pages/ResponsePage.aspx?id=nDWzgpGTfE2g8oFnUal2QQ9d-P8xWc5AgNsykYKKK71UOVpZT1RUMFBRUk5BWjNLSzBCNVhDSVgyNSQlQCN0PWcu).
Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed.
***
** More!
**
The ForePaaS Platform is a very powerful and complete tool. Do you want to know more about its advanced capabilities? 🥊 We have prepared multiple tutorials on the more advanced use-cases in our _Getting Further_ guide series. Click below to check it out!
[Check out the ForePaaS Platform training guides 🤩](https://docs.dataplatform.ovh.net/landing-page-tutorials.md)
Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind.
[Give your opinion on the roadmap 🚀](https://hq.forepaas.io/#/features)
---
url: https://docs.dataplatform.ovh.net/legacy/getting-further-ml-custom-sklearn.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Create a machine learning pipeline using a custom Scikit Learn script
:::warning
This tutorial uses the **Machine Learning Manager**, a service of the ForePaaS Legacy Platform that is not available on OVHcloud Data Platform. See [the current tutorials](https://docs.dataplatform.ovh.net/landing-page-tutorials.md).
:::
This tutorial will show you how to use [custom estimators](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md) in your machine learning [pipelines](https://docs.dataplatform.ovh.net/legacy/ml-pipelines.md).
Contrary to [standard estimators](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-standard-estimator.md), custom estimators let you upload any piece of Python 3+ code to be executed as a training script in your pipelines. The following tutorial will show examples of an estimator from the [Scikit Learn](https://scikit-learn.org/) framework.
:::info
This tutorial also makes use of functions from the [ForePaaS' SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md), which empower you with the tools to connect to the rest of your data Project on ForePaaS.
:::
***
## Pre-requisites
!> If you haven't been through our [Getting Started tutorial for the Machine Learning Manager](https://docs.dataplatform.ovh.net/legacy/getting-started-ml.md), please go through the following pre-requisites.
Before going through this tutorial:
- Download [this zip](https://www.dropbox.com/s/ebiforzq20tjos0/ml-data-csv.zip?dl=1) containing the two data files necessary for this guide: _ml\_dataset.csv_ and _ml\_sample.csv_.
- Load the files into your [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md):
- Load them into the **Sources tab** of your Connectors.
- Extract their metadata in the **Analyzer**.
- Build the data models in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)
- Add both tables (respectively naming them _ml\_dataset_ and _ml\_sample_) to your **Tables tab**.
- Don't forget to enable the corresponding Load actions.
- Make sure _date_ and _station\_id_ are set as primary keys for both.
- Save and Build your model by clicking on the **Play** icon
- Process the physical data in your [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) (DPE):
- Open your DPE and head to the **Workflows tab**.
- Create a workflow containing both newly created Load actions.
- Run the workflow and wait a few minutes for the job to be completed.
- Head back to the Lakehouse Manager, in the list view of the **Tables tab**, and make sure that both tables are filled with data (# of rows is greater than 0).
You're all set! The sample data model you just created will be used by the machine learning model you are about to build.
Ready? Go! 🏃
- [Create a new pipeline](https://docs.dataplatform.ovh.net/legacy/getting-further-ml-custom-sklearn.md#create-a-new-pipeline)
- [Upload a custom training script](https://docs.dataplatform.ovh.net/legacy/getting-further-ml-custom-sklearn.md#upload-a-custom-training-script)
- [Enter custom hyper-parameters](https://docs.dataplatform.ovh.net/legacy/getting-further-ml-custom-sklearn.md#enter-custom-hyper-parameters)
***
## Create a new pipeline
Open the **Machine Learning Manager**. On the welcome page, click on **New Project** and select _Pipeline_.
You just created a new ML pipeline on ForePaaS. Open the **Dataset** panel.
Configure the Data Preparation step of the pipeline [as shown in the Getting Started tutorial **here**](https://docs.dataplatform.ovh.net/legacy/getting-started-ml-dataset.md#prepare-your-datasets), using _ml\_dataset_ as your input table. Don't forget to remove redundant variables.
Once your screen looks like the screenshot below, click on **Training** to move on to the training procedure.
***
## Upload a custom training script
Select **Custom Estimator**. This will let you customize the training procedure for your pipeline.
You then need to specify the framework for your pipeline. Select _Custom Scikit Learn_.
This framework allows you to upload a Python _.py_ file containing an estimator compatible with the Scikit Learn library. The supported libraries typically include (but aren't restricted to): Scikit Learn, XGBoost, lightgbm, ...
:::info
When you select this framework, some packages are imported by default in the pipeline's environment: scikit-learn and the libraries that come out of the box with [ForePaaS' SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
You must now upload a training script in the box below:
Below is a sample custom estimator, including feature engineering that was tailor-made for the _ml\_dataset_ example. Following a [scikit-learn example](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html) and using `sklearn.pipeline`, the script includes:
- use [ForePaaS' SDK functions](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md#common-features-from-sdk) to retrieve:
- your train and test datasets (configured previously)
- your validation configuration (left to the default 80%/20% in the interface)
- and the hyper-parameters grid (that you will register later on)
- scaling of numerical variables
- basic processing (one-hot encoding) of some categorical variables
- fitting a _RandomForestClassifier_ estimator with hyper-parameter tuning and validation
```py
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from forepaas.core.settings import ML_CONFIG
from forepaas.ml import get_train_dataset, get_hyper_parameters, get_train_scoring_function
def my_random_forest(event):
# Retrieve train dataset. No need to split it since it will be done by cross validation
x, y = get_train_dataset()
# Retrieve validation settings.
cv = ML_CONFIG["train"]["execution_options"]["cross_validation"]
if not cv:
cv = get_shuffle_split(x, y)
# Retrieve scoring function used for the validation score, and make it compatible with scikit's gridSearchCV
scoring_function = get_train_scoring_function()
scoring = make_scorer(scoring_function["func"])
# Retrieve hyper parameters.
hyper_parameters = get_hyper_parameters()
# Set Column Transformers
numeric_features = ['cloud_cover','humidity','temperature','wind_speed']
numeric_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler())])
categorical_features = ['station_id','month','week_day']
categorical_transformer = OneHotEncoder(handle_unknown='ignore')
preprocessor = ColumnTransformer(
transformers=[
('num', numeric_transformer, numeric_features),
('cat', categorical_transformer, categorical_features)])
# Append classifier to preprocessing pipeline.
# Now we have a full prediction pipeline.
clf = Pipeline(steps=[('preprocessor', preprocessor),
('classifier', RandomForestClassifier())])
# Start grid search with cross validation
grid_search = GridSearchCV(clf, hyper_parameters, scoring=scoring, cv=cv, return_train_score=True)
grid_search.fit(x, y)
return grid_search
def get_shuffle_split(x, y):
from sklearn.model_selection import ShuffleSplit
ratio = ML_CONFIG["train"]["ratio"]
rs = ShuffleSplit(n_splits=1, test_size=ratio)
return rs.split(x)
```
Save the above snippet as a _.py_ file and upload it in your ForePaaS pipeline.
When a [Training job is launched](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md#training-jobs), this _.py_ file is the file that will be executed. It contains a function that has `event` as its first argument and returns a fitted estimator based on the training dataset.
This function's name, here `my_random_forest`, must be written down in the _function name_ box.
Leave the validation configuration as it is by default: the validation set is a random extract of 20% of the training set.
Now the last step is for you to specify a grid of hyper-parameters to optimize on.
***
## Enter custom hyper-parameters
Move on to the **tuning** step of the pipeline.
If you were using a **standard** Scikit Learn estimator, this page would be pre-populated with all the hyper-parameters for the estimator and their default value. However, since you are using a custom script, this hyper-parameter tuning studio **turns into a registry of key-value objects that you can pass on to your training script**.
?> If you don't change anything here, running the pipeline will train the Scikit Learn model using the default value for all its hyper-parameters.
Create a new hyper-parameter by clicking on **Add**.
The code snippet previously used for the training script encapsulates the `RandomForestClassifier()` inside of a `Pipeline` object, in the _classifier_ step. For this reason, all the parameters that you want to pass on to the RandomForestClassifier should start with _nameOfTheStepInSklearnPipeline\_\__ (double underscore).
In this case, if you want to add a grid search on the hyper-parameter `n_estimators`, name it _classifier\_\_n\_estimators_.
Make sure to switch its type to _Number_.
Now, switch the mode to **Multiple**. Remove the empty option, and add a few values, pressing Enter between each: _5_, _10_, _50_, _100_, _500_, _1000_.
Now, it is time to run your ForePaaS pipeline. Press **Play** at the top-right, and [play all steps](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute.md).
Your ForePaaS pipeline is going to train 6 different combinations, keep the model with the best score on the validation set, score it on the testing set, then deploy it (served by a model API). This can take up to 10 minutes.
Move to the model selection step to view your deployed model when your pipeline execution is finished.
That's it! Your custom ML model is live and accessible by your users. You can add [consumers](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-deployment.md) on the model API to use the model for live or batch predictions. You can also add [triggers](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-execute-triggers.md) to automatically retrain/redeploy.
***
** Congrats! 🎉🎊
**
**You've completed this ForePaaS Getting Further tutorial!** You're now ready to build your own custom machine learning pipelines.
> Please take 3 min ⌛to fill in [this short survey](https://forms.office.com/Pages/ResponsePage.aspx?id=nDWzgpGTfE2g8oFnUal2QTTUCIkDw9dBkuMh8LM8xBBUOVpZT1RUMFBRUk5BWjNLSzBCNVhDSVgyNS4u). It will help us understand 🧐 what worked well and what didn't in your experience, in order to make this tutorial even better.
Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed.
***
** More!
**
Did you like this tutorial? Want more? Don't worry we got you covered, you can deep dive into the customization capabilities of the Machine Learning Manager in our documentation!
[Learn more about custom estimators](https://docs.dataplatform.ovh.net/legacy/ml-pipelines-configure-training-custom-estimator.md)
Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind.
[Give your opinion on the roadmap 🚀](https://hq.forepaas.io/#/features)
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# ForePaaS APIs
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Data Platform APIs](https://docs.dataplatform.ovh.net/landing-page-developers-apis.md).
:::
In addition to the graphical user interface (GUI), you can also interact with your [ForePaaS Project](https://docs.dataplatform.ovh.net/manage-projects.md) through the **endpoints in the ForePaaS APIs**. It provides a developer alternative to the GUI for building and managing ForePaaS Projects.
Below are the API references for each component.
| API reference for | Main context of use |
| :--------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------- |
| [Buckets](https://docs.dataplatform.ovh.net/legacy/api-reference-datastore.md) | Interact with the [buckets of your Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). |
| [Query Builder](https://docs.dataplatform.ovh.net/legacy/api-reference-qb.md) | Interact with your [Query Builder](https://docs.dataplatform.ovh.net/legacy/index.md) and its metadata. |
| [Identity Access Manager](https://docs.dataplatform.ovh.net/legacy/api-reference-iam.md) | Interact with your [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md), typically to manage authentication. |
| [ForePaaS Store](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md) | Interact with the ForePaaS Store, typically to manage your application's modules or application's templates. |
***
## General formats
### Response format
- Data format: JSON see [http://www.json.org](http://www.json.org)
- Character set: UTF-8
- All non-empty responses are sent with the header: `Content-Type: application/json; charset=utf-8`
### HTTP response codes
| Code | Message | Signification |
| :--- | :-------------: | :-------------------------------------------------------------------------- |
| 200 | OK | Default response for successful GET or POST requests with non-empty content |
| 400 | Bad Request | Invalid value for GET or POST parameters |
| 401 | Unauthorized | Invalid Credentials or Rights |
| 403 | Forbidden | Authentication error |
| 404 | Not Found | Action or request not found |
| 500 | Internal Server | A server-side error occurred |
### Error handling
In case of client-side (HTTP status code 4xx) or server-side (HTTP status code 5xx) error, the response body may contain a JSON object describing the error:
```json
{
"error": "Invalid parameter", // code 400, 403, 500
"warning": "Action not found" // code 404
}
```
### Timestamp
All data endpoints return the timestamp associated with the data provided.
```json
{
"timestamp": 1435929960
}
```
### Duration
All data endpoints return the duration attribute. It represents the effective time taken by the system to execute a request in seconds.
```json
{
"duration": 0.056
}
```
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-qb.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# API Reference: Query Builder / Analytics Manager
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
This page contains the API reference for the [Query Builder](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md), now replaced by the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
- [Query endpoint](https://docs.dataplatform.ovh.net/legacy/api-reference-qb.md#query-endpoint)
- [Dataset endpoint](https://docs.dataplatform.ovh.net/legacy/api-reference-qb.md#dataset-endpoint)
?> Like all other ForePaaS endpoints, these endpoints requires you to have an authentication token from the IAM of the Project you are trying to access. To obtain an authentication token, [please follow this subsection](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md#generate-an-authentication-token).\
Make sure you also have corresponding permissions in service _Query Builder Legacy_ (in IAM).
***
## Query endpoint
**Description**
- Executes the given query, using the [ForePaaS JSON query format](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-json-format.md)
**Endpoint**
- POST /qb/query
**Query Parameters**
| Parameter name | required | example | description |
| -------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| data.fields | yes | \{ "income":\["sum"],"expense":\["sum"]} | Fields to return and their compute mode(s) (can be: select, select\_distinct, sum, min, max, avg, count, count\_distinct) |
| data.skip | no | 10 | Number of rows to skip |
| data.limit | no | 100 | Number of rows returned |
| data.hints | no | true/false | Total number of lines corresponding to the query (omitting data.limit) |
| filter | no | \{ "date": \{ "between": \["1420066800", "1428962399"] }, "client\_code": \{ "in": \["08KD", "09CC"] }, "creation\_date": \{ "eq": \["2018-01-01"] }} | Apply filter(s) on the data to query. Filters can be: in, not\_in, between, not\_between, eq, not\_equal, like, not\_like, is\_null, is\_not\_null, gt (\>), gte (\>=), lt (\<), lte (\<=) |
| scale.fields | no | \["date",\["client\_code", "employee\_id"]] | Apply scale (group by) to the query. It can be a computed scale, or more than 1 scale. |
| evol.scale | no | "year" | scale to apply for evolution (weekday;week; month; quarter; semester; year) |
| evol.mode | no | "date""weekday" | date : comparison with same dateweekday : comparison with respecting the same weekday. |
| evol.depth | no | 5 | How much evolutions you want to get back |
| total.all | no | \[]\["client\_code"] | total queries per distinct scales (optional) |
| total.x | no | \["client\_code"] | Total queries for a specific attribute |
| order | no | \{ "date":"asc"} | order query output (asc or desc, asc by default) |
| table\_name | no | "prim\_vehicles" | Force to query in a specific table |
| database\_name | no | "data\_prim""data\_mart" | Force to query in a specific layer |
**Sample query, request body**
```json
{
"data": {
"fields": {
"a": ["sum"],
"b": ["sum", "avg"]
},
"skip": 10,
"limit": 10
},
"filter": {
"h": {
"between": ["1420066800,1428962399"]
},
"c": {
"in": [1, 2, 3]
},
"g": {
"eq": [1]
},
"x": {
"is_null": []
}
},
"scale": {
"fields": ["d", ["e", "f"]]
},
"evol": {
"scale": "year",
"mode": "date",
"depth": 5
},
"total": {
"all": [],
"x": ["d"]
},
"order": { //order module to order query output (asc or desc, asc by default)
"c": "asc"
},
"table_name": "TABLE_NAME" //force table_name for the query
}
```
**Sample query, response**
| Code | Description |
| ---- | :---------- |
| 200 | JSON object |
```json
{
"success": true,
"timestamp": 1435936140,
"duration": 0.09,
"error": null,
"warning": null,
"query_params": {…}, //recap of the executed query
"results": [ //results of the query
{
"data": {
"a": {
"sum": [
{
"month": 1,
"team": "Team Rocket",
"value": 10300
},
{
"month ": null,
"team": null,
"value": null
}
]
}1
},
"scales": {
"month_date_fact": 1,
"team": "Team Rocket"
}
}
],
"total": { //results for each total provided by the query
"all": [{…}]
}
}
```
***
## Dataset endpoint
:::warning
This feature is from the legacy Query Builder and will soon be deprecated. Please check out the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) and [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md) instead.
:::
**Description**
- Returns the dataset from the Query Builder.
**Endpoint**
- GET /qb/dataset
**Responses**
| Code | Description |
| ---- | :---------- |
| 200 | JSON object |
```json
{
"success": true,
"timestamp": 1435934914,
"duration": 0.002,
"data": { //list of available data and the compute mode
"a": {
"computes": ["sum"],
"default": "sum"
}
},
"compute_mods": [ //list of available compute mode
…
],
"orders": [ //list of available fields for sorting queries
…
],
"filters": [ //list of available fields for filtering queries
…
],
"scales": [ //list of available fields for grouping queries
…
],
"evol": [ //list of available evolution mode
…
],
"dictionnary": { //The data dictionary for reporting
…
}
}
```
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-qb-data.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# 'Data' query parameter
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
This parameter is used to select and display the desired data. It is possible to apply on each attribute simlple functions such as the average, the count, the distinct count, the max, the min, the select, the distinct select and the sum.
***
## Select
The _select_ function simply displays the value of the selected attribute. It will select the first value of the list if several values are returned. Let's assume we first want to display the date and amount attributes. In the interface click on the right fields and choose the function _select_. The corresponding JSON is:
```json
...
"data": {
"fields": {
"date": [
"select"
],
"amount": [
"select"
]
}
}
...
```
This will yield the following result:
You can add as many fields as you want as long as they exist in the same table, otherwise the query builder (QB) returns an error. The list of field available in the interface is automatically filtered to only display the other available attributes.
> For more fine grain control, note that you have the possibility to hardcode the table from to recover the data using the attribute "table\_name". This can come in handy when certain attribute names (such as "date") are common to a large number of tables. However, this is not a good practice as the Query Builder is designed to fetch the data with the optimal path in order to reduce runtime.
***
## Avg
The _average_ function is used to calculate the average of an attribute. For instance in our case, let's assume we are trying to return the average reported revenue amount overtime:
```json
...
"data": {
"fields": {
"amount": [
"avg"
]
}
},
...
```
This will yield the following result:
:::warning
Beware, depending on the granularity of your table, the average operation can have different meanings, because it operates an average of the "amount" attribute by giving each line the same weight. We therefore recommend using averages, ratios, etc. via virtual attributes so that you can control the weight (average per order, per customer, per day, per month, etc.).
:::
***
## Count
The _count_ function is used to count the number of occurrences returned by the query. For instance, in our case, let's assume we try to count the number of lines present in our table. The corresponding JSON is:
```json
...
"data": {
"fields": {
"date": [
"count"
]
}
}
...
```
This will yield the following result:
***
## Count Distinct
The _count distinct_ function is used to count the unique different values of an attribute. For instance, in our case, let's assume we try to count the unique number of month present in our table. The corresponding JSON is:
```json
...
"data": {
"fields": {
"month": [
"count_distinct"
]
}
}
...
```
This will yield the following result:
***
## Min and Max
The _min and max_ functions are used to display the minimum and maximum values of an attribute. For instance, in our case, let's assume we are looking for the minimum and maximum date in the table. The corresponding JSON is:
```json
...
"data": {
"fields": {
"date": [
"min",
"max"
]
}
}
...
```
This will yield the following result:
***
## Select Distinct
The _select\_distinct_ functions are used to display the unique different values of an attribute. For instance, in our case, let's assume we are trying to display every month that recorded revenue for the company. The corresponding JSON is:
```json
...
"data": {
"fields": {
"month": [
"select_distinct"
]
}
}
...
```
:::warning
We recommend to use the _scale_ parameter (GROUP BY equivalent in SQL) that we will see in another article to perform similar requests.
:::
***
## Sum
The _sum_ function is used to calculate the total value of an attribute. For instance, in our case, let's assume we are trying to get the total turnover. The corresponding JSON is:
```json
...
"data": {
"fields": {
"amount": [
"sum"
]
}
}
...
```
This will yield the following result:
***
[Learn more about the 'scale' parameter](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-scale.md)
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-qb-scale.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# 'Scale' query parameter
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
Many **query parameters can be fine tuned using sub-parameters**. In this article we will see the main usage of the _scale_ parameter and then explore how to leverage the 'scales' using the _axis_ sub-parameter.
***
## Using the 'scale' parameter
This parameter is used to regroup the values of an attribute ("measure") according to another attribute ("dimension").
For instance displaying the revenue amount per month or the revenue amount per store. You can simply add as much scale as possible by adding attributes in the scale parameters.
Let's assume that we are trying to calculate the total revenue per month. The corresponding JSON is:
```json
{
"data": {
"fields": {
"amount": [
"sum"
]
}
},
"scale": {
"fields": [
"month"
]
},
"filter": {},
"order": {}
}
```
> Note that the query will display one line per combination of unique values of the attributes used as scale, and therefore can rapidly increase the weight of the query's output value. Using 5 scales with each 5 different values, will yield 3125 combinations.
***
## Using the 'axis' sub-parameter
When you make a query split on different dimensions, some graphics will not have the desired rendering.
Let's take for example this query:
```json
{
"data": {
"fields": {
"ca": [
"sum"
]
}
},
"scale": {
"fields": [
"date",
"city"
]
}
}
```
From the "Editor" screen of your Query Builder, you'll get different renderings:
The "Table" one looks good, but when you go to "Area":
The chart creates a bar for each "city - date" combination, which is the default mode. This graph, and most of the default graphs provided by ForePaaS, offer a cutout via an "axis" system, allowing to cut out this rendering differently.
Take the query again and add the field "scale.axis":
```json
{
"data": {
"fields": {
"ca": [
"sum"
]
}
},
"scale": {
"fields": [
"date",
"city"
],
"axis":{
"x":["date"],
"y":["city"]
}
}
}
```
The "x" axis will always correspond to the default field.
The "y" axis differs according to the type of graph and the library used, in the most common case, it will cut the data into separate series.
This gives you a new rendering:
***
[Discover the 'filter' parameter](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-filter.md)
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-qb-filter.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# 'Filter' query parameter
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
This parameter is used to apply filters to the selected values that the query is expected to return. Filters are particularly useful to prevent nulls or unlabeled values to be displayed or taken into account.
***
## List of filter options
Below is the list of all filter options available:
- _in_ - Value contained in reference list.
- _not\_in_ - Value not contained in reference list.
- _between_ - Value is between 2 extrema.
- _not\_between_ - Value is not between 2 extrema.
- _equal_ - Value is equal to reference.
- _not\_equal_ - Value is not equal to reference.
- _like_ - Substring of value is contained in another reference value.
- _not\_like_ - Substring of value is not contained in another reference value.
- _is\_null_ - Value is null.
- _is\_not\_null_ - Value is not null.
- _gt_ (>) - Value is greater than a reference.
- _gte_ (> =) - Value is greater than & equal to a reference value.
- _lt_ (>) - Value is lower than a reference value.
- _lte_ (\<=) - Value is lower than & equal to a reference value.
***
## Simple example
Let's assume that we are now looking to display the turnover only on the month of January of the year 2018.
```json
{
"data": {
"fields": {
"amount": [
"sum"
]
}
},
"scale": {
"fields": []
},
"filter": {
"month": [
1
],
"year": [
2018
]
},
"order": {}
}
```
> To apply filters on numerical values, just fill the JSON without the quotes. For non-numeric values, you will need to use "" around the value. For instance, "filter": \{"date": \["2018-01-01"]}.
If we were looking to display the total revenue between 2 dates say February 1st and February 28th, we would simply use a between filter:
```json
{
"data": {
"fields": {
"amount": [
"sum"
],
"month": [
"select"
],
"year": [
"select"
]
}
},
"scale": {
"fields": []
},
"filter": {
"date": {
"between": [
"2018-02-01",
"2018-02-28"
]
}
},
"order": {}
}
```
***
[Discover all the other query parameters](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-others.md)
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-qb-others.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Discover all other query parameters
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
In this article we will review the remaining less used parameters used to build queries on ForePaaS. In particular we will cover:
- _limit_ - Ability to limit the number of outputs of a query
- _order_ - Ability to order the output of a query based on other attributes
- _table\_name_ - Ability to set from which table is the data retrieved
- _skip_ - Ability to skip a certain number of the first outputs of a query
- _hints_ - Allows to display more metadata in the raw JSON query response
- _total_ - Allows to compute subtotals when a sum is computed with several scales
***
## Limit
Displaying large amount of information at once can freeze the browser as it uses all its live memory. We therefore often want to limit the number of rows that we returned by the query, you can do so by setting the _limit_ sub-parameter to the desired number.
By default, the limit is set to 5000 in order to prevent any risks of overloading the web page that leverages the query. Note that this is not a maximum, you can override that condition with a higher number when setting the "limit" parameter to another value.
Let's assume, we try to display 2 lines of the month of January. The corresponding JSON would be:
```json
{
"data": {
"fields": {
"date": [
"select"
],
"amount": [
"select"
]
},
"limit" : 2
},
"scale": {
"fields": []
},
"filter": {
"month": [
1
],
"year": [
2018
]
},
"order": {}
}
```
This will yield the following result:
***
## Order
The results returned by the query may be ordered based on the order of values of one or more fields in ascending or descending order. To do that you need to add an _order_ parameter to the JSON configuration with the list of fields & their corresponding direction of which to order to the output.
Assuming we are trying to order in ascending order the data of January. The corresponding JSON would be:
```json
{
"data": {
"fields": {
"amount": [
"select"
],
"date": [
"select"
] ,
"month": [
"select"
] ,
"year": [
"select"
]
}
},
"scale": {
"fields": []
},
"filter": {
"month": [
1
],
"year": [
2018
]
},
"order": {
"date" : "asc"
}
}
```
This will yield the following result:
***
## Table Name
As mentioned in the _data_ parameter article, setting the _table\_name_ parameter will enforce the Query Builder to retrieve data from a specific table in the data model.
For instance, assuming we were looking to confirm that the data requested by the Query Builder came from the "prim\_revenue" table, we would use the following JSON configuration:
```json
{
"data": {
"fields": {
"amount": [
"sum"
],
"month": [
"select"
],
"year": [
"select"
]
}
},
"scale": {
"fields": [
"month"
,
"year"
]
},
"filter": {},
"order": {},
"table_name":"prim_ca"
}
```
***
## Skip
The _skip_ sub-parameter allows you to skip certain lines from the final outputted ones.
For instance when the output is ordered in a certain manner, you want to remove the first 5 or the first 10 outputs. In this case, your JSON configuration would look like the following:
```json
{
"data": {
"fields": {
"date": [
"select"
],
"amount": [
"select"
]
},
"skip": 10
},
"scale": {
"fields": []
},
"filter": {},
"order": {}
}
```
***
## Hints
The _hints_ sub-parameter allows to have additional meta data in the JSON response from the Query Builder which provides the total number of lines that the request generates without taking into account the _skip_ and _limit_ parameters. You can add the _hints_ parameter as follow in the JSON configuration:
```json
{
"data": {
"fields": {
"date": [
"select"
],
"amount": [
"select"
]
},
"limit": 2,
"skip": 5,
"hints": true
},
"scale": {
"fields": []
},
"filter": {},
"order": {}
}
```
To see the effect on the query output, you need to visualize directly the raw JSON format. To do saw, in the editing view of the query select the option "JSON" in the output display window:
It will display the following result (we've collapsed the actual query output in the _results_ key which outputs the values as a table of JSON objects):
```json
{
"success": true,
"cache": false,
"timestamp": 1559896936,
"duration": 0.11,
"error": null,
"warning": null,
"hints": 13,
"query_params": {
"data": {
"fields": {
"date": [
"select"
],
"amount": [
"select"
]
},
"limit": 2,
"skip": 5,
"hints": true
},
"scale": {
"fields": []
},
"filter": {},
"order": {}
},
"table_name": "prim_income",
"results": [...]
}
```
***
## Total
Let's assume we are calculating the sum of a measure called "mydata" with respect to 2 scales called "my\_scale1" and "my\_scale2". The query would look like the following:
```
...
"data": {
"fields": {
"mydata": ["SUM"]
}
},
"scale": {
"fields": ["my_scale1","mys_cale2"]
}
...
```
This query will generate the equivalent of the following SQL query:
```
SELECT SUM(mydata) FROM .... GROUP BY my_scale1, my_scale2
```
And you will be able to easily use the output data to create a table such as below:
| | my\_scale1 1 | my\_scale1 2 |
| ------------ | ------------ | ------------ |
| my\_scale2 1 | sum 1x1 | sum 1x2 |
| my\_scale2 2 | sum 2x1 | sum 2x2 |
Adding the total attribute in your query:
```
...
"data": { },
"scale": { },
"total": {
"all": [],
"x": [myscale1],
"y": [myscale2]
}
...
```
Will allow the following rendering:
| X | my\_scale1 1 | my\_scale1 2 | total |
| ------------ | ------------ | ------------ | ------- |
| my\_scale2 1 | sum 1x1 | sum 1x2 | total.x |
| my\_scale2 2 | sum 2x1 | sum 2x2 | total.x |
| total | total.y | total.y | total |
Which would actually require 4 distinct SQL queries:
```
SELECT SUM(mydata) FROM ... GROUP BY my_scale1, my_scale2
SELECT SUM(mydata) FROM ...
SELECT SUM(mydata) FROM .... GROUP BY my_scale1
SELECT SUM(mydata) FROM .... GROUP BY my_scale2
```
***
[You can even send your questions directly by clicking here 👨🏻💻](https://help.ovhcloud.com/csm/fr-home?id=csm_index)
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-qb-json-format.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# ForePaaS JSON query format
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
The ForePaaS JSON query format uses multiple parameters as input and output to describe a query and its results.
The following articles document each individual query parameters, and illustrate their impact using a simple example of the revenue of a company. The available dataset is given as follows:
[Reference: 'data' parameter](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-data.md)
[Reference: 'scale' parameter](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-scale.md)
[Reference: 'filter' parameter](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-filter.md)
[Reference: other parameters](https://docs.dataplatform.ovh.net/legacy/api-reference-qb-others.md)
***
## Complete reference of parameters
The JSON configuration for a query takes in the following parameters.
| Parameter name | required | example | description |
| -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| data.fields | yes | \{ "income":\["sum"],"expense":\["sum"]} | Fields to return and their compute mode(s) (can be: select, select\_distinct, sum, min, max, avg, count, count\_distinct) |
| data.skip | no | 10 | Number of rows to skip |
| data.limit | no | 100 | Number of rows returned |
| data.hints | no | true/false | Total number of lines corresponding to the query (omitting data.limit) |
| filter | no | \{ "date": \{ "between": \["1420066800,1428962399"] }, "client\_code": \{ "in": \["08KD", "09CC"] }, "creation\_date": \{ "eq": \["2018-01-01"] }} | Apply filter(s) on the data to query. Filters can be: in, not\_in, between, not\_between, eq, not\_equal, like, not\_like, is\_null, is\_not\_null, gt (\>), gte (\>=), lt (\<), lte (\<=) |
| scale.fields | no | \["date",\["client\_code", "employee\_id"]] | Apply scale (group by) to the query. It can be a computed scale, or more than 1 scale. |
| evol.scale | no | "year" | scale to apply for evolution (weekday;week; month; quarter; semester; year) |
| evol.mode | no | "date""weekday" | date : comparison with same dateweekday : comparison with respecting the same weekday. |
| evol.depth | no | 5 | How much evolutions you want to get back |
| total.all | no | \[]\["client\_code"] | total queries per distinct scales (optional) |
| total.x | no | \["client\_code"] | Total queries for a specific attribute |
| order | no | \{ "date":"asc"} | order query output (asc or desc, asc by default) |
| table\_name | no | "prim\_vehicles" | Force to query in a specific table |
| database\_name | no | "data\_prim""data\_mart" | Force to query in a specific layer |
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# API reference: Data Manager
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md).
:::
When interacting with the Data Manager through the ForePaaS API, you have to use different endpoint patterns for different purposes.
Below, you will find the reference for these patterns.
- [Create a source](https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md#create-a-source)
- [Add tables to a source](https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md#add-tables-to-a-source)
- [Metas extraction](https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md#metas-extraction)
- [Add virtual attributes](https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md#add-virtual-attributes)
- [Save your table schema](https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md#save-your-table-schema)
- [Build tables](https://docs.dataplatform.ovh.net/legacy/api-reference-data-manager.md#build-tables)
***
## Create a source
Endpoint pattern:
- **POST /datastores/db?token=xxxxx:**
JSON Body example for a Dropbox source:
```json
{
"dbms": "dropbox",
"parameters": {
"path": "/folder/path",
"token": "xxxxxxxxxx"
},
"database_name": "dropbox_source",
"dwhName": "your_dwh_name",
"level_name": "source
}
```
***
## Add tables to a source
Endpoint pattern:
- **POST /metas/tables?dwh\_name=your\_dwh\_name\&token=xxxxx**
JSON Body example to add a table to a Dropbox source:
````json
{
"table_name": "your_file.csv",
"database_name": "dropbox_source"
} ```
You can also add several tables in one shot:
```json
[{
table_name": "file1.csv",
"database_name": "dropbox_source"
},
{
"table_name": "file2.csv",
"database_name": "dropbox_source"
}]
````
***
## Metas extraction
Endpoint pattern:
- **PUT /metas/sources/extract/source\_name/table/table\_name?dwhName=dwh\_name\&token=xxxxxxx**
If you need to extract metas from a file1.csv file from a Dropbox source, you need to search: _/metas/sources/extract/dropbox\_source/file1.csv_
This _call_ accepts settings like:
- sample: to extract metas on a data sample, not all (set it up on "true")
- sample\_max\_lines: if the sample is set up on "true", use it to set up the max number of lines you want for the sample
- flashzone: set up the option as "true" to have an overview of the file / table data on your DataPlant
***
## Add virtual attributes
Endpoint pattern:
- **POST /attributes/\?token=xxxxxxx**
Here is an example of a JSON Body:
```json
{
"attribute_name": "my_attribute",
"parameters": {},
"is_dimension": false,
"is_virtual": true,
"is_summable": true,
"sql": "SELECT COUNT(id) FROM clients.csv",
"is_measure": true
}
```
***
## Save your table schema
Endpoint pattern:
- **POST /logical/object?dwhName=dwh\_name\&token=xxxxxx**
Here is an example of a JSON Body:
```json
[
{
"attributes":[
{
"attribute_name":"conso_id",
"is_dimension":true,
"sources":[
{
"rules":[
],
"attributes":[
"dropbox/consomations.csv/conso_id"
]
}
],
"is_summable":false,
"attribute_path":null,
"fpui":{
"operation":"mapping"
},
"type":{
"main":"num",
"sub":"smallint"
},
"is_measure":false
},
{
"attribute_name":"ca",
"database_name":"data_prim",
"is_dimension":false,
"sources":[
{
"rules":[
],
"attributes":[
"dropbox /consomations.csv/ca"
]
}
],
"is_summable":false,
"table_name":"consomations",
"fpui":{
"operation":"mapping"
},
"type":{
"max":999.99,
"main":"num",
"precision":2,
"min":0.02
},
"attribute_path":null,
"is_measure":true
}
],
"object_name":"consomations",
"display_name":"consomations",
"object_type":"measure",
"to_materialize":true,
"measure_date_field":"date",
"key":[
"conso_id"
],
"date_type":false,
"relations":[
],
"indexes":[
]
}
]
```
***
## Build tables
Below, are the endpoints that have to be executed to build your tables.
Build logical structure of a Data Prim:
- **PUT /logical/build/prim?dwhName=dwh\_name\&token=xxxxxx**
Build logical structure of a Data Mart:
- **PUT /logical/build/prim?dwhName=dwh\_name\&token=xxxxxx**
Build physical structure:
- **PUT /datastores/db/physical?token=xxxxx**
Here is a JSON Body example:
```json
{
"levelsToBuild": "["data_prim"]",
"dwhName": "plw1uzhy_jm224kp5"
}
```
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-datastore.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# API Reference: Buckets
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Data Platform APIs](https://docs.dataplatform.ovh.net/landing-page-developers-apis.md).
:::
The ForePaaS Buckets are fully compatible with **AWS S3** client.
To interact with them, you will need to [generate an API key and Secret key in your Identity Access Manager](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) from a **ForePaaS Directory** account.
- [Using AWS CLI](https://docs.dataplatform.ovh.net/legacy/api-reference-datastore.md#using-aws-cli)
- [Using Rclone](https://docs.dataplatform.ovh.net/legacy/api-reference-datastore.md#using-rclone)
- [Using NodeJS](https://docs.dataplatform.ovh.net/legacy/api-reference-datastore.md#using-nodejs)
- [Using Python](https://docs.dataplatform.ovh.net/legacy/api-reference-datastore.md#using-python)
***
## Using AWS CLI
> In the base URLs in the code snippets below, you will need to replace `my-dataplant` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
1. Install the AWS CLI client from [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/)
2. Configure the AWS CLI, it is suggested to use [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
```bash
$ aws configure --profile mydataplant
AWS Access Key ID [None]: your_access_key
AWS Secret Access Key [None]: your_secret_key
Default region name [None]: forepaas
Default output format [None]:
```
1. Enable _AWS Signature Version 4_ for MinIO server.
```bash
aws configure set s3.signature_version s3v4 --profile mydataplant
```
1. A few command examples (replacing `my-dataplant` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain)):
```bash
# list your buckets
aws --profile mydataplant --endpoint-url https://my-dataplant.forepaas.io/datastore s3 ls
# list contents inside bucket
aws --profile mydataplant --endpoint-url https://my-dataplant.forepaas.io/datastore s3 ls s3://mybucket
# make a bucket
aws --profile mydataplant --endpoint-url https://my-dataplant.forepaas.io/datastore s3 mb s3://mybucket
# add an object to a bucket
aws --profile mydataplant --endpoint-url https://my-dataplant.forepaas.io/datastore s3 cp my-file.csv s3://mybucket
# delete an object from a bucket
aws --profile mydataplant --endpoint-url https://my-dataplant.forepaas.io/datastore s3 rm s3://mybucket/my-file.csv
# remove a bucket
aws --profile mydataplant --endpoint-url https://my-dataplant.forepaas.io/datastore s3 rb s3://mybucket
```
> To manage entrypoint URL inside a named profile, you may look at this documentation page: [configure entry point URL per profile](https://github.com/wbingli/awscli-plugin-endpoint).
***
## Using Rclone
> In the base URLs in the code snippets below, you will need to replace `my-dataplant` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
1. Install [rclone](https://rclone.org/) from [https://rclone.org/downloads/](https://rclone.org/downloads/)
2. Configure rclone (replacing `my-dataplant` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain)).
```bash
rclone config create my-dataplant s3 provider Minio region forepaas env_auth false
rclone config update my-dataplant endpoint https://my-dataplant.forepaas.io/datastore
rclone config update my-dataplant access_key_id YOUR_ACCESS_KEY
rclone config update my-dataplant secret_access_key YOUR_SECRET_KEY
```
1. Check rclone's config file location if needed
```bash
rclone config file
```
1. Check your configuration
```bash
rclone config show my-dataplant
[my-dataplant]
type = s3
provider = Minio
region = forepaas
env_auth = false
endpoint = https://my-datplant.forepaas.io/datastore
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
```
1. Some simple commands (please refer to rclone documentation for more details):
```bash
# listing buckets
rclone lsd my-dataplant:
# create a bucket
rclone mkdir my-dataplant:test-rclone
# generate a local docs for test purpuse
rclone gendocs docs
# copy ./docs/ to test-rclone
rclone copy docs my-dataplant:test-rclone/docs
# what is the size of a bucket or a path ?
rclone size my-dataplant:test-rclone
Total objects: 69
Total size: 239.570 kBytes (245320 Bytes)
# displays a tree
rclone tree my-dataplant:test-rclone
# move files in bucket
rclone move my-dataplant:test-rclone/docs/commands my-dataplant:test-rclone/commands
# syncing source to destination
rclone sync --dry-run docs my-dataplant:test-rclone/docs
# listing files in a human redable way
rclone lsl my-dataplant:test-rclone
# listing files with format options
rclone lsf --format "tsp" --recursive my-dataplant:test-rclone
# deleting files greater than 1k
rclone --min-size 1k lsl my-dataplant:test-rclone/commands
rclone --dry-run --min-size 1k delete my-dataplant:test-rclone/commands
rclone --min-size 1k delete my-dataplant:test-rclone/commands
# copy files in ./commands
rclone copy my-dataplant:test-rclone/commands commands
# delete an entire path
rclone delete my-dataplant:test-rclone/commands/
```
***
## Using NodeJS
> In the base URLs in the code snippets below, you will need to replace `[DATAPLANT_NAME]` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
```sh
# Add aws-sdk to your application with the following command
yarn add aws-sdk
# Or
npm i -s aws-sdk
```
```js
const AWS = require('aws-sdk')
const fs = require('fs')
// Modify those values for your environment
let accessKey = '[ACCESS_KEY]'
let secretKey = '[SECRET_KEY]'
let endpoint = 'https://[DATAPLANT_NAME].eu.dataplatform.ovh.net'
let bucket = '[BUCKET_NAME]'
let region = 'forepaas'
// Constructs a service object. This object has one method for each API operation.
const dataStoreClient = new AWS.S3({
accessKeyId: accessKey,
secretAccessKey: secretKey,
endpoint: `${endpoint}/datastore`,
region: region,
signatureVersion: 'v4',
s3ForcePathStyle: true
})
////////////////////////////////
// Adds an object to a bucket //
////////////////////////////////
let uploadStream = fs.createReadStream('./file.csv')
dataStoreClient.putObject({
Bucket: bucket,
Key: 'file.csv',
Body: uploadStream
}, (err) => {
if (err) console.error(err)
else console.info('File uploaded')
})
/////////////////////////////////////
// Retrieves objects from a bucket //
/////////////////////////////////////
let downloadStream = fs.createWriteStream('./file_downloaded.csv')
dataStoreClient.getObject({
Bucket: bucket,
Key: 'file.csv'
})
.createReadStream()
.on('error', (err) => {
console.error(err)
})
.pipe(downloadStream)
.on('close', () => {
console.info('File downloaded')
})
/////////////////////////////////////
// Returns some or all (up to 1000) of the objects in a bucket.
/////////////////////////////////////
dataStoreClient.listObjects({
Bucket: bucket
}, (err, data) => {
if (data && data.Contents) {
data.Contents.forEach(file => {
console.info(`${file.Key} (${file.Size} bytes)`)
})
}
})
```
***
## Using Python
> In the base URLs in the code snippets below, you will need to replace `[DATAPLANT_NAME]` with your [Project's subdomain](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md#project-subdomain).
```sh
# Install boto3 with the following command:
pip install boto3
```
```python
import boto3
from botocore.client import Config
# Modify those values for your environment
dataplant_url = 'https://[DATAPLANT_NAME].eu.dataplatform.ovh.net'
access_key = '[ACCESS_KEY]'
secret_key = '[SECRET_KEY]'
bucket = '[BUCKET_NAME]'
region = 'forepaas'
# Constructs a service object. This object has one method for each API operation.
datastore = boto3.client(
's3',
endpoint_url='{dataplant}/datastore'.format(dataplant=dataplant_url),
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
config=Config(signature_version='s3v4'),
region_name=region
)
# Adds an object to a bucket
datastore.upload_file(
Bucket=bucket,
Filename='./file.csv',
Key='file.csv')
# Retrieves objects from a bucket
datastore.download_file(
Bucket=bucket,
Filename='./file_downloaded.csv',
Key='file.csv')
# Returns some or all (up to 1000) of the objects in a bucket.
# You can use the request parameters as selection criteria to return a subset of the objects in a bucket.
if datastore.list_objects(Bucket=bucket):
for file in datastore.list_objects(Bucket=bucket)['Contents']:
print('{name} ({size} bytes)'.format(name=file['Key'], size=file['Size']))
```
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-iam.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# API Reference: Identity Access Manager
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the current IAM API reference](https://docs.dataplatform.ovh.net/developers-api-authentication.md).
:::
This page contains the API reference for the [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md). IAM APIs will always look like this:
```
https://`%dataplant_url%/iam
```
- [Login endpoint](https://docs.dataplatform.ovh.net/legacy/api-reference-iam.md#login-endpoint)
- [Logout endpoint](https://docs.dataplatform.ovh.net/legacy/api-reference-iam.md#logout-endpoint)
***
## Login endpoint
### Using a login/password method
This authentication mode allows you to [connect with a login and password](https://docs.dataplatform.ovh.net/iam-auth-providers.md#project) defined in your [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
Example request :
```
curl -X POST \
https://mydataplant.eu.dataplatform.ovh.net/iam/login \
-H 'Content-Type: application/json' \
-d '{
"login":"LOGIN",
"password":"****",
"app_id":"APPID",
"auth_mode":"forepaas_id_resource_owner"
}'
```
Example response :
```json
{
"uid": "John Doe", // The unique id of the logged-in user
"user": "M. John Doe", // The full name of the logged-in user
"firstname": "John", // The first name of the logged-in user
"lastname": "Doe", // The last name of the logged-in user
"civility": "M. ", // The civility of the logged-in user
"login": "john.doe", // The login of the logged-in user
"password_renew": false, // flag to know if the has to change his password "password_expiration": "2020-02-28T17:17:10.274Z ", // password expiration
date
"token": "f54e34a56b897"
}
```
| | | |
| :--------- | :----: | ----------------------------------------------------------------------------------------: |
| login | String | The login of the user, as received in the registration email. |
| password | String | The password of the user, as received in the registration email. |
| app\_id | String | The application identifier (set to 'forepaas' to login to the project's IAM in general) |
| auth\_mode | String | The authentication mode to used. The value must be set to “forepaas\_id\_resource\_owner” |
### API Key Credentials
This authentication mode allows you to [connect with an API and secret key](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md) defined in your [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md).
Example request:
```
curl -X POST \
https://mydataplant.eu.dataplatform.ovh.net/iam/login \
-H 'Content-Type: application/json' \
-d '{
"apikey":"APIKEY",
"secretkey":"****",
"app_id":"APPID",
"auth_mode":"apikey"
}'
```
Example response:
```json
{
"uid": "John Doe", // The unique id of the logged-in user
"user": "M. John Doe", // The full name of the logged-in user
"firstname": "John", // The first name of the logged-in user
"lastname": "Doe", // The last name of the logged-in user
"civility": "M. ", // The civility of the logged-in user
"login": "john.doe", // The login of the logged-in user
"password_renew": false, // flag to know if the has to change his password "password_expiration": "2020-02-28T17:17:10.274Z ", // password expiration
date
"token": "f54e34a56b897"
}
```
| | | |
| :--------- | :-------------------: | --------------------------------------------------------------------------------------: |
| apikey | String \[A-Za-z0- 9-] | The api key provided by the new device registration. |
| secretkey | String \[A-Za-z0- 9-] | The secret key provided by the new device registration. |
| app\_id | String | The application identifier (set to 'forepaas' to login to the project's IAM in general) |
| auth\_mode | String | The authentication mode to used. The value must be set to “apikey” |
***
## Logout endpoint
In order to delete a token, you can call the logout endpoint.
Example request:
```
curl -X DELETE \
https://mydataplant.eu.dataplatform.ovh.net/iam/logout?app_id=*****&token=***** \
-H 'Content-Type: application/json'
```
Example response:
```json
{
"success": true,
"status": 200,
"duration": 0.02
}
```
---
url: https://docs.dataplatform.ovh.net/legacy/api-reference-store.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# ForePaaS Store API Reference
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the current documentation](https://docs.dataplatform.ovh.net/landing-page-services.md).
:::
The [ForePaaS Store](https://docs.dataplatform.ovh.net/legacy/project-marketplace.md) is the collaborative space on the platform where each organization's member can create, share, and/or access code templates and packages.
Each marketplace (application templates, API extensions, etc..) has [its own endpoints](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#list-of-all-store-apis). They all start with:
```
https://store.forepaas.io/
```
After getting an authorization token, **creating a package in the store is done in 2 steps**:
1. create the package in the correct marketplace
2. upload a ZIP file containing the code for the first version
The same API endpoints are used to update package information and version.
- [Getting an authorization token](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#getting-an-authorization-token)
- [Create or update a package](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#create-or-update-a-package)
- [Add a version to your package](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#add-a-version-to-your-package)
- [Get package information](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#get-package-information)
- [Get package version](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#get-package-version)
- [List of all store APIs](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#list-of-all-store-apis)
- [Application templates](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#application-templates)
- [API templates](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#api-templates)
- [Tips and Troubleshooting](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#tips-and-troubleshooting)
:::warning
Currently, only **Application templates** and **API Templates** are fully supported.
:::
***
## Getting an authorization token
For each call referenced below, you will need an **authorization token**.
!> You must have a [ForePaaS account](https://docs.dataplatform.ovh.net/manage-organizations-profile.md), and be part of at least one [organization](https://docs.dataplatform.ovh.net/manage-organizations.md) to be able to upload packages and version.
To get an authorization token,
1. Open the ForePaaS Platform and open a Project
2. Open the **browser console** (_Ctrl + Shift + C_ on Google Chrome)
3. From the App's panel, choose **+** to open the new application dialog box
4. In the browser's console, go to **Network tab** and filter for _store.forepaas.io_
5. There should be one GET for `https://store.forepaas.io/v1`
6. Right click and **Copy as cURL**, that contains Authorization token and will serve as basis for next Store API calls.
***
## Create or update a package
Create/update a package in your **private store**, meaning only the members of your [organization](https://docs.dataplatform.ovh.net/manage-organizations.md) will be able to see it.
> It is currently impossible to upload a package to the public store.
The following reference shows how to upload a package for the _App templates_. See [below](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#list-of-all-store-apis) the list of all APIs for other marketplaces.
**Example request**
```bash
curl 'https://store.forepaas.io/v1/packages/app/templates/my-app' \
-X 'PUT' \
-H 'authority: store.forepaas.io' \
-H 'accept: application/json, text/plain, */*' \
-H 'content-type: application/json;charset=UTF-8' \
-H 'origin: https://store.forepaas.io' \
-H 'Authorization: Bearer ' \
--data-raw '{
"name":"My App",
"owner":"",
"category":"My package",
"i18n":{
"fr":{
"name": "name fr",
"description": "desc fr",
"long_description": "long desc fr"
},
"en":{
"name": "name en",
"description": "desc en",
"long_description": "long desc en"
},
"jp":{
"name": "name jp",
"description": "desc jp",
"long_description": "long desc jp"
}
},
"image":"",
"preview":[, , ...]
}' \
--compressed
```
**Example response**
```json
{
"status": 200,
"message": "Package updated with success",
"id": ":my-app",
"package": {
"image": "",
"preview": [],
"name": "My App",
"i18n": {
"fr": {
"name": "name fr",
"description": "desc fr",
"long_description": "long desc fr"
},
"en": {
"name": "name en",
"description": "desc en",
"long_description": "long desc en"
},
"jp": {
"name": "name jp",
"description": "desc jp",
"long_description": "long desc jp"
}
},
"owner": "",
"category": [
"My package"
],
"type": "modules",
"id": ":my-package",
"versions": [],
"downloaded": 0
}
}
```
:::warning
Make sure to update _YOUR\_AUTH\_TOKEN_ with the [token value retrieved above](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#getting-authorization-token).
:::
| | | |
| ---------: | :------: | :------------------------------------------------------------------------------------------------------------------------------------------------- |
| _name_ | `String` | Technical name of your package |
| _owner_ | `String` | ID of your organization. This package will only be available by Project of that organization |
| _category_ | `String` | Category of the package |
| _i18n_ | `Object` | Display and translations for your package. You can add 3 fields (_name_, _description_ and _long\_description_) in French, English and/or Japanese |
| _image_ | `String` | Image of your package. It has to be in base 64 |
| _preview_ | `Array` | Preview images of your package. It has to be in base 64 |
***
## Add a version to your package
Add a version of your package with a _.zip_ file
**Example request**
```bash
curl 'https://store.forepaas.io/v1/packages/app/templates/:my-app/' \
-X 'PUT' \
-H 'authority: store.forepaas.io' \
-H 'accept: application/json, text/plain, */*' \
-H 'content-type: application/zip' \
-H 'origin: https://store.forepaas.io' \
-H 'Authorization: Bearer ' \
--compressed --data-binary "@"
```
**Example response**
```json
{
"success": true,
"status": 200,
"duration": 0.02
}
```
:::warning
Make sure to update _YOUR\_AUTH\_TOKEN_ with the [token value retrieved above](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md#getting-authorization-token).
:::
***
## Get package information
This endpoint allows to check current information for a given package.
```bash
curl 'https://store.forepaas.io/v1/packages/app/templates/:my-app' \
-X 'GET' \
-H 'authority: store.forepaas.io' \
-H 'accept: application/json, text/plain, */*' \
-H 'origin: https://store.forepaas.io' \
-H 'Authorization: Bearer '
```
***
## Get package version
This endpoint allows to download a given package version.
```bash
curl 'https://store.forepaas.io/v1/packages/app/templates/:my-app/' \
-X 'GET' \
-H 'authority: store.forepaas.io' \
-H 'origin: https://store.forepaas.io' \
-H 'Authorization: Bearer ' \
--output my-app_1.0.0.zip
```
***
## List of all store APIs
### Application templates
The API for [application templates](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md#create-an-application) is:
```
https://store.forepaas.io/v1/packages/app/templates/
```
### API templates
The API for [API templates](https://docs.dataplatform.ovh.net/app-services-apis-deploy.md) is:
```
https://store.forepaas.io/v1/packages/api/templates/
```
***
***
## Tips and Troubleshooting
- when package is hosted in a Git repository, keep the JSON describing the package as a dedicated source file.
- make sure the `` is a usual version number such as `1.0.0` for instance.\
By default, ForePaaS Store will deliver latest version based on version ordering.
- make sure that ZIP file contains only source files and not intermediate code or Git folders (such as `.git` or `node_modules` for instance)
- make sure the ZIP file meets package expectation.\
For instance, in App's template, a common mistake is to have a _root folder_ inside the ZIP file rather than the contents at the root of the ZIP archive.
**Good ZIP archive**
```
.
|-- config
|-- forepaas
|-- public
|-- scripts
`-- src
```
**Baz ZIP archive**
```
my-app
|-- config
|-- forepaas
|-- node_modules
|-- public
|-- scripts
`-- src
```
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/project-marketplace.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# ForePaaS Store
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the connector catalog](https://docs.dataplatform.ovh.net/landing-page-connectors.md).
:::
The platform has been designed so that each user can create, share, and/or access pieces of work through the ForePaaS Store, at each step of the data production chain:
- **[Collect](#connectors-marketplace)** : to easily connect your data sources
- **[Transform](#dpe-marketplace)** : to work on pre-built ETL actions
- **[Data Science](#machine-learning-marketplace)** : to leverage artificial intelligence algorithms
- **[Expose](#api-and-app-marketplace)** : to integrate widgets and dashboards templates
The ForePaaS Store gathers dozens of tools, models, templates, etc. allowing you to save time when creating analytics and AI applications.
***
## Connectors Marketplace
Within the Connectors, the ForePaaS Store lets you collect in a few clicks any type of data, from any source.
[Learn more about connectors](https://docs.dataplatform.ovh.net/landing-page-connectors.md)
***
## DPE Marketplace
Inside the Data Processing Engine, the ForePaaS Store grants you access to 10+ pre-configured actions to let you transform your data in the most efficient way. Pick the action you need (aggregate, load, delete...) or create a custom action.
[Learn more about Data Processing Engine actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md)
***
## Machine Learning Marketplace
ForePaaS lets you integrate all components of an AI solution, whether it is entirely pre-trained models or specific parts of an ML pipeline such as an estimator or a score.
[Learn more about the Machine Learning Manager](https://docs.dataplatform.ovh.net/legacy/ml.md)
***
## API and APP Marketplace
You can create your own templates and share them within an organization or with other ForePaaS users. Easily build your API with our standard NodeJS API template.
[Learn more about API creation](https://docs.dataplatform.ovh.net/app-services-apis-deploy.md)
In the App Manager, create an analytical dashboard from scratch with our React templates.
Finally, the extensions Marketplace allow you to enrich your apps and APIs with new features.
[Learn more about APP creation](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md)
---
url: https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Version 3.0 Changes (2022-01-11)
:::warning
This page describes the **ForePaaS Legacy Platform**, which no longer accepts new signups. If you are using OVHcloud Data Platform, see [the Analytics Manager documentation](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md).
:::
The [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md) is the name of version 3 of the Query Builder on the Platform. This page lists all significant changes.
- [1. Dataset tab](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md#_1-dataset-tab)
- [2. New queries](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md#_2-new-queries)
- [3. What happens to my existing queries?](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md#_3-what-happens-to-my-existing-queries)
- [4. Versioning of queries](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md#_4-versioning-of-queries)
- [5. Query engines](https://docs.dataplatform.ovh.net/legacy/am-v3-changes.md#_5-query-engines)
***
## 1. Dataset tab
The dataset tab from the classic Query Builder no longer exists in the Analytics Manager. All the information about tables names and number of rows are now found in the **[tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) page of the [Data Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)**.
***
## 2. New queries
In the Analytics Manager, it is possible to create and run queries in any of two ways:
- Using a [visual point-and-click builder](https://docs.dataplatform.ovh.net/analytics-manager-queries-visual-builder.md): the interface has been revamped for a much more comfortable use, namely the addition of an Order section to sort your results!
- Using [SQL](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md): new, directly query your tables in SQL!
***
## 3. What happens to my existing queries?
Queries that were created in the classic Query Builder will be migrated to the new Analytics Manager, into the [queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) tab.
If applicable, they will still be linked to your [apps](https://docs.dataplatform.ovh.net/landing-page-app-services-apps.md) through the same _request\_id_, without needing to change anything.
Please note that the new Analytics Manager interface **doesn’t allow** for editing advanced JSON parameters anymore. As such, [such advanced options](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) that were added in your queries in the classic Query Builder will not be editable in the new Analytics Manager. However, the migrated queries will still be saved with them and as such will **return the same result** as in the classic Query Builder upon execution.
***
## 4. Versioning of queries
Queries are now versioned at the [repository](https://docs.dataplatform.ovh.net/analytics-manager-queries.md#versioning-queries) level, instead of at the query level like in the classic Query Builder.
Use the **the platform versioning system** to manage production and development frameworks, and work on your analytics without breaking your deployed applications.
***
## 5. Query engines
It is now possible to activate a [query engine](https://docs.dataplatform.ovh.net/analytics-manager-resources.md) in the Analytics Manager.
Query engines allow you to **scale your queries both vertically and horizontally**, especially for Big Data analytics, and give you access to unique new features: SQL querying, query history, query execution metrics, fine-grained data access control (_coming soon!_), and more.
## Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/legacy/release-notes-before-ga.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Release notes before GA
:::warning
Releases from before general availability, kept for reference and no longer maintained. Every release since GA is on the [current release notes](https://docs.dataplatform.ovh.net/release-notes.md).
:::
## Feb 29th 2024
### Platform
- Several bug fixes and improvements
***
## Feb 22nd 2024
### Platform
- Several bug fixes and improvements
***
## Feb 15th 2024
### Platform
- Several bug fixes and improvements
***
## Feb 8th 2024
### Platform
- Several bug fixes and improvements
***
## Feb 1st 2024
### Platform
- Updated ForePaaS logos to OVHcloud logo
### Analytics Manager
- Fixed history metrics not being returned correctly
***
## Jan 25th 2024
### Data Processing Engine
- Removed logs that are not formatted correctly
***
## Jan 18th 2024
### Analytics Manager
- Fixed queries, APIs and applications returning an error when trying to query data from standard Lakehouse Manager datasets.
### AI App Builder
- Removed repeated column in Knowledge Bases list
- Visual fixes and improvements
### Storage Engines
- Sorted storage engines list in alphabetical order
- Added a 'Location' filter for interface
- Updated 'Engine' filter to filter the interface on the type of engine
### Platform
- Updated Profile & Profile > Security pages to be in auto-save mode
***
## Jan 11th 2024
### Lakehouse Manager
- Removed check on last dataset preventing to delete it
- Added warning message when trying to delete a dataset used in Machine Learning Manager
### Support Portal
- Fixed 'Open in a new tab' option not being available for support cases
- Fixed direct URL opening of a support case redirecting to the portal home
***
## Dec 28th 2023
### Platform
- Several bug fixes and improvements
***
## Dec 21th 2023
### Lakehouse Manager
- Fixed query estimator still appearing when failed
- Fixed display of type in datasets list
- Fixed bugs on the import and export of configurations
### AI App Builder
- Added redirection link to linked app from assistant interface
- Updated possible options from the home of the project
### Identity Access Manager
- Fixed appearance of MFA feature
***
## Dec 14th 2023
### Lakehouse Manager
- Updated 'Associated resources' table in policy tag page to show attributes' table
- Added sections in sidebar of Lakehouse
### Platform
- Added template for AI App Builder
***
## Dec 8th 2023
### Lakehouse Manager
- Fixed name and type of datasets sometimes not displaying correctly
### Platform
- Added confirmation modal before stopping a running job
- Fixed tile sometimes being blank after adding service in project
***
## Nov 30th 2023
### Platform
- Fixed sidebar links not working when there is no project in organization
### Data Processing Engine
- Improved Aggregate action to support manual adding of GROUP BY
***
## Nov 23rd 2023
### Identity Access Manager
- Added OVH logo for emails.
- Bugfixed login without authentication provider except ovh.
### Data Processing Engine
- Added message block when creating pyspark action from action.
### Lakehouse Manager
- Fixed search in groups/users when adding policy tags.
- Fixed display of policy tags in table / tree view.
***
## Nov 16nd 2023
### Lakehouse Manager
- Added interface to grant access to policy tags directly from Lakehouse Manager
- Fixed queries running automatically when opening Explorer
- Fixed attributes being duplicated twice when duplicating a table
- Multiple bug fixes and improvements related to the build status of tables
### Data Processing Engine
- Python 3.6 is now decommissioned. All actions and workflows have been migrated to the next available version.
- Added option to create a PySpark action when a Python Load/Aggregate action is detected to insert data in a Lakehouse Manager dataset.
***
## Nov 9nd 2023
### Lakehouse Manager
- Multiple bug fixes and performance improvements
### Analytics Manager
- Fixed a bug where it was impossible for a new user to login to the deployed-live dashboard URL
- Added a confirmation modal before deactivating a query engine
### Platform
- Updated password validation rules in the profile
***
## Nov 2nd 2023
### Storage Engines
- Reworked pages for _Storage Engines_ service in the interface: _Storage Engines_ can now be opened as a standalone service from either the main sidebar or the services list.
### Lakehouse Manager
- Fixed file size ordering not working in Buckets
- Multiple bug fixes and improvements to Datasets and Tables
### Analytics Manager
- Added a link to the query when an error occurred in a dashboard
### Identity Access Manager
- Fixed a bug when logging-in using login/password would not not work
***
## Oct 26th 2023
### Platform
- Added link to projects' templates' store in the services list
- Added locked visual in sidebar and header if no permission to header
- Removed some unused permissions in organization IAM
- Improved error message when reseting password
### Data Manager
- Data Manager service has been split in **Data Catalog** and **Lakehouse Manager**.
- Data Manager graphical interface is depreciated
### Lakehouse Manager
- Introducing Lakehouse Manager!
- Lakehouse Manager is a serverless data warehouse to store and organize any dataset
- You can now create custom datasets
- You can now create policy tags for Advanced Data Access Control
### Data Catalog
- Introducing Data Catalog!
- Data Catalog allows you to define connections and data assets across your organization
### Data Processing Engine
- Added a new action: _CSV to PostgreSQL_
### Identity Access Manager
- Removed incorrect display of 'Log History' feature for DataplantID and API key authentication methods
***
## Oct 19th 2023
### Platform
- Multiple bug fixes and improvements
### Applications
- Fixed metric of FPU in APIs/apps not displaying correct value
***
## Oct 12th 2023
### Platform
- "Organization not found" page will be showed for invalid link.
- Increased text area size on Support portal page.
- Fixed default configuration of storage engine when create new project.
### Data Manager
- Fixed secret+key label missing for 'S3-compatible storage' source
### Data Processing Engine
- Changed source icon size.
- Time format updated.
### Machine Learning Manager
- Time format updated.
### Analytics Manager
- Analytics Manager is now generally available
***
## Oct 5th 2023
### Analytics Manager
- Analytics Manager Dashboards are now in Beta release
- Fixed access to live deployed dashboard
### Data Processing Engine
- Fixed duration not showing when it is under 1 second
### Application Services
- Fixed certificate management for apps
- Fixed configuration for login screens disappearing
***
## Sep 28th 2023
### Platform
- Fixed long projects/organizations names being shortened with ellipsis
### Data Processing Engine
- Fixed bug for repo version override
- Removed app\_id & type query strings from the code snippet to launch an action/workflow
### App Manager
- Fixed a bug where it was impossible to remove a tag
***
## Sep 21st 2023
### Platform
- Fixed a bug where the Storage Engines page would be accessible but empty if user has no permission on storage engine instances. The page is now un-accessible in this case
- Fixed bug where the list of invites (to organizations) on the profile page would be inaccurate
- Fixed support page layout
### Data Processing Engine
- Fixed default version in actions being Python 3.11
***
## Sep 14th 2023
### Platform
- We are rolling out a major update to the backend of the organization's IAM! Performances and stability are much improved on all of the platform pages.
- Updated sign-in screen when session is timed out inside a project
***
## Sep 7th 2023
### Platform
- Add option to sign in / sign up to ForePaaS using OVHcloud account
### Application Services
- Fixed automatic deployment when creating app/api
***
## Aug 31th 2023
### Data Processing Engine
- Added possibility to order/limit when listing objects by API
### Data Manager
- Added possibility to order/limit when listing objects by API
### Machine Learning Manager
- Added possibility to order/limit when listing objects by API
***
## Aug 24th 2023
### Data Processing Engine
- The version of PySpark is now displayed in Custom PySpark actions
- The minor version number for Python is no longer displayed in the interface
### Machine Learning Manager
- The minor version number for Python is no longer displayed in the interface
### Analytics Manager
- Fixed displaying of paywall based on query engine activation
### Control Center
- Added support for quotas. You can manage Control Center quotas in the Settings of the CC
- Fixed redirection to Control Center in the header which sometimes opened the wrong organization
***
## Aug 17th 2023
### Platform
- Various bug fixes and improvements
***
## Aug 10th 2023
### Data Manager
- Various bug fixes and improvements
***
## Aug 3rd 2023
### Data Manager
- Added two new connectors: **Shadow Drive**, and **Google Cloud Storage**.
- Fixed verify\_ssl option for OpenSearch connector to be false by default
- Added a data type for virtual attributes on creation
- Fixed a bug when creating an index with 2 attributes on a PostgreSQL table
### Data Processing Engine
- Fixed a bug where the service would be softlocked if Control Center is unresponsive
***
## Jul 27th 2023
### Platform
- Fixed status member pending till he creates his account
- Fixed access to project being available if creation aborted
### Analytics Manager
- Added support to `eq` and `ne` operators for JSON queries
***
## Jul 20th 2023
### Platform
- Bug fixes and performances improvements
***
## Jul 13th 2023
### Storage Engines
- Introducing a new engine in Alpha release: Polar Data. Polar Data is a modern lakehouse engine built by ForePaaS.
### Data Manager
- Explorer can now be used without activating a query engine _(currently only available on projects deployed on OVHcloud provider)_
***
## Jul 6th 2023
### Platform
- Multiple improvements to the display performances of home page
- Added tooltips on kpis of project home
- Make editing message on project home always visible
- Fixed status of project when adding a service
- Fixed sidebar projects sometimes not up to date
- Fixed free fpu included in organization overview
***
## Jun 29th 2023
### Platform
- Performance improvements and multiple bug fixes to the new home pages and navigation
### Analytics Manager
- Added new option to export query result to ForePaaS Buckets, and receive the download link by email. Use this option if your query results are bigger than 10 MB and you cannot use the local download.
***
## Jun 22th 2023
### Platform
- Introducing ForePaaS Home v2!
- Dataplants are now called Projects! They can be made of any number of services bundled together
- A new header allowing to navigate between services, projects and organizations
- A new sidebar allowing to navigate between services
- A new organization home page introducing widgets and a 'recently opened' section
- A new project home page where you can edit the layout and get detailed metrics
- Projects can now be created from templates, allowing to get started with only some services before expanding to others
***
## Jun 15th 2023
### Control Center
- Fixed bug on logs streaming endpoint
***
## Jun 9th 2023
### Platform
- Fixed opening of versioning panel in Data Processing Engine, Analytics Manager and Machine Learning Manager
### Analytics Manager
- Migrated all queries from Query Builder Legacy to Analytics Manager, in a new repository "QB Queries"
- Query Builder Legacy is now removed from the console and decommissioned from the ForePaaS Platform
### Data Processing Engine
- Added possibility for Custom actions and Load actions using a streaming-type connector to have no timeout, if the job's execution mode is Always-up
- Fixed an issue on MQTT connector related to the acknowledgement of message consumption that would sometimes cause a timeout of the connection
***
## Jun 1st 2023
### Platform
- Fixed buttons sometimes disappearing on lists if a column was enlarged a lot
### Analytics Manager
- Fixed attributes from MART Tables not appearing in the choice of the query editor
- History loading is now better handled in Analytics Manager & Data Mnaager Explorer
### Data Processing Engine
- Statistics about insert/update are now disabled by default in order to optimize performances
- Parameter `statistics` is now false by default in the _bulk\_insert_ SDK function
- Statistics about insert/update are now computed only if the log level of a job is set to _DEBUG_
***
## May 25th 2023
### Data Manager
- Fixed preview table button sometimes previewing wrong attributes
### Analytics Manager
- Fixed regrouped attributes which sometimes would appear in wrong order on a bar chart
***
## May 11th 2023
### Data Manager
- Added a warning when linking a storage engine that is not in the same location as the Dataplant
- Fixed translation issues in some modals
- Fixed arguments not being correctly required on MongoDB connector
### Analytics Manager
- Fixed queries not running if Japanese characters were in the title
- Fixed virtual attributes sometimes not appearing in the options for the visual query builder
- Fixed gauge chart not taking the dashboard's chart into account
***
## May 4th 2023
### Analytics Manager
- Added a new chart type: **gauge chart**. Make sure your projects are on-track and monitor how far KPIs are from reaching targets with this new visualization type!
- Added new option to break the X-axis down into multiple attributes, in order to regroup bar charts, line charts and area charts by specific Scale attributes
- Added more fine-grained _origin_ types for the query history, with hyperlinks to the relevant object
### Data Processing Engine
- Increased the size of a workflow's visual interface, in order to make the name of its actions more visible
- Fixed a bug on the workflow version override which would sometimes not take effect
### Identity Access Manager
- Fixed a bug on the service selection in the role condition builder
***
## Apr 27th 2023
### Data Manager
- Added a _type_ parameter for virtual attributes
- Fixed "port" parameter in MQTT connector
### Analytics Manager
- Added a more fine-grained origin for queries in History
- Added a more fine-grained status for queries in History
- Tiles/filters are now automatically removed from dashboards if the related query is deleted
### Data Processing Engine
- Added a new action: _MySQL to Parquet_
- Fixed a bug where the status of a workflow would sometimes not reflect the status of stages
***
## Apr 20th 2023
### Data Manager
- Bug fixed on MQTT connector
### Analytics Manager
- Small cosmetic improvements to dashboards
***
## Apr 13th 2023
### Data Processing Engine
- Python 3.6.5 is now depreciated. Please migrate all your actions to the next stable version of Python: 3.9.9
- Python 3.6.5 will be fully decommissioned on **October 19th 2023**
- Fixed multiple git conflicts
### Data Manager
- Added Location column when binding a new storage engine, and 'recommended' tag if the storage engine is in the same location as dataplant
- Fixed logs in console being swapped when launching another job
### Machine Learning Manager
- Python 3.6.5 is now depreciated. Please migrate all your pipelines to the next stable version of Python: 3.9.9
- Python 3.6.5 will be fully decommissioned on **October 19th 2023**
***
## Apr 6th 2023
### Data Processing Engine
- Added support for **Python 3.11**
- Added merge conflict user warning for Git versioning
### Analytics Manager
- Keep attributes sorted in the same order when previewing a table from the Data Manager
### Application Services
- Multiple bug fixes on timeout after building
- Better handle when build finish job
***
## Mar 30th 2023
### Data Processing Engine
- Alerts integration in DPE Workflow UI.
- Confirm popin if creating a workflow with actions that use different Python versions.
### Analytics Manager
- Improved animation to display edit chart settings panel in dashboard.
- For Pie charts group data together if more than 20.
- Fixed reload jobs and logs in logs explorer
### Control Center
- Reload of jobs in Logs Explorer has been fixed.
***
## Mar 23rd 2023
### Analytics Manager
- Added inversion evolution colors and displaying dimension.
- Bug fixed where offset percentiles labels so that they don't overlap.
### Data Processing Engine
- Alerts tab in Actions to create Control Center Alerts.
- Actions/workflows names can be clicked in Jobs tab to redirect to the action/workflow.
### Machine Learning Manager
- Pipeline names can be clicked in Jobs tab to redirect to the pipeline.
***
## Mar 16th 2023
### Analytics Manager
- Added _trend line_ option in the settings of scatterplot
- Fixed red error message if user has no access to IAM
- Fixed duplicate attributes appearing in query creation choices if both virtual and physical attribute have the same name
### Applications
- Multiple bug fixes
### Platform
- Fixed Azure storage engine creation
- Added new filters to the Jobs page statuses
***
## Mar 9th 2023
### Analytics Manager
- Added a new chart type: [_Scatterplot_](https://docs.dataplatform.ovh.net/analytics-manager-queries-charts.md#use-a-scatterplot), to represent the correlation between two metrics as colored bubbles
- Added possibility to edit all chart settings directly from a dashboard
- Added info in query history that the query has been launched from a deployed dashboard
- Fixed a bug when drilling down on a chart which already had an Order set at query level
### Data Processing Engine
- Added a new action: _PostgreSQL to Parquet_
- Renamed action _table-to-files_ in interface and improved its visual aspect, to add an option to choose the output format
- Added ability to disable stats calculation on `bulk_insert` function from SDK, in order to reduce the computing time
- Disabled Always-up option for PySpark actions
***
## Mar 2nd 2023
### Analytics Manager
- Introducing the possibility to [drill-down](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-edit.md#drill-down-on-the-dashboard) on dashboards' charts! Click on a part of any chart in order to filter everything on related values, or to further explore relevant data.
- Added option to download the content of a dashboard's tile as a CSV
- Fixed timepicker's display reseting when opening it from a dashboard
- Fixed timepicker's WTD option not starting on Monday in some countries
### Control Center
- Updated interface for Logs Explorer in streaming mode in order to stick to the bottom of page when new logs appear
***
## Feb 23th 2023
### Analytics Manager
- Added _Download as a CSV_ option to export the result of query as a tab-separated .CSV local file (limited to 10 MB)
### Data Processing Engine
- Disabled possibility in GUI to mix standard and PySpark actions in the same workflow
- Fixed environement variables automatically opening in preferences
- Fixed language version override on workflows sometimes not displaying
### Control Center
- Fixed logs sometimes being displayed on same line when streaming
- Redirecting to the logs explorer from another component now switches Streaming mode on if it is a currently running job
### Platform
- Fixed drag and dropping repositories which would sometimes randomize their order
- Renamed job status from Pending to Running
- Fixed search if category selected
***
## Feb 20th 2023
### Data Processing Engine
- Added a parameter _apply\_rules_ (default value = _false_) in function `extract` of the SDK in order to apply blueprint of rules.
### Analytics Manager
- Added _Copy to clipboard_ option to copy the result of query to the clipboard (limited to 1 MB)
- Replaced the dashboard subtitle by the dashboard's description
- Improved the default visual aspect of some charts
- Updated the options for the size of query engine. 4 FPU-S is now XS, and added a new coming soon option: XL (64 PFU-S).
- Fixed visual display bugs on dashboard tiles (tooltip or options that would sometimes go behind the tile)
### Data Manager
- Fixed a bug in Trino connector where it would return an error `'Trino' object has no attribute '_read_sql'`
### Control Center
- In Logs Explorer, clicking on refresh now refreshes the list of executions for jobs
### Platform
- Added repository\_id in url of following pages: DPE Actions, MLM Pipelines, AM Queries
- Fixed a bug where double clicking on folder would create subfolder with same name
***
## Feb 9th 2023
### Platform
- Fixed drag and drop of files into folder which was sometimes failing
- Fixed display of delete confirmation modal when too many objects are in it
- Increased size of description ticket in Support portal
### Analytics Manager
- Added option to compare to previous period in [Metric](https://docs.dataplatform.ovh.net/analytics-manager-queries-charts.md#use-a-metric) chart!
- Improved visual aspect of big number
- Improved drag and drop of tiles to include an auto-scroll at the top and bottom of dashboard
- Fixed filter dropdown that would sometimes go behind a tile
- Fixed "Filter on" option at the bottom of the filter
- Updated text if no deployed dashboard in the dashboards portal
### Data Processing Engine
- Updated PySpark version from 3.1.1 to **3.2.1**
- Fixed bug when committing changes in repo connected to git
- Fixed the display Building status when simply saving action
### Machine Learning Manager
- Made the pipeline ID in a job clickable
### Control Center
- Added icons to show status of execution id's in Log Explorer
- Removed delete icon for ForePaaS monitoring panels (in multi-select)
***
## Feb 2nd 2023
### Platform
- Updated label color and button's grey.
- Fixed table width.
### Analytics Manager
- Added dashboards portal.
- Optimized dashboards render.
- Fixed redirection to logs in the Control Center from a resource.
### Data Manager
- Fixed loader in Table load.
***
## Jan 26th 2023
### Analytics Manager
- Added a new chart type: [Metric](https://docs.dataplatform.ovh.net/analytics-manager-queries-charts.md#use-a-metric), to display a query result as a single big number!
- Greatly improved vertical resizing of dashboard tiles to be pixel perfect instead of by steps
- You can now choose to add a time to your date filter in a dashboard if the filter is set on [_Date & Time_](https://docs.dataplatform.ovh.net/analytics-manager-dashboards-edit.md#edit-advanced-filter-options)
- Increased the limit of tiles per row from 4 to 8 (tiles can now be smaller)
- Switched icon for successful autosave in dashboard to green
- Fixed scroll in query preview if error
- Fixed message when query does not exist in a dashboard
- Fixed labels being hidden in a pie chart in dashboards with no reason
### Control Center
- The Control Center is now in General Availability!
- Infrastructure Monitoring Legacy interface has been decommissioned
- Added possibility to notify subscribers when an alert is resolved
### Data Processing Engine
- Fixed a bug where a workflow would be stuck at a stage (despite all its actions successfully finishing) and would timeout
### Application Services
- Fixed creation of app using custom template
### Platform
- Improvements of the linking of storage engines with Dataplants
- Added a tutorial on [how to upload custom app templates to the ForePaaS Store](https://docs.dataplatform.ovh.net/legacy/api-reference-store.md)
***
## Jan 19th 2023
### Analytics Manager
- You can now filter absolute and relative dates in your dashboards using the **Date** and **Date & Time** filter types!
- Added Boolean filter type
- Added possibility to change filter type in the advanced filter options
- Fixed sql query not executed in dashboards if configuration empty
- Fixed display button back to dashboard if no dashboard
### Data Processing Engine
- Add log for operator elected
- Fixed watch\_jobs ignore not initialised or stopped job
- Fixed skip instead of stop if error happens
***
## Jan 16th 2023
### Platform
- Improved performances for support interface
***
## Jan 12th 2023
### Analytics Manager
- Introducing [Dashboards](https://docs.dataplatform.ovh.net/analytics-manager-dashboards.md)! A modern and simple BI tool for super fast dataviz and reporting. Using the next-gen Analytics Manager query engines, your data visualisation and querying can easily scale up to Big Data and beyond.
- Dashboards are currently in Alpha release in Analytics Manager.
### Identity Access Manager
- Added compatibility for CEL conditions on _Analytics Manager_ service.
***
## Jan 6th 2023
Happy New Year! ForePaaS and OVHcloud wish you the most successful data projects for 2023! 🎉🎉
### Platform
- Performances improvements
***
## Week 52
**Date:** 29/12/2022
### Platform
- Bug fixes and performances improvements
***
## Week 51
**Date:** 22/12/2022
### Platform
- Bug fixes and performances improvements
***
## Week 50
**Date:** 15/12/2022
### Platform
- Bug fixes and performances improvements
***
## Week 49
**Date:** 08/12/2022
### Data Processing Engine
- Added option to override repository version of actions inside workflow
### Analytics Manager
- Fixed chart Runtime Percentile on Overview page
### Control Center
- Removed storage engine overtime chart in storage engine monitoring
- Added OAuth2 support for Custom Webhook alert consumers in the interface
### Identity Access Manager
- Advanced CEL conditions (technical name, etc.) are now available for services _Data Manager_ and _Control Center_.
***
## Week 48
**Date:** 01/12/2022
### Data Manager
- Added a new connector in general availability: **InfluxDB**
### Control Center
- Added OAuth2 support for Custom Webhook alert consumers in the API
***
## Week 47
**Date:** 24/11/2022
Happy Thanksgiving! 🎉
### Data Manager
- Added a new connector in general availability: **CockroachDB**
- Fixed translation when creating a new storage engine
- Fixed responsive on Buckets files list
### Identity Access Manager
- Advanced CEL conditions (technical name, etc.) are now available for service _Data Processing Engine_.
### Control Center
- Fixed URL when redirecting from other services to the Logs Explorer
- Fixed format date of storage engines
***
## Week 46
**Date:** 17/11/2022
### Platform
- Implemented a major backend update to dramatically improve the performance and availability of the ForePaaS console. Among other things, this update significantly increases the stability and speed of access to your Dataplants and other organization-level services.
### Data Processing Engine
- Logs window now prompts to redirect to the Control Center Logs Explorer
### Identity Access Manager
- Advanced CEL conditions (technical name, etc.) are now available for service _Machine Learning Manager_.
### Control Center
- Fixed display of tooltip for RPS and Latency charts in monitoring panels
- Added automatic default values on triggers when creating an alert
- Replaced 0 by 'Terminated' in column 'Age' of service in monitoring panels if status of pod is terminated
### Machine Learning Manager
- Logs window now prompts to redirect to the Control Center Logs Explorer
- Fixed bug on dataset generation for unstructured data
### Data Manager
- Logs window now prompts to redirect to the Control Center Logs Explorer
***
## Week 45
**Date:** 10/11/2022
### Control Center
- Added all query filters in URL of Logs Explorer
- Fixed a bug when logs would not match chosen timeframe after editing the datepicker to a smaller date
### Data Processing Engine
- Added a new tutorial: [how to load streaming data from Apache Kafka](https://docs.dataplatform.ovh.net/tutorials-kafka.md)
### Machine Learning Manager
- Fixed download model link
### Identity Access Manager
- Added compatibility for CEL conditions on _Analytics Manager_ and _Control Center_ services
***
## Week 44
**Date:** 03/11/2022
### Control Center
- You can now search logs using a regex match (as well as doesn't regex match, contains, or doesn't contain) in the Logs Explorer.
- Log search can now be done using Control Center API
- Navigation in the header no longer blocked if user has rights on service.
### Identity Access Manager
- Added compatibility for CEL conditions on _Data Processing Engine_, _Data Manager_ and _Machine Learning Manager_ services
- Updated date format for creation date of api key in users and service account.
### Data Manager
- Fixed opening of advanced panel when update table in Data Manager.
### App Service
- Updated return status when user is unauthorized (403 instead of 401).
- Updated to return 403 if users has no rights on applications instead of 200 and empty list.
### Data Processing Engine
- Fixed concurrent job execution amongst other bugs.
***
## Week 43
**Date:** 27/10/2022
### Control Center
- Added storage engine charts in monitoring.
### Identity Access Manager
- Fixed disabled when edit legacy role condition.
- Updated user name with civility translated.
- Translated civility (in english) in mails.
- Fixed bug about group duplicate in backend.
### Machine Learning Manager
- Fixed autosave when edit preferences of model.
- Fixed getters prim and mart table from store.
### Data Manager
- Added most queried tables chart in overview.
### Data Processing Engine
- Changed settings to not ask to update environment on first launch if version deployed.
### Platform
- Added loader when switch organization.
***
## Week 42
**Date:** 20/10/2022
### Control Center
- Replaced FPU consumption KPI with Dataplant count KPI
### Identity Access Manager
- Bug fixes and improvements to CEL role conditions editor
- Added a new tutorial: [How to set a role condition on a specific bucket](https://docs.dataplatform.ovh.net/tutorials-bucket-role-conditions.md)
### Data Manager
- Fixed reseting search filter when selecting a bucket
### Analytics Manager
- Fixed count of queries in overview
### Data Processing Engine
- Fixed use of databases and tables in Actions
- Added visual display of when job fails because oomkill
- Added tooltip on disabled options for perimeter and segmentation
- Updated trigger translations
### Platform
- Prevented accidentally closing the modal when adding/editing storage engine
- Fixed empty size when creating new storage engine
***
## Week 41
**Date:** 13/10/2022
### Identity Access Manager
- Added compatibility for CEL conditions on _Data Store_ and _App Service_ services
### Data Manager
- Fixed 'magnifying glass' button for a table to open a preview of the table
- Fixed button switch table views
- Added horizontal scrolling on the metadata analysis page
### Data Processing Engine
- Fixed perimeter option not being in the correct place in the UI
- Fixed missing English translation in quotas
- Fixed buttons in the versioning panel
- Fixed database\_name in tables
***
## Week 40
**Date:** 06/10/2022
### Control Center
- Added refresh button next to the date picker
- Fixed alert's metrics page to filter table only on the alert
- Added all authentication modes (Azure AD, etc.) in Total Logins KPI in overview
### Data Processing Engine
- Improved stability and performance of the execution of jobs
- Increased default limit of maximum jobs that can be launched at the same time to 42
- Jobs can no longer be put to the queue, an error will be returned if the maximum limit is reached
- Added new "Automatic segmentation" mode
### Data Manager
- On a Kafka source, added possibility to choose between watching from the earliest or the latest offset for each topic
- Fixed error in chart in Overview page when storage engine is missing
- Moved toggle near to text in wide screen when configuring sources
- Added message if trying to analyze table of source in Analyzer but no tables in source
### Analytics Manager
- Fixed list of attributes in table details
- Removed counting Explorer queries in some charts and KPIs in overview
### Machine Learning Manager
- Fixed display success rate heatmap in metrics notebook and pipeline
- Fixed adding modules in notebook by type of notebooks (PySpark vs Python)
***
## Week 39
**Date:** 03/10/2022
### Control Center
- Fixed organization-level view not available despite being authenticated in IAM
***
## Week 38
**Date:** 22/09/2022
### Identity Access Manager
- Introducing new [CEL role conditions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md)! Use the new visual condition builder to give fine-grained permissions on resources, or write your role conditions using Common Expression Language (CEL)
- Currently, CEL conditions are only available for roles containing permissions on IAM service solely.
### Control Center
- Automatically add author of the alert to the subscribers
- Added alert categories
- Added alert status next to the title in UI
### Data Processing Engine
- Removed segmentation options not available depending on the connector type
- Improved job execution time by improving the build system
### Data Manager
- Added historical values of storage size in Overview
- Improved job execution time by improving the build system
***
## Week 37
**Date:** 15/09/2022
### Control Center
- We revamped the log exploration interface! Now, you can scroll down to progressively load logs as they go
- Added two new alert consumers: Amazon SNS and Opsgenie
- Updated email text content
### Data Processing Engine
- Multiple bug fixes and performance improvements
### Data Manager
- Warning: `/metas` API endpoint is now deprecated. Please make sure to stop using it when calling your ForePaaS Dataplants from outside.
- Multiple bug fixes and performance improvements
***
## Week 36
**Date:** 08/09/2022
### Control Center
- Fixed external consumers unexpected coercion on configuration field
- Fixed slack config (remove not working fields Channel, username, user icon)
### Data Processing Engine
- Stability and performances improvements
- UI changes to environments to force user to click on confirm before saving
### Data Manager
- Stability and performances improvements
### Analytics Manager
- Fixed autosave right after opening a query
### Identity Access Manager
- Several interface bug fixes and improvements
- Prevented access to service interface from header and url if no rights
- Updated title and message in authentication page for Dataplant application
- Added default message in authentication page when creating new application
***
## Week 35
**Date:** 01/09/2022
### Platform
- Updated title in tabs of browser when being on the Overview page of a component
### Control Center
- We added tooltips on each KPI on the overview! Learn how a specific metric was obtained by hovering over it for a couple of seconds.
- Overview: Added ability to filter panel Consumption by providers or dataplants
- Fixed chart consumption in organization level that wouldn't stack
- Fixed jobs heatmap in overview when filtering by timeframe, and to only display only CRON-based
- Updated Y-axis for cpu / ram charts to adjust to the value
- Optimized display of items for DPE and ML dashboards
### Data Manager
- Added shortcut file upload when create new source from overview
- Updated selectbox style in Explorer
### Data Processing Engine
- Disabled create action button on the onboarding welcome page if no rights
### Analytics Manager
- Updated selectbox style in queries
- Added origin explorer in filter of history
- Fixed fields not showing all attributes in visual builder when adding first filter
### Identity Access Manager
- Added button to copy application’s url to clipboard
***
## Week 34
**Date:** 25/08/2022
### Control Center
- The Control Center is now in Beta release!
- Fixed display of days in the date picker
- Metric "Login per hour" is now "Total logins", and metric "RPS" is now "Current RPS"
### Data Processing Engine
- Fixed behavior of [environment variables](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#environment-variables)
### Data Manager
- The "Directly Queryable" feature has been renamed to [Direct Query](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md#make-your-data-source-directly-queryable). Use the Analytics Manager to run queries on remote sources without having to import the data!
- Added endpoint _health_
### Identity Access Manager
- Fixed sessionExpired when calling IAM after login with APIkey/secret key
***
## Week 33
**Date:** 18/08/2022
### Control Center
- Fixed scroll bar in alert editor, monitoring and job’s triggers pages.
- Number of failed execution now includes timed out executions.
- Added full time in tooltip of network charts.
### Data Processing Engine
- Added retry when retrieving DPE action/workflow configuration.
***
## Week 32
**Date:** 11/08/2022
### Platform
- Fixed size tooltip in charts.
### Control Center
- Fixed overview chart storage when organisation view.
- Improved chart consumption in overview.
- Forced display tooltip for charts event if no value.
- Updated tracker front.
- Fixed logout.
- Updated auto generate name for new alert.
- Fixed update FPU for items from old infra monitoring.
### Data Manager
- Fixed delete file when buckets not loaded.
### Machine Learning Manager
- Allowed adding link for notebook dependencies.
### Identity Access Manager
- Added permission DM for DM & QB Editor.
- Added role Control Center Editor.
- Created Infrastructure Managers group.
- Updated name of Identity Access Managers group.
***
## Week 31
**Date:** 04/08/2022
### Control Center
- Prevented second click when create alert + Autogeneration of name.
- Removed info from selectbox to select existing consumer + prevented adding it if required field is not filled.
- Showed preview of instances when toggle HA in storage engine dashboard.
- Added tracking UI.
- Refreshed time when click on refresh button in logs explorer.
- Jobs: Fixed long loader.
- Jobs: Added missing filter "Execution method".
- Jobs: Added more options for filter "Status".
- Jobs: Fixed failed case in graphs.
- Jobs: Added hyperlink for Object name of job.
- Jobs: Fixed wrong icon.
- Jobs: Fixed duplicate objects in table.
- Jobs: Fixed numbers of executions column.
- Jobs: Fixed edit ml trigger.
- Logs: Fixed wrong icon.
- Logs: Sorted the list of object by numbers of executions
- Logs: Fixed toggle streaming button
- Logs: Added show numbers of search result
- Logs: Fixed filter cleared after refreshing logs
- Logs: Sorted and filter execution\_id in the filter options
### Data Manager
- Fixed related items in attributes pages
### Machine Learning Manager
- Update select box for depencies if pyspark notebook.
- Update icon when notebook starting or stopping.
### Data Processing Engine
- Removed stopped deployments from active deployments table.
***
## Week 30
**Date:** 28/07/2022
### Control Center
- Removed not necessary actions in top of treeview in alerts.
- Optimized call in dashboarding default (call was send on all data plant instead of one).
- Added style to redirect on name in table (like in DPE).
- Added name of dataplant in tooltip of charts.
- Displayed first row of charts in one third for panel in overview.
- Updated chart of alerts active over time (fix xAxis and add 0 in green).
- Removed warning when edit fpu from API / APP / AM dashboarding.
- Fixed alignment in input-fpu.
### Data Processing Engine
- Fixed size and add tooltip on tags in treeview.
### Query Builder
- Fixed size and add tooltip on tags in treeview.
### Machine Learning Manager
- Fixed size and add tooltip on tags in treeview.
***
## Week 29
**Date:** 21/07/2022
### Control Center
- Update title in tabs browser
- Add some tooltips
- Update xAxis on charts
- Overview: Update current alert triggerings table
- Overview: Fix xaxis for active alert over time chart
- Monitoring: Fix chart network not reloaded when update datepicker
- Monitoring: Fix redirect to item when click on name from table
- Monitoring: Fix update FPU of item
- Monitoring: Add search by tags in tree view
- Monitoring: Fix UI in storage engine dashboard
- Alerts: Add "Only watch CRON-triggered" feature
- Alerts: Fixed target saving conditionals
- Alerts: Added a Metrics shortcut from the treeview
- Alerts: Fixed alert duplicate
- Alerts: Added alert status in alert title
- Alerts: Health check ability to select all resources
- Alerts: Trigger - Default value by default
- Alerts: Subscribers pagination fix
- Alerts: Ability to search for subscribers from the added list
- Settings: Ability to edit an external consumer
### Analytics Manager
- Fixed scroll in query fields when vertical view
### Data Manager
- Fixed scroll in query fields when vertical view
***
## Week 28
**Date:** 13/07/2022
### Control Center
- The [Control Center](https://docs.dataplatform.ovh.net/landing-page-control-center.md) is now available in Alpha release! Get full observability on each service and component in your data project.
- Monitor metrics across your data pipelines, applications, and deployed instances
- Drill-down on logs to troubleshoot errors
- Set up alerts when issues arise
- Ensure scalability by adding more resources on any service as needed
### Identity Access Manager
- Bug fixes and improvements to session management
***
## Week 27
**Date:** 07/07/2022
### Platform
- Updated CSV file connector.
### Data Processing Engine
- Enabled concurrent execution button for serverless environments.
***
## Week 26
**Date:** 30/06/2022
### Platform
- Support of multi factor authentication (MFA).
### Data Manager
- Connectors (Druid, OpenSearch, Prometheus)
- New Explorer tab released in Alpha.
- Optimized query execution to be able to use virtual attributes in SQL mode
### Data Processing Engine
- Point-and-click PySpark Load and Aggregate actions are now available in Alpha release
- Optimized provisioning time with ForePaaS actions.
- Multiple concurrent executions in Serverless mode.
***
## Week 25
**Date:** 23/06/2022
### Platform
- Fixed version api on serviceaccounts that can be on error while sent to kubernetes.
### Data Processing Engine
- Disabled concurrent execution button for serverless environment.
- Uniformized Information panel when editing an element.
***
## Week 24
**Date:** 16/06/2022
### Platform
- Updated Japanese translations.
### Machine Learning Manager
- Update open Notebook CTA.
***
## Week 23
**Date:** 09/06/2022
### Platform
- Bug fixing and improvements.
***
## Week 22
**Date:** 02/06/2022
### Platform
- Fixed Japanese some translations.
- Updated list of restricted words for tables/attributes names.
- Bug fixing and improvements.
***
## Week 21
**Date:** 26/05/2022
### Platform
- Bug fixing and improvements.
***
## Week 20
**Date:** 19/05/2022
### Platform
- Removed big shadow on white buttons.
- Improved tooltip display.
### Data Processing Engine
- Fixed folder on some treeviews.
- Added advised bucket size.
### Data Manager
- API: created a v4/metrics endpoint to aggregate metrics from DM, storage volume.
- Hotfixed metrics endpoint.
- API metrics extract: added possibility to update table parameters depending of connector func
***
## Week 19
**Date:** 12/05/2022
### Data Processing Engine
- Actions treeview bug fixed.
- Actions folder treeview bug fixed.
***
## Week 18
**Date:** 05/05/2022
### App Manager
- Fixed copy and remove with new Minio gateway.
### Identity Access Manager
- Fixed populate of undefined when getRolesAcls for users.
- Fixed generate password too long.
- Update format date generated password
### Data Processing Engine
- Introducing [Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#execution-modes)! 🎉 Execute your jobs faster with the new Always-up execution mode in Data Processing Engine
- In addition to the standard _Serverless_ execution mode, now you can also keep dedicated resources always up and ready to run your jobs instantly.
- Added placeholder for empty table in metrics.
### Machine Learning Manager
- Added placeholder for empty table in metrics.
### Analytics Manager
_This component is currently released under open beta._
- Added limit in query configuration.
- Put select be default when select data.
- Added a shortcut to preview a table directly from a table.
***
## Week 17
**Date:** 29/04/2022
### Platform
- Added link to have info on OVH acquisition in no new account page.
- Added modal confirm when delete member from editor.
- Added status of job in job detail.
- Fixed condition to display provider logo in dataplant block.
- Added link to documentation on how to manage members in request to join organisation email.
### Data Manager
- Added column in sources.
- Added tooltip for generate action in table list.
### Data Processing Engine and Machine Learning Manager
- Fixed message in FPU settings for environment.
***
## Week 16
**Date:** 22/04/2022
### Data Manager
- Removed virtual attributes from Tables page
- Added info on system bucket
- Fixed bug to attach sandbox in DM + can unlink sandbox
### Analytics Manager
_This component is currently released under open beta._
- Display more attributes in selectbox data and scale
- Updated function to set sum or select by default
- Added placeholder in filters
- Hiden ML tables in table details in query editor
- Added shift + Enter to run query And wait after saving to run it
- Added loader when run from SQL editor from keyboard
- Added download as a json
- Fixed size of SQL in history table
- Fixed switch in filter from between to in
- Added created\_at and updated\_at column in queries list
### Data Processing Engine
- Ability to view workflow from action lineage page
- Bug fixed: workflows list of actions was wrong when using repo versioning
### Identity Access Manager
- Updated icon in active users table.
- Formated the result of cron with UTC
### Platform
- Added option to resize columns in many tables
- Fixed active lang in header.
***
## Week 15
**Date:** 14/04/2022
### Data Manager
- Fixed edition of attribute name after creating it.
### Analytics Manager
_This component is currently released under open beta._
- You can now run query by pressing cmd/ctrl + Enter in SQL mode.
### Data Processing Engine
- Fixed display error when start action/workflow.
### Machine Learning Manager
- You can now export your notebooks as DPE Custom Actions, MLM Estimators or MLM Scoring Functions.
### Identity Access Manager
- Fixed generate pwd when the username is empty.
### Platform
- New header.
***
## Week 14
**Date:** 07/04/2022
### Data Manager
- You can now activate _Directly Queryable_ mode for supported sources: run SQL queries from the Analytics Manager on those sources without having to load their data into ForePaaS.
### Analytics Manager
_This component is currently released under open beta._
- Added free-trial paywall for resources.
### Data Processing Engine
- Fixed display error when start action/workflow.
- Added max length to filenames.
### Platform
- Added in trial paywall when trying to update FPU in monitoring when free-trial mode.
***
## Week 13
**Date:** 29/03/2022 and 31/03/2022
### Platform
- Updated the home page of Dataplants: the Data Processing Engine is now located below the Data Manager instead of being to its left.
- Added new cookies management solution
### Data Manager
- Added a shortcut to create or upload a source.
- Updated style of storage engine status.
- Fixed long naming when table is created by drag and drop.
- Improved handling of small error from sentry report.
### Data Processing Engine
- Added a shortcut to create or upload a custom action.
- Updated the maximum value of the FPU slider for actions and workflows.
### Machine Learning Manager
- Updated the maximum value of the FPU slider for pipeline steps.
### Identity Access Manager
- Fixed the positions of the permissions modal filter
***
## Week 12
**Date:** 25/03/2022
### Platform
- FP Store
- Added Japanese translations
- Fixed search
### Analytics Manager
_This component is currently available under open beta release._
- Fixed reactivity when activating Trino.
- Added API endpoints for exporting and importing AM configuration.
- Bug about updating chart configuration in Query fixed.
### Data Manager
- Fixed reactivity when changing default storage engine.
- Updated Clickhouse connector.
- Renamed MemSQL to Singlestore.
### App Manager
- Removed Owned and Not owned from categories
---
url: https://docs.dataplatform.ovh.net/landing-page-data-platform.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Platform
## What is Data Platform?
Data Platform is a managed platform for building data products, with the infrastructure decisions already made: there is no cluster to size, no orchestrator to run and no deployment pipeline to build. Connecting sources, storing and modelling data, processing it, analyzing it and exposing the result as an API or an application are all provisioned and operated by the platform, behind four levels of automation, so effort goes into business logic rather than logistics.
A [project](https://docs.dataplatform.ovh.net/manage-projects.md) is the unit of work and the unit of isolation. It holds the sources, tables, buckets, pipelines, queries, dashboards, APIs and apps that make up one data system, together with the access rules over them. Creating a project deploys more than 60 microservices and provisions machines in the selected cloud region in under eight minutes, with no further input.
:::info
Access to a project is usually shared collaboratively along role lines: one group works on ingestion and transformation, another on analytics, another on the application layer. The [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md) decides who can read and change what, down to individual table attributes.
:::
If Data Platform is new to you, the fastest way in is the six-step guide that builds a working application on sample data. From there, the [services](https://docs.dataplatform.ovh.net/landing-page-services.md) documentation covers each component in the order data moves through it.
[### Getting started→
Build a first end-to-end application in six steps.
](/landing-page-getting-started)[### Tutorials→
Complete worked use cases, from Kafka to Superset.
](/landing-page-tutorials)[### Services→
The seven services a project is made of, in data order.
](/landing-page-services)[### Developers→
Public APIs and the SDKs behind them.
](/landing-page-developers)[### Manage your platform→
Organizations, projects, pricing and finding your way around.
](/landing-page-manage)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/release-notes.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Release notes
## October 19, 2026
**General**
- **New guide:** [Analyse online store sales with Apache Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md), from Shopify and Stripe orders to a sales dashboard.
- **[User creation](https://docs.dataplatform.ovh.net/iam-users.md#application-end-users):** new users receive an email with a single-use link to set their password.
- **Password reset:** a single-use reset or activation link is sent by email after you confirm, and it returns you to the application you came from.
- A link that is no longer valid now says so.
- Changing a password signs out every other session.
- **Organizations:** an organization that still contains active projects can no longer be deleted.
- **Fixes:**
- the code editor shows the file you switch to, instead of the previous file of the same type.
**Lakehouse Manager**
- **[Buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md):** new buckets are encrypted at creation (SSE-S3).
**Data Processing Engine**
- **[Spark resources](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md#pyspark-actions):** set the driver size independently of the executors, see the total DPU of the execution, and choose the library and Spark [log levels](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#logs-level).
- **[Always-up execution mode](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#execution-modes):** now labelled beta for PySpark actions.
- **Job status:** an action or workflow that reaches its own time limit is reported as `TIMEOUT` instead of a generic failure, and a start refused because the organization DPU quota is exceeded now says so.
- **Build failures:** the reason now appears in the execution log.
- **Fixes:**
- running jobs are no longer interrupted by cluster scale-down.
- downloading the files of a segmented job works again.
**Control Center**
- **[Audit trail](https://docs.dataplatform.ovh.net/control-center-auditing.md#export) export:** download the audit and security events of an organization or project as a file, with clearer errors when an export fails.
- **[Security events](https://docs.dataplatform.ovh.net/control-center-auditing.md#security-events):** a new tab on the auditing pages lists security events, with a detail panel for each one.
- **[Resources consumption chart](https://docs.dataplatform.ovh.net/landing-page-control-center.md):** usage is read one day at a time, so long date ranges no longer miss data.
- **Application DPU:** each application's DPU in the resources panel links to its configuration page.
- **Fixes:**
- the audit trail filter works again.
**Apps and API**
- **App Service pages:** the overview and the other App Service pages have an improved layout and experience.
## October 5, 2026
**Lakehouse Manager**
- **Fixes:**
- updating a table that carries table-level policy tags no longer fails.
- copying a partitioned table works again.
- dropping or renaming an attribute used by a current or previous partition key is now refused.
**Data Processing Engine**
- **Job logs:** execution logs now show only what concerns your job.
- Platform internals and duplicate errors are filtered out.
- A failed action reports its error once.
- **[Always-up deployments](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#always-up):** PySpark deployments can be kept always up again.
- **Fixes:**
- stopping and redeploying a deployment is more reliable.
- opening a notebook whose server is already running no longer reports a start failure.
- the Python 3.13 Data Science notebook image keeps numpy 2.
**Apps and API**
- **Fixes:**
- a build whose builder image cannot be pulled now fails.
- a build stopped for exceeding its deadline now reports `FAILURE` instead of `ABORTED`.
- build duration is now always bounded, even when the deadline is misconfigured.
## September 22, 2026
**General**
- **Fixes:**
- the cookie preferences control now stays at the bottom of the page.
- project tiles now show a rounded DPU total instead of a long decimal value.
- the message shown when a new quota value is below current usage now explains the problem clearly.
- sign-in and permission checks are more stable and faster under load.
**Data Processing Engine**
- **Logs from your own packages:** the logs emitted by the Python packages you install are no longer dropped in Spark jobs, notebooks and Python jobs alike.
- Warnings and errors from your packages now reach the job logs, while the platform's own libraries stay quiet.
- For more detail, set `lib_log_level` to `info` or `debug` in the action's parameters, in advanced mode.
- **Faster environments:** the Python packages of an environment now install in parallel, so it is ready sooner.
- **Fixes:**
- loading data into a partitioned table with a [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) no longer fails.
## September 14, 2026
**General**
- **Cookie consent:** cookie preferences now use the standard OVHcloud consent manager, reachable at any time from the cookie icon in the header.
- **Fixes:**
- permission checks for project creation are corrected.
- a clearer error page is shown when your session cannot be verified.
**Lakehouse Manager**
- **[Row filters](https://docs.dataplatform.ovh.net/lakehouse-manager-row-filters.md):** restrict which rows of a table a user or group can see, without duplicating the table. The table keeps one copy of its data, and each person queries it through their own filters.
- Write the condition in the default builder mode (column, operator, value, combined with _Match ALL (AND)_ or _Match ANY (OR)_), or switch to SQL mode for conditions the builder cannot express.
- The impact panel reports how many rows the filter would keep and how many it would hide, before anyone is assigned to it.
- Filters are enforced on Trino queries, so they cover Analytics Manager queries and dashboards and Trino consumers. A notebook reading through PySpark or PyIceberg, and direct storage access, are not filtered.
- A filter can be saved inactive, switched on and off from the list, duplicated onto another table, and travels with a project configuration export.
- **Fixes:**
- deleting a project now also removes the dataset shares it granted, so a consumer project no longer lists a shared dataset whose source project is gone.
**Data Processing Engine**
- **Spark job memory:** Spark jobs now receive their full allotted memory, overhead included, which reduces out-of-memory failures.
- **[Custom action](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) logs:** `print()` and standard error output from action code now show up in the job logs, and a missing action produces a clearer error message.
- **[Spark UI and History](https://docs.dataplatform.ovh.net/dpe-spark-history-server.md):** Spark applications now carry the job's name instead of a generic one.
- **Fixes:**
- stopping a job now terminates its Spark application.
- a job whose pod crashes is detected and reported instead of hanging.
- running jobs are no longer archived out of the job history.
- jobs that finish while you watch them now appear correctly in executed jobs, and pipeline stages display reliably.
- Iceberg table procedures such as `expire_snapshots` and `remove_orphan_files` work again in PySpark jobs.
**Analytics Manager**
- **[SQL editor](https://docs.dataplatform.ovh.net/analytics-manager-queries-sql-editor.md) autocomplete:** the editor now suggests SQL keywords, ready-made snippets, and every available Trino function with its description.
- **Fixes:**
- the runtime shown in the [query history](https://docs.dataplatform.ovh.net/analytics-manager-history.md) is now formatted correctly.
**Control Center**
- **[Audit trail](https://docs.dataplatform.ovh.net/control-center-auditing.md) search:** a new full-text search finds an event quickly, and the layout is clearer.
**Apps and API**
- **App resources:** each DPU allocated to an app now gives it 1 vCPU and 4 GB of RAM, matching the [DPU definition](https://docs.dataplatform.ovh.net/manage-pricing.md#billing-units). Apps previously received half of that. The change applies on the app's next restart.
- **Fixes:**
- a build that gets stuck now reports why, with a status message and an event log.
- two builds of the same app can no longer run at the same time.
- an application again has exactly one active version.
## August 31, 2026
**Lakehouse Manager**
- **[Views](https://docs.dataplatform.ovh.net/lakehouse-manager-views.md):** save a SQL `SELECT` query as a named virtual table inside a dataset and query it like a table, with results always computed live from the source tables.
- Columns are derived automatically from the SQL, which is validated on every create and edit with human-readable errors.
- Views track their own **Valid** or **Broken** status: they are re-checked automatically when a source table changes and recover on their own once the source is restored.
- Policy tags, dataset permissions, and the audit trail apply to views just like to tables, and views created by external tools are discovered automatically.
- **[Filtered tabs](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#filtered-tabs):** filtered views in the Tables section are renamed to filtered tabs, to avoid confusion with the new views.
**Data Processing Engine**
- **[Deduplicate action](https://docs.dataplatform.ovh.net/dpe-actions-deduplicate.md):** new ready-to-use action to remove duplicate rows from a lakehouse or PostgreSQL table, keeping exactly one row per deduplication key.
- The key is pre-filled from the table's identifier fields, a read-only scan previews how many rows a run would remove, and you choose which row survives (any, most recent, or oldest).
- An optional backup table protects the run; on lakehouse tables a run can also be undone with Time Travel.
- **Parallel job runs:** launching the same job several times in parallel is now safe: each run keeps its own data and results.
- **[Spark History](https://docs.dataplatform.ovh.net/dpe-spark-history-server.md):** the Spark application history is now reachable from older jobs too, not only the most recent ones.
- **Fixes:**
- when a job fails because a secret cannot be read, the actual cause is now shown instead of an opaque error.
- the Rebuild action confirmation now shows the name of the table being rebuilt.
## August 24, 2026
**Lakehouse Manager**
- **[Lineage](https://docs.dataplatform.ovh.net/lakehouse-manager-lineage.md):** a new built-in lineage view maps how data flows through your project, from the pipelines that feed each table down to the tables built from it.
- Open it from the new **Lineage** section or straight from a table's details page, and drill down to column-level lineage.
- Lineage now covers PySpark too: automatic for Load and Aggregate PySpark actions, opt-in through the [SDK](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md) for Custom PySpark actions and PySpark notebooks.
- **Fixes:**
- checkboxes in the Tables list view are now correctly aligned.
- sharing a dataset with a consumer project that does not exist now returns a clear error instead of a storage engine failure.
**Data Processing Engine**
- **[Notebook export as a custom action](https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md#export-a-notebook-as-a-custom-action):** the Data Platform extension can now turn a notebook into a custom action directly from JupyterLab, with the notebook's pip installs automatically converted into action dependencies.
- **Fixes:**
- environment variables already defined on a job are no longer overridden when the job starts.
**Control Center**
- **Fixes:**
- job metric charts now display over any time range: the measurement precision adapts to the selected window instead of failing on wide ranges.
## August 3, 2026
**General**
- **New guide:** [Trino data types and casts](https://docs.dataplatform.ovh.net/tutorials-trino-types-and-casts.md), covering the mapping between Lakehouse types and Trino types and how to fix `TYPE_MISMATCH` errors when writing through Trino.
- **Expanded docs:** deeper Lakehouse Manager pages on data types, naming rules, the storage model, and cross-project sharing, plus new Data Processing Engine docs on code sharing between actions, triggering notebooks from workflows, and CI/CD.
**Lakehouse Manager**
- **Fixes:**
- sharing a dataset in read-only no longer downgrades the owner's own read-write access to that dataset.
- the API now returns a 404 error instead of a 500 when a resource is missing or was deleted.
**Data Processing Engine**
- **[Data Platform Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-data-platform-extension.md):** a new JupyterLab extension adds a side panel to browse your project's datasets and storage buckets, run quick read-only SQL queries, and start from a gallery of ready-to-use notebooks wired to your own data.
- **marimo extension:** the marimo notebook extension is now preinstalled in all notebook images.
- Both extensions are new: try them out and share your feedback.
- **[Custom action helpers](https://docs.dataplatform.ovh.net/dpe-actions-custom.md#use-the-helper-panel):** the helper panel in the Custom action and [Custom PySpark](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md#use-the-helper-panel) editors now offers ready-to-use scenarios organized by category, SDK guides, a tab to browse your data, and a FAQ.
**Control Center**
- **[Audit trail](https://docs.dataplatform.ovh.net/control-center-auditing.md):** new event types record configuration export and import operations.
- **Fixes:**
- job logs in the [Logs Explorer](https://docs.dataplatform.ovh.net/control-center-logs-explorer.md) now display in the correct order without gaps or duplicated lines, streaming resumes after an interruption, and a loading error is shown instead of an empty log view.
- the **Open logs** button now opens the Logs Explorer with the correct parameters instead of doing nothing.
**Apps and API**
- **Application scheduling (beta):** automatically start and stop an application (apps, APIs, Superset) during weekly activity windows, to avoid paying for idle time.
- Pick a timezone and add one or more windows with days and start and stop times; outside these windows the application is stopped.
- Transitions are evaluated about every 5 minutes, so a start or stop can be delayed by up to 5 minutes.
- **In-memory caching (beta):** deploy a Valkey or Redis cache alongside an application service.
- Choose the cache engine version and the share of DPU allocated to it (10 to 50%); the change takes effect after restarting the deployed version.
- **SQLite backups (beta):** applications that store data in SQLite can now enable automatic backups.
- Set the daily retention in days and how many weekly and monthly backups to keep, and open stored backups directly in the Lakehouse.
## July 20, 2026
**Lakehouse Manager**
- **[Iceberg bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md#iceberg-bucket):** the Buckets screen now lists your project's Iceberg bucket, the storage behind your Lakehouse tables, in read-only mode.
**Data Processing Engine**
- **[Overview](https://docs.dataplatform.ovh.net/landing-page-dpe.md) heatmap:** job activity is now grouped by day and 4-hour slot, with run counts shown per cell and only finished jobs counted.
- **Build logs:** logs from the build phase of a job are now available, so errors during a build are visible and easier to debug.
- **Notebook import and export:** notebooks are now imported and exported as ZIP archives.
- **[Workflows](https://docs.dataplatform.ovh.net/dpe-workflows.md):** a workflow must now contain at least one action to be started; empty workflows can no longer be run.
## July 6, 2026
**General**
- **Configuration export and import:** each service module can now be imported directly from the UI, without calling an endpoint to import a configuration.
**Lakehouse Manager**
- **Fixes:**
- **Explore snapshot** from Time Travel now redirects to the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) with the snapshot loaded.
- searching a dataset by its full name (for example `default_dataset`) now returns the dataset instead of "You have no dataset".
- deleting a shared dataset from a receiving project no longer leaves the Explorer without a dataset name.
**Data Processing Engine**
- **[Overview](https://docs.dataplatform.ovh.net/landing-page-dpe.md) metrics:** new **Resource consumption**, **Top consumers**, and **Spark History Server** metrics, with a selectable last 7, 30, or 90 day range and faster loading.
- **Fixes:** the [Rebuild action](https://docs.dataplatform.ovh.net/dpe-actions-rebuild.md) now mirrors an attribute's state when it is marked **Required + Identifier**.
**Control Center**
- **[Overview](https://docs.dataplatform.ovh.net/landing-page-control-center.md) metrics:** new **Total resource consumption**, **Data scanned over time**, **DB storage (tables)**, and **Object storage (buckets)** metrics, shown both as current values and over time.
- Metrics on the Overview and [Jobs Central](https://docs.dataplatform.ovh.net/control-center-jobs-central.md) now load faster.
- **Fixes:**
- the [Alerts](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md) page now shows an error notification when alerts fail to load, instead of an empty "No alerts" state.
- the [Audit trail](https://docs.dataplatform.ovh.net/control-center-auditing.md) now loads further entries when paging past the cached results, at both project and organization level.
## June 22, 2026
**General**
- **App Service component count:** the organization home now shows the number of deployed App Service components on each project box.
- **New tutorial:** [Build an IoT fleet-monitoring pipeline](https://docs.dataplatform.ovh.net/tutorials-iot-fleet-monitoring.md), an end-to-end guide streaming simulated sensor data from a Custom action through Kafka into a live Superset dashboard.
**Connectors**
- **Database connector pagination:** the database connector list is now paginated, making large lists easier to browse.
- **Fixes:** sorting by column header now works on the Trino consumers table.
**Lakehouse Manager**
- **[Required and Identifier fields](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints):** attributes can now be marked **Required** (never NULL) or **Required + Identifier** (part of the row's uniqueness key for upserts).
- Required fields cannot be added once a table is built; use the new [Rebuild action](https://docs.dataplatform.ovh.net/dpe-actions-rebuild.md) to add them. Identifier fields cannot be `double` or `decimal`.
- **Table metrics:** tables now have a **Metrics** section surfacing key statistics about the table.
- **Fixes:** in the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md), the **Export as CSV** button is disabled with an explanatory message when no query engine is active.
**Data Processing Engine**
- **[Write strategy](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md#write-strategy):** Load and Aggregate actions (Python and PySpark) can now upsert using a table's identifier fields.
- Choose **Upsert**, **Insert when not matched**, or **Insert all** in the action preferences.
- Actions default to **Insert all**; select **Upsert** manually to enable matching behavior.
- **[Rebuild action](https://docs.dataplatform.ovh.net/dpe-actions-rebuild.md):** rebuild a table from scratch to change field constraints, restructure partitioned data, and clean its metadata.
- Set a default value when promoting a field to Required/Identifier; optionally keep a backup of the previous table.
- Removes all table history; Time Travel is no longer available after a rebuild.
- **[SQL action](https://docs.dataplatform.ovh.net/dpe-actions-sql.md):** write and run a SQL query against your datasets and tables on the Trino engine, as a schedulable, workflow-ready action.
- Supports reads, metadata, DML, DDL, and `CREATE TABLE AS SELECT`, which also builds the new table in the Lakehouse Manager.
- Validate before creating, with destructive-query confirmation and a re-validation check on every run.
- **[Spark History Server](https://docs.dataplatform.ovh.net/dpe-spark-history-server.md):** inspect the execution history of your PySpark actions and workflows from a new DPE page.
- Start the server on demand to browse recent runs; it auto-stops when idle and is billed per DPU only while running.
- Open each run's **Spark UI**, or analyze it with **DataFlint** performance insights, live or after completion.
**Analytics Manager**
- **Fixes:** in [Queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md), the **Export as CSV** button is disabled with an explanatory message when no query engine is active.
**Control Center**
- **[Logs Explorer](https://docs.dataplatform.ovh.net/control-center-logs-explorer.md) performance:** the Logs Explorer now issues a single metrics and jobs request instead of one per project, speeding up loading.
- **[Alerting](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md):** the **Create alert** action is available again from the project homepage.
- **Fixes:**
- logs no longer display "Invalid date".
- refreshing the [Auditing](https://docs.dataplatform.ovh.net/control-center-auditing.md) trail no longer duplicates rows.
**Apps and API**
- **App Service UI (beta):** a redesigned, more unified interface for the App Service is now available as a beta, bringing it in line with the rest of the platform.
- We will ship improvements and updates continuously over the coming weeks and months. Your feedback is welcome.
- **Fixes:** creating an environment variable no longer fails with a 400 error.
## June 2, 2026
**Connectors**
- **[Trino consumers](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md):** create a Trino connection string in a few clicks from the **Consumers** tab, instead of setting up a service account, API keys, and the connection string by hand.
- Pick a **READ** or **READ & WRITE** access level per consumer.
- The connection string appears only once, at creation; each consumer has a dedicated service account.
## May 25, 2026
**General**
- **Contact Support:** the **Contact Support** link now opens an in-platform support form instead of redirecting to the external OVHcloud Help Centre.
**Lakehouse Manager**
- **[Cross-project dataset sharing](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md#cross-project-sharing-beta) (beta):** datasets can now be shared read-only with other projects in the same organization.
- Admins share from the source project via the dataset's **Configuration** tab → **Manage share settings** → pick one or more target projects.
- Receiving projects see the dataset in their Lakehouse Manager **Datasets** menu with a shared icon and the type **Lakehouse Manager (shared)**. Tables can be previewed but not modified.
- In the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md), shared tables appear under a new **Shared Datasets** category in the _list by dataset_ view. SQL editor only.
**Data Processing Engine**
- **Fixes:** importing a DPE configuration into another project now correctly rewrites repository references in workflows. Previously, workflows imported into the destination project still pointed to the source project's `repository_id` for the **Default** repository, breaking action references.
**Control Center**
- **[Auditing](https://docs.dataplatform.ovh.net/control-center-auditing.md):** notebooks are now covered by the audit trail.
## May 5, 2026
**Connectors**
- **New connectors:**
- [HubSpot](https://docs.dataplatform.ovh.net/connectors-sources-hubspot.md): extract contacts, companies, deals, tickets, pipelines, forms, marketing emails, and more.
- [Odoo](https://docs.dataplatform.ovh.net/connectors-sources-odoo.md): extract contacts, CRM leads, sales orders, invoices, products, inventory, HR, and projects from your Odoo ERP instance. Compatible with Odoo 14–19.
- [Salesforce](https://docs.dataplatform.ovh.net/connectors-sources-salesforce.md): extract standard and custom sObjects, run SOQL and SOSL queries, list reports, and inspect org limits.
- [Shopify](https://docs.dataplatform.ovh.net/connectors-sources-shopify.md): extract products, customers, orders, inventory, marketing, B2B, payments, and content via the Shopify GraphQL Admin API.
- [Stripe](https://docs.dataplatform.ovh.net/connectors-sources-stripe.md): extract customers, charges, payment intents, subscriptions, invoices, balance transactions, and events.
- Each new connector ships with a companion **Technical Reference** page covering authentication, the full endpoint catalogue, pagination, rate limits, output format, and limitations.
**Data Processing Engine**
- **Fixes:**
- selecting resource allocation when creating an action or workflow no longer freezes the screen.
- updating the environment on an existing action no longer freezes the screen.
**Control Center**
- **Fixes:** the [Auditing](https://docs.dataplatform.ovh.net/control-center-auditing.md) refresh icon now preserves the selected time range instead of resetting to the last 6 hours.
## April 20, 2026
**General**
- **Maintenance banner:** a banner is now shown in the [header](https://docs.dataplatform.ovh.net/manage-navigation.md#maintenance-banner) for the duration of any scheduled platform maintenance. While active, the platform is read-only; run operations remain allowed.
**Connectors**
- **OpenLineage consumer:**
- auth token is now optional.
- new **Disable Trino events** toggle - skip Trino-generated lineage (Explorer, Analytics Manager) and forward only DPE batch lineage.
- **Fixes:** schemas no longer displayed incorrectly in Marquez for forwarded lineage events.
**Lakehouse Manager**
- **Explorer:**
- **Export to CSV:** scale fields now come before data fields in visual-query CSV exports (download and bucket export).
- all non-numeric values are now quoted in CSV exports, which avoids parsing issues with commas, quotes and line breaks.
- **Fixes:** **Preview table** now opens the Explorer with the selected table loaded.
**Data Processing Engine**
- **Git dependency auto-update:** [Custom](https://docs.dataplatform.ovh.net/dpe-actions-custom.md) and [Custom PySpark](https://docs.dataplatform.ovh.net/dpe-actions-custom-pyspark.md) actions now support `@latest` in Git-hosted Python dependencies:
- Append `@latest` to any `git+` dependency URL to automatically resolve the most recent published release at build time.
- Click **Force Build** to pick up a new version. No more manual tag or commit hash updates across actions.
- Note: the latest release is not detected automatically at runtime, a manual **Force Build** is required to pull the update.
- **Custom action lineage SDK:** new `lineage_run` context manager, auto-emits `START` / `COMPLETE` / `FAIL` and generates `run_id`.
- **Custom action lineage helpers:** new `schema_facet` (declare schemas) and `column_lineage_facet` (column-to-column mappings).
- **Auto schema detection:** pass `connector=` to `lineage_run` to extract schemas automatically.
- Updated docs: [Track data lineage in a Custom action](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md).
**Analytics Manager**
- **Queries:**
- **Export to CSV:** scale fields now come before data fields in visual-query CSV exports (download and bucket export).
- all non-numeric values are now quoted in CSV exports, which avoids parsing issues with commas, quotes and line breaks.
**Control Center**
- **New: [Auditing](https://docs.dataplatform.ovh.net/control-center-auditing.md) (initial release):** logs significant create / update / delete activity on the platform. Coverage will expand in upcoming releases.
- Project-level view in Control Center; filter by service (Connectors, Lakehouse Manager, DPE, Analytics Manager, IAM), resource and time window.
- Organization-level view in Organization Settings. Currently covers organization IAM and member management; billing, quotas and similar are coming soon.
- Two new roles at each level: **Audit Trail Admin** and **Audit Trail Viewer**.
- **Alerting (beta):** [alerting](https://docs.dataplatform.ovh.net/landing-page-control-center-alerting.md) has been reworked and re-enabled. Please report any issues you run into.
- **Fixes:** Logs Explorer now displays execution logs for DPE actions and workflows correctly.
## April 7, 2026
**Lakehouse Manager**
- **OpenLineage compliance:** Data Platform now tracks and exposes data lineage events using the [OpenLineage](https://openlineage.io/) standard. Lineage events are recorded automatically for Load and Aggregate actions, and opt-in for Custom actions.
- **Fixes:** Query Gateway now correctly auto-infers `catalog.schema` for Direct Query catalogs. This only affected tables with Direct Query enabled.
**Connectors**
- **New Consumer:** Added the [OpenLineage consumer](https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md), connect your own OpenLineage-compatible lineage solution (e.g. Marquez) by providing an endpoint URL and auth token.
**Data Processing Engine**
- **New Action:** Added [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md), forward data lineage events from Data Platform to your external lineage backend. Supports cron scheduling or always-up continuous mode.
- **SDK:** Added documentation for [tracking data lineage in Custom actions](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md) using the `emit_lineage` function. Lineage is opt-in for Custom actions, declare inputs/outputs and emit `START`, `COMPLETE`, and `FAIL` events to integrate with the platform's OpenLineage graph.
- **Fixes:**
- Cron jobs are now active by default upon creation.
- Execution logs for newly created actions and workflows are now displayed correctly.
## March 23, 2026
**Data Processing Engine**
- Improved **Spark** job launch and workflow stability.
- **Actions & workflows:** Added option to deactivate and reactivate **cron** jobs instead of deleting them.
- **Fixes:** Environment variables field no longer overflows the layout for long values.
**Lakehouse Manager**
- **Attributes:** Attribute naming rules are now displayed.
- **Explorer:**
- Fixed export of **CSV** results to bucket for visual queries.
- Fixed incorrect alert icon for direct-query sources after metadata extraction.
## March 9, 2026
**Connectors**
- **New Connector:** Added support for the **FHIR** connector.
- **Updates:**
- Updated **ClickHouse** connector.
- Fixed SSL issues in the **MySQL** connector.
- **Kafka** connector certificates are now encrypted at creation time.
- Updated: **Kafka** nested JSON fields extraction; metadata extraction may need to be re-run if your Kafka source contains nested fields.
**Data Processing Engine**
- Improved `SparkSession` acquisition for Custom PySpark Actions.
- **⚠️ PySpark 3.3.2 Deprecation:** Creating new actions with **PySpark 3.3.2** is no longer possible. This version is being deprecated, please migrate to a newer PySpark version.
**Lakehouse Manager**
- **Trino:** Updated Trino version to **479**.
- **Direct Query in Explorer:** Added direct query support in the Lakehouse Explorer. If your source is direct-query enabled, its tables and extracted attributes are now visible directly in the Explorer.
- **Fixes:** Fixed encryption of nested parameters (particularly for Kafka databases).
**Analytics Manager**
- **Fixes:**
- Resolved an issue with the `dashboard/deployed` endpoint.
- Ensured resource groups are created during seed even when no JSON body is provided.
**Control Center**
- **Fixes:** Fixed `endDate` handling in the Control Center Logs Explorer.
## February 23, 2026
**General**
- **Fixes:**
- Environment selected in workflow preferences now properly retained.
- Clicking metrics icon now redirects directly to the metrics page.
- Deployment `start_date` now properly taken into account in Control Center Logs Explorer.
- IAM auth provider `active` button now correctly reflects state.
**Lakehouse Manager**
- **Updates:** Updated Trino version from **476** to **478**.
- **Performance:** Improved catalog reload time by approximately 2-3×.
- **Fixes:** Fixed 401 errors on queries immediately after project creation.
## February 9, 2026
**General**
- **Fixes:**
- Fixed infinite loading spinner issue in Consumers section.
- Enhanced platform stability with backend improvements and dependency upgrades.
**Connectors**
- **API Connectors:** Improvements and fixes for API connectors to enhance reliability and user experience.
**APPs & APIs**
- **Superset Theming:** Introduced a new theming module for Superset, allowing for better customization and branding options.
- **Updates:** Updated Superset Template to version 6.
**Lakehouse Manager**
- **Performance:** Minor improvements to query stability and monitoring capabilities.
## January 26, 2026
**General**
- **UI Refresh:** Rename **Data Catalog** to **Connectors** across the entire platform for improved clarity and consistency.
- Update all navigation menus, documentation links, and sidebar references to reflect this change.
- The functionality remains the same - this is purely a naming update to better represent the service's purpose as your central repository for data source connections.
- **Fixes:**
- Fix issue where Current Project usage had multiple supersets being displayed. Now displays just one superset column with total consumption, similar to how it's shown for apps and APIs.
- Fix issue where users were unable to grant write access on policy tags.
- Added a visual indicator when sorting table columns.
- Improved overall platform stability and performance.
- Enhanced user interface responsiveness across various components.
**Data Processing Engine**
- **Fixes:**
- Show more accurate last 24 hours view in DPE metrics.
- Fix issue where environment variables section was missing on PySpark actions.
**Documentation Updates**
- Updated all documentation references from "Data Catalog" to "Connectors"
- Refreshed navigation and sidebar structure to reflect the new naming convention
- Use "Connectors" terminology consistently in all tutorials and guides
## January 12, 2026
**General**
- **Fixes:**
- Fix infinite redirect loop that occurred when navigating away from the MFA (Multi-Factor Authentication) validation flow, ensuring a smoother authentication experience.
- Resolve issue where users at the organization level could not add specific services to favorites in the sidebar.
- Improved API dashboard precision by truncating numeric values to two decimal places. This makes the data display clearer and more consistent.
- Enhance error handling and user-facing messages for quota management. Provide more informative feedback when quota limits are reached or exceeded.
**Lakehouse Manager**
- **Time Travel:** Introduced retention period for time-travel feature, by default it's **14 days**. [Read more on the doc](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md).
- **Trino Optimization:** Optimized Trino memory usage to prevent Out-of-Memory (OOM) errors, improving query stability and performance for large datasets.
- **Table Maintenance:** Enabled the table maintenance job in production. This daily automated job removes old snapshots and orphan files, helping to optimize storage usage and maintain data lakehouse health.
**Data Processing Engine**
- **Reverse-ETL Actions:** Added support for protocol as a destination option in reverse-ETL actions.
- **PySpark Load/Aggregate:** Fix issue with XLS file processing in classic PySpark load and aggregate actions.
**APPs & APIs**
- **Deployment Status:** Improved deployment status updates with more accurate real-time information during app and API deployment processes.
- **View Logs:** Added a new "View Logs" button that appears when an app or API is deployed.
- **Open Button:** Display "Open" button only once app or API is fully deployed, preventing proxy error messages that previously appeared when attempting to access services before deployment completion.
:::details **2025
**
**December 17, 2025**
**General**
- **Fixes:**
- Resolved an issue where boolean values were not displayed in query results when executing queries in Lakehouse Manager Explorer or Analytics Manager.
- Fixed an issue where incorrect data was displayed when switching between 7 days and 24 hours views in job metrics.
- Group names are now correctly shown in the UI after granting access to a group on a policy tag.
**Data Catalog**
- **Google Analytics Connector:** Updated the Google Analytics connector with enhanced functionality. [Learn more about the Google Analytics Connector](https://docs.dataplatform.ovh.net/connectors-sources-google-analytics.md).
**Lakehouse Manager**
- **Time Travel:** Introduced time travel feature for tables, enabling you to query historical versions of your data. [Learn more about Time Travel](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md).
- **Explorer:** Enhanced SQL query capabilities to support Iceberg snapshot version syntax using `FOR VERSION AS OF SNAPSHOT_ID`.
- Improved Trino stability for more reliable query execution.
**Data Processing Engine**
- **Metrics:** DPE Actions and Workflows job metrics now include timed-out executions alongside successful and failed ones, for more comprehensive monitoring.
- **Fixes:**
- Resolved an issue where loading multiple Parquet files from a bucket with segmentation would fail when running multiple instances in parallel.
- Restored the missing Python code sample in the dialog box when viewing API examples under DPE Actions and Workflows **Preferences > Triggers**.
**December 1, 2025**
**General**
- **Project-Level Search:** Introduced a new search functionality within the project level to quickly find menus and sections. Use **CMD+K** on Mac or **CTRL+K** on Windows to access this feature.
- **Quotas:** New Quotas feature now available in the organization settings page, allowing administrators to manage resource allocation limits. [Learn more about Quotas](https://docs.dataplatform.ovh.net/manage-organizations-settings.md#quotas).
- **Performance:** Improved job performance across the platform.
**Lakehouse Manager**
- **REST Catalog:** Deployed a new open-source REST catalog service (Polaris) for Apache Iceberg to prepare for future integrations, such as enhanced table metadata access and more granular RBAC.
- Fixed ACLs issue with policy tags.
**Data Processing Engine**
- **Notebooks:** Python 3.9 notebooks can no longer be created; **Python 3.13** is now the default version.
- **Fixes:**
- Resolved an issue where `S3Store.bucket_exists()` encountered errors when saving status with multiple workers during `get_bucket` operations.
- Fixed PySpark linked environment parameter handling to properly support number and boolean types (previously only string values worked correctly).
**November 17, 2025**
**General**
- During project creation, hostnames no longer accept underscores to ensure better compatibility.
**Data Catalog**
- **Kafka Connector:** Improved performance with faster and more efficient batch loading into datasets.
- Fixed timezone handling in metadata extraction for more accurate data processing.
**Data Processing Engine**
- **Workflows:** Added `Force Rebuild` option, matching the functionality previously available in actions (useful for troubleshooting and ensuring clean re-execution).
- **Custom Actions:** Added support for **Python 3.13**.
- **PySpark Actions:** Fixed environment variable handling to properly support number and boolean types when using linked environments (previously only string values worked correctly).
- **Spark Custom Actions:** Enhanced boilerplate templates for improved developer experience.
**November 3, 2025**
**⚠️ Important Actions Required ⚠️**
- **Front-end Applications with MFA:** Due to a security patch, applications using project-level authentication with Multi-Factor Authentication (MFA) require a manual update. Users must update the `client-authority-manager` dependency to version `1.5.0` in their `forepaas.json` file (located in the root folder of the Front App), then rebuild and redeploy the application.
**General**
- Platform-wide infrastructure improvements for enhanced stability.
- **Security:** Improved token rotation when switching between organizations and increased authenticator secret length for better MFA compatibility.
**Data Catalog**
- **Kafka Connector:** Now returns flattened JSON by default for easier processing.
- **Source Management:** Columns using [reserved keywords](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md#reserved-words) are now automatically renamed with an `_r` suffix to prevent SQL query and table build failures.
**Data Processing Engine**
- **Jupyter AI Extension:** Notebooks now include the Jupyter AI extension for conversational assistance. Setup is required with your OVHcloud AI endpoint key.
- **Notebooks:**
- Improved startup time with asynchronous post-start processes.
- New resource usage metrics (CPU, Memory, Storage) are now displayed.
- Fixed an issue preventing the override of existing dependencies.
- Updated example notebooks and boilerplates.
- Resolved compatibility issues with Spark versions 3.4.4 and 3.5.5.
**Lakehouse Manager**
- **SDK Update:** Added a new [`create_from_physical`](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md#create_from_physicaltable-datasetdefault_dataset-kwargs) method to create logical objects from existing physical tables.
- Improved stability for configuration imports by correctly setting table statuses.
**Analytics Manager**
- Fixed out-of-memory issues in Trino to enhance query performance and stability.
**APPs & APIs**
- There are no specific updates for this section in this release beyond the security patch mentioned above.
**Documentation Updates**
- **Tutorials Hub:** Replaces the previous Tutorials section, organizing learning materials into clear categories such as project setup, data ingestion, transformation, export, analysis with Trino, and business intelligence.
- **[Jupyter AI Extension](https://docs.dataplatform.ovh.net/dpe-notebooks-jupyter-ai-extension.md):** New guide for the built-in Jupyter AI extension, including setup with your OVHcloud AI endpoint key and using conversational assistance in notebooks.
- **[SQL Transformation Guide](https://docs.dataplatform.ovh.net/tutorials-sql-transformation.md):** Step-by-step examples showing how to perform SQL-based data transformations within notebooks on the data platform.
- **SDK Reference:** Updated with new [dataset connector methods](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-dataset.md); more updates coming in future releases.
- **Link Fixes:** Several broken links across the documentation have been corrected for better navigation.
**October 20, 2025**
**General**
- Platform-wide dependency upgrades for enhanced stability and security.
- Fixed an issue where the Power BI connector was returning the wrong subdomain when sharing Project URLs.
**Data Processing Engine**
- Default Spark version updated to **3.5.5** for load and aggregate actions
- Updated SDK to latest versions of **Pandas 2.x** and **SQLAlchemy 2.x**, improving notebook compatibility.
- Added support for **Python 3.13** with updated base, PySpark, TensorFlow, and DataScience notebook images
- New example notebooks included with every notebook image.
**Data Catalog**
- Added **NDJSON** format support for JSON uploads.
**October 6, 2025**
**⚠️ Important Actions Required ⚠️**
- **For Apache Superset (using the SQLite template) built before October 6th:** Clients must rebuild and redeploy the environment to ensure compatibility and support.
- **For the Front Application built before October 6th:** Clients must update the SDK to version **1.3.0** in `forepaas.json`, then rebuild and redeploy the application.
**General**
- Improved SSO login flow to only display auto-provisioned options where applicable
- Fixed navigation, loading, and multi-button hover issues in UI
- Database connectors now display warnings and documentation links
**Data Processing Engine**
- Fixed issues with PySpark actions and aggregate action reliability
- Updated Oracle and Polar connectors for improved compatibility
- New warning when changing destinations with multiple sources
**Lakehouse Manager**
- Fixed missing access rights for job endpoints (users now need explicit rights)
- Improve stability and performance
- Added Metastore service to handle metadata more efficiently
- Fixed logical attribute build issue with incremental updates.
- Polaris catalog deployed for open-source REST compatibility
**Analytics Manager**
- Fixed ACL-related constraints for endpoints
- Configuration service updated for improved Trino catalog compatibility
**Apps & APIs**
- Fixed automatic deployment issues
- Optimized Superset builds.
- Updated store templates and SDK modules
**September 22, 2025**
**General**
- Minor bug fixes and improvements
**September 8, 2025**
**General**
- Improved error handling and log sanitization to make troubleshooting smoother and more secure.
- Fixed errors with invalid parameters in the Log Explorer.
**Data Catalog**
- Improved OVHAPI Connector with enhanced V2 compatibility
**Data Processing Engine**
- Updated Python environment to _v3.13.7_ for improved security and compatibility.
- Enhanced logs for easier debugging.
- Improved PySpark execution reliability, reducing warnings and errors during workload runs.
- Fixed an issue with SQL Server installation dependencies to ensure smoother deployments
**Lakehouse Manager**
- Better error message handling when importing configurations to simplify troubleshooting
- Stability improvements across Lakehouse query services.
**August 25, 2025**
**General**
- [**Consumers**](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) have now been moved to the [Lakehouse Manager](https://docs.dataplatform.ovh.net/landing-page-lakehouse-manager.md)
- Improved logs for the Lakehouse Manager "all-in-one" build action.
- Fixed an issue where "all-in-one" action reports a successful metadata update (update metas successful) but returns a result of -1.
**August 11, 2025**
**General**
- Trino version upgrade v461 to v476
- PowerBI consumer is now available for use. guide on how to use is available [here](https://docs.dataplatform.ovh.net/connectors-consumers-power-bi.md)
- Fixed minor issue with the edit icon not appearing on DPE actions/workflows/environments.
**July 28, 2025**
**General**
- Improved ACL for actions for Data Processing Engine
- New tutorial available for a FinOps uses case with FOCUS and superset - [here](https://docs.dataplatform.ovh.net/tutorials-finops-focus-superset.md)
**July 15, 2025**
**General**
- Apache Superset is now availalable in the App Service
- Adjusted Spark log levels and optimized processing for better performance
- Introduced the [NYC Taxi dataset tutorial](https://docs.dataplatform.ovh.net/tutorials-pyspark.md) with two practical examples and a [PySpark cheat sheet](https://docs.dataplatform.ovh.net/tutorials-pyspark-cheat-sheet.md) for enhanced learning and quick reference.
**Data Processing Engine**
- Added new metrics to monitor notebook usage and performance
- Improved notebook import/export experience for better usability
- Improved documentation for [PySpark modules](https://docs.dataplatform.ovh.net/developers-python-sdk-connect-spark.md) and PySpark [use cases](https://docs.dataplatform.ovh.net/developers-python-sdk-quick-start-spark.md), making it easier to leverage distributed processing.
- Added comprehensive documentation for previously missing Data Processing Engine (DPE) [actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md), expanding available guides.
**UI / Platform**
- Fixed an issue where Log Explorer wouldn’t display the bottom of the window correctly
- Resolved a WebSocket issue when multiple users accessed the same DPE
- You can now add widgets directly without going through the edit menu
- Updated DPE Notebooks interface for a smoother experience
- S31-compatible storage now automatically lowercases the region field for consistency
**June 30, 2025**
**General**
- Project's home: update "APPLICATION SERVICES" tile by merging "Apps" and "APIs" tiles
- Improve OAuth integration with MFA
**Data Processing Engine**
- All Actions: Added a **Force Rebuild** option, useful for troubleshooting and ensuring a clean re-execution.
- PySpark Actions: added selection of appropriate "Spark Log Level"
- PySpark Custom Action: added selectition of PySpark version (currently 3.3.2, 3.4.4, 3.5.5)
**Lakehouse manager**
- various optimisation and bugfixes
**Control Center**
- Logs Explorer improvements
- Jobs Central: fix Last 30 days display
**June 9, 2025**
**Data Processing Engine**
- PySpark Load: allow multiple files loading from Bucket
**Control Center**
- Logs Explorer: fix when last second of logs were sometimes not displayed
**May 26, 2025**
**Data Catalog**
- Added JSONPath support to OVH API sources for simpler data navigation.
- Improved performance of OVH API sources for quicker data access.
- Made it easier to manage multiple endpoints in OVH API sources.
**Data Processing Engine**
- New Action - Export Table to Source: Export selected tables from your lakehouse to an external data source.
- Fixed the Time Breakdown view in job details for accurate insights into job performance.
- Resolved an issue preventing users from leaving the editing page in certain actions without refreshing.
- Improved notebook usability with a pending status display when stopping a notebook.
- Enhanced metrics for better monitoring.
- Optimized Spark processing for improved performance and reliability.
- Adjusted Spark log levels for clearer debugging and monitoring.
- Fixed an issue with incorrect start dates in Logs Explorer for accurate job tracking.
- Fixed incomplete display of last 24 hours metrics for accurate monitoring.
- Reworked notebook access controls to include read-only support for better collaboration.
**Analytics Manager**
- Improved query menu interface with a clear display of visual or SQL mode on the main page.
- Enhanced handling of queries to support virtual attributes, ensuring reliable performance.
**Lakehouse Manager**
- Fixed an issue where copying tables with a large number of attributes caused errors.
- Resolved errors during Physical Build stops for smoother operations.
- Standardized metadata extraction for better data consistency.
- Added physical build status checks by job type for improved monitoring.
**May 12, 2025**
**General**
- New [tutorial](https://docs.dataplatform.ovh.net/tutorials-billing-analysis.md) to use OVH API Connector for billing analytics
**Data Catalog**
- Better handling of date time string with UTC time zone
- Improved Analyzer for streaming sources
- Improved OVH API connector
- Improved handling of PostgreSQL source using Direct Query and integration with Trino
**Data Processing Engine**
- Python SDK: we have fixed package **boto3 \< 1.36.0.**
- **⚠️ Note:** For compatibility reasons, please do not include the boto3 library in your custom dependencies
**April 28, 2025**
**General**
- Improved login page text for a better experience.
- Tutorials and documentation now open in a new tab from the homepage.
- Updated text on the Analytics Manager and SQL Queries screens.
- Simplified organization invitations by removing the "create an organization" option.
- Fixed occasional issues when searching in the policy tag list.
- Fixed action links in Notebooks to better reflect user's permissions.
- Improved user list visibility when managing organization/project invites.
- Improved display of logs in the Control Center.
**Data Processing Engine**
- Fixed toolbar icon display in Advanced Mode for better usability.
- Improved stability after notebook crashes, with better disk restoration.
**Lakehouse Manager**
- Improved dataset status updates.
- Improved performance of physical builds.
**April 14, 2025**
**General**
- Several UI fixes in Organization's pages and Identity Access Manager
**Data Proessing Engine**
- Fix File Transfer Action
- In logs, add Action's name when just Action's ID was visible
- Fix some logs visibility in Action / Workflow / Job view.
**Lakehouse Manager**
- Improve dataset creation time
- Better handling of Explorer when Trino is not available
**Control Center**
- Fix display of some Monitoring views
**March 28, 2025**
**General**
- Enhanced dataset creation flow for a more user-friendly experience.
- Fixed minor bugs related to policy tags.
- Resolved an issue where Control Center logs were not displaying correctly.
**Data Processing Engine**
- Enhanced logging for PySpark actions to improve traceability and debugging.
- Improved OVHcloud API connector for better handling of multiple endpoints.
- Notebooks:
- Added support for Python 3.11
- Introduced new Data Science and TensorFlow notebook types.
**March 18, 2025**
**General**
- Documentation is now available at [https://docs.dataplatform.ovh.net/](https://docs.dataplatform.ovh.net/)
- Improve handling of logins mixing upper and lower cases
- Improve readability of some error messages
- Update project is ready e-mail
- Remove all UI references to BETA
**Data Processing Engine**
- Streamline events exchange with UI
- Notebooks: update Data Platform Python SDK
**Lakehouse Manager**
- Improve all-in-one action to better handle attributes names
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
```sql
SHOW SCHEMAS FROM CATALOG_NAME
```
:::info
**Note:** Replace `CATALOG_NAME` with the actual catalog you selected in the previous step.
:::
This command will display a list of available schemas within your chosen catalog.
Copy and save the desired schema name as `SCHEMA_NAME`.
:::info
Upon creation of a new project, the `default dataset` is provided. If you have not established any additional datasets within your project, this is the appropriate selection.
:::
2. Open the _Identity Access Manager_ and create a new service account for the purpose of this transfer.
:::info
Make sure to add _Platform Admin_ as the group for this service account
:::
Once the service account has been successfully create, you need to generate a key for the _API Access_. Click on the add button as shown in the picture below, followed by create.
You will be provided with the _API Key_ and the _Secret Key_. Make sure to copy and save them securely as you wont be able to access the _Secret Key_ once you leave this page.
You have successfully completed all the steps required at the source project; the next part of the tutorial will continue on the destination project.
### Steps to follow at destination project
1. Proceed to the _Connectors_ and add a new Trino Source.
2. Fill in the details as given below and click on connect
- **Host:** replace by your _**source**_ _Project\_Name_ + `-query.eu.dataplatform.ovh.net`
- **Port:** 443
- **Catalog:** replace with your _CATALOG\_NAME_
- **Schema:** replace with your _SCHEMA\_NAME_
- **Username:** replace with _API\_KEY_
- **Password:** replace with _SECRET\_KEY_
- **SSL:** activated
3. Once the Trino Connection has been successfully established, the list of available tables will be populated. Click on the box next to table name you desire to extract from the source and click create.
## Using the extracted data
Once all the previous steps have been completed, you can visit the _Analyzer_ tab within the _Connectors_ to verify the extracted tables.
To use this extracted data you can follow **step three** of the [Getting Started](https://docs.dataplatform.ovh.net/getting-started-prepare-data.md) Tutorial.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Integrating OVHcloud Data Platform with External Tools using Trino
## Objective
:::info
**Looking for the quick way?** You can now generate a ready-to-use Trino connection string in a few clicks with a [**Trino consumer**](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md). This guide covers the **manual method**: useful when you want to set up the service account, role, and connection string yourself, or need fine-grained control over the catalog and permissions.
:::
This guide provides a comprehensive walkthrough for creating a Trino integration to be used in a third-party tool. You will generate the necessary credentials and construct a connection string to access your data via Trino.
## Steps to Establish a Trino Connection
### Step 1: Create a New Service Account
1. Log in to the Data Platform.
2. Navigate to **IAM** (Identity and Access Manager).
3. Go to the **Roles** section and click **New Role**.
4. Provide a name for the role (e.g., `TrinoServiceAccountRole`).
5. Assign the **Advanced Data Access Control** permissions to this role. This role will be assigned to your service account to grant the necessary Trino access.
6. Go to **Service Accounts**.
7. Create a new Service Account and give it a name (e.g., `Apache Superset Service Account` or `Metabase Service Account`) and an email address.
8. Assign the newly created role (`TrinoServiceAccountRole`) to this account.
9. Click **Create**.
:::info
For better security and access management, create a dedicated service account when allowing external tools to access your data.
:::
### Step 2: Generate an API Key and Secret Key
1. Once you have created the _Service Account_, you will see the section for **API Access**. Click on **+ ADD**.
2. Here you can modify the duration of the keys and also if you want to send a copy to your registered email.
3. **Critical**: Once the modal is closed, the keys cannot be retrieved again. Be sure to save them immediately in a secure location, such as a password manager. These API Key and Secret Key will serve as your `Username` and `Password` for the Trino connection.
### Step 3: Prepare the connection string
The connection string you will be able to use in your third-party tools will be as follows:
```text
trino://{Username}:{Password}@{Project}-query.eu.dataplatform.ovh.net:{Port}/{Catalog}
```
- **Project**: Replace `{Project}` with your specific project name. You can find this name by navigating to any of your existing projects, then clicking the Information icon represented by ℹ️ in the top right corner of the interface. Your project name will be displayed there.
- **Port**: Replace `{Port}` with the default Data Platform port (`443`) or a custom port provided by your setup.
- **Catalog**: Replace `{Catalog}` with the catalog you want to query. You can find the catalog you want by executing this SQL command in the explorer of the Lakehouse Manager: `SHOW CATALOGS`.
- **Example `SHOW CATALOGS` output:**
```text
Catalog
-------
tpcds
tpch
db_m8obb5ch_default_dataset
```
In this example, `db_m8obb5ch_default_dataset` is a valid catalog name.
- **Username**: Use the _API Key_ generated earlier from IAM in Step 2.
- **Password**: Use the _Secret Key_ generated earlier from IAM in Step 2.
**Example:**
```text
trino://XXXX:YYYYYY@demo-query.eu.dataplatform.ovh.net:443/db_m8obb5ch_default_dataset
```
### Step 4: Use the connection string in your third-party tool
Once you have prepared your connection string, you can use it to connect your preferred third-party tool (e.g., Apache Superset) to Trino on OVHcloud Data Platform.
**Returning to a Specific Tool Guide?**
- If you came from the Apache Superset integration guide, click [here](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) to continue.
- If you came from the Metabase integration guide, click [here](https://docs.dataplatform.ovh.net/tutorials-metabase-trino.md) to continue.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-trino-types-and-casts.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Trino data types and casts
## Objective
Whatever path you take to query your project over [Trino](https://trino.io/), the SQL editor of the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md), a [Trino consumer](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), or an external tool, the types you manipulate are the **Trino types**, not the logical types you see in the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-attributes.md). Most of the time the two line up, but a few cases require an explicit `CAST` at `INSERT` time.
## Data types: Lakehouse to Trino
The table below maps each Lakehouse type to the Trino type it is exposed as:
| Lakehouse type | Trino type | Notes |
| ---------------- | --------------- | --------------------------------------------- |
| `integer` | `INTEGER` | 32-bit signed integer |
| `bigint` | `BIGINT` | 64-bit signed integer |
| `real` / `float` | `REAL` | 32-bit floating point |
| `double` | `DOUBLE` | 64-bit floating point |
| `decimal` | `DECIMAL(p, s)` | Precision and scale are preserved |
| `varchar` | `VARCHAR` | Unbounded unless a length was set at creation |
| `boolean` | `BOOLEAN` | |
| `date` | `DATE` | |
| `timestamp` | `TIMESTAMP` | Timezone-naive |
:::info
The exact Trino type of a column, including any length or precision, is whatever the table was **created** with. When in doubt, run `SHOW CREATE TABLE 
- Upload the `yellow_tripdata_YEAR_MONTH.parquet` and `taxi_zone_lookup.csv` files, then click create.
- Go to the Analyzer tab and perform meta extraction to infer schemas for the uploaded files.
### Step 2: Create Tables in Lakehouse Manager
- Go to _Tables_ tab → click on **Create from a Connectors source**.
- Select the `yellow_tripdata_YEAR_MONTH.parquet` file → and only select the option "Automatically build the table".
- Repeat for the `taxi_zone_lookup.csv` file, select all the options since this table is smaller and can be fully loaded by Iceberg quickly.
### Step 3: Use Load PySpark action in the DPE
- Go to _Actions_ tab → Create new action → Use **Load PySpark**.
- Set:
- **Source**: `yellow_tripdata_YEAR_MONTH`
- **Destination**: `yellow_tripdata_YEAR_MONTH` Table in the Lakehouse Manager.
- Ensure schema mapping is correct (the source columns should align with the table’s schema)
- Save and run the action.
### Step 4: Create a Jupyter Notebook in the DPE
- Go to the "Notebooks" tab → New notebook → Select **PySpark Notebook**.
- Name it as `NYC yellow taxi notebook`.
After launching, the platform will take a few moments to initialize a JupyterHub instance. Once it's ready, click the **Open in Jupyter** button to access the notebook interface.
With the data environment now fully configured and all sources loaded into the Lakehouse, we’re ready to dive into the PySpark notebook and begin our analysis.
## PySpark Notebook Walkthrough
The notebook below includes the full PySpark code, structured step by step with inline comments and explanations. You can download it and upload it directly into your Jupyter environment on the OVHcloud Data Platform. Each step is explained in detail to help you follow the analysis process, and a summary is provided at the end for quick reference.
:::info
[NYC_Yellow_Taxi_Dataset_Analysis.ipynb
](/images/public-cloud/data-platform/getting-further/pyspark/resources/NYC_Yellow_Taxi_Dataset_Analysis.ipynb)
:::
### What’s Inside the Notebook?
The notebook walks you through:
- Connecting to the Lakehouse Manager and loading the dataset
- Cleaning and joining trip data with zone lookups
- Performing EDA with Matplotlib and Seaborn
- Building and comparing three ML models (Linear Regression, Random Forest, GBT)
- Outputting performance metrics and insights
Below are a few sample outputs from the notebook:
## NYC Taxi Trip Duration Prediction – Summary
### Objective
The objective of this analysis was to predict `trip_duration` using machine learning models trained on NYC Yellow Taxi data. These predictions can help optimize taxi dispatch strategies, improve fare estimates, and support operational decision-making.
### Models Compared
We evaluated three regression models to understand their effectiveness on this task:
- **Linear Regression** – a simple and interpretable baseline
- **Random Forest** – an ensemble method that captures non-linear relationships
- **Gradient Boosted Trees (GBT)** – a boosting-based approach aimed at maximizing predictive accuracy
### Data Preparation
To ensure high-quality input data and better model performance, the following steps were performed:
- Removed extreme outliers (trips shorter than 60 seconds or longer than 3600 seconds)
- Encoded categorical variables (e.g., `pickup_borough`)
- Scaled and assembled numerical features into a unified feature vector
### Features Used
- `trip_distance`
- `pickup_hour`
- `day_of_week`
- `pickup_borough` (indexed)
### Data Split
- **Training set:** 80% (\~2.2 million records)
- **Test set:** 20% (\~556,000 records)
### Model Performance
| Model | RMSE (seconds) | R² |
| ----------------- | -------------- | ----- |
| Linear Regression | 373.1 | 0.675 |
| Random Forest | 318.6 | 0.763 |
| GBT | 288.7 | 0.806 |
**Interpretation:**\
GBT achieved the best performance overall, with the lowest Root Mean Squared Error (\~4.8 minutes) and the highest R² score (80.6% variance explained).
### Feature Importance (Random Forest)
| Feature | Importance (%) |
| ---------------------- | -------------- |
| `trip_distance` | 77.4 |
| `pickup_borough_index` | 19.8 |
| `pickup_hour` | 2.4 |
| `day_of_week` | 0.4 |
**Insight:**\
Trip distance was by far the most influential predictor. Pickup borough had some impact, while time-based features (hour and weekday) had minimal effect on trip duration.
### Impact of Data Cleaning
- Removed records with negative or unrealistic values
- Reduced noise and improved model reliability
- Final dataset (\~2.78 million records) was more representative
- Resulted in lower RMSE and higher R² across all models, especially GBT
## Conclusion
This tutorial demonstrated how PySpark and the OVHcloud Data Platform can be used effectively to process, explore, and model large-scale taxi trip data. Through structured data preparation, visual analysis, and model evaluation, we identified key drivers of trip duration and built a predictive model with strong performance.
Among the models tested, Gradient Boosted Trees (GBT) offered the best results in both accuracy and robustness, highlighting its value for real-world forecasting tasks.
The workflow outlined here, covering ingestion, transformation, exploration, and machine learning, can serve as a reusable framework for similar large-scale analytics projects across different domains.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-pyspark-multi-dataset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# NYC Taxi Comparative Analysis Across Types with PySpark
## Objective
This tutorial extends our previous analysis to compare trip patterns across four different NYC transportation datasets for **January 2025**: Yellow Taxi, Green Taxi, For-Hire Vehicle (FHV), and High Volume FHV Trip Records. We'll perform a comprehensive comparative analysis to understand the differences in service usage across NYC's ride-sharing ecosystem.
These datasets represent distinct segments of NYC's transportation network:
- **Yellow Taxi**: The iconic yellow cabs, primarily street-hail services operating mainly in Manhattan
- **Green Taxi**: Street-hail taxis specifically focused on outer boroughs (Brooklyn, Queens, Bronx, Staten Island)
- **FHV**: For-hire vehicles (including Uber, Lyft) booked via mobile apps, covering all boroughs
- **High Volume FHV**: High-volume providers like Uber and Lyft with exceptionally large trip counts
By comparing trip volume, average trip duration, and pickup zones across boroughs, we'll uncover key differences in service usage patterns. This analysis provides valuable insights for urban planners, taxi operators, and ride-sharing companies analyzing market dynamics and operational strategies.
The tutorial builds on cleaning and joining techniques from our previous Yellow Taxi analysis, performs comprehensive **exploratory data analysis (EDA)**, and creates **visualizations** to highlight usage patterns across different transportation modes.
## Requirements
Before starting this comparative analysis, ensure you have:
- **Data Setup**: All four datasets (`yellow_tripdata_2025_01.parquet`, `green_tripdata_2025_01.parquet`, `fhv_tripdata_2025_01.parquet`, `fhvhv_tripdata_2025_01.parquet`) and the `taxi_zone_lookup.csv` file are loaded in your Connectors and accessible in the Lakehouse Manager. You can download these datasets from the [official NYC TLC Trip Record Data website](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
- **Environment**: A PySpark-enabled Jupyter notebook configured on the OVHcloud Data Platform
- **Previous Tutorial**: Completed the **[NYC Yellow Taxi Dataset Analysis](https://docs.dataplatform.ovh.net/tutorials-pyspark-single-dataset.md)** tutorial for foundational understanding
If you haven't completed the data setup, follow **Steps 1-3** from the previous tutorial to:
1. Upload datasets to the Connectors
2. Create tables in the Lakehouse Manager
3. Configure load actions in the Data Processing Engine (DPE)
## Step-by-Step Tutorial
The complete PySpark code is provided below with detailed explanations for each step. Copy and paste this into a new Jupyter notebook on the OVHcloud Data Platform. Each section includes comprehensive comments and explanations to guide you through the comparative analysis process.
### Step 1: Environment Setup and Data Connection
We initialize our PySpark session and establish connections to access all four transportation datasets.
```python
import logging
from forepaas.dwh import connect
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, col, hour, dayofweek, unix_timestamp, avg, count, sum, when
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
# Configuration variables
DATASET = "default_dataset"
PROJECT_ID = "PROJECT_ID" # Replace with your actual Project ID
YEAR = "2025"
MONTH = "01"
# Set up logging for debugging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Initialize SparkSession
try:
spark = SparkSession.builder.appName("NYC_Taxi_Comparative_Analysis").getOrCreate()
logging.info(f"Spark Version: {spark.version}")
except Exception as e:
logging.error(f"Failed to initialize SparkSession: {e}")
raise
# Connect to Lakehouse
try:
cn_prim = connect(f"dwh/{DATASET}/")
logging.info(f"Connected to Lakehouse - Dataset: {DATASET}")
except Exception as e:
logging.error(f"Failed to connect to Lakehouse - Dataset: {DATASET} | {e}")
raise
```
**What this does:**
- Imports necessary libraries for data processing and visualization
- Configures logging for debugging and monitoring
- Initializes a PySpark session optimized for large-scale data processing
- Establishes connection to the OVHcloud Data Platform Lakehouse
### Step 2: Data Loading and Initial Inspection
We load all four transportation datasets and the taxi zone lookup table, then verify successful loading and inspect their schemas.
```python
# Load all datasets from the Lakehouse
yellow_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.yellow_tripdata_2025_01")
green_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.green_tripdata_2025_01")
fhv_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.fhv_tripdata_2025_01")
fhvhv_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.fhvhv_tripdata_2025_01")
taxi_zones_df = cn_prim.query(f"SELECT LocationID, Borough, Zone FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.taxi_zone_lookup")
# Cache DataFrames for improved performance
yellow_df.cache()
green_df.cache()
fhv_df.cache()
fhvhv_df.cache()
taxi_zones_df.cache()
# Verify successful data loading
logging.info(f"Yellow Taxi Records: {yellow_df.count():,}")
logging.info(f"Green Taxi Records: {green_df.count():,}")
logging.info(f"FHV Records: {fhv_df.count():,}")
logging.info(f"High Volume FHV Records: {fhvhv_df.count():,}")
logging.info(f"Taxi Zones Records: {taxi_zones_df.count():,}")
# Inspect dataset schemas
print("=== DATASET SCHEMAS ===")
print("\nYellow Taxi Schema:")
yellow_df.printSchema()
print("\nGreen Taxi Schema:")
green_df.printSchema()
print("\nFHV Schema:")
fhv_df.printSchema()
print("\nHigh Volume FHV Schema:")
fhvhv_df.printSchema()
print("\nTaxi Zones Schema:")
taxi_zones_df.printSchema()
```
**Expected Output:**
- Yellow Taxi: \~3,475,226 records
- Green Taxi: \~48,326 records (notably smaller, possibly partial dataset)
- FHV: \~1,894,659 records
- High Volume FHV: \~20,405,666 records (largest dataset)
- Taxi Zones: 265 records
**Why this matters:**
- Caching improves performance for repeated operations, especially critical for the High Volume FHV dataset
- Schema inspection reveals column differences that need standardization
- Record counts help understand the relative scale of each transportation mode
### Step 3: Data Cleaning and Standardization
We clean each dataset to remove invalid records and standardize column names and data types for consistent analysis across all transportation modes.
```python
# Clean Yellow Taxi DataFrame
yellow_df_clean = yellow_df.filter(
(col("tpep_pickup_datetime").isNotNull()) &
(col("tpep_dropoff_datetime").isNotNull()) &
(col("passenger_count").isNotNull()) &
(col("passenger_count") > 0) &
(col("trip_distance") > 0) &
(col("fare_amount") > 0)
).withColumn("pickup_datetime", col("tpep_pickup_datetime")) \
.withColumn("dropoff_datetime", col("tpep_dropoff_datetime")) \
.withColumn("trip_duration", unix_timestamp("tpep_dropoff_datetime") - unix_timestamp("tpep_pickup_datetime")) \
.withColumn("pickup_hour", hour("tpep_pickup_datetime")) \
.withColumn("day_of_week", dayofweek("tpep_pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Clean Green Taxi DataFrame
green_df_clean = green_df.filter(
(col("lpep_pickup_datetime").isNotNull()) &
(col("lpep_dropoff_datetime").isNotNull()) &
(col("passenger_count").isNotNull()) &
(col("passenger_count") > 0) &
(col("trip_distance") > 0) &
(col("fare_amount") > 0)
).withColumn("pickup_datetime", col("lpep_pickup_datetime")) \
.withColumn("dropoff_datetime", col("lpep_dropoff_datetime")) \
.withColumn("trip_duration", unix_timestamp("lpep_dropoff_datetime") - unix_timestamp("lpep_pickup_datetime")) \
.withColumn("pickup_hour", hour("lpep_pickup_datetime")) \
.withColumn("day_of_week", dayofweek("lpep_pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Clean FHV DataFrame (Note: PULocationID is string, cast to double)
fhv_df_clean = fhv_df.filter(
(col("pickup_datetime").isNotNull()) &
(col("dropoff_datetime").isNotNull()) &
(col("pulocationid").isNotNull())
).withColumn("trip_duration", unix_timestamp("dropoff_datetime") - unix_timestamp("pickup_datetime")) \
.withColumn("pickup_hour", hour("pickup_datetime")) \
.withColumn("day_of_week", dayofweek("pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Clean High Volume FHV DataFrame (uses trip_miles instead of trip_distance)
fhvhv_df_clean = fhvhv_df.filter(
(col("pickup_datetime").isNotNull()) &
(col("dropoff_datetime").isNotNull()) &
(col("trip_miles").isNotNull()) &
(col("trip_miles") > 0) &
(col("pulocationid").isNotNull())
).withColumn("trip_duration", unix_timestamp("dropoff_datetime") - unix_timestamp("pickup_datetime")) \
.withColumn("trip_distance", col("trip_miles")) \
.withColumn("pickup_hour", hour("pickup_datetime")) \
.withColumn("day_of_week", dayofweek("pickup_datetime")) \
.withColumn("PULocationID", col("pulocationid").cast("double"))
# Apply consistent outlier filtering and capping for all datasets
yellow_df_clean = yellow_df_clean.filter(
(col("trip_duration") >= 60) &
(col("trip_distance") >= 0.1) &
(col("trip_distance").isNotNull()) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))) \
.withColumn("trip_distance", when(col("trip_distance") > 50, 50).otherwise(col("trip_distance")))
green_df_clean = green_df_clean.filter(
(col("trip_duration") >= 60) &
(col("trip_distance") >= 0.1) &
(col("trip_distance").isNotNull()) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))) \
.withColumn("trip_distance", when(col("trip_distance") > 50, 50).otherwise(col("trip_distance")))
fhv_df_clean = fhv_df_clean.filter(
(col("trip_duration") >= 60) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration")))
fhvhv_df_clean = fhvhv_df_clean.filter(
(col("trip_duration") >= 60) &
(col("trip_distance") >= 0.1) &
(col("pickup_hour").isNotNull()) &
(col("PULocationID").isNotNull())
).withColumn("trip_duration", when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))) \
.withColumn("trip_distance", when(col("trip_distance") > 50, 50).otherwise(col("trip_distance")))
# Verify cleaned data counts
logging.info("=== CLEANED DATA COUNTS ===")
logging.info(f"Cleaned Yellow Taxi Records: {yellow_df_clean.count():,}")
logging.info(f"Cleaned Green Taxi Records: {green_df_clean.count():,}")
logging.info(f"Cleaned FHV Records: {fhv_df_clean.count():,}")
logging.info(f"Cleaned High Volume FHV Records: {fhvhv_df_clean.count():,}")
```
**Key Cleaning Steps:**
- **Remove invalid records**: Null datetimes, zero distances, negative passenger counts
- **Standardize columns**: Create consistent `pickup_datetime`, `trip_duration`, `pickup_hour`, `day_of_week`, and `PULocationID`
- **Handle schema differences**: Cast `PULocationID` to double for FHV datasets, rename `trip_miles` to `trip_distance` for High Volume FHV
- **Cap outliers**: Limit trip durations to 1 hour (3600 seconds) and distances to 50 miles to reduce impact of extreme values
- **Filter edge cases**: Remove trips shorter than 60 seconds or distances less than 0.1 miles
### Step 4: Geographic Context with Taxi Zone Joins
We join each cleaned dataset with the Taxi Zone Lookup table to add borough and zone information for geographic analysis.
```python
# Join all datasets with taxi zones for pickup location context
yellow_df_clean = yellow_df_clean.join(taxi_zones_df, yellow_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
green_df_clean = green_df_clean.join(taxi_zones_df, green_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
fhv_df_clean = fhv_df_clean.join(taxi_zones_df, fhv_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
fhvhv_df_clean = fhvhv_df_clean.join(taxi_zones_df, fhvhv_df_clean.PULocationID == taxi_zones_df.LocationID, "left") \
.withColumnRenamed("Borough", "pickup_borough") \
.withColumnRenamed("Zone", "pickup_zone") \
.drop("LocationID")
# Filter out records with invalid or unknown geographic information
yellow_df_clean = yellow_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
green_df_clean = green_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
fhv_df_clean = fhv_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
fhvhv_df_clean = fhvhv_df_clean.filter(
(col("pickup_zone") != "Unknown") &
(col("pickup_borough") != "Unknown") &
(col("pickup_borough") != "N/A") &
(col("pickup_borough").isNotNull())
)
# Verify final cleaned data after geographic filtering
logging.info("=== FINAL CLEANED DATA COUNTS ===")
logging.info(f"Final Yellow Taxi Records: {yellow_df_clean.count():,}")
logging.info(f"Final Green Taxi Records: {green_df_clean.count():,}")
logging.info(f"Final FHV Records: {fhv_df_clean.count():,}")
logging.info(f"Final High Volume FHV Records: {fhvhv_df_clean.count():,}")
```
**Why Geographic Context Matters:**
- **Market Analysis**: Understanding which boroughs each service primarily serves
- **Operational Insights**: Identifying high-demand zones for resource allocation
- **Competitive Analysis**: Comparing service penetration across different areas
- **Urban Planning**: Supporting transportation infrastructure decisions
### Step 5: Comparative Exploratory Data Analysis
Now we perform comprehensive comparative analysis across all four transportation modes to identify usage patterns, market share, and operational characteristics.
#### Analysis 1: Trip Volume Comparison by Borough
```python
# Add trip_type identifier to each dataset
yellow_df_clean = yellow_df_clean.withColumn("trip_type", lit("Yellow Taxi"))
green_df_clean = green_df_clean.withColumn("trip_type", lit("Green Taxi"))
fhv_df_clean = fhv_df_clean.withColumn("trip_type", lit("FHV"))
fhvhv_df_clean = fhvhv_df_clean.withColumn("trip_type", lit("High Volume FHV"))
# Create unified dataset for comparative analysis
# Note: Excluding trip_distance as FHV dataset lacks this field
combined_df = yellow_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type") \
.union(green_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type")) \
.union(fhv_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type")) \
.union(fhvhv_df_clean.select("pickup_borough", "trip_duration", "pickup_hour", "day_of_week", "pickup_zone", "trip_type"))
# Calculate trip volume by borough and taxi type
trip_volume_by_borough = combined_df.groupBy("trip_type", "pickup_borough") \
.agg(count("*").alias("num_trips")) \
.orderBy("pickup_borough", "trip_type")
# Convert to Pandas for visualization
trip_volume_by_bpd = trip_volume_by_borough.toPandas()
# Create comprehensive visualization
plt.figure(figsize=(14, 8))
sns.barplot(data=trip_volume_by_bpd, x="pickup_borough", y="num_trips", hue="trip_type", palette="Set1")
plt.xlabel("Pickup Borough", fontsize=12)
plt.ylabel("Number of Trips", fontsize=12)
plt.title("Trip Volume Comparison by Borough and Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.xticks(rotation=45, ha='right')
plt.legend(title="Transportation Type", bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Log key insights
logging.info("=== TRIP VOLUME INSIGHTS ===")
for borough in trip_volume_by_bpd["pickup_borough"].unique():
borough_data = trip_volume_by_bpd[trip_volume_by_bpd["pickup_borough"] == borough]
dominant_service = borough_data.loc[borough_data["num_trips"].idxmax()]
logging.info(f"{borough}: Dominant service is {dominant_service['trip_type']} with {dominant_service['num_trips']:,} trips")
```
#### Analysis 2: Average Trip Duration Comparison
```python
# Calculate average trip duration by transportation type
duration_by_type = combined_df.groupBy("trip_type") \
.agg(avg("trip_duration").alias("avg_duration_seconds")) \
.orderBy("avg_duration_seconds", ascending=False)
# Convert to Pandas and add minutes column
duration_by_type_pd = duration_by_type.toPandas()
duration_by_type_pd["avg_duration_minutes"] = duration_by_type_pd["avg_duration_seconds"] / 60
# Create visualization
plt.figure(figsize=(12, 6))
bars = plt.bar(duration_by_type_pd["trip_type"], duration_by_type_pd["avg_duration_minutes"],
color=['#FF6B6B', '#4ECDC4', '#45B7D1', '#96CEB4'])
plt.xlabel("Transportation Type", fontsize=12)
plt.ylabel("Average Trip Duration (Minutes)", fontsize=12)
plt.title("Average Trip Duration by Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.xticks(rotation=45, ha='right')
plt.grid(axis='y', linestyle='--', alpha=0.7)
# Add value labels on bars
for bar, value in zip(bars, duration_by_type_pd["avg_duration_minutes"]):
plt.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.2,
f'{value:.1f}', ha='center', va='bottom', fontweight='bold')
plt.tight_layout()
plt.show()
# Log duration insights
logging.info("=== TRIP DURATION INSIGHTS ===")
for _, row in duration_by_type_pd.iterrows():
logging.info(f"{row['trip_type']}: Average duration {row['avg_duration_minutes']:.1f} minutes")
```
#### Analysis 3: Top Pickup Zones by Transportation Type
```python
# Calculate top pickup zones for each transportation type
zone_trips_by_type = combined_df.groupBy("trip_type", "pickup_zone") \
.agg(count("*").alias("num_trips")) \
.orderBy("trip_type", "num_trips", ascending=[True, False])
# Select top 5 zones per transportation type using window function
windowSpec = Window.partitionBy("trip_type").orderBy(col("num_trips").desc())
zone_trips_top5 = zone_trips_by_type.withColumn("rank", row_number().over(windowSpec)) \
.filter(col("rank") <= 5) \
.drop("rank")
# Convert to Pandas for visualization
zone_trips_top5_pd = zone_trips_top5.toPandas()
# Create comprehensive visualization
plt.figure(figsize=(16, 10))
sns.barplot(data=zone_trips_top5_pd, x="num_trips", y="pickup_zone", hue="trip_type", palette="Set3")
plt.xlabel("Number of Trips", fontsize=12)
plt.ylabel("Pickup Zone", fontsize=12)
plt.title("Top 5 Pickup Zones by Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.legend(title="Transportation Type", bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(axis='x', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Log top zones for each service
logging.info("=== TOP PICKUP ZONES ===")
for trip_type in zone_trips_top5_pd["trip_type"].unique():
top_zones = zone_trips_top5_pd[zone_trips_top5_pd["trip_type"] == trip_type].head(3)
logging.info(f"\n{trip_type} - Top 3 zones:")
for _, zone in top_zones.iterrows():
logging.info(f" {zone['pickup_zone']}: {zone['num_trips']:,} trips")
```
#### Analysis 4: Hourly Usage Patterns
```python
# Analyze hourly usage patterns by transportation type
hourly_usage = combined_df.groupBy("trip_type", "pickup_hour") \
.agg(count("*").alias("num_trips")) \
.orderBy("trip_type", "pickup_hour")
# Convert to Pandas
hourly_usage_pd = hourly_usage.toPandas()
# Create line plot showing hourly patterns
plt.figure(figsize=(14, 8))
for trip_type in hourly_usage_pd["trip_type"].unique():
data = hourly_usage_pd[hourly_usage_pd["trip_type"] == trip_type]
plt.plot(data["pickup_hour"], data["num_trips"], marker='o', linewidth=2, label=trip_type)
plt.xlabel("Hour of Day", fontsize=12)
plt.ylabel("Number of Trips", fontsize=12)
plt.title("Hourly Usage Patterns by Transportation Type (January 2025)", fontsize=14, fontweight='bold')
plt.legend(title="Transportation Type")
plt.grid(True, linestyle='--', alpha=0.7)
plt.xticks(range(0, 24))
plt.tight_layout()
plt.show()
# Identify peak hours for each service
logging.info("=== PEAK HOUR ANALYSIS ===")
for trip_type in hourly_usage_pd["trip_type"].unique():
data = hourly_usage_pd[hourly_usage_pd["trip_type"] == trip_type]
peak_hour = data.loc[data["num_trips"].idxmax()]
logging.info(f"{trip_type}: Peak hour is {peak_hour['pickup_hour']}:00 with {peak_hour['num_trips']:,} trips")
```
## Key Findings and Insights
### Market Share Analysis
**Manhattan Dominance:**
- **Yellow Taxis**: Maintain strong presence in Manhattan, particularly in Midtown areas
- **High Volume FHV**: Significant market penetration across all boroughs, with highest absolute numbers
**Outer Borough Patterns:**
- **Green Taxis**: Concentrated in Brooklyn and Queens as intended by regulation
- **FHV Services**: Provide important connectivity to areas less served by traditional taxis
### Service Characteristics
**Trip Duration Patterns:**
- **Shortest**: Green Taxis (optimized for local trips in outer boroughs)
- **Longest**: FHV services (often include airport and longer-distance trips)
- **Medium**: Yellow Taxis and High Volume FHV (balanced mix of trip types)
**Peak Usage Times:**
- **Morning Rush**: 8-9 AM across all services
- **Evening Rush**: 6-7 PM with variation by service type
- **Late Night**: High Volume FHV maintains stronger presence than traditional taxis
### Geographic Insights
**High-Demand Zones:**
- **Airport Access**: JFK and LaGuardia dominate FHV pickup zones
- **Transit Hubs**: Penn Station, Grand Central prominent across services
- **Business Districts**: Midtown Manhattan remains critical for Yellow Taxis
## Conclusion
This comparative analysis reveals distinct usage patterns and market positioning across NYC's transportation ecosystem. **High Volume FHV services** dominate by sheer volume, while **Yellow Taxis** maintain their traditional Manhattan stronghold. **Green Taxis** successfully serve outer borough markets, and **FHV services** provide crucial connectivity for longer trips and airport access.
The analysis demonstrates how different transportation modes complement each other, serving distinct geographic areas, trip purposes, and time patterns. This insight is valuable for:
- **Urban Planners**: Understanding transportation demand patterns
- **Service Operators**: Optimizing fleet deployment and pricing strategies
- **Policy Makers**: Evaluating the effectiveness of transportation regulations
- **Researchers**: Analyzing the evolution of urban mobility
The PySpark-based approach showcases how large-scale transportation data can be efficiently processed and analyzed to extract meaningful insights for data-driven decision making in urban transportation planning.
## Next Steps
To extend this analysis further, consider:
- **Temporal Analysis**: Comparing patterns across different months or seasons
- **Demand Forecasting**: Building predictive models for trip volume by zone and time
- **Network Analysis**: Examining origin-destination patterns and flow dynamics
- **Economic Analysis**: Incorporating fare and revenue data for financial insights
- **Weather Impact**: Analyzing how weather conditions affect different transportation modes
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-pyspark-cheat-sheet.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PySpark Cheat Sheet: Essential and Advanced Functions on OVHcloud Data Platform
## Objective
This cheat sheet provides a concise guide to essential and advanced PySpark functions for data processing on the **OVHcloud Data Platform**. It uses the NYC Yellow Taxi Trip Records for January 2025 (`yellow_tripdata_2025_01.parquet`, \~3.5M records) and the Taxi Zone Lookup Table (`taxi_zone_lookup.csv`, 265 records) as practical examples.
Aimed at intermediate users, it covers core functions (`filter`, `select`, `groupBy`, `join`, `udf`) and advanced ones (`window`, `pivot`, `approx_count_distinct`, `collect_list`, `explode`, `regexp_replace`) for complex transformations and analytics. These examples demonstrate practical applications, making this a versatile reference for any dataset.
## Requirements
Before diving in, ensure you have:
- **Datasets**: (`yellow_tripdata_2025_01.parquet`, `taxi_zone_lookup.csv`) available in your Connectors and accessible in the Lakehouse Manager. You can download these from the [official NYC TLC Trip Record Data website](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
- **Notebook**: A PySpark-enabled Jupyter notebook.
## Setup Instructions
1. **Connectors**: Create a source named "NYC-taxi", upload your files (`yellow_tripdata_2025_01.parquet`, `taxi_zone_lookup.csv`), and extract their schemas.
2. **Lakehouse Manager**: Create corresponding tables in the Lakehouse Manager.
3. **DPE (Data Processing Engine)**: Ensure the data is loaded into these tables within the DPE.
4. **Notebook**: Start a PySpark Jupyter notebook within the OVHcloud Data Platform.
## PySpark Functions Cheat Sheet
### Step 1: Initializing Spark and Loading Data
**Code Block**
```python
from forepaas.dwh import connect
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, unix_timestamp, hour, when, count, avg, udf, approx_count_distinct, collect_list, explode, regexp_replace, row_number
from pyspark.sql.types import FloatType
from pyspark.sql.window import Window
from forepaas.dwh.common import request as dwh_request, DwhRequestException
import io
import logging
# Setup logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Set up variables (DATASET, PROJECT_ID, YEAR, MONTH)
DATASET = "default_dataset"
PROJECT_ID = "PROJECT_ID" # Make sure to replace this with your actual PROJECT_ID
YEAR = "2025"
MONTH = "01"
# Initialize SparkSession
spark = SparkSession.builder.appName("PySpark_Advanced_Cheat_Sheet").getOrCreate()
logging.info(f"Spark Version: {spark.version}")
# Connect to Lakehouse
cn_prim = connect("dwh/default_dataset/")
# Load data
taxi_df = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.yellow_tripdata_{YEAR}_{MONTH}")
zones_df = cn_prim.query(f"SELECT LocationID, Borough, Zone FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.taxi_zone_lookup")
taxi_df.cache()
zones_df.cache()
logging.info(f"Taxi Records: {taxi_df.count()}")
logging.info(f"Zones Records: {zones_df.count()}")
```
**Functions Used in this Code Block:**
- `SparkSession.builder.appName().getOrCreate()`:
- **What**: Initializes the PySpark session and DataFrame API, which is the entry point for using Spark functionalities.
- **Benefit**: Sets up the environment for distributed data processing.
- `cn_prim = connect("dwh/default_dataset/")`:
- **What**: Establishes a connection to your specified Lakehouse dataset using the `forepaas.dwh` library.
- **Benefit**: Allows you to query and interact with tables stored in your Lakehouse.
- `cn_prim.query(sql_query)`:
- **What**: Executes an SQL query against tables accessible via the `cn_prim` connection. It retrieves the data as a Spark DataFrame.
- **Benefit**: Provides a simple way to load data from your Connectors and Lakehouse into a Spark DataFrame.
- `DataFrame.cache()`:
- **What**: Marks the DataFrame to be cached in memory the first time it's computed. Subsequent actions on this DataFrame will read from cache.
- **Benefit**: Significantly speeds up operations on the same DataFrame, especially useful for iterative algorithms or multiple transformations on a large dataset (\~3.5M records).
- `DataFrame.count()`:
- **What**: Triggers the computation and returns the total number of rows in the DataFrame.
- **Benefit**: Used here to quickly verify that the data has been loaded and to check the record count.
**Output of this Code Block:**
- Spark Version (e.g., `Spark Version: 3.4.1`)
- `Taxi Records: ~3,500,000` (actual count may vary slightly)
- `Zones Records: 265`
### Step 2: Clean and Transform Data
**Code Block**
```python
# Clean and transform
cleaned_df = taxi_df \
.filter(
(col("tpep_pickup_datetime").isNotNull()) &
(col("fare_amount") > 0) &
(col("trip_distance") > 0)
) \
.select(
"tpep_pickup_datetime",
"tpep_dropoff_datetime",
"trip_distance",
"fare_amount",
col("pulocationid").cast("double").alias("pulocationid"),
regexp_replace(col("store_and_fwd_flag"), "^[Yy]$", "Yes").alias("store_and_fwd_flag")
) \
.withColumn(
"trip_duration",
unix_timestamp("tpep_dropoff_datetime") - unix_timestamp("tpep_pickup_datetime")
) \
.withColumn(
"pickup_hour",
hour("tpep_pickup_datetime")
) \
.withColumn(
"trip_duration",
when(col("trip_duration") > 3600, 3600).otherwise(col("trip_duration"))
) \
.filter(col("trip_duration") >= 60)
logging.info(f"Cleaned Records: {cleaned_df.count()}")
```
**Functions Used in this Code Block:**
- `DataFrame.filter(condition)`:
- **What**: Filters rows of the DataFrame based on a given condition, returning a new DataFrame with only the rows that satisfy the condition.
- **Benefit**: Essential for data cleansing, removing invalid or irrelevant records.
- `pyspark.sql.functions.col(column_name)`:
- **What**: References a column in a DataFrame, allowing you to apply various transformations and operations to it.
- **Benefit**: Provides a way to build expressions involving DataFrame columns.
- `DataFrame.select(columns)`:
- **What**: Projects a set of expressions (columns or column-based transformations) and returns a new DataFrame containing only these selected columns.
- **Benefit**: Useful for column subsetting and reducing the DataFrame's size by selecting only relevant fields.
- `Column.cast(dataType)`:
- **What**: Converts the data type of a column to the specified `dataType`.
- **Benefit**: Ensures data type compatibility for calculations or downstream processes (e.g., casting `pulocationid` to `double`).
- `pyspark.sql.functions.regexp_replace(column, pattern, replacement)`:
- **What**: Replaces all occurrences of a string pattern within a column's text data with a specified replacement string.
- **Benefit**: Excellent for data standardization and cleaning string fields (e.g., converting "Y" to "Yes").
- `DataFrame.withColumn(colName, col)`:
- **What**: Returns a new DataFrame by adding a new column or replacing an existing column with the specified expression.
- **Benefit**: Key for feature engineering and creating new derived columns on the fly.
- `pyspark.sql.functions.unix_timestamp(timestamp_column)`:
- **What**: Converts a timestamp string or timestamp column into Unix timestamp (seconds since 1970-01-01 00:00:00 UTC).
- **Benefit**: Facilitates numerical time calculations, such as computing trip durations by subtracting timestamps.
- `pyspark.sql.functions.hour(timestamp_column)`:
- **What**: Extracts the hour component from a timestamp column.
- **Benefit**: Useful for temporal analysis, allowing you to identify hourly patterns or trends in the data.
- `pyspark.sql.functions.when(condition, value).otherwise(other_value)`:
- **What**: Implements conditional logic. If the `condition` is true, the column takes `value`; otherwise, it takes `other_value`. Can be chained.
- **Benefit**: Effective for handling outliers, applying business rules, or categorizing data based on specific conditions (e.g., capping `trip_duration`).
**Output of this Code Block:**
- `Cleaned Records: ~2,700,000 – ~2,800,000` (actual count may vary based on data quality).
### Step 3: Custom Logic with UDF (User-Defined Function)
**Code Block**
```python
# UDF for fare efficiency (fare per minute)
def fare_efficiency(fare, duration):
return fare / (duration / 60) if duration > 0 else 0.0
fare_efficiency_udf = udf(fare_efficiency, FloatType())
# Apply UDF
transformed_df = cleaned_df \
.withColumn("fare_efficiency", fare_efficiency_udf(col("fare_amount"), col("trip_duration")))
transformed_df.show(5)
```
**Functions Used in this Code Block:**
- `udf(func, returnType)`:
- **What**: Registers a Python function as a User-Defined Function (UDF) in PySpark. This allows you to apply custom Python logic to Spark DataFrame columns.
- **Benefit**: Enables tailored, complex calculations that might not be available in native PySpark functions (e.g., calculating "fare per minute" using custom logic).
- `FloatType()` (from `pyspark.sql.types`):
- **What**: Specifies the return data type of the UDF as a Float (single-precision floating-point number).
- **Benefit**: Ensures that the output of your custom function is correctly typed in the Spark DataFrame.
- `DataFrame.withColumn(colName, col)`:
- **What**: (Re-used from Step 2) Adds a new column or replaces an existing one based on the result of an expression.
- **Benefit**: Here, it's used to apply the newly defined `fare_efficiency_udf` to create the `fare_efficiency` column.
**Output of this Code Block:**
- A `DataFrame.show(5)` output, displaying the first 5 rows of `transformed_df`, including the newly added `fare_efficiency` column.
- Example: If `fare_amount` is 15.0 and `trip_duration` is 600 seconds (10 minutes), `fare_efficiency` would be 1.5 ($/min).
### Step 4: Advanced Window Functions and Joins
**Code Block**
```python
# Define window for ranking trips by fare within borough
window_spec = Window.partitionBy("pickup_borough").orderBy(col("fare_amount").desc())
# Join with zones and rank trips
joined_df = transformed_df \
.join(
zones_df,
transformed_df.pulocationid == zones_df.LocationID,
"left"
) \
.withColumnRenamed("Borough", "pickup_borough") \
.drop("LocationID") \
.filter(col("pickup_borough").isNotNull()) \
.withColumn("fare_rank", row_number().over(window_spec))
logging.info(f"Joined Records: {joined_df.count()}")
joined_df.filter(col("fare_rank") <= 3).show()
```
**Functions Used in this Code Block:**
- `Window.partitionBy(*cols).orderBy(*cols)`:
- **What**: Defines a window specification. `partitionBy` divides the rows into groups, and `orderBy` defines the logical order of rows within each partition.
- **Benefit**: Crucial for enabling advanced analytical operations (like ranking, lead/lag, cumulative sums) that operate on a defined subset of rows.
- `pyspark.sql.functions.row_number()`:
- **What**: A window function that assigns a unique, sequential number to each row within its partition, based on the ordering defined in the window specification.
- **Benefit**: Perfect for ranking records (e.g., identifying the top N records based on a metric, like top fares).
- `DataFrame.join(other_df, on=None, how=None)`:
- **What**: Combines two DataFrames based on a specified join condition (`on`) and join type (`how`, e.g., "inner", "left", "right").
- **Benefit**: Enriches data by bringing together related information from different sources (e.g., joining taxi trip data with zone lookup data).
- `DataFrame.withColumnRenamed(existing, new)`:
- **What**: Returns a new DataFrame by renaming an existing column.
- **Benefit**: Helps clarify schema and improves readability, especially after joins where column names might be ambiguous.
- `DataFrame.drop(*cols)`:
- **What**: Returns a new DataFrame with the specified columns dropped.
- **Benefit**: Helps manage DataFrame size and complexity by removing unnecessary columns, saving memory.
**Output of this Code Block:**
- `Joined Records: ~2,600,000 – ~2,700,000` (actual count may vary).
- A `DataFrame.show()` output, displaying rows where `fare_rank` is less than or equal to 3, showing the top 3 fares per borough.
### Step 5: Aggregate and Pivot Data
**Code Block**
```python
# Aggregate: unique zones and trips per borough
agg_df = joined_df \
.groupBy("pickup_borough") \
.agg(
approx_count_distinct("pulocationid").alias("unique_zones"),
count("*").alias("num_trips")
)
# Pivot: avg fare by hour and borough
pivot_df = joined_df \
.groupBy("pickup_hour") \
.pivot("pickup_borough") \
.agg(avg("fare_amount")) \
.orderBy("pickup_hour")
agg_df.show()
pivot_df.show()
```
**Functions Used in this Code Block:**
- `DataFrame.groupBy(*cols)`:
- **What**: Groups the DataFrame by one or more specified columns, preparing for aggregate computations.
- **Benefit**: Enables summarization and analysis of data based on distinct categories or dimensions.
- `DataFrame.agg(*exprs)`:
- **What**: Applies aggregate functions to the grouped data, computing summary statistics.
- **Benefit**: Used to calculate metrics like counts, sums, averages, etc., for each group.
- `pyspark.sql.functions.approx_count_distinct(column)`:
- **What**: Returns an approximate count of distinct items in a group. It uses the HyperLogLog++ algorithm.
- **Benefit**: Significantly faster and more memory-efficient than `countDistinct` for very large datasets when an exact count isn't strictly necessary.
- `pyspark.sql.functions.count(column)`:
- **What**: Counts the number of non-null values in a column or, with `count("*")`, counts all rows in a group.
- **Benefit**: Tallies the size of each aggregated group or the occurrences of specific values.
- `DataFrame.pivot(pivot_column)`:
- **What**: Rotates (pivots) a DataFrame, transforming unique values from a specified column into new columns. Requires a subsequent aggregation.
- **Benefit**: Creates wide tables that are often more suitable for reporting and cross-sectional analysis, allowing direct comparison of values across categories.
- `pyspark.sql.functions.avg(column)`:
- **What**: Computes the average value of a numerical column.
- **Benefit**: Provides a central tendency measure for quantitative data within each group.
- `DataFrame.orderBy(*cols, ascending=True)`:
- **What**: Sorts the rows of the DataFrame based on one or more columns in ascending or descending order.
- **Benefit**: Organizes the output for better readability and to present data in a logical sequence.
**Output of this Code Block:**
- `agg_df.show()`: Displays a table with `pickup_borough`, `unique_zones` (approximate), and `num_trips` (e.g., Manhattan might show \~60 unique zones and \~2M trips).
- `pivot_df.show()`: Displays a pivoted table showing `pickup_hour` as rows and `pickup_borough` as columns, with the average `fare_amount` in each cell.
### Step 6: Collect and Explode Lists
**Code Block**
```python
# Collect zones per borough
list_df = joined_df \
.groupBy("pickup_borough") \
.agg(collect_list("Zone").alias("zones_list"))
# Explode zones list
exploded_df = list_df \
.select("pickup_borough", explode(col("zones_list")).alias("zone"))
exploded_df.show(10)
```
**Functions Used in this Code Block:**
- `pyspark.sql.functions.collect_list(column)`:
- **What**: An aggregation function that gathers all non-null values from a specified column within each group into a Python list.
- **Benefit**: Useful for creating array-like structures where each element corresponds to a record from the original group.
- `pyspark.sql.functions.explode(array_column)`:
- **What**: Transforms a column containing arrays (lists) or maps into individual rows for each element in the array/map. If an array has `N` elements, it creates `N` rows for that original row.
- **Benefit**: Flattens nested data structures, allowing individual elements to be processed or viewed as separate records.
**Output of this Code Block:**
- A `DataFrame.show(10)` output, displaying a table with `pickup_borough` and an `exploded` `zone` column, where each distinct zone for a borough gets its own row (e.g., for Manhattan, you'd see multiple rows like "Manhattan | Midtown", "Manhattan | Upper East Side", etc.).
### Step 7: Save DataFrame to Bucket and Create Table (Advanced)
**Code Block**
```python
def create_table_from_this_dataframe(dataframe, dataset, table_name, bucket, source_bucket):
logging.info(f"We will create a source (bucket) - {source_bucket} where we will store the new table - {table_name} - and automatically load it")
# Creating bucket to store the table
cn_datastore = connect('data_store')
logging.info(f"{cn_datastore.list()} - Before creating new bucket")
cn_datastore.create_bucket(bucket)
logging.info(f"{cn_datastore.list()} - After adding new bucket")
cn_bucket = connect('data_store/' + bucket)
get_dbs = dwh_request(f"v4/databases", method="GET")
data = get_dbs.json()
db_exist = next((item['_id'] for item in data if item.get('display_name') == source_bucket and item.get("package") == "data-store"), None)
if db_exist is None:
# Creating source where we will use the new bucket created to get access to the table
new_source = {"type":"protocol","package":"data-store","parameters":{"path":"","bucket":bucket},"default":False,"level":"source","display_name":source_bucket}
new_source_bucket = dwh_request(f"v4/databases", method="POST", json=new_source)
logging.info(f"New source added with the bucket: {bucket} - source name: {source_bucket}")
else:
logging.info("Source already exist")
# Call to API - To get dataset id
get_database_id = dwh_request(f"v4/databases", method="GET")
database_all = get_database_id.json()
# Filter to get the corresponding _id for the database
dataset_id = next((item['_id'] for item in database_all if item.get('name') == dataset), None)
logging.info(f"dataset_id : {dataset_id}")
# Convert table to Pandas and serialize to CSV in BytesIO
try:
# Convert to Pandas DataFrame
table = dataframe.toPandas()
# Create BytesIO buffer and write CSV
data = io.BytesIO()
table.to_csv(data, index=False, encoding='utf-8')
data.seek(0) # Reset buffer position
# Upload to bucket
file_path = f"{table_name}.csv"
etag = cn_bucket.put(file_path, data, data.getbuffer().nbytes)
logging.info(f"DataFrame uploaded to bucket {bucket}/{file_path} with ETag: {etag}")
# Verify bucket contents
files = cn_bucket.list()
logging.info(f"Bucket contents: {files}")
except Exception as e:
logging.error(f"Failed to save to bucket: {e}")
raise
# Call to API - To add the file into the source
table_config = {"display_name":file_path,"progress":None,"physical_status":None,"parameters":{},"filename":file_path}
res_table = dwh_request(f"v4/databases/{source_bucket}/tables/{file_path}", method="PUT", json=table_config)
# Call to API - To get the corresponding ID for the file_path added in the source
get_template_catalog_object = dwh_request(f"v4/tables", method="GET")
data = get_template_catalog_object.json()
file_path_source_id = next((item['_id'] for item in data if item.get('filename') == file_path), None)
logging.info(f"file_path_source_id : {file_path_source_id}")
# In case the table already exist and you made some modification on it
auto_build_table_DELETE = dwh_request(f"v4/logical/objects/{table_name}", method="DELETE")
# Call to API - To launch the build of the table on the corresponding dataset and load the specific data
config_build = {"database_id":dataset_id,"display_name":table_name,"name":table_name,"type":"prim","load_data":True,"build_table":True,"templated_from":"data_catalog","template_catalog_object":file_path_source_id}
auto_build_table = dwh_request(f"v4/logical/objects", method="POST", json=config_build)
logging.info(f"You can check the build of the table {table_name} on the Lakehouse Manager screen")
# Use the function create_table_from_this_dataframe:
create_table_from_this_dataframe(pivot_df,"default_dataset","taxi_pivot_table","new_bucket","new_source_bucket")
```
**Functions Used in this Code Block (and within `create_table_from_this_dataframe`):**
- `DataFrame.toPandas()`:
- **What**: Converts a Spark DataFrame to a Pandas DataFrame. This collects all distributed data to the driver node.
- **Benefit**: Enables the use of Pandas-specific functions for local data manipulation and file serialization (e.g., `to_csv`).
- **Warning**: Use with caution on very large datasets as it can cause out-of-memory errors on the driver.
- `Pandas_DataFrame.to_csv(path_or_buffer, index=False, encoding='utf-8')`:
- **What**: Writes the Pandas DataFrame to a comma-separated values (CSV) file.
- **Benefit**: Serializes the data into a standard text format suitable for storage and retrieval.
- `io.BytesIO()`:
- **What**: A class from Python's `io` module that creates an in-memory binary stream, acting like a file object.
- **Benefit**: Allows you to write and read bytes as if you were interacting with a physical file, which is useful for direct data transfer to services without saving to disk.
- `cn_datastore.create_bucket(bucket_name)` (from `forepaas.dwh`):
- **What**: Creates a new storage bucket within the OVHcloud Data Platform's data store.
- **Benefit**: Provides a dedicated location to store files, including intermediate or final processed data.
- `cn_bucket.put(file_path, data, size)` (from `forepaas.dwh`):
- **What**: Uploads data (typically from a BytesIO buffer) to a specified path within a connected bucket.
- **Benefit**: Persists your processed data (e.g., the CSV from `pivot_df`) into OVHcloud cloud storage.
- `cn_bucket.list()` (from `forepaas.dwh`):
- **What**: Retrieves a list of files and subdirectories within a connected bucket.
- **Benefit**: Used for verification that files have been successfully uploaded to the bucket.
- `dwh_request(path, method, json)` (from `forepaas.dwh.common`):
- **What**: A utility function to make direct HTTP API calls to the OVHcloud Data Platform's backend services.
- **Benefit**: This function provides granular control to automate tasks like creating data sources, registering files in the Connectors, and triggering table builds in the Lakehouse Manager programmatically, which are not always exposed via the higher-level `connect` object.
**Explanation of this Step's Process:**
This advanced step showcases how to persist a processed Spark DataFrame (`pivot_df`) into a new table in the OVHcloud Data Platform's Lakehouse. It leverages a combination of PySpark, Pandas, and direct API calls:
1. **Bucket Creation**: A new bucket (`new_bucket`) is created in the OVHcloud data store if it doesn't already exist.
2. **Source Creation**: A new data source (`new_source_bucket`) is configured and linked to this bucket, allowing the Connectors to discover files within it.
3. **Data Serialization & Upload**: The `pivot_df` is converted to a Pandas DataFrame, then serialized into a CSV format within an in-memory buffer (`io.BytesIO`). This CSV data is then uploaded to the newly created bucket.
4. **Table Registration & Build**: Via direct API calls (`dwh_request`), the uploaded CSV file is registered as a "physical table" in the Connectors, and a "logical table" (`taxi_pivot_table`) is created in the Lakehouse Manager, triggering a data load from the registered source.
:::warning
**Warning**: This is a temporary solution for saving and creating tables, relying on direct API interactions. The OVHcloud Data Platform SDK is expected to be updated to include more native and simplified SQL transformation and direct storage capabilities in the future, reducing the need for manual API calls and Pandas conversions.
:::
### Step 8: Verify Table Existence
**Code Block**
```python
# Verify table exists in Lakehouse
try:
test_result = cn_prim.query(f"SELECT * FROM db_{PROJECT_ID}_{DATASET}.{DATASET}.taxi_pivot_table")
logging.info(f"taxi_pivot_table Records: {test_result.count()}")
except Exception as e:
logging.error(f"Failed to query table - please ensure it's the correct table")
```
**Functions Used in this Code Block:**
- `cn_prim.query(sql_query)`:
- **What**: (Re-used from Step 1) Executes an SQL query against tables in your Lakehouse through the established `cn_prim` connection, retrieving the result as a Spark DataFrame.
- **Benefit**: Used here to confirm that the `taxi_pivot_table` was successfully created and is queryable in the Lakehouse.
- `DataFrame.count()`:
- **What**: (Re-used from Step 1) Returns the total number of rows in the queried DataFrame.
- **Benefit**: Confirms that the newly created table contains the expected data and records.
**Output of this Code Block:**
- If successful: `taxi_pivot_table Records: [Number of rows in pivot_df]` (e.g., `taxi_pivot_table Records: 24` if 24 hours are present).
- If unsuccessful: An error message from the `logging.error` indicating that the table could not be queried.
### Step 9: Clean Up and Stop Spark Session
**Code Block**
```python
spark.catalog.clearCache()
spark.stop()
logging.info("Spark cache cleared and Spark session stopped.")
```
**Functions Used in this Code Block:**
- `SparkSession.catalog.clearCache()`:
- **What**: Clears the Spark's internal cache, releasing memory occupied by cached DataFrames and RDDs.
- **Benefit**: Essential for freeing up resources, especially after complex operations or when you no longer need the cached data. It helps prevent out-of-memory issues in long-running applications or interactive sessions.
- `SparkSession.stop()`:
- **What**: Terminates the SparkSession, shutting down the SparkContext and releasing all associated resources.
- **Benefit**: Ensures a clean shutdown of your Spark application, freeing up cluster resources. It's crucial to call this at the end of your Spark application to avoid resource leaks.
**Output of this Code Block:**
- `INFO - Spark cache cleared and Spark session stopped.`
- This message confirms that the cache has been cleared and the Spark session has been gracefully terminated. You will typically see further output from the notebook environment indicating that the kernel is idle or has shut down.
## Conclusion and Next Steps
This PySpark cheat sheet has provided a hands-on overview of essential and advanced functions for data processing on the OVHcloud Data Platform. You've seen how to initialize Spark, load and clean data, apply custom logic with UDFs, perform complex aggregations with window functions and pivots, and even manage data persistence to your Lakehouse.
This guide serves as a practical reference to accelerate your PySpark development. To continue your journey and apply these skills to real-world scenarios, we encourage you to:
- **Explore the Tutorials**: Dive into the comprehensive [NYC Taxi Dataset Analysis tutorials](https://docs.dataplatform.ovh.net/tutorials-pyspark.md) to build end-to-end data pipelines and machine learning models.
- **Deepen Your Knowledge**: Consult the [official PySpark Documentation](https://spark.apache.org/docs/latest/api/python/) for in-depth details on all functions.
Happy coding and data analyzing!
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-ai-endpoints.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Enhance Your Data Analysis with AI Endpoints on the Data Platform
## Objective
No more worries about imperfect datasets: import, clean, and visualize your data with the **Data Platform**. But that's not all! Benefit from detailed analysis thanks to AI and go even further by asking your LLM to find a Machine Learning rule that fits perfectly with your data.
This guide will demonstrate how to take advantage of [**AI Endpoints**](https://endpoints.ai.cloud.ovh.net/) to enhance your data analysis capabilities on the Data Platform. You'll learn how to start from scratch and generate meaningful predictions from your data.
Here you can find all the **resources** used (CSV files, Python scripts, etc.) for the purpose of this guide. Please download all required files before you begin:
| **Type** | **Name** |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Source file | 
3. **Then, create the "Delete" action as follows:**
- Configure the delete action settings according to your requirements (this action will ensure you are using a clean table every time)
4. **Finally, define the processing workflow:**
- Set up your workflow to connect the delete, custom action and aggregate action
Your dataset is now clean.
### step 8 - Explore clean tables
Now, you can check if everything's okay:
1. **From the Lakehouse Manager Tables**, you should see your created tables
2. **From the Lakehouse Manager [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md)**, you can explore the `clean_cafe_sales` table
### step 9 - Data visualization using custom attributes
Now we can take an even more closer look at our data visually with the help of [queries](https://docs.dataplatform.ovh.net/analytics-manager-queries.md) in the [Analytics Manager](https://docs.dataplatform.ovh.net/landing-page-analytics-manager.md)
1. **Create quantity\_per\_item query**
- You should see the average quantity sold per day for each item
2. **Create total\_spent\_per\_item query**
- You could see the average total amount sold per day for each item
3. **Create sales\_per\_location query**
- You could see the average total amount sold per day for each location
You can go further by creating more queries and then add them to a Dashboard which you can then share with others.
## Option 2: Step-by-Step Processing with LLM
### step 10 - Clean dataset and save it into S31
bucket
If you want to follow the data cleaning process step by step, you can create a new notebook from the Data Processing service.
1. **Create a "Base notebook" with:**
- Python 3.9 version
- Install `numpy`, `pandas`, `matplotlib`, and `seaborn` dependencies
2. **Drag and drop the or click upload, to upload the file `cafe_sales_preprocessing.ipynb` into the jupyterhub file siebar**
### step 11 - One-shot data analysis with LLM AI Endpoints
A single request is enough to generate all the Python code you need for dataset analysis!
1. **Use the Llama 3.3 70B Instruct model** with the Python code that will allow you to analyze your dataset easily
2. **Generate an AI Endpoints API key** and use it in the designated line as seen below in the screenshot
- Access [here](https://endpoints.ai.cloud.ovh.net/)
3. **Use the generated Python code** to analyze your data and display graphs
The advantage of asking an LLM to generate code in Python is that you will be able to reuse it when you add new data to your coffee sales in subsequent months, ensuring a consistent analysis method.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-billing-analysis.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# OVHcloud API: Billing Data Analysis Configuration
## Objective
This guide provides step-by-step instructions on configuring multiple endpoints for the OVHcloud Billing API. You'll learn how to retrieve billing data and integrate it into your applications for accurate and streamlined financial tracking.
## Overview
The OVHcloud API supports multiple endpoint configurations, allowing you to fetch and integrate data from interconnected resources. This guide provides a comprehensive walkthrough for configuring and utilizing the OVHcloud Billing API endpoints. By following these steps, you'll be able to programmatically retrieve, process, and analyze your billing information efficiently.
## Requirements
Before beginning, ensure you have:
- An active OVHcloud account with billing access
- API credentials, specifically:
- Application Key (App Key)
- Application Secret (App Secret)
- Consumer Key
- Access to the OVHcloud API Developer Portal
- Basic understanding of API endpoint interactions
## Step-by-Step Billing Endpoint Configuration
## 1. Initialize API Connection
Configure your OVHcloud API connection by following the authentication setup process detailed in the OVHcloud API Connector Documentation. This involves obtaining and securely inputting your Application Key, Application Secret, and Consumer Key through the connector's user interface. For comprehensive instructions on acquiring and setting up these credentials, consult the [API Connection Details](https://docs.dataplatform.ovh.net/connectors-sources-ovhcloud-api.md) section in the connector documentation.
## 2. Define the billing endpoints
### 2.1. Retrieve Account Bills IDs
- Endpoint: `/me/bill`
- Purpose: Fetch a list of all bills IDs for your account
Steps:
1. Select "Add New Endpoint" in the connector
2. Enter `/me/bill` as the endpoint path
3. Configure query parameters (e.g., filtering by year) - for this example we have used key:`date.from` and value:`2023`, to get the records from the year 2023.
4. Make sure that endpoint has a unique name `bills_ids_from_2023`
5. Click "Confirm" and "Save"
:::info
We can include an additional step to confirm whether the connected endpoint is correct. While this step is not mandatory, it can be helpful the first time to get an idea of the data you are looking for.
:::
#### Validate Endpoint Data
1. Navigate to the Analyzer tab
2. Extract metadata for the new endpoint
3. Verify bill records are correctly retrieved
### 2.2. Retrieve Account Bills Details IDs
- Endpoint: `/me/bill/{billId}/details`
- Purpose: Fetch the list of IDs for each invoice line details
Steps:
1. Click "Add Endpoint"
2. Enter `/me/bill/{billId}/details` as the endpoint path
3. Configure multiple endpoint selection
4. Link to the previous bills endpoint
5. Select `{billId}` as the path parameter
6. Make sure that endpoint has a unique name `bills_details_ids_from_2023`
7. Save the configuration
:::info
Use unique endpoint names to simplify source management. This will be useful for step 4. Import your data in the lakehouse.
:::
:::warning
**Note**: Currently, only one path parameter can be dynamically selected. Manual entry might be required for complex endpoints with multiple path parameters.
:::
Returning to the main modal by clicking on the Settings tab, you'll notice that the path parameter and endpoint fields are now non-editable. This is because their values are being derived from another endpoint. If you need to make changes, simply go back to the multiple endpoints section and select 'No multiple endpoint' from the endpoint selection dropdown.
### 2.3. Retrieve Account Bills Details
- Endpoint: `/me/bill/{billId}/details/{detailsID}`
- Purpose: Fetch the details of each invoice line
Steps:
1. Click "Add Endpoint"
2. Enter `/me/bill/{billId}/details/{detailsID}` as the endpoint path
3. Configure multiple endpoint selection
4. Link to the previous bills details ID endpoint
5. Select `{billId}` and `{detailsID}` as the path parameter
6. Make sure that endpoint has a unique name `bills_details_from_2023`
7. Save the configuration
### 2.4. Retrieve services IDs
- Endpoint: `/services`
- Purpose: Fetch the IDs of all the services of your accounts
Steps:
1. Click "Add Endpoint"
2. Enter `/services` as the endpoint path
3. Make sure that endpoint has a unique name `services_ids`
4. Save the configuration
### 2.5. Retrieve services information
- Endpoint: `/services/{serviceId}`
- Purpose: Fetch information about your services to enrich the data in the pipeline
Steps:
1. Click "Add Endpoint"
2. Enter `/services/{serviceId}` as the endpoint path
3. Configure multiple endpoint selection
4. Link to the previous service IDs endpoint
5. Select `{serviceId}` as the path parameter
6. Make sure that endpoint has a unique name `services`
7. Save the configuration
The configuration of the endpoints is now over, you should see the following:
## 3. Metadata extraction
With the endpoints now established, navigate to the analyzer tool within the Connectors. Once there, select the endpoints you have recently configured. Allow some time for the system to initiate and complete the processing jobs associated with these endpoints.
:::warning
The duration of this process may vary depending on the complexity of the tasks and the volume of data being processed.
:::
## 3.1. Reformat data
The data extracted from the sources can be modified to match the exact format we want, or to bucketize it through the blueprints. This is what we are going to do for 2 endpoints:
- `bills_details_from_2023`
- `services`
### 3.1.1. Reformat bills details
The field `periodstart` from the API is a value with the format YYYY-MM-DD, with the DD that can be any day of the month, depending on the day you purchased the service. We are going to use a blueprint to override this day to be set on the first of each month.
Click on top right of the data analysis on `New Rule`, then select `Date format`.
You are now on the configuration of the blueprint, please fill as the following:
- Attribute: `periodstart`
- Current Date Format: `%Y-%m-%d`
- Replace by: `%Y-%m-01`
- Condition: `Is not null`
Click on confirm, you can now see the `periodstart` has 1 blueprint.
### 3.1.2. Reformat services
Many datetime fields from the `services` endpoint contains the timezone information. We want to remove this timezone information from the datetime, so we will reformat all the dates with the `Date Format` blue print, as seen previously.
Go to the `services` endpoint, click `New rule`, select `Date format`.
You are now on the configuration of the blueprint, please fill as the following:
- Attribute: `billing_next_billingdate`
- Current Date Format: `%Y-%m-%dT%H:%M:%S%z`
- Replace by: `%Y-%m-%dT%H:%M:%S`
- Condition: `All`
You can repeat the operation for the following attributes:
- `billing_expirationdate`
- `billing_lifecycle_current_terminationdate`
- `billing_lifecycle_current_creationdate`
- `billing_renew_current_nextdate`
## 4. Import your data in the lakehouse
With the data sources now configured, we can proceed to construct the tables. You can go into the `Lakehouse Manager`, and the `Tables` section. Click on the `+` and `Create from a Connectors source`.
The first table will be populated using the `bills_details_from_2023` data source. Click `Next`, rename the table `raws_bills_details`, then click `Create`.
You can now also import the services into a dedicated table:
## 5. Add a tag on services
In order to categorize the expenses, we want to categorize the services with tags, that can represent either your technical environments (e.g. production, dev) or your company's departments or even your customers.
In order to extract the first list of services to categorize, you can go into the `Explorer` section inside the `Lakehouse Manager`.
In the Data, select the `domain`, click on `select` and replace by `select_distinct` and then click on `RUN`.
You should have a result like the following:
You can now click on `Download` and `As a CSV`. You will now have a first CSV file with all the resources to tag. You can open it, rename the column `select_distinct_domain` by `domain`, and then add a second column `tag` and for each domain you want to tag, add the value that you want. You should end up with a CSV file like the following:
It is now time to add a new datasource in the `Connectors` with this CSV file, and create a table in the lakehouse out of it. You can go into `Connectors` -> `sources` -> `New Source` -> `File Upload`. You can name your new source `Domains Tagged` and Add the CSV file and hit the `create` button.
You can new go to the analyzer to check the data is well evaluated, and also add a blueprint on the tags, so we will replace the empty values by `unknown`. Click on `new rule`, `Replace By`. Fill the form as follow:
- Attribute: `tag`
- Replace by: `unknown`
- Condition: `Equal`
- Equal: leave empty
Confirm the blue print.
You can now return to the `Lakehouse Manager`and `Tables` section, and create a table from this new source. Be careful to name the table `domains_tags`.
## 6. Create an aggregation table
Now we have a lot of information about our services, theirs bills and also tags. We are going to build an aggregation table to link all the information together.
Inside the `Lakehouse Manager`and `Tables` section, click on the `+` and `Create a new empty table`. Name it: `enriched_bills_details`.
You should now have the following tables in your lakehouse:
- `services`
- `domains_tags`
- `raws_bills_details`
- `enriched_bills_details`
We are going to drag some fields from the tables we created from sources, and drop them into the `enriched_bills_details`. The fields to drag and drop are:
- From `raws_bills_details`:
- `domain`
- `periodstart`
- `totalprice_value`
- From `domains_tags`:
- `tag`
- From `services`:
- `resource_displayname`
- `resource_product_name`
- `resource_product_description`
Your `enriched_bills_details` table should now look like:
You can now click on `Build` to build this new table.
Now we have the structure of this new table, we are going to create a job to extract the data from the previous 3 tables and load it into this new table. We need to go now inside the `Data Processing Engine` and `Actions` section. Click on `New Action` and select `Aggregate`.
We are now going to map fields from different table into the `enriched_bills_details`, for that, you need first to select the sources and destination:
- Sources:
- `raws_bills_details`
- `services`
- `domains_tags`
- Destination: `enriched_bills_details`
For the sources, we need to explain how the data from the 3 tables are linked together through the SQL JOIN syntax. We are going to add then the following joins:
- Between `raws_bills_details` and `services`:
```sql
raws_bills_details.domain = services.resource_name
```
- Between `raws_bills_details` and `domains_tags`:
```sql
raws_bills_details.domain = domains_tags.domain
```
To finish the aggregate, we need to configure how to select and group attributes of the tables into enriched\_bills\_details. Map the attributes as shown in the screenshot below.
You can now create the aggregate, and run the action.
## 7. Prepare the queries and build the Dashboard
Now we have all the data ready to be exploited, we can go into the `Analytics Manager` and `Queries`.
Let's create some queries.
### Cost evolution vs previous month
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `periodstart`
- Order by: `periodstart desc`
For the visualization:
- As a metric
- Customization: enable all the toggle buttons
### Cost repartition by tag
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `tag`
For the visualization:
- As a pie chart
- Customization:
- Show category
- Show % of total
- Position: outside
- Hide legend
### Cost repartition by product
For the query:
- data: `totalprice_value` sum FROM `enriched_bills_details`
- scale: `resource_displayname`
For the visualization:
- As a pie chart
- Customization:
- Show category
- Show % of total
- Position: outside
- Hide legend
### Cost repartition by product type
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `resource_productdescription`
For the visualization:
- As a pie chart
- Customization:
- Show category
- Show % of total
- Position: outside
- Hide legend
### Cost per month and tag
For the query:
- Data: `totalprice_value` sum FROM `enriched_bills_details`
- Scale: `periodstart` `resource_product_description` `tag`
- Filter on: `periodstart` `Not null`
- Order by: `periodstart` `asc`
For the visualization:
- As bar charts
- Customization:
- Display stacked
- Hide legend
- X-axis: regroup by periodstart
### Create the Dashboard
You can now go into the dashboard section, create a new one and put all your charts into this dashboard by clicking the + button.
On top of the dashboard, you can also add filters to ease the naviation into the data. Consider adding the following filters:
- `periodstart`
- `tag`
- `resource_product_description`
## Conclusion
By following these steps, you can effectively configure and utilize the OVHcloud Billing API endpoint to retrieve, process, and analyze billing information programmatically.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Deploying Apache Superset on OVHcloud Data Platform
## Objective
This guide provides a comprehensive walkthrough for deploying Apache Superset on the OVHcloud Data Platform, covering installation, configuration, and Trino integration for a secure and reliable setup.
**Before you begin:**
You will need an active Trino connection string. The easiest way is to create a [**Trino consumer**](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) in the Connectors; alternatively, build one manually by following [Integrating Trino with External Tools](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md). This connection string is crucial for configuring Trino within Apache Superset.
## Steps to Connect Apache Superset to Data Platform via Trino
### Step 1: Add Apache Superset into Application Services
1. Navigate to your project.
2. Click **Create an application** in the tile **Application Services**.
3. Select **Apache Superset**.
4. In the Apache Superset Store, select the Template you want (for this tutorial, we will select v5.0.0 with SQLite).
:::info
The SQLite file will be stored in the **appservice**
folder of the S31
bucket (Lakehouse Manager section). We highly recommend backing up this file regularly.
:::
5. Provide a name for the new application (e.g., `dashboard`).
6. Build the application.
7. Once built, deploy the application by clicking on the **Deploy** button.
### Step 2: Customize the Template (Editor Tab)
In the **Editor** tab of your application, you have access to the full source code of the template. You can modify it as needed.
This includes:
- The overridden **Apache Superset source code** used in the template.
- A technical **`README.md`** file that explains the structure and customization points of the template.
- The **`entrypoint.sh`** script, which manages backup and restoration of the **`sqlite.db`** file.
- The standard **`forepaas.json`** file, present in all AppService deployments, which defines:
- The version of Apache Superset being used
- The configuration for **role mapping** and runtime parameters
:::info
Modifying this template allows advanced customization of Superset behavior, layout, and deployment logic.
:::
### Step 3: Configure Trino in Apache Superset
1. Open Apache Superset by clicking on the **Open** button.
2. Click **Sign In With DataPlatformSSO**. Once redirected, you will be connected to Superset.
3. In the **Settings** menu, click **Database Connections**.
4. Click **+ Database**.
5. Select **Trino** in the **Supported databases** list.
6. Paste the **full Trino connection string** (from your [Trino consumer](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), or built manually per the [Integrating Trino with External Tools](https://docs.dataplatform.ovh.net/tutorials-trino-external-tools.md) guide) into the **SQLAlchemy URI** field and validate the form.
### Step 4: Create Your First Chart
1. Click **`Datasets`** in the main menu.
2. Click the **`+ Dataset`** button.
3. Once the **New dataset** page is loaded, fill out the form with the following:
- **Database**: The Trino database you have just created.
- **Schema**: The catalog you want to query (e.g., `db_m8obb5ch_default_dataset`).
- **Table**: A table from your selected catalog (e.g., `dataset_history`).
4. Click the **Create dataset and create chart** button.
5. Once created, you will be redirected to the **Create a new chart** form.
6. Select a chart (e.g., **Pie Chart**).
7. Complete the **Query** fields (e.g., **Dimensions**: **`cat_temperature`** & **Metric**: **`Sum(rides)`**).
8. Click the **Create chart** button.
:::info
For more information, see the [Apache Superset documentation](https://superset.apache.org/docs/intro/).
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-finops-focus-superset.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# FinOps uses case with FOCUS and superset
## Objective
This guide provides step-by-step instructions on configuring your Data Project to take your first steps inside the FinOps with a FOCUS file.
## Overview
[FOCUS](https://focus.finops.org/) is a specification inside the FinOps world to expose data consumption throughout multiple cloud providers. This guide will provide an example on how to consume this kind of file inside the Data Platform, and leverage the power of superset to create useful dashboards.
## Requirements
Before beginning, ensure you have:
- An active OVHcloud account with billing access
## Step-by-Step configuration
## 1. Retrieve the FOCUS file
We need to have a FOCUS formatted file to develop this use case. You can easily extract a CSV file with your OVHcloud account consumption by following this link: [OVHcloud Focus Extractor](https://finops.labs.ovh.net/).
You can login on the interface through the OVHcloud SSO. This tool is a POC developped by our OVHcloud internal team meanwhile we are working to provide a more standard way to extract a FOCUS formatted file.
Once logged, you can click on "Load Services Consumption".
When the table is created, you can now click on "Export to CSV" to extract a complete file with your consumption.
You have now a FOCUS formatted CSV that can be loaded inside the Data Platform
## 2. Load the FOCUS file inside the Lakehouse
### 2.1. Create the Data Source
You can now go to the [`Data Platform`](https://eu.dataplatform.ovh.net), inside the `Project` of your choice, then `Connectors` section, and to finish `Sources` in order to create a new `File Upload` source.
You can name your source `focus`, and add the `export.csv` file you downloaded inside the previous step. You can now click on `Create`.
Now the file is loaded, you can go into the `Explorer` section, and then click on your new source `finops` and then start the analysis of the file on clicking `export.csv`. After few seconds, your file should be analyzed successfully.
### 2.3. Load the data into the Lakehouse
You can now go into the `Lakehouse Manager` section, and then `Tables`. You can hit `New Table` and `Create from a Connectors source`. Select your `export.csv` file, then hit `next`, rename the table into `consumption` and `create`.
## 3. Create the Superset Application
Now all the data is ready to be analyze in the Lakehouse, you need to create the Apache Superset Application. You can do so by following [this tutorial](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md).
Considering this tutorial, we consider you followed it until the `Step 5: Create your first Chart` and until the \`3. New Dataset' where you create the Dataset for:
- Database: `trino`
- Schema: `Default dataset`
- Table: `consumption`
You can now hit `Create Dataset and create dashboard`
## 4. Create the Charts
For now, we are going to create multiple charts, always from the same process:
- Click `Charts`, then `+ Chart`
- Chose the dataset `consumption`
We are going through multiple chart you can create:
### Effective cost - current
- Type: `Big number with Trendline`
- Temporal axis: `billingperiodstart`
- Metric: `Billed Cost` aggregate `SUM`
- Comparison Period Lag: `1`
- Show timestamp: true
### Effective cost - previous month
- Type: `Big number`
- Metric: `Billed Cost` aggregate `SUM`
- Filters: `billingperiodstart` with `Range type: advanced` as per the screenshot
- Display settings
- Subtitle: `1 month ago`
### Effective cost - 2 month
- Type: `Big number`
- Metric: `Billed Cost` aggregate `SUM`
- Filters: `billingperiodstart` with `Range type: advanced` and `- 3 month` and `-2 month`
- Display settings
- Subtitle: `2 month ago`
### Most popular region by effective cost
- Type: `Word Cloud`
- Dimension: `regionname`
- Metric: `billedcost` with aggregate `SUM`
### Total accounts
- Type: `Big number`
- Metric: `billingaccountname` aggregate `COUNT DISTINCT`
- Filters: `billingperiodstart` with `Last Month`
- Display settings
- Subtitle: `Previous month`
### Total services
- Type: `Big number`
- Metric: `resourceid` aggregate `COUNT DISTINCT`
- Filters: `billingperiodstart` with `Last Month`
- Display settings
- Subtitle: `Previous month`
### Total providers
- Type: `Big number`
- Metric: `resourceid` aggregate `COUNT DISTINCT`
- Filters: `billingperiodstart` with `Last Month`
- Display settings
- Subtitle: `Previous month`
### Effective cost per service
- Type: `mixed chart`
- X-Axis: `billingperiodstart`
- Time Grain: `Month`
- Query A:
- Metrics: 'billedcost' with aggregate 'SUM'
- Dimension: `Custom SQL` with `CONCAT(servicecategory, '-', servicesubcategory)`
- filters: `billingperiodstart`
- Query B:
- Metrics: 'billedcost' with aggregate 'SUM'
- filters: `billingperiodstart`
- Customize:
- Query A:
- Serie type: `Bar`
- Stack serie: `true`
### Cost repartition per service category and subcategory
- Type: `sunburst chart`
- hierarchy: `servicecategory` and `servicesubcategory`
- primary metric: `billed cost`
- Filters:
- `billingperiodstart`: 'Last month'
### Service Category Effective cost per region
- Type: `Sankey chart`
- Source: `region name`
- Target: `servicecategory`
- Metric: `billedcost`
- Filters: `billingperiodstart`
### Effective cost per sub account name
- Type: `Partition Chart`
- Metrics: `billedcost` aggregate by `SUM`
- Filters: `billingperiodstart`
- Dimensions: `billingaccountname`, `subaccountname`, `servicecategory`
## 5. Create the Dashboard
Now you have a lot of charts, you can put them together into a single place: the Dashboards. Click on `Dashboard`, then `Create`. You can name your dashboard `Billing Summary`, and drag and drop all the charts you previously created to finalize the view.
## Conclusion
By following these steps, you can effectively configure and utilize the OVHcloud Data Platform to implement your first advanced FinOps use case based on the FOCUS format.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Prepare online store orders for a sales dashboard
## Objective
This guide walks you through loading two years of orders from an online store (about 1 million order lines) into the Lakehouse, concatenating them into a single table and cleaning them with a Custom action. You upload the files with [Connectors](https://docs.dataplatform.ovh.net/connectors-sources-upload-files.md), create the tables in [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md), then concatenate and clean the orders with [Data Processing Engine](https://docs.dataplatform.ovh.net/dpe-actions-load.md).
By the end you will have the table `online_retail_ii_clean`, ready to query from Apache Superset. The second part of the guide, [Build a sales dashboard for an online store in Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md), turns it into a sales dashboard.
## Requirements
To follow this guide you need:
- A Data Platform project. This guide uses a project named `ecommerce-sales`.
- The orders of the store and the cleaning script of the Custom action:
| Type | Name |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Orders, December 2009 to December 2011 | [Online Retail II](https://archive.ics.uci.edu/dataset/502/online+retail+ii), UCI Machine Learning Repository |
| Cleaning Python script | 
3. Select the file `online_retail_II_200910.xlsx` from the source `online_retail_ii`, then click **Next**.
4. Keep the default settings and click **Create**. The table is named `online_retail_ii_200910`, in the dataset `default_dataset`.
5. Repeat with `online_retail_II_201011.xlsx`. The table is named `online_retail_ii_201011`.
Both tables appear with 8 attributes and about 525k and 542k rows.
#### Create the table for all orders
1. [Duplicate](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#duplicate) the table `online_retail_ii_200910`. Hover over its row and click the **Duplicate** icon.
2. Name the new table `online_retail_ii`, in the dataset `default_dataset`, and click **Create**.
3. [Build](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#build-table) the table.
The table has the same 8 attributes, and no rows yet.
### Step 4: Concatenate the two years
#### Create the Load actions
1. Open **Data Processing Engine** > **Actions** and click **New action**.
2. Search for `load`, select the _Load_ action and click **Select**.
3. Fill in the action:
| Setting | Value |
| ------------ | -------------------------------------------------------------------------- |
| Source table | `online_retail_II_200910.xlsx` from `online_retail_ii` |
| Destination | `online_retail_ii` from `default_dataset` |
| Attributes | each source attribute mapped to the destination attribute of the same name |
4. Click **Create**.
5. Create a second _Load_ action, with `online_retail_II_201011.xlsx` as source and the same destination.
The list of actions shows your two _Load_ actions, next to the two actions generated automatically when you created the yearly tables.
#### Run both actions with a workflow
1. In **Data Processing Engine** > **Workflows**, click **New workflow**.
2. Name it `Concat`.
3. In _Stage 1_, add your two _Load_ actions.
4. Click **Create**, then run the workflow.
The workflow takes a couple of minutes. When its status turns green, open **Lakehouse Manager** > **Tables**: `online_retail_ii` holds the rows of both yearly tables, **1,067,371 rows**.
:::warning
Run the `Concat` workflow once. Running it again may add the same order lines a second time. After each run, check that `online_retail_ii` holds 1,067,371 rows.
To run the workflow again, you can create a [Delete action](https://docs.dataplatform.ovh.net/dpe-actions-delete.md) that empties `online_retail_ii`, and run it before the workflow.
:::
### Step 5: Clean the orders
#### Create the clean table
1. In **Lakehouse Manager** > **Tables**, duplicate `online_retail_ii`.
2. Name the new table `online_retail_ii_clean` and build it.
You now have 4 tables. `online_retail_ii_clean` has the same 8 attributes and no rows yet.
#### Create the cleaning action
1. In **Data Processing Engine** > **Actions**, click **New action**.
2. Search for `custom`, select the _Custom_ action and click **Select**.
3. Fill in the action:
| Setting | Value |
| -------------------- | ---------------------------------- |
| Action name | `online_retail_cleaning` |
| Function name | `online_retail_cleaning` |
| Development language | `Python 3.11` |
| Python requirements | `pandas` |
| File | upload `online_retail_cleaning.py` |
4. Click **Create**, then run the action.
:::info
**Note:** the function name is the entry point of the script and must be `online_retail_cleaning`. The other functions of the file are helpers: setting one of them as the function name returns `'Event' object is not subscriptable`.
:::
The action reads `online_retail_ii`, applies the cleaning rules and writes the result to `online_retail_ii_clean`. Open its logs: each rule reports how many rows it affects, and the last line gives the number of rows written to `online_retail_ii_clean`: **1,021,128 rows**.
| Rule | A row is removed when |
| ------------------------- | --------------------------------------------------------- |
| 1. Exact duplicates | it is identical to another row on the 8 columns |
| 2. Invalid invoice number | the invoice is not 6 digits, with an optional `C` |
| 3. Invalid stock code | the stock code is not 5 digits, with optional letters |
| 4. Missing description | the description is empty |
| 5. Inconsistent quantity | a sale is not positive, or a cancellation is not negative |
| 6. Zero or negative price | the price is not strictly positive |
| 7. Missing invoice date | the date is empty |
A final step, **8. Harmonized descriptions**, gives each stock code its most frequent description, so a product never shows under two names.
#### Create an action to empty the clean table
The cleaning action adds rows to `online_retail_ii_clean`. Before running it again, empty the table with a [Delete action](https://docs.dataplatform.ovh.net/dpe-actions-delete.md), as for `online_retail_ii` before running the `Concat` workflow again.
1. Click **New action**, search for `delete`, select the _Delete_ action and click **Select**.
2. In _Table list_, move `online_retail_ii_clean` to _Table to empty_.
3. Click **Create**.
To clean the data again, run this _Delete_ action first, then `online_retail_cleaning`.
### Step 6: Connect Superset
All the data is now ready in the Lakehouse. To analyse it, create an Apache Superset application:
1. Create a Trino connection by following the [Generate a Trino connection string for external tools](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) guide.
2. Create the application and connect it to Trino by following Steps 1 to 3 of the [Deploying Apache Superset on OVHcloud Data Platform](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) guide.
#### Create the dataset in Superset
1. In Superset, open **Datasets** and click **+ Dataset**.
2. Select the database connection you created, named `Trino DP` in this guide, then the schema `default_dataset` and the table `online_retail_ii_clean`.
3. Check the columns, then click **Create and explore dataset**.
Every chart of the dashboard then starts from **Charts** > **+ Chart**: choose the dataset `online_retail_ii_clean`, then the chart type.
Continue with [Build a sales dashboard for an online store in Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md).
## Good to know
- **You can connect a real store.** This guide uses a public dataset, uploaded as files. To analyse the orders of your own store, create a [Shopify](https://docs.dataplatform.ovh.net/connectors-sources-shopify.md) or [Stripe](https://docs.dataplatform.ovh.net/connectors-sources-stripe.md) source instead of the File Upload source: in **Connectors** > **Sources**, both appear in the _New source_ window. The tables then follow the attributes of that source, and the cleaning script must be adapted to them.
## What you have built
You have a raw table with every order line of the store, a clean table that follows the UCI description of the variables, and a Superset dataset on top of it.
The next part, [Build a sales dashboard for an online store in Superset](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md), turns the clean table into a dashboard: key figures, revenue over time, best-selling products, sales by country and customer loyalty.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales-superset-dashboard.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Build a sales dashboard for an online store in Superset
## Objective
This guide walks you through turning the cleaned order lines of an online store into a Superset dashboard: key figures, revenue over time, best-selling products, sales by country and customer loyalty.
It is the second part of [Prepare online store orders for a sales dashboard](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md). Superset queries the Lakehouse through [Trino](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md), so all SQL on this page is in the **Trino dialect**.
## Requirements
To follow this guide you need:
- The table `online_retail_ii_clean` in the dataset `default_dataset` of your project, produced by the cleaning Custom action in Step 5 of [Prepare online store orders for a sales dashboard](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md#step-5-clean-the-orders).
- An Apache Superset application connected to your project. If you have not deployed it yet, follow the [Deploying Apache Superset on OVHcloud Data Platform](https://docs.dataplatform.ovh.net/tutorials-install-apache-superset.md) guide.
- The Superset dataset `online_retail_ii_clean`, created in Step 6 of [Prepare online store orders for a sales dashboard](https://docs.dataplatform.ovh.net/tutorials-ecommerce-sales.md#step-6-connect-superset).
## What you will build
At the end of this part, your dashboard looks like this:
| Section | Content |
| --------------- | --------------------------------------------------------------------------------------- |
| Key figures | 4 tiles: net revenue, orders, average order value, customers |
| _Time_ tab | Net revenue by month, seasonality by year |
| _Products_ tab | Top 10 products by revenue, top 10 products by units sold |
| _Geography_ tab | United Kingdom vs rest of world, top 10 countries outside the United Kingdom, world map |
| _Customers_ tab | 3 loyalty tiles, purchase frequency, recency segments, days between purchases |
Every chart follows the same path in Superset: **Charts** > **+ Chart**, pick the dataset and the chart type, fill in the settings, click **Create chart**, then **Save** with the chart name. Leave **Add to dashboard** empty: you create the dashboard and add the charts to it in [Step 8](#step-8-create-and-assemble-the-dashboard).
:::info
**Note:** amounts are in pounds sterling. For every amount, in the _Customize_ tab, set **Currency format** to `GBP` with the `Prefix` position, so values read `£18.9M` and not `18.9M £`.
:::
## Instructions
### Step 1: Create the customer datasets in SQL Lab
The customer charts need one row per customer or per return visit, which a chart cannot compute from order lines. You prepare these rows once, with two queries saved as virtual datasets.
#### Create the virtual dataset customer\_summary
1. Open **SQL** > **SQL Lab**.
2. Select your database and the schema `default_dataset`.
3. Paste the query below and click **Run**.
4. In the query toolbar, click the **Save or Overwrite Dataset** icon and save the dataset as `customer_summary`.
```sql
WITH purchase_days AS (
SELECT DISTINCT customer_id, CAST(invoicedate AS DATE) AS purchase_day
FROM default_dataset.online_retail_ii_clean
WHERE customer_id IS NOT NULL
AND invoice NOT LIKE 'C%'
),
customers AS (
SELECT
customer_id,
COUNT(*) AS purchase_days,
MIN(purchase_day) AS first_purchase,
MAX(purchase_day) AS last_purchase
FROM purchase_days
GROUP BY customer_id
)
SELECT
customer_id,
purchase_days,
LEAST(purchase_days, 10) AS purchase_days_capped,
CASE WHEN purchase_days >= 2 THEN 1 ELSE 0 END AS is_repeat,
first_purchase,
last_purchase,
date_diff('day', last_purchase, DATE '2011-12-09') AS days_since_last_purchase,
CASE
WHEN date_diff('day', last_purchase, DATE '2011-12-09') <= 90 THEN '1. Active (90 days)'
WHEN date_diff('day', last_purchase, DATE '2011-12-09') <= 180 THEN '2. At risk (91 to 180 days)'
WHEN date_diff('day', last_purchase, DATE '2011-12-09') <= 365 THEN '3. Lapsed (181 to 365 days)'
ELSE '4. Lost (over a year)'
END AS recency_segment
FROM customers
```
The result has one row per identified customer: **5,852 rows**.
#### Create the virtual dataset purchase\_intervals
Repeat these steps with the query below and save the dataset as `purchase_intervals`.
```sql
WITH purchase_days AS (
SELECT DISTINCT customer_id, CAST(invoicedate AS DATE) AS purchase_day
FROM default_dataset.online_retail_ii_clean
WHERE customer_id IS NOT NULL
AND invoice NOT LIKE 'C%'
),
with_previous AS (
SELECT
customer_id,
purchase_day,
lag(purchase_day) OVER (PARTITION BY customer_id ORDER BY purchase_day) AS previous_purchase
FROM purchase_days
)
SELECT
customer_id,
purchase_day,
date_diff('day', previous_purchase, purchase_day) AS days_since_previous_purchase
FROM with_previous
WHERE previous_purchase IS NOT NULL
```
The result has one row per return visit: **27,026 rows**.
:::info
**Note:** in SQL Lab, select the schema first. Without it, Trino returns `Schema must be specified when session schema is not set`. You can also prefix the table, as in the queries above: `default_dataset.online_retail_ii_clean`.
:::
### Step 2: Add calculated columns to online\_retail\_ii\_clean
Open **Datasets**, edit `online_retail_ii_clean`, then go to the _Calculated columns_ tab. For each column below, click **Add item**, then fill in its name and SQL expression. Start with `revenue`, the simpler one. Click **Save** once both columns are added.
#### Add revenue
| Setting | Value |
| -------------- | ------------------ |
| Column name | `revenue` |
| SQL expression | `quantity * price` |
`revenue` is the amount of each order line. A cancellation has a negative quantity, so its revenue is negative, and a plain sum gives the net revenue. Every amount in the dashboard is built on this column.
#### Add country\_code
| Setting | Value |
| -------------- | -------------------- |
| Column name | `country_code` |
| SQL expression | the expression below |
`country_code` converts the country name to its ISO 3166-1 alpha-3 code, for the world map:
```sql
CASE country
WHEN 'United Kingdom' THEN 'GBR'
WHEN 'EIRE' THEN 'IRL'
WHEN 'Germany' THEN 'DEU'
WHEN 'France' THEN 'FRA'
WHEN 'Netherlands' THEN 'NLD'
WHEN 'Spain' THEN 'ESP'
WHEN 'Switzerland' THEN 'CHE'
WHEN 'Belgium' THEN 'BEL'
WHEN 'Portugal' THEN 'PRT'
WHEN 'Australia' THEN 'AUS'
WHEN 'Italy' THEN 'ITA'
WHEN 'Norway' THEN 'NOR'
WHEN 'Sweden' THEN 'SWE'
WHEN 'Cyprus' THEN 'CYP'
WHEN 'Finland' THEN 'FIN'
WHEN 'Austria' THEN 'AUT'
WHEN 'Denmark' THEN 'DNK'
WHEN 'Greece' THEN 'GRC'
WHEN 'Japan' THEN 'JPN'
WHEN 'USA' THEN 'USA'
WHEN 'Poland' THEN 'POL'
WHEN 'United Arab Emirates' THEN 'ARE'
WHEN 'Israel' THEN 'ISR'
WHEN 'Hong Kong' THEN 'HKG'
WHEN 'Singapore' THEN 'SGP'
WHEN 'Malta' THEN 'MLT'
WHEN 'Iceland' THEN 'ISL'
WHEN 'Canada' THEN 'CAN'
WHEN 'Lithuania' THEN 'LTU'
WHEN 'RSA' THEN 'ZAF'
WHEN 'Bahrain' THEN 'BHR'
WHEN 'Brazil' THEN 'BRA'
WHEN 'Thailand' THEN 'THA'
WHEN 'Korea' THEN 'KOR'
WHEN 'Lebanon' THEN 'LBN'
WHEN 'Bermuda' THEN 'BMU'
WHEN 'Nigeria' THEN 'NGA'
WHEN 'Czech Republic' THEN 'CZE'
WHEN 'Saudi Arabia' THEN 'SAU'
ELSE NULL
END
```
### Step 3: Build the key figures
The four tiles use the dataset `online_retail_ii_clean` and the chart type _Big Number_.
#### Net revenue
| Setting | Value |
| --------------- | --------------------- |
| Metric | `SUM(revenue)` |
| Number format | `SMART_NUMBER` |
| Currency format | `GBP`, prefix |
| Subheader | `after cancellations` |
Expected value: **£18.9M**.
#### Orders
| Setting | Value |
| ------------- | ---------------------------------------------------------- |
| Metric | _Simple_ tab: column `invoice`, aggregate `COUNT_DISTINCT` |
| Filters | _Custom SQL_: `invoice NOT LIKE 'C%'` |
| Number format | `SMART_NUMBER` |
Expected value: **39.5k**.
#### Average order value
| Setting | Value |
| --------------- | ------------------------------------------------------ |
| Metric | _Custom SQL_: `SUM(revenue) / COUNT(DISTINCT invoice)` |
| Filters | _Custom SQL_: `invoice NOT LIKE 'C%'` |
| Number format | `,.2f` |
| Currency format | `GBP`, prefix |
| Subheader | `sale invoices, before cancellations` |
Expected value: **£497.08**.
:::warning
Do not use `AVG(revenue)` for the average order value. It returns the average amount of an order line, about £18.53, not of an order.
:::
#### Customers
| Setting | Value |
| ------------- | -------------------------------------------------------------- |
| Metric | _Simple_ tab: column `customer_id`, aggregate `COUNT_DISTINCT` |
| Number format | `SMART_NUMBER` |
| Subheader | `identified customers` |
Expected value: **5.88k**.
### Step 4: Build the Time tab
#### Net revenue by month
| Setting | Value |
| --------------- | ----------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Line Chart_, or _Area Chart_ |
| X-axis | `invoicedate` |
| Time grain | `Month` |
| Metric | `SUM(revenue)` |
| Currency format | `GBP`, prefix |
#### Seasonality by year
| Setting | Value |
| --------------- | ------------------------------------------------------------------ |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Area Chart_ |
| X-axis | _Custom SQL_: `date_format(invoicedate, '%m - %M')`, label `month` |
| Dimensions | _Custom SQL_: `CAST(year(invoicedate) AS VARCHAR)`, label `YEAR` |
| Metric | `SUM(revenue)` |
| Currency format | `GBP`, prefix |
You get one area per year. The month number at the start of each label keeps January to December in order.
### Step 5: Build the Products tab
Both charts use the dataset `online_retail_ii_clean` and the chart type _Table_. The cleaning action gives each stock code a single description, so a product never appears twice.
#### Top 10 products by revenue
| Setting | Value |
| ---------- | ------------------------------- |
| Dimensions | `stockcode`, `description` |
| Metrics | `SUM(revenue)`, renamed `total` |
| Row limit | `10` |
| Sort by | `total`, descending |
In the _Customize_ tab, under **Conditional formatting**, add a rule:
| Setting | Value |
| ------------ | ----------------- |
| Column | `total` |
| Color scheme | `success` |
| Color type | background colour |
| Use gradient | yes |
#### Top 10 products by units sold
| Setting | Value |
| ---------- | ----------------------------------- |
| Dimensions | `stockcode`, `description` |
| Metric | `SUM(quantity)`, renamed `quantity` |
| Row limit | `10` |
| Sort by | `quantity`, descending |
Apply the same conditional formatting as the previous table, on the `quantity` column. Placed side by side, the two tables show that the products sold in the largest volumes are not the ones that bring in the most revenue.
### Step 6: Build the Geography tab
#### United Kingdom vs rest of world
| Setting | Value |
| ---------- | --------------------------------------------------------------------------------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Pie Chart_, with **Donut** enabled |
| Dimensions | _Custom SQL_: `CASE WHEN country = 'United Kingdom' THEN 'United Kingdom' ELSE 'Rest of world' END` |
| Metric | `SUM(revenue)` |
| Label type | `Category and Percentage` |
#### Top 10 countries outside the United Kingdom
| Setting | Value |
| ---------- | ------------------------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _Pie Chart_ |
| Dimensions | `country` |
| Metric | `SUM(revenue)` |
| Filters | _Custom SQL_: `country <> 'United Kingdom'` |
| Row limit | `10` |
| Label type | `Category and Percentage` |
#### Revenue by country
| Setting | Value |
| ------------------ | --------------------------- |
| Dataset | `online_retail_ii_clean` |
| Chart type | _World Map_ |
| Country column | `country_code` |
| Country field type | `ISO 3166-1 alpha-3 (cca3)` |
| Metric | `SUM(revenue)` |
### Step 7: Build the Customers tab
These charts cover identified customers and sale invoices, from December 2009 to December 2011. Recency is measured from 9 December 2011, the last day of the data. A customer comes back when they buy on another day: several invoices on the same day count as one purchase.
#### Repeat customer rate
| Setting | Value |
| ------------- | -------------------------------------- |
| Dataset | `customer_summary` |
| Chart type | _Big Number_ |
| Metric | _Custom SQL_: `AVG(is_repeat)` |
| Number format | `.1%` |
| Subheader | `customers with two purchases or more` |
Expected value: **71.4%**.
#### Median purchases per customer
| Setting | Value |
| ------------- | ----------------------------------------------------- |
| Dataset | `customer_summary` |
| Chart type | _Big Number_ |
| Metric | _Custom SQL_: `approx_percentile(purchase_days, 0.5)` |
| Number format | `.0f` |
| Subheader | `purchases per customer` |
Expected value: **3**.
#### Median days between purchases
| Setting | Value |
| ------------- | -------------------------------------------------------------------- |
| Dataset | `purchase_intervals` |
| Chart type | _Big Number_ |
| Metric | _Custom SQL_: `approx_percentile(days_since_previous_purchase, 0.5)` |
| Number format | `.0f` |
| Subheader | `days, across all return visits` |
Expected value: **32**.
#### Customers by number of purchases
| Setting | Value |
| ------------ | ----------------------------- |
| Dataset | `customer_summary` |
| Chart type | _Bar Chart_ |
| X-axis | `purchase_days_capped` |
| Metric | `COUNT(*)` |
| X-axis title | `Purchases (10 = 10 or more)` |
| Y-axis title | `Customers` |
If the bars are unevenly spaced, enable **Force categorical** on the X-axis.
#### Customers by recency
| Setting | Value |
| ---------- | ------------------ |
| Dataset | `customer_summary` |
| Chart type | _Bar Chart_ |
| X-axis | `recency_segment` |
| Metric | `COUNT(*)` |
The four bars add up to 5,852 customers. The number at the start of each segment keeps them in order, from active to lost.
#### Days between purchases
| Setting | Value |
| ---------- | --------------------------------------------------- |
| Dataset | `purchase_intervals` |
| Chart type | _Histogram_ |
| Column | `days_since_previous_purchase` |
| Filters | _Custom SQL_: `days_since_previous_purchase <= 365` |
| Bins | `10` |
| Normalize | enabled |
The filter leaves out the rare returns after more than a year, which would stretch the axis and squeeze every other bar to the left.
### Step 8: Create and assemble the dashboard
1. Open **Dashboards** and click **+ Dashboard**. The new dashboard opens in edit mode.
2. Click its title and rename it `Online store sales`.
3. Add a header `Key figures`, drag the 4 key figure tiles from the right panel and place them on one row.
4. Add a header `Trends and breakdowns`, then a tabs component with 4 tabs: `Customers`, `Time`, `Products` and `Geography`.
5. Drag each of the other charts from the right panel into its tab. In the _Customers_ tab, put the 3 tiles on one row, the 2 bar charts side by side and the histogram at full width.
6. Click **Save**.
## What you have built
You have a dashboard that answers the questions of an online store manager: how much the store sells and to how many customers, how sales evolve over the year, which products and countries matter, and how often customers come back.
From here you can add dashboard filters on `country` or on a date range, or apply the same pipeline to your own order data. For another Superset dashboard built on Data Platform, see the [FOCUS and Apache Superset use case](https://docs.dataplatform.ovh.net/tutorials-finops-focus-superset.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-tutorials-data-export.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data export & management
## Exports, external access and permissions
Exporting folders with a custom date builds export paths that carry the run date, which is what downstream systems and archives usually expect. Setting role conditions on one bucket narrows an IAM role to a single bucket rather than the whole Lakehouse, using CEL conditions. Custom event handling actions let an outside system trigger a workflow with a payload, which is the inbound half of the same integration problem.
Accessing buckets with S31
-compatible tools covers the other direction, reading and writing project buckets from the AWS CLI, Rclone, NodeJS or Python instead of from the interface.
Together they cover data lifecycle work: scheduling exports, shaping their layout, reaching data from outside, and keeping permissions tight around all of it.
[### Export folders with a custom date→
Export paths that carry the run date.
](/tutorials-dated-export-folders)[### Set role conditions on one bucket→
Narrow an IAM role to a single bucket with CEL.
](/tutorials-bucket-role-conditions)[### Custom event handling action→
Let an outside system trigger a workflow with a payload.
](/tutorials-custom-event-handling)[### Access your buckets with S3-compatible tools→
AWS CLI, Rclone, NodeJS and Python against a project bucket.
](/tutorials-bucket-s3-access)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/tutorials-dated-export-folders.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Export folders with a custom date
## Objective
This tutorial demonstrates how to add a custom date to your folder names during the creation of the [Data Processing Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md).
:::info
**Note:** This functionality is only available to the following actions: _Load to Bucket_, _MySQL to Parquet_, and _PostgreSQL to Parquet_.
:::
## Introduction
### Requirements
To follow this tutorial, you need to be familiar with the [Data Processing Engine](https://docs.dataplatform.ovh.net/landing-page-dpe.md) and you should preferably have done the [Getting Started guide](https://docs.dataplatform.ovh.net/landing-page-getting-started.md). The essential is that you know what [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) are for and how to use them.
### Concepts Overview
When you create actions, you have the choice to name your folder which will hold your exported files. The _$now_ token enables you to give custom dates easily using a few basic patterns as part of the folder name.
There are 4 patterns available on the Data Platform:
- **$now(date)**
- **$now(datetime)**
- **$now(timestamp):** Timestamp will be printed in [UNIX epoch time](https://en.wikipedia.org/wiki/Unix_time)
- **$now(_insert sequence here_):** Create custom date sequences with the available [format codes](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Now let's see how all of this works in practice!
## Use the $now token
For this tutorial we shall only explore how to use the _$now_ token. To know how to create and run _Actions_, there is an in depth documentation on [Actions](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) available.
For the ease of understanding we will choose to create an action - _Load to Bucket_, which is meant to run daily for the foreseeable future - Which will in turn lead to folders being generated on a daily basis. To help sort and easily access the correct folder, the **$now** token can be used.
At the stage where the option to input the name for the folder becomes available, You can use the _$now_ token like shown in the example below.
On the picture below, you can see that the folder has been successfully generated in the _bucket_ with the naming schema we expected. It is not necessary to have additional text in the folder name like used in the example, the token can be used independently as well.
bFor the scenario mentioned above, here is an example of how the _bucket_ will look with the folders generated over the past few days. It becomes easy to quickly identify the folder created on a certain date.
## Examples
Here are some more examples of how the folder name will appear with different use cases:
- **$now(date):** 2023-08-15
- **$now(datetime):** 2023-08-15\_20-51-04
- **$now(timestamp):** 1692132664 _(Unix epoch time)_
- **$now(%a-%d-%b-%Y\_%H-%M%):** Tue-15-Aug-2023\_20-53
:::info
Since folder names shouldn't have special characters like _colon_, _slash_, _asterisk_, etc. the _dash_ character is used for the time and date format.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/tutorials-bucket-role-conditions.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# How to set a role condition on a specific bucket
## Objective
The Data Platform [Identity Access Manager (IAM)](https://docs.dataplatform.ovh.net/landing-page-iam.md) allows you to set fine-grained conditions in your Project:
- on resources properties
- on users properties
- on data, through Advanced Data Access Control
The purpose of this tutorial is to show how to give users in your Project restricted access to only the [Lakehouse Manager bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) called _my\_bucket_.
## Requirements
You will need:
- an [IAM role](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) that grants access to the resource `bucket` (such as the default "DataStore Editor" role)
## Tutorial
To restrict access to specific [buckets in your Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md), you must set a condition on the _**technical name**_ when binding a role granting access to _Data Store buckets_.
:::info
Data Store is the original name of the buckets on Data Platform.
:::
[Bind](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#bind-a-role-to-a-user-service-account-or-group) such a role to the group/user of your choice.
Then, add a [condition](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#add-a-condition-when-binding-a-role) to the role binding.
The condition must be set on the _technical name_ of the resource, which intuitively would look like the following:
```text
Name == "my_bucket"
```
However, just adding this condition would filter **the entire role** to the resources called "my\_bucket", including resources that are not buckets (like queries, dashboards, pipelines, etc..). In other words, if your role gives access to any other resource, they will be filtered just like buckets.
Consequently, it is recommended to set conditions as follows.
### Using the visual builder
If your initial role grants access to other resources, it is important to start by adding a first **rule** which will allow access to all resources that are not buckets which are planned by the role.
:::info
Through conditions, you cannot give access to resources/actions that were not initially permitted by the role. Conditions can only filter a role further.
:::
- Field: _Resource_
- Operator: _Not equals_
- Value: _Bucket_
Then, add a **branching** with an _OR_ operator.
In the branching, add two rules linked by an _AND_ operator.
First rule:
- Field: _Resource_
- Operator: _Equals_
- Value: _Bucket_
Second rule:
- Field: _Technical name_
- Operator: _Equals_
- Value: _my\_bucket_
Press **Confirm**. The user/group will now have access only to the bucket "my\_bucket", and access to all other resources contained in the role.
:::info
This branching system can also be used to restrict _write_/_delete_ access to only some objects (for example, the ones that have the tag "dev") while allowing _read_ access to the remaining objects.
:::
### Using the CEL editor
In CEL, the condition would be the following:
```text
Resource != "bucket"
||
(
Resource == "bucket"
&&
Name == "my_bucket"
)
```
To go further, see the customization capabilities of the Identity Access Manager in our documentation.
[
- **Advanced** mode does not add extra controls; it is for typing the filter manually as text, for example `gender=female&birthdate=...` with parameters separated by `&`. Use the syntax your FHIR server expects.
### Linked tables
Some tables depend on others. When you select a table, the UI may auto-select related tables that are required for a consistent dataset.
For example, selecting **`patient_name`** can also select **`patient`** and **`human_name`**. The exact rules follow the product implementation.
:::info
Review the full selection (including any tables added automatically) before you click **Add**. Dependencies can widen the scope of the data you ingest.
:::
When your selection is ready, click **Add**. Use **Cancel** to close without adding tables.
### Name the source
Before you finish creating the source, enter a name in the field provided by the UI.
:::warning
The **technical name** cannot be changed after you create the source. It is used when you open the source with the [Data Platform SDK](https://docs.dataplatform.ovh.net/landing-page-developers-python-sdk.md).
:::
## Use ingested data
After you ingest data, you can work with it like other Platform sources. Load it into [Lakehouse Manager tables](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) using [Data Processing Engine load actions](https://docs.dataplatform.ovh.net/dpe-actions-load.md).
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-connectors-analyzer.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connectors Analyzer
## What is the Analyzer?
The Analyzer is the first quality gate on incoming data. It runs against a registered source before any of that data reaches a project table, so problems in structure, typing and formatting are caught at the door rather than after a load has written them into the Lakehouse.
It does two things.
**Metadata extraction** reads the shape of the source objects: the fields they expose, their types, and the volumes involved. Every downstream [Load action](https://docs.dataplatform.ovh.net/dpe-actions-load.md) builds on that metadata, so a source has to be analyzed before its objects can be loaded.
**Blueprint rules** are the cleaning layer on top. A blueprint is a named set of conditions and transformations applied to a source object as its data passes through, so date formats, casing, numeric precision and null handling follow one standard across every source feeding the project.
Extract the metadata first. Blueprint rules are written against fields the Analyzer has already found.
[### Extract the metadata→
Read the fields, types and volumes a source object exposes.
](/connectors-analyzer-extract-metadata)[### Blueprint rules→
Normalize incoming values with conditions and transformations.
](/connectors-analyzer-blueprint-rules)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-analyzer-extract-metadata.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Extract the metadata of a source
## Objective
To extract the metadata of a source for the first time, simply select a source object on the left-hand panel. This will parse a sample of 1,000 rows from the source to obtain the following information:
- attribute names
- attribute types
- attribute minimum and maximum values
- attribute card., i.e. the number of unique values in the sample
- attribute null values
To extract the metadata of a source again, you can click on the **Extract Metadata** button at the top right. Further, choose _Sample data_ or _Complete data_ to extract on the first 1,000 rows or on the entire table respectively.
:::info
A _complete_ metadata extraction can take a while depending on the size of the selected table
:::
:::warning
If your metadata extraction fails, investigate why by checking out the **logs in the Jobs tab** in the Connectors.
:::
Once the meta are extracted, you can start working to set standard structure formats for your data by adding blueprint rules to any attribute.
[Learn how to add blueprint rules](https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-analyzer-blueprint-rules.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Clean data automatically with blueprint rules
## Objective
The Platform provides default "blueprint rules" which consist of simple cleaning logic (like dates formatting, thousands of separators, etc.) to help users run simple data preparation. You can choose from available transformations in the Platform store and apply them if the applied condition is met.
It is not mandatory to use blueprint rules when using the platform. If used, the rules will be added automatically to the [Data Processing Engine Load actions](https://docs.dataplatform.ovh.net/dpe-actions-load.md) you will define to ingest the data inside your dataset.
## How a rule works
A rule applies to each line independently and allows several things.
If a condition is specified, the rule will apply only under certain conditions expressed in `condition` and `condition_value(s)`.
If a transformation is filled, the transformation action on each line will be performed, otherwise, it is considered to be a filtering rule (`keep`).
If no _condition_ is filled, the one set by default is `ALL`, which allows any line.
If no _transformation_ nor _condition_ is filled, then we consider it to be a _default transformation_. Otherwise, if no transformation is filled and there is one _condition_, the action by default is _keep_. The action applies only to lines that respond to the rule defined by _condition_.
### Fields
| Attribute name | Required | Example | Comment |
| ----------------- | ----------------------------------------------------------------------------- | ----------------------------------- | --------------------------------- |
| attribute\_name | Yes if the format is a list, no if the format is an object | `ca_ttc`, `date` | |
| condition | No (defaults to `ALL`) | `EQUAL`, `IN` | See the condition list below |
| condition\_value | Yes, except if condition is `IN`, `NOT IN` or `ALL` | `1`, `"Categorie1"` | |
| condition\_values | Yes if condition is `IN` or `NOT IN` | `["ca_ttc", 2, 3]`, `["categorie"]` | |
| action | No | `keep`, `concat` | See the transformation list below |
| action\_value | Yes if action is `default` or `split` | `1`, `"Categorie1"` | |
| action\_values | Yes, except if action is `default`, `split`, `action_error`, `keep` or `skip` | `["ca_ttc", 2, 3]`, `["categorie"]` | |
| else\_value | If the condition is not met, replaces the value by `else_value` | `"ca2"` | |
## Configure a blueprint rule
Rules require 3 steps to be configured:
1. **Information:** Name of the attribute
2. **Transformation:** If the condition is confirmed, which transformation action to apply to the data (can be chosen from the Platform store)
3. **Condition:** which rule to apply for the condition
To add a rule on an attribute, on the relevant field, click on the following button:
### Transformations
Different **transformations** are available in the Platform store:
- _Replace by_: Replaced values by a single value
- _Substring_: Values will be truncated
- _Date format_: Replace date formats
- _Keep line_: Keep matching values
- _Skip line_: Skip matching values
- _Throw error_: Matching values will trigger an error
- _Replace substring_: Replace matching substrings
- _Apply regexp_: A new regexp will be applied to matching values
- _Stop workflow_: Stop the ongoing workflow
### Conditions
Different **conditions** are available:
The Platform provides you a variety of Conditions you can apply to your chosen transformation. The list includes the following: _Equal_, _Not Equal_, _Between_, _Not Between_, _Lower than_, _Lower or Equal than_, _Greater than_, _Greater or Equal than_, _REGEXP_, _NOT REGEXP_, _Is a number_, _Is not a number_, _Is a date_, _Is not a date_, _Is null_, _Is not null_
:::info
Once the transformation and condition are set, the rule will be applied to the selected attribute.
:::
## Reference
The names below are the values written in the rule itself, so they are the ones you use when you edit rules as JSON, for example in the [Load advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-load-advanced-mode.md) or the [Aggregate advanced mode](https://docs.dataplatform.ovh.net/dpe-actions-aggregate-advanced-mode.md).
:::details Condition list
| Condition's name | Description |
| ---------------- | ---------------------------------------------------------------------------------------------------------------- |
| ALL | Keeps all the lines |
| EQUAL | Valid lines: those whose _attribute\_name_ is equal to _condition\_value_ |
| NOT EQUAL | Valid lines: those whose _attribute\_name_ is not equal to _condition\_value_ |
| IN | Valid lines: those whose _attribute\_name_ is in the list _condition\_values_ |
| NOT IN | Valid lines: those whose _attribute\_name_ is not in the list _condition\_values_ |
| LTE | Valid lines: those whose _attribute\_name_ is lower or equal to _condition\_value_ |
| LT | Valid lines: those whose _attribute\_name_ is lower than _condition\_value_ |
| GTE | Valid lines: those whose _attribute\_name_ is higher or equal to _condition\_value_ |
| GT | Valid lines: those whose _attribute\_name_ is higher than _condition\_value_ |
| REGEXP | Valid lines: those whose _attribute\_name_ corresponds to the regexp _condition\_value_. Example: `^([0-9]{4})` |
| NOT REGEXP | Valid lines: those whose _attribute\_name_ does not correspond to the regexp _condition\_value_ |
| BETWEEN | Valid lines: those whose _attribute\_name_ is between _condition\_values\[0]_ and _condition\_values\[1]_ |
| NOT BETWEEN | Valid lines: those whose _attribute\_name_ is not between _condition\_values\[0]_ and _condition\_values\[1]_ |
| OR | Valid lines: those where one of the values contained in _condition\_values_ is not empty (empty, null, False, 0) |
| AND | Valid lines: those where all values contained in _condition\_values_ are not empty (empty, null, False, 0) |
| IS NULL | Valid lines: those whose _attribute\_name_ is null |
| IS NOT NULL | Valid lines: those whose _attribute\_name_ is not null |
| IS INT | Valid lines: those whose _attribute\_name_ is an integer |
| IS NOT INT | Valid lines: those whose _attribute\_name_ is not an integer |
| IS DATE | Valid lines: those whose _attribute\_name_ is a date |
| IS NOT DATE | Valid lines: those whose _attribute\_name_ is not a date |
| IS DEC | Valid lines: those whose _attribute\_name_ is a decimal |
| IS NOT DEC | Valid lines: those whose _attribute\_name_ is not a decimal |
| IS NUM | Valid lines: those whose _attribute\_name_ is a number |
| IS NOT NUM | Valid lines: those whose _attribute\_name_ is not a number |
:::
:::details Transformation list
- _keep_: Keep the line
- _skip_: Skip the line
- _default_:
Replace the value by _action\_value_.
For dates, there is:
- `$today` returns yyyy-mm-dd
- `$now` returns yyyy-mm-dd hh:mm:ss
- `$yesterday` returns yyyy-mm-dd
- _regexp\_replace_:
Replace the value corresponding to regexp _action\_values\[0]_ by _action\_values\[1]_
- _replace_:
Replace the value corresponding to _action\_values\[0]_ by _action\_values\[1]_
- _substring / substr_:
- If _action\_values\[0]_ >= 0:
Starts at the \*action\_values\[0]\*th character
- If _action\_values\[0]_ \< 0:
Only takes into account the _action\_values\[0]_ last characters
- If _action\_values\[1]_ is filled:
The 2nd figure represents the max number of characters wanted
- _date\_replace_:
Replace the date format _action\_values\[0]_ by _action\_values\[1]_.
Examples for _action\_values_:
- `["%d/%m/%Y", "%Y-%m-%d"]`
- See the date format directives below
- _add_:
Add several fields or hard numbers, automatically detecting if for each value of _action\_values_ it is an attribute available in the line, otherwise it is considered to be a number. Examples for _action\_values_:
- `["ca_ht", "taxes"]`
- `["nb_clics", 1]`
- _sub_: Substract _action\_values\[0]_ from all the other data present in _action\_values_
- _mul_: Multiply _action\_values\[0]_ by _action\_values\[1]_
- _div_: Divide _action\_values\[0]_ by _action\_values\[1]_
- _mod_: Modulo _action\_values\[0]_ by _action\_values\[1]_
- _concat_: Concatenate one or several fields with hard strings. Each element of _action\_values_ detects if it is the name of a column available in the line, if not we consider it to be a string of hard characters. Examples for _action\_values_:
- `["site_name", " - ", "country"]`
- `["civilite", " name", "lastname"]`
- _ceil_: Rounded up _action\_value_
- _floor_: Rounded down _action\_value_
- _disaggregate / split_:
Used to create multiple lines if a field contains multiple values that should be split into separate lines.
There can only be one _disaggregate_ rule per DPE action, and this rule always applies last.
The field is split on the character _action\_value_, and for each split element only non-empty values are kept.
- _action\_error_: Leads to an error in the action, and consequently considers that all the lines are false
- _stop\_workflow_: Stop the workflow
:::
:::details Date format directives
| **Directive** | **Meaning** | **Example** |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| %w | Weekday as a decimal number, where 0 is Sunday and 6 is Saturday. | 0, 1, ..., 6 |
| %d | Day of the month as a zero-padded decimal number. | 01, 02, ..., 31 |
| %m | Month as a zero-padded decimal number. | 01, 02, ..., 12 |
| %y | Year without century as a zero-padded decimal number. | 00, 01, ..., 99 |
| %Y | Year with century as a decimal number. | 1970, 1988, 2001, 2013 |
| %H | Hour (24-hour clock) as a zero-padded decimal number. | 00, 01, ..., 23 |
| %I | Hour (12-hour clock) as a zero-padded decimal number. | 01, 02, ..., 12 |
| %p | Locale's equivalent of either AM or PM. | AM, PM (en\_US); am, pm (de\_DE) |
| %M | Minute as a zero-padded decimal number. | 00, 01, ..., 59 |
| %S | Second as a zero-padded decimal number. | 00, 01, ..., 59 |
| %f | Microsecond as a decimal number, zero-padded on the left. | 000000, 000001, ..., 999999 |
| %z | UTC offset in the form +HHMM or -HHMM (empty string if the object is naive). | (empty), +0000, -0400, +1030 |
| %Z | Time zone name (empty string if the object is naive). | (empty), UTC, EST, CST |
| %j | Day of the year as a zero-padded decimal number. | 001, 002, ..., 366 |
| %U | Week number of the year (Sunday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0. | 00, 01, ..., 53 |
| %W | Week number of the year (Monday as the first day of the week) as a decimal number. All days in a new year preceding the first Monday are considered to be in week 0. | 00, 01, ..., 53 |
| %% | A literal '%' character. | % |
:::
:::details Rule examples
Replace a date format:
```json
[
{
"attribute_name": "date",
"action": "date_replace",
"action_values": ["%d/%m/%Y", "%Y-%m-%d"]
}
]
```
Add 1 to the _year_ attribute, only on the lines where it is 2015 or 2016:
```json
[
{
"attribute_name": "year",
"condition": "IN",
"condition_values": ["2015", "2016"],
"action": "add",
"action_values": ["year", 1]
}
]
```
Set _year_ to 2015 on the lines where it is lower than 2015:
```json
[
{
"attribute_name": "year",
"condition": "LT",
"condition_value": "2015",
"action": "default",
"action_value": "2015"
}
]
```
Set the _date_ attribute to today's date:
```json
[
{
"action": "default",
"action_value": "$today",
"attribute_name": "date"
}
]
```
Convert a temperature from Kelvin to Celsius:
```json
[
{
"action": "sub",
"action_values": [
"temp_min",
273.15
],
"attribute_name": "temp_min"
}
]
```
Normalise a timestamp:
```json
[
{
"action": "date_replace",
"action_values": [
"%a %b %d %H:%M:%S +0000 %Y",
"%Y-%m-%d %H:%M:%S"
],
"attribute_name": "statuses_created_at"
}
]
```
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Connector consumers
## What is a consumer?
A consumer is an external tool granted read access to project data. Where [sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) bring data in, consumers expose it back out: a BI report built on project tables, a lineage backend receiving OpenLineage events, or any SQL client connecting through Trino.
Creating one is a single step. Under **New consumer**, pick the tool from the store, and the platform registers a matching [IAM application](https://docs.dataplatform.ovh.net/iam-application-authentication.md) in the project with the OAuth redirect URLs and secrets already configured for it. [Trino consumers](https://docs.dataplatform.ovh.net/connectors-consumers-trino.md) are the exception: they are created from their own **Trino consumers** section rather than that button.
Power BI and OpenLineage support one consumer each per project. Trino consumers have no such limit, so create as many as you need, each with its own connection string.
Access is then governed entirely by the project's [Identity Access Manager](https://docs.dataplatform.ovh.net/landing-page-iam.md). People connecting from the external tool authenticate against project IAM and see only the tables and queries their [roles and groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md) allow, which means every external user, including people who never open the platform itself, has to exist in the project's IAM. Granting the _Analytics Consumer_ role gives full read access to the project's data.
:::warning
Permissions are not scoped per consumer. A role that grants access to a table grants it through every consumer attached to the project.
:::
[### Power BI→
Connect a Power BI workspace to project tables and queries.
](/connectors-consumers-power-bi)[### OpenLineage→
Send lineage events to an OpenLineage-compatible backend.
](/connectors-consumers-openlineage)[### Trino→
Expose a Trino endpoint for any external SQL client.
](/connectors-consumers-trino)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-consumers-power-bi.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Plug Power BI to your Data Platform
## Objective
Are you or your teammates already proficient with [Microsoft Power BI](https://powerbi.microsoft.com/en-us/getting-started-with-power-bi/)? If you want to use it as your analytics solution on top your Data Platform, this guide is for you.
The configuration is done in two parts. First you need to [configure the consumer on Data Platform](#how-to-configure-power-bi-with-your-data-platform) and then [configure the connector on Power BI Desktop](#configuration-on-power-bi).
## How to configure Power BI with your Data Platform
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) _Power BI_ consumer in the Connectors. Else, just edit it.
Since the Trino connector for Power BI is an open sourced project and not yet certified by Microsoft, you have to **download the extension**, either manually from [this link](https://raw.githubusercontent.com/CreativeDataEU/PowerBITrinoConnector/refs/heads/master/Trino/bin/AnyCPU/Debug/Trino.mez) or from the interface shown below.
Power BI users will have to be registered in a Project's Identity Access Manager (IAM) in order to access its data. They will only access the data associated to their [API Secret Key](https://docs.dataplatform.ovh.net/tutorials-generate-api-keys.md).
You can **easily grant full access to a new Power BI user** by assigning them the _Analytics Consumer_ role. This is a default role that can be assigned to a group, user, or service account and provides the appropriate permissions to view data from an external consumer.
Alternatively, if you prefer to fine-tune permissions, you can go to your project's Identity Access Manager and grant the user the following specific permissions:
- _Data Manager | Source | Read_ and _Data Manager | Table | Read_: to import a table's data inside Power BI
- _Advanced Data Access Control | Read_: to access the data inside the project
:::warning
Any user who has the role of **Admin** will also have access to the Project's data
:::
## Configuration on Power BI
Once you have [downloaded](https://raw.githubusercontent.com/CreativeDataEU/PowerBITrinoConnector/refs/heads/master/Trino/bin/AnyCPU/Debug/Trino.mez) the connector _.MEZ_ file, follow this procedure to use it on **Power BI Desktop**.
### Allow your Power BI Desktop to use a custom connector
Open _Power BI Desktop_. Click on **File** in the header.
Open the **Options and settings** section, then select **Options**.
Find the **Global > Security** section, then activate the following option: _(Not Recommended) Allow any extension to load without validation or warning_.
This will allow you to get data from a source not yet certified by Microsoft, in this case the Trino Connector.
Now, **close Power BI Desktop** for the changes to be applied.
### Install the connector
Create this folder in your user directory:
```yaml
C:\Users\[Your_Windows_User_Name]\Documents\Microsoft Power BI Desktop\Custom Connectors
```
If the _Microsoft Power BI Desktop\Custom Connectors_ folders already exists, then you don't have to create it.
:::warning
Make sure that the folders have exactly the same names as above, or you won't be able to find the Trino connector inside Power BI Desktop.
:::
Copy the connector _.MEZ_ file that you have previously downloaded into this new directory.
You can now restart Power BI Desktop.
### Import your first data from Data Platform
Click on **Get Data**.
Search for _Trino_, and select the connector.
You now have to enter the [URL of the Data Platform Project](https://docs.dataplatform.ovh.net/manage-projects-config-variables.md) you want to connect to. You can easily find this URL on the page of the consumer on the Data Platform project. The port is 443.
If you are not yet authenticated to this Project, you will be asked to authenticate at this step.
Choose the Basic mode and use **API key** for User name and **Secret key** for Password as configured to your Service-Account/User in IAM.
When you are successfully authenticated, press **connect** to continue.
Then you will see your datasets, schemas and tables in preview mode.
You can now create your first visualizations on your data from your Data Platform project using Power BI Desktop.
:::info
If the data is updated on Data Platform, graphs can be manually refreshed with the latest data by pressing the refresh button in the Power BI header.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-consumers-openlineage.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Forward lineage events to an external tool with OpenLineage
## Objective
The **OpenLineage consumer** lets you forward data lineage events generated by your Data Platform project to an external lineage solution of your choice. Once connected, the platform continuously streams [OpenLineage](https://openlineage.io/)-compliant events to your lineage backend so you can visualize and explore your data lineage graph there.
:::info
Data Platform also includes a built-in lineage viewer: see [Lineage](https://docs.dataplatform.ovh.net/lakehouse-manager-lineage.md) in the Lakehouse Manager. The OpenLineage consumer is for teams that want the same events in their own lineage solution, see [compatible tools](#compatible-lineage-solutions) below.
:::
:::warning
**Compatibility notice:** At this time, [Marquez](https://marquezproject.ai/) is the only lineage solution that has been validated to work reliably with Data Platform's OpenLineage events. Other OpenLineage-compatible tools may receive events but could provide a degraded experience. We recommend using Marquez for the best results.
:::
## Compatible lineage solutions
The following lineage solution has been validated to work with Data Platform:
| Tool | Description |
| :------------------------------------ | :---------------------------------------------------------- |
| [Marquez](https://marquezproject.ai/) | Open-source metadata service built natively for OpenLineage |
Any tool that exposes an OpenLineage HTTP API endpoint can technically be used, but only Marquez is officially supported at this time.
## Configure the OpenLineage consumer
If you haven't done so already, [add a new](https://docs.dataplatform.ovh.net/landing-page-connectors-consumers.md) **OpenLineage** consumer in the Connectors. Else, just edit it.
You will need to provide the following values:
| Field | Required | Description |
| :----------------------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint URL** | Yes | The HTTP(S) URL of your lineage solution's OpenLineage API (e.g. `https://marquez.example.com/api/v1/lineage`) |
| **Auth token** | No | A bearer token used to authenticate requests to your lineage backend. Leave empty if your backend does not require authentication |
| **Keep Events** | No | If true, events are stored in the bucket after being sent for history. Otherwise they are deleted. (Default: false) |
| **Disable Trino events** | No | If true, lineage events generated by Trino queries (Explorer, Analytics Manager) are not forwarded to your lineage backend. Only events from DPE actions (Load, Aggregate, Custom) are sent. Use this to reduce noise if you only care about batch lineage. (Default: false) |
Once a lineage consumer is configured, every lineage events will be stored on the [Lakehouse Manager Bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) **DWH** under `openlineage/events/queued`. Events will be forwarded to your lineage solution each time the [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) DPE action runs.
## Send lineage events from the DPE
Lineage events are forwarded by the [**Send OpenLineage Events**](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) action in the Data Processing Engine. You can configure it to run:
- **As a cron job**: on a fixed schedule (e.g. every hour, every night)
- **As an always-up action**: continuously, to keep your lineage solution in sync in near real time
See the [Send OpenLineage Events](https://docs.dataplatform.ovh.net/dpe-actions-send-openlineage-events.md) documentation for setup details.
:::info
For Custom actions, lineage tracking is opt-in. See [Track data lineage in a Custom action](https://docs.dataplatform.ovh.net/developers-python-sdk-lineage.md) to learn how to emit lineage events from your own Python code.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/connectors-consumers-trino.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Generate a Trino connection string for external tools
## Objective
A **Trino consumer** generates a ready-to-use Trino connection string in a few clicks. Previously, connecting an external tool to your project over Trino meant creating a service account, generating API keys, and assembling the connection string by hand. A Trino consumer does all of that for you.
You'll find them in the Connectors service, under the **Consumers** tab, in the **Trino consumers** section of **Registered consumers**.
:::info
Unlike the PowerBI and OpenLineage consumers (which you connect once), you can create as many Trino consumers as you need. Each with its own name, permission level, and credentials.
:::
## Create a Trino consumer
1. In the **Trino consumers** section, click **+ New Trino consumer**.
2. Give it a **name**, an optional **description**, and optional **tags**.
3. Set the **permission**: the access level this Trino consumer has on your data:
| Permission | Access |
| :--------------- | :-------------------------------- |
| **READ** | Query and retrieve data only. |
| **READ & WRITE** | Query, retrieve, and modify data. |
4. Click **Create**.
5. Your **Trino connection string** appears.
:::warning
Copy the connection string now and store it somewhere safe (for example, a password manager). You can't view it again after you close this window, if you lose it, you'll have to create a new Trino consumer.
:::
## Manage the linked service account
Each Trino consumer has a dedicated [service account](https://docs.dataplatform.ovh.net/iam-service-accounts.md) in your project's Identity Access Manager. After you create the consumer, click **Go to service account settings** to view and manage it.
The project **admin also receives an email** containing the Trino connector credentials.
## Use the connection string
The connection string is ready to paste into any external Trino-compatible client or BI tool. It embeds the credentials and follows this shape:
```text
trino://
2. Select the policy tag you want to assign and specify the access type: **Read** for read-only access, or **Read and Write** for both.
3. Assign the policy tag values (e.g., `access_level: level 1`) to the user or group.
4. Confirm and save the access settings.
:::warning
**Important limitation**: The Grant Access UI generates a CEL condition without Resource-level scoping. This means the condition is evaluated at **every level** of the gate chain (dataset, table, attribute). If the policy tag is only applied at the table level, the dataset-level check will fail and the user will be denied access. **You must manually edit the CEL condition** in the IAM to add Resource scoping. See [Writing CEL Conditions Manually](#writing-cel-conditions-manually) below.
:::
:::info
**Note**: Users without **Advanced Data Access Control** permissions cannot read or write to any dataset, table, or attribute by default. This can be verified in the [IAM settings](https://docs.dataplatform.ovh.net/landing-page-iam.md) for the user.
:::
### Writing CEL Conditions Manually
To correctly scope policy tag access, you must edit the CEL condition on the user's role binding in the [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions).
The Grant Access UI generates a CEL condition like this:
```text
Service == "adac" && (PolicyTags.sensitivity == "high")
```
This condition checks `PolicyTags.sensitivity == "high"` at **every** resource level. If the dataset doesn't have `sensitivity: high` (only the table does), the dataset check fails and the user is denied.
To fix this, edit the CEL condition to scope the policy tag check to specific resource levels using `Resource`:
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
This condition means:
- At the **dataset** level: always pass (no tag check needed)
- At the **table** level: require `sensitivity: high`
- At the **attribute** level: always pass (no tag check needed)
To edit the CEL condition:
1. Open the [Identity Access Manager](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions) and navigate to the user or group.
2. Find the ADAC role binding (the one with Service `adac`).
3. Click **Edit** on the condition.
4. Switch to the **CEL editor** and replace the generated condition with the Resource-scoped version.
5. Save.
:::info
Note the comment `// Permissions are managed in Lakehouse Manager. Do not update here!` on line 1. This is auto-generated by the Grant Access UI. You can safely replace the CEL content below it with your Resource-scoped condition.
:::
### Common CEL Patterns
Below are ready-to-use CEL patterns for common scenarios. In each pattern, replace tag keys and values with your own.
:::warning
**Critical:** Any resource level with a plain pass-through (e.g., `|| Resource == "dataset"`) has **no tag enforcement** at that level. Only use pass-throughs for levels where you intentionally want to skip tag checks. All patterns below list conditions in gate chain order: dataset → table → attribute.
:::
**1. Enforce tags at ALL levels (dataset, table, and attribute), recommended for full access control:**
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "attribute")
)
```
This checks tags at every level. A user with `level 1` + `level 2` can access the dataset, tables tagged up to `level 2`, and attributes tagged up to `level 2`. Attributes tagged `level 3` will be denied.
**2. Enforce tags at table level only (dataset and attribute pass-through):**
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
:::info
Use this when you only tag tables and don't need dataset-level or attribute-level restrictions. Note: attribute-level tags are **not enforced** with this pattern.
:::
**3. Grant access to tables that do NOT have a specific tag:**
```text
Service == "adac" && (
Resource == "dataset"
|| (!has(PolicyTags.sensitivity) && Resource == "table")
|| Resource == "attribute"
)
```
**4. Enforce at dataset level only (all tables and attributes pass through):**
```text
Service == "adac" && (
(PolicyTags.sensitivity == "high" && Resource == "dataset")
|| Resource == "table"
|| Resource == "attribute"
)
```
:::info
Use this when you only need to gate access at the dataset level. All tables and attributes within the dataset are accessible if the dataset gate passes.
:::
**5. Enforce at dataset and table levels (attribute pass-through):**
```text
Service == "adac" && (
(PolicyTags.department == "finance" && Resource == "dataset")
|| (PolicyTags.sensitivity == "high" && Resource == "table")
|| Resource == "attribute"
)
```
:::info
Attribute-level tags are **not enforced** with this pattern.
:::
**6. Enforce at attribute level only:**
```text
Service == "adac" && (
Resource == "dataset"
|| Resource == "table"
|| (PolicyTags.sensitivity == "high" && Resource == "attribute")
)
```
**7. Combine multiple tag values (user needs any of the listed values):**
```text
Service == "adac" && (
Resource == "dataset"
|| (PolicyTags.sensitivity in ["high", "medium"] && Resource == "table")
|| Resource == "attribute"
)
```
:::info
For more information on CEL syntax, operators, and the visual condition builder, see [Roles and Conditions, CEL Conditions](https://docs.dataplatform.ovh.net/iam-roles-conditions.md#new-cel-conditions).
:::
## Best Practices
- **Always tag at the dataset level first.** Since the gate chain starts at the dataset, an untagged dataset with tagged tables creates a confusing setup. Set a baseline tag on the dataset.
- **Use Resource-scoped CEL conditions.** Never rely on the default CEL generated by the Grant Access UI for production access control. Always edit the CEL to include `Resource` scoping.
- **Order CEL checks as dataset → table → attribute.** This matches the gate chain evaluation order and makes conditions easier to read and debug.
- **Keep tag structures simple.** Use a single tag key (e.g., `sensitivity`) with a small number of values (e.g., `public`, `internal`, `restricted`). Avoid deeply nested tag combinations.
- **Grant cumulative tags.** Remember that users need tags for every level in the chain. If a user needs `restricted` table access, they also need whatever tag is on the parent dataset.
- **Test with a dedicated user.** Before rolling out access policies, create a test user and verify each level of the gate chain independently. See [Testing Access Control](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-testing.md).
- **Audit CEL conditions regularly.** As policy tags change, CEL conditions on role bindings may become stale. Review them when modifying tag bindings.
- **Document your tag schema.** Keep a record of which tag keys and values are in use, what they mean, and which resources they're bound to.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-testing.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Testing & Troubleshooting
## Objective
This page covers how to test policy tag access control and troubleshoot common errors. For an overview of policy tags and the gate chain, see [Policy Tags Overview](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md). For CEL condition patterns, see [CEL Conditions & Access Patterns](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md).
## Table of Contents
1. [Testing Access Control](#testing-access-control)
2. [Troubleshooting and Common Errors](#troubleshooting-and-common-errors)
## Testing Access Control
This walkthrough uses the dataset from the [Getting Started tutorial](https://docs.dataplatform.ovh.net/landing-page-getting-started.md) and the `access_level` policy tag.
### Prerequisites
You need a user account with the following roles assigned:
1. **Organization Level**: `Organization Viewer`: Allows the user to view the platform and projects.
2. **Project Level**: `AM Admin`: Grants permissions to access the Explorer and run SQL queries.
### Setup
Bind the following policy tags:
- **Dataset level:** `access_level: level 1` on the entire dataset.
- **Table level:** `access_level: level 2` on the table `chicago_calendar_full`.
- **Attribute level:** `access_level: level 3` on the `bridge` attribute in `chicago_calendar_full`.
- **No Policy Tag:** The `stations_rides` table has no tag (it will inherit `level 1` from the dataset).
Verify the bindings in the policy tag menu:
### Test 1: User with `level 1` only
Grant the user `access_level: level 1` using the [Grant Access UI](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#using-the-grant-access-ui).
Then, edit the CEL condition in the IAM to enforce tags at **all three levels**:
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level == "level 1" && Resource == "table")
|| (PolicyTags.access_level == "level 1" && Resource == "attribute")
)
```
This CEL checks `level 1` at every level. Since the dataset has `level 1` and untagged tables/attributes inherit it, this user can access inherited resources but not `level 2` or `level 3` resources.
Open the [Explorer](https://docs.dataplatform.ovh.net/lakehouse-manager-explorer.md) in SQL mode as the test user.
**Show catalogs:**
```
SHOW CATALOGS
```
Expected: The dataset appears in the catalog list.
**Read an untagged table (inherits `level 1` from dataset):**
```sql
SELECT * FROM stations_rides
```
Expected: **Success**. The table has no tag, inherits `level 1` from the dataset, and the user's CEL matches `level 1` at all levels.
**Read the `level 2` table:**
```sql
SELECT * FROM chicago_calendar_full
```
Expected: **Error**. The user passes the dataset gate (`level 1`) but fails the table gate (table has `level 2`, CEL only checks for `level 1`).
### Test 2: User with `level 1` + `level 2`
Grant the user `access_level: level 2` in addition to `level 1`.
Update the CEL condition to include `level 2` at the table and attribute levels:
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "attribute")
)
```
This CEL enforces tags at all three levels. At the attribute level, it checks for `level 1` or `level 2`, but `bridge` has `level 3`, so it will be denied.
**Read the `level 2` table with SELECT \*:**
```sql
SELECT * FROM chicago_calendar_full
```
Expected: **Error**. The user passes the dataset and table gates, but `SELECT *` includes the `bridge` attribute which has `level 3`. The CEL only allows `level 1` or `level 2` at the attribute level.
**Read the table excluding the restricted attribute:**
```sql
SELECT date, public_holiday, christmas_holidays, weekend, sport_holidays, summer_holidays, month, week_day_label, autumn_holidays, spring_holidays, temperature, cloud_cover, humidity, week_day, wind_speed FROM chicago_calendar_full
```
Expected: **Success**. All selected attributes either have `level 1` or `level 2` tags (or inherit from the table), and the CEL matches.
**Read only the restricted attribute:**
```sql
SELECT bridge FROM chicago_calendar_full
```
Expected: **Error**: `bridge` has `level 3`, and the CEL only allows `level 1` or `level 2` at the attribute level.
### Test 3: Demonstrating that pass-throughs disable enforcement
To understand the difference between enforced and pass-through levels, try this CEL with the same user (`level 1` + `level 2`):
```text
Service == "adac" && (
(PolicyTags.access_level == "level 1" && Resource == "dataset")
|| (PolicyTags.access_level in ["level 1", "level 2"] && Resource == "table")
|| Resource == "attribute"
)
```
Note the **attribute level is now a pass-through** (`|| Resource == "attribute"`).
```sql
SELECT bridge FROM chicago_calendar_full
```
Expected: **Success**, even though `bridge` has `level 3` and the user doesn't have `level 3`, the CEL does not check tags at the attribute level. The pass-through means all attributes are accessible.
:::warning
This test demonstrates why pass-throughs must be used carefully. If you need attribute-level restrictions, the CEL **must** check tags at `Resource == "attribute"`.
:::
### Further testing
Try these variations:
- Grant `level 3` to the user (updating the CEL to include `level 3` at all levels) and verify `SELECT *` works on `chicago_calendar_full`.
- Remove `level 1` from the user's CEL and verify they **cannot access anything**, not even the `level 2` table, because the dataset gate requires `level 1`.
- Try accessing a table with only `level 2` (no `level 1`) to confirm the gate chain blocks at the dataset level.
## Troubleshooting and Common Errors
### User cannot access a table despite having the correct tag
**Most common cause:** The CEL condition is missing Resource scoping. The Grant Access UI generates a CEL without `Resource` constraints, so the tag check runs at every level of the gate chain. If the tag is only on the table (not the dataset), the dataset-level check fails.
**Fix:** Edit the CEL condition in the IAM to include Resource scoping. See [Writing CEL Conditions Manually](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags-cel-conditions.md#writing-cel-conditions-manually).
### User has the table-level tag but gets denied
**Cause:** The gate chain is cumulative. The user must also satisfy the dataset-level tag. Having only the table tag is not sufficient. The dataset gate must pass first.
**Fix:** Ensure the user's CEL covers all levels in the chain, or grant the user the dataset-level tag as well.
### `SELECT *` fails but selecting specific columns works
**Cause:** One or more attributes in the table have a higher-level policy tag that the user doesn't have. `SELECT *` includes all attributes, so the restricted attribute's gate fails.
**Fix:** Explicitly list the columns the user has access to, excluding the restricted attribute(s). Or grant the user the required attribute-level tag.
### User can see the dataset in `SHOW CATALOGS` but cannot query any tables
**Cause:** The dataset-level gate may pass, but every table in the dataset has tags that the user doesn't have.
**Fix:** Verify the user has tags matching both the dataset and the target table(s). Remember that untagged tables inherit the dataset's tag.
### CEL condition uses `PolicyTags.
- You can also type a new path, and the system will create the necessary folders if they don't already exist.
3. **Run the Action:** Once both your source and destination are configured, click the "Run" button to initiate the file transfer.
## Verify Your Transfer
After the _File Transfer_ action completes successfully, you can visually confirm the changes by navigating to the **Buckets** section within the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). Here, you will see the copied files and folders appear in your destination bucket, reflecting the structure preserved from the source.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-load-to-bucket.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Load to Bucket Action
## Objective
The _Load to Bucket_ action **extracts raw data from sources and puts it into a file** in your Project bucket. You can choose the file format.
:::info
**Note:** This action supports the functionality to have customised date in the folder name. You can find the tutorial [here](https://docs.dataplatform.ovh.net/tutorials-dated-export-folders.md).
:::
## Configure a Load to Bucket action
To create a new action, navigate to the **Action tab** of the Data Processing Engine (DPE), click on **New Action** and select the _Load to Bucket_ action.
### Choose a source and a destination
To set-up the action, start by selecting a source from the list that you have already set-up in the [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) tab of the Connectors.
Then, choose the _destination bucket_ from the list of [Project buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
You will then be prompted to choose the file format and the destination folder. If you leave the folder name empty, it will automatically be named as the source name.
:::warning
Note that **the following sources are NOT supported**
in this action: Azure Blob Storage, Data Store, Dropbox, FTP, Google Drive, HTTP Files, HTTP REST, One Drive, S31
-compatible-storage, Amazon S3 and SFTP.
:::
:::info
If you choose a **PostgreSQL source** your action will be have optimised performance similarly to the [PostgreSQL to Parquet action](https://docs.dataplatform.ovh.net/dpe-actions-postgresql-to-parquet.md).
:::
### Destination file
Your data will be put into the destination folder in a file named _part-1_ with the format you specified. If your file is too big, it will be split into multiple files named _part-1_, _part-2_, _part-3_ and so on.
The order of your rows will be preserved in this process. Hence, the first rows will be in the _part-1_ file and the following rows will be in the _part-2_ file.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
1
: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.
---
url: https://docs.dataplatform.ovh.net/dpe-actions-export-table-to-source.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Export Table to Source Action
## Objective
The _Export Table to Source_ action allows you to **export one or more tables from your Lakehouse Manager directly to a compatible Connectors source.** This action facilitates moving processed or raw data from your managed Lakehouse environment into external databases or other systems supported by Connectors connectors.
## Configure an Export Table to Source action
To create a new _Export Table to Source_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"Export Table to Source"**.
### Choose Tables for Export
First, you need to select the tables from your Lakehouse Manager that you wish to export.
1. A list of available tables from your [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-tables.md) will be displayed.
2. You can select **one or multiple tables** from this list for export. Simply click on the table names to highlight them for selection.
### Select Destination Source
Next, you will define the external source where the selected tables will be exported.
1. A dropdown or list will appear, showing available compatible sources from your [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md).
2. **Compatibility Note:** Currently, this action supports Connectors sources of **type database and protocols**.
3. **Important:** If no destinations are shown, it means you have not yet connected any compatible database or protocol sources in your Connectors. Please ensure you have configured at least one such source before attempting to use this action.
4. Select the desired destination source from the list.
5. click confirm and run.
:::warning
**Important Note on Export Table to Source:** The _Export Table to Source_ action might **fail** if the destination database you are exporting to has a configuration requiring primary keys for tables being created or inserted. For example, some database systems or specific table settings (like `REQUIRE PRIMARY KEY` in certain MySQL/MariaDB setups or stricter NoSQL configurations) might enforce this.
:::
:::info
By default, tables within the Lakehouse Manager **do not inherently have primary keys defined**. If your destination database requires a primary key and the incoming table schema does not provide one, the export operation will fail. To successfully export to such databases, you may need to adjust the destination database's configuration to temporarily or permanently relax the primary key requirement.
:::
## Verify Your Export
After the _Export Table to Source_ action completes successfully, you can verify the export by connecting to your chosen destination database or system (e.g., using a SQL client for a database) and checking for the presence of the newly exported tables and their data. You can even check the connectors source to check for the update.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-csv-to-postgresql.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# CSV to PostgreSQL Action
## Objective
The _CSV to PostgreSQL_ action allows you to **ingest data from a CSV file (sourced from any compatible connection in your Connectors) into a table within a PostgreSQL database.** By default, this action appends data to an existing table, with an explicit option to entirely overwrite its contents.
## Configure a CSV to PostgreSQL action
To create a new _CSV to PostgreSQL_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"CSV to PostgreSQL"**.
### Choose a CSV File for Transfer
First, you need to select the specific CSV file you wish to transfer. This list will comprise CSV files accessible from **all compatible sources** configured in your Connectors.
1. You will be presented with a unified list of available CSV files. These files are drawn from all [Connectors sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) that contain CSV data.
2. Browse or search through this list to locate the CSV file you want to transfer.
3. Select the desired CSV file.
4. **Choose Delimiter (Optional):** You can specify the delimiter used in your CSV file. By default, the system will attempt to **auto-detect** the delimiter. Alternatively, you can explicitly select one of the following common delimiters: `,` (comma), `;` (semicolon), or `|` (pipe).
### Select Destination PostgreSQL Table and Options
Next, you will define the target PostgreSQL table and how the data should be loaded. You will see a consolidated list of all tables from all connected PostgreSQL sources.
1. **Select Destination Table:** You will see a comprehensive list of all PostgreSQL tables available across all your connected PostgreSQL [Connectors sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md).
- For easier identification, the **name of the PostgreSQL source** will be mentioned below each table name.
- **Limitation:** Currently, this action supports pushing data only into **existing tables** in your PostgreSQL database.
- **To load into a new table:** You must first create an empty table in your PostgreSQL database (using your preferred PostgreSQL client or tools) and then select it from this list.
2. **Loading Option:** By default, the action will **append** the new data from your CSV file to the selected PostgreSQL table.
- You will also have an explicit option to **Overwrite** the table. If selected, this will completely delete all existing data from the chosen PostgreSQL table before inserting the new data from your CSV file. Use this with extreme caution, as it will erase all previous data in that table.
3. **Run the Action:** Once your CSV file, destination PostgreSQL table, and loading option are configured, click the "Run" button to initiate the transfer.
## Verify Your Transfer
After the _CSV to PostgreSQL_ action completes successfully, you can verify the transfer by connecting to the respective PostgreSQL database (e.g., using a SQL client) and querying the destination table to confirm the presence and correctness of the loaded CSV data.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-mysql-to-parquet.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# MySQL to Parquet Action
## Objective
The _MySQL to Parquet_ action provides a streamlined way to **transfer a table from a MySQL source, configured in your Connectors, directly into a Project bucket.** The data from your MySQL table will be stored in a highly efficient Parquet file format within your chosen object storage bucket.
## Configure a MySQL to Parquet action
To create a new _MySQL to Parquet_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"MySQL to Parquet"**.
### Choose a MySQL Table
First, you need to select the specific MySQL table you wish to transfer.
1. A list of available tables from your connected MySQL sources in the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) will be displayed.
2. **Important:** If this list is empty, it means you do not have any MySQL sources connected or configured in your Connectors. Please ensure you have added and connected your MySQL database as a source before attempting to use this action.
3. Select the desired MySQL table from this list.
### Select Destination Bucket and Path
Next, you will define the destination bucket and the folder path where your Parquet file(s) will be stored.
1. **Choose Destination Bucket:** Select an available [Project bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) from the list where your Parquet file(s) will be stored.
2. **Specify Folder Path:** In the field below the destination bucket selection, you will specify the desired folder path within the bucket.
- **Example:** If you want your data to be stored in a folder named `my_mysql_data` inside a `raw_data` directory, you would enter `raw_data/my_mysql_data/`.
- The system will automatically create any necessary folders if they don't already exist in the specified path.
- **File Naming Convention:** The exported data will always be stored in one or more files named `part-1.parquet`, `part-2.parquet`, and so on, depending on the size of your table.
- **Default Behavior:** If you leave the folder path empty, the action will, by default, create a folder named after the MySQL table you selected (e.g., if your table is named `users`, it will create a `users/` folder at the root of the bucket). Within this default folder, the data will be stored as `part-1.parquet`, `part-2.parquet`, etc.
3. **Run the Action:** Once both your MySQL table and destination are configured, click the "Run" button to initiate the transfer.
## Verify Your Transfer
After the _MySQL to Parquet_ action completes successfully, you can visually confirm the transfer by navigating to the **Buckets** section within the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). Here, you will find your new Parquet file(s) at the specified path within your destination bucket.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-postgresql-to-csv.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PostgreSQL to CSV Action
## Objective
The _PostgreSQL to CSV_ action allows you to **transfer a table from a PostgreSQL source, configured in your Connectors, directly into a Project bucket.** The data from your PostgreSQL table will be stored in a comma-separated values (CSV) file format within your chosen object storage bucket.
## Configure a PostgreSQL to CSV action
To create a new _PostgreSQL to CSV_ action, follow these steps within the Data Processing Engine (DPE):
1. Navigate to the **Action tab** within the Data Processing Engine (DPE) interface.
2. Click on the **"New Action"** button.
3. From the list of available actions, select **"PostgreSQL to CSV"**.
### Choose a PostgreSQL Table
First, you need to select the specific PostgreSQL table you wish to transfer.
1. A list of available tables from your connected PostgreSQL sources in the [Connectors](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) will be displayed.
2. **Important:** If this list is empty, it means you do not have any PostgreSQL sources connected or configured in your Connectors. Please ensure you have added and connected your PostgreSQL database as a source before attempting to use this action.
3. Select the desired PostgreSQL table from this list.
### Select Destination Bucket and Path
Next, you will define the destination bucket and the folder path where your CSV file(s) will be stored.
1. **Choose Destination Bucket:** Select an available [Project bucket](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) from the list where your CSV file(s) will be stored.
2. **Specify Folder Path:** In the field below the destination bucket selection, you will specify the desired folder path within the bucket.
- **Example:** If you want your data to be stored in a folder named `my_postgres_data` inside an `analytics` directory, you would enter `analytics/my_postgres_data/`.
- The system will automatically create any necessary folders if they don't already exist in the specified path.
- **File Naming Convention:** The exported data will always be stored in one or more files named `part-1.csv`, `part-2.csv`, and so on, depending on the size of your table.
- **Default Behavior:** If you leave the folder path empty, the action will, by default, create a folder named after the PostgreSQL table you selected (e.g., if your table is named `products`, it will create a `products/` folder at the root of the bucket). Within this default folder, the data will be stored as `part-1.csv`, `part-2.csv`, etc.
3. **Run the Action:** Once both your PostgreSQL table and destination are configured, click the "Run" button to initiate the transfer.
## Verify Your Transfer
After the _PostgreSQL to CSV_ action completes successfully, you can visually confirm the transfer by navigating to the **Buckets** section within the [Lakehouse Manager](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md). Here, you will find your new CSV file(s) at the specified path within your destination bucket.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-postgresql-to-parquet.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# PostgreSQL to Parquet Action
## Objective
The _PostgreSQL to Parquet_ action extracts data from a PostgreSQL database and puts it into a [Parquet file](https://parquet.apache.org/) in your Project bucket. You can choose the file format.
:::info
**Note:** This action supports the functionality to have customised date in the folder name. You can find the tutorial [here](https://docs.dataplatform.ovh.net/tutorials-dated-export-folders.md).
:::
## Configure a PostgreSQL to Parquet action
To create a new action, navigate to the **Action tab** of the Data Processing Engine (DPE), click on **New Action** and select the _PostgreSQL to Parquet_ action.
### Choose a source and a destination
To set-up the action, start by selecting a source from the list that you have already set-up in the [Sources](https://docs.dataplatform.ovh.net/landing-page-connectors-sources.md) tab of the Lakehouse Manager.
Then, choose the _destination bucket_ from the list of [Project buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md).
### Destination file
You will then be prompted to choose the destination folder. If you leave the folder name empty, it will automatically be named as the source name.
Your data will be put into the destination folder in a file named _part-1_. If your file is too big, it will be split into multiple files named _part-1_, _part-2_, _part-3_ and so on.
The order of your rows will be preserved in this process. Hence, the first rows will be in the _part-1_ file and the following rows will be in the _part-2_ file.
:::info
Note that this action has its **transfer speed optimised** for the PostgreSQL source.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-diff.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Convert cumulative values to daily deltas with the Diff action
## Objective
When you have tables which contain dated attributes describing total values (total daily orders, total monthly revenue, etc.) at a given time, the diff action lets you compute the difference of these values between consecutive time increments.
For example, let's consider the _total\_fans_ attribute in the table below calculated each day for a certain social media profile page. Assuming the _total\_fans_ describes the total number of fans since the creation of this profile, the **diff** action will let you compute the number of new fans acquired or lost between each day.
Please note that first dates of each ids won't have their first incremental value computed, as the previous date is not known.
There is an optional advanced parameter if you have the date of creation so that the first incremental date will be computed as equal to the total attribute.
## Action Configuration
### Action's configuration interface
When configuring the **diff** action from the interface, you will be asked to specify:
- the **table name** which contains the total value
- the **timeline attribute** which will be used to get the time increments between 2 total values
- the **measure** to compute, there can be several of them for a given table, for each specify the **cummulative value attribute** and the **incremental value attribute**
The primary keys of the table are automatically assigned.
### Action's example JSON configuration
Like any other DPE actions, you can also configure the action in "Advanced Mode" by editing its JSON configuration, a snippet of which is shown below.
```json
{
"table": "dwh/default_dataset/posts",
"attribute_timeline": "date",
"attribute_total": ["total_impressions"],
"attribute_diff": ["incr_impressions"],
"table_attribute_created_at": "ref_posts",
"attribute_created_at": "post_createdat",
}
```
### Action's Parameters
Below is a short description of all available attributes in the JSON configuration.
| Parameter | Description | Example | Required |
| :-------------------------------: | ----------------------------------------------------- | ----------------------- | -------- |
| **table** | Name of the table on which to run the diff action | Dwh/agr\_site\_date | Yes |
| **attribute\_timeline** | Name of the attribute of timeline | date | Yes |
| **attribute\_total** | List of attribute names with totals. | \["total\_impressions"] | Yes |
| **attribute\_diff** | List of attribute names with incremental values. | \["incr\_impressions"] | Yes |
| **table\_attribute\_created\_at** | Optional table containing date of creation of ids | ref\_posts | No |
| **attribute\_created\_at** | Optional attribute containing date of creation of ids | post\_createdat | No |
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-delete.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Empty tables with the Delete action
## Objective
This action is used to empty one, or multiple, tables in the database. Using the perimeter in the preferences section, you can limit the deletion to a specific subset of data. The perimeter can be preformed on:
- Based on predefined set (A table attribute)
- Date perimeter of the workflow
- Based on attribute of the Project (An attribute of the Project)
:::info
_Pro-tip_
The perimeter type _Based on predefined set_ can have multiple values, but only one attribute can be selected. Whilst unorthodox, you can use segmentation to act as an additional attribute based on a table attribute.
:::
## Setup example
```json
{
"params": {
"tables": [
"dwh/default_dataset/table_1",
"dwh/analytics_dataset/table_2"
]
}
}
```
## Parameters
| Parameter | Description | Example | Required |
| :--------: | ----------------- | ---------------------------------------------------------------- | -------- |
| **tables** | Tables to delete. | `["dwh/default_dataset/table1", "dwh/analytics_dataset/table2"]` | Yes |
\*This parameter should be used carefully, as if the action terminates early, it will result in partial deletion of data, and if multiple tables are selected, some tables may not be deleted at all.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-delete-diff.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Delete Diff Action
## Objective
This action is used to delete obsolete entries that existed at one time but no longer exist. Compatible with all Data Platform data engines.
## Usecase example
An erroneous data entry was made to a source file. The DPE automatically inserted the entry into the primary and aggregate tables. Once the incorrect entry was discovered, it was deleted from the source, however, it still remains in the primary and aggregate tables.
## How to use it
When you need to delete items that no longer exist, we recommend creating a temporary table that will serve as a work zone, containing only the data you wish to keep. This will allow you to set up a perimeter on the date field which allows you to act only for a given duration. If you have to act on all the data, then the easiest way is not to use delete diff, but to simply empty the tables beforehand with flush tmp and to fill it with the new data via the source.
Create a delete diff action for each table you wish to remove the records from, using the temporary table as the reference table. Select the attribute you wish to preform the comparison on. Optionally, set a date perimeter via the preferences section.
## Advanced mode example
```json
{
"diffToCheck" : [
{
"tablecheck": "dwh/default_dataset/testfile",
"tabletodelete": "dwh/default_dataset/testfile2",
"uniqFields": "attr1,attr2"
}
]
}
```
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-delete-bucket.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Delete Bucket Action
## Objective
The _Delete Bucket_ action allows you to **remove data from a bucket, including both folders and files.** You can choose to delete the entire bucket contents or target specific paths for deletion.
## Configure a Delete Bucket action
To create a new action, navigate to the **Action tab** of the Data Processing Engine (DPE), click on **New Action** and select the _Delete Bucket_ action.
### Choose items for deletion
Upon selecting the _Delete Bucket_ action, you will be presented with a list of your [Project buckets](https://docs.dataplatform.ovh.net/lakehouse-manager-buckets.md) on the left side of the screen. This panel allows you to browse and select the data you wish to delete.
1. **Select the target bucket:** Locate the bucket from which you want to delete data.
2. **Expand bucket contents (optional):** To see the folders and files inside a bucket, click on the **name of the bucket**. This will expand the view, showing its contents.
3. **Choose items for deletion:**
- To delete an **entire bucket** and all its contents, click the **right arrow icon** (▶) next to the bucket's name.
- To delete a **specific folder** (and all its contents), navigate into the bucket and click the **right arrow icon** (▶) next to the desired folder's name.
- To delete an **individual file**, navigate into the bucket and folder where the file resides, then click the **right arrow icon** (▶) next to the file's name.
4. **Review selected items:** As you click the right arrow, the selected buckets, folders, or files will be moved to the right column. This column indicates all items that are currently marked for deletion.
Carefully review the items in the right column to ensure you are deleting only what is intended. Once satisfied with your selection, proceed with the final steps as prompted by the DPE interface to confirm the deletion. **Be aware that deletion actions are permanent and cannot be undone.**
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-rebuild.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Rebuild a table from scratch with the Rebuild action
## Objective
The _Rebuild_ action **entirely rebuilds an existing table** by creating a new physical table with the same schema and inserting all of its data into it.
It is mainly used to change field properties that cannot be modified once a table is built, most notably adding a **Required** or **Required + Identifier** constraint to an attribute (see [attribute constraints](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints)).
## Why rebuild a table
Rebuilding a table has several benefits:
- **Change field properties** such as _Required_, which [cannot be done once a table is built](https://docs.dataplatform.ovh.net/lakehouse-manager-manage-tables.md#attribute-constraints).
- **Restructure partitioned data.** If the table uses partitions, all data is rewritten using those partitions. Note that when partitions are added to an existing table, only new data is stored under those partitions. The Rebuild action lets you properly restructure _all_ of the data.
- **Clean metadata and storage.** All metadata and history is removed to keep a single clean version of the table. This can reduce storage and improve performance, as Iceberg optimizes the way it stores data.
:::warning
The Rebuild action removes **all of the table's history**. Once it runs, you can no longer use the [Time Travel](https://docs.dataplatform.ovh.net/lakehouse-manager-time-travel.md) feature to roll back to a previous version. It is a fresh start.
:::
## Configure a Rebuild action
To create a new _Rebuild_ action, navigate to the _Actions_ tab of the Data Processing Engine (DPE), click on **New Action**, and select the _Rebuild_ action. Then select the table to rebuild and adjust its field configuration (required flag, identifier flag, and default value).
### Default values
When changing an attribute's property to **Required** or **Required + Identifier**, you must set a **default value**. This value replaces any NULL values found for that attribute during the rebuild, ensuring the new required constraint is satisfied.
### Keep backup table
The **Keep backup table** option keeps a backup of the previous table, _with its history_. The backup table is named with the prefix `_back_{hash}` (for example, `my_table_back_d26433d4`). You can find this name in the execution logs.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-actions-deduplicate.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Remove duplicate rows with the Deduplicate action
## Objective
The _Deduplicate action_ removes duplicate rows from a table, **in place**, keeping exactly one row per **deduplication key**. The typical use case is a table that was loaded in _append_ mode when _upsert_ was intended: deduplicating on the table's identifier fields produces exactly the table that upsert would have produced. Like any other action, it can be run on demand, scheduled with a cron trigger, or orchestrated inside a [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md).
The action works on tables stored in **lakehouse (Iceberg)** and **PostgreSQL** datasets.
:::warning
Deduplicate modifies the target table itself. On lakehouse tables the previous state remains available as a snapshot, so a run can be undone with time travel. On PostgreSQL tables the removal is **permanent**: enable the backup option before deduplicating anything you are not ready to lose.
:::
## Configure a Deduplicate action
The configuration screen has two panels: **Table configuration** on the left, where you pick the table, the deduplication key and the row to keep, and **Backup & scan** on the right, where you protect the run and preview its effect before creating the action.
### Choose the table
Select the target table from the **table selector**. Each entry displays the table name together with the dataset it belongs to. The action name is prefilled from the selected table, for example _Deduplicate - orders_.
### Choose the deduplication key
The deduplication key is the set of columns that defines a unique row: after the run, the table contains exactly one row per distinct combination of these columns.
When the table declares **identifier fields**, they are displayed above the key selector and the key is prefilled with them. This is the recommended key: it deduplicates the table exactly the way an _upsert_ load would have deduplicated it on ingestion.
You can select any set of existing columns instead. If your selection differs from the table's declared identifier fields, a warning reminds you that rows which are not duplicates by the table's own definition may be removed. If the table declares no identifier field, pick the column or columns that define a unique row for your data.
:::info
Rows whose key columns are empty group together, following standard SQL `GROUP BY` semantics: two rows with NULL in the same key column count as duplicates of each other.
:::
### Choose the row to keep
When several rows share the same key, the **Row to keep** setting decides which one survives:
- **Any remaining row** (default): the survivor is arbitrary. This is the right choice when duplicated rows are identical copies. If duplicated rows differ outside the key, the kept row is arbitrary.
- **Most recent / oldest entry**: deterministic. Choose **Most recent** or **Oldest**, then select the **Order by column**: a date, timestamp or numeric column that reflects recency, such as an update timestamp, a version number, or an auto-increment ID. Rows with an empty ordering column rank last: a row with a value always wins over a row without one.
### Keep a backup
Turn on **Keep a backup of this table** to create a full copy of the table before anything is removed. The copy is named `
2. **Start Configuration:** Click on _⚙️ Start Here_ within the chat panel to begin the configuration process.
3. **Define Completion Model:**
- **3.1 Completion Model:** Select `OpenAI (General Interface)`. This will reveal further configuration settings for the model ID, base URL, and API key.
- **3.2 Model ID:** Enter the ID of your desired model from the [AI Endpoints Catalog](https://endpoints.ai.cloud.ovh.net/catalog). For example, let's use `Qwen2.5-Coder-32B-Instruct`.
- **3.3 Base API URL:** Enter the base URL for your OVH AI Endpoints, such as `https://oai.endpoints.kepler.ai.cloud.ovh.net/v1/`.
- **3.4 API Keys:** Enter your OVH AI Endpoints API Key in the designated field.
4. **Save and Return:** Click _Save Changes_, and then click the back arrow to return to the Jupyter AI Chat panel. Your Jupyter AI extension is now configured and ready for use.
## Using Jupyter AI in your Notebook
Once your OVH AI Endpoint is configured with Jupyter AI, you can interact with it directly within your notebook environment using the Jupyter AI Chat Panel.
### Jupyter AI Chat Panel
The **Jupyter AI chat panel**, powered by _Jupyternaut_, enables conversational interaction directly within JupyterLab. This is your primary interface for interacting with your configured OVH AI Endpoint. You can use it to get programming help, generate notebooks, and much more with ease.
You can:
- Ask questions about your data or code using the chat box.
- Generate new Jupyter notebooks from text prompts using the `/generate` command.
- Fix errors in selected notebook cells using the `/fix` command.
- Export chat history to a Markdown file using the `/export` command.
- Teach Jupyternaut about files on your system using the `/learn` command.
- Use `@file` to include selected file contents in your question for more accurate responses, providing context to the AI directly from your files.
For more detailed information on all capabilities, explore the [Jupyter AI documentation](https://jupyter-ai.readthedocs.io/).
## Example Workflow: Generating a Machine Learning Notebook from Kaggle Hub Data
Let's demonstrate a powerful feature: using the `/generate` command to create a new notebook that fetches data, performs data exploration, and builds a basic machine learning model. For this example, we'll use a commonly available version of the Kaggle Titanic dataset, accessible via a direct URL.
1. **Ensure your Jupyter AI is configured** as described in the "Configure Jupyter AI with OVH AI Endpoints" section, using your chosen model (e.g., `Qwen2.5-Coder-32B-Instruct`).
2. **Open the AI Chat Panel** (if not already open) by clicking the "Chat" icon on the left sidebar.
3. **Type the `/generate` command** followed by your detailed prompt into the chat input field:
```text
/generate Create a Python notebook to perform binary classification on the Titanic dataset. The notebook should:
1. Download the 'train.csv' dataset from 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv' using pandas.
2. Load the downloaded data into a pandas DataFrame.
3. Display basic information (first 5 rows, data types, descriptive statistics).
4. Handle missing values (e.g., fill 'Age' with the mean, 'Embarked' with the mode, and drop the 'Cabin' column).
5. Encode categorical features ('Sex', 'Embarked').
6. Split the data into training and testing sets (80/20 split), with 'Survived' as the target variable.
7. Train a RandomForestClassifier model.
8. Evaluate the model using accuracy and a classification report.
```
4. **Press Enter** to send your prompt. Jupyter AI will process your request using the configured OVH AI Endpoint. After a short time, it will generate a new `.ipynb` file in your current directory (e.g., `Titanic Binary Classification Notebook.ipynb`), pre-populated with Python code cells performing the requested steps.
5. **Open the generated notebook** and run the cells. You can then modify, expand, or debug the code as needed, using the Jupyter AI chat panel for further assistance. This significantly accelerates your data science workflow.
## Troubleshooting
- **`API Key not found` or `Authentication Error`:** Double-check the API Key entered during Jupyter AI configuration. Ensure it is correct and has the necessary permissions for your OVH AI Endpoint.
- **`Could not connect to the model` or `Bad URL`:** Verify the **Base API URL** provided during Jupyter AI configuration. Ensure the URL is correct and your OVH AI Endpoint is running and accessible.
- **No AI panel/button in JupyterLab:** If you don't see the AI panel or button, try refreshing your browser tab or restarting the Jupyter server. While Jupyter AI is pre-installed, sometimes a fresh load helps.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-environments.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Reuse execution configurations with environments
## Objective
Environments allow you to save and re-use quickly a set of [configurations and execution preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) in actions and workflows.
_Figure 1: The environment panel_
Environments are not permanent. At any time you can [change a jobs environment](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) even if one had already been applied.
:::info
If you make changes to any of the environments parameters, the changes will be reflected in all the actions and workflows to which the environment is assigned.
:::
- [Create an environment](https://docs.dataplatform.ovh.net/dpe-environments.md#create-an-environment)
## Create an environment
When you first access the **Environment** tab, you will be greeted with a blank canvas for you to fill your custom environments.
Click on **New environment** and a reduced version of the [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) section will appear.
From here, select your preferred _resources_, _options_ and _environment variables_. Don't forget to give your environment a name and a description.
[How to configure the execution preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Data Processing Engine jobs
## What is a job?
A job is a single execution of an action or a workflow. Every job carries its own execution environment: the [resources](https://docs.dataplatform.ovh.net/dpe-jobs-resources.md) allocated to it, its [segmentation](https://docs.dataplatform.ovh.net/dpe-jobs-segmentation.md) and [perimeter](https://docs.dataplatform.ovh.net/dpe-jobs-perimeter.md) settings, and its environment variables. Those come from the [preferences](https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md) of the action or workflow, or from a shared [environment](https://docs.dataplatform.ovh.net/dpe-environments.md).
The Data Processing Engine is job-based. When a run is triggered, the platform builds a container matching the environment defined, executes it, then shuts it down, so resources are consumed only for the duration of the run.
The Jobs screen shows three states.
**Running** jobs stream their logs live, and the number that can run at once is capped by the project [quotas](https://docs.dataplatform.ovh.net/dpe-settings.md#quotas).
**Queued** jobs are runs ordered beyond that cap. Queuing is deactivated by default and released on request through support. Queued jobs start in the order shown on screen, top to bottom, as each running job finishes.
**Finished** jobs keep 30 days of history, each with per-run CPU and RAM statistics, and a result of successful, failed, or manually interrupted. A job that exceeds its timeout is flagged as such, to distinguish it from other failures. Runs of PySpark actions, and of the workflows containing them, open directly in the [Spark History Server](https://docs.dataplatform.ovh.net/dpe-spark-history-server.md).
Segmentation and perimeter carry vocabulary of their own. The two references below define the terms as they appear in the interface and in advanced mode.
[### Resources→
Size a job vertically, or spread it across workers.
](/dpe-jobs-resources)[### Segmentation→
Split one job into parallel tasks by date, value or account.
](/dpe-jobs-segmentation)[### Perimeter→
Filter the data a job processes before it runs.
](/dpe-jobs-perimeter)[### Execution preferences→
Timeouts, service account, write strategy, triggers and execution modes.
](/dpe-jobs-preferences)
## Segmentation and perimeter vocabulary
:::details Segmentation and perimeter vocabulary
**Segments**:
Subset of a data source. Segments for row / column sources can be defined based on a fixed number of lines, an attribute of the source (a date or a value) or a user account (for account based sources).
**Segmentation / To segment**:
Action to split a DPE action into multiple tasks that handle one or more segments.
**Segmentation values**:
Referenced values used to divide an action into smaller tasks.
**Segmentation type**:
Technical name in advanced mode: _segmentationValues_. Determines the process according to which the list of segmentation values are retrieved.
**Source Attribute / (Segmentation Variable)**:
Technical name in advanced mode: _segmentationVarName_. Determine which column in the source is used to apply the filter to the segmentation values
**Perimeter**: Filter the source of a DPE action.
**Perimeter type**: Technical name in advanced mode: _perimeterValues_. Determines how the list of perimeter values is retrieved.
**Perimeter values**: List of values to apply the scope.
**Source Attribute / (Perimeter Variable)**: Technical name in advanced mode: _perimeterVarName_. Determines which column in the source is used to apply the filter to the perimeter values.
**Bucket Size**:
Technical name in advanced mode: _segmentationChunkSize_. The number of segmentation values that each task handles.
:::
## Bucket size in detail
:::details Bucket size in detail
At any time it is possible to **group some tasks in buckets (chunk) to manage several segments of the data source**. When the list of segments (values, date or social network accounts) is consistent, it avoids generating too many tasks.
The bucket size sets how many segments each task handles:
- If you set the bucket size to 2, each task will handle 2 segments
- If you set the bucket size to 10, each task will handle 10 segments
- Etc.
Below is an illustration of the impact of setting the bucket size to 30 compared to a bucket size of 1. Assume we’re running the segmentation based on the date of the workflow which runs from Jan 1st of 2018 to Dec 31st of 2019.
| w/out Segmentation Chunk Size | with Segmentation Chunk Size |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ChunkSize = 1 | ChunkSize = 30 |
| 
| 
|
In the first case the DPE action will be split in 730 tasks (365 x 2), one for each day for 2 years. In the second case, the bucket size is set to 30, in which case the DPE action will only be split in 25 tasks (365 x 2 / 30) as each task handles 30 days.
:::
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/dpe-jobs-preferences.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Configure job execution preferences
## Objective
Each [action](https://docs.dataplatform.ovh.net/landing-page-dpe-actions.md) or [workflow](https://docs.dataplatform.ovh.net/dpe-workflows.md) in the Data Processing Engine has several configuration settings which can be managed via the **Preferences** tab.
When an _action_ is executed, it will use those preferences to run.
When a _workflow_ is executed, its preferences will **override** the preferences of all the actions it contains. If a certain preference option is deactivated at workflow level (such as segmentation or perimeter) it will fall back to each action's preferences during the workflow's execution.
Those preferences can be templated to be re-used in one click every time by saving them in an [environment](https://docs.dataplatform.ovh.net/dpe-environments.md).
## Environments
Environments on the Platform are a set of pre-defined preferences that you can quickly assign to your actions or workflows, so you don't need to configure it every time.
[
| 
| 
|
Assuming, you want to aggregate “income” data per the date of orders “date\_order”. Let's say that the “tmp\_date” table serves as the reference table of all dates during which there was ever an order made, e.g. it contains a list of dates from 2019-01-01 to 2019-01-03.
By activating this segmentation mode, it will automatically generate a pre-stage to your main stage which contains your action, which will list the unique list of dates available in order to split your action in separate tasks as describe in the diagram below:
By doing so, the data from your Source Table “orders” will therefore be loaded to the Destination Table “aggr\_date” on day at a time.
### Compatibility
This segmentation option is only compatible with the following actions and sources:
| Action Types | Source Types |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Load; Aggregate; Diff; Delete\_Diff; Delete; Custom | **Databases**: MySQL, PostgreSQL, SQLServer, Impala, Hive, BiqQuery, ElasticSearch, Cassandra, Redshift, Oracle |
### Bucket Size
If your source contains a significant number of unique values within the attribute used for segmentation, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.
To avoid generating too many tasks, you can make each task manage several values of your segmentation attribute at once by using the “bucket size” option. Simply set a size for your bucket to decide how many values at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:
- If you do not set the bucket size, each task will handle 1 value
- If you set the bucket size to 2, each task will handle 2 values
- If you set the bucket size to 10, each task will handle 10 values
- And so on...
If you’re still unclear about what the “bucket size” does, find out more in the [bucket size reference](https://docs.dataplatform.ovh.net/landing-page-dpe-jobs.md#bucket-size-in-detail).
### Configuration
To configure this segmentation option select _Based on an attribute of a Project table_ in the "Segmentation Type" dropdown.
If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:
```json
"segmentation": {
"active": true,
"type": "dwh_attributes",
"var_name": "id",
"attribute": "name",
"object": "s3.deldiff/delete_diff.csv",
"attribute_type": "map",
"chuncksize": 2
}
```
[
- **Comparison**: a side-by-side diff of the resource _before_ and _after_ the action, so you can see exactly what changed.
## Organization-level auditing
From **Organization Settings**, open the **Auditing** tab to see organization-level activity.
Currently tracked at the organization level:
- Organization IAM (roles, policies, authentication providers)
- Member management (invitations, removals, role assignments)
Coming in upcoming releases:
- Billing activity
- Quotas
- Other organization-level settings
The organization-level view is scoped strictly to organization-level resources. For Project-internal activity, open the Project and use its own Auditing page.
## Security events
Both auditing pages have a **Security events** tab next to the audit trail. It lists the security events recorded at that scope. Click the **info icon** at the end of a row to open the event detail panel.
## Export
Click the **export** button above the event list to download the events currently listed as a file. The export is available on the audit trail and on the **Security events** tab, at Project and organization level. If an export fails, the error message tells you why.
## Roles
Auditing access is gated by two dedicated roles. The same two role names exist at **both levels** (organization and Project):
| Role | Capabilities |
| :--------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| **Audit Trail Admin** | Full management of auditing: view all events and configure audit settings at the scope of the role (organization or Project) |
| **Audit Trail Viewer** | Read-only access: view audit events at the scope of the role (organization or Project) |
A user needs the organization-level role to see the organization Auditing page, and the project-level role to see a Project's Auditing page. Assign them the same way as any other role from the [Users, Roles & Groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md) page.
## Search
Every audit view has a search field above the event list. Type any text to keep only the events that contain it, in any of their fields: a user name, a resource name, an action, or a value recorded in the event payload. The search combines with the [filters](#filters) below, so you can narrow to a service and a time window first, then search within the result.
## Filters
### Services (project level)
At the Project level, events can be filtered by service:
- **Connectors**
- **Lakehouse Manager**
- **Data Processing Engine**
- **Analytics Manager**
- **Identity Access Manager**
### Resources
Once a service is selected, the resource filter exposes the resource types available for that service. The full list of audited resource types is:
| Category | Resource types |
| :----------------------------------------- | :----------------------------------------------------------------------------------------------- |
| Lakehouse Manager & Connectors | Source, Table, Attribute, Rule, Dataset, Virtual Attribute, View, Policy Tag, Policy Tag Binding |
| Data Processing Engine & Analytics Manager | Query, Dashboard, Repository, Workflow, Action, Environment, Notebook, Application |
| Identity Access Manager | Role, Service Account, User, Group |
### Time window
Every audit view lets you scope events to a specific time window. You can pick one of the preset ranges or define your own:
- **Today**
- **Yesterday**
- **Last 1 hour**
- **Last 6 hours**
- **Last 24 hours**
- **Last 7 days**
- **Last 30 days**
- **Custom**: pick exact **From** and **To** dates and times on the calendar, with a selectable timezone.
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/landing-page-iam.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Identity Access Manager
## What is the Identity Access Manager?
The Identity Access Manager decides who can do what. It exists as two almost entirely independent modules: one for each [organization](https://docs.dataplatform.ovh.net/manage-organizations.md), one for each [project](https://docs.dataplatform.ovh.net/manage-projects.md).
[Organization IAM](https://docs.dataplatform.ovh.net/iam-organization.md) governs the organization itself: the [authentication settings](https://docs.dataplatform.ovh.net/iam-organization-authentication.md) for members signing in to Data Platform, single sign-on through an external provider, enforced multi-factor authentication, join requests, and the access rights members hold over organization-level resources and settings.
[Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md) governs one project, and reaches further. It covers the team members working inside the project and the end users of the applications it deploys. Access is role-based and granular: [roles](https://docs.dataplatform.ovh.net/iam-roles-conditions.md) are granted to users, to service accounts or to groups, and evaluated against project resources, down to individual table attributes where [policy tags](https://docs.dataplatform.ovh.net/lakehouse-manager-policy-tags.md) are involved. Project IAM also owns the login page of every deployed application and the authentication providers offered on it.
Three providers exist by default on every application: Data Platform IDs, Project IDs and API/Secret keys. Additional SSO or MFA providers are configured per project, and adding one means editing each application's authentication configuration before users see it in the login menu.
Authenticating against the platform programmatically, rather than through a login page, is covered by the [Authentication API](https://docs.dataplatform.ovh.net/developers-api-authentication.md).
Service accounts exist only in Project IAM. Every Data Processing Engine job runs as one, which is how a pipeline's permissions stay separate from those of the person who wrote it.
[### Organization IAM→
Members and their rights over organization-level resources.
](/iam-organization)[### Authentication settings→
SSO, enforced MFA and join requests for the organization.
](/iam-organization-authentication)[### MFA setup guide→
Enrol a second factor on your own account and recover it if you lose the device.
](/iam-organization-mfa-setup)[### Project IAM→
Access inside one project, for teams and for end users.
](/landing-page-iam-project)[### Users, roles and groups→
The principals in a project and the permissions they hold.
](/landing-page-iam-users)
## Go further
For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/en-gb/professional-services/) page to request a quote and have your project analysed by our experts.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud).
If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).
Join our [community of users](https://community.ovhcloud.com/).
---
url: https://docs.dataplatform.ovh.net/iam-organization.md
---
> For AI agents: the complete documentation index is available at https://docs.dataplatform.ovh.net/llms.txt, the full documentation bundle is available at https://docs.dataplatform.ovh.net/llms-full.txt.
# Manage organization authentication and access
## Objective
The Organization IAM is where you can manage all authentication requirements for members of your organization as well as their level of access to the organization's resources.
In the Organization IAM you will be able to:
- Set-up **authentication providers** to enable single-sign-on to your Data Platform Organization via external solutions.
- Enforce multi-factor authentication (MFA)
- Manage Organization members **access rights to the Organization-level resources and settings**.
You can **access this service from the sidebar**, as shown in the picture below:
## Resources managed by the Organization IAM
When using the Organization IAM, you will create roles and groups with permissions to access the following resources:
- [Billing Settings](https://docs.dataplatform.ovh.net/manage-pricing.md)
- [Control Center organization-level resources (alerts, logs, panels, consumers, etc)](https://docs.dataplatform.ovh.net/landing-page-control-center.md)
- [Organization IAM users, roles and groups](https://docs.dataplatform.ovh.net/landing-page-iam-users.md)
- [Organization's Projects deployed services and admins](#link-between-project-iam-and-organization-iam)
### Link between Project IAM and Organization IAM
While the Organization IAM provides manages access to projects within the organization, it does not allow management of access-rights of the resources within a project. In other words, the Organization IAM provides IT admins with a high-level access-rights management of the projects.
Below are the project-related permissions in the Organization IAM:
- _Projects | Project | Admin_: special permission that makes user become admin of the Project's IAM.
- _Projects | Project | List_: ability to view a list of existing projects in the organization and their basic information.
- _Projects | Project | Get_: ability to view basic information about a project
- _Projects | Project | Create_: ability to create a project.
- _Projects | Project | Update_: ability to update services inside a project.
- _Projects | Project | Delete_: ability to delete a project.
:::warning
Note that the **Project Admin permission is the only permission to overlap between both IAMs**. A user that has the _Project Admin_ permission in the Organization IAM has the _Admin_ permission in the underlying Project IAM(s).
:::
### Other Organization IAM permissions
Beyond the Project permissions, the access-rights to the following resources are managed in the Organization IAM via read, write and delete permissions:
- Organisation IAM
- _Users_
- _Roles_
- _Groups_
- _Authentication Settings_
- Control Center (organization-level)
- _Monitoring_
- _Alert_
- _Job_
- _Logs_
The only exception to this rule are the [datasets](https://docs.dataplatform.ovh.net/lakehouse-manager-datasets.md) whose permissions work slightly differently:
- **Create**: Create a dataset
- **Read**: See the dataset(s) of the organization
- **Update**: Change the amount of DPU allocated to the dataset(s)
- **Delete**: Delete the dataset(s)
## Authentication Settings
In the **Authentication Settings** tab, you can set-up & configure additional login requirements for all the users in the organization. To enforce security you can for instance delegate authentication to your third-party enterprise Single-Sign-On provider and require all users to verify their credentials on it to access the platform. Additionally, you can require all users to use multi-factor authentication to use projects or even allow users from a given email domain to request to join the organization.
[Authentication Settings](https://docs.dataplatform.ovh.net/iam-organization-authentication.md)
## Users, Groups and Roles
The Organisation IAM uses the same role-based access-rights control system of the [Project IAM](https://docs.dataplatform.ovh.net/landing-page-iam-project.md): permissions are granted by giving users roles and adding them to groups.
[