#What is the Data Processing Engine?

The Data Processing Engine builds and runs the data pipelines of a project, from raw sources through to the tables analysts query. It covers extraction and loading from a registered source, transformation of data already held in the project, and the scheduling and triggering around both.

Four objects make up the model:

• An action is one unit of work: load a table, aggregate it, run SQL, run custom Python or PySpark.
• A workflow chains actions into stages with a given execution order.
• An environment is a reusable set of execution settings, covering resources, perimeter, segmentation and variables, shared across actions and workflows.
• A job is one execution of an action or a workflow.

Execution is job-based rather than long-running. When an action or workflow is triggered, the platform builds a worker with the resources requested, runs the job, then shuts the worker down, so compute is billed for the execution only. An always-up execution mode is available for workloads that cannot wait for provisioning.

Every job runs as a service account, and that identity's IAM roles decide what the job can read and write. Notebooks are the interactive counterpart: Jupyter instances for exploring data before its logic becomes an action. For PySpark jobs, the Spark History Server keeps the Spark UI of past runs available, so a finished job can still be inspected stage by stage.

#Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.