Data Processing Engine actions
Twenty-one action types, one JSON document each.
What is an action?
An action is one unit of work in the Data Processing Engine: read from somewhere, do something, write somewhere. Around twenty action types cover the common cases, from loading a registered source into a Lakehouse table to running arbitrary Python or PySpark.
They group by what they do. Loading brings data in: Load, Load PySpark, Load to Bucket, File Transfer, CSV to PostgreSQL. Transformation reshapes data already held in the project: Aggregate, Aggregate PySpark, SQL, Custom, Custom PySpark, Diff. Export sends data back out: Export Table to Source, MySQL to Parquet, PostgreSQL to CSV, PostgreSQL to Parquet. Housekeeping keeps the project tidy: Delete, Delete Diff, Delete Bucket, Rebuild, Maintenance actions, Send OpenLineage Events.
Every action is configured by a JSON document. The graphical editor writes that document for you, and advanced mode edits it directly, which is where fields with no UI equivalent live. Load, Aggregate and Workflows each document their JSON separately.
Actions are held in repositories, and the repository is what gets versioned, either through the platform's own version control or by syncing the repository with Git. An action always runs inside a job, under the preferences and resources set on the action, on its workflow, or on a shared environment.
Creating an action, versioning a repository and syncing it with Git are covered together in Create and manage actions.
Create and manage actions→
Create one, version the repository, sync it with Git, and know what triggers a rebuild.
Load→
Extract, map and load data from a source into a Lakehouse Manager table.
Load PySpark→
The same load, distributed with PySpark for large datasets.
Load to Bucket→
Load raw data straight into an object storage bucket.
File Transfer→
Copy files between buckets, or from a source to a bucket, keeping the path structure.
CSV to PostgreSQL→
Ingest a CSV file from any source into an existing PostgreSQL table.
Aggregate→
Read a table, apply transformations, write the result to another table.
Aggregate PySpark→
Distributed aggregation between Lakehouse Manager tables.
SQL→
Run SQL against datasets and tables on Trino, including DML and DDL.
Custom→
Run your own Python for ETL or validation logic.
Custom PySpark→
Run your own PySpark code on a Spark cluster.
Diff→
Build a change log between datasets over time.
Export Table to Source→
Push tables back out to a compatible external source.
MySQL to Parquet→
Transfer a MySQL table into a bucket as Parquet files.
PostgreSQL to CSV→
Transfer a PostgreSQL table into a bucket as CSV files.
PostgreSQL to Parquet→
Extract from PostgreSQL and store the result as Parquet.
Delete→
Delete an object entirely, such as a table.
Delete Diff→
Delete rows that no longer exist in a differential comparison.
Delete Bucket→
Remove files, folders or entire contents from a bucket.
Rebuild→
Rebuild a table to change constraints, repartition and clean metadata.
Maintenance actions→
Refresh project metadata and flush component caches.
Send OpenLineage Events→
Forward lineage events to an external OpenLineage backend.
Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.
If you need support with your OVHcloud services, create a request in our Help Centre.
Join our community of users.