# Actions Actions are the core component of the Data Processing Engine. They can perform multiple data processing services, from loading your data into a table, to complex custom Python code. ![etl-interface-action](picts/actions-dashboard.png) * [Create an action](/en/product/dpe/actions/index?id=create-an-action) * [Available action types](/en/product/dpe/actions/index?id=available-action-types) * [Advanced mode](/en/product/dpe/actions/index?id=advanced-mode) * [Manage actions](/en/product/dpe/actions/index?id=manage-actions) * [Versioning repositories of actions](/en/product/dpe/actions/index?id=versioning-repositories-of-actions) * [Use the platform version control system](#use-the-platform-version-control-system) * [Using a repository linked to Git](#using-a-repository-linked-to-git) * [Linking a repository to Git](#linking-a-repository-to-git) * [Commiting and pushing](#committing-and-pushing) * [Pulling and merging](#pulling-and-merging) * [Organising and sharing code across actions](#organising-and-sharing-code-across-actions) * [Triggering a Notebook from an action](#triggering-a-notebook-from-an-action) * [Managing actions programmatically (CI/CD)](#managing-actions-programmatically-cicd) * [Build behaviour](#build-behaviour) --- ## Create an action When a repository is empty, the first thing you'd like to do is **create an action**. From the main page, click the *New action* button on the top right of the screen. You will then be prompted to choose the type of action you wish to perform from the platform Store. ![etl-tag-plan](picts/typeofactions.png) > Note that editing actions is always on autosave mode. Use [versions](/en/product/dpe/actions/index?id=versioning-repositories-of-actions) if you don't want to tamper with your deployed production actions. ### Available Action Types The Data Processing Engine offers a variety of action types to address diverse data processing needs, organized for a logical workflow: - [Load](/en/product/dpe/actions/load/index): Extract, map, and load data from a source into a [Lakehouse Manager table](/en/product/lakehouse-manager/tables/index) using standard Python logic. - [Load PySpark](/en/product/dpe/actions/load-pyspark/index): Load data into a Lakehouse Manager table using PySpark for distributed processing, ideal for large datasets. - [Load to Bucket](/en/product/dpe/actions/load-buckets/index): Load raw data directly into an object storage bucket. - [CSV to PostgreSQL](/en/product/dpe/actions/csv-to-postgresql/index): Ingest data from a CSV file (from any Connectors source) into an existing PostgreSQL table. - [File Transfer](/en/product/dpe/actions/file-transfer/index): Copy files and folders between Project buckets, or from a Connectors source to a Project bucket, preserving the source path structure. - [Aggregate](/en/product/dpe/actions/aggregate/index): Read data from a Lakehouse Manager table, apply transformations, and write the result to another table. - [Aggregate PySpark](/en/product/dpe/actions/aggregate-pyspark/index): Perform distributed aggregation and transformation between Lakehouse Manager tables using PySpark for complex computations. - [SQL](/en/product/dpe/actions/sql/index): Write and run a SQL query against your datasets and tables on the Trino engine, including reads, metadata, DML, DDL, and table creation. - [Custom](/en/product/dpe/actions/custom/index): Run custom data logic written in Python for flexible ETL or validation tasks. - [Custom PySpark](/en/product/dpe/actions/custom-pyspark/index): Execute custom PySpark code on a Spark cluster for heavy-duty transformations or computations beyond standard actions. - [Diff](/en/product/dpe/actions/diff/index): Build a differential (change log) between datasets over time, useful for tracking data evolution. - [Export Table to Source](/en/product/dpe/actions/export-table-to-source/index): Export selected tables from Lakehouse Manager to a compatible external Connectors source (e.g., databases). - [MySQL to Parquet](/en/product/dpe/actions/mysql-to-parquet/index): Transfer a table from a MySQL Connectors source and store it as Parquet files in a Project bucket. - [PostgreSQL to CSV](/en/product/dpe/actions/postgresql-to-csv/index): Transfer a table from a PostgreSQL Connectors source and store it as CSV files in a Project bucket. - [PostgreSQL to Parquet](/en/product/dpe/actions/postgresql-to-parquet/index): Extract data from PostgreSQL and convert it into Parquet format stored into a bucket. - [Delete](/en/product/dpe/actions/delete/index): Delete a specific object entirely (e.g., a table in Lakehouse Manager). - [Delete Bucket](/en/product/dpe/actions/delete-bucket/index): Remove files, folders, or entire contents from an object storage bucket. - [Delete Diff](/en/product/dpe/actions/delete_diff/index): Delete entries from a table that no longer exist in a differential comparison. - [Rebuild](/en/product/dpe/actions/rebuild/index): Rebuild an existing table from scratch to change field constraints (such as Required), restructure partitioned data, and clean its metadata and history. - [Flush & Update Metadata](/en/product/dpe/actions/flush-update-metas/index): This action group includes operations to refresh metadata and clear various caches: - [Update Metadata](/en/product/dpe/actions/flush-update-metas/update-metas.md): Refresh metadata for a project to reflect the latest changes. - [Flush Cache: Analytics Manager](/en/product/dpe/actions/flush-update-metas/flush-cache-qb.md): Clear Analytics Manager-related caches to ensure fresh previews or visualizations. - [Flush Cache: App Services](/en/product/dpe/actions/flush-update-metas/flush-cache-api.md): Flush frontend caches related to visualizations or dashboards. - [Send OpenLineage Events](/en/product/dpe/actions/send-openlineage-events/index): Forward data lineage events from Data Platform to an external OpenLineage-compatible lineage solution (e.g. Marquez). Runs as a cron job or always-up action. ### Advanced mode Actions in the Data Processing Engine are all configured using a set of inputs specified via a JSON configuration file. There are 2 ways for users to update an action's inputs: - either using the graphical user interface - or using an **advanced mode** which lets you directly edit the JSON configuration file Using the "advanced mode" offers a bit more flexibility and options which might come in handy for advanced users. ![advanced-mode](picts/advanced-mode.png) > Note that the advanced mode is also very practical for [Custom actions](/en/product/dpe/actions/custom/index) as it offers an IDE-like experience in the web. This will allow you to code custom Python script directly inside of the platform without managing files locally on your computer. --- ## Manage actions As your Project matures, structure and organization becomes increasingly important. To facilitate this, the Platform provides several methods to better organize and manage your actions: - [Repositories](/en/product/dpe/actions/index?id=versioning-repositories-of-actions): Allow compartmentalizing your actions into different sections that can be versioned - [Versions](/en/product/dpe/actions/index?id=versioning-repositories-of-actions): the platform offers an online versioning experience for each repository of actions individually. The platform has its own version control system, but you can also synchronize each repository with Git - **Descriptions & Tags**: Every action can be assigned a *description* and/or *tags*. This allows you to document your Project, particularly useful when collaborating in large teams. - **Lineage**: Every action contains a lineage sub-tab, accessible when editing an action. It displays every [workflow](/en/product/dpe/workflows/index) that contains the action. Workflows allow you to orchestrate the execution of multiple actions and launch them in a specific order using stages. ### Versioning repositories of actions Editing actions is always on autosave mode. When you are trying to iterate on existing actions, it is recommended to create several versions of the actions' code. This lets you incrementally introduce variations to experiment and test your work, while keeping the version in production always intact. **On the platform, this versioning happens at the level of the repositories**. The repositories are the tabs you see at the top of the Actions' tree view. ![repos](picts/repos.png) You cannot version an individual action, but rather a whole repository of actions. The versioning panel can be accessed at the top right of the repository header: ![etl-versioning](picts/versioning-panel.png) ?> This functionality is also available on other components that implement repositories, such as [Machine Learning Manager Pipelines](en/product/ml/pipelines/index.md). It works in the same way, just look for the equivalent UI elements in the Pipelines screen. There are 2 versioning systems on the Platform: - [the platform version control system](/en/product/dpe/actions/index?id=use-the-the-platform-version-control-system): the default versioning system on all repositories - [Git versioning](/en/product/dpe/actions/index?id=use-a-repository-linked-to-git): if you decide to synchronize your repository with Git Now let's see how all of this works in practice! --- ### Use the Platform Version Control System By default, each repository of actions on the platform has an integrated version control system, allowing you to manually manage different versions of the code (i.e., actions) they contain. Repositories now have: - a **deployed version**, which is the version served when [executing the job](/en/product/dpe/jobs/index). - a **latest version**, which is the current version being edited through the editor panel. ?> If your repository has *only one version*, that version is both the deployed and editable version, meaning changes affect production immediately. If your repository has *two or more versions*, only the latest version can be edited. All other versions, including the deployed version, will be in read-only mode. To create a new version for the repository, open the versioning panel at the top right and click on the **+** icon. ![etl-versioning](picts/versioning-fp1.png) Choose which version to duplicate to create the new version. ![etl-versioning](picts/versioning-fp2.png) After creating the new version, the *deployed version* will not change. You are able to edit the new version directly. ![etl-versioning](picts/versioning-fp3.png) Click on the **Play** button to set a new version as the deployed version: the version of the code used when actions of the repo are executed. ![etl-versioning](picts/versioning-fp4.png) ### Use a repository linked to Git Each repository can be linked to an external Git repository in which case versions are synced with Git commits. This allows you to update & test actions continuously without affecting the version deployed in production. #### Linking a repository to Git To link your repository on the platform to Git, you must click on the gear icon to edit an existing repository or create a new a one. Either of this actions will open the repository configuration window. ![img1](picts/repository_configuration_window.png) Click on *Connect to Git* to expand the window. ![img1](picts/repository_configuration_full.png) Here the first thing you need to enter is the Repository SSH URL. It is generally found on repository cloning options of common Git solutions. On GitHUB, for instance, it can be found on Code > Clone > SSH. ![github](picts/github.png) Next click on *FETCH* to get the distant branches and select the branch you desire to use. Finally, copy the Public SSH key given at the bottom and paste it in the list of SSH keys associated to your account on your Git solution. For GitHUB, you can follow [this tutorial](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account). #### Committing and pushing After changing an object in a repo, your repo will display the uncommitted changes as shown in the image below. You can commit and push them by clicking on the arrow pointing upwards indicated in the image. ![commit](picts/commit_push.png) ?> If there is a conflict when you push, your changes will not be pushed to the branch you configured, instead they will be pushed to a new branch with same name of the current branch and a suffix of *_conflictN*, where N is the number of the conflict. For example, if your branch is named *current-branch* and your push causes a conflict, the new branch will be named *current-branch_conflict1*. If there is a conflict again when you push, the new branch will be named *current-branch_conflict2*. #### Pulling and merging Pull distant changes by clicking on the arrow pointing downwards indicated in the image, a popup window will prompt you to confirm your decision. All uncommitted changes will be lost and both branches will be merged together. ![pull](picts/pull_merge.png) --- ## Organising and sharing code across actions An action is a single `.py` file with one entry function (`customfunc(event)` by default), added through drag & drop or the advanced IDE. There is **no multi-file package structure inside a single action**. To reuse code across actions, use one of the documented paths: * **Reference a Git repository** in the *Python Requirements* field: ``` git+https://github.com/OWNER/REPO.git@ git+https://github.com/OWNER/REPO.git@latest ``` * **Version your whole actions repository through Git**: [link it over SSH](#linking-a-repository-to-git), fetch a branch, and paste the public SSH key on the Git side. This is the intended mechanism for sharing code between actions. ?> There are **no global Python variables** shared between actions: each job runs in a fresh container that shuts down when the job finishes. Reusable values are passed through **Environments** (environment variables), which can be assigned to several actions and workflows at once. --- ## Triggering a Notebook from an action Triggering a Notebook from a Custom action is **not supported**. Notebooks are Jupyter instances with their own manual Start/Stop lifecycle, and the documented flow is the reverse one: converting a notebook into an action. **Recommended pattern:** convert the notebook code into a Custom action, which can then be orchestrated in a [workflow](/en/product/dpe/workflows/index) or through an API call. --- ## Managing actions programmatically (CI/CD) There is **no dedicated REST endpoint** for creating, updating or deleting actions. The official approach is [Git synchronisation](#use-a-repository-linked-to-git): link the actions repository to a Git repository over SSH, then commit and push from the platform, or pull changes made outside of it. For CI/CD, this means that changing an action is a **commit into the linked Git repository**, followed by a pull from the platform. [Workflows](/en/product/dpe/workflows/index) can be run manually or through an API call, so triggering is available even though the CRUD of actions goes through Git. --- ## Build behaviour * **What triggers a build.** Worker builds are triggered by **dependency changes**: updating dependencies redeploys the environment. For Git dependencies, click **Force Build** to reinstall. * **No image cache.** Pre-built workers and image caching for actions are not available today (unlike the preconfigured Notebook images: Base, Data Science, PySpark, TensorFlow). * **Autosave and versions.** Editing an action is always in autosave mode. To avoid affecting production, create a new [version](#versioning-repositories-of-actions): it becomes the editable *latest*, while the *deployed* version stays frozen. ?> To keep rebuilds to a minimum, factor shared dependencies into a reusable **Environment** instead of redeclaring them per action. --- ### Need help? 🆘 > At any step, you can create a ticket to raise an incident or if you need support at the [OVHcloud Help Centre](https://help.ovhcloud.com/csm/fr-home?id=csm_index). Additionally, you can ask for support by reaching out to us on the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559). There is a step-by-step guide in the [support](/en/support/index.md) section.