Scale Data Processing Engine resources
The Platform is designed to scale easily as the requirements of your data Project evolve
Objective
The Platform is designed to scale easily as the requirements of your data Project evolve.
In the Data Processing Engine, each individual action or workflow can be scaled both horizontally and vertically, meaning you can parallelize processing tasks and add more computing power to each indefinitely.
Scale your jobs horizontally
You can parallelize the workload of your action or workflow by selecting the number of parallel instances in which your job will be ran. Typically, multiple instances will be used for the following processes:
- Using segmentation for a job
- Running workflows with multiple actions in the same stage
- Running multiple concurrent occurrences of the same Always-up action/workflow
Failure to allocate multiple instances for the above processes will simply have the broken-down tasks executed back-to-back on the single instance.
To change the amount of parallel instances allocated to an action or a workflow, go to its Preferences tab.
Change the number of instances in the Resources panel.
Scale your jobs vertically
You can allocate more computing power to an action or a workflow by increasing the number of Data Platform Units (DPU) of each computing instance. The DPU is a unit of processing capability, representing access to a certain amount of CPU and memory, as such:
- DPU: - corresponds to approximately 1 vCPU and 4 GB of RAM, based on hardware availability.
By default, newly created actions or workflows are allocated 2 DPU.
To change the amount of DPU allocated to an action or a workflow, go to its Preferences tab.
Use the slider to set the desired DPU size of each parallel instance for the execution.
The final total amount of DPU allocated to a DPE job is the number of parallel instances times the DPU size of each.
If you are using the Always-up execution mode, you still are allowed to update your resources. The update will come into effect shortly after you change your settings and your workflows/actions will not be interrupted in the process.
Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.
Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.
If you need support with your OVHcloud services, create a request in our Help Centre.
Join our community of users.