Split large jobs with segmentation#

#Objective

When executing an action in the Data Processing Engine, it is important to think about the amount of data that will be processed. For large volumes, there is a risk to reach the maximum compute power of a worker. To be able to process these actions, you can split the execution of actions into a set of smaller tasks. This broken-down workload can then be shared between multiple workers concurrently.

The segmentation options allow you to specify how to break down the data workload. The execution will be split into multiple tasks, each task managing a smaller set of values at a time.

There are several segmentation modes available inside the Platform depending on the data source and the type of action you are using:

1. Based on the date perimeter of the workflow
2. Based on an attribute of a Project table
3. Based on a predefined set
4. Based on a fix number of lines
5. Based on the source's accounts
6. Based on files
7. Automatic segmentation

#1. Based on the date perimeter of the workflow

If your source data contains a date attribute, it is possible to split the action in separate processes which will individually address a specific date or a group of dates (bucket). In this case, each task will apply a filter on the date that it needs to extract.

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#Bucket Size

If your source contains a significant number of dates, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time. This is particularly common when dealing with datetime type attributes that can record time information up to the milli-second unit.

To avoid generating too many tasks, you can make each task manage several dates at once by using the “bucket size” option. Simply set a size for your bucket to decide how many dates at a time you want your task to manage. The default “bucket size” value is 1, that is:

• If you do not set the bucket size, each task will handle 1 date
• If you set the bucket size to 2, each task will handle 2 dates
• If you set the bucket size to 10, each task will handle 10 dates
• And so on...

If you’re still unclear about what the “bucket size” does, find out more in the bucket size reference.

#Configuration

To configure this segmentation option select Based on the date perimeter of the workflow in the "Segmentation Type" dropdown, and select the attribute.

If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:

      "segmentation": {
        "active": true,
        "type": "workflow_dates",
        "var_name": "id2",
        "chunksize": 1
      }

Get further with this type of segmentation with our tutorial series 👩‍🏫

#2. Based on an attribute of a Project table

This option lets you segment an action using a list of values contained in the column of a given table (usually a Reference Table) from your data model. For instance, let’s consider the following case where we’re trying to set the segmentation options for an action which loads the data from the Source Table “orders” to the Destination Table “aggr_date”:

Reference Table	Source Table	Destination Table

Assuming, you want to aggregate “income” data per the date of orders “date_order”. Let's say that the “tmp_date” table serves as the reference table of all dates during which there was ever an order made, e.g. it contains a list of dates from 2019-01-01 to 2019-01-03.

By activating this segmentation mode, it will automatically generate a pre-stage to your main stage which contains your action, which will list the unique list of dates available in order to split your action in separate tasks as describe in the diagram below:

By doing so, the data from your Source Table “orders” will therefore be loaded to the Destination Table “aggr_date” on day at a time.

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#Bucket Size

If your source contains a significant number of unique values within the attribute used for segmentation, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.

To avoid generating too many tasks, you can make each task manage several values of your segmentation attribute at once by using the “bucket size” option. Simply set a size for your bucket to decide how many values at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:

• If you do not set the bucket size, each task will handle 1 value
• If you set the bucket size to 2, each task will handle 2 values
• If you set the bucket size to 10, each task will handle 10 values
• And so on...

If you’re still unclear about what the “bucket size” does, find out more in the bucket size reference.

#Configuration

To configure this segmentation option select Based on an attribute of a Project table in the "Segmentation Type" dropdown.

If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:

      "segmentation": {
        "active": true,
        "type": "dwh_attributes",
        "var_name": "id",
        "attribute": "name",
        "object": "s3.deldiff/delete_diff.csv",
        "attribute_type": "map",
        "chuncksize": 2
      }

Get further with this type of segmentation with our tutorial series 👩‍🏫

#3. Based on a predefined set

Use this mode in order to use a set of hard-coded values as your segmentation criteria. For this you must choose the column of the source table on which the segmentation will be applied & manually input the set of values. For each value entered, a task will be generated (when the bucket size is set to 1).

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#Bucket Size

If your source contains a significant number of unique values within the attribute used for segmentation, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.

To avoid generating too many tasks, you can make each task manage several values of your segmentation attribute at once by using the “bucket size” option. Simply set a size for your bucket to decide how many values at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:

• If you do not set the bucket size, each task will handle 1 value
• If you set the bucket size to 2, each task will handle 2 values
• If you set the bucket size to 10, each task will handle 10 values
• And so on...

If you’re still unclear about what the “bucket size” does, find out more in the bucket size reference.

#Configuration

To configure this segmentation option select Based on predefined set in the "Segmentation Type" dropdown.

If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:

      "segmentation": {
        "active": true,
        "type": "predefined_set",
        "var_name": "id",
        "chunksize": 1,
        "values": [
          "1"
        ]
      }

Get further with this type of segmentation with our tutorial series 👩‍🏫

#4. Based on a fixed number of lines

This segmentation mode allows you to split the loading of the data from a source simple using a fixed number of rows defined by the bucket size variable.

Each task will handle a specific number rows equal to the "bucket size" .

• If the bucket size is set to 1000, each task will extract 1000 rows.
• If the bucket size is set to 50000, each task will extract 50000 rows.
• Etc …

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#Configuration

To configure this segmentation option select Based on a fix number of lines in the "Segmentation Type" dropdown.

If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:

      "segmentation": {
        "active": true,
        "type": "lines",
        "chunksize": 1000
      }

Get further with this type of segmentation with our tutorial series 👩‍🏫

#5. Based on the source's accounts

For sources like social media which are user account specific, it is possible to configure multiple accounts for the same source within the Connectors. You can then leverage these accounts to split a DPE action using that source in several tasks, where each task will retrieve the data from one or more accounts.

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#Bucket Size

If your source contains a significant number of accounts, it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.

To avoid generating too many tasks, you can make each task manage several accounts at once by using the “bucket size” option. Simply set a size for your bucket to decide how many accounts at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:

• If you do not set the bucket size, each task will handle 1 account
• If you set the bucket size to 2, each task will handle 2 accounts
• If you set the bucket size to 10, each task will handle 10 accounts
• And so on...

If you’re still unclear about what the “bucket size” does, find out more in the bucket size reference.

#Configuration

To configure this segmentation option select Based on the date perimeter of the workflow in the "Segmentation Type" dropdown.

If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:

      "segmentation": {
        "active": true,
        "type": "accounts",
        "chunksize": 1000
      }

Get further with this type of segmentation with our tutorial series 👩‍🏫

#6. Based on files

For sources with the same structure in a distant source folder, this segmentation mode allows the DPE to automatically find all files in a folder, then create tasks with a specific number of files. It will replace the source file in “load_from.source” actions for each files.

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#Bucket Size

If your source contains a significant number of files , it might lead to generating a multitude of small tasks and not be particularly beneficial to your action run time.

To avoid generating too many tasks, you can make each task manage several files at once by using the “bucket size” option. Simply set a size for your bucket to decide how many accounts at a time you want your task to manage. Note that the default “bucket size” value is 1, that is:

• If you do not set the bucket size, each task will handle 1 file
• If you set the bucket size to 2, each task will handle 2 files
• If you set the bucket size to 10, each task will handle 10 files
• And so on...

If you’re still unclear about what the “bucket size” does, find out more in the bucket size reference.

#Configuration

To configure this segmentation option select Based on files in the "Segmentation Type" dropdown.

If you are using the expert mode interface, you will need to add the following fields to your action JSON configuration:

      "segmentation": {
        "active": true,
        "type": "files",
        "values": ["a.*"],
        "chunksize": 10
      }

Get further with this type of segmentation with our tutorial series 👩‍🏫

#7. Automatic Segmentation

With this option, you do not need to worry about configuring your segmentation settings, it will be automatically configured for you. Unfortunately this option isn't available with all Data Sources or all Action types.

Below a list of supported Data Sources for the supported Actions as well as an explanation on how many workers to set for the Jobs.

Supported Data Sources with the Table to files Action

• PostgreSQL: maximum number of workers is not limited.

Supported Data Sources with the Load Action

• Apache Kafka: maximum of one worker per topic partition.

Check-out additional information about our Segmentation and Perimeter options for DPE action in the Additional Remarks articles of the DPE Action settings. Segmentation and perimeter vocabulary

#Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.

¹: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.