Discover segmentation based on a predefined set of values#

#Objective

Your actions can be executed on a set of values of your choice while segmenting on them at the same time.

Examples of usage :

• Segmenting on a precise preset of shopping sites or items
• Applying a business rule A to subset of data A with aggregate action A, and a business rule B to a subset of data B in an aggregate action B.

#Requirements

Before using this segmentation type, some points must be checked:

#1. Is the var_name attribute indexed in the source table?

If not, the extraction query will be much slower.

#2. Is there enough CPU on the source and destination databases?

• Keep in mind that select and insert operations are CPU intensive.
• For example, if your database has only 1 CPU, it's probably a bad idea to set 6 workers to select and insert simultaneously against the DBMS instance...

#3. Avoid generating too much tasks in the same stage.

To ensure the Data Processing Engine (DPE) performs well, we advise against having stages with more than 500 tasks.

This is not a hard limit but performance degradations can be observed with increasing number of tasks above 500. In order to reduce the number of tasks, you can set a higher bucket size, so that each task will manage more values. (meaning less tasks at the end).

#Compatibility

This segmentation option is only compatible with the following actions and sources:

#How to use this type of segmentation?

Here is a quick view between the visible mode and the advanced JSON mode.

Source's attribute / var_name: SQL attribute that will be used for the filtering of the source.

Bucket size / Chunksize: Number of values to be filtered for each task.

Predefined set / values: Set of hard-coded values to filter from.

#Other tips

#You can use SQL formulas

For segmentation.var_name (in the attribute last part), you can use SQL formulas, as long as they are compatible with your DBMS. Please note that the DBMS can change over time and so you might have to recheck and correct those SQL formulas if you decide to use them.

For instance, you may have :

{
  "segmentation": {
    "active": true,
    "type": "predefined_set",
    "var_name": "DATE(datetime)",
    "values": ["2020-01-01","2020-01-02"]
    "chunksize": 1
  }
}

#How does it work behind the scenes?

This segmentation mode works the same as segmentation based on an attribute of a Project table. The only difference is that the values are already set in the configuration, rather than a Lakehouse Manager table attribute, so there is no pre-stage needed.

#Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.