Packages preloaded in Data Platform action containers#

#Objective

All actions & workflows when launched are built and run in containers which come with preloaded packages. That way, especially in the case of custom actions, you don't have to always manually specify which librairies you need to run your code.

Currently Data Platform jobs are only compatible with Python 3+, there is therefore only 2 default packages for you to manage:

• Python Libraries: If you're about to write your own custom action, no need to give you a lecture on Python libraries... in the background, we're simply running a pip install command.
• Data Platform Modules: In order to keep a light weight image of the containers all Data Platform connectors are not loaded automatically. To use a specific connector using the SDK, please specify the required modules from our internal supported list.

#Default Python Librairies

Below is the list of all the Python libraries and corresponding version number installed by default in the Data Platform DPE containers:

nameko==2.12.0
pyopenssl==19.1.0
minio==5.0.8
bson==0.5.8
requests==2.22.0
timedelta==2019.4.13
coverage==4.5.4
pycryptodome==3.9.4
kubernetes==10.0.1
sqlalchemy==1.3.8
mysqlclient==2.0.1
prometheus_client==0.7.1
google-cloud-storage==1.36.2
pandas==1.0.0

However, you can absolutely install any other libraries on top of the list provided above by adding them in "Requirements" field of your custom action configuration whether in the UI or in the JSON file. Simply use the same format as when using a pip install command to specify the version's library.

Explore all Python libraries on Pypi's official website

#Data Platform Modules

When you create a new custom action using the Data Platform Data Processing Engine you will certainly be connecting to external data sources and use a variety of file formats. Data Platform possesses a pre-existing list of modules to handle both data formats & connector types inside your code.

The list of modules preinstalled by default includes primarily support for generic file format & other component of the platform (data warehouse & data store): mySQL, S3¹, CSV, XML, XLS, XLSX, JSON. Parquet file formats are not installed by default because of the size of the module (approximately 400 MB) which would add latency for the build of the workers.

To install Data Platform modules for your custom actions you can either do it via the Custom action configuration GUI under the Requirements section. For instance, if you use the SDK functions to connect to an external Dropbox source but have not installed the Dropbox module, your code will throw an error unless you configure your custom action as below:

Or you can directly input the list of the modules in the JSON configuration of the action under the key modules:

{
  "modules": [
    {
      "type": "action",
      "id": "aggregate",
      "version": "latest"
    },
    { ... }
  ],

Below is the exhaustive list of additional available Data Platform modules that can be installed in DPE workers (which corresponds to the list you can find in the Lakehouse Manager's source's marketplace):

#Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.

¹: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.