Custom Action with Connectors source directly

This example will show you how to get a file directly from a Connectors Source, treat it and put the treated data into a Lakehouse Manager Table

Objective

This example will show you how to get a file directly from a Connectors Source, treat it and put the treated data into a Lakehouse Manager Table.

The code is written in the Custom Action context and it uses the chicago_files source from the Getting Started Tutorial. If you did that tutorial, you can just copy and paste the code below to test it, otherwise you need to adapt it to your tables and data sources.

Tip

This works with any Source protocol such as FTP, Dropbox, etc.

import sys
from forepaas.dwh.connect import connect
from forepaas.dwh import bulk_insert
import logging

logger = logging.getLogger(__name__)

def customfunc(event):
    try:
        # here we are connecting to a source named 'chicago_files'
        source_address = "dwh/chicago_files_artur/"

        # specify unsupported filename from list of files in source
        filename_w_extension = "stations_rides.csv"

        # connect to to file to get the address 
        source_file_connector = connect(source_address + filename_w_extension) 

        # connect to file directly
        file_address = source_file_connector.get()
        file_connector = connect(file_address) 

        # Extract and treat the file so it is usable
        df = file_connector.extract(return_type='dataframe') 

        # Treat the data
        # - - - - 

        # connect to the Lakehouse Manager
        dm_connector = connect("dwh/default_dataset/")

        # insert into an existing destination table
        stats = bulk_insert(dm_connector, "chicago_calendar_full", df) 

        logger.info(stats)

        # disconnect from datastore and remote source
        del source_connector
        del dm_connector

    except Exception as err:
            raise Exception(f"err:{err} L:{sys.exc_info()[2].tb_lineno}")

Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.

Custom Action with Connectors source directly#

#Objective

#Go further

Custom Action with Connectors source directly

Objective

Go further