# Enhance Your Data Analysis with AI Endpoints on the Data Platform
## Introduction
No more worries about imperfect datasets: import, clean, and visualize your data with the **Data Platform**. But that's not all! Benefit from detailed analysis thanks to AI and go even further by asking your LLM to find a Machine Learning rule that fits perfectly with your data.
This guide will demonstrate how to take advantage of [**AI Endpoints**](https://endpoints.ai.cloud.ovh.net/) to enhance your data analysis capabilities on the Data Platform. You'll learn how to start from scratch and generate meaningful predictions from your data.
Here you can find all the **resources** used (CSV files, Python scripts, etc.) for the purpose of this guide. Please download all required files before you begin:
| **Type** | **Name** |
| ---------------------------- | --------------------------------------- |
| Source file | dirty_cafe_sales.csv |
| Preprocessing Python script | custom_action_cafe_sales_processing.py |
| Preprocessing notebook & LLM request | cafe_sales_preprocessing.ipynb |
---
## Table of Contents
- [Tutorial Steps](#tutorial-steps)
- [step 1 - Upload CSV file](#step-1-upload-csv-file)
- [step 2 - Analyze the source and define blueprint rules](#step-2-analyze-the-source-and-define-blueprint-rules)
- [step 3 - Create table from source](#step-3-create-table-from-source)
- [step 4 - Duplicate the table](#step-4-duplicate-the-table)
- [step 5 - Create an empty table for future data visualization](#step-5-create-an-empty-table-for-future-data-visualization)
- [step 6 - Create new attributes](#step-6-create-new-attributes)
- [Processing Options](#processing-options)
- [Option 1: Automated Processing - Data Platform](#option-1-automated-processing)
- [Option 2: Step-by-Step Processing with LLM - AI endpoints](#option-2-step-by-step-processing-with-llm)
---
## Tutorial Steps
### step 1 - Upload CSV file
The first step is to access the [Connectors](/en/product/connectors/index.md), create a new source, and choose `File Upload` to upload the CSV file. For the purpose of this guide we will be using the - *Dirty Cafe Sales Dataset*
### step 2 - Analyze the source and define blueprint rules
Access the [Analyser](/en/product/connectors/analyzer/index.md) from the Connectors and click on the newly added source to extract the metadata.
Once complete you can start to define the following [blueprint rules](/en/product/connectors/analyzer/add-blueprint-rules.md) for each of the mentioned attributes:
| **Attribute** | **Blueprint rule** |
|-----------|---------------|
| **payment_method** | - **Skip line** if *Is null*
- **Skip line** if *Equal* "ERROR"
- **Skip line** if *Equal* "UNKNOWN" |
| **location** | - **Skip line** if *Is null*
- **Skip line** if *Equal* "ERROR"
- **Skip line** if *Equal* "UNKNOWN" |
| **transaction_date** | - **Skip line** if *Is null*
- **Skip line** if *Equal* "ERROR"
- **Skip line** if *Equal* "UNKNOWN" |
### step 3 - Create table from source
1. Open the tables section by going to the [Lakehouse Manager](/en/product/lakehouse-manager/index.md).
2. Click on **Create from a Connectors source** and select the source *`dirty cafe sales.csv`*.
3. Name it dirty_cafe_sales (default name)
3. Click on **Create** to build and create the table.
4. Verify that the table has been created correctly. If there are any issues, check the logs for errors.
### step 4 - Duplicate the table
Duplicate this table to get a new empty table with the same attributes and name it `clean_cafe_sales`. Launch the "Build" and check that everything's okay after completion.
> As noted, the duplicated table is currently empty. This occurs because only the table attributes have been duplicated, while the corresponding data still needs to be loaded separately.
### step 5 - Create an empty table for future data visualization
Create a new empty table called `clean_dataset_history` and add the following attributes:
>💡 **Tip**: You can switch to the Canvas view to easily drag and drop the attributes, instead of creating them individually.
- item
- location
- payment_method
- quantity
- total_spent
- transaction_date
### step 6 - Create new attributes
Head to the [Attributes](/en/product/lakehouse-manager/attributes/index.md) in the Lakehouse Manager, create new Virtual Attributes for data visualization:
| **Attribute Name** | **Virtual Attribute Definition** |
|----------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| **avg_quantity_per_day_per_item** | SUM(CAST(quantity AS DECIMAL(10,2))) / COUNT(DISTINCT CONCAT(CAST(transaction_date AS VARCHAR), CAST(item AS VARCHAR))) |
| **avg_sales_per_day_per_item** | SUM(CAST(total_spent AS DECIMAL(10,2))) / COUNT(DISTINCT CONCAT(CAST(transaction_date AS VARCHAR), CAST(item AS VARCHAR))) |
| **avg_sales_per_day_per_location** | SUM(CAST(total_spent AS DECIMAL(10,2))) / COUNT(DISTINCT CONCAT(CAST(transaction_date AS VARCHAR), CAST(location AS VARCHAR))) |
| **yearmonth** | SUBSTR(CAST(transaction_date AS VARCHAR), 1, 7) |
it should look like so:
---
## Processing Options
You now have two options to process and analyze your data:
### Option 1: Automated Processing (Steps 7-9)
- Automate processing with workflow and visualize data for custom analysis
- Create a Data Processing workflow with custom action using a Python script + an Aggregate action (for automation and efficient dataset processing)
- Create Queries from Analytics Manager to do data visualization using the previously created attributes
### Option 2: Step-by-Step Processing with LLM (Steps 10-11)
- Process data step-by-step and benefit from LLM API endpoints for a complete data analysis
- Use a Jupyter notebook to process your dataset step-by-step
- Benefit from powerful LLMs available on AI Endpoints to do a complete data analysis and generate a report
---
## Option 1: Automated Processing
### step 7 - Clean dataset using your own data processing workflow
If you want to process data quickly and be able to launch it with a single click, you can create a workflow with custom and aggregate actions from the [Data Processing Engine](/en/product/dpe/index.md).
1. **First, create a new "Custom" action:**
- Upload your Python file `custom_action_cafe_sales_processing.py`
- name the custom action as `cafe_sales_data_processing`
- Choose Python 3.9 version
- Add `numpy` and `pandas` dependencies (type in the library name and press enter) and click on **Create**
- Now you can **RUN** the custom action.
Also, you should see logs similar to below to double check if what you have done is correct:
```
2025-02-27 08:59:09 [NOTICE] "cafe_sales_data_processing" SUBMITTED
2025-02-27 08:59:09 [NOTICE] "cafe_sales_data_processing" QUEUED
2025-02-27 08:59:10 [NOTICE] "cafe_sales_data_processing" BUILD SUBMITTED
...
2025-02-27 09:00:41 [NOTICE] "cafe_sales_data_processing" Stop cleaning
2025-02-27 09:00:41 [NOTICE] "cafe_sales_data_processing" END (custom) action. Duration: 6.65 sec
...
2025-02-27 09:00:44 [NOTICE] "Flush dataplant cache" END (forepaas-flushall) action. Duration: 2.14 sec
```
2. **Then, create the "Aggregate" action as follows:**
- Configure the aggregate action settings according to your requirements
3. **Then, create the "Delete" action as follows:**
- Configure the delete action settings according to your requirements (this action will ensure you are using a clean table every time)
4. **Finally, define the processing workflow:**
- Set up your workflow to connect the delete, custom action and aggregate action
Congratulations! Your dataset is now clean ✨
### step 8 - Explore clean tables
Now, you can check if everything's okay:
1. **From the Lakehouse Manager Tables**, you should see your created tables
2. **From the Lakehouse Manager [Explorer](/en/product/lakehouse-manager/explorer/index.md)**, you can explore the `clean_cafe_sales` table
### step 9 - Data visualization using custom attributes
Now we can take an even more closer look at our data visually with the help of [queries](/en/product/am/queries/index.md) in the [Analytics Manager](/en/product/am/index.md)
1. **Create quantity_per_item query**
- You should see the average quantity sold per day for each item
2. **Create total_spent_per_item query**
- You could see the average total amount sold per day for each item
3. **Create sales_per_location query**
- You could see the average total amount sold per day for each location
You can go further by creating more queries and then add them to a Dashboard which you can then share with others.
## Option 2: Step-by-Step Processing with LLM
### step 10 - Clean dataset and save it into S3 bucket
If you want to follow the data cleaning process step by step, you can create a new notebook from the Data Processing service.
1. **Create a "Base notebook" with:**
- Python 3.9 version
- Install `numpy`, `pandas`, `matplotlib`, and `seaborn` dependencies
2. **Drag and drop the or click upload, to upload the file `cafe_sales_preprocessing.ipynb` into the jupyterhub file siebar**
### step 11 - One-shot data analysis with LLM AI Endpoints
A single request is enough to generate all the Python code you need for dataset analysis!
1. **Use the Llama 3.3 70B Instruct model** with the Python code that will allow you to analyze your dataset easily
2. **Generate an AI Endpoints API key** and use it in the designated line as seen below in the screenshot
- Access [here](https://endpoints.ai.cloud.ovh.net/)
3. **Use the generated Python code** to analyze your data and display graphs
The advantage of asking an LLM to generate code in Python is that you will be able to reuse it when you add new data to your coffee sales in subsequent months, ensuring a consistent analysis method.
---
### Need help? 🆘
> At any step, you can create a ticket to raise an incident or if you need support at the [OVHcloud Help Centre](https://help.ovhcloud.com/csm/fr-home?id=csm_index). Additionally, you can ask for support by reaching out to us on the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559). There is a step-by-step guide in the [support](/en/support/index.md) section.