# API Reference: Buckets

The Data Platform Buckets are fully compatible with **AWS S3** client.
To interact with them, you will need to [generate an API key and Secret key in your Identity Access Manager](/en/getting-further/generate-api-key/api-secret-key) from a **Data Platform Directory** account.

> ℹ️ **Note:** Currently, all root buckets are named as `"project-projectID"`.  
> You can find your `projectID` by clicking on the info icon at the top-right corner of the platform.  
> **Example:** `project-abcdef`

* [Using AWS CLI](#using-aws-cli)
* [Using Rclone](#using-rclone)
* [Using NodeJS](#using-nodejs)
* [Using Python](#using-python)

---
## Using AWS CLI

> In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](/en/product/project/config-ids?id=project-subdomain).

1. Install the AWS CLI client from  https://aws.amazon.com/cli/
1. Configure the AWS CLI, it is suggested to use [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
```bash
aws configure --profile myproject
aws Access Key ID [None]: your_access_key
aws Secret Access Key [None]: your_secret_key
Default region name [None]: forepaas
Default output format [None]:
```
1. Enable *AWS Signature Version 4* for MinIO server.
```bash
aws configure set s3.signature_version s3v4 --profile myproject
```
1. A few command examples (replacing `my-project` with your [Project's subdomain](/en/product/project/config-ids?id=project-subdomain) and 'mybucket' with your desired bucket name):

```bash
# list your buckets
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 ls

# list contents inside bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 ls s3://mybucket

# make a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 mb s3://mybucket

# add an object to a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 cp my-file.csv s3://mybucket

# delete an object from a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 rm s3://mybucket/my-file.csv

# remove a bucket
aws --profile myproject --endpoint-url https://my-project.eu.dataplatform.ovh.net/datastore s3 rb s3://mybucket

```

> To manage entrypoint URL inside a named profile, you may look at this documentation page: [configure entry point URL per profile](https://github.com/wbingli/awscli-plugin-endpoint).

---
## Using Rclone

> In the base URLs in the code snippets below, you will need to replace `my-project` with your [Project's subdomain](/en/product/project/config-ids?id=project-subdomain).

1. Install [rclone](https://rclone.org/) from https://rclone.org/downloads/
1. Configure rclone (replacing `my-project` with your [Project's subdomain](/en/product/project/config-ids?id=project-subdomain)).
```bash
rclone config create my-project s3 provider Minio region forepaas env_auth false
rclone config update my-project endpoint https://my-project.eu.dataplatform.ovh.net/datastore
rclone config update my-project access_key_id YOUR_ACCESS_KEY
rclone config update my-project secret_access_key YOUR_SECRET_KEY
```
1. Check rclone's config file location if needed
```bash
rclone config file
```
1. Check your configuration if it matches the following below
```bash
rclone config show my-project
[my-project]
type = s3
provider = Minio
region = forepaas
env_auth = false
endpoint = https://my-project.eu.dataplatform.ovh.net/datastore
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
```
1. Some simple commands (please refer to rclone documentation for more details):

```bash
# listing buckets
rclone lsd my-project:

# create a bucket
rclone mkdir my-project:test-rclone

# generate a local docs for test purpuse
rclone gendocs docs

# copy ./docs/ to test-rclone
rclone copy docs my-project:test-rclone/docs
# what is the size of a bucket or a path ?
rclone size my-project:test-rclone
Total objects: 69
Total size: 239.570 kBytes (245320 Bytes)

# displays a tree
rclone tree my-project:test-rclone
# move files in bucket
rclone move my-project:test-rclone/docs/commands my-project:test-rclone/commands

# syncing source to destination
rclone sync --dry-run docs my-project:test-rclone/docs

# listing files in a human redable way
rclone lsl my-project:test-rclone
# listing files with format options
rclone lsf  --format "tsp" --recursive my-project:test-rclone

# deleting files greater than 1k
rclone --min-size 1k lsl my-project:test-rclone/commands
rclone --dry-run --min-size 1k delete my-project:test-rclone/commands
rclone --min-size 1k delete my-project:test-rclone/commands
# copy files in ./commands
rclone copy my-project:test-rclone/commands commands
# delete an entire path
rclone delete my-project:test-rclone/commands/
```




---
## Using NodeJS

> In the base URLs in the code snippets below, you will need to replace `[DATAPLANT_NAME]` with your [Project's subdomain](/en/product/project/config-ids?id=project-subdomain).

> ℹ️ **Note:** Currently, all root buckets are named as `"project-projectID"`.  
> You can find your `projectID` by clicking on the info icon at the top-right corner of the platform.  
> **Example:** `project-abcdef`


```sh
# Add aws-sdk to your application with the following command
yarn add aws-sdk
# Or
npm i -s aws-sdk
```
```js
const AWS = require('aws-sdk')
const fs = require('fs')

// Modify those values for your environment
let accessKey = '[ACCESS_KEY]'
let secretKey = '[SECRET_KEY]'
let endpoint = 'https://[DATAPLANT_NAME].eu.dataplatform.ovh.net'
let bucket = '[BUCKET_NAME]'
let region = 'forepaas'

// Constructs a service object. This object has one method for each API operation.
const dataStoreClient = new AWS.S3({
  accessKeyId: accessKey,
  secretAccessKey: secretKey,
  endpoint: `${endpoint}/datastore`,
  region: region,
  signatureVersion: 'v4',
  s3ForcePathStyle: true
})


////////////////////////////////
// Adds an object to a bucket //
////////////////////////////////
let uploadStream = fs.createReadStream('./file.csv')
dataStoreClient.putObject({
  Bucket: bucket,
  Key: 'file.csv',
  Body: uploadStream
}, (err) => {
  if (err) console.error(err)
  else console.info('File uploaded')
})


/////////////////////////////////////
// Retrieves objects from a bucket //
/////////////////////////////////////
let downloadStream = fs.createWriteStream('./file_downloaded.csv')
dataStoreClient.getObject({
    Bucket: bucket,
    Key: 'file.csv'
  })
  .createReadStream()
  .on('error', (err) => {
    console.error(err)
  })
  .pipe(downloadStream)
  .on('close', () => {
    console.info('File downloaded')
  })


/////////////////////////////////////
// Returns some or all (up to 1000) of the objects in a bucket.
/////////////////////////////////////
dataStoreClient.listObjects({
  Bucket: bucket
}, (err, data) => {
  if (data && data.Contents) {
    data.Contents.forEach(file => {
      console.info(`${file.Key} (${file.Size} bytes)`)
    })
  }
})
```


---
## Using Python

> In the base URLs in the code snippets below, you will need to replace `[DATAPLANT_NAME]` with your [Project's subdomain](/en/product/project/config-ids?id=project-subdomain).

> ℹ️ **Note:** Currently, all root buckets are named as `"project-projectID"`.  
> You can find your `projectID` by clicking on the info icon at the top-right corner of the platform.  
> **Example:** `project-abcdef`


```sh
# Install boto3 with the following command:
pip install boto3
```

```python
import boto3
from botocore.client import Config
import os

def customfunc(event):
    # Modify these values for your environment
    dataplant_url = 'https://[DATAPLANT_NAME].eu.dataplatform.ovh.net'
    access_key = '[ACCESS_KEY]'
    secret_key = '[SECRET_KEY]'
    bucket = '[BUCKET_NAME]'
    region = 'forepaas'
    
    try:
        # Create the S3 client (named datastore)
        datastore = boto3.client(
            's3',
            endpoint_url=f'{dataplant_url}/datastore',
            aws_access_key_id=access_key,
            aws_secret_access_key=secret_key,
            config=Config(signature_version='s3v4',request_checksum_calculation='when_required'),
            region_name=region
        )

        # List all available buckets
        buckets_response = datastore.list_buckets()
        print("Available buckets:")
        if 'Buckets' in buckets_response:
            for b in buckets_response['Buckets']:
                print(b['Name'])
        else:
            print("No buckets found.")

        # Test connection by listing objects in the specified bucket
        response = datastore.list_objects(Bucket=bucket)
        print("Connection successful!")
        if response and 'Contents' in response:
            print("Bucket contents:")
            for obj in response['Contents']:
                print(f"{obj['Key']} ({obj['Size']} bytes)")
        else:
            print("Bucket is empty or no objects returned.")

        # Ensure the local file exists
        local_filename = './file.csv'
        if not os.path.exists(local_filename):
            print(f"Local file {local_filename} not found.")
            return

        # Upload a file using a key that places it in an allowed subfolder.
        upload_key = 'dwh/uploads/file.csv'
        datastore.upload_file(
            Bucket=bucket,
            Filename=local_filename,
            Key=upload_key
        )
        print(f"Uploaded local file '{local_filename}' to bucket '{bucket}' with key '{upload_key}'")

        # Download a file from the bucket.
        # Using a file key from your logs:
        download_key = 'dwh/uploads/clean_cafe_sales.csv'
        local_download_filename = './clean_cafe_sales_downloaded.csv'
        datastore.download_file(
            Bucket=bucket,
            Filename=local_download_filename,
            Key=download_key
        )
        print(f"Downloaded '{download_key}' from bucket '{bucket}' to local file '{local_download_filename}'")

        #Here we have used a file with very specific directory using the download key, this can be done manually or either using the step above to list all the files and picking from that
        #for reference here the dpe folder exists inside the bucket at root level.

    except Exception as e:
        print("Connection failed:", e)

# Example usage:
if __name__ == "__main__":
    customfunc(event={})

```