# Data Platform Buckets connector

?> In the current Data Platform SDK, the **Datastore connector** is used to interact with the Data Platform Buckets. You may think of the Datastore simply as a bucket container.

* [Connect to the Datastore](#connect-to-the-datastore)
* [Datastore Connector methods](#datastore-connector-methods)
    * [datastore.list](#datastorelistreturn_type--array)
    * [datastore.get_buckets](#datastoreget_buckets)
    * [datastore.get_bucket](#datastoreget_bucketname)
    * [datastore.create_bucket](#datastorecreate_bucketname)
    * [datastore.remove_bucket](#datastoreremove_bucketname)
    * [datastore.bucket_exists](#datastorebucket_existsname)
* [Bucket Connector methods](#bucket-connector-methods)
    * [bucket.list](#bucketlistbool-metadatatrue-bool-recursivetrue-kwargs)
    * [bucket.list_filename](#bucketlist_filenamereturn_typearray-contains-recursivetrue-kwargs)
    * [bucket.get](#bucketgetfile_name-kwargs)
    * [bucket.fget](#bucketfgetobject_name-file_path-kwargs)
    * [bucket.put](#bucketputobject_name-data-int-length-kwargs)
    * [bucket.fput](#bucketfputobject_name-file_path-kwargs)
    * [bucket.put_request](#bucketput_requesturl-path-data-methodget-headers-kwargs)
    * [bucket.delete](#bucketdeletepath-kwargs)
    * [bucket.fcopy_to](#bucketfcopy_tonew_bucket-object_name-object_source-kwargs)
    * [bucket.exists](#bucketexistsfilename-kwargs)
    * [bucket.get_content](#bucketget_contentfile_name-kwargs)
    * [bucket.get_path](#bucketget_pathpath)
    * [bucket.remove_path](#bucketremove_pathpath)
* [Deprecated Methods](#deprecated-methods)

---
## Connect to the Datastore

In order to interact with the Datastore you have to connect to it first, as shown in the code below:

```python
from forepaas.dwh import connect

cn_datastore = connect('data_store')
```

After that you can use the `cn_datastore.list()` method to see the buckets available in your Datastore and then connect to the bucket of your choice to interact with it.

You can connect directly to a specific bucket in the Datastore as shown in the code below:

```python
from forepaas.dwh import connect

bucket_name = "name"
cn_bucket = connect('data_store/' + bucket_name)
```

The datastore connector will return a Data Store Connector object and connecting directly to a bucket will return a Bucket Connector object.

See the next section of this article for additional details on the methods of each connector.

---
## Datastore Connector methods

### datastore.list(return_type = 'array')
Lists all buckets in the Datastore.

**Input Parameters**

| Name  | Type  | Description | Example |
| :---  | :---: | :---        | :---    |
| return_type | str | Determine the type you want to get `array` or `str` | |

**Output**

| Type  | Description | Example |
| :---: | :---        | :---    |
| `str` or `array` | List of buckets in Data Store |   |

### datastore.get_buckets()
Gets all buckets from the Datastore

**Output**

| Type  | Description | Example |
| :---: | :---        | :---    |
| `list[bucket]` | list of `bucket` instances |  |

### datastore.get_bucket(name)
Gets a bucket instance from its name.

**Input Parameters**

| Name  | Type  | Description | Example |
| :---  | :---: | :---        | :---    |
| name | str | Bucket name |  |

**Output**

| Type  | Description | Example |
| :---: | :---        | :---    |
| `bucket` | Bucket instance to handle files |  |

### datastore.create_bucket(name)
Adds a bucket in the Data Store.

**Input Parameters**

| Name  | Type  | Description | Example |
| :---  | :---: | :---        | :---    |
| name | str | Bucket name |  |

**Output**

| Type  | Description | Example |
| :---: | :---        | :---    |
| `boolean` | Success of operation |  |

### datastore.remove_bucket(name)
Removes a bucket from the Data Store.

**Input Parameters**

| Name  | Type  | Description | Example |
| :---  | :---: | :---        | :---    |
| name | str | Bucket name |  |

**Output**

| Type  | Description | Example |
| :---: | :---        | :---    |
| `boolean` | Success of operation |  |

### datastore.bucket_exists(name)
Finds out if a bucket exists or not.

**Input Parameters**

| Name  | Type  | Description | Example |
| :---  | :---: | :---        | :---    |
| name | str | Bucket name |  |

**Output**

| Type  | Description | Example |
| :---: | :---        | :---    |
| `boolean` | Success of operation |  |

---
## Bucket Connector methods

### bucket.list(bool metadata=True, bool recursive=True, **kwargs)
Lists files from Data Store's bucket.

**Input Parameters**

| Name      | Type | Description                                   | Example             |
|-----------|------|-----------------------------------------------|---------------------|
| metadata  | bool | (optional) Get metadata for all files listed  | True                |
| recursive | bool | (optional) List recursively through folders    | True                |
| **kwargs  |      | Additional arguments passed to list_objects_v2 |                     |

**Output**

| Type         | Description               | Example |
|--------------|---------------------------|---------|
| list[Object] | List of bucket files      |         |

**Short Example**

```python
from forepaas.dwh.connect import connect
import logging

bucket_name = "name"
bucket = connect('data_store/' + bucket_name)
files = bucket.list()
logger.info(f"Bucket contents: {files}")
```

### bucket.list_filename(return_type='array', contains='', recursive=True, **kwargs)
Lists filenames from Data Store's bucket.

**Input Parameters**

| Name         | Type | Description                                                 | Example |
|--------------|------|-------------------------------------------------------------|---------|
| return_type  | str  | Return type format: 'array' or 'str'                        | 'array' |
| contains     | str  | Filter filenames containing this value                     | '2023'  |
| recursive    | bool | (optional) List recursively through folders                | True    |
| **kwargs     |      | Additional arguments passed to list_objects_v2             |         |

**Output**

| Type          | Description                    | Example             |
|---------------|--------------------------------|---------------------|
| str or [str]  | List of filenames in bucket    | ['file1.csv']       |

**Short Example**

```python
filenames = bucket.list_filename()
logger.info(f"Filenames: {filenames}")
```

### bucket.get(file_name, **kwargs)
Gets raw file content from a Data Store bucket.

**Input Parameters**

| Name       | Type | Description                        | Example            |
|------------|------|------------------------------------|--------------------|
| file_name  | str  | Name of the file to retrieve       | "data/file.csv"   |

**Output**

| Type                           | Description                   |
|--------------------------------|-------------------------------|
| urllib3.response.HTTPResponse | HTTP response with file data |

**Short Example**

```python
file_data = bucket.get('uploads/file.csv')
logger.info(f"Stream: {file_data.stream(1024)}")
```

### bucket.fget(object_name, file_path, **kwargs)
Gets an object from Datastore's bucket to local path.

**Input Parameters**

| Name         | Type | Description                             | Example              |
|--------------|------|-----------------------------------------|----------------------|
| object_name  | str  | Name of the object in the bucket         | "data.csv"          |
| file_path    | str  | Local path where file will be saved      | "./downloads/data.csv" |

**Output**

| Type   | Description              |
|--------|--------------------------|
| Object | Object stat information  |

**Short Example**

```python
bucket.fget("uploads/file.csv", "/tmp/file.csv")
logger.info("File downloaded to /tmp/file.csv")
```

### bucket.put(object_name, data, int length, **kwargs)
Puts an object to Data Store's bucket.

**Input Parameters**

| Name         | Type         | Description                        | Example         |
|--------------|--------------|------------------------------------|-----------------|
| object_name  | str          | Name to assign the object          | "uploaded.csv"  |
| data         | io.RawIOBase | Data stream                        | stream          |
| length       | int          | Length of the data                 | 2048            |

**Output**

| Type | Description              |
|------|--------------------------|
| str  | Object ETag from server  |

**Short Example**

```python
import io
data = io.BytesIO(b"name,age\nJohn,30")
etag = bucket.put("people.csv", data, data.getbuffer().nbytes)
logger.info(f"Uploaded with ETag: {etag}")
```

### bucket.fput(object_name, file_path, **kwargs)
Puts a file to Data Store's bucket.

**Input Parameters**

| Name         | Type | Description                        | Example         |
|--------------|------|------------------------------------|-----------------|
| object_name  | str  | Name of object to be created       | "backup.csv"   |
| file_path    | str  | Path to the file on local system   | "./backup.csv" |

**Output**

| Type | Description              |
|------|--------------------------|
| str  | Object ETag from server  |

**Short Example**

```python
bucket.fput("people.csv", "/tmp/people.csv")
logger.info("Uploaded /tmp/people.csv")
```

### bucket.put_request(url, path, data={}, method='GET', headers={}, **kwargs)
Gets an object from an HTTP request and upload it to Datastore's bucket.

**Input Parameters**

| Name     | Type   | Description                                        | Example        |
|----------|--------|----------------------------------------------------|----------------|
| url      | str    | Source URL to download the object from             | "https://..."  |
| path     | str    | Path to store the file in the bucket               | "raw/data.csv" |
| data     | dict   | Request body data (if any)                         | {}             |
| method   | str    | HTTP method to use                                 | 'GET'          |
| headers  | dict   | Custom headers for the request                     | {'Auth': '...'}|

**Short Example**

```python
bucket.put_request(
    url="https://example.com/file.csv",
    path="remote/file.csv"
)
logger.info("File fetched from URL and uploaded to bucket.")
```

### bucket.delete(path, **kwargs)
Deletes multiple/single file in Data Store's bucket.

**Input Parameters**

| Name       | Type        | Description                 | Example             |
|------------|-------------|-----------------------------|---------------------|
| path       | str or list | Path(s) of file(s) to delete | "data/file.csv"     |

**Short Example**

```python
bucket.delete("people.csv")
logger.info("File deleted from bucket.")
```

### bucket.fcopy_to(new_bucket, object_name, object_source, **kwargs)
Copies file from bucket to a new bucket in Data Store.

**Input Parameters**

| Name          | Type | Description                              | Example            |
|---------------|------|------------------------------------------|--------------------|
| new_bucket    | str  | Target bucket name                       | "archive"         |
| object_name   | str  | New name for the copied object           | "file_backup.csv" |
| object_source | str  | Original object's name in current bucket | "file.csv"        |

**Short Example**

```python
bucket.fcopy_to("archive", "people_backup.csv", "people.csv")
logger.info("File copied to archive bucket.")
```

### bucket.exists(filename, **kwargs)
Checks whether a file exists in the bucket.

**Input Parameters**

| Name       | Type | Description                            | Example         |
|------------|------|----------------------------------------|-----------------|
| filename   | str  | Name or path of the file to check      | "logs/2024.csv" |
| `**kwargs` |      | Additional options for internal checks |                 |

**Output**

| Type | Description               |
|------|---------------------------|
| bool | Whether the file exists   |

**Short Example**

```python
if bucket.exists("people.csv"):
    logger.info("File exists.")
```

### bucket.get_content(file_name, **kwargs)
Retrieves the full content of a file from the bucket.

**Input Parameters**

| Name        | Type | Description                          | Example         |
|-------------|------|--------------------------------------|-----------------|
| file_name   | str  | Name of the file in the bucket       | "people.csv"    |
| `**kwargs`  |      | Additional options (e.g., version)   |                 |

**Output**

| Type        | Description             |
|-------------|-------------------------|
| bytes / str | Raw content of the file |

**Short Example**

```python
content = bucket.get_content("people.csv")
logger.info(f"File content: {content.decode()}")
```

### bucket.get_path(path)
Returns the full qualified path (URL or reference) of an object in the bucket.

**Input Parameters**

| Name | Type | Description                   | Example               |
|------|------|-------------------------------|-----------------------|
| path | str  | Path or object name in bucket | "reports/summary.csv" |

**Output**

| Type | Description             |
|------|-------------------------|
| str  | Full path to the object |

**Short Example**

```python
full_path = bucket.get_path("people.csv")
logger.info(f"Full path: {full_path}")
```

### bucket.remove_path(path)
Removes a specific path from the bucket.

**Input Parameters**

| Name | Type | Description                      | Example            |
|------|------|----------------------------------|--------------------|
| path | str  | Path to the object to be removed | "uploads/file.csv" |

**Output**

| Type | Description            |
|------|------------------------|
| bool | Success of the removal |

**Short Example**

```python
bucket.remove_path("people.csv")
logger.info("Removed specific path from bucket.")
```
---

## Deprecated Methods

> The `bucket.stat()` method is no longer supported and has been deprecated. Use `exists()` or `get_content()` as alternatives depending on your use case.

---

## Additional methods

The Data Platform Datastore is built on [Minio](https://min.io/) technology. Please refer to the [Minio Technical Documentation](https://docs.min.io/docs/python-client-api-reference.html) for more information on the advanced settings of the SDK functions.