# Datasets Datasets are logical groupings of tables, which allow them to share some default parameters.the Platform supports **Standard** as well as **External** Datasets to manage your tables. By default you will be given one standard dataset named **default_dataset**. This standard dataset will be using the *Lakehouse Manager Engine* which is based on [Apache Iceberg](https://iceberg.apache.org/). ![LHM-datasets](picts/datasets-1.png) * [Datasets](#datasets) * [Standard Dataset](#standard-dataset) * [How data is stored](#how-data-is-stored) * [Dataset Management](#dataset-management) * [Organising your data: the medallion approach](#organising-your-data-the-medallion-approach) * [Cross-project sharing (beta)](#cross-project-sharing-beta) * [What sharing does, and does not, do](#what-sharing-does-and-does-not-do) * [Permissions and roles](#permissions-and-roles) * [Share a dataset](#share-a-dataset) * [On the receiving project](#on-the-receiving-project) * [In the Explorer](#in-the-explorer) * [Auditing](#auditing) !> Once you create a dataset, there is a waiting period of 60 seconds before it can be used for queries. --- ## Datasets Datasets were designed with the mindset of encouraging users to manage the parameters of a table at a group/dataset level. You can Edit, Duplicate and Delete datasets from the list view. ### Standard Dataset The [Lakehouse Manager](en/product/lakehouse-manager/index.md) does the dual role of being the data management service but also the storage engine for the standard dataset. You can learn more about this engine by clicking on the *discover* link during the creation of a dataset. Click on the **New Dataset** button to start creating a new dataset, then select *standard dataset*. Fill in all the required dataset fields and click **Create**. ![LHM-datasets](picts/datasets-2.png) ![LHM-datasets](picts/datasets-3.png) --- ## How data is stored The Lakehouse Manager separates the **logical** organisation of your data from where the bytes actually live. Three concepts are easy to confuse, so it helps to keep them apart: * **Datasets**: a logical envelope that groups tables together. A dataset does **not** store any data itself. * **Tables**: the physical objects, stored in [Apache Iceberg](https://iceberg.apache.org/) and managed by the Lakehouse Manager. * **Buckets**: S3-compatible object storage for unstructured files, independent from the Lakehouse. See [Buckets](/en/product/lakehouse-manager/buckets/index.md). A dataset is therefore **not** a bucket: it is a logical grouping of tables. ?> Because a dataset only holds metadata, sharing or moving a dataset never copies data: it changes what the Lakehouse points to. See [Cross-project sharing](#cross-project-sharing-beta). --- ## Dataset Management The configuration page shows all the information you need to know about your dataset. You can edit the description and the tags for your datasets, you can also view the data residency policies - storage engine used and data residency location. ![LHM-datasets](picts/datasets-6.png) In the tables tab, you can view all the associated tables that are a part of the dataset. It is important to remember that this is only a view, and you cannot edit any of the tables from this menu. ![LHM-datasets](picts/datasets-7.png) The [policy tags](en/product/lakehouse-manager/policy-tags/index.md) menu allows you assign dataset wide policy tags, which will be further applied to all the tables part of the dataset. You can choose from the available policy tags, or even create policy tags on spot. [Here](en/product/lakehouse-manager/policy-tags/index.md#binding-policy-tags-to-data) is the step-by-step guide to binding policy tags. ![LHM-datasets](picts/datasets-8.png) --- ## Organising your data: the medallion approach The Data Platform does **not** prescribe a Bronze / Silver / Gold ("medallion") architecture: you are free to model your lakehouse the way that fits your use case. By default, a dataset is provisioned for you (**default_dataset**). That said, managing medallion layers **with datasets** is a valid pattern and works naturally with the rest of the platform. A typical pipeline looks like this: 1. **Bronze**: raw data, ingested through [Connectors](/en/product/connectors/sources/index). 2. **Silver**: cleaned, validated and deduplicated by [Data Processing Engine](/en/product/dpe/index) actions. 3. **Gold**: enriched, joined and aggregated by Data Processing Engine actions, ready for consumption in the [Analytics Manager](/en/product/am/index) or your BI tool. Each layer is a dataset; the transformations between layers are actions orchestrated in a [workflow](/en/product/dpe/workflows/index). --- ## Cross-project sharing (beta) > This feature is currently in beta. Please report any issues you run into. By default, every project on the Data Platform is fully isolated: there is no implicit way to reach a dataset from another project. Cross-project sharing lets the owner of a dataset expose it, from the **source project**, to one or more **receiving projects** in a controlled way, without breaking the existing security boundaries. ### What sharing does, and does not, do * Sharing happens **at the dataset level**, between projects **within the same organization**. You cannot share with another organization. * A shared dataset is a **singleton, not a copy**. Reads, writes, builds, and use in notebooks and custom actions apply to the same underlying data on both the source side and the receiving side. * For now, sharing is **full**: the entire dataset is exposed to the receiving project. * Isolation between unrelated projects stays strictly enforced. ### Permissions and roles Sharing is managed from the source project. * A shared dataset is exposed in **READ & WRITE**. * The **LM & AM Editor** role is required to share a dataset or to remove an existing share. !> Access to a shared dataset is always evaluated **in the context of the receiving project**, not the source project. When user *B* in project *B* reads a dataset shared from project *A*, the platform asks "does user *B* have access to this dataset **in project B?**", never "in project A". Since user *B* does not exist in project *A*, that second question would always return false. Your access rules therefore stay anchored in the receiving project. ?> A finer-grained **Advanced Data Access Control** module is under study for more granular permissions on shared objects. ### Share a dataset 1. From the source project, open the dataset you want to share and go to its **Configuration** tab. 2. Click **Manage share settings**. ![LHM-cross-project-share](picts/share-1.png) 3. Select one or more projects from the available list of projects in your organization, then confirm. ![LHM-cross-project-share](picts/share-2.png) 4. You will receive an email confirming that the dataset has been shared. The source project shows usage information for each project the dataset has been shared with, so you can keep track of who is consuming the shared data. ### On the receiving project Shared datasets appear in the **Datasets** menu of the receiving project's Lakehouse Manager, alongside the project's own datasets. They are flagged with a shared icon and the type **Lakehouse Manager (shared)**. ![LHM-cross-project-share](picts/share-3.png) Opening a shared dataset shows its details in **read-only** mode: the dataset configuration and table structure stay managed from the source project. The associated tables can be previewed by clicking the preview icon. ![LHM-cross-project-share](picts/share-4.png) The shared dataset can be used as a source in the Data Catalog and in Trino, so users of the receiving project work with its data exactly as they would with a dedicated dataset. ### In the Explorer Inside the [Explorer](/en/product/lakehouse-manager/explorer/index.md), use the **list by dataset** option to find a new **Shared Datasets** category. All tables coming from datasets shared with this project are listed here. ![LHM-cross-project-share](picts/share-5.png) > Shared dataset tables can only be queried using the **SQL editor**. Visual builder support is not available yet. --- ### Need help? 🆘 > If you are logging-in with an OVHcloud account, you can create a ticket to raise an incident or if you need support at the [OVHcloud Help Centre](https://help.ovhcloud.com/csm/fr-home?id=csm_index). Additionally, you can ask for support by reaching out to us on the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559). There is a step-by-step guide in the [support](/en/support/index.md).