Understand the medallion architecture format

 

Bronze layer

The bronze or raw layer of the medallion architecture is the first layer of the lakehouse. It's the landing zone for all data, whether it's structured, semi-structured, or unstructured. The data is stored in its original format, and no changes are made to it.

Silver layer

The silver or validated layer is the second layer of the lakehouse. It's where you'll validate and refine your data. Typical activities in the silver layer include combining and merging data and enforcing data validation rules like removing nulls and deduplicating. The silver layer can be thought of as a central repository across an organization or team, where data is stored in a consistent format and can be accessed by multiple teams. In the silver layer you're cleaning your data enough so that everything is in one place and ready to be refined and modeled in the gold layer.

Gold layer

The gold or enriched layer is the third layer of the lakehouse. In the gold layer, data undergoes further refinement to align with specific business and analytics needs. This could involve aggregating data to a particular granularity, such as daily or hourly, or enriching it with external information. Once the data reaches the gold stage, it becomes ready for use by downstream teams, including analytics, data science, or MLOps.

Customize your medallion architecture

Depending on your organization's specific use case, you may have a need for more layers. For example, you might have an additional "raw" layer for landing data in a specific format before it's transformed into the bronze layer. Or you might have a "platinum" layer for data that's been further refined and enriched for a specific use case. Regardless of the names and number of layers, the medallion architecture is flexible and can be tailored to meet your organization's particular requirements.


ccna certification training courses malaysia

Comments

Post a Comment

Popular posts from this blog

Azure built-in roles for tables

Azure RBAC provides built-in roles for authorizing access to table data using Microsoft Entra ID and OAuth. Built-in roles that provide permissions to tables in Azure Storage include: Storage Table Data Contributor: Use to grant read/write/delete permissions to Table storage resources. Storage Table Data Reader: Use to grant read-only permissions to Table storage resources. To learn how to assign an Azure built-in role to a security principal, see Assign an Azure role for access to table data. To learn how to list Azure RBAC roles and their permissions, see List Azure role definitions. For more information about how built-in roles are defined for Azure Storage, see Understand role definitions. For information about creating Azure custom roles, see Azure custom roles. Only roles explicitly defined for data access permit a security principal to access table data. Built-in roles such as Owner, Contributor, and Storage Account Contributor permit a security principal to manage a storage acc...
Read more

Explore Dataflows Gen2 in Microsoft Fabric

  In Microsoft Fabric, you can create a Dataflow Gen2 in the Data Factory workload or Power BI workspace, or directly in the lakehouse. Since our scenario is focused on data ingestion, let's look at the  Data Factory  workload experience. Dataflows Gen2 use Power Query Online to visualize transformations.  1. Power Query ribbon Dataflows Gen2 support a wide variety of data source connectors. Common sources include cloud and on-premises relational databases, Excel or flat files, SharePoint, SalesForce, Spark, and Fabric lakehouses. Then there are numerous data transformations possible, such as: Filter and Sort rows Pivot and Unpivot Merge and Append queries Split and Conditional split Replace values and Remove duplicates Add, Rename, Reorder, or Delete columns Rank and Percentage calculator Choose Top N and Bottom N You can also create and manage data source connections, manage parameters, and configure the default data destination in this ribbon. 2. Queries pane The ...
Read more

Select and configure an appropriate method for access to Azure Blobs

Azure Storage supports using Microsoft Entra ID to authorize requests to blob data. With Microsoft Entra ID, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, group, or application service principal. The security principal is authenticated by Microsoft Entra ID to return an OAuth 2.0 token. The token can then be used to authorize a request against the Blob service. Authorization with Microsoft Entra ID is available for all general-purpose and Blob storage accounts in all public regions and national clouds. Only storage accounts created with the Azure Resource Manager deployment model support Microsoft Entra authorization. For optimal security, Microsoft recommends using Microsoft Entra ID with managed identities to authorize requests against blob, queue, and table data, whenever possible. Authorization with Microsoft Entra ID and managed identities provides superior security and ease of use over Shared Key author...
Read more