This solution accelerator provides a unified data foundation with integrated data architecture leveraging Microsoft Fabric, OneLake, Microsoft Purview, and Azure Databricks to deliver a unified, integrated, and governed analytics platform.
Built with principles of medallion lakehouse architecture, the solution accelerator supports data mesh concepts with a sample implementation. It provides domain schemas and sample data as a framework for shared domains (customer, product), finance, and sales across multiple channels. You can easily adopt the framework and update the domain models with your own. The solution seamlessly integrates sales channel data hosted in Azure Databricks, utilizing Fabric's data mirroring and shortcut to bring this data into the gold tier for unified analytics and reporting. Pre-built Power BI dashboards showcase advanced sales analytics. Advanced data governance is powered by Microsoft Purview, ensuring compliance and transparency.
This solution accelerator demonstrates how organizations can unify, govern, and analyze data across multiple domains and platforms using modern lakehouse architecture and robust governance, enabling rapid development of analytics solutions for diverse business needs.
We have built flexibility and options so that you do not have to provision all three platforms (Microsoft Fabric, Microsoft Purview, and Azure Databricks) all at once. Please see details in the Solution Overview section below for more details.
Key use cases include:
- Unified data platform uses Microsoft Fabric as core architecture components, with choices for integration with other platforms
- Customer and product master data management (data models, source file processing, data validation, consolidations)
- Sales analytics across data stored in Fabric and Azure Databricks
- Finance data management and foundation for reporting
- Power BI dashboards for sales data with actionable business insights
We have built a flexible and configurable architecture with plug-and-play options, providing you with four architecture choices. You can deploy any of the following four architectures based on your organizational needs:
- Core Medallion Architecture in Microsoft Fabric only
- Core Medallion Architecture in Microsoft Fabric + Microsoft Purview
- Core Medallion Architecture in Microsoft Fabric + Azure Databricks
- Core Medallion Architecture in Microsoft Fabric + Microsoft Purview + Azure Databricks
For a detailed feature description of each architecture option, please refer to Solution Architecture and Options.
The architecture below illustrates the solution architecture built with option 4.
 |
|---|
If you'd like to customize the solution accelerator, here are some common areas to start:
You can modify the data models and notebooks in different folders under the src folder. Please note if any part is modified, you will need to modify the associated parts accordingly, as the data model (schemas and tables), notebooks, Power BI semantic models, Power BI dashboards, and sample data are a cohesive set of resources working together as designed.
Update Data Management Notebooks
Update Bronze to Silver Data Processing Notebooks
Update Silver to Gold Data Processing Notebooks
Update Runner Notebooks if Notebooks have been added or dropped
What's New in Microsoft Fabric
Microsoft Fabric Power BI Adoption roadmap
Click to learn more about the key features this solution enables
-
Core Medallion Architecture in Fabric
Core medallion architecture in unified Microsoft Fabric Platform, with cross-domain data models covering shared (customer, product), finance, and sales across multiple channels. The solution is packaged with 48 Fabric PySpark Notebooks and 2 SQL scripts. All of them are deployed to the Fabric workspace with an automated deployment process. -
Raw data in Bronze to Silver Lakehouse Tables with Automated Execution
Complete and automated process for raw data processing from bronze to validated data populated to silver tables. -
Silver Lakehouse Data Flows into Gold Lakehouse with Automated Execution
Completed and automated process for validated data in silver tables flowing into gold tables for enrichment and utilization. -
Power BI Semantic Models and Sales Analysis Dashboard
Power BI semantic models using gold tables, producing dashboards with comprehensive sales analysis. -
Integration with Azure Databricks
Integration with Azure Databricks with Mirroring and Shortcut to eliminate the need for data movement. Additional sales data from Azure Databricks is made available to Fabric via data mirroring and shortcut. -
Microsoft Purview for Data Governance
Microsoft Purview reviews and governs selected resources in the Microsoft Fabric workspace, providing capabilities such as scanning data, data discovery, and metadata for the data stored in the Fabric workspace gold tier.
If you choose option 1, "Core Medallion Architecture in Microsoft Fabric only", please follow the quick deploy steps in guide to deploy medallion architecture with PowerBI dashboard in Fabric. Otherwise, please follow the instructions provided in deployment guide for all options.
You have followed the provisioning guidance pages for Provisioning Fabric, Provisioning Azure Databricks, Provisioning Purview based on your choice of architecture.
To deploy this solution accelerator, ensure you have access to an Azure subscription with the necessary permissions to create resource groups, resources, app registrations, and assign roles at the resource group level. This should include Contributor role at the subscription level and Role Based Access Control role on the subscription and/or resource group level. Follow the steps in Azure Account Set Up.
Licensing and cost for establishing Fabric can be found at Microsoft Fabric concepts and licenses and Microsoft Fabric Pricing.
Consumption model and pricing information for Azure Databricks can be found at Serverless DBU consumption by SKU - Azure Databricks.
Microsoft Purview billing models can be found at Microsoft Purview billing models.
Note: This is not meant to outline all costs as selected SKUs, scaled use, customizations, and integrations into your own tenant can affect the total consumption of this sample solution. The sample pricing sheet is meant to give you a starting point to customize the estimate for your specific needs.
| Product | Description | Cost |
|---|---|---|
| Microsoft Fabric | Core Medallion Architecture in Microsoft Fabric, and Unified Data Platform for integration with other platforms such as Azure Databricks and Snowflakes. | Pricing |
| Azure Databricks | Azure Databricks stores sales data for one channel and the data is used by Microsoft Fabric through data mirroring and shortcut. | Pricing |
| Microsoft Purview | Data Governance, data security, and risk management. | Pricing |
⚠️ Important: To avoid unnecessary costs, remember to take down your app if it's no longer in use, either by deleting the resource group in the Portal or runningazd down.
- Data Engineer creates or updates PySpark notebooks to complete data processing and data management tasks
- Data Engineer creates or updates T-SQL scripts to manage the data using Fabric SQL End Points
- Data Engineer tests end-to-end data flow
- Sales Analysts create or update Power BI Semantic models based off Gold tier lakehouse
- Sales Analysts create or update Power BI dashboards.
 |
|---|
The data engineer can also upload additional source data to the bronze lakehouse and recreate the data processing flow from bronze to silver and then to gold.
Click to learn more about what value this solution provides
-
Build Modern Lakehouse Architecture
Demonstrate how organizations can unify, govern, and analyze data across multiple domains and platforms using modern lakehouse architecture and robust governance, enabling rapid development of analytics solutions for diverse business needs. The solution accelerator provides a complete set of sample data for testing. -
Processing Raw Source Data in Bronze Lakehouse and Validate in Silver Lakehouse, and Enrich Data In Gold Lakehouse
Provide core solution with domain models, data processing and data management code with automated execution to copy raw source data in Bronze lakehouse to populate Silver Lakehouse data models and then Gold Lakehouse Data models, to prepare data for analysis. -
Seamless Integration with Azure Databricks with Data Mirroring and Shortcut, Microsoft Purview for Data Governance
Provide architecture options to integrate with Azure Databricks and Microsoft Purview for data governance.
-
Semantic Models and Power BI Dashboards
Provide Semantic Models and Power BI Dashboards for Sales Analysis, such as year-over-year sales analysis, best-selling products, and sales distribution across customer segments.
This template uses Azure Key Vault to store all connections to communicate between resources.
This template also uses Managed Identity for local development and deployment.
To ensure continued best practices in your own repository, we recommend that anyone creating solutions based on our templates ensure that the GitHub secret scanning setting is enabled.
You may want to consider additional security measures, such as:
- Enabling Microsoft Defender for Cloud to secure your Azure resources.
- Protecting the Azure Container Apps instance with a firewall and/or Virtual Network.
Click here to learn more about common questions about this solution.
Check out similar solution accelerators
| Solution Accelerator | Description |
|---|---|
| Agentic applications for unified data foundation | Agentic AI application that provides nature language query of the data using unified data foundation. Description of solution accelerator |
Have questions, find a bug, or want to request a feature? Submit a new issue on this repo and we'll connect.
Please refer to Transparency FAQ for responsible AI transparency details of this solution accelerator.
To the extent that the Software includes components or code used in or derived from Microsoft products or services, including without limitation Microsoft Azure Services (collectively, “Microsoft Products and Services”), you must also comply with the Product Terms applicable to such Microsoft Products and Services. You acknowledge and agree that the license governing the Software does not grant you a license or other right to use Microsoft Products and Services. Nothing in the license or this ReadMe file will serve to supersede, amend, terminate or modify any terms in the Product Terms for any Microsoft Products and Services.
You must also comply with all domestic and international export laws and regulations that apply to the Software, which include restrictions on destinations, end users, and end use. For further information on export restrictions, visit https://aka.ms/exporting.
You acknowledge that the Software and Microsoft Products and Services (1) are not designed, intended or made available as a medical device(s), and (2) are not designed or intended to be a substitute for professional medical advice, diagnosis, treatment, or judgment and should not be used to replace or as a substitute for professional medical advice, diagnosis, treatment, or judgment. Customer is solely responsible for displaying and/or obtaining appropriate consents, warnings, disclaimers, and acknowledgements to end users of Customer’s implementation of the Online Services.
You acknowledge the Software is not subject to SOC 1 and SOC 2 compliance audits. No Microsoft technology, nor any of its component technologies, including the Software, is intended or made available as a substitute for the professional advice, opinion, or judgement of a certified financial services professional. Do not use the Software to replace, substitute, or provide professional financial advice or judgment.
BY ACCESSING OR USING THE SOFTWARE, YOU ACKNOWLEDGE THAT THE SOFTWARE IS NOT DESIGNED OR INTENDED TO SUPPORT ANY USE IN WHICH A SERVICE INTERRUPTION, DEFECT, ERROR, OR OTHER FAILURE OF THE SOFTWARE COULD RESULT IN THE DEATH OR SERIOUS BODILY INJURY OF ANY PERSON OR IN PHYSICAL OR ENVIRONMENTAL DAMAGE (COLLECTIVELY, “HIGH-RISK USE”), AND THAT YOU WILL ENSURE THAT, IN THE EVENT OF ANY INTERRUPTION, DEFECT, ERROR, OR OTHER FAILURE OF THE SOFTWARE, THE SAFETY OF PEOPLE, PROPERTY, AND THE ENVIRONMENT ARE NOT REDUCED BELOW A LEVEL THAT IS REASONABLY, APPROPRIATE, AND LEGAL, WHETHER IN GENERAL OR IN A SPECIFIC INDUSTRY. BY ACCESSING THE SOFTWARE, YOU FURTHER ACKNOWLEDGE THAT YOUR HIGH-RISK USE OF THE SOFTWARE IS AT YOUR OWN RISK.