How to solve semantic layers in Databricks

Databricks metric views are a good start for AI self-service analytics. Why the semantic layer still has to sit across platforms and in the hands of the business.

Many companies are now rolling out self-service analytics with AI. Business users get Genie, Copilot or an internal agent and "talk to their data" directly, instead of waiting for the BI team to build the next dashboard. On Databricks this usually starts with a Genie space on top of Unity Catalog.

Whether people can trust the answers depends on whether the agent knows how the business defines and calculates its KPIs. Providing that knowledge is the job of the semantic layer, which in practice often becomes a long engineering project the business doesn't fully trust. Below I look at what Databricks offers today and where it falls short once metrics span several platforms and their meaning is owned by the business.

One deep blue watercolour form divided into vertical bands of different blues.

Unity Catalog metric views

Databricks' own semantic layer is Unity Catalog metric views. A metric view separates measures from the dimensions used to group and filter them, so revenue is defined once and can be sliced by region, product or month without a new view for every combination.

Metric views are written in YAML or SQL DDL, support star and snowflake joins, composable measures, window measures for period-over-period comparisons, and materialization. You can add synonyms and display names so Genie understands that "turnover" means revenue. They can be queried from SQL, notebooks, AI/BI dashboards, Genie and alerts, and from external BI tools like Power BI and Tableau.

If all your data sits in Databricks and your data team is comfortable maintaining YAML, metric views cover most of what you need.

A blue watercolour rectangle torn into three pieces, each resting on a different coloured ground.

Limits of metric views

Almost no company I meet has all its metrics in one platform. Revenue and margin come from the ERP and land in Databricks. Pipeline lives in the CRM and is reported from Power BI. Headcount sits in an HR tool. Budget and forecast are still in Excel, owned by controlling. Some business units run on Snowflake or Fabric because of an acquisition or a regional IT decision.

A metric view can only describe data that Databricks can query. As soon as a KPI depends on data outside the lakehouse, for example plan versus actual or revenue per FTE, its definition ends up split between YAML in Unity Catalog and a DAX measure or spreadsheet formula somewhere else, and nobody can see the full calculation in one place.

Ownership is the second problem. Metric views are written by engineers, while the people who decide what "net revenue" means sit in finance and controlling and don't work with YAML files or pull requests. The data team ends up interpreting the definition on its own, and finance may only notice months later that returns were treated differently than in the monthly report.

A blue branching watercolour form above a grid of muted squares, joined by a bright blue band.

Metric views also require thinking in the structures of the data platform. Every measure is defined against source tables, joins and columns. Business users don't think about their KPIs that way. They think about the KPIs they care about, often the ones their bonus depends on, and about the other KPIs needed to calculate them, such as price, volume, discounts or cost of goods sold. A metric view has no concept of this value driver tree, so a translation layer is needed between KPI calculations and value drivers on one side and data schemas on the other.

AI agents make both problems more visible. To answer "why did margin drop in Q3?", an agent has to know which drivers feed margin and in which system each of them is calculated, along with the definition the CFO approved. When that knowledge is spread across platforms and people, the agent guesses.

Blue watercolour bleeding down from a wide top field into four separate fields below.

Requirements for a cross-platform semantic layer

Metric definitions are business knowledge, so they shouldn't be a feature of one data platform. They belong one level above Databricks, Snowflake or Power BI and get pushed down into each of them from there. The platform that holds them needs the following properties.

Cross-platform by design. One place holds every metric, regardless of whether it is computed in Databricks, Snowflake, Fabric, Power BI or a controlled spreadsheet. For each metric it records where the implementation lives on each platform. Databricks metric views become one target among several, generated from that central definition and kept in sync with it.

Built for business users. Controllers and analysts need to be able to propose and approve definitions without writing YAML or SQL. That requires plain-language definitions with the driver tree behind each KPI visible, and a review step where finance signs off before a change goes live.

Focused on metrics and their drivers. Enterprise-wide ontologies and knowledge graphs tend to stall because nobody in the business has time to maintain them. KPIs get management attention every month, so the business does care about getting them right. Modelling the metrics together with their slices and the value driver tree that connects them gives an agent enough context to understand how the business works.

A translation layer between KPIs and data schemas. The value driver tree and the calculation logic are modelled in business terms, independent of table structures, and each KPI is then mapped to its implementation on the data platform. When a source system is migrated or a table is renamed, the business definition stays the same and only the mapping changes.

Ready for agents. Genie, Copilot, ChatGPT and internal agents should all read the same definitions through a standard interface such as MCP, so the answer to "what is our churn rate?" is the same whichever agent a team uses.

A practical way to get there

  1. Start from the management report, not the tables. Take the KPIs the leadership team looks at every month. That list is usually 30 to 80 metrics, and it is where disagreements cost the most.
  2. Write each definition in business language first. Agree with finance on what the number means, which slices apply and which drivers feed it. Record who owns it and who approved it.
  3. Map each metric to where it is computed today. For many metrics that is Databricks, for others Power BI, Snowflake or a planning spreadsheet. This shows the gaps and the duplicates.
  4. Generate the platform implementations from the central definition. In Databricks these are metric views, in Power BI semantic models. Copies maintained by hand in each tool drift apart over time.
  5. Route every change through a review. A new metric or a changed formula is proposed, reviewed by the business owner and then rolled out to all platforms at once.
  6. Point your agents at the governed layer. Connect Genie, Copilot and any other agent to the same definitions, and test them with the questions business teams and management actually ask.

Conclusion

Databricks metric views work well as the execution layer for metrics computed in the lakehouse. In most companies, though, metrics span more systems than Databricks, and the people who own their meaning don't work in YAML. The business definitions should therefore be maintained in a platform that spans all data platforms and that finance, sales or supply chain teams can work in directly, with metric views generated from it.

Sources