Published on

The Open Standards within Microsoft Fabric and Purview

Authors

There is a tendency to describe Microsoft Fabric and Purview as if they were completely self-contained Microsoft platforms. That is not quite how I see them.

Both products bring together open formats, open source projects, and standard interfaces. Microsoft is adding a managed service, an integrated user experience, and a set of proprietary features around those foundations. That distinction matters when thinking about portability and lock-in.

Microsoft Fabric

Fabric is Microsoft's unified analytics platform. It combines data engineering, data science, real-time analytics, and business intelligence in one service. The Microsoft offering is the integration: a shared workspace, OneLake, managed compute, permissions, deployment, and a common interface for these workloads.

The underlying components are less proprietary than the product experience.

Delta Lake

The main table format in Fabric is Delta Lake. OneLake, the shared storage layer at the centre of Fabric, uses Delta tables for Lakehouse data.

Delta Lake is an open source project, originally developed by Databricks and now hosted by the Linux Foundation. It is not a Microsoft-specific format. Other engines that support Delta can read the same tables.

That makes the data layer more portable than a proprietary binary format would be. It does not make the whole Fabric implementation portable: workspace metadata, permissions, shortcuts, pipelines, and other service features still belong to Fabric.

Apache Parquet

Delta Lake itself uses Apache Parquet, an open columnar file format for structured data.

The data files in OneLake are .parquet files. They can be read by the many tools that support Parquet, although the Delta transaction log is also needed when treating the files as a table rather than as individual files.

# The physical files in OneLake look like this:
myworkspace.dfs.core.windows.net/
  mylakehouse.Lakehouse/
    Tables/
      mytable/
        _delta_log/
        part-00000-abc.snappy.parquet
        part-00001-def.snappy.parquet

The Delta log is a JSON transaction log that tracks changes. The data files themselves are plain Parquet.

Apache Spark

The compute engine in Fabric Notebooks and Spark Job Definitions is Apache Spark.

Fabric runs a Microsoft-managed Spark runtime, but the code is standard PySpark or Scala Spark. That makes the skills and much of the notebook code transferable to Databricks, Azure HDInsight, or a self-hosted Spark cluster. The runtime version, libraries, authentication, and Fabric-specific helpers still need to be dealt with separately.

# Standard PySpark — runs identically in Fabric or Databricks
from pyspark.sql.functions import col, count

df = spark.read.format("delta").load("Tables/orders")
df.groupBy("region").agg(count("*").alias("total")).show()

This is the value Microsoft is providing here: managed Spark, integrated storage, and a polished UI rather than a new compute model.

Apache Kafka

Eventstream is Fabric's real-time data ingestion feature. It can ingest events from Kafka-compatible sources.

Fabric also exposes Kafka-compatible endpoints in parts of its real-time offering. Existing Kafka producers can therefore use a familiar protocol, while Microsoft supplies the hosted endpoint, security, scaling, and integration with the rest of Fabric.

{
  "bootstrap.servers": "<fabric-eventhouse>.servicebus.windows.net:9093",
  "security.protocol": "SASL_SSL",
  "sasl.mechanism": "PLAIN"
}

The configuration pattern is familiar from Azure Event Hubs and other Kafka-compatible brokers. Compatibility with Kafka does not mean that every broker feature or operational detail is identical.

Apache Arrow

Fabric uses Apache Arrow for some in-memory data interchange. Arrow provides a language-independent columnar memory format, which helps data move between Spark, Python, and the Fabric runtime without repeated serialisation.

You are unlikely to interact with Arrow directly. It is one of the open components beneath the managed runtime, rather than a Fabric feature that you need to learn separately.

Microsoft Purview

Purview is Microsoft's data governance service. It handles cataloguing, lineage, classification, scanning, and access policies.

Here the distinction is especially important: the standards describe how metadata and lineage can be represented, while Purview supplies the catalog, connectors, identity integration, governance experience, and Azure operations around them.

Apache Atlas

Purview exposes an API based on the entity model from Apache Atlas. Atlas is an open source metadata and governance framework from the Apache Software Foundation, originally developed for the Hadoop ecosystem.

An Atlas-compatible model gives tools a familiar way to represent entities, classifications, and relationships. It should not be read as a promise that Purview and an arbitrary Atlas deployment are interchangeable: the service, connectors, authentication, and Microsoft-specific extensions remain part of Purview.

GET https://{purview-account}.purview.azure.com/catalog/api/atlas/v2/entity/guid/{guid}
Authorization: ******

If you have worked with Atlas before, the Purview data model will look familiar. The API compatibility is useful, but it is not the same thing as being able to move a complete Purview catalog elsewhere without migration work.

OpenLineage

OpenLineage is an open standard for tracking data lineage: who produced a dataset, when, and from what inputs.

Purview supports ingesting OpenLineage events, allowing pipelines built outside Azure to report lineage in a standard format. Spark jobs, dbt models, and Airflow DAGs can emit OpenLineage events.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-07-25T10:00:00Z",
  "run": {
    "runId": "a1b2c3d4-..."
  },
  "job": {
    "namespace": "my-spark-cluster",
    "name": "transform_orders"
  },
  "inputs": [{"namespace": "s3://datalake", "name": "raw/orders"}],
  "outputs": [{"namespace": "s3://datalake", "name": "processed/orders"}]
}

This means lineage in Purview does not have to be limited to data flowing through Azure services. The Microsoft contribution is the ingestion endpoint and the way those events are presented alongside Purview's scans and catalog.

OpenAPI

Purview APIs are documented using OpenAPI and exposed over REST. This is not unique to Purview, but it is useful: catalog and scanning operations can be automated without relying only on the portal. The API is still a Microsoft service API, with Azure identity, permissions, resource names, and service-specific behaviour.

Why This Matters

There are a few practical implications worth drawing out.

The data layer is more portable than the service layer. Because OneLake Lakehouses use Delta Lake on Parquet, other engines can read the data without an export step. You can point Databricks, a local Spark environment, or DuckDB at the same storage account and query supported tables directly.

# DuckDB reading Delta tables from OneLake directly
import duckdb

conn = duckdb.connect()
conn.execute("INSTALL delta; LOAD delta;")
df = conn.execute("""
    SELECT * FROM delta_scan('abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/orders')
""").fetchdf()

Skills and integrations transfer better than the branding suggests. PySpark skills developed on Fabric apply to other Spark environments. Knowledge of the Atlas entity model applies to other Atlas-based catalogs. OpenLineage integrations are not inherently tied to Purview.

Lock-in has not disappeared. The workflows, UX, managed runtimes, identity model, permissions, connectors, and convenience features are where Fabric and Purview become Microsoft-specific. Open formats reduce the cost of leaving, but they do not remove the work of replacing those services.

Conclusion

Microsoft Fabric and Purview are useful because Microsoft has packaged these components into managed services. The integration, identity, operations, connectors, and user experience are the product; the open formats and protocols are the foundation.

That is a more useful way to evaluate them than treating either product as magic or as a completely proprietary black box. Open standards give you options. Microsoft's managed layer gives you convenience. The architecture decision is about how much of each you want, and how deliberately you keep the service-specific parts replaceable.

References