time, constructing a medallion structure in Microsoft Cloth meant stitching collectively a small orchestra of transferring components: notebooks for the transformations, pipelines for orchestration, schedules for refresh, customized code for knowledge high quality checks, and the Monitor Hub for maintaining a tally of whether or not something really labored. Each layer labored \u2013 till one thing didn\u2019t, and then you definately had to determine which layer broke, why, and which downstream layers acquired affected alongside the best way.<\/p>\n
In the event you\u2019ve ever tried to debug a silver layer that didn\u2019t replace as a result of the bronze pocket book failed three hours in the past, you realize precisely what I\u2019m speaking about.<\/p>\n
Then, at FabCon Atlanta in March 2026, materialized lake views (MLVs) went usually accessible. And the story they\u2019re telling is easy: what in case your complete medallion pipeline may very well be a number of SELECT statements?<\/p>\n
Let me stroll you thru the entire thing \u2013 what they’re, how they work, what modified between preview and GA, and the place they match (and the place they don\u2019t) in your structure.<\/p>\n
Materialized Lake View \u2013 WHAT?<\/h2>\n
A materialized lake view is a endured, robotically refreshed view outlined in Spark SQL or PySpark. You write a SELECT question that describes the transformation you need, and Cloth takes care of execution, storage, refresh, dependency monitoring, and knowledge high quality enforcement.<\/p>\n
The result’s saved as a Delta desk in your lakehouse. So downstream customers, akin to Energy BI Direct Lake, Spark notebooks, SQL endpoints, can question it identical to another Delta desk. No particular dealing with, no totally different syntax.<\/p>\n
To place it in plain English: an MLV is nothing else however a SELECT assertion that discovered to materialize itself, handle its personal dependencies, schedule its personal refresh, and test its personal knowledge high quality.<\/p>\n OK, that\u2019s good. However what does that really change?<\/p>\n That\u2019s a good query. Earlier than MLVs, constructing a single bronze-to-silver-to-gold move appeared roughly like this: you\u2019d write a pocket book for every transformation, arrange a Knowledge Manufacturing facility pipeline to name them in the best order, configure schedules, construct customized validation logic, after which wire up the Monitor Hub to observe for failures. 5 totally different surfaces, 5 various things to debug when one thing breaks.<\/p>\n With MLVs, all of that collapses into declarative SQL. You describe what you need. Cloth figures out the remainder.<\/p>\n Each MLV strikes via 4 phases. In keeping with the\u00a0Microsoft documentation, understanding them is the muse for all the pieces else:<\/p>\n Create\u00a0\u2013 You write the Spark SQL (or PySpark) that defines the transformation. Cloth shops the definition and materializes the preliminary outcome as a Delta desk.<\/p>\n Refresh\u00a0\u2013 When supply knowledge modifications, Cloth chooses the optimum technique: incremental (course of solely modifications), full (rebuild), or skip (no modifications detected).<\/p>\n Question\u00a0\u2013 Any software or software reads the materialized outcome. They don\u2019t know \u2013 and don\u2019t have to know \u2013 that it\u2019s an MLV.<\/p>\n Monitor\u00a0\u2013 Refresh historical past, execution standing, knowledge high quality metrics, and lineage are all tracked and visualised natively in Cloth.<\/p>\n Now let\u2019s dive into every bit.<\/p>\n Right here\u2019s the total Spark SQL pseudo-code syntax for creating an MLV,\u00a0straight from the Microsoft Be taught reference:<\/p>\n CREATE [OR REPLACE] MATERIALIZED LAKE VIEW [IF NOT EXISTS]<\/p>\n [workspace.lakehouse.schema].MLV_Identifier<\/p>\n [(CONSTRAINT constraint_name CHECK (condition) [ON MISMATCH DROP | FAIL], …)]<\/p>\n [PARTITIONED BY (col1, col2, …)]<\/p>\n [COMMENT \u201cdescription\u201d]<\/p>\n [TBLPROPERTIES (\u201dkey1\u201d=\u201dval1\u201d, …)]<\/p>\n AS select_statement<\/p>\n An actual instance \u2013 cleansing order knowledge joined from merchandise and orders, with a knowledge high quality constraint and partitioning:<\/p>\n CREATE OR REPLACE MATERIALIZED LAKE VIEW silver.cleaned_order_data<\/p>\n (<\/p>\n CONSTRAINT valid_quantity CHECK (amount > 0) ON MISMATCH DROP<\/p>\n )<\/p>\n PARTITIONED BY (class)<\/p>\n COMMENT \u201cCleaned order knowledge joined from merchandise and orders\u201d<\/p>\n AS<\/p>\n SELECT<\/p>\n p.productID, p.productName, p.class,<\/p>\n o.orderDate, o.amount, o.totalAmount<\/p>\n FROM bronze.merchandise p<\/p>\n INNER JOIN bronze.orders o ON p.productID = o.productID<\/p>\n Two issues value flagging immediately. First, MLV names are case-insensitive (MyView\u00a0turns into\u00a0myview). Second, all-uppercase schema names (like\u00a0MYSCHEMA) aren\u2019t supported, so use both blended or lowercase.<\/p>\n You additionally want a schema-enabled lakehouse and Cloth Runtime 1.3 or larger. In case your lakehouse doesn\u2019t have schemas turned on, MLVs aren\u2019t accessible, that\u2019s the very first prerequisite.<\/p>\n Right here\u2019s the place MLVs cease being intelligent and begin being sensible.<\/p>\n When supply knowledge modifications, Cloth\u2019s optimum refresh engine appears at each MLV within the lineage and asks a collection of questions: Did something really change? Can I course of simply the modifications? Or do I have to rebuild from scratch?<\/p>\n Three potential outcomes:<\/p>\n Skip refresh\u00a0\u2013 supply knowledge hasn\u2019t modified. Don\u2019t waste compute. Transfer on.<\/p>\n Incremental refresh\u00a0\u2013 course of solely the brand new or modified rows. Quick, low-cost, very best.<\/p>\n Full refresh\u00a0\u2013 rebuild the entire thing. Slowest path, used when incremental isn\u2019t secure or potential.<\/p>\n However, and that is vital, incremental refresh isn\u2019t free. It has conditions:<\/p>\n The Delta change knowledge feed (CDF) should be enabled on each supply desk referenced by the MLV (delta.enableChangeDataFeed=true).<\/p>\n The supply should be a Delta desk. Non-Delta sources all the time get a full refresh.<\/p>\n The information should be append-only. In case your supply has updates or deletes, Cloth falls again to a full refresh.<\/p>\n The question should use solely supported SQL constructs (extra on this in a second).<\/p>\n With out CDF enabled, optimum refresh can solely select between skip and full. With CDF on, the total incremental path opens up. Enabling CDF in your supply tables has no measurable storage or efficiency affect for append-only workloads, so there\u2019s little or no purpose to not flip it on:<\/p>\n ALTER TABLE bronze.orders SET TBLPROPERTIES (delta.enableChangeDataFeed = true);<\/p>\n ALTER TABLE bronze.merchandise SET TBLPROPERTIES (delta.enableChangeDataFeed = true);<\/p>\n Can it get higher than this? Really, sure! And, that is the place the GA story actually begins.<\/p>\n MLVs had been launched in preview at Construct 2025. Between then and GA in March 2026, Microsoft closed crucial gaps. 5 main modifications turned MLVs from \u201cfascinating\u201d into \u201cproduction-ready\u201d:<\/p>\n Multi-schedule assist<\/p>\n Broader incremental refresh protection<\/p>\n PySpark authoring (preview)<\/p>\n In-place updates with Exchange<\/p>\n Stronger knowledge qc<\/p>\n Let me take them one by one.<\/p>\n In preview, you would solely refresh all MLVs in a lakehouse on a single schedule. Want finance to replace hourly, however analytics to replace each six hours? You needed to work round it with notebooks, which broke dependency consciousness, error reporting, and retry logic. Pocket book-triggered refreshes don\u2019t floor MLV error particulars. Failures seem solely in cell output, and dependent views haven’t any consciousness of them. Errors can persist week after week with out anybody figuring out the pipeline is damaged.<\/p>\n Now you possibly can outline named schedules inside a single lakehouse, every concentrating on a particular subset of views. Finance pipeline hourly. Analytics each six hours. Advertising and marketing each quarter-hour. All in the identical lakehouse, no customized code.<\/p>\n When a named schedule runs, Cloth nonetheless refreshes all upstream dependencies within the right order, runs unbiased views in parallel, and surfaces errors centrally. If a run is already in progress when a schedule fires, the brand new run is skipped, and the subsequent window proceeds as anticipated \u2013 so that you don\u2019t have to fret about overlapping runs stomping on one another.<\/p>\n Incremental refresh used to fall again to full very often, as a result of the record of \u201csupported\u201d SQL constructs was slender. At GA, that record expanded considerably. MLVs now refresh incrementally when the definition consists of:<\/p>\n Aggregations like\u00a0COUNT\u00a0and\u00a0SUM\u00a0with\u00a0GROUP BY<\/p>\n Left outer joins and left semi joins<\/p>\n Widespread desk expressions (CTEs)<\/p>\n That\u2019s a significant change. Most real-world medallion pipelines I\u2019ve labored on use precisely these patterns, and now they qualify for incremental processing with out being rewritten. With optimum refresh, a built-in choice engine examines every refresh, evaluates the quantity of modified knowledge in opposition to the price of a full recomputation, and robotically chooses the sooner path.<\/p>\n I hear you, I hear you: Nikola, what occurs if my question makes use of one thing the engine can\u2019t deal with incrementally? Don\u2019t fear, it\u2019s a lot simpler than it sounds:) Utilizing unsupported constructs doesn\u2019t forestall you from creating the MLV. It solely implies that Cloth makes use of a full refresh as an alternative of an incremental one. Optimum refresh robotically falls again to full when wanted, so that you don\u2019t usually have to drive it. In the event you do need to drive one (for instance, to reprocess knowledge after a correction), there\u2019s a one-liner for that:<\/p>\n REFRESH MATERIALIZED LAKE VIEW silver.cleaned_order_data FULL;<\/p>\n This one is big! SQL is nice till your transformation logic includes a customized Python library, an ML inference name, or a UDF that wraps complicated enterprise guidelines. You then\u2019d hit a wall as MLVs had been SQL-only.<\/p>\n With PySpark authoring, now you can create, refresh, and change MLVs from Cloth notebooks utilizing PySpark and the acquainted DataFrameWriter API. The\u00a0fmlv\u00a0module exposes a decorator-based sample,\u00a0documented within the official PySpark MLV reference:<\/p>\n import fmlv<\/p>\n from pyspark.sql import features as F<\/p>\n @fmlv.materialized_lake_view(<\/p>\n title=\u201dLH1.silver.customer_silver\u201d,<\/p>\n remark=\u201dCleaned & enriched buyer silver MLV\u201d,<\/p>\n partition_cols=[\u201dyear\u201d, \u201ccity\u201d],<\/p>\n table_properties={\u201ddelta.enableChangeDataFeed\u201d: \u201ctrue\u201d},<\/p>\n change=True<\/p>\n )<\/p>\n @fmlv.test(title=\u201dnon_null_sales\u201d, situation=\u201dgross sales IS NOT NULL\u201d, motion=\u201dDROP\u201d)<\/p>\n def customer_silver():<\/p>\n df = spark.learn.desk(\u201dbronze.customer_bronze\u201d)<\/p>\n cleaned_df = df.filter(F.col(\u201dgross sales\u201d).isNotNull())<\/p>\n enriched_df = cleaned_df.withColumn(\u201dsales_in_usd\u201d, F.col(\u201dgross sales\u201d) * 1.0)<\/p>\n return enriched_df<\/p>\n A number of PySpark gotchas value figuring out about:<\/p>\n PySpark MLVs are nonetheless in preview on the time of writing.<\/p>\n Right this moment, PySpark-authored MLVs all the time carry out a full refresh. Optimum refresh for PySpark is on the roadmap, however not right here but.<\/p>\n The\u00a0@fmlv\u00a0decorator doesn\u2019t assist dynamic parameters or variables. All parameters should be hardcoded.<\/p>\n You possibly can\u2019t create an MLV from a PySpark short-term view (createOrReplaceTempView) \u2013 the engine can\u2019t see session-scoped views. Use bodily Delta tables or different MLVs as sources.<\/p>\n Don\u2019t delete the pocket book the place the MLV is outlined. Scheduled refresh fails with out it.<\/p>\n So in case your transformation could be expressed cleanly in SQL, SQL continues to be the higher selection for efficiency. PySpark MLVs unlock the circumstances the place SQL alone gained\u2019t do.<\/p>\n Enterprise logic modifications. A filter shifts. A be part of good points a column. An aggregation provides a metric. In preview, updating an MLV definition required dropping and recreating it, which misplaced refresh historical past and compelled downstream customers to reconnect.<\/p>\n Now, with the Exchange functionality, you replace an MLV\u2019s definition in place. Cloth validates the brand new logic, swaps it in, and preserves the view\u2019s identification, metadata, and lineage. Downstream dependencies stay intact. Works for each SQL (CREATE OR REPLACE) and PySpark (change=True).<\/p>\n That is a kind of \u201cbeneath the radar\u201d GA options that doesn\u2019t get headlines however issues enormously day-to-day. In the event you\u2019ve ever needed to coordinate dropping and recreating a closely consumed desk whereas manufacturing is operating, you realize the ache. That goes away with this.<\/p>\n Knowledge high quality constraints are nothing new in MLVs, however at GA, they acquired a severe improve. Now you can:<\/p>\n Use expression-based logic that mixes a number of columns<\/p>\n Apply arithmetic and built-in features inside a single rule<\/p>\n Invoke session-scoped user-defined features for validation logic that lives in Python fairly than SQL<\/p>\n Mix that with the auto-generated knowledge high quality experiences, and also you get one thing near a built-in knowledge observability layer. You possibly can rapidly spot which guidelines fail most frequently, which views they have an effect on, and the way tendencies shift over time, with out constructing a separate monitoring pipeline.<\/p>\n When one MLV references one other (or a desk), Cloth infers the connection robotically. No handbook configuration, no exterior orchestration software. The dependencies are found out of your SQL.<\/p>\n That dependency graph turns into a visible lineage in your lakehouse. Every node represents a change. Arrows present execution order. Cloth makes positive that when bronze knowledge lands, the bronze-to-silver MLV runs first, then the silver-to-gold MLV runs in opposition to the freshly up to date silver.<\/p>\n That is the place the declarative method actually pays off. You\u2019re not writing pipelines. You\u2019re not defining orchestration. You\u2019re writing what every layer ought to seem like, and Cloth figures out the order. That is the great thing about a declarative method:)<\/p>\n A number of helpful behaviors to find out about:<\/p>\n Impartial views run in parallel<\/p>\n Errors floor centrally as an alternative of getting misplaced in pocket book cell output<\/p>\n The lineage view auto-refreshes each two minutes when a run is in progress<\/p>\n All shortcuts are handled as supply entities within the lineage view<\/p>\n You possibly can connect a customized Spark setting to materialized lake views lineage to optimise efficiency and useful resource utilization throughout refresh<\/p>\n I touched on this above, nevertheless it deserves its personal part as a result of it\u2019s one of many issues that makes MLVs really feel totally different from a hand-built pipeline.<\/p>\n Each MLV can have a number of knowledge high quality constraints connected:<\/p>\n CREATE OR REPLACE MATERIALIZED LAKE VIEW silver.valid_orders<\/p>\n (<\/p>\n CONSTRAINT positive_quantity CHECK (amount > 0) ON MISMATCH DROP,<\/p>\n CONSTRAINT valid_date CHECK (orderDate >= \u20182020-01-01\u2019) ON MISMATCH FAIL<\/p>\n )<\/p>\n AS<\/p>\n SELECT * FROM bronze.orders<\/p>\n Two motion sorts:<\/p>\n DROP\u00a0\u2013 violating rows are eliminated, the rely is logged within the lineage view, and the pipeline retains going<\/p>\n FAIL\u00a0\u2013 the refresh stops on the first violation. That is additionally the default if you happen to don\u2019t specify<\/p>\n If a number of constraints are current and each behaviors are configured, FAIL takes priority.<\/p>\n Violations floor within the lineage view and run particulars. Superb, however what does that really seem like in apply? Properly, within the knowledge high quality report, you\u2019ll see counts by constraint, by view, over time. So if a constraint that usually drops 0.1% of rows instantly drops 15%, you\u2019ll see the spike and know precisely which rule failed and which view it belongs to. That\u2019s a top quality sign you\u2019d in any other case should construct by hand.<\/p>\n The Microsoft docs additionally observe that the brand new expression-based constraints assist built-in Spark\/SQL features like\u00a0UPPER(),\u00a0LOWER(),\u00a0TRIM(),\u00a0COALESCE(),\u00a0INITCAP(), and\u00a0DATE_FORMAT(), so your CHECK circumstances could be richer than simply easy comparisons.<\/p>\n MLVs aren’t a hammer for each nail. The Microsoft documentation is unusually direct about the place they match and the place they don\u2019t.<\/p>\n Use MLVs when you could have:<\/p>\n Incessantly accessed aggregations (every day totals, month-to-month metrics) the place precomputed outcomes beat re-running costly queries<\/p>\n Complicated joins throughout a number of massive tables that must be constant for all customers<\/p>\n Knowledge high quality guidelines you need to apply uniformly, declaratively<\/p>\n Reporting datasets that mix knowledge from a number of sources and profit from computerized refresh<\/p>\n Medallion design sample \u2013 bronze to silver to gold outlined as SQL transformations<\/p>\n Don\u2019t use MLVs when:<\/p>\n The question runs as soon as or hardly ever \u2013 precomputing gained\u2019t assist<\/p>\n Transformations are easy and quick already<\/p>\n You want non-SQL logic like ML inference, API calls, or complicated Python processing \u2013 notebooks are nonetheless higher (although PySpark MLVs are beginning to bridge this hole)<\/p>\n You want sub-second latency for streaming \u2013 that\u2019s Actual-Time Intelligence territory<\/p>\n I\u2019ll add a private observe right here. I\u2019m at the moment deep in a Microsoft Cloth engagement the place the silver layer design selection \u2013 Warehouse vs. Lakehouse with MLVs \u2013 is actively on the desk. And what I maintain coming again to is that this: MLVs aren\u2019t competing with the Warehouse the best way some individuals body it. They\u2019re competing with the spaghetti of notebooks-and-pipelines that you simply\u2019d in any other case construct contained in the Lakehouse. In case your staff is already SQL-fluent and your transformations reside naturally in SELECT statements, the case for MLVs because the silver layer in a Lakehouse-based structure is genuinely sturdy.<\/p>\n I\u2019d be doing you a disservice if I painted MLVs as a silver bullet. They’ve significant limitations, a few of which is able to matter to your structure:<\/p>\n No cross-lakehouse lineage and execution\u00a0\u2013 all sources, MLVs, and dependencies should reside in the identical lakehouse. In the event you\u2019re utilizing a Cloth Knowledge Warehouse desk as a supply, it’s a must to create a shortcut to it in your lakehouse first.<\/p>\n No DML statements\u00a0\u2013 you possibly can\u2019t\u00a0INSERT,\u00a0UPDATE, or\u00a0DELETE\u00a0into an MLV. The information is regardless of the SELECT produces.<\/p>\n No time-travel queries within the definition\u00a0\u2013\u00a0VERSION AS OF\u00a0and\u00a0TIMESTAMP AS OF\u00a0aren\u2019t allowed.<\/p>\n No UDFs within the SQL definition\u00a0\u2013 although PySpark authoring fills this hole with session-scoped UDFs.<\/p>\n No short-term views as sources\u00a0\u2013 the SELECT can reference bodily tables and different MLVs, however not temp views. This is applicable to PySpark too:\u00a0createOrReplaceTempView()\u00a0outputs aren\u2019t seen to the MLV engine.<\/p>\n Session-level Spark properties don\u2019t apply throughout scheduled refresh\u00a0\u2013 set them on the lakehouse or workspace stage as an alternative.<\/p>\n Schema title casing issues\u00a0\u2013 all-uppercase schema names aren\u2019t supported. Use blended case or lowercase.<\/p>\n Area availability\u00a0\u2013 on the time of writing, MLVs aren\u2019t accessible within the South Central US area.<\/p>\n None of those are showstoppers for many pipelines. However they\u2019re value figuring out earlier than you commit an structure to MLVs and uncover the limitation midway via.<\/p>\n In the event you\u2019ve been constructing medallion design patterns in Cloth utilizing notebooks and pipelines, MLVs are value a severe look. They collapse 5 surfaces into one declarative layer. The dependency administration is computerized. The information high quality is in-built. The lineage is seen. And as of FabCon Atlanta, they\u2019re production-ready.<\/p>\n The roadmap from Microsoft is obvious: optimum refresh for PySpark-authored MLVs is coming, extra SQL operators will change into incremental-refresh-eligible, and deeper integration with different Cloth workloads is on the best way. It is a milestone, not the end line \u2013 and I\u2019m curious to see how MLVs evolve over the subsequent few quarters, particularly round PySpark incremental refresh and any cross-lakehouse story Microsoft would possibly inform.<\/p>\n Two takeaways I\u2019d maintain onto:<\/p>\n The \u201cT\u201d in your ELT simply acquired rather a lot simpler to jot down, schedule, and belief \u2013 if that \u201cT\u201d is SQL.<\/p>\n MLVs don\u2019t change each pocket book, each pipeline, or each Warehouse. However for declarative transformations that want lineage, refresh, and knowledge high quality baked in, they\u2019re now a legit default in Microsoft Cloth.<\/p>\n Thanks for studying!<\/p>\n<\/div>\n![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/image-284-1024x484.png\")
The 4 phases of an MLV\u2019s life<\/h2>\n
Create: the syntax<\/h2>\n
Refresh: the mind of MLVs<\/h2>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/image-285-1024x670.png\")
What\u2019s new within the Basic Availability section?<\/h2>\n
1. Multi-schedule assist<\/h3>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/image-287-1024x540.png\")
2. Broader incremental refresh<\/h3>\n
3. PySpark authoring (preview)<\/h3>\n
4. In-place updates (Exchange)<\/h3>\n
5. Stronger knowledge high quality<\/h3>\n
The lineage view \u2013 Dependencies without cost<\/h2>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/image-288-1024x503.png\")
Knowledge high quality \u2013 Declared!<\/h2>\n
When MLVs shine and once they don\u2019t<\/h2>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/image-289-1024x559.png\")
The positive print \u2013 What to know earlier than you soar in<\/h2>\n
Wrapping up<\/h2>\n