Microsoft announces Mirroring, a way to copy databases

Microsoft today announced “Mirroring,” a way for enterprise customers of its Microsoft Fabric product to replicate external databases they use, and to manage them from within Microsoft’s data warehousing experience.

The capability is arguably the most significant in a series of updates Microsoft announced today to Microsoft Fabric, because it allows for easier sharing of data from databases even with proprietary data formats. This in turn allows customers to more easily consolidate their data within Microsoft Fabric, and save costs.

The move continues to place pressure on Microsoft’s main cloud competitor, Amazon, which has not moved as quickly to embrace open formats – something that analysts praised Microsoft for when we covered Fabric’s announcement May. It also pressures Google, another cloud competitor: While Google is also working hard to create open formats (see our exclusive OneTable news today), Microsoft’s bigger customer base and more robust range of services, gives it an upper hand. But with Mirroring, Microsoft is also taking the gloves off against upcomer analytics competitors Databricks and Snowflake, which have gained momentum recently by trying to make data sharing easier. 

It’s another reinforcement of Microsoft Fabric’s ambitious effort to help companies recognize, access, and analyze data uniformly wherever that data lives. On the other hand, the openness can be seen as becoming table stakes for cloud data providers, most of which are providing various off-ramps to replicate or exit data from established on-premise databases, said Tony Baer, an analyst at dbInsight. “To that extent, the Mirroring announcement is an example of keeping up with the Joneses,” he said.

Other updates to Fabric today include the following:

• Microsoft Fabric is moving to general availability; 
• Fabric already has 25,000 customers since being announced in May in preview model
• Fabric’s Copilot feature, which lets people use natural language to create dataflows and pipelines, write SQL statements, build reports, or develop machine learning models, is now in public preview. 
• The expansion of Microsoft’s ISV ecosystem with Microsoft Fabric.

Mirroring is the latest move by Microsoft to create a unified, open data environment

Mirroring is a completely new feature, and helps companies avoid having to make costly SQL calls to databases outside of Microsoft’s warehousing tool, Fabric’s Synapse Data Warehouse. It also helps customers avoid complex migration projects, or what Microsoft calls the integration tax. This can happen when they add new databases through things like acquisitions.

A key design element within Fabric is OneLake, a single data repository where Microsoft customers can keep all of their data in the open file format Apache Parquet, and open data table format Delta.

Mirroring essentially allows customers to replicate existing cloud data warehouses and databases into the open source Parquet and Delta formats, for direct use within Fabric’s warehouse experience – even when those external databases have proprietary data formats. Microsoft said that the replication technology will start working today for customers of Azure Cosmos DB, Azure SQL DB, Snowflake, and Mongo DB. More data sources will be added 2024, the company said.

“The reason we’re doing this is we really believe that all data needs to be open so that you can get business value,” said Arun Ulagaratchagan, Microsoft’s corporate VP, Azure Data, in an interview with VentureBeat. 

The Fabric updates push forward Microsoft’s vision of an open “Cloud 2.0”

Mirroring is an extension of the “Shortcuts” feature that Microsoft first announced in May, as a way to support multicloud scenarios. Using shortcuts, Fabric’s OneLake can virtualize data storage in Amazon’s S3 and Google’s storage, without having to move or replicate the data. As of today, that feature is now generally available. But shortcuts works only on databases in open source data formats like CSV and Parquet.

The London Stock Exchange (LSE), which has one of the largest data repositories in the  financial industry, is working closely with Microsoft to test the Fabric offering, including the mirroring technology. Dmitri Sedov, Global Group Head, Data Intelligence, said in an interview with VentureBeat that use of the Fabric is still early and that the proof of will be in the pudding, but that the early indication is that Microsoft will realize its vision of a “Cloud 2.0” in Fabric.

Fabric shifts the focus away from data storage, which is becoming cheaper and more commoditized, Sedov said, and allows customers to focus on what they do with their data. And with mirroring and shortcuts, Microsoft’s Fabric becomes more attractive to customers because it lets them analyze and access data wherever it resides.

“They’ve seen quite a lot of attempts by competitors to come in and eat their lunch, like Databricks and Snowflake…who came and said ‘Well, hold on, we’re going to do data sharing easier’, said Sedov. However, Microsoft’s latest moves have upped the game against these players in a big way. “In Microsoft’s true nature, you know, they don’t move fast, but when they move, they really move.”

It’s early, but Microsoft says Fabric is growing like crazy

That even some of Microsoft’s biggest customers like LSE are still in early days experimenting with Fabric suggests it’s too early to know for sure just how the Fabric technology will play out across the industry. However, Microsoft’s Ulagaratchagan said 67% of the Fortune 500 are using Fabric, and 85% of the customers are using three or workloads in Fabric, and says this enough to say that Fabric “is growing like crazy.”

Ulagaratchagan also provided more details on how Mirroring works: Mirroring uses publicly documented APIs from database and data warehouse vendors to create and maintain a snapshot of the database in near real time, and taps into the change data capture feed of the databases to keep them in sync, he said.

Once the source database is attached, features like shortcuts, Direct Lake mode in Power BI, and our universal security model work instantly, he said. “It gives you massive performance acceleration and significant cost reduction, because there’s no SQL queries being run,” he said. So when a customer uses Microsoft’s Power BI, for example, that app simply goes to OneLake and pages the data into memory, he said. “That now applies to all of the mirrored databases.”

He said customers like the fact that generative AI capabilities that are powered by Azure Open AI are built into Fabric, that the data architecture is simplified to not have to ingest data so many times, and that Fabric’s unified business model means that customers only have to buy one product, Fabric, and have it power all of their data workloads.

A race against Snowflake and Databricks to be the enterprise data provider of choice

Microsoft is eager to get as much data into Fabric and its supporting cloud platform, Microsoft Azure, as possible, said Amalgam Insights analyst Hyoun Park. This includes pulling data from Snowflake, which had been growing quickly, but which offers its own proprietary data environment. Microsoft would also like to exploit the perception from some customers that Snowflake’ costs are “out of control,” Park said.

Microsoft Fabric’s approach to Databricks is interesting as well, Park said. Microsoft partners with Databricks, but letting Azure customers work closely with Databricks’ lake house via product called Azure Databricks. Databricks has a multi-billion dollar run rate and the the strategic importance of being a data lake built to support AI, Park said. “Microsoft Fabric is both being shaped to work with the fastest growing emerging data pillars with an eye towards both growing Azure and monetizing in the Era of AI.”