We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 - 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!
Dataops is the set of processes and technologies that aims to promote a “culture of continuous improvement” in the area of data analytics. First proposed in 2014 by Lenny Liebmann in a piece for IBM’s Big Data & Analytics Hub, dataops has matured from a collection of practices into an entire approach to data analytics, encompassing not only data preparation and reporting but all related information technology operations. According to a January 2020 report from 451 Research, 91% of companies already had — or were in the process of defining — a formal dataops strategy, while 86% planned to increase spending or development germane to dataops in the next 12 months.
Hypothetically, dataops can provide the tools that an organization needs to deal with an increasing amount of data, for example streamlining database maintenance through automation. But blockers stand in the way of fully realizing the promises of dataops. In a separate survey published in August 2021, 451 Research found that 90% of organizations don’t have an “optimized” dataops strategy and that few believe that they’ve achieved dataops maturity.
Vendors like Atlan claim to simplify dataops by offering managed solutions that abstract away many of the complexities involved in deployment. For instance, Atlan — which recently raised $50 million in series B funding led by Insight Partners, Salesforce Ventures, and Sequoia Capital India at a $450 million post-money valuation — performs automatic profiling of a company’s data to identify outliers, missing values, and anomalies. It also correlates business terms with data objects to generate a common understanding of the data and how to use it, revealing how data has evolved through its lifecycle to predict how it will change going forward.
Data analytics tools
Atlan started out as an internal initiative at “data for good” firm SocialCops and was incubated across over 200 data projects, including India’s National Data and Analytics Platform and the United Nations SDGs National Data Platforms. By acting as a hub for assets ranging from tables and dashboards to models and code, the goal is to enable teams to create a source of truth while collaborating via integrations with data warehouses, chat apps like Slack, and business and data science tools.
Prukalpa Sankar and Varun Banka founded Singapore-based Atlan in 2018, after launching SocialCorps in 2012. Prior to teaming up with Sankar, Banka held a software engineering role at Microsoft and served on Barclays’ operations and cross-product technology team.
“Today, data assets are not just tables, but code, models, business intelligence dashboards, and pipelines,” Sankar said in a previous statement. “At Atlan, we are reimagining the human experience with data — why can’t data assets be shared as easily as sharing a link on Google Docs, or if Google Analytics can tell you usage on a website, why can’t we do the same for our data?”
Atlan can be configured to send alerts to stakeholders in the event of a data problem. In-line chats and annotations ostensibly help users stay on the same page, as do Excel-type queries like filters, aggregations, and grouping of data from data lakes and warehouses. (A data lake is a centralized repository for data stored in its raw format, while a data warehouse collects data from a range of sources to provide business insights.) Atlan also offers a “bot ecosystem” with bots that, for example, read through database contents to detect personally identifiable (PPI) information and recommend descriptions for data in databases on the basis of past data.
Challenges in dataops
Hurdles in this space can be challenging to surmount for many organizations, as revealed in a 2021 survey commissioned by Data.World and DataKitchen. In the survey, the vendors — which, it should be noted, have ulterior motives in giving the impression that dataops is difficult to adopt — found that only 46% of companies considered their dataops efforts to be both mature and successful. Respondents said that data governance policies and data requests with unreasonable expectations made their day-to-day jobs “very difficult.”
Sankar asserts that Atlan can help lighten the burden on engineers — a sales pitch that’s evidently resonated with customers and investors. Teams at large enterprises like Unilever, Scripps Health, and Postman use Atlan. And to date, Atlan has raised $69 million in venture capital. Salesforce Ventures and Sequoia Capital India participated in Atlan’s series B. The company previously landed $16.5 million in a series A financing tranche in May 2021.
VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn more about membership.