Big data platform startup dbt Labs Inc. said today it’s reinvigorating its flagship dbt Cloud offering with new enhancements and capabilities that it says will enable every employee to participate in data transformation.

New innovations include dbt Explore, Cloud CLI and a revamped dbt Semantic Layer, plus the launch of the new dbt Mesh paradigm for companies looking to adopt a “data mesh architecture,” the company said.

Based in Philadelphia, dbt Labs is the creator of the dbt data transformation tool. Data transformation refers to the process of altering data so it’s easier to process and analyze. It might include something such as consolidating multiple spreadsheets into a single file to simplify analysis, filtering inaccuracies from a dataset, or changing the way information is formatted.

Dbt Lab’s tool can be deployed in various data warehouse platforms, including Snowflake, Databricks and Google BigQuery. Users can write data transformation workflows using the Structured Query Language syntax, which many developers are familiar with. The startup has attracted a lot of funding, with its most recent $222 million raise in February 2022 bringing its value to $4.2 billion.

A new data paradigm

At dbt Labs’ Coalesce 2023 event today, the company announced it’s focused on eliminating the silos and bottlenecks and low data quality that often arises when organizations are reliant on a single, centralized data platform such as a data warehouse or data lake. It’s doing so with the creation of a “data mesh,” which is an architectural framework that solves advanced data security challenges through distributed, decentralized ownership.

With dbt Mesh, the startup said, it’s creating a new paradigm where different teams can build and maintain their own data products without affecting governance or creating silos. This data mesh architecture will make it quick and easy for data products to build on each other, the company said. Instead of business logic being centralized, teams will be able to make their own decisions.

Data warehouses and data lakes have proven to be too inflexible, constraining and slow-moving, Constellation Research Inc. analyst Doug Henschen told SiliconANGLE. He explained that a data mesh can free the far-flung teams within large organizations to manage and maintain their own data as they see fit, while also providing more consistent standards and centralized governance and control over data.

“With dbt Mesh, dbt Labs is supporting this approach by giving central data teams the ability to make platform-level decisions and set global standards for governance,” Henschen said. “So, for example, they can define interfaces for all dbt contributors incorporating model access levels, model contracts and model versions. They can also set up dependencies across dbt projects rather than requiring a single, monolithic dbt project for the entire organization.”

Rob Strechay, managing analyst at theCUBE Research, said dbt Labs is trying to transform itself from the default translation layer for SQL to the API layer for data intelligence. He said dbt Mesh will enable it to bridge the untransformed data with that which has already been transformed, and thereby build an ecosystem that can become a major player in next-generation data platforms and modern data stacks. “Depending on how it is priced, dbt Mesh, with other data warehouses and data lakes, will enable organizations to keep data where it lands, transform it and help build data products across data platforms,” the analyst explained.

Revamped semantics

The latest version of dbt Semantic Layer incorporates functionality that was gained through the company’s acquisition of Transform Inc. in February 2023. With the update, it becomes possible for companies to centrally define business metrics in dbt and query them from integrated analytics tools including Tableau, Google Sheets, Hex and Mode. As a result, organizations can ensure that critical definitions such as “revenue,” “customer count” and “churn rate” are universally consistent across every data project, for every downstream application and user.

Dbt Labs founder and Chief Executive Tristan Handy said that’s an important development for the dbt Sematic Layer. “First, our customers now have access to the fully integrated best-in-class technology that we acquired in the Transform acquisition,” he explained. “Second, our customers can use the dbt Semantic Layer with two of the most widely used data analysis tools, Tableau and Google Sheets, with additional integrations available right now and more integrations coming soon.”

Henschen said dbt’s semantic layer is being enhanced by the capabilities of Transform, weaving business metric definition into its data modeling environment. “The improved semantic layer supports powerful new capabilities for organizations, including the ability to create new metrics and dynamic joins, to optimize query planning and support complex metric types,” he explained.

In practical terms, Strechay said, dbt Semantic Layer’s new support for business intelligence will act as the glue to create complex, query-modeled definitions of company’s most complex data representations. “One could envision this being used with the dbt Mesh and Explorer to provide visibility into a company’s definition of a customer or a customer’s journey through a sales and support cycle to renewal,” he explained.

Feature upgrades

The dbt Explorer is a brand new documentation and lineage visualization tool that supports the dbt Mesh paradigm and makes it easy for anyone within an organization to discover, understand and reuse dbt assets from across different teams. In this way, dbt Explorer helps to reduce friction within data development workflows, the startup said.

As for the Cloud CLI, or command line interface, it’s a new addition to dbt Cloud aimed at developers. Previously, developers could only build applications based on dtb data using an integrated development environment. With the Cloud CLI, experienced developers now have the ability to contribute via any terminal or the IDE software of their choice, the company said.

Finally, dbt Labs is expanding its ecosystem of cloud data platforms with upcoming adapters for Microsoft Azure Synapse and Microsoft Fabric. With their launch, Azure Synapse and Fabric users will be able to take full advantage of dbt Cloud’s data transformation capabilities. The support for Fabric is currently in private preview, and support for Synapse is set to be added before the end of the year.

According to dbt Labs, the combination of today’s announcements will provide companies with a centralized data transformation platform that everyone can contribute to, helping to increase the velocity, quality and consistency of their data.

Handy explained that the original purpose of dbt Cloud was to help data analysts and data engineers productionize their dbt deployments. “With today’s announcements, dbt Cloud customers can create a mesh of interconnected, domain-owned, dbt codebases,” he promised. “The developments we’ve made this year are central to enabling collaboration across multiple projects, a requirement for managing dbt at scale.”

Photo: Rob Strechay/SiliconANGLE