This article was originally written by Avanade alum Dael Williamson.

The role of data teams in organizations are changing and expanding. With data and artificial intelligence (AI) becoming critical areas of focus for every organization, the need for businesses to unlock value and insights faster has never been more in demand. Today, businesses can harness data and AI to predict behavioral patterns, understand operations, create new business revenue, improve their customer experience, and even simulate new drug candidates.

From a technology perspective, we are seeing the incremental evolution of data architectural patterns shift more rapidly. From a data warehouse (30+ years ago) to a data lake (10+ years ago) to a data hub (5+ years) to new constructs like the Lakehouse and the Data Mesh (in the last 18 months), the shifts are happening faster, and that means we are going to need to enable our data teams and our businesses to get closer, more unified and adaptive to the future shifts that are certainly coming.

Learning from the past to make way for the future
The cloud platform engineering approach, driven mostly by cloud economics and the need for incremental (and early) value realization, has altered the way organizations engineer data platforms and software more broadly.

In the past, organizations would invest in a big server and add massive, monolithic software. They would then hire expensive, in-demand and uniquely skilled people to customize their centralized, monolithic data application. However, the returns were slow, the ongoing effort and annual true-up was expensive and the turnaround for insights often came too late to provide value.

Depending on the size of the organization, the same data processes can be implemented by other business units. Interestingly, after a decade or so, often the divergent and inconsistent approaches taken by different teams within the same organization remain and are mind blowing. Within the same platform, it’s easy to see where different teams’ standards were applied over time – not unlike a walk through a museum.

How cloud and agile are changing the approach of Data Teams
Cloud has a different form of economics. You are now able to switch your infrastructure and pay for storage and usage based on consumption of “rented” hardware, and increasingly, rented software – called native services. Engineering is now more akin to the assembly of different components and allows for building logic into the gaps. As you may be aware, the number of native services and components can be overwhelming for organizations since there is now an abundance of technology choices.

Infrastructure automation tooling allows teams to continuously test, integrate and deploy the software they build. The Agile Data teams approach allows teams to focus on use cases and features which accelerate time to value provided that there data is available. Insights can be gained in parallel to engineering whereby data scientists work with data engineers to ensure that the data pipelines are independently automated, reducing the 80% of time wasted to prepare data.

Simultaneously, data modelers are shaping, organizing and aggregating data using clever, emerging patterns like product thinking and domain-driven design. Data visualization experts can also rapidly shape data into dashboards and reports for business consumption.

Unifying the experience of the team
Changing how people work together, analyze, and treat data should never be underestimated. If you truly want to become a data and AI-powered organization, it’s not only about the data, but also about how your team utilizes tools to prepare data and insights for your organization‘s consumers.

Ensuring that data teams have a consistent experience is key. In the past few years, we have seen a movement to unify the experience of the main members: the data engineer, the data modeler, and the data scientist. All these personas should be able to use the same interface to work on their part of the data supply chain. However, while this addresses the technical data team, there is still a chasm that exists between business and informational technology (IT).

Cross-functional teams are not a new concept and nowadays I’ve heard many different terms to define the same thing. There are success stories where companies have transformed for the better by bridging the divide. However, in most companies, the major chasm still exists, and the divide has only been partly bridged across the functional technology teams. I still witness data teams missing representation from the business domain context, roles like business data analysts for a start.

Further unifying the user experience
To address the divided between data teams, we are excited to see the inclusion of the SQL Analytics experience being added to the Databricks capability. The experience of this new capability is one that is familiar to business and data analyst users who will be able to take their place in a truly cross-functional data team. Unifying business and IT objectives is key to value realization and having teams working together is a significant part of the story. This allows teams to collaborate in a similar interface, which connects to the same environment. No longer do data teams need to copy data into another space (thereby potentially compromising data integrity) or experience complex, slow integrations with tools that don’t perform.

The journey doesn’t stop there
The magic under the hood that allows the SQL Analytics experience to be possible is driven by another innovation by Databricks, which has led to the Lakehouse movement. We have been watching this movement over the past year and it has been another incredible evolution to the Data Lake construct. Applying the Databricks Delta Lake capability not only reduces the effort and logic required to organize data in your Data Lake, it also reduces the effort to detect issues with data quality and simplifies the ability to query data.

While the approach brings teams closer together, this does not negate the need for Enterprise Analytics tools like the Microsoft Power Platform. The new SQL Analytics capability is not just an interface to further unify your data teams, it also provides the integration capability to connect enterprise tools like PowerBI directly to your Data Lake. This supports the pre-existing security features and policies applied to the underlying Lake storage, while also reducing the latency that was previously experienced when performing complex analytics reporting and visualization.

A significant step forward for unifying data teams
We see the launch of Databricks SQL Analytics being a significant step towards unifying data teams by bringing more business-centric skills into these teams and (with some luck) even business users themselves. The significant advances under the hood through the Lakehouse and Delta Lake, and the necessary integrations to connect Enterprise-level analytics tools like the Microsoft Power Platform, open up a world of possibilities, which were previously painful, slow and challenging to accomplish.

We are proud to be a Databricks partner and look forward to exploring their innovative approach to unifying data experiences further.