This article was originally written by Avanade alum Dael Williamson.

Data science is rapidly becoming an area of focus for every business and it’s a passion of mine. Data scientists can help an organization in numerous ways. Businesses can harness data and artificial intelligence (AI), based on data science, to predict behavior patterns, understand operations, create new business revenue, improve the customer experience and even simulating new candidate drugs.

As someone coming from an engineering background who did a fair bit of data science in academia, I look forward to scintillating conversations about all things technical, and if there’s one thing I’ve learned, there’s a right way and a wrong way to engage data scientists in the journey to a digital transformation.

A very important consideration is to understand how a data scientist works. This helps you fully understand how to put the right tools in place so as not to disrupt but to efficiently focus their attention on where there’s the most potential to drive business value. 

"A data scientist is someone who knows how to extract meaning from and
interpret data, which requires both tools and methods from statistics and
machine learning, as well as being human."

As you can imagine, the total process can take weeks or months. Traditional spreadsheet-driven computations or manual data prep combined with languages like Python and R are unable to keep up with the demand. Working across an abundance of tools that don’t integrate efficiently — as well as focusing too much on data collection and not enough on action — often makes data science too cumbersome, and a lot of time is wasted. 

Putting data science teams in the driver seat
Changing how people work together, analyze and treat data should never be underestimated. If you truly want to become a data and AI-powered organization, it’s not only about the data, it’s also about how the consumers of your data (in this instance, your data science teams) work today with the tools they need to prepare for tomorrow.  

For one client in the travel insurance industry where we were creating a data-driven pricing engine, we started their journey by creating a day in the life of a pricing analyst from the business with technical knowledge (mostly R and SQL). We followed the users through the day-to-day basic steps and phases of data analytics projects, from raw data preparation to building both their statistical pricing models and machine learning models. 

We then conducted and designed working sessions, not demos, based on business-driven requirements and user-identified needs, expectations and frustrations. Working with Azure Databricks, a unified analytics platform, we put the pricing team users in the driver seat, helping them validate the data, figure out what they needed quickly, and design models more efficiently with the added benefit of helping the data engineers to organize the data in a more consumer-led manner. 

The pricing users could collaborate within a shared notebook and could see for themselves faster processing speeds and scalable data extraction, transformation and loading (ETL) on the cloud with Azure Data Lake and Databricks Delta Lake. This helped users see gains in performance and gave them great ability to analyze big data analytics workloads, all the while the data platform was being built by the Data Engineers as they worked.

We were also able to gather usage data to benchmark usage patterns and therefore forecast consumption costs and infrastructure setup more accurately long before launch of the new data-driven pricing engine.

Fully engaged in the potential for innovation
The technical business teams we worked with quickly became vocal advocates. They could model and test a protype in hours, compared with weeks or months, and reduced data processing time from about an hour to 10 minutes. 

Many users reported increased gains in productivity. Regardless of skill set, users experienced the power of shared collaboration and limitless potential for innovation.

The journey to an AI-driven organization
But creating an effective AI-driven organization is not as simple as hiring a few data scientists and data engineers. No one person can be an expert in mathematics, statistics, computer science, communications, machine learning and data visualization.

Developing a data strategy is about deploying an ecosystem where getting to the right data, metrics and resources is easy. It’s about understanding how people work today and then helping them use technology to work smarter, faster and better. You also need to foster a culture that shares information, enabling data science teams to quickly solve problems and build new innovative business models in response to a rapidly changing world.  

Find out more about how to power your data science teams with data and advanced analytics and help your business thrive in a changing world.