It is the best of data science times, it is the worst of data science times…

Data science is high in the hype cycle and considered the sexiest job of the 21st century. Data scientists are in demand, and the tools to do our jobs are getting better by the month. It’s a great time to be in data science!

However… managing a data science group often feels like an uphill battle. The problems are not the data science work itself, they are about doing data science within organizations. At first I thought it was me. But the more data science managers I talked to, the more I heard the same stories over and over again. My conclusion from is that there are three root causes to many of the barriers we face: marginality, mystery, and mis-alignment.

Marginality

Marginality is the perception (or reality) of others that data science is tangential to the core work of the organization.

A clinical diagnostics company may have a group dedicated to the complex processing required to turn raw instrument results into a meaningful scientific result. Even though they recognize that this is a necessary step in developing diagnostics, the core business of the company is manufacturing and selling diagnostic tests, not analyzing data.

Mystery

Mystery is the difficulty people throughout the organization have understanding what data science really is.

It’s not uncommon for data science groups to come into existence like this:

“We have data. We need to get more from our data. Let’s hire data scientists…”

“Hooray, now we have data science! But what do they actually do? And can I ask without seeming dumb?“

Misalignment

Misalignment is the poor placement of data science groups within the organizational structure.

This can manifest in at least three ways:

  • Data science being entirely in the wrong place – such as housed in IT even though all their stakeholders are elsewhere and they don’t share a mission with IT.
  • “Half-alignment” - in which the data science group reports up through a part of the company in which they do some work, but where the most valuable work they deliver is to other departments.
  • Data science groups being pushed down within the organization. Because the groups often start out small, and are sometimes intimidating for non-technical people to manage, they can be placed so far down in the org structure that they don’t have the eyeline or influence to be effective.

Barriers to Data Science Impact

These three issues can keep data science teams from having the hoped-for impact. They lead to difficulty getting access to data, adding headcount to teams, and being free to focus on the most impactful work.

As a maturing field, we need to take on the challenge of making data science work effectively within organizations.

Some of my thoughts about ways to do this by borrowing from Agile practices are in this post.