Discovery is half of the project: Our survey results

Share this on your networks

By Jane Steen, Head of Engagement at ETL.

Every data project has a phase at the beginning where the build hasn’t started yet. Instead, it’s about discovery: opening tables, reading schemas, asking which of three customer records is the real one, following a tangle of systems and discovering that a field called “status” means different things depending on the system.

We wanted to know how much of the project that phase actually consumes. So we asked 102 data practitioners at Data Decoded in Manchester, anonymously, about their last data project.

The answer is more than half the project

How much of the project timeline went on simply finding out what the data and systems really looked like?

Hardly any: 13% of data practitioners. Around a quarter of the project: 32% of data practitioners. About half of the project: 32% of data practitioners. More than half the project: 23%.

So 55% – 56 of the 102 respondents – spent half the project or more on discovery. Nearly a quarter spent more than half the time. Only one in eight spent hardly any time on it.

Read that as a budget line and it’s remarkable. On many of these projects, the single largest activity was establishing the state of something that the organisation already owned.

The people doing the work think you don’t know

At the end of the survey we asked each data practitioner, optionally, one thing that their CTO or IT Director doesn’t understand about their projects. Most of them answered with a version of the same thing:

“Lack of understanding of the data quality.”

“How ungoverned the different data stores were.”

“No such thing as one version of the truth.”

Another simply wrote: “Garbage in, garbage out.”

Those are responses to a question about leadership, not about discovery, but they point to the same thing. The people working on the project tend to believe that the people approving the plan don’t know what state the data’s in.

What this does and doesn’t prove

You might expect the projects with the longest discovery to be the ones that overran. In this data they weren’t, particularly: half of the heavy-discovery group went over budget, but apart from that there isn’t a link to report. A larger sample size would be required.

Based on this sample, it looks like discovery isn’t a risk that sometimes blows the budget. It’s a cost that can be expected to be there, whether or not anything goes wrong.

The awkward consequence

It looks as though a big chunk of work consists of exploring the state of what’s already there. It means that, potentially, the true timeline and costs are initially unknown quantities. That can be uncomfortable if the project’s budget is requested before the discovery phase has happened. No one particularly wants to go to a board with “we’d like some money to find out how much money we need”. However, the alternative is having that conversation later, possibly in a rush, when discovery has happened anyway and consumed the contingency budget.

What to take from it

In scoping conversations, one question that really helps is: when did someone last actually look thoroughly at this data? The answers aren’t always reassuring. Often the real answer is that nobody has looked at the data in depth since the system was built, and the person who built it has left.

That isn’t a reason avoid starting a project. It’s something to find out before estimating the timeline. It’s about treating discovery as a funded piece of work with its own output, rather than something that the delivery team tries to absorb in the first weeks.

About the survey

102 data practitioners completed a six-question anonymous survey on our stand at Data Decoded, Manchester, on 13 and 14 October 2026. No email address or identifying detail was collected. Respondents were self-selected visitors to a single event, so this is not a representative sample of the UK market and I wouldn’t present it as one. The eleven comments were an optional free-text answer; spelling has been tidied, wording has not.