By Jane Steen, Head of Engagement at ETL.
Many data integration project estimates are wrong in a particular and slightly unfair way. It isn’t that someone was careless with the arithmetic, or was too optimistic about how fast the project will go. The issue is that the estimate describes the data as documented, while the actual project meets the data as it’s used in the real world. Those are two different sets of things, and only one of them has ever been written down.
So the overrun, when it comes, doesn’t always look like an overrun. Instead, it often looks like a series of reasonable discoveries.
The extra costs of a data integration
- The first is often a field that hasn’t actually meant what it’s called since about 2011. Someone needed to record something the system had no room for, so they used the notes field, or the second address line, or a status code that had gone spare. It worked, and it’s been working ever since. Nobody wrote it down because it wasn’t an actual decision. And now it’s hard to map, because mapping it means first deciding what it is, and that wasn’t included in the technical brief.
- Then there’s the time that you didn’t budget for because it isn’t actually yours. You’ve costed your own team carefully. But you haven’t costed the two days a week you need from the person who owns the source system, who has a full-time job already and no line in your costings. This is one constraint that ends up governing the schedule, and it often isn’t included in the plan as a cost or a risk.
- The legacy vendor is a related surprise. Getting data out of a system you’re leaving is work that someone has to do, and the someone is often a supplier with no commercial interest in you leaving. The extract is available, but it’s chargeable, and the quote often arrives at a point in the project where declining it isn’t an option. And don’t forget that actually getting that data from a disinterested vendor can be very slow indeed.
- The one that catches people the most, though, is the gap between moving data and proving you’ve moved it. The migration is a technical job with a clear end. However, reconciliation is a different job entirely, and the people who have to sign it off aren’t the people who did the work. They will want to know why the record count differs by eleven, and the answer will sometimes take a fortnight to establish. On a lot of projects, this phase is longer than the migration and it appears in the plan as a single line called “validation”, without a budget.
Underneath all of this is the thing nobody puts in a business case, which is that you’re not just moving data. You’re also adjudicating fifteen years of fudges and exceptions. Every one of them made sense to somebody at the time. But resolving them needs decisions from business people who have no time to make them and no particular reason to prioritise your deadline.
None of this is a disaster, it’s just how integration projects often work. But it does affect you differently, depending on where you sit. If you’re running the project, the cost is the week you lose while waiting for feedback or help. If you signed off the number, the cost is the reconciliation of budgets later. Neither of you was wrong. The estimate was simply made before anyone had looked at the data properly, which is when estimates are usually made.
What to do about it
What helps is fairly boring. Do the discovery early, on a real extract rather than a schema, when finding problems is still cheap. (We often insist on a landscaping phase before embarking on the actual integration project.) Put other people’s time in the plan as an item with a name attached to it. Give reconciliation its own phase and its own duration, rather than letting it hide inside the last week. And keep a list of the exceptions you find, because it tends to become the most useful document the project produces and nobody ever sets out to write it.
We’ve spent a long time straightening out these kinds of projects, and the pattern varies less than you’d think. If you’re part way through your own project and recognising some of this, feel free to book time with us here.
