By Jane Steen, Head of Engagement at ETL.
Most data integration problems don’t begin as integration problems; they begin as growth. One market, then another. A new project. A handful of new sites, then a few hundred, with each one running the system that suited them at the time. By the time anyone steps back to look at the whole network, the data sits in a dozen different systems, and only some of it lines up.
A straightforward approach would be to tackle this as a standardisation project. If everyone used the same system, the thinking goes, the data would look after itself. But in many networks, from car dealers to parts networks, that option rarely exists. Operators own their own choices and markets have their own established systems. Asking third party organisations to tear out something that works isn’t often going to happen.
We saw this kind of problem with Snap-on Business Solutions, who needed to get service, repairs, counter sales and parts data out of different systems across their network. The project showed that a few things tend to make the difference in practice.
- Start by deciding what “the same” actually means. That’s partly a question of format – agreeing the canonical shape of a service record or a parts sale before you touch any system – but it’s also about sameness in the harder sense: two systems will often hold what is meant to be one event, recorded slightly differently, and you have to decide the rule that says those are the same thing and not two. Pin that down early and you stop reconciling the same disagreement every month.
- Keep the work off the endpoint while you’re at it – the more processing you ask a third party’s system to do, the more fragile the feed becomes, so a light extract with the shaping done centrally tends to hold up better.
- That also keeps the transformation logic in one place, which is what makes adding up a new system closer to a change than a rebuild.
- It’s worth bearing in mind, too, that often a failure isn’t wrong data but missing data: a feed that goes silent overnight and isn’t spotted. An automated check that every expected extract has actually arrived earns its keep very well.
- None of this needs proving at scale on day one. Two systems in one market is enough to see whether the process holds, and if it does, the rest is mostly repetition.
So if you’re looking at a network held together by different systems in different places, the useful question isn’t how to make them all the same. It’s whether enough of the methodology questions have been made up-front, so that scale becomes a matter of repeating that methodology, carefully. This comes up a lot when a network has grown faster than its data. The full Snap-on write-up is here.
