By Jane Steen, Head of Engagement at ETL.
Often, data integration technology decisions are framed as build versus buy. But the useful question isn’t which approach is better. Instead, it’s about which parts of your problem each approach suits best.
When off-the-shelf platforms are a good idea
We create bespoke data integration systems, and even we have to agree that off-the-shelf technology is a good option. If you’re moving data between well-known systems with documented APIs, in formats those systems already agree on, at volumes that are not unusual, a platform will do it faster and more cheaply than anything written from scratch. You get monitoring, retries, alerting and a support contract, without paying to have them built. For many organisations, that covers a decent share of the IT estate. The connectors in particular can be very useful, although it’s worth being clear-eyed about how much adjustment they might need.
Where off-the-shelf tends to fall short
In our experience, a few patterns come up again and again.
Systems that were never meant to talk to each other (or to anyone at all). A legacy platform, a heavily customised ERP, something built in-house fifteen years ago by someone who has since left. There was no demand for a connector, so nobody built one.
Fields that agree on their names but not their meaning. Two systems both have a customer. In one, it’s the legal entity that gets invoiced; in the other it’s a physical site. One customer in the first system can be a dozen in the second, and no mapping tool will tell you which one is most important to the business. Someone has to decide what the rule is, and then that rule has to live somewhere maintainable.
Inaccurate data that has to be made usable. Off-the-shelf tooling tends to assume that the source data is broadly correct. When it’s not, the cleansing logic has to go somewhere, and that somewhere is often a growing pile of scripts that nobody owns.
Many-to-many, across markets. One manufacturer, several markets, a different system in each and reporting that has to reconcile all of them. Each additional variant multiplies the mapping.
None of these issues make an off-the-shelf platform the wrong choice. But they mean that the platform can’t easily do everything that’s required, and the remainder is the part that’s in danger of overrunning.
The majority problem
Imagine a hypothetical tool that covers 80% of the requirement convincingly. The final 20% of the requirement – the exceptions, the system that’s hardest to connect to, the reconciliation logic, the market that does things differently – usually takes considerably more than 20% of the effort. It gets solved with workarounds, a staging database here, a nightly script there, a spreadsheet that somebody maintains. Each workaround makes sense on its own. Together, they become a second integration layer that’s not documented, not monitored and doesn’t appear on anyone’s architecture diagram.
The upshot is that the platform didn’t remove the hard work. Instead, it moved it onto your team.
Questions to ask before you commit
Whichever route you take, these questions tend to get to the real answer.
Which of our systems does this support out of the box, by name? Not “supports ERP integration”, but which ERP, which version, which modules? The gap between the marketing page and your actual IT estate is the gap that the project is going to have to include.
Where will the business logic sit? If your mapping rules are genuinely complex, the question is how to make sure that they end up somewhere documented and maintainable.
What happens when a source system changes? Vendors upgrade, whether or not it suits you. Who notices, who fixes it and how quickly? This is the difference between an ongoing, planned cost and an ongoing problem.
Could we take the mappings with us? If you moved platforms in four years, can all your work come too, or do you have to rebuild it?
What’s the five-year cost, including our own time? This includes licence escalation, internal effort and specialist skills that you would need to hire or retain. The off-the-shelf option usually still wins this on the ordinary flows. It’s the awkward integrations where the sums change.
Who carries the risk if it doesn’t work? This is something to be direct about. If the answer is your own organisation, that’s a decision that’s best made deliberately.
The answer from our perspective
We’re not an off-the-shelf provider, so most estates that we work with end up with a purpose-built system that’s highly specific to the needs of that organisation. Usually, that’s the way that the initial problem was always going to be resolved.
It’s worth saying that bespoke doesn’t mean that we write everything from scratch. Usually it means deliberate, documented, owned work doing the job that no product was ever going to do, sitting alongside tooling that handles the rest. There’s little risk in our custom code. The risk is in undocumented custom code that appeared by accident because nobody planned for the last 20%.
A good place to start
Before commissioning anything, list your sources and be honest about which are standard and which are odd. The odd ones decide the approach. If most of the estate consists of common systems in common formats, buy an off-the-shelf option and don’t over-engineer it. If a meaningful share of your estate is legacy, customised or inconsistent, there will be bespoke work required somewhere. It’s far better to scope it deliberately at the start of the project, than to discover it in month seven.
We discuss this in early design conversations. If you need some help with your own situation, feel free to ask us a question.
