Data migration is a complex undertaking, and the processes and software used are evolving and improving continually. Our pragmatic approach incorporates modern data migration best practice, with an emphasis on making the data manager’s life a little easier.
The realities of data migration
First of all, a quick definition of what we mean by data migration. The term usually refers to the movement of data from an old or legacy system to a new system. Importantly, the old system will be turned off. Data migration is typically part of a larger programme, where a business need drives the implementation of a new system for the business’s users. A data migration project can be triggered by a merger or acquisition, a business decision to standardise systems, or modernisation of an organisation’s systems.
In the typical business view of data migration, the data is picked up from the old system and deposited into the new system. The process usually happens at the end of the programme, just before the old system turns off, and so is often started late into the programme. In reality, before it is moved, the data needs to be read, understand and translated, while obsolete elements need to be discarded.
Create a methodology
An efficient methodology is an important part of data migration best practice. According to a report by Bloor, 38% of data migration projects run over time or budget. The report identifies an effective methodology as one of the ways to minimise these risks. However, industry-standard data migration methodologies are scarce. One option is the Practical Data Migration methodology developed by industry expert Johny Morris, which consists of training and certification. Alternatively, most companies who provide data migration services have their own methodology; ours consists of pre-migration scoping, project assessments and a core migration process.
The complexity of data migration means that a chosen methodology can seem like a sea of steps, which can be difficult to get all the stakeholders to buy into. Focus on the most startling element of the migration – the fact that the legacy system will be turned off – and the attention of the stakeholders is guaranteed.
Develop a system retirement plan
If you’re going to turn a large system off, you need a plan. Best practice requires careful planning for all these elements:
- Recovering software licences, so that the organisation isn’t paying for software that is no longer used
- Machine disposal
- Cancellation of support contracts
- Redeployment and retraining of staff
- Closure of any buildings that are no longer needed, for instance those that housed specialist hardware or staff
- Agreement and awareness from all users that the system will be decommissioned.
Understand the legacy landscape
The data migration team needs to develop a thorough understanding of the system that’s to be turned off. The first step is to model the legacy system in its entirety. Even an organisation with an apparently simple legacy system may be hiding, for instance, departments running data in Excel or in an Access database. There may be satellite locations with their own separate systems. As you start to talk to stakeholders across the organisation, you’ll discover more connections which need to be brought into the new system. You’ll also discover parts of the legacy system which are no longer used – often there is no need to migrate these to the new system.
Build confidence with incremental migration
Data migration best practice helps to generate confidence across the organisation. To create the best chance of the organisation running smoothly during and after the migration, consider these steps:
- Create a rollback plan in case of major issues
- Carry out dummy runs of the migration to identify problems before they occur
- Consider an incremental approach to the migration.
Project scoping and dry runs sometimes indicate that a big bang approach to the migration will not run smoothly. This is where changing to an incremental approach can be the best course of action. With an incremental approach, the old and new systems run in parallel, with a data highway keeping both up-to-date. The migration occurs piece by piece, and the legacy system is turned off once all data and processes have been migrated.
Testing, testing
Testing will always be required, so gain users’ buy-in by involving them early on in the project. Ask if the providers of the new system have tools that will help give users confidence, such as repeatable tests of their system. Tests should ideally be configured so that expert users are not required to input too much detail.
Communicate widely
Data migration usually involves a host of different teams, often competing: users; new system providers; old system experts; migration experts and project managers. Encourage teams to communicate as they find issues, and create a central repository for data quality and system issues so that they are recorded somewhere that everyone can see. This improves visibility and avoids a silo mentality.
You need people to agree to turn the old system off, so identify these people and bring them with you. Decide what documents they will sign off and ensure that these are neither too large nor hard to understand.
A full set of tools
Using tools is a recognised best practice. It is far more efficient than trying to carry out tasks such as migration scripts by hand. Tools can be used throughout the migration, for instance for data profiling, discovery and quality, for core migration and for testing. But your team will require skills in using the toolsets.
In summary, start the data migration project early in the programme and review all the options available to you. Incorporate tools and a methodology into your planning to minimise risk. The project is primarily concerned with continuity, so adopting best practices ensures that the new system operates as smoothly as possible.
Find out more about data migration planning in our article on essential planning components.
Further reading
