Migrating data is a specialist activity that demands a detailed plan – especially if the project involves complex data. Our data migration service uses a clear process to mitigate risk and maximise the opportunity for project success. This process has been applied by our consultants to migrations of even the most complex data. Here are the six stages that we consider.
Stage 1: Project scoping
If the parameters of the project are unclear, or if you haven’t conducted a data migration before, you will benefit from a scoping exercise. Draw up a plan before the project starts that sets out critical areas of the project’s structure. Elements to include are:
- Stakeholders and their required deliverables
- Business domain knowledge, system expertise and migration expertise
- Communication plans and reporting requirements
- Budget and deadlines.
If using an external provider to run the project, be clear about your own dataset or you could end up changing requirements and incurring additional fees. Be honest about the scale of the project and have a clear understanding of its scope before agreeing costs with a supplier. This also aids budgeting of an internally-run project.
Our consultants also carry out a review of the migration itself to ensure that all its aspects are functionally correct. This includes:
- Ensuring the routes of communication are defined
- Ensuring storage and versioning of project artefacts is available
- Ensuring hardware is available and accessible
- Ensuring that our consultants have access to any required sites or buildings.
Stage 2: Resource evaluation
A clear methodology is essential if you want a staged, well-managed and robust approach to data migration. Our proven methodology includes thorough assessments of the project and a core migration process (see details of the process here). Consider incorporating standards into your project. Standards are used to identify problem areas early on, making sure that you don’t reach the final stages with a hundred different issues to sort out. For instance, at ETL Solutions we have the Prince2 management standard, and use ISO standards where appropriate to underpin our data migration methodology.
You will also need to evaluate the migration tools available to you. Apart from evaluation of its features, the critical questions to consider about the data migration software used for the project include: how flexible is your preferred migration tool? Is there a fit with the skills of the people working on the project? If using an external company to manage the project, confirm whether the software is included in the cost, or whether there will be an additional fee.
Initial assessment of staff competency and training requirements can reduce reliance on external experts and boost the confidence of the project team. Will the people carrying out the project be there for the duration? Are they skilled and knowledgeable in the toolsets and methodology they’ll be using? For individuals, this process provides clarity about their role within the data migration.
Stage 3: Migration design
This stage plans the extraction, verification and transformation of the data. These core steps are included in our bespoke data migration methodology to enable an uninterrupted flow of data during the migration. (See more details of the methodology here.)
The migration itself is dependent on key artefacts being available at this point in the migration project. The migration design should include:
- How the data is extracted, held and verified
- Mapping rules
- How data is loaded into the new system
- Recovery plans for each stage of the migration
- A schedule of the actions required to go live.
Stage 4: Testing design
The testing design stage defines an overall test plan for all stages of the migration. An initial overview should assess the tools, reporting, structures and constraints involved with testing. The overview usually includes how each stage will be tested at unit level, followed by how the entire migration will be tested from start to end to ensure that the data flows accurately.
Unit test specifications will define test groups, which contain individual tests for that particular area of the migration. Each test should be broken down into its component steps, including description and expected results.
Stage 5: Development
Our consultants use an agile methodology to develop a data migration project in stages. This has proved particularly successful in migrations where a number of stakeholders are involved. An agile approach, which is clearly visible across all teams, ensures that risks are mitigated as soon as they occur. It also provides test data relatively early in the process.
We also create a test framework. This framework allows tests to be run regularly down to unit level, highlighting any potential issues.
Stage 6: Execution
Dry runs are often carried out to test the go-live strategy, enabling the go-live plan to be adjusted if necessary. When you’re ready to go live, you might consider implementing the migration at the weekend to reduce disruption to the organisation. Alternatively, you could run the old and new systems concurrently and transfer data piece-by-piece. Our preference, if business objectives allow, is a parallel migration. This can increase budgets and timescales, but enables your team to address any issues that occur, with minimal disruption.
This process helps ensure that the project is delivered successfully with minimum risk.