Gartner recently reported that 83 per cent of data migrations either fail outright or exceed their budgets and implementation schedules.
Transformation programmes, usually driven by an ambitious business change initiative, are a huge investment of time, resources and money. Delay or even failure can be very expensive and can adversely impact the business growth strategy.
Most transformation programmes experience delay, with some delays being very long and costly; data migration is usually at the top of the list of the reasons why, and often some team members are replaced as a result.
Having many years’ experience leading complex data migrations across many sectors, including with several housing providers, I want to share my understanding of the most common mistakes and ‘gotchas’ where things, even with the best of intentions, can go wrong.
Often overlooked, data migration is a complex and specialist activity. It requires a specific skillset, in-depth experience, a best-practice methodology and a proven toolset to deliver an end-to-end solution.
Common problems and how to avoid them
Start early
Don’t leave data migration until the last minute. It’s a key priority for success and is often overlooked until it is far too late to turn around, thereby delaying the published go-live date.
Aim to start your data migration activities as soon as possible. As a rule, the migration should be ramping up during the early design phase and in full flow by the detailed design phase. Follow a structured and proven data migration methodology as the basis of a detailed delivery plan; it will help to ensure that the right tasks are completed in the correct sequence and fundamental activities aren’t missed.
Use a data platform
A data migration platform is essential for successful delivery. Don’t build it yourself; we know it’s tempting and might excite some of your IT people, but it will take longer, cost more and ultimately be thrown away after go-live. It will also be a huge distraction from delivering the programme itself.
Select a platform that has been specially designed for data migration, rather than using a data reporting/query or system integration toolset that you might have already licensed. A proven specialist data migration solution will save time and, in the long run, prove to be a very cost-effective investment. Take advantage of the software-as-a-service subscription models available, and just pay for what you need for only as long as you need it.
Data quality
Don’t blithely assume that the quality of your data is okay. High-quality, accurate data is fundamental to a successful implementation, while bad data is unlikely to even load into the new application. Poor data leaves key stakeholders disappointed and underwhelmed, and sometimes even leads them to question the value of the entire transformation programme. Data migration should be considered as a great opportunity to completely transform your data into an accurate, consistent and fit-for-purpose asset.
The data-quality capability should enable both simple and complex business data-quality rules to be defined, grouped, reported and tracked over time. It should include dashboards to contextualise information so that senior management can focus on key priorities, plus detailed error reports/listings for your operational resources to use for data cleansing. Automated and regular reporting is needed to track progress, view trends and drive the data-cleansing initiative through to a successful go-live.
The data team
Don’t use inexperienced resources. Data migration is complicated, with many moving, interrelated parts; it simply can’t be delivered on time and within budget without specialist experience.
A strong and multi-disciplined data team, combining technical expertise and business knowledge, is essential to the successful delivery. Create a dedicated data team responsible for all migration activities with an experienced data lead at the helm.
The data lead should report direct to the programme manager/director, rather than just tagged onto a particular workstream. Data requires a high level of focus, governance and control and affects the entire programme. Avoid having data team members embedded in other workstreams or ‘business as usual’ activities; they will usually be given unrelated work and data migration will take a back-seat.
Clear task ownership
Don’t assume that all of your data-migration activities are being handled by an implementation partner. Ensure that every activity is clearly owned and there are no gaps, especially data transformation, data quality and cleansing. Delays mostly arise from confusion and assumptions on the responsibilities and ownership of the activities across third parties, the business itself and your internal technical teams.
As soon as possible, ensure a full ownership matrix is agreed across all activities; it’ll save time and avoid gaps and subsequent delays. Commercial contracts and SoWs should include all of these.
Some unscrupulous third parties may even prefer the migration ownership to be ambiguous in order to buy themselves time and have a ‘get out of jail free’ card when things do overrun and to avoid any commercial penalties.
Data security
Don’t weaken your data security protocols. Moving high volumes of data off your corporate applications can weaken the sophisticated security and protection that is provided by the applications themselves. For a controlled and secure process, the data-migration platform must be the central repository for all data-migration activities.
A data platform must be able to tightly control user access, allowing specific authorisation to limited datasets and limiting authorisation to only those that require access. The flow of data must be controlled and enable data to be segregated into clearly-defined groups (usually by business area) to control who has access to what, rather than a blanket authorisation to all.
Data scope
Don’t just migrate everything. The scoping decision around the data migration is often that everything should be migrated. That might appear, at first glance, to be the easiest option, but don’t forget that the more data you migrate, the more data there is to clean, test, load, reconcile and sign-off. Therefore aim to only migrate what is necessary from a legal, fiscal or critical business-operations perspective.
Knowing what is being migrated and what is not must be established as early as possible to avoid the emergence of extremely late requirements or having expensive work completed on unnecessary activities. Ensure the data migration scope is fully agreed, clearly documented and signed-off as early as possible, ideally during the design phase. Data migration is often left out of the formal programme change process, but formal change control is essential. The data team must be involved in decisions relating to changes to the solution design to assess the impact of any changes on the data migration.
Common oversights
Don’t forget documents (such as PDF and Word); they are usually very high volume and complex to migrate. Some document types will be closely related to your structured data and without them the business process could fail.
Don’t break the interfaces. This applies both internally, across the integrated application landscape, and with third parties, where they provide a service that depends on data exchange. Third-party data may need to be updated and aligned with new number ranges, codes, data formats and lengths that are all compatible, so ensure you include third parties in testing.
Avoid manual activities wherever possible because these often cause single points of failure. Aim to automate the full data-migration process because the failure of a single step could derail the entire migration. Check the full data-migration process can run within the go-live window, typically over a weekend, from business freeze to data sign-off.
Don’t skip the testing phases for data migration, so include the data migration as part of the system testing and in all formal test phases. This will flush out unexpected problems and allow time for investigations and corrections. It will also prove the new application actually works with real data, across a multitude of scenarios, rather than using data manufactured just to pass the test.
Being aware of these common problems in advance will really help the data migration delivery be a success. Unfortunately, in my experience, there are no short-cuts; they just lead to more risk, unexpected delays, and in the end a higher overall cost.
Graeme Cox is the co-founder of IntoZetta.
