Data Migration best practices, common issues & advice

What are some of the reasons why a business might want to consider data migration?

There are various reasons, but I think the obvious ones are:

Mergers and acquisitions: If one company has bought another, and you've got data spanning numerous systems, you want to bring that across into a single system or indeed multiple systems
Obsolete and unsupported systems: Older legacy technology, for instance, that may have high license costs and might no longer be supported. A recent example is Microsoft ERPNavsion 2013 to Dynamics 365 Business Central
Compliance or regulatory reasons: You may have fragile systems that might have security concerns, perhaps with previous data breaches, and they may need to move to a more secure platform

Some reasons might not materialise until later in the project:

Cost reduction: If you’re moving to cloud and managed services, you might traditionally have on premise systems that are either managed by a third party or have high license costs so you may want to move to a more scaleable and cost effective platform over the longer term
Competitive advantage: If the organisation is trying to get a competitive advantage over somebody else in the industry. If I worked for a large house builder for instance, and they wanted a much better customer and user experience, you might need business process streamlining, or to move to a new system that has better features all in one place, that will give them that competitive edge from a sales, marketing and customer care perspective. A more modern platform may also offer more value to your customers, or provide more services

What are some of the best practices for data migration?

Stakeholder Engagement

I think it's important to try to build allies as soon as possible by identifying who the key decision makers are and getting the key business stakeholders on your side. There might also be departments or functions in the business that aren’t directly ringfenced within the project that also need to be considered. If I've got the data owners, and I know who the subject matter experts and the third parties are, I can start doing a data landscape analysis, so that I can better understand the complexity, the structure and the volumes, to get to the bottom of what we know within the business and where the key gaps are.

Don't use data migration as an opportunity to start looking at all of the data quality issues and cleansing them, that’s an auxiliary piece of work that will certainly get the attention of the business stakeholders. Try to gather evidence rather than just relying on anecdotal issues that might be talked about in the business.

One of the best bits of advice I was offered recently on a couple of projects, was to try and hunt down the right person to speak to in the finance department so you can keep them updated on daily findings. They usually have a vested interest in your progress, in case the changes hurt the business from a financial point of view, but if they can see some of the benefits, that's a key way to get senior stakeholders engaged and involved.

Defining the scope

The scope needs to be well defined as early as possible so that you're not spending too much time tackling issues which are never going to be part of the new system. If you identify your data retention policy, you don't want to be fixing, for example, data quality issues when that data is not going to be brought across. Having a well defined scope is a much more efficient use of everyone’s time.

Phased or iterative approach

You might want to take a more iterative approach rather than waterfall, depending on the target solution, for example with MS Dynamics you might want to limit customisations to iterative updates which is less risky because you’re not trying to do everything at once. You should look at where the most gain will come from, what’s critical on day 1, in order to decide on your MVP. Perhaps for your business, customer care is the most important function, because that’s where you have your competitive edge, or in a higher education setting it might be administration, or perhaps your business needs to primarily focus on onboarding new cohorts of customers rather than managing those already in your system. You should be looking for where the most revenue is.

What are the common challenges businesses face during a data migration project?

People First approach

When you come in as a consultant working within the project, but not heavily embedded in the project, you're expected to provide solutions and answers, but we can't do that on our own. Getting access to the right people is a really common challenge; you need to be considerate of people’s time, and get approval from department or business leads so that you’re not taking away key resources when they’re needed.

You really need to take a people first approach, embed yourself in the organisation and really show that you understand how that business functions, and identify some of the consequences that could impact the business, so everyone is involved as early as possible. It’s often seen as a technical first role, rather than a business decision led activity, so you need to overcome that mindset, and bring the business with you on the whole journey from beginning to end.

When business stakeholders are disengaged that can also be a real challenge, so if I’m presenting something, I try to tone down the language so it’s not too technical and there’s no jargon.

Don’t take short cuts on Testing

Another really big lesson learned and challenge to overcome is when there is insufficient or ineffective testing to ensure that the data has been properly migrated. Early on, during the test phases, you can rely on synthetic data, but as soon as you start bringing across real data, you want a good depth and breadth of data to fulfil a number of scenarios, and if you don't get approval from information security, for example, then that can delay things later on when you want to start migrating live data.

If you haven't justified the purpose and usage of why you need to move live data, and if you’re moving to a new system it’s obvious it’ll have to be migrated eventually, so you need to get that agreement, think about how you would test and whether you need to have some information like a data security policy for instance, then it’s about escaping the data, and analysing the data and what the impact on testing could be. If you anonymise it too far, you might not even be able to test, because if, for example, you are testing a sale, and you need to look at sensitive, personal identifiable data, how do you know that sale is attached to the right individual if it’s anonymised?

In the move to big cloud solutions you’re looking at both structured and unstructured data. In its simplest form, structured data would just be the rows of records within the system that you want to bring across, things like customers, sales and vendors. The unstructured data could be things like documents in any format - Word documents, PDFs, or any other complex structures that you’d certainly want bring across to the new target system, anything that you’d want to find in a search.

Demarcation of responsibilities

Another big challenge is when you haven't got a proper agreement between the data migration workstream and the new system vendor. It’s important to get the roles and responsibilities and the demarcation of responsibility clear. In, for example, a statement of work, it should be clear what the customer is responsible for, what the supplier does, and what the vendor of the target system expects from the project and vice versa.

What are some of the consequences of a failed data migration?

The more obvious ones would be financial impact due to regulatory fines, loss of reputation, loss of customers and loss of sales. If there's any data loss whatsoever that’s critical from day one, you could lose customers and sales. Loss of reputation could come from bad data quality if you've migrated it across incorrectly, and you haven't tested it sufficiently. The UK has regulatory fines for data breaches and data privacy issues, so if, for example, you're testing in an integrated test environment, make sure that certain workflows are disabled, or that you haven't got live email addresses for example, because you don't want to be sending emails to customers or suppliers during testing, and you certainly don't want to be sending sensitive data out.

Lack of confidence can come from both within the project and the business, which is why the people first approach is important. Stakeholders may be sceptical, or simply not understand why the whole organisation is going through this change. During mergers and acquisitions in particular, or when moving to a platform, if you start if you lose confidence from the business, and lose the assurance that we're following best practice, the project could be delayed or become an afterthought, and this is typically when we see projects failing. The lack of confidence could impact timescales quite drastically and cause the migration to stall.

Do you have an example of a time when you needed to resolve data migration issue?

In one project I was given insufficient system access so I couldn't perform a data landscape analysis exercise satisfactorily. It’s wise to do data profiling, and a data quality assessment to make sure the data is fit for purpose structurally in the quality of the data before you move it to a new system, so if you haven't got agreement upfront for third party system access, that will delay things - you don't want any nasty surprises too far on in the project timeline. It’s really important to get those requests in as early as possible, get key decision makers on your side, and find out, maybe through the change board, who can give you approval access.

If you are looking at sensitive data, make sure you identify and classify it as "sensitive" as quickly as possible, and that way you can get justification for why you need system access. I would predominantly say to set up a staging environment where you can take that data offline, off the live system, and repeatedly refresh it into a much safer staging environment.

When the scope hadn't been properly defined from the beginning you can end up spending a lot of time looking at issues which you just shouldn't be spending time looking at, and if the scope isn't clearly agreed, you need to try not to let it become a design by committee. The way around it is to get the key people and do some interviews, and run some workshops with a number of different departments, so you can learn what's important to them and get that agreement across the board.

If in the first week of data migration things have gone missing like important vendors, or customers, to mitigate that it’s important that you do dry runs and dress rehearsals.

Another example of where things can go wrong and a way to mitigate it, is having both key business people and technical people involved in what the release implementation might look like. You should start by identifying the owners, establishing clear timescales, then building out the detail in the project and implementation plans. Have meetings with clear agendas, and action owners with their names attached to specific tasks, so you know you’re delegating to the right person who can make those decisions.

Do you have any advice for a business looking to find the right data migration partner?

From my point of view it's always good to get some independent advice early, prior to mobilising teams on a project, and when it's both technology and process agnostic. An independent advisor with previous migration experience should offer you project assurance in setting out that strategy and a framework early. If you're working with some of the bigger consultancies or vendors, they might have an idea of how that migration should go ahead, but they possibly haven't been involved in migrations before, so be sure you get somebody that's done it before and actually learned some lessons from past migrations.

I would usually ask the business what their constraints are and their expectations from the project, and then how we can align the two. It is usually useful to perform a pre-migration assessment, understand the legacy system and data landscape and get an understanding of what we know so far, and what artifacts we have in place like architecture or integration interface documents and diagrams if they exist because there's usually a lot of commonality between those workstreams. It might also be good to start with a list of key stakeholders, and get access to existing data catalogues and current knowledge. Getting third party system access, recommendations from your vendors and database access agreed up front would be useful, and starting to think about any constraints you may have with cyber security.

You should start by keeping things technology agnostic, but think about where your data is held, and what the current investment in technologies and tools is. Think about the skillsets you have in house and how you can upskill people and bring them on that process. As much as possible it’s about getting the key business people to sit with you and make those decisions.

Daniel Allen, a Director at DTA Consulting Limited, is a Lead Data Migration consultant with a strong development background specialising in Data Integration and Migration combined with best practice Data Governance, Data Management and Data Quality principles. He is also a co-founder of DnA Hub, which helps organisations manage their people, cultural change and challenges.

If you're looking to hire a data migration expert like Daniel Allen for your business you can contact us here or visit our IT Infrastructure page to discover other consultants like Daniel.