Top 3 challenges for data migration projects

Data migration projects have a notorious reputation for being both highly challenging and prone to failure. In fact, according to a Gartner report, as many as 83% of data migration projects exceed their deadline and/or budget and in some cases, fail entirely¹.

Now, obviously failure isn’t an option but there is good news! Over the last few years there has been an increased presence of dedicated migration tools and specific methodologies which in turn is helping to improve the success rate of data migration projects. These tools and methodologies alone though will not guarantee you success with your project. Firstly, you must understand the main challenges that can be faced and then what to do about them.

So, if you are wondering how to make your data migration a success read on to learn more about the top 3 challenges for data migration projects and, more importantly, what to do about them.

What is a data migration?

A data migration is defined as “the process of moving data from one system to another”, which sounds simple, so where do the challenges arise? In this next section we have picked out the top three challenges for you to be aware of and how you can approach these challenges to ensure your project is a success.

Ensure project success with our data migration checklist.

Data Migration Checklist

• Email Address
• Comments
  This field is for validation purposes and should be left unchanged.

Δ

Challenge 1 – Lack of business engagement

All too often the decision to move systems or undertake a data migration has been handed to the IT department to ‘deal with’. Whilst the IT department may understand the legacy data, they may also have gaps in their knowledge of the business processes involved in specific business areas. This, in turn, can lead to assumptions being made which may cause problems later down the line. These problems usually manifest themselves in the form of data load failures or User Acceptance Testing failing due to the system not being able to complete an expected task.

The type of data migration project that you are undertaking will have an impact into just how much involvement other areas of the business will need to have. For example, moving from one ERP system to another will require much more involvement from the business than an IT centric migration such as moving from one cloud server to another.

The need for business engagement is clear, but what happens when it’s difficult to engage with the key stakeholders due to their time commitments? When the time that these stakeholders can grant you is limited it pays to plan for this early to avoid losing momentum.

Consider using sample data that can be queried and analysed in real time, allowing you to walk through scenarios with the stakeholder to identify any questions and corrections that can be raised immediately, helping to avoid future delays.

Challenge 2 – Lack of legacy data knowledge

The success of any data migration project will always rely upon a good understanding of the data that you already hold. Often, the reasons for moving to a new system is due to the lack of understanding of the data in existing systems, and, as time has gone on, people have worked on workarounds to get the data they need. This leads to ‘Shadow IT’ creeping into businesses, using spreadsheets to collect and enhance data, but this data is not documented or governed in a central system.

Many of the original employees that set up a legacy system may have already moved on without properly documenting how, and what, data is stored and for what purpose. With these employees now gone, how can you possibly understand the data that is held? Without this understanding there is a high degree of certainty that the effort required may be underestimated leading to unexpected costs further down the line.

By using the right tools, you can gain a new understanding of your data and should ideally be done early on in any data migration project. This will then give you a complete, accurate and current view of your data and will determine the associated effort required to ensure that any legacy data is fit for purpose.

Challenge 3 – Data quality issues

Assuming the data you hold, in the system that you are migrating away from, has no quality issues can be a costly mistake, which multiplies when you are merging data from multiple systems. It is highly unlikely that the existing data has no issues at all and will fit beautifully into your new system. Resolving these issues manually is often not feasible and can be highly resource intensive, especially without the appropriate tools or expertise.

Take for example financial information, the existing source system may not require validation of bank account numbers, in contrast the new system can be more sophisticated and validate bank account numbers based on the regional details of the customer e.g. a US customer has a UK IBAN. In this case an import will fail resulting in a delay while the issue was resolved.

So, how do you avoid problems with data? Test, test, and test again! Using tools, such as PhixFlow, you will be able to test the process on segments of data, helping you to identify any areas that fail. This should be an iterative process that allows you to make changes to the validation rules and translations before pressing the green light on the final migration.

Summary

Undertaking a data migration project is daunting, especially if you are in a position where there are many sources of data that need merging into one, consolidated new system. The key to success is in the preparation. Having a detailed understanding of your legacy data and having engagement from each area of the business involved will ensure the success of your project.

• Tip 1: During the early project stages, use small sample datasets that can be quickly processed multiple times and reviewed with your business users. The old adage that 20% of the data will reveal 80% of the issues does really hold in our experience.
• Tip 2: Use tools which enable the migration logic to be built up and automated in steps. Migrations are projects of data discovery and exception handling.
• Tip 3: Use tools that enable rules to be configured to correct the data and can allow manual corrections to be recorded so that they can be automatically applied during subsequent migration runs.

For more information on how to approach your data migration project and the challenges that you may need to consider, please download a copy of our free ebook, Data Migration Checklist.