Data Migration Series: What is data migration?

Put simply, data migration is the process of moving your data from source systems to a target location. Sounds easy right? After all we are all used to moving files and data around in our daily lives. In practice, data migrations are far more complex than that.

Independent research from companies such as Bloor Group and Gartner confirms this:

“83% of data migration projects either fail or exceed their budgets and schedules.” – Gartner

“More than 80% of data migration projects run over time and/or over budget. Cost overruns average 30%. Time overruns average 41%.” – Bloor Group

So, what makes data migrations so challenging?

The challenges of data migration

To ensure a successful outcome when undertaking a data migration project, it’s essential that you understand the challenges that may be faced. Therefore, it is imperative that you have this understanding way before attempting any significant migration. The main challenges are:

• Poor Data quality
• Data complexity
• Security of data
• Stakeholder engagement

Let’s take a look at each in more detail.

Poor data quality

The more data you have, and the longer that data has existed, the higher the likelihood that your data will have quality issues. It goes without saying that when installing, or upgrading to, new software solutions in your business there will be an expectation that these new systems will make life easier and more efficient. For that to happen you will need to address your data quality issues beforehand. After all, Garbage in = garbage out.

Moving inconsistent or incomplete records to a new system will inevitably result in the same issues that you had before upgrading and negate the benefits of changing systems.

Data complexity

Usually, the complexity of the data increases with the amount of data that you have and the various types of data you have. Different systems will have different ways of storing the same information and your target system may have another way of storing the information.

For example, a sales CRM system may store a phone number as three separate fields, country code, area code and number. A separate billing system may want that information as a single string. Finally, a customer care system may require the number as the country code in one field, with the area code and number in another field.

These inconsistencies in the data will need to be addressed during the transform phase of a migration. Having the knowledge of the data architecture for all source systems as well as the target system is a must. Time spent researching and planning up front will pay dividends later.

PhixFlow allows you to examine the data from all sources, create Entity Relationship Diagrams (ERD’s) and business rules to normalise the data before your final migration.

Security of data

Data security is a hot topic these days. There’s been numerous news stories relating to data breaches, and the consequences of getting it wrong can be disastrous. Huge fines, legal action, reputation and customer confidence can all be shattered in an instant.

When dealing with data migrations it’s often the case that you can be dealing with some highly sensitive data. Privacy concerning customer details and financial data, for example, must be treated with the utmost care.

It is vital, therefore, that you choose your data migration tools and approach wisely. Check what security measures you have in place to protect the data, Assess the security credentials of any software solutions that you plan to implement for the migration.

Stakeholder engagement

Upgrades to legacy systems and system infrastructure maintenance falls under the control of the IT department. Therefore, it’s no surprise that data migration projects end up with a problem for ‘IT to deal with’. From a technical standpoint this makes sense, as they are more likely to understand the process of moving and transforming the data. However, they will have gaps in their knowledge, as they are often detached from the business processes that affect the data in various business areas.

Getting key stakeholders that utilise the data involved at the earliest opportunity will ensure a smoother, and quicker, transition to a new system. Increased involvement from the stakeholders allows you to take a collaborative approach to building the business rules required when transforming the data.

With PhixFlow, you can increase stakeholder involvement by providing them with easy-to-use dashboards, reports, data correction forms, and automated workflows. This greatly simplifies the process of working and engaging with them in the creation of transformation rules whilst getting valuable feedback on the new datasets at an early stage.

The data migration process

So, once you have researched the potential pitfalls and chosen your preferred tool for the migration, what are the stages in a typical data migration project?

You’ve probably heard of the acronym ETL, which stands for Extract Transform and Load. On the face of it ETL describes exactly the process of migrating data. Or does it?

Partly, but not entirely. ETL is quite a simplistic way of looking at a data migration. However, we think there’s more than just those three stages, especially for it to be a successful project.

With ETL, you have simply moved the data from one place to another. However, there will be data that needs corrections, or enriching, data that is missing and there will be no audit trail or reconciliation.

The PhixFlow Low-Code data migration process

At PhixFlow, we believe that there should be a few extra steps, including the data preparation, data corrections, data enrichment, reconciliation, and a full audit trail.

Ok, so this hasn’t got a nice fancy acronym (we’re working on it!) and it looks like there are lots of steps. That’s because data migrations are complicated, which is why so many of them fail. All too often they are viewed as a simple task The reality is an integrated solution enabling end-to-end data extraction, analysis, correction, transformation, reporting and load capability with a full audit trail from the first touch of the data through to the upload, together with an independent reconciliation process to ensure no data is left behind, is required.

Why use PhixFlow Low-Code?

By utilising our experience, coupled with our Low-Code platform we have developed a methodology to ensure a successful outcome for your data migration project. It is this proven methodology, combined with the PhixFlow integrated platform and experienced team that enables us to deliver successful migration projects, time after time.

In our approach, we start with an examination of the data, across all systems. This is done utilising small data sets, to prove the full end-to-end migration process. Then we take an incremental approach to building the business rules, on increasingly larger data sets, making small adjustments and checking the results to build out the final solution

Reconciling a data migration

PhixFlow supports an independent reconciliation (R) process to analyse, verify and report on the data and migration progress, with five reconciliation points: R1 providing an integrated report on the data ingested from multiple sources. R2 reporting on the filtered data to show what has been rejected (and why), what has been highlighted for further investigation, what has been transformed using automated rules and what was passed straight through. R3, a report on the final data being provided to the target system(s) reconciled to the initial data, from this it is possible, in conjunction with the audit trail to achieve complete insight into your data migration.

PhixFlow goes further. With an R4 reconciliation we read the data back out of the target system to ensure it has been correctly read in and stored, providing a full end-to-end check. For some organisations a further reconciliation is beneficial, R5. An R5 check looks at the output of the target system results and compares them to output of the source systems, compensated by any data not migrated across, to ensure not only is the data recorded correctly but the system performance is as expected e.g., a bill-to-bill reconciliation.

This approach ensures that everyone involved in the project can see that the process is working. They will see the current stage of the project, drill-down into the detail, simplify a complex process into a series of manageable steps, and improve the robustness of any migration. All of this gives confidence in the process and project timelines.

Get your data migration project off to the right start with our free data migration checklist.

Alternatively, contact us for a demonstration on how the PhixFlow Low-Code platform can be the catalyst for success with your project.