Data Migration Series: The importance of reconciliation during data migration

Reconciliation is a vital step in measuring the success of any data or system migration. Whether your data is static or dynamic (transactional) a reliable and fast means of determining the accuracy and progress of a migration is critical. Failure to reconcile this data will result in costly delays or, even worse, a migration with missing and incomplete data.

Before we look at how Low-Code applications offer a more flexible and accurate reconciliation, we must first examine the basics of reconciliation.

What is data reconciliation?

Data reconciliation is a phase of a data migration project where target and source data are compared to ensure that all data has been transferred correctly.

Why is data reconciliation important?

There are many reasons that can cause a data migration to fail. Without adding data reconciliation stages many issues can go unnoticed. This in turn can lead to poor customer service, production delays, and inaccurate data leading to poor decision making.

Some of the more common issues can be:

• Missing customer records
• Incorrect values
• Record duplications
• Badly formatted data
• Missing or broken relationships

These issues can be caused by many reasons, such as errors made in the initial data mapping logic, errors in the source data or even network issues when performing the migration.

How to perform a data reconciliation ?

There are many ways that you can perform reconciliations on your migration data. For example, manually checking using spreadsheets, using automated solutions and third-party tools.

Manual checks are resource intensive, slow, and susceptible to human error. Automated solutions can be too rigid, not providing the flexibility needed to suit your business needs. Using multiple 3rd party tools means further transformations and/or processing of the data is required. This means there can be no end-to-end audit trail.

Using a Low-Code platform to reconcile your data, and if possible, to perform the end-to-end migration will result in a less complex landscape and a faster resolution to any errors found.

How to use Low-Code data reconciliation in an end-to-end migration

With Low-Code platforms, such as PhixFlow, your data migration application will become the central point for all the data. Business rules, corrections, and the location in which the data is prepared for the final ‘load’ phase of the migration.

Within this approach we would recommend that you implement multiple reconciliation points to ensure that your project is successful. By adding these data reconciliation points you will be able to produce dashboards and reports, using drag and drop functionality. Each reporting screen will allow users to drill-down into each record and examine the details to ensure that all information has been captured and merged successfully.

These reports enable the results to be shared at regular intervals, allowing key members and data owners to become familiar with the results and issues well ahead of any dry runs. Also, these reports will provide the key evidence in the final go/no-go decision prior to switching operations to the migrated system.

PhixFlow supports an independent reconciliation (R) process to analyse, verify and report on the data and migration progress, with five reconciliation points: R1 providing an integrated report on the data ingested from multiple sources. R2 reporting on the filtered data to show what has been rejected (and why), what has been highlighted for further investigation, what has been transformed using automated rules and what was passed straight through. R3, a report on the final data being provided to the target system(s) reconciled to the initial data, from this it is possible, in conjunction with the audit trail to achieve complete insight into your data migration.

PhixFlow goes further. With a R4 reconciliation we read the data back out of the target system to ensure it has been correctly read in and stored, providing a full end-to-end check. For some organisations a further reconciliation is beneficial, R5. An R5 check looks at the output of the target system results and compares then to output of the source systems,

The benefits of using Low-Code for data reconciliation

Increased agility

Reconciliation at each step of the migration allows you to take an iterative approach, avoiding issues further down the line. Your teams and subject matter experts will be able to resolve any errors before moving on to the next step.

Increased business engagement

Using interactive screens and reports allows subject matter expert involvement throughout the entire process, from design to completion. Having them on board early ensures the right business rules are applied, facilitating greater trust in the project overall.

Reduction in time and costs

By their very nature, Low-Code platforms are designed to make the process of creating applications quick and easy. With data migration the same principles apply. This means you can build your data models and business rules, iterate and apply in no time at all.

Improved data quality

Having multiple reconciliation points, will uncover data quality issues that you may not have known about. Identifying these issues early on gives you the time to fix them before migrating the data to a new system. This, in turn, leads to a better outcome and improved data quality within your new systems.

Want to find out more?

If you’re interested in exploring how PhixFlow can assist you with your data migration project or would like to learn more on how the PhixFlow Low-Code platform can increase your chances of project success, please request a demo.