Some estimates suggest that 90% of the world’s data was only created in the past two years. With a massive influx of new data, we’ve also seen huge growth in the demand for big data systems and business intelligence (BI) tools for businesses to make effective decisions. 

While it’s great that we have systems that are capable of storing these vast volumes of data, being able to take the data sets stored within, and use them for a new tool is still one of the biggest challenge.

Many business leaders get caught up in the excitement of a digital transformation program, and the opportunity to replace their legacy systems with the latest CRM, BI tool, or credit reporting system.BHowever, consolidating data from numerous platforms can be a major challenge when these systems have widely varying source and target structures. Poor data quality also makes the migration of data into a new system incredibly difficult, and drastically limits the effectiveness of the new system you’re trying to implement. 

Poor data quality has a variety of different causes including:

• duplicates 
• incorrect data
• differing formats between source and target 

In the haste to implement a new system, we see many businesses underestimating the complexity of a data migration and the level of data preparation involved to ensure it is successful. Failing to prepare your data adequately may potentially cost you hundreds (or thousands) of lost man-hours, and you might never see the return on your investment in a new technology solution.

You can’t control what you can’t see

What many people fail to realise is that data is rarely ever at the level of quality required for use in analytics when it’s extracted from a source repository. Before anyone in your business can actually begin building dashboards and generating reports, the data needs to be:

• cleansed
• standardised
• deduplicated
• transformed specifically for analysis

This is the process of data preparation, and it includes all the activities required for bringing source data up to the level of quality required for actual analysis and decision making. Simply put, the better the quality of data, the smoother your transformation process will be.

36 percent of organisations surveyed are planning data preparation projects in the next 12 months (up from 27% the prior year), according to our 2018 global data management benchmark report. Key drivers of these data preparation activities are most likely to be:

• analytics (48%)
• operations (45%)
• financial reporting (44%)
• business intelligence (40%) 

For any migration into a new system, it’s essential to gain full visibility into your data so that you can exercise real control over the preparation process. With the right data quality management solution, you will be empowered to:

• analyse all of your data upfront
• expose all of the data quality issues 
• generate mapping specifications
• de-risk and streamline your migration project
• move your data to the new environment, on time and on budget

Alan Thornton
General Manager, Data Quality Southeast Asia