Conversion of data from one form to another, or from one system to another, is a common task in IT organizations. Upgrading systems, merging companies, installing data warehouses, and extending the life of legacy systems all require data migration. Some situations may even require converting paper documents into digital form. While many aspects are technical in nature, data migration should not be left to technicians. Without careful managerial oversight, failures in data migration can cause project cost and schedule overruns, as well as quality problems with far-reaching effects in the organization.
This Research Byte is a summary of our full report, Data Migration Requires Process, Tools, and Expertise.
When converting data from one form to another, the most important criterion of success both during and after the transition is maintaining the quality of the data. Success requires following a clearly defined process, automating procedures as much as possible, and paying close attention to detail to keep risks and costs within bounds. These activities include mapping, extracting, cleansing, loading, and verifying, as outlined in Figure 1.
The process begins with mapping data formats from the old system to the corresponding formats in the new system. Next, the data is extracted from the old system. At this point, thorough data cleansing is essential to improve performance on the new system, boost data quality, and remove obsolete or redundant data. After the data is loaded into the new system, the transferred information should be carefully verified to eliminate errors. One way to do this is to run the old and new systems in parallel and compare the results. Expect that the cleanse, load, and verification steps will need to be repeated, as this is where the majority of errors tend to occur.
These steps are analyzed in more detail in the full version of this report, which also provides an overview of the data migration process and some of the decisions and issues managers need to address. We also examine the variety of open source tools that can help to automate the task. Finally, we conclude with our recommendations for managing a smooth data migration process.
Regardless of whether data migrations are performed in-house or carried out partially or completely by an outside service provider, data conversion is often on the critical path for major application development projects. Yet data migration is often under-managed and treated as a low-level technical activity. Experienced IT leaders appreciate the critical importance of data migration and provide sufficient resources to get the job done right.
This Research Byte is a brief overview of our report on this subject, Data Migration Requires Process, Tools, and Expertise. The full report is available at no charge for Computer Economics clients, or it may be purchased by non-clients directly from our website at https://avasant.com/report/data-migration-requires-process-tools-and-expertise-2008/ (click for pricing).