It's the data stupid! (data quality)(The Last Boy Scout)(Column)

As far as I know, the office of Chief Information Officer (CIO) is not an elective position in any organization. But if there were a company where the CIO was elected by the employees, I would run for that office on the slogan, "It's the data stupid!"

Whenever I start a Data Warehouse project, I try to warn the clients that they are in for a major shock when they start looking at data they have extracted from there legacy systems. They will find much of the data they thought they had is missing or wrong. They will find that when they run queries against the data, the results will not match with their operational reports from the legacy systems. Finally, I warn them that when the people who maintain the legacy system are confronted with the poor quality data extracted from their systems, they will blame the extraction process, claiming the problems are due to the data not being extracted correctly.

There is no reason for the people who maintain the legacy system to be so defensive and try to blame the messenger for the bad news. Many reasons exist for the poor data in legacy systems, none of which is due to carelessness or lack of morality on the part of those who maintain these systems. The prime reason for poor quality data is that the people who are charged with collecting that data are not the people who use the data. Therefore they have no involvement in the entire system process; hence no vested interest in it being correct.

A prime example is the warranty card that comes with many appliances. Many manufactures think this is a great instrument to collect data about their customers. But many consumers are annoyed by these cards and feel the questions are "none of the business of the manufacturer." As a result, many consumers throw the cards away. Others delight in deliberately putting down wrong information. For example, a friend of mind filled out a card claiming he was under 16 years old, had an income of over $100,000 and like to spend time with his grandchildren.

Within an organization, the same problems occur. I have seen many order entry screens that contain fields for data that are only of concern to the Marketing Department. But the Order Department's only concern is with taking and fulfilling an order. If the marking data is incomplete or incorrect, the product will still get to the customer, and it is unlikely that much will happen if the marketing department complains. But a lot will happen if the wrong product gets shipped, or the product gets shipped to the wrong place, so this data gets special attention.

Another reason for poor data quality is poorly designed systems. It is not that there are bugs in the software that mutilate the data, the problem is that the system is designed without any thought being given to data quality control.

Back in the old days of computer systems, when data was entered on punch cards. There were several ...