Data quality and record linkage techniques

This book offers a practical understanding of issues involved in improving data quality through editing, imputation, and record linkage. The first part of the book deals with methods and models, focusing on the Fellegi-Holt edit-imputation model, the Little-Rubin multiple-imputation scheme, and the Fellegi-Sunter record linkage model. The second part presents case studies in which these techniques are applied in a variety of areas, including mortgage guarantee insurance, medical, biomedical, highway safety, and social insurance as well as the construction of list frames and administrative lists. This book offers a mixture of practical advice, mathematical rigor, management insight and philosophy.

Author Notes

Thomas N. Herzog, Ph.D., ASA is the Chief Actuary at the U.S. Department of Housing and Urban Development
Fritz J. Scheuren, Ph.D., is a Vice President for Statistics with the National Opinion Research Center at the University of Chicago
William E. Winkler, Ph.D., is Principal Researcher at the U.S. Census Bureau

Preface	p. v
About the Authors	p. xiii
1 Introduction	p. 1
1.1 Audience and Objective	p. 1
1.2 Scope	p. 1
1.3 Structure	p. 2
Part 1 Data Quality: What It is, Why It is Important, and How to Achieve It
2 What Is Data Quality and Why Should We Care?	p. 7
2.1 When Are Data of High Quality?	p. 7
2.2 Why Care About Data Quality?	p. 10
2.3 How Do You Obtain High-Quality Data?	p. 11
2.4 Practical Tips	p. 13
2.5 Where Are We Now?	p. 13
3 Examples of Entities Using Data to their Advantage/Disadvantage	p. 17
3.1 Data Quality as a Competitive Advantage	p. 17
3.2 Data Quality Problems and their Consequences	p. 20
3.3 How Many People Really Live to 100 and Beyond? Views from the United States, Canada, and the United Kingdom	p. 25
3.4 Disabled Airplane Pilots - A Successful Application of Record Linkage	p. 26
3.5 Completeness and Accuracy of a Billing Database: Why It Is Important to the Bottom Line	p. 26
3.6 Where Are We Now?	p. 27
4 Properties of Data Quality and Metrics for Measuring It	p. 29
4.1 Desirable Properties of Databases/Lists	p. 29
4.2 Examples of Merging Two or More Lists and the Issues that May Arise	p. 31
4.3 Metrics Used when Merging Lists	p. 33
4.4 Where Are We Now?	p. 35
5 Basic Data Quality Tools	p. 37
5.1 Data Elements	p. 37
5.2 Requirements Document	p. 38
5.3 A Dictionary of Tests	p. 39
5.4 Deterministic Tests	p. 40
5.5 Probabilistic Tests	p. 44
5.6 Exploratory Data Analysis Techniques	p. 44
5.7 Minimizing Processing Errors	p. 46
5.8 Practical Tips	p. 46
5.9 Where Are We Now?	p. 48
Part 2 Specialized Tools for Database Improvement
6 Mathematical Preliminaries for Specialized Data Quality Techniques	p. 51
6.1 Conditional Independence	p. 51
6.2 Statistical Paradigms	p. 53
6.3 Capture-Recapture Procedures and Applications	p. 54
7 Automatic Editing and Imputation of Sample Survey Data	p. 61
7.1 Introduction	p. 61
7.2 Early Editing Efforts	p. 63
7.3 Fellegi-Holt Model for Editing	p. 64
7.4 Practical Tips	p. 65
7.5 Imputation	p. 66
7.6 Constructing a Unified Edit/Imputation Model	p. 71
7.7 Implicit Edits - A Key Construct of Editing Software	p. 73
7.8 Editing Software	p. 75
7.9 Is Automatic Editing Taking Up Too Much Time and Money?	p. 78
7.10 Selective Editing	p. 79
7.11 Tips on Automatic Editing and Imputation	p. 79
7.12 Where Are We Now?	p. 80
8 Record Linkage - Methodology	p. 81
8.1 Introduction	p. 81
8.2 Why Did Analysts Begin Linking Records?	p. 82
8.3 Deterministic Record Linkage	p. 82
8.4 Probabilistic Record Linkage - A Frequentist Perspective	p. 83
8.5 Probabilistic Record Linkage - A Bayesian Perspective	p. 91
8.6 Where Are We Now?	p. 92
9 Estimating the Parameters of the Fellegi-Sunter Record Linkage Model	p. 93
9.1 Basic Estimation of Parameters Under Simple Agreement/Disagreement Patterns	p. 93
9.2 Parameter Estimates Obtained via Frequency-Based Matching	p. 94
9.3 Parameter Estimates Obtained Using Data from Current Files	p. 96
9.4 Parameter Estimates Obtained via the EM Algorithm	p. 97
9.5 Advantages and Disadvantages of Using the EM Algorithm to Estimate m- and u-probabilities	p. 101
9.6 General Parameter Estimation Using the EM Algorithm	p. 103
9.7 Where Are We Now?	p. 106
10 Standardization and Parsing	p. 107
10.1 Obtaining and Understanding Computer Files	p. 109
10.2 Standardization of Terms	p. 110
10.3 Parsing of Fields	p. 111
10.4 Where Are We Now?	p. 114
11 Phonetic Coding Systems for Names	p. 115
11.1 Soundex System of Names	p. 115
11.2 NYSIIS Phonetic Decoder	p. 119
11.3 Where Are We Now?	p. 121
12 Blocking	p. 123
12.1 Independence of Blocking Strategies	p. 124
12.2 Blocking Variables	p. 125
12.3 Using Blocking Strategies to Identify Duplicate List Entries	p. 126
12.4 Using Blocking Strategies to Match Records Between Two Sample Surveys	p. 128
12.5 Estimating the Number of Matches Missed	p. 130
12.6 Where Are We Now?	p. 130
13 String Comparator Metrics for Typographical Error	p. 131
13.1 Jaro String Comparator Metric for Typographical Error	p. 131
13.2 Adjusting the Matching Weight for the Jaro String Comparator	p. 133
13.3 Winkler String Comparator Metric for Typographical Error	p. 133
13.4 Adjusting the Weights for the Winkler Comparator Metric	p. 134
13.5 Where are We Now?	p. 135
Part 3 Record Linkage Case Studies
14 Duplicate FHA Single-Family Mortgage Records: A Case Study of Data Problems, Consequences, and Corrective Steps	p. 139
14.1 Introduction	p. 139
14.2 FHA Case Numbers on Single-Family Mortgages	p. 141
14.3 Duplicate Mortgage Records	p. 141
14.4 Mortgage Records with an Incorrect Termination Status	p. 145
14.5 Estimating the Number of Duplicate Mortgage Records	p. 148
15 Record Linkage Case Studies in the Medical, Biomedical, and Highway Safety Areas	p. 151
15.1 Biomedical and Genetic Research Studies	p. 151
15.2 Who goes to a Chiropractor?	p. 153
15.3 National Master Patient Index	p. 154
15.4 Provider Access to Immunization Register Securely (PAiRS) System	p. 155
15.5 Studies Required by the Intermodal Surface Transportation Efficiency Act of 1991	p. 156
15.6 Crash Outcome Data Evaluation System	p. 157
16 Constructing List Frames and Administrative Lists	p. 159
16.1 National Address Register of Residences in Canada	p. 160
16.2 USDA List Frame of Farms in the United States	p. 162
16.3 List Frame Development for the US Census of Agriculture	p. 165
16.4 Post-enumeration Studies of US Decennial Census	p. 166
17 Social Security and Related Topics	p. 169
17.1 Hidden Multiple Issuance of Social Security Numbers	p. 169
17.2 How Social Security Stops Benefit Payments after Death	p. 173
17.3 CPS-IRS-SSA Exact Match File	p. 175
17.4 Record Linkage and Terrorism	p. 177
Part 4 Other Topics
18 Confidentiality: Maximizing Access to Micro-data while Protecting Privacy	p. 181
18.1 Importance of High Quality of Data in the Original File	p. 182
18.2 Documenting Public-use Files	p. 183
18.3 Checking Re-identifiability	p. 183
18.4 Elementary Masking Methods and Statistical Agencies	p. 186
18.5 Protecting Confidentiality of Medical Data	p. 193
18.6 More-advanced Masking Methods - Synthetic Datasets	p. 195
18.7 Where Are We Now?	p. 198
19 Review of Record Linkage Software	p. 201
19.1 Government	p. 201
19.2 Commercial	p. 202
19.3 Checklist for Evaluating Record Linkage Software	p. 203
20 Summary Chapter	p. 209
Bibliography	p. 211
Index	p. 221

Available:*

On Order

Summary

Summary

Author Notes

Table of Contents