Febrl - Freely extensible biomedical record linkage

Peter Christen [1], Tim Churches [2]

[1] Department of Computer Science
Australian National University
Canberra ACT 0200
Australia
Email: peter.christen@anu.edu.au

[2] Centre for Epidemiology and Research
New South Wales Department of Health
Locked Mail Bag 961
North Sydney NSW 2059 Australia
Email: tchur@doh.health.nsw.gov.au

Release 0.3.1
July 1, 2005

Front Matter
Contents
1. Acknowledgments
2. Introduction
- 2.1 Performance
3. Data Cleaning and Record Linkage
4. System Overview
5. Configuration and Running Febrl using a Module derived from 'project.py'
6. Data Cleaning and Standardisation
- 6.1 Name and Address Cleaning and Standardisation
- 6.2 Output Fields
- 6.3 Name Cleaning and Standardisation using a Rules Based Approach
- 6.4 Name Cleaning and Standardisation using a Hidden Markov Model Based Approach
- 6.5 Address Cleaning and Standardisation using a Hidden Markov Model Based Approach
- 6.6 Date Cleaning and Standardisation
- 6.7 Phone Number Cleaning and Standardisation
- 6.8 Field Passing
- 6.9 Record Cleaning and Standardisation
- 6.10 Starting a Standardisation Process
7. Hidden Markov Models for Data Standardisation
- 7.1 Hidden Markov Model Implementation Module 'simplehmm.py'
8. Hidden Markov Model Training
- 8.1 Program 'tagdata.py'
- 8.2 Program 'trainhmm.py'
9. Record Linkage and Deduplication
- 9.1 Indexing
- 9.2 Field Comparison Functions
- 9.3 Record Comparator
- 9.4 Example Field and Record Comparator Initialisation
- 9.5 Classification
  - 9.5.1 Fellegi and Sunter Classifier
  - 9.5.2 Flexible Classifier
- 9.6 Starting a Linkage or Deduplication Process
10. Geocoding
- 10.1 G-NAF - A Geocoded National Address File
- 10.2 Outline of the Febrl Geocoding Process
- 10.3 Program 'process-gnaf.py'
  - 10.3.1 Memory Usage and Performance of 'process-gnaf.py'
- 10.4 Geocoding Project Module 'project-geocode.py'
- 10.5 Auxiliary Geocoding Programs
11. Output
- 11.1 Record Pair One-To-One Assignment Restrictions
12. Auxiliary Programs
- 12.1 Program 'randomselect.py'
- 12.2 Data Set Generator Program 'generate.py'
- 12.3 File Analysis Program 'fileanalysis.py'
13. Data Set Access
- 13.1 COL Data Set Implementation
- 13.2 CSV Data Set Implementation
- 13.3 SQL Data Set Implementation
- 13.4 Shelve Data Set Implementation
- 13.5 Memory Data Set Implementation
14. Look-up and Frequency Tables
- 14.1 Correction List
- 14.2 Tagging Look-up Table
- 14.3 Frequency Look-up Table
- 14.4 Geographic Location Look-up Table
- 14.5 Neighbouring Region Look-up Table
15. Logging and Console Output
16. Installation
17. Parallelism
A. Hidden Markov Model States
B. List of Tags
C. Rule-based Name Segmentation
- C.1 Input
- C.2 Process Overview
  - C.2.1 Step 1: Allocating the elements into one of the five sub-lists
  - C.2.2 Step 2: Parse each sub-list and assign into appropriate output name component
- C.3 Output
D. Manifest
E. To-Do: Outstanding Development Tasks, Possible Additions and Enhancements
- E.1 Data Cleaning and Standardisation
- E.2 Record Linkage and Deduplication
- E.3 Febrl System
F. Version History
G. Support Arrangements
H. ANU - Open Source License
Bibliography
Index
About this document ...