E.1 Data Cleaning and Standardisation

Fix problem in HMM address standardisation with warning 'This should never happen!' in which case a record is not standardised properly.

Add a single letter tag for address standardisation, similar to the II (initial) tag used in name standardisation. This will allow HMMs to more easily classify single-letter words correctly, and then allow their transformation at the output stage. For example, it is hazardous to transform every instance of the string ' r ' into ' road ' during the initial cleaning stage, but it is quite safe to transform any instance of 'r' into 'road' in the output stage where those instances have been classified as the wayfare type.

Current input data is split into words at space boundaries. It might be useful to allow word splits at other, user-configurable boundaries (at least all whitespace characters).

Provide a simple user interface to the HMM training module to make the checking of training data sets faster and easier. At the moment, these training data sets need to be edited in a text editor and a lot of time and mental energy is wasted scrolling the cursor to the correct place in the file to add or correct states. It is anticipated that the curses module will be used to provide a text-mode user interface on Unix and Linux platforms, and the Tkinter module will be used to provide a graphical user interface on all platforms which support this (including Microsoft Windows and Apple Macintosh).

Soft code the observation tags and hidden states used by the HMM standardiser so that users can specify their own sets of tags and states without having to modify the Python code.

Allow more than one standardisation HMM to be trained and used, and use the forward algorithm to choose the best HMM for each input record. For example, input data may contain a mixture of addresses from different countries, each of which has quite different conventions for specifying address, each requiring a different HMM.

Continue to develop the example standardisation training data and tagging look-up tables so that they work well with a wide range of typical Australian name and address data. Accept tagging and training data for other countries from contributors provided the tagging look-up tables are free from copyright restrictions.

Explore the utility of the Baum-Welch EM (expectation maximisation) algorithm in optimising the probabilities in the HMM used for standardisation. Probably use LogiLab's fast C code implementation of this (see http://www.logilab.org/projects/hmm/).

Explore the use of the forward-backwards algorithm for developing HMMs without explicitly specifying the hidden states.

Add fuzzy look-up of words in the tagging lists so that the user does not have to specify all possible mis-spelling of abbreviations of a word or group of words for it to be tagged correctly (although the current ability to do that is very useful). It is quite tricky to do a fuzzy lookup efficiently, but worth considering for the next version. Real-life testing has shown that words with minor mis-spellings which should have been tagged as a locality, institution type etc. occur very commonly. However, the HMM still manages to get the output state for most of them correct, but would do even better if they were tagged correctly in the first place by a fuzzy lookup of the tagging tables.

Add the ability to specify output transformation rules for the output fields, e.g. if the unit_number output field starts with 'unit' (as in, for example, 'unit42', then remove the 'unit' prefix from the unit_number field and make the unit_type field equal to 'unit'. There is no need to invent a new rule specification language to implement this - Python is already easy enough for users of Febrl to write their own transformation rules. All the user needs to do is specify a function name for the transformation of the output field. However, some more object-oriented refactoring would make this easier to implement, e.g. if the output was an object (class instance) which could be passed to such transformation functions.

Extend the above idea to input processing as well, to allow more than just substitution.

Provide an improved, more robust version of the tcsv.py delimited file parsing module.

It is not always sensible to concatenate fields which are already quite well segmented in a database record, only to try to parse them out again. One way not to lose such pre-existing segmentation without having to modify Febrl very much would be to add an option to add a comma (or some other character) as a delimiter between fields in the concatenated name or address string which is then presented to the Febrl parser. Such a delimiter is not wanted between every field, so there would need to be some way of specifying where to insert it. For example, if the original record was:

      Field1 = Wayfare number
      Field2 = Wayfare name and type
      Field3 = Locality
      Field4 = State/Territory
      Field5 = Postcode

then it would be useful to be able to re-concatenate that as:

      Field1 Field2, Field3 Field4 Field5

and present that to the parser. Of course, the HMMs would have to be trained using data in that format (i.e. with comma interpolation between fields 2 and 3 switched on), but at least we would then be able to distinguish between:

      23 Smith St North, Fairfield NSW 2345

and

      23 Smith St, North Fairfield NSW 2345

At the moment, this is presented to the standardiser as:

      23 Smith St North Fairfield NSW 2345

Another idea is to use a dictionary of wayfare names in each locality, derived from the phone book or similar to distinguish property names and wayfare names, e.g.

      Wymallee Arthur St Gundagai NSW 2345
      Windy Willows Ave Littleville NSW 2345

Now we know that 'Wymallee' is the property name (because it sounds like one), and 'Arthur' is the wayfare name (also, 'Wymalle Arthur' would be a strange name for a street), but for the second address, 'Windy Willows' is the wayfare name and there is no property name. Now, if it were possible to look up all streets in e.g. Gundagai, and a match was found for 'Arthur', but not 'Wymallee Arthur', then it is possible to infer, as a deterministic post-processing rule, that 'Wymallee' should be re-assigned to the property name output field. All this sort of processing, which is domain and site-specific, can just be done by users writing their own Python functions or methods to post-process the standardised data - like a plug-in. So all we need to do is provide a hook for such plug-ins, and a few examples.

Exploration of the use of Maximum Entropy Markov Models as described in http://citeseer.nj.nec.com/mccallum00maximum.html instead of HMMs for name and address segmentation. We see this as a very low priority, because HMMs work well enough, but we don't want to lose the idea of investigating them one day, or someone else investigating them for Febrl, since they are an elegant generalisation of HMMs but very similar. In fact, the only difference is that the emission matrix probabilities are tied to state transitions (edges), not to states (nodes) - hence they are depend on the current state and the previous state, not just the current state. This effectively allows states to overlap wrt the observations they emit. Nice!