eVACS was developed by, and is a trademark of, Software Improvements and was used in the 2001 ACT election. eVACS consisted of two major components: a system for casting votes via a computer, and a system for counting votes by computer according to the Hare-Clark method. Some of the original source code for this system is publicly available from the ACT Electoral Commission (ACT EC) website. The source code is distributed under the terms of the GNU General Public License (GPL) and the copyright is held by Software Improvements. In particular, the code that implements vote counting according to Hare-Clark is "open" to the public. Making the source code available in this way allows for public scrutiny of an electronic voting process. As part of our investigation we took this code and built the necessary extra infrastructure to run the vote counting code.
These notes briefly describe the process and experience of getting the eVACS counting code to work with real and test data. We wanted to do this as easily as possible and make no modifications to the eVACS counting code because our primary aim was to run the counting code to obtain test and comparison results. The prose below contains a reasonable amount of technical detail, all of which may not be accessible to the average reader.
eVACS ran on a customised Debian linux system and used PostgreSQL as the database engine. We used Red Hat Linux which includes PostgreSQL as an optional part of the install image.
The election data from the ACT 2001 election were obtained from ACT EC. The ACT EC supplies CSV data files (comma separated records in text files) and the corresponding Microsoft Access database files with table and record documentation on a CD-ROM for AU$31.50. Note that the tables and fields in the Access database do not directly correspond with the structure of the eVACS Postgres database. We generated additional test data sets by sampling the real data, as well as by inventing new data.
We tested the counting program with real data. The first anomaly we found is that the program produces no intermediate set of results, but instead creates postscript files containing the results. That is, the code "draws" the results but ignores the concept of page boundaries or position and placement. Thus it is not possible to view or print the results without further manipulation of the postscript files.
Initially we edited these postscript files by hand to make the results viewable, but later we wrote a script to enhance the postscript file and make it easier to convert to other formats like acrobat/pdf using freely available utilities.
Other script files in the fragment of eVACS made publicly available indicate that another script was provided to the end-user so that the result file could be sent to a specific printer. Other than this the details of how the results are obtained is unclear. The overall output process appears to be unusually convoluted, including the fact that a separate perl script is provided just to modify some of the text decoration in the output files. The output process also seems to indicate that there is no simple machine readable form of the results and that the spreadsheet results available on the ACT EC website would have been entered by hand!
As expected, the counting program results for real data agreed with the published results.
We ran a small number of other simple tests using data in the same format as the data supplied by the ACT EC for the 2001 ACT election. The invented test scenario was the "Teletubbies vs. US Presidents" election: In the electorate "Test" there are two parties and a total of seven candidates. Each candidate has a different number of primary votes. There are no further preferences.
When the "Test" electorate has 5 seats available, the eVACS counting code reports only 4 as being elected. In fact when the number of seats is 4, 5 or 6 the eVACS counting code reports as elected one less candidate than the number of seats to be filled. When the number of seats is 1,2,3 or 7 the eVACS counting code reports the correct number of elected candidates. The tests reveal that unless there are as many seats as candidates, "TinkyWinky" never gets elected even though he should be the fourth candidate elected in all these cases.
We traced this defect to confirm that it was not just a fault of the report generating code. The function used by the counting code to report an elected candidate is never called for the missing elected candidate. We did not trace this bug further, but see the notes on inspecting the counting code below.
Of lesser importance is the fact that when there are more seats than candidates, the system produces a segmentation fault (crashes). This is also the case when the number of candidates becomes less than the number of seats due to the death of some number of candidates. In both cases the ACT Electoral Commissioner would intercede and the remaining candidates would be declared elected (Sections 111 & 112, Electoral Act 1992 (ACT)), however a count for statistical purposes would be impossible. For mission-critical systems, a more defensive style is appropriate: identifying and reporting on unlikely or even seemingly impossible errors or states serves as a safeguard, and can indicate possible faults in some other part of the system.
Scrutiny sheets for the Teletubbies Test Election are available here. The CSV data corresponding to the Teletubbies Test Election is included in the source code collection available from the bottom of this page.
To further understand what the counting code does, we attempted a simple code review. We found that in reality the eVACS counting code alone is too big to fully scrutinise by eye within a reasonable time frame because it consists of approximately 1000 lines of C code in one file. It is generally accepted that an efficient and effective code review is easier with "Unit" code containing a small specific set of data definitions and functionality. We spent four to five hours scanning the code and performing a brief code review:
Determining a separation of concerns in the actual code would be the next step in breaking down the process of interpreting the code. The most obvious improvement is to refactor the code to make it more modular.
The fragment of the eVACS system made publicly available provides an excellent way of maintaining public scrutiny in a complex process that has not traditionally used information technology. The code provided by the ACT Electoral Commission is not the complete eVACS code. Moreover, the amount of code provided is not sufficient to compile and run the software in order to perform testing.
While the eVACS vote counting fragment itself is easy to obtain, we had to write further code to construct and test it. Notably there is little documentation, the counting system is difficult to fathom, and while some of this is due to the complexity of the Hare-Clark method, the construction of the code omits basic software engineering concepts that encourage maintainability and make it understandable. While these issues may result from initial design, the overall impression is that they probably result from the code being completed in a limited amount of time.
The anomalous reporting in the Teletubbies Test Election examples indicates a bug in the counting algorithm. This bug has now been acknowledged, traced, and fixed by the ACT Electoral Commission and Software Improvements. Note this bug was found with only minimal scrutiny of the code fragments of the eVACS system. It is quite feasible that a more detailed scrutiny could expose other bugs.
We reiterate that the results produced by our own in-house vote counting code for the ACT 2001 elections is in full agreement with the official results. However, note that the fundamental difference between an empirically tested system and a verified system is that the verified system will comply with the original specification in every respect. This covers all possible cases, test or real.
The modified eVACS source code package and test data used is available here: revacs.tar.gz. This package contains further notes on building and running the system.