Evaluating online handwritten signature verification (HSV) systems is more challenging than evaluating most other biometrics systems, such as fingerprint systems. There can be significant variation in an individual’s signature even if it is written repeatedly at the same time. Furthermore, a person’s signature often changes over time. The device used for capturing a signature can also influence the quality of the signature. For example, a signature captured when a person is sitting down and signing on a page-sized tablet is likely to be different than one captured when a person is standing up and signing on an iPhone-sized device or at a checkout counter. Furthermore, some signatures are very simple, whereas some can be very complex. The signer’s cultural background also influences how simple or complex his or her signature is.

Perhaps because of these reasons, there have been only a few HSV performance competitions since 2004. Two were held in 2009. This paper describes one of those: the BioSecure Signature Evaluation Campaign (BSEC). The other was held at the 10th International Conference on Document Analysis and Recognition (ICDAR). These competitions used different signature databases; therefore, their results cannot be compared. The ICDAR competition used a signature database of 100 individuals, captured using a digitizing tablet. BSEC used two databases of signatures captured from the same 382 individuals, once with a digitizer and again using a personal digital assistant (PDA). The databases are publicly available.

The paper presents the results of two different tasks. One task studied the impact of the capturing device on performance. The other task evaluated (using a personal entropy measure) how the relative difficulty of recognizing an individual’s signature impacts performance.

The 12 systems tested by BSEC vary considerably. Some used only a few local features, while one of the systems used 100 global features.

An evaluation of the systems shows a large variation in their performance. The results of several situations are presented. If we only consider the first task, on the first dataset (DS2) when random forgeries (signatures of other people) were used, the equal error rate (EER)--when the false acceptance rate is equal to the false rejection rate--varied from 0.51 percent to 24.24 percent, although most systems performed below 3 percent. Using signatures captured from a mobile PDA, performance was similar, except in one system that performed much worse. When skilled forgeries were used, the performances of many systems were worse on the PDA database compared to their performances when signatures were captured from a digitizer.

The approach taken and the results obtained from BSEC should help HSV researchers build more reliable systems.