Abstract—This paper presents an approach for evaluating video-based enforcement systems for motorway toll collection, which has been applied to the case of Portugal’s largest motorway operator, Brisa. The results of this evaluation have contributed to the design of a new LPR system, denoted advanced license plate recognition (ALPR), also described in this paper. The ALPR is currently being deployed not only by Brisa, but also by other operators that use the Via Verde system. A significant decrease in the need for human intervention has been an important improvement, in which the introduction of a tunable and trustable confidence level in the LPR process has played a key part.

I. INTRODUCTION IDEO-based enforcement systems in motorways have become increasingly widespread, due to the fact that

they allow efficient data capture 24 hours a day, as well as the easy storage, handling and transmission of the data in digital form [1,2,3]. Many of the subsequent enforcement procedures, based on License Plate Recognition (LPR) technology, can be automated, reducing operating costs and enabling greater effectiveness [4,5,6].

The role of such systems ranges from detecting and documenting traffic violations to providing enforcement in the electronic toll collection. The focus of the present paper is on the latter role, for the case of the Via Verde toll collection system. This work is integrated in the wider scope of the Intelligent Transport System - Interoperability Bus (ITS-Ibus), which is undergoing development [7].

The Via Verde was introduced in 1991 (which makes it a pioneering electronic toll collection system worldwide), on

Manuscript received March 13, 2006. This work was supported in part by Brisa Auto-Estradas de Portugal, SA.

J. G. Silva is with the Multimedia and Machine Learning Group, DEETC, ISEL, R. Conselheiro Emídio Navarro, 1950-062 Lisbon Portugal ( e-mail: jgs@ isel.ipl.pt ). G. C. Marques is with the Multimedia and Machine Learning Group, DEETC, ISEL, R. Conselheiro Emídio Navarro, 1950-062 Lisbon Portugal ( e-mail: gmarques@ deetc.isel.ipl.pt ). P. M. Jorge is with the Multimedia and Machine Learning Group, DEETC, ISEL, R. Conselheiro Emídio Navarro, 1950-062 Lisbon Portugal ( e-mail: pmj@ deetc.isel.ipl.pt). A. J. Abrantes is with the Multimedia and Machine Learning Group, DEETC, ISEL, R. Conselheiro Emídio Navarro, 1950-062 Lisbon Portugal (phone: +351-21-8317251; fax: +351-21-8317114; e-mail: aja@ deetc.isel.ipl.pt ).A. L. Osório is with the GIATSI Research Group, DEETC, ISEL, R. Conselheiro Emídio Navarro, 1950-062 Lisbon Portugal ( e-mail: aosorio@ deetc.isel.ipl.pt).

J. S. Gomes is with the DIT, Brisa Auto-Estradas de Portugal, SA, Quinta da Torre da Aguilha, Edifício Brisa, 2795-599 S. Domingos de Rana, Portugal ( e-mail: jorge.gomes@ brisa.pt). J. C. Braga is with Brisa Auto-Estradas de Portugal, SA, Quinta da Torre da Aguilha, Edifício Brisa, 2795-599 S. Domingos de Rana, Portugal ( e-mail: j.braga@ brisa.pt).

motorways operated by Brisa Auto-Estradas de Portugal, having subsequently expanded its coverage to other motorway operators and to other applications, e. g. controlling access to parking lots.

From the inception it was understood that, as a gateless system, the Via Verde would require enforcement in order to be successful. In regular conditions, toll collection is performed through Dedicated Short Range Communication (DSRC) at microwave frequencies (5.8 GHz) between antennas at the toll plaza and a tag, also called On-Board Unit (OBU), fixed to the windshield of the passing vehicle. In the event of an irregularity, for example a vehicle without an OBU or a communication failure, a rear photograph of the vehicle is taken and forwarded to the toll clearing company, Via Verde Portugal (VVP), for further processing.

From the photograph, it is possible to obtain the license plate number and match it to the client database, pursuing legal action if necessary, or simply initiating a normal toll charge in case the transaction failure was due to technical reasons (e. g. low battery on the OBU).

Given the number of photographs per day (around 15,000), manually reading the license plate numbers from all photographs is not a feasible option. In order not to remain limited to processing a small sample of the violations, and since in any case manually reading license plates is a tedious and time-consuming (and therefore expensive) process, a semi-automatic procedure based on optical license plate recognition (LPR) technology was adopted. The current system, based on rear capture, has been in operation for approximately three years.

The procedure is not fully automatic because currently available LPR methods still have a significant error rate. It should be noted that even an error rate of 5%, corresponding to 95% correctly recognized license plates, might lead to thousands of fines being wrongly issued per year, which would seriously undermine the system’s credibility.

Also, there are specific difficulties regarding trucks, trailers and vehicles with logotypes and written signs, which are often mistaken for license plates by LPR systems. Even when manually processing the photograph, in such cases (which amount from 3% to 5% of the total number of pictures) it often happens that the visible, rear plate belongs to the trailer instead of the vehicle itself, rendering identification of the vehicle owner considerably more difficult.

Evaluation of an LPR-Based Toll Enforcement System on Portuguese Motorways

Jorge G. Silva, Gonçalo C. Marques, Pedro M. Jorge, Arnaldo J. Abrantes, António L. Osório, Jorge S. Gomes and José C. Braga

V

Therefore, human intervention remains necessary, although the automatic recognition result can still be used to simplify the operator’s task, by presenting on-screen a suggested license plate number, which the operator may either confirm with a single keystroke, or edit if needed.

Even in a semi-automatic setting, an increase in the correct recognition rate will reduce the frequency of edits, thus mitigating the amount of operator intervention.

Moreover, it is possible to devise a strategy where “easy” photographs need not be subject to human intervention at all. This requires a criterion for deciding on fully automatic or manual processing for each individual picture.

It is proposed to treat the entire procedure as a decision system, taking into account a quantitative measure of uncertainty of the LPR result: a confidence level. Furthermore, it is intended not only to raise the overall recognition rate but also to narrow the situations where even manual processing is fruitless, by using frontal as well as rear capture. This raises issues concerning the privacy of the driver and passengers, which are addressed. The work reported here has led to a new design for the LPR system, called advanced license plate recognition (ALPR).

The remainder of this paper is organized as follows: the following sections describe the ALPR architecture and the proposed methodology for the evaluation of its performance, as well as that of similar systems. This is followed by a discussion of the decision system and the proposed confidence level. Next, the previous LPR architecture and

the new ALPR are subjected to comparative evaluation based on a sample of real motorway enforcement images, leading to a discussion of the results and some concluding remarks.

II. THE ALPR SYSTEM Whereas the previous LPR design relied on single capture

from the rear, the new ALPR design performs double capture, from the front and rear. Also, the ALPR uses both the visible and the Infra-Red (IR) spectra: namely, visible plus IR from the rear and IR-only from the front, generating a total of three images that are composed to form a single composite picture, encoded in JPEG format.

The physical setup is shown in Figure 1. There are two modules, front and rear. The rear module contains a computer equipped with a multi-channel frame grabber, an IR illuminator, a color camera operating in the visible spectrum and an IR camera with a filter for blocking visible light as much as possible. The front module consists only of a similar IR camera and an IR illuminator, connected to the rear module by coaxial and power supply cables. The computer in the rear module provides triggers to the cameras and the IR illuminators. It also runs the LPR, which is based on pattern recognition technology for real-time Optical Character Recognition (OCR), specifically an Artificial Neural Network (ANN).

The motivation behind this architecture, namely the option for double capture, is to deal with the problem of

Rear module

Front module

Color cameraIR illuminatorIR camera with filter

Color camera IR illuminator Computer with frame grabber

Fig. 1. Physical setup of the ALPR system. Above: the front and rear modules. Below: details of the rear module.

trailers while also improving the reliability of the LPR, by having two license plates available per vehicle. Moreover, it is desirable to decouple the license plate image acquisition process from that of the overall panoramic vehicle view, because the images serve different purposes and have distinct requirements.

The license plate images, front and rear, are intended for OCR and therefore should have high contrast, which is why they are acquired in IR. Combined with fast (1 / 4,000 s) shutters synchronized with pulsed IR illuminators, this enables nearly constant image contrast, by day or night [8]. The IR images should also zoom in to the license plate region as much as possible (so that resolution is maximized), although not so much that the cameras might miss the plates for vehicles passing at varying speeds (and with different plate locations).

These features, however, make the IR images unsuitable for providing context. It may be necessary, for instance at a court of law, to visually establish the surroundings, make and model of the vehicle. For this purpose, a rear view of the vehicle, in color and with a wider angle, is more adequate. That is the task of the visible spectrum color camera.

One of the reasons for using rear instead of front capture for the panoramic color view is that the latter would raise issues of privacy, since it might allow identification of the vehicle driver and passengers, which is not desirable. Therefore, a rear view is preferable. As for the front IR picture, the very features that make it appropriate for LPR processing would render it difficult to identify any passengers. Nevertheless, and since only the license plate is

relevant for the LPR, the region that contains it is cropped from both the front and rear IR images, which are then discarded - in fact, they are never written to disk.

Figure 2 shows the final composite photograph, where the cropped license plate sub-images are overlaid on the color rear view. Additional data, including the text output of the LPR system as well as a timestamp and vehicle class information from the AVDC (Automatic Vehicle Detector and Classifier), are placed in the JPEG header. The information flow, as well as the overall enforcement design are schematized in Figure 3. In an enforcement event, the ALPR and the AVDC are responsible for enabling answers to the questions “who should pay?” and “how much should be charged?”, based on the composite photograph, the LPR result and the detected vehicle class.

47 12 XT 47 12 XT

ALPRgenerated

picture

ALPRgenerated

picture

<other data>

LPRLPR

Fig. 2. Example of a composite picture generated by the ALPR, in JPEG format. The text strings containing the license plates, a timestamp, AVDC information and other data are place in the JPEG file header.

AVDC

IR1

IR2

V

LPRJPEG

Via VerdeCentral

Services

How much?

Who?

Fig. 3. Enforcement system architecture. When an enforcement event occurs, the ALPR creates a composite JPEG photograph to allow identification of the vehicle owner (“whose should pay?”), while the AVDC determines the vehicle class (“how much should be charged?”).

The decision, by the central toll clearing services, on how to proceed depends significantly on the reliability of the ALPR, which should therefore be evaluated

III. METHODOLOGY FOR LPR EVALUATION It is common, when evaluating not only LPR systems but

also classification systems in general, to place most of the emphasis on the recognition rate. While no doubt important, this is not the only relevant measure. There is an inherent degree of uncertainty to the recognition process that should be quantified and taken into account for enforcement decisions.

An enforcement system should act as a deterrent, by being able to detect and identify the perpetrators of violations. On the other hand, the detection and identification process should be reliable, particularly when fines and other legal actions are involved, in order not to be counterproductive.

It is possible to define the following parameters, as illustrated in Figure 4:

• The number of photographs taken, N; • The percentage of photographs that do not

correspond to violations, c; • The percentage of violation photographs where

the plate is legible, b; • The percentage of violation photographs with

legible plates that can be automatically processed, p.

Ideally, there would be no violations (c=1). For a given c, the quantity that should be maximized is N(1-c)bp, which is the number of enforcement violations that can be automatically processed, without operator intervention. The parameters that depend on LPR performance are b and p. Therefore, the proposed criterion for evaluation of LPR-based enforcement systems is the product bp.

The impact of the new ALPR design on b comes from using double capture with IR, thus increasing the percentage of legible pictures. Parameter p, on the other hand, must be

increased by applying decision theory and probabilistic reasoning based on a measure of uncertainty for the LPR result. That measure is called a confidence level.

IV. DECISION SYSTEM USING THE CONFIDENCE LEVEL Given a recognized plate, the enforcement system must

decide whether or not a given photograph can be processed without human intervention. It is unknown whether or not the recognition result, denoted s, is correct. Let z∈{0,1} be a random variable with value 1 or 0 for correct or incorrect recognition, respectively. Thus, the recognition rate is the probability P(z=1).

Moreover, let η denote a confidence level for the recognition result, and let T be a threshold value for η. The decision system, as illustrated in Figure 5, accepts or rejects the license plate s depending on whether or not η > T. There are four possible outcomes, corresponding to the possible combinations between z (correct/incorrect) and the decision (accept/reject). There are different costs, λ00, λ01, λ10, λ11, associated with each of the λ. For instance, it is more serious to accept an incorrect license plate (λ01), wrongly issuing a fine (a false positive), than to reject a correct plate (λ10 a false negative). The costs are defined by the motorway operator. Following Bayesian decision theory [9], the optimum threshold T is the one that minimizes the Bayesian risk. This optimum is obtained through the following expression:

1+=

rrT ;

1110

0001

λλλλ

−−

=r . (1)

For (1) to be valid, however, the confidence level should

be a close approximation of the posterior probability P(z=1 |η ), that is

)|1( ηη =≈ zP , (2)

meaning that license plates with confidence level, say, η=80% should be correctly recognized in approximately 80% of all cases. Therefore, it is crucial to estimate the

N A

B

C

D E

F G

c

1-c

b 1-b

p 1-p

N(1-c)bp

Nc

N(1-c)b(1-p)

N(1-c)(1-b)

N(1-c)

N(1-c)b

Preferred Situation

Fig. 4. Parameters for evaluating an enforcement system.

0 1

Decision

0

1

TrueState

λ00 λ01

λ10 λ11

ThresholdSelection

LPR DecisionIs

η

Accept/RejectT

0 1

Decision

0

1

TrueState

λ00 λ01

λ10 λ11

0 1

Decision

0

1

TrueState

λ00 λ01

λ10 λ11

ThresholdSelection

ThresholdSelection

LPR DecisionIs

η

Accept/RejectT

Fig. 5. Decision system. I is the input photograph.

confidence level appropriately [10]. This is done using the outputs of the LPR software to

train a neural network for a large set of annotated images, setting the network target to 1 or 0 for each image, depending on whether the plate is correct or not. It is known [11], for neural networks with a single output non-linearity taking values in the interval [0,1], that the output approximates the posterior probability in (2). A graph illustrating this result empirically, based on approximately 26,000 photographs, can be seen in Figure 6.

V. EVALUATION RESULTS The new ALPR has been evaluated using a sample of

several thousand real enforcement images. The results are shown in Table I. There was a significant improvement in

relation to the previous LPR system (see Table II). Nevertheless, it should be noted that the images used for testing the new ALPR were obtained in a single toll booth, where the prototype of the system was first implemented.

Furthermore, the results in Table II are based on a much larger sample from several toll booths all over the country. Therefore, the results presented for the new system are preliminary and a direct comparison with the ones in Table II should be made with some caution. Nevertheless, there are several points worth considering. The new system has a constant recognition rate for day and night time images, which is encouraging because it shows that the dependence on the lighting conditions has been greatly reduced. This was a problem in the previous design, as shown in the over- and under-exposed photographs in Figure 7.

Another factor is the number of illegible images. Even in ideal lighting conditions, the images shown in Figure 7 would be considered illegible since there is no vehicle license plate present, only that of the trailer (and in one case, none at all). However, in the new design, with frontal capture using pulsed IR, the image contrast is ideal for recognition, and in case the rear plate is not present, the front plate can still be used (see Figure 8). This contributes towards lowering the percentage of illegible images and thus to an improvement on the global LPR performance stemming from an increase in parameter b (the percentage of legible images), as well as from the overall image quality due to the new capture setup.

Also, there is now the possibility of automatically processing the photographs whose confidence level is

TABLE I RECOGNITION RATES FOR THE ALPR SYSTEM

Total Day Night Front Rear

Nº of images 9144 2034 505 4483 4661 Recognition

Rate 94% 94% 95% 95% 94%

TABLE II RECOGNITION RATES FOR PREVIOUS LPR SYSTEM

Total Day Night

Nº of images 28688 6029 1369 Recognition

Rate 68% 70% 57%

a)

b)

c)

Fig. 7. Unrecognized photographs due to a) night under-exposure, b) daylight over-exposure and c) hidden plate.

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Cor

rect

Rec

ogni

tion

Rat

e

Confidence Level

Fig. 6. Recognition rate vs. the estimated confidence level, based on approximately 26,000 photographs. The straight line is the desirable result.

sufficiently high. Moreover, the double capture scheme affords a degree of redundancy that can be taken advantage of. This corresponds to an increase in parameter p.

VI. CONCLUSION An approach has been proposed for evaluating video-

based enforcement systems using LPR technology. The case of toll enforcement on Portuguese motorways covered by the Via Verde, using the new ALPR system, has been described.

The impact of the double capture setup, using both the IR and visible spectra, has been discussed. Additionally, a set of quantitative measures for performance evaluation was introduced and the need for using a properly estimated confidence level in the decision process was established.

Preliminary results, based on a sample of approximately 9,000 enforcement photographs, indicate that the new ALPR design leads to a better overall enforcement performance, compared to the solution previously being used in the Via Verde system.

ACKNOWLEDGMENT The authors would like to thank Bruno Basílio and

Ricardo Cruz for valuable inputs.

REFERENCES [1] Michael Brosnan, Philipp Jordi, VERA 2: Video

Enforcement for Road Authorities, 2004 [2] Simon Benz, Bernhard Oehry, Systems for automatic

traffic enforcement using digital cameras, 10th World Congress and Exhibition on Intelligent Transport Systems and Services, Madrid, November 2003

[3] Larry O’Gorman, Pattern Recognition in Traffic Engineering, IAPR Newsletter, Vol. 27, n. 3, 2005

[4] Y. Hofman, License Plate Recognition-A Tutorial, http://www.licenseplaterecognition.com, Accessed 27 February 2006

[5] A. Rahman, A. Radmanesh, A Real Time Vehicle’s License Plate Recognition, Proceedings of the IEEE on Advanced Video and Signal Based Surveillance, 2003

[6] T. Duc Duan, T. Hong Du, T. Vinh Phuoc, N. Viet Hoang, Building an Automatic Vehicle License-Plate Recognition System, Int. Conf. in Computer Science, RIVF’05, Vietnam, February 2005

[7] L. Osório, M. Barata, C. Gonçalves, P. Araújo, A. Abrantes, P. Jorge, J. Sales Gomes, G. Jacquet, A. Amador, Interoperability among ITS Systems with ITS-Ibus Framework, in Emerging Solutions for Future Manufacturing Systems, editor L. Camarinha-Matos, Springer Science, 2005.

[8] M. Constant, CCTV Information – ANPR, http://www.cctv-information.com, Accessed 27 February 2006

[9] Richard O. Duda, Peter E. Hart, David G. Stork, Pattern Classification, Wiley, 2001

[10] Gethin Williams and Steve Renals, Confidence measures from local posterior probability estimates, Computer Speech and Language, 13. pp. 395-411

[11] M. D. Richard and R. P. Lippmann, “Neural network classifiers estimate Bayesian a posteriori probabilities,” Neural Comput., vol. 3, no. 4, pp.461–483, 1991.

Fig. 8. Front IR capture, where the contrast is optimized for recognition, and not for providing context (vehicle model, surroundings).