REETS D3.1 Definition of Toolbox KPIs V3.1

REETS D3.1 Definition of Toolbox KPIs V3.1.docx

i

REETS-TEN

Activi ty 3: Key Performance Indicators

E3: Key Performance Indicators

D3.1 Definit ion of KPIs Def in i t i on o f Too lbox KP Is f o r EETS


ii

Date Version Description Document

Status Responsible

17.12.2013 0.3 Initial draft for WP3 participant comments Draft JM / MH / MH / TA

13.01.2014 0.4 Revised draft incorporating WP3 partici-pant comments Draft JM / MH

24.01.2014 0.5 Revised draft incorporating amendments agreed at WP3 meeting Draft MH

0.6 Not Used

25.03.2014 0.7 Revised draft incorporating amendments agreed at WP3 meeting Draft MH

02.04.2014 0.8 Revised draft incorporating additional amendments and cross-referencing. Draft MH

24.04.2014 0.9 Revised draft incorporating comments on V0.8 Draft MH

23.05.2014 1.0 Revised draft incorporating adjustments for consistency with D3.2 Draft MH

16.06.2014 2.0 Revised draft to reflect changes to D3.2 Initial release MH

26.06.2014 2.1 Formatting corrections Updated re-lease MH

16.07.2014 3.0 Final corrections and glossary Release for SC JM

29.10.2014 3.1 Typographical corrections Update to released version

MH


iii

The sole responsibility of this publication lies with the author. The European Union is not responsible for any use that may be made of the information con-

tained therein.


iv

Information on document administration:

How to complete Document revision history table:

Date: Insert the date of the document revision in the following format: dd.mm.yyyy Version: Insert the current version-number of the document (see explanation below) Description: Insert a brief description of the document changes Document Status: Insert either Draft or Released (see explanation below) Responsible: Insert your name (the name of the person responsible for document revision)

Information on Document Version and Status administration:

Document Status:

Draft: the document is being processed and has not been finished yet. Released: the document has been checked and released; it can only be modified if the version

number is updated. Versions: Take place in two stages. Checked and released documents receive the next higher

integral version number.

0.1, 0.2, etc.: not released version, with the status "Draft" 1.0: first checked and released version, with the status "Released" 1.1, 1.2, etc.: Revision of first released version, with the status "Draft". 2.0: second checked and released version, with the status "Released".

Creation of the document name (Nomenclature):

REETS TEN_D.x.x_Name of the deliverable_v0.1_2013-09-01_comments

REETS TEN_ D x.x_ Name of the deliverable_ v0.1_ 2013-09-01_ comments

Project name

Deliverable Nr. e.g. D 1.1 e.g. D 2.1 e.g. D 4.1

Use Keywords e.g. contractual framework e.g. certification-approaches and practices e.g. Def. BOI and tests

Version Number

Date format: yyyy-mm-dd

Any infor-mation you wish to add (Responsi-ble person, etc.).


v

Content 1 Introduction ............................................................................................................................. 1

1.1 Scope, objective and Structure of the Document ............................................... 1 1.2 Purpose of KPIs ...................................................................................................... 2 1.3 Application of ISO/TS 17444-1 Electronic Fee Collection – Charging

Performance – Part 1 Metrics ................................................................................ 4 1.4 Coverage of Business Processes ......................................................................... 4

2 Methodology and Selection Criteria ..................................................................................... 5 3 Abbreviations .......................................................................................................................... 5 4 KPI toolbox .............................................................................................................................. 6

4.1 Goal .......................................................................................................................... 6 4.2 Approach ................................................................................................................. 6 4.3 Scope ....................................................................................................................... 6 4.4 KPIs .......................................................................................................................... 7 4.4.1 Technology independent KPIs: ................................................................................. 7 4.4.2 DSRC-related KPI: ................................................................................................... 7 4.4.3 KPIs for GNSS/CN systems: .................................................................................... 7 4.5 Usage of KPIs ......................................................................................................... 8 4.6 Performance Mitigation Methods .......................................................................... 8

5 Technology Independent KPIs .............................................................................................. 9 5.1 EETS Interface Service Quality (Timeliness) ....................................................... 9 5.1.1 EETS Interface Service Quality (Informaton Transmission Timeliness) ................... 9 5.1.2 EETS Interface Service Quality (Information Receipt Timeliness) ......................... 11 5.2 EETS Interface Service Quality (Correctness) ................................................... 13 5.3 Payment Settlement Delay ................................................................................... 15 5.4 Correctness of OBE Personalisation Data ......................................................... 16 5.5 Service User Claim Response ............................................................................. 18

6 DSRC-related KPIs ................................................................................................................ 19 6.1 OBE DSRC Transaction Quality .......................................................................... 19

7 KPIs for GNSS/CN Systems ................................................................................................. 22 7.1 Detection Performance ........................................................................................ 22 7.1.1 Missed Recognition Events .................................................................................... 22 7.1.2 Data Provision for Detection of Charged Objects ................................................... 24 7.1.3 Accuracy of Usage Parameters .............................................................................. 26 7.2 False Positive Events ........................................................................................... 27 7.3 DSRC Compliance Checking Communication Performance ............................ 28

Annex A Status of KPI development..................................................................................... 30 Annex B Existing EETS KPIs..................................................................................................... 34 Annex C Glossary........................................................................................................................ 49

REETS D3.1 Definition of Toolbox KPIs V3.1.docx Page 1 of 51

1 Introduction 1.1 Scope, objective and Structure of the Document The scope of this deliverable is to document the outcome of sub-activity 3.1 within the REETS pro-ject. Specifically, the deliverable provides a ‘toolbox’ of KPI definitions which reflect the needs of both Toll Chargers and EETS Providers for the measurement of EETS service performance. The objective of the document is to define KPIs which can be used within contracts between Toll Chargers and EETS Providers, in order to monitor the performance within the main business pro-cesses required for EETS operation. The remaining chapters of the deliverable provide:

- a description of the methodology used and the criteria used to select the KPIs included in the

Toolbox.

- a list of definitions for abbreviations used

- a list of the KPIs included in the Toolbox

- descriptions for KPIs according to their applicability to either DSRC or GNSS based Toll do-

mains or both technology options.

Information is included in Annex A, describing the current status of development of KPIs for EETS within the Toll Domains of the Toll Chargers participating in the REETS project. In Annex B, a summary is included of the specific EETS KPIs already developed and published by the participating Toll Chargers at the start of the REETS project.


1.2 Purpose of KPIs EETS establishes a long-term partnership between Toll Chargers (TCs) and EETS Providers (SPs) which should have the goal of providing the best possible service for users and ensuring correct Toll payment. To permanently improve the quality of the service, TCs and SPs need to co-operate to develop Key Performance Indicators (KPIs) in order to monitor how the service is delivered by either party in ac-cordance with the expectations of the other party, as well as those of users. TCs expect: to be paid the due amount of toll revenue in a correct and secure way, to minimise oper-ational problems linked to the service and to keep trust in their relations with SPs. They also expect to combat fraud. SPs are willing to offer the best service to their customers by minimising problems in toll collection. This includes having reliable equipment, and good quality data and information exchange with TCs. A TC is expecting 100% compliance rate for all service elements provided by SP and vice versa. In practical operation however, a 100% compliance level is never achievable. The acceptable and achievable performance level therefore has to be negotiated between TC and SP for each service element depending on the specific requirements of the toll context. The KPI is the method to meas-ure performance, in order to monitor the quality of service provided. The overall aim of both parties is therefore to achieve the continuous improvement of the service quality. The aim of improving service quality is in turn to, improve users’ satisfaction and confidence, to reduce wasted effort and expense, minimise risks and safeguard revenue. From the perspective of service providers, awarding penal-ties [linked to reduced levels of performance] is seen as a consequence arising when continuous improvement is not taking place, and the negotiated performance level is not achieved [by one of the parties]. Where applicable, SPs need billing details on a regular (at least daily) basis to be able to manage their financial risk exposure and give their customer the regulatory details for their toll in-voices. Therefore, the following aspects are, among others, of key importance: (Note that the implementation of the required business processes may vary between Toll Domains, and therefore not all of the fol-lowing aspects are / will be applicable in all Toll Domains).

• SPs expect to receive correct data from the TC to be able to verify the Toll invoices.

• Where applicable, SPs need billing details on a regular (at least daily) basis to be able to

manage their financial risk exposure.

• TCs need to provide relevant information on procedures to enable the SP to help their

customers in understanding the toll scheme and if necessary to address relevant claims if

the correct toll is not applied.

• EETS-Providers require a timely and accurate management of black lists by the TC.

• SPs contribute in fighting against fraud.

• TCs expect from the EETS Providers that the OBE are working and are maintained in a

sufficient way

• SPs have to secure transparency about the toll capturing process for the TCs


• SPs have to secure a correct and timely payment of the collected toll

These requirements are further described as business processes (see REETS work package 4 De-liverable D4.1). The operation of each process, involves specific performance requirements which are relevant to either the SP or the TC or both according to the process being performed. Service quality is of key importance to users. The outcomes of business processes performed by the SP and / or the TC impact directly on the quality of service for users [of the service]. Service users expect for example:

• OBE to be reliable

• the service provided by the SP to be European-wide

• a good level of service on Toll Gates (dedicated lanes, priority of passage compared to

other users paying with other means of payment, smooth transit in lanes),

• to have an easy, fast and standard way to register where necessary (for tax payment or

for getting discounts),

• that there will be a practical and secure fall-back procedure when the OBE is not working

and not to be penalized incorrectly and

• to receive information enabling them to check their toll invoice and expenses.

However in this document, we consider only performance indicators relating to businesses processes in the SP – TC relationship, and not measurements for service quality as perceived by service users. Overall, KPI’s should be considered primarily as a measurement tool for assuring and improving the service and stakeholders’ expectations as mentioned above. KPI’s should use available data without creating complexity to generate them and keep statistical significance in their interpretation. It should also be taken into consideration, that the SP may have additional information which could be used for improving toll determination. Such as, for example, positioning information that could be used to find missing free flow detections; or, information in other Toll Domains as a comparison to help explaining non-conformity detected through KPI’s. SP’s are also required to monitor their own system, and would prefer to use the same toolbox of KPIs in all toll domains. In SPs view, their internal monitoring should be treated as a valid reference for the evaluation of the SP’s [equipment] performance and used as a comparator with the TC’s monitoring data. This would lead to identification of descrepancies and the possibility to make EETS perfor-mance more consistent at European level. To ensure trust in an SP’s internal monitoring information, the SP monitoring system should be subject to audit from a trusted third party.1 Overall, Work Package 3 of the REETS project has defined a set of KPIs that can be used in the dif-ferent Toll Domains across Europe without the introduction of individual complex measurement

1 It should also be noted that all equipment used as Interoperability Constituents are presumed to be certified and tested for suitability for use before entering in operation and, in the case of OBE, are used in many different Toll Domains. Therefore non-compliance would also provide information to certification procedures and relevant notified bodies. A major failure on the EETS Provider side [to comply with its obligations regarding the service] would have European wide consequences especial-ly in the event that such circumstances led to the disqualification of an EETS Provider.


methods in individual member states or Toll Domains, whilst still addressing the needs of TCs and SPs. 1.3 Application of ISO/TS 17444-1 Electronic Fee Collection – Charging Performance – Part 1 Metrics The international standard ISO/TS 17444 “Electronic Fee Collection – Charging Performance – Part 1 Metrics”, is a ‘toolbox’ standard. It describes a set of metrics which may be used in order to meas-ure the performance of electronic fee collection (EFC) systems in terms of the level of errors associ-ated with charging computation. The actual values that define the required acceptable performance are not covered in ISO/TS 17444-1. The charging performance metrics defined in ISO/TS17444-1 are technology and charging scheme independent. In accordance with ISO 17573, and with the associated interface standards ISO/TS 17575-1 and ISO 12855, metrics defined in ISO/TS17444-1 are related to the six levels below:

- End to End Charging Performance Metrics, defined at a level which determines the overall charging performance of a toll scheme across all interfaces on the overall systems level for a group of users

- User Account Metrics, are related to the number of User Complaints related to Charging re-ceived by the Toll Service Provider

- Payment Claim Metrics, as transmitted from the Toll Charger to the Service Provider (ISO 12855:2012)

- Billing Details Metrics, as transmitted from the Toll Charger to the Service Provider (ISO 12855:2012)

- Toll Declaration Metrics, as transmitted from the Service Provider to the Toll Charger for au-tonomous systems (ISO 12855:2012)

- Charge Report Metrics, as transmitted from the Front End to the Service Provider’s Back End (ISO/ TS 17575 -1:2012)

Some of the metrics defined in ISO/TS17444-1 are part of the KPIs toolbox proposed in the delivera-ble D3.1. The KPIs toolbox proposed in the deliverable D3.1 utilises of some of the KPI’s defined in ISO/TS17444-1, and also propose other KPIs which are not considered in ISO/TS17444-1 but which are based on the knowledge and experience of the SPs and TCs involved in the REETS project. Note, not all of the metrics defined in ISO/TS17444-1 were selected for inclusion and definition in this REETS Deliverable D3.1. The reason for this is that the focus for this KPI toolbox, was to clarify which KPIs were commonly used or required for the Toll Domains rSPresented in the REETS project. 1.4 Coverage of Business Processes Overall the KPIs included in the toolbox are those which are applicable to, and required in order to monitor the performance of, the main business processes which need to be established between SP and TC. A complete analysis of the main business processes can be found in REETS Deliverable D4.1


2 Methodology and Selection Criteria The Methodology used to develop the toolbox definitions for EETS KPIs was as follows:

1) The participating Toll Chargers were asked to provide information about the KPIs for

which they have already developed and in some cases published KPI definitions in thier

national schemes for the purposes of EETS.

2) The information provided was grouped and aligned where possible according to the defini-

tions which were previously developed by the Stockholm group and published within the

report ’EETS Quality Monitoring and Assurance – Report on KPI Definition’.

3) The grouped KPIs were reviewed within the drafting group and a list of generic KPIs was

identified in order to provide the basis to be able generate generic descriptions for the ma-

jority of KPIs identified.

4) Generic descriptions were drafted for the KPIs identified in the list making reference to the

metrics defined in EN ISO 17444. Note that it was decided that information about the data

to be monitored and the calculation method would be included in the REETS Deliverable

D3.2 covering KPI measurement rather than in this Deliverable D3.1. However, the

descriptions of the KPIs in D3.1 were reviewed with the intent of ensuring that practical

and consistent methods of measurement can be defined.

5) The draft list and descriptions were circulated for comment to all participants of the work-

ing group on KPIs and a revised version of the list and descriptions was prepared.

3 Abbreviations ADU Application Data Unit CN Cellular Network DSRC Dedicated Short Range Communications SP [EETS] Service Provider GNSS Global Navigation Satellite System KPI Key Performance Indicator LAC Location Augmentation Communication OBE OnBoard Equipment RSE Roadside Equipment TC Toll Charger WP Work Package Further explanation of these and other terms can be found in the REETS Glossary document.


4 KPI toolbox 4.1 Goal The KPI toolbox which follows in this document consists of KPIs that are recommended for use in the relationship between TCs and SPs. The overall goal of these KPIs being to provide definitions that can be adopted in any such relationships thereby facilitating increased consistency and improvement of EETS Service Quality across all Toll Domains. The aim of the toolbox is to reduce the need for customization of KPIs for individual Toll Domains, thereby providing the benefit of improved consistency across Toll Domains through greater harmoni-sation of definitions, reduced effort for Toll Chargers in the definition of KPIs and corresponding measurement methods, and reduced cost and complexity of implementation of monitoring systems for EETS providers. 4.2 Approach The KPIs listed herein are the result of work carried out within Work Package 3. The Work Package participants have carried out a review of KPI definitions under development and in some cases al-ready published by the Toll Chargers participating in REETS. This means that the Toolbox has been both driven and validated by practical considerations of implementation of monitoring methods as well as by the desired outcomes. 4.3 Scope This document focusses on harmonisation of KPI definitions. Calculation methods, data require-ments and possible methods of measurement are considered in Deliverable 3.2 of Work Package 3. The list does not contain KPIs that may be used locally by individual TCs for purposes other than in relation to EETS services. The list only consists of KPIs for business processes forming part of EETS service operations. Re-quirements relating to the certification of SPs or to testing of interoperability constituents are not con-sidered as KPIs in this context.


4.4 KPIs The recommended KPIs included in the Toolbox, are listed below.

4.4.1 Technology independent KPIs: These indicators will be used to monitor and improve:

• the quality of the interface supporting the business processes, with regard to the timeliness

and correctness of the files exchanged between SP and TC (for example: toll declarations,

exception lists, etc.,..)

• payment delay;

• correctness of the OBE personalisation data; and

• performance of TCs in supporting SPs inquiry process on user claims related to toll state-

ments.

The technology independent KPIs are :

• EETS Interface Service Quality : Timeliness

o Timeliness of provision

o Timeliness of processing

• EETS Interface Service Quality : Correctness

• Payment settlement delay

• Correctness of OBE Personalisation Data

• Service User Claim Response

4.4.2 DSRC-related KPI:

One indicator (OBE Transaction Quality) is included specifically in order to monitor and improve the DSRC transaction quality between OBEs from a specific SP within a toll domain. This KPI can be applied in multilane free-flow systems as well as in DSRC tolling systems with barriers.

4.4.3 KPIs for GNSS/CN systems: Five indicators are included specifically in order to monitor and improve aspects of performance in GNSS/CN based toll domains:

• the quality of a service with respect to the capture of all Chargeable Events of vehicles (within

the responsibility of the SP in a Toll Domain);

• same as above but for all Segments that are passed in a Toll Domain;

• the accuracy of ‘measured usage’ parameters for charging in a Toll Domain;


• the service with respect to the declaration of false positive Events of vehicles (within the re-

sponsibility of the SP in a Toll Domain); and

• the DSRC performance of the Compliance Checking Communication function of the OBE.

The KPIs included in the Toolbox specifically for GNSS/CN are:

• Missed Recognition Events

• Data provision for Detection of Charged Objects

• Accuracy of usage parameters

• False Positive Events

• DSRC Compliance Checking Communication Performance

4.5 Usage of KPIs The KPIs are formulated with the intention that they are primarily to be used to maintain and improve overall EETS service quality by influencing the performance of SPs and TCs within the relevant busi-ness processes. Whether these KPIs are suitable to be used in a contractual agreement between TC and SP in com-bination with penalties has not been considered in the process of assembling the toolbox. It should also be noted that not all KPIs may be applicable in all toll domains – specifically some of the technology-specific KPIs will only apply where the particular technology (DSRC or GNSS) is used [for charging purposes]. The toolbox provides a ‘menu’ of KPIs that are applicable for use in EETS service monitoring and the suitability of each KPI should be evaluated for each Toll Domain. 4.6 Performance Mitigation Methods Actions that could positively influence the performance (e.g. in case of not fulfilling the performance requirements) are documented as ideas for mitigation for each KPI. Particularities of toll domains need to be taken into account when considering any of the mitigating actions. It is recommended to establish a regular process for review of the KPI values by TC and EP. A core element of this process is to define mitigating actions when target values are not met. Further definition of the process, when mitigation actions must be taken and what could happen if either party does not take any mitigation action, is not part of this document.


5 Technology Independent KPIs

5.1 EETS Interface Service Quality (Timeliness) Looking the exchange of data at the back-office-interface the timeliness of data exchange can be distinguished based on two perspectives:

• Perspective of the recipient who wants to have specific data available at a specific time

• Perspective of the sender who wants that the recipient takes the responsibility for processing

of specific data at a specific time

The following two KPIs (5.1.1 and 5.1.2) are created to measure this.

5.1.1 EETS Interface Service Quality (Informaton Transmission Time-liness)

KPI EETS Interface Service Timeliness (Information Transmission Timeli-ness)

Aim of the KPI

The aim of this KPI is to monitor the timeliness of the information packages communicated via an interface as an indicator of service quality provided on the EETS Interface. Note : Together with the KPI described in section 5.2, the overall aim of the KPI is also to provide a harmonised approach to the monitoring of EETS in-terface performance by introducing ‘timeliness’ and ‘correctness’ as parame-ters to measure the performance of the interface rather than measuring ‘availability’ as link on / off events. The KPI applies to the measurement of the performance from the perspective of the recipient (either TC or SP) of transmitted information packages (i.e. the KPI applies to processes operated by the sender of information).

Description of process

The process to be monitored by this KPI is the exchange of information using the defined interface between SP and TC in order to support the required business processes for EETS in a specific Toll Domain.

Sender Recipient

InformationPackage

InformationPackage

FeedbackData Processing

Time MeasuringPoint for Provisionof Information(KPI 5.1.1)

Time MeasuringPoint for Processing of Information (KPI 5.1.2)

Back-‐Office-‐Interface


Specifically, this KPI applies to transmission processes initiated by the send-er and which are mainly important to the sender.

What will be monitored?

The KPI should monitor the quality of the back office interface in supporting the business processes required for EETS within the specific Toll Domain. Specifically this KPI will monitor the timeliness of the provision of information across the interface according to the requirements agreed between the SP and TC in the specific Toll Domain to support the identified business pro-cesses (see REETS Deliverable D4.1).

Definition of KPI

The KPI is defined as B/A (expressed as a percentage) where B = the quantity of information exchanges which are verified as received in accordance with the timing requirements agreed between the SP and TC for a specific interface and where A = the total quantity of information exchanges which should have been pro-vided by the sender within the measurement period. The method for measuring the quantity of information exchanges included within the KPI calculation shall be determined by the SP and TC. Guidelines for this will be described in REETS deliverable D3.2 The information exchange types shall include for example those associated with the exchange of:

1) Toll Context Data 2) Toll Declarations 3) Billing details 4) Exception lists 5) User Details 6) Payment Claims

Mitigation ideas

• improve / enforce service level agreement for communication link

e.g. availability of communications link.

• Improve delay of back-office processing of information required for

transmission

• Modify packet size of transmitted information


5.1.2 EETS Interface Service Quality (Information Receipt Timeliness)

KPI EETS Interface Service Quality (Information Receipt Timeliness)

Aim of the KPI

The aim of this KPI is to monitor the timeliness of information packages communicated via an interface as an indicator of service quality provided on the EETS Interface. In this case the information has to be received and ac-cepted (hand shake procedure) by the recipient of the information. Note : Together with the KPI described in section 5.2, the overall aim of the KPI is also to provide a harmonised approach to the monitoring of EETS in-terface performance by introducing ‘timeliness’ and ‘correctness’ as parame-ters to measure the performance of the interface rather than measuring ‘availability’ as link on / off events. The KPI applies to the measurement of the performance from the perspective of the sender (either TC or SP) of transmitted information packages (i.e. the KPI applies to processes operated by the receiver of information).


The process to be monitored by this KPI is the exchange of information using the defined interface between SP and TC in order to support the required business processes for EETS in a specific Toll Domain. Specifically, this KPI applies to transmission processes which are mainly im-portant for the sender.


The KPI should monitor the quality of the back office interface in supporting the business processes required for EETS within the specific Toll Domain. Specifically this KPI will monitor the timeliness of the processing of infor-mation provided across the interface according to the requirements agreed between the SP and TC in the specific Toll Domain to support the identified business processes (see REETS Deliverable D4.1).

Definition of KPI

The KPI is defined as B/A (expressed as a percentage) where B = the quantity of information exchanges which are acknowledged as re-ceived in accordance with the timing requirements agreed between the SP and TC and where A = the total quantity of information exchanges which are provided by the sender. The method for measuring the quantity of information exchanges included within the KPI calculation shall be determined by the SP and TC. Guidelines for this will be described in REETS deliverable D3.2 The information exchange types shall include for example those associated with the exchange of:

1) Toll Context Data 2) Toll Declarations 3) Billing details 4) Exception lists 5) User Details 6) Payment Claims


Mitigation ideas

• improve / enforce service level agreement for communication link

e.g. availability of communications link.

• Improve delay of back-office processing of information required for

transmission

• Modify packet size of transmitted information


5.2 EETS Interface Service Quality (Correctness)

KPI EETS Interface Service Correctness

Aim of the KPI

The aim of this KPI is to monitor the correctness of the information packages communicated via the interface as an indicator of service quality provided on the EETS Interface. Note: Together with preceeding KPI, the overall aim of the KPI is also to pro-vide a harmonised approach to the monitoring of EETS interface perfor-mance by introducing ‘timeliness’ and ‘correctness’ as quality parameters to measure the performance of the interface rather than measuring ‘availability’ as link on / off events. The KPI applies to the measurement of the performance from the perspective of the recipient (either TC or SP) of the transmission of information packages.


The process to be monitored by this KPI is the exchange of information using the defined interface between SP and TC in order to support the required business processes for EETS in a specific Toll Domain. This KPI is therefore bi-directional i.e. it is applied to information sent by ei-ther SP or TC according to the relevant elements of the applicable business processes.


The KPI should monitor the quality of the back office interface in supporting the business processes required for EETS within the specific Toll Domain. Specifically this KPI will monitor the correctness of the provision of infor-mation across the interface according to the requirements agreed between the SP and TC in the specific Toll Domain to support the identified business processes (see REETS Deliverable D4.1). The metrics defined in ISO/TS 17444-1 may be applicable to the measure-ment of this KPI. For example, CM-BD-7 Rejected Billing Details Rate.

Definition of KPI

The KPI is defined as B/A (expressed as a percentage) where: B = the quantity of correct and complete information packages which are ver-ified as received in accordance with the quality requirements for correctness agreed between the SP and TC and where A = the total quantity of information packages which are transmitted using the interface following a corresponding request for information from the recip-ient. The method for measuring the quantity of information packages within the KPI shall be determined by the SP and TC. Guidelines for this will be de-scribed in REETS deliverable D3.2 The information exchange types shall include for example those associated with the exchange of:

1) Toll Context Data 2) Toll Declarations 3) Billing details


4) Exception lists 5) User Details 6) Payment Claims

Where applicable, the correctness of information packages shall take into account the following parameters:

a) Information packages: i. Missing packages: packages not received or gaps in the se-

quence of packages, absence of acknowledgment... ii. Corrupted packages or packages badly addressed: packages

cannot be opened and exploited or are sent to the wrong ad-dress

b) Records compliance: i. Corrupted records: record within a package not compliant with

the protocol, or not readable, or not in the right file… ii. Missing records: gaps in the sequence of received records or

absence of functional acknowledgement for received records where needed…

c) Data correctness: i. Incorrect format ii. Incorrect value iii. Incorrect parameter

Mitigation ideas

o utilise error checking in transmission process

o Implement technical safeguards within interface design

and security poilicies

o intensify interface tests (for next interface update)

o train operations team


5.3 Payment Settlement Delay

KPI Payment settlement delay

Aim of the KPI

The aim of this KPI to provide an effective measurement of the performance of SPs in transferring Toll Payments to the Toll Charger.


The KPI is concerned with the specific process of the payment settlement by the SP to the TC, following the issue of required documentation (for example invoice, billing details or toll declarations as agreed in the specific Toll Do-main) by the TC to the SP.


This KPI will monitor the time taken to transfer toll payments to the TC by the SPs and specifically any delays arising within this process.

Definition of KPI

• The KPI focuses on the payments which have been settled within the measurement period.

• This KPI is equal to the number of days of delay occurring for the payment settlement, either punctually or during a monitoring period. Alternatively it can be expressed as a percentage. The exact method of calculation of the KPI will be further described in REETS Deliverable D3.2 ‚KPI Measurment Methods’.

Mitigation ideas

• - Process improvements by SP for example the usage of workflow systems

for approval and release of payments

• - Provision of correct and complete information by TC


5.4 Correctness of OBE Personalisation Data KPI Correctness of OBE Personalisation Data

Aim of the KPI

The aim of the KPI is to assure the quality of data within the process of SPs personalising on-board equipment (based on information provided by users).


The process being monitored is the correct personalisation of the on-board unit by the SP (based on information provided by users) to ensure that the correct data (for example emission class) is stored in the OBE, thereby en-suring correct charging and enforcement operations result. Note that the requirement for users to register in each Toll Domain is not mandatory and requirements for registration vary according to the individual requirements of each Toll Domain.


To improve the performance of the personalisation process, the correctness of data held in OBE shall be monitored by SPs using this KPI. The data that are required to be correctly personalised by the SP during the personalisation process could include for example:

• the licence plate (incl. country of registration),

• vehicle class (UNECE vehicle class) and number of axles of the vehi-

cle

• vehicle EURO emission class (if required)

• vehicle weights

• any other required data

This means that the SP must carry out checks on the information and data provided by the user, the registered user and vehicle data and on the data stored in the onboard equipment (if agreed in the contract between SP and TC). The TC should inspect the SP’s operational processes in order to verify the correct interpretation, collection and storage of data provided by users. In the course of a regular inspection, it is determined how many of the OBEs issued by the SP and registered on the TC’s network deviate in one of these features from the features actually identified in the inspection process. The SP must send the TC information of proof on any deviations found. If the personalisation of the OBE matches information of proof, then the deviation was caused by the misbehaviour of the User. If a deviation between the per-sonalisation of the On-Board Unit and the information of proof is established during the inspection, then the deviation was caused by the misbehaviour of the SP. The basis for the distinction of responsibilities is EU Decision 2009/750 art.4 item 5 and art.9 item 2. The metric CM-UA-5 defined in ISO/TS 17444-1 is directly applicable to the


measurement of this KPI.

Definition of KPI

• The KPI is defined as B/A (expressed as a percentage) where

• B = the number of incorrectly personalised items of On-Board Equipment

(based on data provided at personalisation by users) and where

• A = the total number of items of On-Board Equipment in the corresponding

sample.

Guidance on the selection of samples for measurement of this KPI and on the data to be considered as part of the determination of correct personalisa-tion is considered further in REETS deliverable D3.2.

•

Mitigation ideas

• Improved checking by SP during personalization process:

• User: The user is not declaring the correct data

• SP: The SP has erroneous data within his database


5.5 Service User Claim Response KPI Service User Claim Response

Aim of the KPI

The aim of the KPI is to measure the performance of the process of providing responses to Service Users. In particular, this KPI aims to measure the per-formance of TCs in supporting SPs inquiry process on user claims relating to Toll Statements.


The SP is the main contact for his users/customers in case of complaints on the invoice and the toll statement. The TC solely decides to accept or refuse the user complaint on applied toll charges. After first review by the SP, the SP is responsible for transmitting the user’s complaint with all necessary justifications to the TC for instruction and deci-sion. The KPI measures the conditions and quality of the process in respect to the agreed time frame between the SP and the TC for user claims instruc-tion.


Number of transmitted requests by the SP towards the TC in comparison to number of correctly and timely answered requests by the TC Metrics for this KPI are not defined in ISO/TS 17444-1.

Definition of KPI

The KPI is defined as follows: 1) A= Number of claims per month, B=Number of tolled km or number of

passenges in a DSRC operated tolling system: The Ratio A/B gives by comparison with other comparable toll domains, an indication on the difficulties felt by user to get the right toll charges. This could lead to improving the quality of the information delivered to users or the way the toll charges are determined for a certain category of users.

2) C= Number of accumulated days taken by the TC to answer to open claims, D= Number of open claims transmitted to the TC The Ratio C/D defines the average number of days taken by the TC to answer claims and is compared to the TC commitment set in the Con-tract.

3) E= Number of claims by the TC, F= Number of claims sent to the TC for which instruction is closed Ratio E/F gives an indication of the quality of the pre-processing made by the SP and the flexibility of the TC in accepting claims.

Mitigation ideas

• Improvements in SP reviewing claims before forwarding to TC

• Improvements in details relating to claim provided by SP to TC

• Note the above mitigations relate to SP processes and not to TC pro-

cesses but TC performance in responding to claims is largely dSPendant

on the information provided by the SP

• Temporarily or permanently increase resources in treating claims or train

resources at SP and/ or TC.

• Process improvement TC


6 DSRC-related KPIs 6.1 OBE DSRC Transaction Quality The following KPI shall be applicable in multi-lane free flow DSRC tolling systems, as well as in DSRC tolling systems with barriers, and in GNSS tolling systems for Compliance Checking Commu-nication if a statistically significant reference group is available. Note the SP Front-End may have additional real time information, using the GNSS/CN facility of the OBE, related to incorrect DSRC transactions which can be used to identify the origin of the malfunc-tion and to determine the right rate of toll. The bilateral agreement between the TC and the SP can define how to use these complementary data for toll determination process improvement KPI OBE DSRC Transaction Quality

Aim of the KPI

Monitor the transaction quality between the RSE and the DSRC interface of OBE from any SP. The KPI is therefore applicable to monitoring of SPs (OBE performance) by TCs.


The process being monitored by this KPI is the process of completing a DSRC transaction or Compliance Checking Communication (if applicable) between OBE and RSE. In ETC systems where DSRC communications between OBE and RSE are used to generate Toll Transactions, correctly completed DSRC transactions are necessary in order to correctly identify the vehicle and SP to which the corresponding billing details should be sent by the TC.


The KPI will monitor the performance of the DSRC transactions of a specific group (= e.g. all OBE of one SP) or type of OBE within and compared to a statistically representative group of reference OBE circulating in a specific Toll Domain. The comparison with a reference group can help to statistically eliminate the influence of Road Side Equipment errors and Service User errors occurring during the measurement period. The monitoring of this KPI will be done by calculating an overall DSRC Error Ratio (=KPI 1). This overall error rate shall be based on monitoring of the rate of occurrence of different types of transactions as defined in the following section. Sub-KPIs may be implemented for individual transaction types to provide more detailed monitoring. Metrics for this KPI are not defined in ISO/TS 17444-1 explicity at this level of detail although consequences may be covered by other metrics.

Definition of KPI

The KPI is defined as B/A (expressed as a percentage) where : B = the total number of Complete DSRC transactions recorded for the sam-ple group in the measurement period and A = the total number of Complete DSRC transactions recorded for the sam-ple group in the measurement period plus the total number of Incomplete and missing transactions which can be associated with the SP (note no transaction shall be counted twice).


The sample group is either the specific group (e.g. all OBE of one SP) or the statistically representative group of reference OBE. Parameter definitions:

• Complete transactions: those transactions for which data trans-mission was complete and error-free.

• Incomplete transactions: all DSRC transactions detected as in-complete / incorrect (for which the required communication cycle was not properly terminated, insofar as an incomplete transaction can be confirmed as relating to a specific OBE unit).

• Missing transactions (note : the following definitions may not be valid in every tolling context): • Reconstructed transactions : in the event of gaps

between two or sometimes also more toll transactions, missing toll transactions can be synthetically reproduced (retroactively calculated) following a plausibility check of the driving time between the toll sections preceding and following the gap. Such transactions can be used within the KPI calculation to determine the number of missing DSRC transactions.

• Missing transactions detected by enforcement equipment: a missing transaction of an OBE exists when a vehicle passing a stationary or mobile enforcement equipment without a complete or incomplete transaction has a valid contract with an SP, the OBE was not blocked and where applicable that the visual inspection by the Toll Charger showed that the OBE was mounted correctly.

• Videobased transactions defined as a Transaction

created (in back office systems) based on:

1. extraction of an registration number from a picture tak-

en in a Lane (RSE)

2. subsequent lookup of data about the service provider.

• Manually keyed in transactions (where allowed)

defined as a Transaction where the OBE Primary Ac-

count number (PAN) is keyed in manually.

Manually keyed in transactions are an indicator that no DSRC transaction occurred.

The reference group will be determined by the respective Toll Charger for his Toll Domain, whereby it is recommended that the reference group either con-sists of all OBE types that are actively operated in post-pay mode in the re-spective Toll Domain or at least of those OBE types that are mainly (with a high population) used in the Toll Domain to make the reference group suffi-ciently big and statistically representative. In certain Toll Domains a further division of the DSRC Error Ratio is neces-


sary as the effects of the above listed incomplete and missing transactions differ. Whilst e.g. incomplete, reconstructed, manually keyed and video based transactions do not lead to a loss of tolling income, the missing transactions detected by enforcement equipment do lead to a loss of income in the tolling system if they cannot be fully recovered by a degraded mode system (e.g. a video based recognition at every toll point). In addition to that, the effort for a Toll Charger to reconstruct types of incom-plete or missing transactions is different for the individual transaction types. All or some of the following individual sub-KPIs, will therefore be calculated in order to have the possibility of an individual and different assessment: The envisaged sub-KPIs are:

• Ratio of incomplete transactions (= KPI 1a)

• Ratio of reconstructed transactions (= KPI 1b)

• Ratio of missing transactions detected by enforcement

equipment (= KPI 1c)

• Ratio of videobased transactions (= KPI 1d)

• Ratio of manually keyed transactions (= KPI 1e)

Similar to the general DSRC Error Ratio, also those individual sub-KPIs will be calculated for the OBE issued by an SP as well as one for the reference OBE (reference group) and compared.

Mitigation ideas

In case one OBE type has an issue:

• Improved OBE development and test processes

• Improved OBE manufacturing process quality

In case most of the OBE types have an issue:

• Modify the RSE implementation

• Modify RSE software to improve detection


7 KPIs for GNSS/CN Systems The following KPI are specifically applicable to toll domains using GNSS/CN technology. A GNSS/CN technology based tolling system collects the toll more or less based on three process steps which can be executed by the SP or the TC. It is necessary to understand this process and the split of responsibility between SP and TC to understand the following KPIs.

7.1 Detection Performance

7.1.1 Missed Recognition Events Note: this KPI is applicable for discrete charging systems (see EN ISO 17444) Reference EN ISO/TS 17444: “TD – TSP Event Detection” (CM – TD – 4)

This KPI is based on the split of responsibility as follows:

• The SP is responsible for the collection of geo-data and other data which are required by the algorithm for detecting charged objects

• The SP is responsible for the detection of charged objects • The TC is responsible for the rating of toll declaration data based on the detected

chargeable events (vehicle passages at a chargeable event) and other usage data

Toll Collection Process of a GNSS/CN System

Collection ofgeo-‐data and otherrelevant data forthe detection ofcharged objects

Detection ofcharged objects via a specific softwareand maps/geo-‐information

Rating:Putting a price toeach detectedcharged objectbased on usageparameters (e.g. vehicle class)





Responsibility of Service Provider Responsibility of Toll Charger


In the German Toll Domain this KPI is used even though the rating process step is also executed by the SP. KPI Missed Recognition Events

Aim of the KPI

Measure the quality of a service with respect to capture all Chargeable Events (e.g. passages of segments, entry and exit of a zone) of vehicles within the responsibility of the SP on the charged road network of the Toll Domain.


A vehicle is passing a Charged Object (any object that is part of the toll context description that may be charged for its use under certain condi-tions e.g. Segments) and has to capture the passage and has to collect all data necessary to charge this passage.


Detection of all Chargeable Events of vehicles which are within the re-sponsibility of an SP performed at the charged network of the Toll Do-main. Impact factors of the Toll Charger (Erroneous toll context data – the er-roneous toll context data should be detected during the tests which have to be performed before going in production with the toll context data) and of the User (wrong mounting of the OBE and willing manipulation) are excluded..

Definition of KPI A: Chargeable Events detected for the usage of the tolled network divid-ed by B: the total number of Chargeable Events that should have been detected.

Mitigation ideas

• mount additional LAC

• improve detection algorithm

• modify the Toll Context Data by improving the definition of some

Charge Objects or their localization

Note: the determination of chargeable events is dependent on the correct definition and implementa-tion of the toll context data, the localization and the precise definition of Charging Objects, the pres-ence of LAC, algorithms for determining the chargeable events from the information collected by OBE, level of precision of the OBE. The level of precision of the OBE is also verified during the ac-creditation process. The toll context data need to take into account the average performances of all accredited OBE.


7.1.2 Data Provision for Detection of Charged Objects

Note: this KPI is applicable for discrete charging systems (see EN ISO 17444)

KPI 7.1.2 is based on the following split of responsibilities:

• The SP is responsible for the collection of geo-data and other data which are required by the algorithm for detecting charged objects

• The TC is responsible for the detection of charged objects • The TC is responsible for the rating of toll declaration data based on the detected

chargeable events (vehicle passages at a chargeable event) and other usage data In the ECOTAX Toll Domain the detection of charged objects is executed by Ecomouv’ on behalf of the TC. KPI Data Provision for Detection of Charged Objects

Aim of the KPI Measure the quality of a service with respect to capture all charged ob-jects that are passed by vehicles within the responsibility of the SP on the charged road network of the Toll Domain.


A vehicle is passing through a charged object (any area described in the toll context data) and has to capture the passage and has to collect all data according to the rules determined by the TC e.g. in the toll domain statement.


Detection of all charged objects (including the required data) passed by vehicles which are within the responsibility of an SP in the charged net-work of the Toll Domain. Impact factors of the Toll Charger (Erroneous toll context data – the er-roneous toll context data should be detected during the tests which have to be performed before going into production with the toll context data) and of the User (wrong mounting of the OBU and willing manipulation) are excluded.

Definition of KPI A: segments detected for the usage of the tolled network with correct and complete data (according to the requirement) divided by B: the total number of segemetns that should have been detected.

Mitigation ideas

• mount additional LAC


• modify the Toll Context Data by improving the definition of some

Charge Objects or their localization





Responsibility ofService Provider

Responsibility of Toll Charger


Note: the detection of charged objects is dependent on the correct definition and implementation of the toll context data, the presence of LAC and the level of precision of the OBE.


7.1.3 Accuracy of Usage Parameters

Note: this KPI is applicable for continuous charging systems (see EN ISO 17444) KPI Accuracy of Usage Parameters Aim of the KPI Measure the accuracy of the measured usage parameters. Description of process

A vehicle is passing through an area defined in the Toll Context Data and has to capture the charging parameters according to the rules de-termined by the TC e.g. in the toll domain statement.


The Accuracy of the measured charging parameter, representing the usage of the tolled network or area.

Definition of KPI A: Usage parameter effectively measured for a specific charge event (e.g. trip) divided by B: the usage parameter correctly measured for this specific charge event.

Mitigation ideas • mount additional LAC

• improve measurement algorithm

Note: the accuracy of measuring the usage parameters is dependent on the presence of LAC, algo-rithms for determining the usage parameters and the level of precision of the OBE.


7.2 False Positive Events Reference to EN ISO/TS 17444: “TD – TSP False Positive”, (CM-TD-5) KPI False Positive Events

Aim of the KPI Measure the quality of a service with respect to the declaration of False Positive Events of vehicles within the responsibility of the SP.


A vehicle is passing a road section that is not part of the charged net-work and the SP should not declare this passage as Chargeable Event to the Toll Charger of this Toll Domain.


The declaration of False Positive Events.

Definition of KPI A: False Positive Events declared to the Toll Charger (tolled network was not used) divided by B: the total number of correctly dedected GNSS Chargeable Events of the SP.

Mitigation ideas • mount additional LAC


Note: the determination of toll events are dependent on the correct definition and implementation of the toll context data, the presence of LAC, algorithms for determining the chargeable events from the information collected by the OBE’s, level of precision of the OBE.


7.3 DSRC Compliance Checking Communication Performance The following KPI measures the performance of the DSRC compliance checking communication (CCC) of the OBE. This KPI was created in addition to OBE (Toll Charging) DSRC Transaction Quali-ty KPI (section 6.1) because it is not possible within the current system setup of the German toll do-main, to establish the same approach as that specified in section 6.1 (based on a significant refer-ence group.. KPI DSRC CCC Performance

Aim of the KPI Measure the performance of the DSRC compliance checking communi-cation of the OBU.


A vehicle is passing a fixed or portable compliance checking station and is generating a compliance checking communication event.


The performance of the OBU with respect to the DSRC Compliance Checking Communication. The KPI is based on a sample (time period) measuring.

Definition of KPI

A: The total number of DSRC Compliance Checking Events (at a specific compliance checking station during a defined time period) which can be linked to a GNSS Chargeable Event of the SP divided by B: the total number of Chargeable Events of an SP at the compliance checking sta-tion within a defined time period. (The time and the compliance checking stations have to be communicated to the SP)

Mitigation ideas • Instruct users of better/ correct mounting of OBE

• Modify installation of RSE for CCC


Annex A Status of KPI develop-ment

The following section provides a summary of the current status of KPI development for EETS in each of the Toll Domains participating in the REETS project. The KPIs that have been already selected or that were already under consideration by each TC prior to the REETS project are included in Annex B. A.1 Austria Status of KPI development: ASFINAG has developed and published EETS KPIs in

Annexes to its EETS Domain Statement.

General information pertaining to the EETS Domain Austria:

Toll ChargerTC = ASFINAG

CEN DSRC Multi-lane free-flow system

Vehicles liable to toll: heavy goods vehicles > 3.5 t on highways and expressways with mandatory

OBU

Fully electronic distance-based toll system

Toll transactions are performed with Security Level 1 access protection according to EN 15509

The amount of the toll rate is

(i) route-specific, and also depends on

(ii) the time at which the section is used,

(iii) the location of the specific toll section as well as

(iv) the number of axles and

the EURO emission class

A.2 Denmark / Easygo Status of KPI development:

KPIs are awaiting final confirmation in the EasyGo Steering Committee

General information pertaining to the EETS Domain in Denmark

Toll ChargerTC = A/S Storebælt

Toll stations with barriers operating DSRC

Classification parameters: EU vehicle classes (up to 3.5 t or above 3.5 t) in combination with the

measured length and height of the vehicle

Toll transaction policiy: Storebælt is currently a DSRC based toll station with barriers. Before the bar-rier opens, the vehicle has been controlled for valid payment. When a vehicle with an OBU is regis-


tered, a control is carried out to determine if it is a known EETS Provider and if the OBU is valid. If the OBU is valid the barrier opens and information about the OBU and price will be passed on to the EETS Provider according to the agreed procedures and data formats.

A.3 France (ASFA, TIS-PL) Status of KPI development: KPIs are in use in the TIS/PL toll domain. For each bi-

lateral frame (18 for a given ETSP; 5 for a given Toll ChargerTC), the KPI evaluation is done periodically for each OBU model (ManufacturerID + EquipmentClass), weekly, monthly and annually.

A.4 France (MEEDDAT, Ecotaxe) Status of KPI development: MEEDDAT has developed KPIs for the launch of Eco-

taxe. The Ecotaxe KPI’s have been determined by the French State at the whole system level which includes both Ecomouv and SHT’s (and future EETS Providers).

A.5 Germany (BAG) Status of KPI development: BAG has been intensively working on the definition of

quality criteria for EETS Providers including KPI. A binding document will be release in the near future.

A.6 Italy Status of KPI development: Vision on KPI in the Italian EETS; Areas of KPI’s have

been formulated (not specific KPIs)

A.7 Poland Status of KPI development: KPIs are under development in parallel with the finalisa-

tion of statutory rules for the implementation of EETS in Poland which are at the stage of legislative work prepa-ration.

A.8 Spain Status of KPI development: A preliminary list of EETS’KPIs has been defined

A.9 Switzerland Status of KPI development: List of requirements. not been implemented to date,

therefore requirements without proof of feasibility.

General information pertaining to the EETS Domain ‘Switzerland’:

Toll ChargerTC = Federal Customs Administration (FCA)

Collection of performance-related heavy vehicle fee (HVF)

Vehicles liable to toll: EETS can be used to levy the HVF only for vehicles (whose maximum permis-

sible laden weight exceeds 3.5 tonnes) registered abroad

Due date of toll: The fee becomes due on exiting Switzerland. Each journey between entering

Switzerland and leaving Switzerland is assessed separately.


HVF is measured by the distance covered, the emission rating (Euro class) and the relevant weight

of the vehicle. The latter corresponds to the lowest of the following three possible units of weight:

• Maximum permissible laden weight of the towing vehicle plus maximum permissible

laden weight of the trailer or, unladen weight of the articulated towing vehicle plus

maximum permissible laden weight of the semi-trailer; or

• maximum permissible laden weight of the vehicle train; or

• national weight limit in Switzerland (40 tonnes)

EETS journey: An EETS journey is understood to mean a journey by a foreign vehicle through Switzerland for which an EETS contract is used as the method of payment.

Figure 1. Swiss EETS toll domain

1. Prior to entering Switzerland, the EETS User must ensure that the master data is

correctly saved in the EETS OBE and that any attached trailer is correctly declared

in the EETS OBE.

2. On entering Switzerland, a CCC transaction (DSRC) and a LAC-DSRC transaction is

concluded between the EETS OBE and the HVF border beacon, through which:

o the entry into Switzerland is registered by the beacon

o the suitability for recording of the EETS OBE is verified (red/green)

o the fee-relevant vehicle data is retrieved

o the EETS contract is retrieved and verified

o the LAC position information is written to the EETS OBE

The verification of the EETS contract is done by individual tests


3. In the event of changes in the trailer status during the journey in Switzerland, the

• highest permissible laden weight occurring during the journey is relevant.

•

4. On leaving Switzerland, a LAC-DRSC transaction is concluded between the EETS

OBE and the HVF border beacon, in which the LAC position information is written to

the EETS OBE. Moreover, the EETS OBE must detect the exit autonomously and record it.

5. The EETS Provider communicates the declaration data for the EETS journey to the

FCA without being asked to do so. In addition, the FCA can optionally request the

position data relevant for the EETS journey from the EETS Provider.

6. The FCA conducts the assessment of the EETS journey and issues the assessment decision.

7. The EETS Provider is obliged to disclose the assessment data to the EETS User.


Annex B Existing EETS KPIs Note in the following tables, the best match to the definitions proposed within the Stockholm Group report‚ ’EETS Quality Monitoring and Assurance – Report on KPI Definition’ has been included to assist the reader in comparing similarity between definitions. However, it should be noted that in some cases the definitions used are not identical to those previously described in the report of the Stockholm Group. This in itself illustrates the difficulty of the task of trying to achieve completely harmonised KPIs definitions across all Toll Domains.


Austria KPI according to ASFINAG

Best match to Stock-holm Group KPI

Description

OBE Transaction Quality

DSRC Transaction DSRCTRANS

In order to statistically eliminate the influence of RSE errors and user errors, the assessment of the error rates makes reference to the equivalent, calculated error rates (= rate of erroneous transactions, rate of synthetic transactions, rate of missing transactions detected by enforcement equipment) of reference OBE. The refe-rence OBE are defined as those On-Board Unit types that are mainly used in the local toll system of ASFI-NAG. The TC sends all information on erroneous, retroactive and missing transactions per OBE of the SP to the SP on a regular basis, but at least once a month, in order to enable a check on collection quality. The 3 error rates are calculated as follows:

Rate of erroneous transactions = !""#$!#%& !"#$%#&!'($%!"#$%&'& !"#$%#&!'($%

Rate of retroactive transactions = !"#!$%&#'(" !"#$%#&!'($%!"#$%&'& !"#$%#&!'($%

Rate of missing transactions = !"##"$% !"#$%#&!'($% !"#$%&'& !"#$%#&!'($%

The rate of erroneous transactions, the rate of retroacti-ve transactions and the rate of missing transactions of the reference OBE and the OBE of an SP in use will be reported not later than 1 month after the end of an ob-servation period and used for the comparison. If the rate of erroneous transactions exceeds the refe-rence value including a tolerance limit, the SP must pay the TC a fee (=fee rate for determination/correction) for the increased expense for each erroneous transaction that lies above the tolerance limit. The same applies for rate of retroactive transactions as well as for the rate of missing transactions. The fee for the latter also includes a lost toll fee in addition to the fee rate for determinati-on/correction.


KPI according to ASFINAG


Description

Quality of the data stored in the OBE

Correctness of user and vehicle data CORRDATA

The SP must ensure that the licence plate (incl. country of registration), vehicle class (UNECE vehicle class) and number of axles of the vehicle are correctly stored in the OBE. The possible incorrect storage of one of these features will be verified in the course of an inspection process. In the course of a regular inspection, it is de-termined how many of the OBE issued by the SP and registered on the TC’s network deviate in one of these features from the features actually identified in the in-spection process. The SP must send the TC the documents of proof on any deviations found. If the per-sonalisation of the On-Board Unit matches the documents of proof, then the deviation was caused by the misbehaviour of the EETS User. If a deviation between the personalisation of the On-Board Unit and the documents of proof is established during the inspec-tion, then the deviation was caused by the misbehaviour of the SP.

Service quality of the SP data inter-face to the TC

No equivalent The service quality of the SP data interface to the TC is assessed on the following factors with respect to the SP data interface: • Uptime • Compliance with response times • Verifiable and impermissible refusal of the TC’s data transmissions • Verifiably incorrect messages from the SP • Missing entries in the current User List (White List) Limit values are set for the factors above. If those are exceeded, a penalty will become due.

Invoicing error rate

No equivalent The TC is entitled to verify the correctness of the in-

voices issued by the SP to the EETS User on behalf of

the TC on the basis of random samples or individually in

the event of concrete suspicion.

These random samples are used to determine an error

rate for the invoicing, which shows the proportion of the

erroneous invoices of the SP within the quantity of che-

cked instances.


Denmark / Easygo KPI according to Easygo


Description

OBE Transaction Quality

DSRCTRANS DSRC Transaction

Transaction groups in systems with barriers:

1. Automatic is the cases where the OBE

works correctly

2. Manual keyed transactions at the road-

side

3. Manual keyed transactions at the back of-

fice (Similar to video based transactions) however only

very few cases

KPI OK % = 1 / (1+2+3)

KPI not OK % = (2+3) / (1+2+3)

The KPI monitoring are not only done on a overall level

but are also divided in to subgroups in order to detect

problems in a group of OBE or a specific toll lane.

Group of OBE divided by:

• TSP

• Year

• Model and DSRC application

Group of Toll lane divided by:

• Dedicated DSRC only

• Mixed lane

• No DSRC beacon at RSE (Could be an infor-

mation issue if SU drives in that lane)

The main quality requirement of performance are the

amount of OBE from a SP that are able to be read au-

tomatically without manual interference. The require-

ment is 99.95 % to be read correctly.


France (ASFA, TIS-PL) KPI according to ASFA


Description

Abnormal transac-tions (degraded mode)


Indicates the quality of the communication between the OBE and the TC RSE. Abnormal transactions occur when no electronic com-munication between OBE and RSE can take place and a manual transaction can take place instead: - Abnormal mode in payment lanes (close and open system) - Abnormal mode in entry lanes (close system) Abnormal mode may occur for one or several reason(s): OBE behaviour, RSE behaviour, driver behaviour, OBE installation, …

OBE considered out of order (not detected)

EETS availability AVAIL

Indicates the quality of the OBE of the provider. An OBE is considered “Out of Order”, when a given number of consecutive transactions are in “abnormal mode”. The OBE is identified via the PAN. This periodic verification (weekly and monthly) is done by the Toll Charger and/or the ETS Provider. The SP has to proceed to the exchange of the OBU in a given period.

Trips tariffed on the maximum possible value (Trajet le Plus Cher /TLPC)


Indicates the quality of the communication between the OBE and the TC RSE. In TIS/PL, missing entrance de-tections may (depending on the back office processing) imply paying the maximum possible toll. A “TLPC” is an abnormal event due to the behaviour of the OBU, and/or the entry RSE, and/or driver’s behav-iour

Transactions with abnormal events


Indicates the quality of data stored in the OBE. Errors can occur due to.

• Not conform security mechanisms (Authentica-tors, transactions counter sequence)

• Inconsistency of the license plate of the vehicle with the one read in the OBU

• Inconsistency of the vehicle category as detect-ed by the Toll Charger and the one as obtained with the data of OBU.


Qualitative indicators

These indicators are monitored and evaluated monthly, quarterly and annually), based on weekly rSPorts received from each Toll Charger and from each ETS Provider. For more details see enclosed

REETS WP3 TIS PL KPI 20130930.pptx


France (MEEDDAT, Ecotaxe) KPI according to MEEDDAT


Description

Performance de la collecte / Detec-tion Rate

GNSS System Over-charging GNSSOVER

Indicates the quality of the equipment and processes of the SP for the detection of road segments subject to toll (no overcharging). The KPI is calculated in a sample by dividing

-‐ the number of intersections of toll points detect-ed by the EETS-Provider with correctly transmit-ted elements that are necessary for calculating the tax base,

by

-‐ the number of all intersections of toll points de-tected by the EETS-Provider.

The performance of the detection of correct tolling events is measured and calculated by Ecomouv '. The measurement is done using the reference fleet of Ecomouv '. The EETS-Provider must equip parts of the reference fleet of Ecomouv’.

Taux de factura-tion erronée de taxe nominale / rate of incorrect billing

No match Indicates the quality of forwarding the billing details from the TC to the user. The performance rate is calculated quarterly, by dividing

-‐ the number of requests for refunds of taxpayers which were rejected by DGDDI due to an error made by the SP because the SP made an error in the transfer of the tax certificate provided by Ecoumouv',

and

-‐ the number of tax certificates and receipts that were sent by the SP to the taxpayer.

The basis for calculation are tax certificates per vehicle per quarter (even if the taxpayer receives a tax certifica-te for several vehicle). Excluded from the calculation are errors due to the fact that the taxpayer has made false statements and errors that are beyond the control of the SP.

Trips tariffed on the maximum possible value (Trajet le Plus Cher /TLPC)


Indicates the quality of the communication between the OBE and the TC RSE. In TIS/PL, missing entrance de-tections imply paying the maximum possible toll.

Transactions with abnormal events


Indicates the quality of data stored in the OBE. Errors can occur due to e.g. security mechanism, inconsistent license plate, inconsistent vehicle category.


Germany (BAG) KPI according to BAG


Description

GPS Capture Rate

GNSSUNDER, GNSS System Undercharging

Indicates the quality of the equipment and processes of the SP for the detection of road segments subject to toll. Erroneously detecting road segments not subject to toll as toll roads is not measured here.

DSRC capture rate

GNSS System En-forcement Transaction GNSSENF

Indicates the quality of the DSRC Enforcement inter-face. This is also used to verify the toll declaration data (GPS Capture Rate).

Black List Han-dling

No match Indicates the quality of the black list by comparing en-forcement transactions with black list entries. Vehicles with a compliant OBE (status green) must not be placed on a black list (in this case the OBE status should be red).

White List Han-dling


Indicates the correctness of the white list data, as well as the timeliness of the transmission of user/ vehicle data. DSRC transactions of vehicles that can be linked to a specific EETS-Provider must have a white list entry after a certain timeframe.

Toll declaration data

Transmit GNSS tolling transactions GNSSDECL

Indicates the timeliness of the transmission of toll decla-ration data. The time between the usage of a road seg-ment and the transmission of this event to the TC in the toll declaration data is fixed in the contract.

Billing details Sending and conclud-ing billing details BILLDETAILS

Indicates the timeliness of the transmission of billing details from the SP to the TC.

Payment an-nouncement

No match Indicates the timeliness of the transmission of payment announcement from the SP to the TC.


Italy KPI according to AISCAT


Description

Technical interop-erability

TBD Indicates the quality of the EETS-Provider’s OBE to per-form correctly in the toll domain. KPI could be defined for the certification of equipment, average behavior of the equipment, extreme behavior of the equipment.

Management in-teroperability

TBD Indicates the performance in publishing critical data (TCD, Lists,…) as well as respecting timings and proce-dures for the correct management of the system includ-ing the support for enforcement procedures.

End-to-end data exchange

TBD Indicates the quality of application data exchange from OBE to the final interfaces (ISO 17575, EN ISO 12855).

Economic aspects TBD Indicates the performance in economic aspects such as solvency, dispute handling, “European” behavior.


Poland KPI according to Poland


Description

Interface Availabi-lity

Percentage time during measurement period that SP’s side of interface 12855 is fully available to TC

This issue falls outside of statutory regulations and will be regulated in the contract with GDDKiA.

Payment Failure of SP to pay invoices issued by TC on time and / or in full. For discussion as to whether this is per payment or measured over a period.

SPs should be treated identically to EETS users and therefore Polish regulations may provide for SPs to ope-rate their accounts with the TC as pre-pay accounts or as post-pay accounts (the choice of form of payment will be a decision made solely by the SP). This KPI can only apply to accounts operated as post-pay accounts. In the event that an invoice for a post-pay account is paid late, then regulations permit the guarantee associ-ated with that post-pay account to be drawn on. This results in the post-pay account being converted into a pre-pay account. Since Polish law does not differentiate (discriminate) between individual users and SPs, this procedure can also be applied to the accounts of SPs. In this situation, if the balance on such account will fall below zero, all OBE assigned to this account will signal a violation having been committed. In consequence, financial penalties will be imposed on the SP.

OBE Transaction Reliability

Percentage of DSRC transactions fully com-pleted in measurement period. Necessary to ensure that there is no deterioration over time of transactions from approved OBE

Need to ensure that implementation of KPI does not contradict Decision requirements relating to conformity to specification and / or suitability for use approval

Enforcement In-formation Provisi-on

Measure of accuracy and timeliness of SP’s response to requests issued over 12855 by TC for user informati-on required for en-forcement purposes

It is planned to include in statutory level regulations re-quirement for the SP to provide TC or Enforcement Bo-dies with relevant user information for enforcement pur-poses, i.e. for purposes of administrative proceedings governed by Enforcement Bodies. This KPI applies in the situation where administrative penalties are imposed on the SP’s EETS Users/ clients.”


Spain KPI Best match to Stock-

holm Group KPI Description

Delay between transaction trans-mission and pay-ment settlement

Payment PAY Evaluation of the the delay between transaction trans-mission and payment settlement by comparing it to the TC/SP Contract stated maximum delay

OBE’s PAN man-ual entries vs. electronic transac-tions (per issuer)

DSRC Transaction DSRCTrans

OBE behaviour evaluation obtained by comparing the dregraded mode rate of the issuer’s OBE to the average same rate of the rest of issuers


Switzerland KPI according to FCA


Description

Accuracy of dis-tance recording

GNSSUNDER, GNSS System Undercharging and GNSSOVER GNSS System Overcharging

The accuracy of the distance recording by the odometer of the EETS OBE is verified on the basis of the position data and all further information available to the FCA. The deviation may not exceed the prescribed value of +/- 4% in any EETS journey.

Quality of the po-sition data


The quality of the position data is measured on the basis

of the percentage of EETS journeys completely docu-

mented by position data. The percentage of completely

documented EETS journeys is determined as follows:

• 100% / number of EETS journeys for which posi-

tion data was transmitted x number of journeys

completely documented by position data

The percentage of completely documented EETS jour-neys must be at least 98%.

Quality of the DSRC-based recognition of the fee period


The quality of the DSRC-based recognition of the fee

period is measured on the basis of the percentage of

EETS journeys for which a LAC transaction and a CCC

transaction are protocolled in the declaration data on

entry and a LAC transaction is protocolled in the decla-

ration data on exit:

• 100% / (number of EETS journeys) x (number of

EETS journeys with protocolled LAC and CCC

transaction on entry and protocolled LAC trans-

action on exit)

The percentage of border crossings with complete DSRC-based recognition of the fee period must be at least 99.5%.

Quality of the au-tonomously detec-ted fee period


The quality of the autonomously detected entries and

exits is measured on the basis of the correctly docu-

mented (autonomously detected) entries and exits:

• 100% / (number of EETS journeys x 2) x (num-

ber of correctly documented entries/exits)

The percentage of correctly documented entries and exits must be at least 99.5%.


KPI according to FCA


Description

Compliance with deadlines

No equivalent Compliance with deadlines is measured on the basis of

two individual parameters. Percentage of declaration

data transmitted following the end of the fee period with-

in the required deadline:

• 100% / number of declaration data transmitted in

total x number of declaration data transmitted

within the required deadline

The percentage of transmissions within the required

deadline must be at least 99.8%. Percentage of individ-

ual requests from the FCA for declaration and position

data responded to within the deadline:

• 100% / number of individual requests for decla-

ration and position data in total x number of

transmissions of declaration and position data (in

response to an individual request) within the re-

quired deadline.

The percentage of transmissions within the required

deadline must be at least 99 %.

Degree of compli-ance with com-pulsory declarati-on

GNSSUNDER, GNSS System Undercharging (missing toll declarati-ons)

The degree of fulfilment of compulsory declaration is

measured on the basis of the percentage of EETS jour-

neys for which the SP transmits declaration data. The

percentage is calculated as follows:

• 100% / number of EETS journeys x number of

EETS journeys for which the SP transmits decla-

ration data.

The percentage of EETS journeys with declaration data from the SP must be at least 99.9%.


KPI according to FCA


Description

Quality of the master data


The quality of the master data is measured on the basis

of the percentage of matches between the master data

in accordance with the declaration data and the master

data in accordance with the vehicle documents:

• 100% / number of EETS journeys in total x num-

ber EETS journeys for which no difference is

found between the master data in accordance

with the declaration data and the master data in

accordance with the vehicle documents.

The percentage of correct master data must be at least

99.8%.


AETIS KPI according to AETIS


Description

no match General data and report on activity volumes per toll do-main and per SP.

DSRCTRANS DSRC Transaction

Number of OBE’s/vehicles using a degraded mode for toll transaction and identified as belonging to a certain SP, related to the total number of OBE seen in the toll domain of the TC, and explanations regarding the rea-son of the degraded mode transaction

Sending and conclud-ing billing details BILLDETAILS or Transmit GNSS tolling transactions GNSSDECL

Number of missed toll declaration compared to the total number of toll declaration for a specific SP (where missed toll declarations are clearly identified as belong-ing to a customer of the SP), and explanations regarding the reasons of the missing transactions and whether it was possible to reconstitute or not the missing transac-tions

Sending and conclud-ing billing details BILLDETAILS or Transmit GNSS tolling transactions GNSSDECL

Number of erroneous toll declarations compared to the total number of toll declaration from a specific SP and explanations for the mistakes

GNSS System Toll Context Data CON-TEXT

Errors in the toll domain definition updates in a GNSS tolling system (made by SP in implementing or by TC in the definition)

no match Zones of incorrect localization (GNSS tolling) without presence of LAC

no match Number of customer claims and time for instruction, number of accepted claims compared to the total num-ber of claims for a TC

no match Number of incidents in the back-office data exchange, files impacted and explanations for the non-compliances

no direct match Time for transmitting data compared to the contractual obligations

no match Time for reaction, analysis and corrective, preventive measures to be taken for non-compliances. Repeated non-compliances.


Annex C Glossary

Glossary V2.0

No. Terminus Ab-brev.

(short) description

1 Service Provider SP Company / Entity offering the services of an

EETS-Provider but not necessarily formally registered as an EETS-Provider. Since the REETS Project shall facilitate the transition to EETS, it is recommended, to gen-erally use "Service Provider (SP)", except if "EETS-Provider shall be explicitly addressed (e.g. in the context of registration).

2 EETS-Provider EP

A legal entity fulfilling the requirements of Art 3 and registered in a Member State where it is established, which grants access to EETS to an EETS user (see Art 2 b) Decision 2009/750/EC).

3 Member State MS

EU Member State

4 European Electronic Toll Service

EETS

The abbreviation EETS stands for European Electronic Toll Service. It is a service that ena-bles the payment of tolls with a single contract at a single EETS provider and just one on-board unit throughout the European Union.

5 Regional European Electronic Toll Service

REETS

The REETS-TEN project aims at deploying EETS compliant services in a cross-border regional project. The Project shall cover the electronically toll network of 7 Member States (Austria, Denmark, France, Germany, Italy, Poland and Spain) and Switzerland.

6 Toll Charger TC

Public or private organisation which levies tolls for the circulation of vehicles in a toll domain (see Art 2 k) Decision 2009/750/EC)

7 User Physical or legal person who subscribes a con-tract with a Service Provider in order to have access to EETS compliant services (see Art 2 c) Decision 2009/750/EC).


8 On Board Equipment OBE The complete set of hardware and software components required for providing EETS com-pliant services which is installed in a vehicle in order to collect, store, process and remotely receive/transmit data (see Art 2 e) Decision 2009/750/EC)


(short) description

9 Interoperability constit-

uents Any elementary component, group of compo-

nents, subassembly or complete assembly of equipment incorporated or intended to be in-corporated into EETS upon which the interop-erability of the service depends directly or indi-rectly, including both tangible objects and in-tangible objects such as software, see Article 2 of the EETS Decision. Examples of interopera-bility constituents are on-board equipment (in-cluding connected back office systems), road-side equipment (including charging beacons, localization augmentation beacons and en-forcement devices), EETS Providers’ and Toll Chargers’ back-office data exchange systems.

10 Toll A charge, tax or duty levied in relation with cir-culating a vehicle in a toll domain (see Art 2 j) Decision 2009/750/EC)

11 Toll domain

An area of EU territory, a part of the European road network or a structure (such as a tunnel, a bridge, a ferry,..) where toll is collected (see Art 2 n) Decision 2009/750/EC).

12 Tariff class

The set of vehicles treated similarly by a Toll Charger (see Art 2 g) Decision 2009/750/EC).

13 Vehicle classification parameters

The vehicle related information according to which tolls are calculated based on the Toll Context Data (see Art 2 q) Decision 2009/750/EC).

14 Certification

Certification is defined as an EETS Provider's or its representative's official written statement that its interoperability constituents comply with the associated specified (technical) require-ments.


15 Technical accreditation Technical accreditation covers the technical aspects of the accreditation of an already regis-tered EETS Provider in individual toll domains under responsibility of a Toll Charger (or a clus-ter of Toll Chargers).

16 Technical requirements for registration

Requirements defined by the Member State responsible for the registration to check against Article 3b of the EETS decision


(short) description

17 Toll domain independ-

ent specifications Technical specifications for interoperability

constituents that are defined by technical standards or other regulations or specifications independently from individual toll domain re-quirements

18 Toll domain specific specifications

Technical specifications for interoperability constituents that comprise requirements that are specific to the needs of a toll domain

19 Security Policy

A Security Policy is a set of requirements and applicable counter measures specified by the party responsible for the security in a system exposed to threats. These counter measures are based upon a risk analysis of the system in order to protect those data exposed to threats in the relationships between TC and SP.


20 Cluster A cluster of ETC Toll Domains is a set of Toll Domains, interconnected or not, which feature the same or very similar ETC toll collection context(s) in a contractual framework like Memorandum Of Understanding or any other agreement between the Toll Domain repre-sentatives, i.e. the Toll Chargers.

This agreement specifies the rules regarding interoperability and its management within that cluster of ETS Toll Domains; it includes refer-ences to mutually agreed and shared detailed contractual, procedural and operational docu-mentation as well functional and technical specifications (particularly, interfaces for OBU // RSE and for Toll Charger // Service Provider central systems). A cluster of Toll Domains may have a unique representative for some common subjects.

Relationship between Toll Domains and Ser-vice Providers are fixed by bilateral contracts. Common validity periods of bilateral contracts with a given ETC Provider allow the interoper-ability for the global cluster.


(short) description

21 Accreditation The Accreditation covers the whole procedure

(contractual and technical) to be successfully fulfilled by a Service Provider in order that its technical system could be accepted on a Toll Domain and that the TC entrusts the SP with the toll collection and the invoicing process to the SU. When the Accreditation is successfully com-pleted, the Service Provider is “accredited” in the relevant Toll Domain.

REETS D3.1 Definition of Toolbox KPIs V3.1

Documents

Transcript of REETS D3.1 Definition of Toolbox KPIs V3.1