UNIT B5 Help Desk: estat-support-eDAMIS@ec.europa.eu

DDAATTAA TTRRAANNSSMMIISSSSIIOONN

CCOOUURRIIEERR

Issue 91

March 2010

1

Latvia: Change of Local Coordinator

Mr. Alvis Galvins

Central Statistical Bureau of Latvia has replaced Mr. Pavels ONUFRIJEVS as Local Coordinator for Latvia.

eDAMIS Web Application (eWA)

Version 2.9.8 available

In the previous issue of the DTC, we explained that the various reports in eWA 2.9.7 have the following behaviour concerning the sort order used:

- By default, the sort order in the different reports is not defined. When a file is sent, it may not appear in the short history report if the sort order has not been defined beforehand.

- The user can define the sort order by clicking on the little arrow near the column title. The sort order is active during the entire session.

- The sort buttons are inverted, i.e. the up arrow sorts the report in the descending order and vice versa.

These defects have been corrected in eWA version 2.9.8 that is available for download from the eDAMIS Help Centre.

http://circa.europa.eu/irc/dsis/edamis/info/data/website/tools/ewa/downloads.htm

eDAMIS Traffic Monitoring

Data files traffic report

The ‘Reports’ menu in the eDAMIS Web Portal (eWP) groups all reports about traffic monitoring, the inventory (themes, modules, domains, datasets), and user rights.

There are four families of traffic reports:

• Data files traffic

• Timetable between dates (detailed and summary)

• Timetable by occurrences (detailed and summary)

• Volume and number of files transmitted

In this issue of the Data Transmission Courier we publish an article on the data files traffic report. The other reports will be described in future issues.

eDAMIS https://webgate.ec.europa.eu/edamis

eDAMIS Support estat-support-edamis@ec.europa.eu

eDAMIS Help Centre https://webgate.ec.europa.eu/edamis + "eDAMIS Help Centre"

EUROSTAT - B5 Data Transmission Courier March 2010

2

Single Entry Point (SEP) Coverage Indicator Calculation date: 23/02/2010

Nov 08 - Jan 09 Nov 09 - Jan 10 Received Not Rcvd Total Coverage Received Not Rcvd Total Coverage

Change

EU AT 148 42 190 78% 156 38 194 80% 3% BE 114 85 199 57% 118 83 201 59% 1% BG 145 45 190 76% 150 43 193 78% 1% CY 128 54 182 70% 138 48 186 74% 4% CZ 148 37 185 80% 155 32 187 83% 3% DE 159 44 203 78% 169 36 205 82% 4% DK 114 70 184 62% 119 68 187 64% 2% EE 161 17 178 90% 166 15 181 92% 1% EL 126 55 181 70% 116 69 185 63% -7% ES 144 55 199 72% 152 49 201 76% 3% FI 157 32 189 83% 171 22 193 89% 6% FR 134 56 190 71% 145 48 193 75% 5% HU 150 50 200 75% 168 35 203 83% 8% IE 117 63 180 65% 107 77 184 58% -7% IT 123 64 187 66% 131 59 190 69% 3% LT 160 29 189 85% 169 24 193 88% 3% LU 115 77 192 60% 137 57 194 71% 11% LV 150 28 178 84% 156 26 182 86% 1% MT 114 56 170 67% 125 49 174 72% 5% NL 155 39 194 80% 166 29 195 85% 5% PL 168 37 205 82% 180 27 207 87% 5% PT 145 57 202 72% 154 51 205 75% 3% RO 179 28 207 86% 176 34 210 84% -3% SE 147 49 196 75% 146 53 199 73% -2% SI 145 34 179 81% 159 24 183 87% 6% SK 152 44 196 78% 171 29 200 86% 8% UK 120 77 197 61% 122 78 200 61% 0% EU Total 3818 1324 5142 74% 4022 1203 5225 77% 3%

CC HR 48 43 91 53% 62 36 98 63% 11% MK 4 26 30 13% 11 30 41 27% 13% TR 34 52 86 40% 43 49 92 47% 7% CC Total 86 121 207 42% 116 115 231 50% 9%

EFTA CH 36 34 70 51% 39 39 78 50% -1% IS 23 49 72 32% 25 51 76 33% 1% LI 13 24 37 35% 19 17 36 53% 18% NO 78 39 117 67% 82 42 124 66% -1% EFTA Total 150 146 296 51% 165 149 314 53% 2%

The calculation takes into account, for each reference period, the datasets having their deadline between the beginning and the end of the concerned period. The concerned datasets are the ones that are being expected to be received by Eurostat in this period. Amongst these datasets, a count is made of how many have been received via eDAMIS. The coverage indicator is calculated as number of first dataset occurrences received via eDAMIS / expected number of dataset occurrences.

Detailed results for the last reference period and an explanation on the calculation method are available on the eDAMIS Help Centre ( In Practice > Monitoring of Transmission ).

EUROSTAT - B5 Data Transmission Courier March 2010

3

Single Entry Point (SEP) Coverage Indicator by NSI Calculation date: 23/02/2010

Nov 09 - Jan 10 Received Not Rcvd Total Coverage EU AT NSI STAT 136 20 156 87% BE NSI INS 70 62 132 53% BG NSI BNSI 118 33 151 78% CY NSI CYSTAT 136 47 183 74% CZ NSI CSO 122 28 150 81% DE NSI DESTATIS 129 28 157 82% DK NSI DS 104 23 127 82% EE NSI SE 149 13 162 92% EL NSI NSSG 90 30 120 75% ES NSI INE 65 18 83 78% FI NSI STFI 115 10 125 92% FR NSI INSEE 78 13 91 86% HU NSI HCSO 135 22 157 86% IE NSI CSO 73 62 135 54% IT NSI ISTAT 99 31 130 76% LT NSI SL 151 21 172 88% LU NSI STATEC 107 47 154 69% LV NSI CSBL 136 23 159 86% MT NSI NSO 125 49 174 72% NL NSI CBS 154 22 176 88% PL NSI PCSO 154 22 176 88% PT NSI INE 125 39 164 76% RO NSI INS 165 27 192 86% SE NSI SCB 109 22 131 83% SI NSI SURS 142 23 165 86% SK NSI SKSO 152 17 169 90% UK NSI ONS 75 30 105 71% CC HR NSI DZSHR 62 36 98 63% MK NSI SSOM 11 30 41 27% TR NSI TURKSTAT 26 31 57 46% EFTA CH NSI OFS 36 37 73 49% IS NSI STATICE 12 39 51 24% LI NSI AS 5 17 22 23% NO NSI SSB 68 37 105 65%

The calculation takes into account, for each reference period, the datasets having their deadline between the beginning and the end of the concerned period. The concerned datasets are the ones that are being expected to be received by Eurostat in this period. Amongst these datasets, a count is made of how many have been received via eDAMIS. The coverage indicator is calculated as number of first dataset occurrences received via eDAMIS / expected number of dataset occurrences.

Detailed results for the last reference period and an explanation on the calculation method are available on the eDAMIS Help Centre ( In Practice > Monitoring of Transmission ).

EUROSTAT - B5 Data Transmission Courier March 2010

4

Single Entry Point (SEP) Coverage Indicator Calculation date: 23/02/2010

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

NO

LI

IS

CH

TR

MK

HR

UK

SK

SI

SE

RO

PT

PL

NL

MT

LV

LU

LT

IT

IE

HU

FR

FI

ES

EL

EE

DK

DE

CZ

CY

BG

BE

AT

Nov 08 - Jan 09

Nov 09 - Jan 10

EU27: 77 %

EUROSTAT - B5 Data Transmission Courier March 2010

5

Traffic by client and by format from 01/11/2009 to 31/01/2010

eWA68%

eWP20%

E-MAIL to SEP0%

STATEL0%

eWF12%

GESMES29%

SDMX-ML7%

XML1%

Proprietary26%

Compressed6%

Non-Proprietary25%

Encrypted6%

EUROSTAT - B5 Data Transmission Courier March 2010

6

Data files traffic report

The ‘Reports’ menu in the eDAMIS Web Portal (eWP) includes all reports on traffic monitoring, the inventory (themes, modules, domains, datasets), and user rights.

Fig.1: Traffic Monitoring

There are four families of traffic reports:

• Data files traffic • Timetable between dates (detailed and summary) • Timetable by occurrences (detailed and summary) • Volume and number of files transmitted

This article focuses on the data files traffic report that shows the incoming and outgoing files passing through the Eurostat Single Entry Point (via eWA, eWP or eWF) during a certain period.

Defining the contents of a report:

Fig.2: Selection criteria

EUROSTAT - B5 Data Transmission Courier March 2010

7

The contents of a report is defined by two hierarchical sets of selection criteria and by a period of time (see Fig.2). By default, ‘Country group’ and ‘Country’ and are pre-selected according to the user’s country. This pre-selection can be changed.

It is advised to filter the report sufficiently in order to avoid performance problems. By default, 1000 lines are displayed. Above this limit, an export in CSV format is proposed. Choosing a narrower range of dates and restricting the selection will speed up the display of the report.

Displaying a report

Once the selection criteria have been specified, the user clicks on a ‘View’ button to display the report. If the selection criteria are modified later, the user must click again on the ‘View’ button to refresh the screen.

In the default presentation of the report, all the columns are not displayed on the screen. Keeping the mouse pointer over a column header shows the complete header label.

Fig.3: Default presentation

Additional columns can be added by clicking on the list next to the ‘View’ button (Fig.4).

Fig.4: More columns

Columns can be removed by clicking on the ‘x’ in the column header (Fig.5).

Fig.5: Quick Filters

There are ‘Quick Filters’ under the column headers (see Fig.5). A ‘Clear filter(s)’ command allows to remove all filters at once (not shown on Fig.5).

The report can be sorted ascending or descending by clicking on the column headers. The sorting order will be indicated by a plus sign (ascending) or by a minus sign (descending) in the column header. The user can toggle through the available sorting orders by clicking on the column header. Only one column can be sorted at a time. By sorting another column, the report will be refreshed and the quick filters will be deactivated.

The user’s selections are stored in eDAMIS and linked to the user-id. They are used by default the next time the user consults a report.

EUROSTAT - B5 Data Transmission Courier March 2010

8

Explanation of the contents of the report

An example of the report is presented, divided into several parts for a better clarity.

Part 1

Data files are uniquely identified using the following fields:

• Dataset • From • Year • Period • Version

The columns ‘Year’ and ‘Period’ indicate the reference period of the data file.

Part 2

End of ref. Per. (End of reference period)

Date of the end of the reference period.

The transmission period starts the day after (see " Quick view on the management of dates in eDAMIS " at page 11).

Acknowledged

Date and time when the data file was received and acknowledged by the Single Entry Point.

EUROSTAT - B5 Data Transmission Courier March 2010

9

Indic. Deadline

The indicative deadline for the transmission of a dataset occurrence is obtained by applying the Maximum Delay after the end of the reference period.

The column is left empty if the Maximum Delay has not been specified in the dataset inventory.

The columns Acknowledged and Delay (days) are displayed in green when the data file is received before or on the day of the indicative deadline, and in red when the data file is received later.

Delay (days)

Difference, in days, between the indicative deadline and the acknowledgement date.

Maximum Delay

Number of months and days specifying the period during which Eurostat expects to receive a dataset occurrence after the end of the reference period.

Trans. Means (Transmission means)

Indicates the application used to transmit the data file.

Format

Indicates the format of the received data file. If eDAMIS is not able to recognise the file format, the following is displayed:

• ‘ENC’ when the file is encrypted. • ‘UNK’ when the file is not recognised at all.

Part 3

Sent as

Original name of the file sent by the data provider.

Delivered as

File name ascribed by eDAMIS to the received data file before forwarding it to the Eurostat production units. This file name is created according to the Dataset Naming Convention (see page 12).

Vol. (bytes)

Size, in bytes, of the received file.

Status Dat.

Activation date of the dataset.

EUROSTAT - B5 Data Transmission Courier March 2010

10

Other columns, not illustrated

Org(organisation)

Organisation that has sent the data file.

User

User-id of the data provider that sent the file.

Sender eMail

Email address of the data provider.

To

Country that will receive the data file (for outgoing datasets).

Grouped Dataset

Grouped dataset name (used with Web Forms).

Free comments

Comments attached by the data provider when sending the file.

Explanatory note

File name of explanatory notes attached to the data file by the sender.

Icons, legend, extraction and printing symbols

Icons:

Next to a domain or dataset, indicates that it is partly confidential. It is confidential only for some countries (and/or some datasets in case the icon is next to a domain).

Next to a domain or dataset, indicates that it is fully confidential. It is confidential for all countries (and all datasets in case the icon is next to a domain).

Next to a domain or dataset, indicates that the domain has a public key. The public key can be downloaded by clicking on the icon.

Next to a dataset, indicates that it is active and that transmissions for that dataset are possible.

Before the ‘acknowledged’ date in the Traffic Monitoring reports, indicates a delay compared to the indicative deadline.

Next to a dataset, indicates that a workflow is active for the dataset occurrence.

Next to a dataset, indicates that monitoring information has been corrected. Clicking on the icon displays the old and new values.

EUROSTAT - B5 Data Transmission Courier March 2010

11

Extraction and printing symbols:

To extract the content of the report in SDMX-ML format.

To extract the structure of the SDMX-ML format.

To extract the content of the report in CSV format.

To print the reports.

Quick view on the management of dates in eDAMIS

Jan Feb Mar Apr May JunOct Nov Dec

End of reference period

31/12

Beginning of the transmission period

01/01 31/05

Indicative deadline

Reference period(3 months = 1 quarter)

Transmission period

Maximum delay (5 months)

• 1 occurrence = 1 dataset + CY + YEAR + Period.

• 1 data file = 1 occurrence + Version.

• Indicative deadline = End of reference period + Maximum delay.

• The indicative deadline is based on the time-table of the dataset.

• The time-table is calculated based on the dataset timeliness attributes.

• The time-table does not exist for non-periodic datasets impossible to calculate the indicative deadline.

• The transmission period starts immediately after the end of the reference period.

EUROSTAT - B5 Data Transmission Courier March 2010

12

The eDAMIS Dataset Naming Convention (DSNC)

Managing data transmissions is not only a technical issue. Above all, excellent organisation is required to enable an effective cooperation between the parties involved: senders, receivers and applications. To this end, the DSNC is used by all eDAMIS applications (eWA, eWP and eWF) and by all components which interact with eDAMIS, i.e. SDMX-ML formatted messages, Eurostat SDMX-ML registry, validation engines and Web Services linked to third party applications.

The DSNC is a central aspect of eDAMIS and constitutes the common shell which will serve during all maintenance activities on the eDAMIS inventory.

The main concepts used are the ‘Dataset_Id’, the ‘Dataset_Occurence_Id’ and the ‘Data file’:

For each planned data transmission, it is very important to clearly identify the datasets to be sent in accordance with the DSNC. This must be done as soon as possible (already at the Working Group discussions stage, when applicable).

The ‘Dataset_Id’ is composed of ‘Domain_Id’ + ‘Dataset structure_Id’ + ‘Periodicity/ies’.

The ‘Dataset_Occurrence_Id’ is composed of ‘Dataset_Id’ + ‘Country Code’ + reference ‘Year’ + reporting ‘Period’.

For a same Dataset_Occurrence_Id, there is usually only one ‘Data file’ that is transmitted. In the case of several data transmissions for a data file (e.g. updated data file) for the same Dataset_Occurrence_Id, eDAMIS will automatically indicate the version of the datafile.