D3.7 Detailed Processing Model FIRE CCI

fire_cci

D3.7 Detailed Processing Model (DPM) Version 2

Project Name ESA CCI ECV Fire Disturbance (fire_cci)

Contract N° 4000101779/10/I-NB

Project Manager Arnd Berns-Silva

Last Change Date 08/09/2014

Version 2.2

State Final

Author Bernardo Mota, Jose Miguel Cardoso Pereira, Duarte Oom, Itziar Alonso, Andrew Bradley, Kevin Tansey, Thomas Krauß, Kurt Günther, Rupert Müller, Veronika Gstaiger

Document Ref: Fire_cci_Ph3_ISA_D3_7_DPM_v2_2

Document Type: Public

fire_cci

Doc. No.:Fire_cci_Ph3_ISA_D3_7_DPM_v2_2

Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page II

Project Partners

Distribution

Affiliation Name Address Copies

ESA-ECSAT Stephen Plummer (ESA – ECSAT) [email protected] electronic copy

Project Team Emilio Chuvieco, (UAH)

Itziar Alonso-Canas (UAH)

Stijn Hantson (UAH)

Marc Padilla Parellada (UAH)

Dante Corti (UAH)

Arnd Berns-Silva(GAF)

Christopher Sandow (GAF)

Stefan Saradeth (GAF)

Jose Miguel Pereira (ISA)

Duare Oom (ISA)

Gerardo López Saldaña (ISA)

Kevin Tansey (UL)

Andrew Bradley

Oscar Pérez (GMV)

Luis Gutiérrez (GMV)

Ignacio García Gil (GMV)

Andreas Müller (DLR)

Martin Bachmann (DLR)

Martin Habermeyer (DLR)

Kurt Guenther (DLR)

Veronika Gstaiger (DLR)

Eric Borg (DLR)

Martin Schultz (JÜLICH)

Angelika Heil (JÜLICH)

Florent Mouillot (IRD)

Julien Ruff (IRD)

Philippe Ciais (LSCE)

Patricia Cadule (LSCE)

Chao Yue (LSCE)

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

martin.habermeyer@dlr

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

electronic copy

Prime Contractor/

Scientific Lead

- UAH - University of Alcalá de Henares (Spain)

Project Management - GAF AG (Germany)

System Engineering Partners - GMV - Aerospace & Defence (Spain)

- DLR - GermanAerospace Centre (Germany)

Earth Observation Partners - ISA - Instituto Superior de Agronomia (Portugal)

- UL - University of Leicester (United Kingdom)

- DLR - GermanAerospace Centre (Germany)

Climate Modelling Partners - IRD-CNRS - L’Institut de Recherche pour le Développement - Centre National de la

RechercheScientifique (France)

- JÜLICH - Forschungszentrum Jülich GmbH (Germany)

- LSCE - Laboratoire des Sciences du Climat et l’Environnement (France)

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page III

Summary

This document is the Detailed Processing Model Document, version 2, for the fire_cci project. It

provides a structured and environment-independent description of the computer programming

algorithms.

Affiliation/Function Name Date

Prepared ISA, UAH, UL, DLR Andrew Bradley, Kurt Günther, Rupert Müller, Thomas Krauß, Bernardo Mota, Duarte Oom, Itziar Alonso,

17/09/2013

14/10/2013

22/10/2013

05/09/2014

Reviewed GAF Arnd Berns-Silva 08/09/2014

Authorized UAH/ Prime Contractor Emilio Chuvieco

Accepted ESA/ Project Manager Stephen Plummer

Signatures

Name Date Signature

Signature of authorisation and overall approval Emilio Chuvieco

Signature of acceptance by ESA Stephen Plummer

Document Status Sheet

Issue Date Details

1.0 12/07/2012 First Document Issue

1.1 27/11/2012 Addressing ESA comments according to CCI-FIRE-EOPS-MM-12-0048

2.0 25/09/2013 Final Processing Model of prototype; Addressing ESA comments according to CCI-FIRE-EOPS-MM-13-0025.pdf

2.1 16/01/2014 Addressing ESA comments according to CCI-FIRE-EOPS-MM-13-0040.pdf

2.2 08/09/2014 Addressing ESA comments according to CCI-FIRE-EOPS-MM-14-0015.pdf

Document Change Record

# Date Request Location Details

1.1 07/12/2012 DLR Section 3.2 Amended

Section 3.3 Separate workflow for test sites and global processing introduced

Section 3.4.1 Amended

Section 3.4.2 Figure 4 updated


Section 3.7.2 Figure 7 introduced

Section 3.8.1 Figure 8 introduced


ISA Section 4.3.1 Figure 11 updated

Section 4.3.5 Sections 4.3.5.1 and 4.3.5.2 introduced





fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page IV


Section 5.3.5 Section 5.3.5.1 and 5.3.5.2 introduced






UAH Section 6.1 Figure 26 updated

Section 6.2 Referring to pre-processing described in section 3

Section 6.3 Figure 27 updated


Section 6.5.2 Figure 29/30 updated, Figure 31 introduced

Section 6.6.2 Figure 32/33 updated



Section 6.9 Post-processing introduced

UL Section 7.1 Figure 36 introduced

Section 7.3.2 Introducing the 2nd Merge step and Figure 37

Section 7.3.5 Updating and amending “Processing steps”

Section 7.3.5.1 Figure 38 introduced




GAF Section 1.3 Description of symbols introduced; Reference to respective algorithm theoretical base documents applied

Whole document Typos and layout consistency

2.0

11/10/2013

DLR

Section 2

Text modified as response to ESA comments;

Figure 2 modified

Section 3 Text modified as response to ESA comments

Section 3.3 Figure 2 and 3 modified

Section 3.4.2 Figure 4 modified


Section 3.6.5 Including land-water masking pseudo code




Section 3.8.5 Including atmosphere correction pseudo code

Section 4.2

Amendment of pre-processing description;

Updating Figure 10

Section 4.3

Adapting overview description;

Modifying Figure 11;

Modifying Equations 4.3 – 4.7

Adapting Tables 2 and 3

Section 4.3.5 Adapting overview description;

Section 4.3.7

Restructured and text amendment; Adapting pseudo-code; Figure 14 modified;

Section 4.3.8

Introducing “Spatial p_scoring”, overview and pseudo-code description; Figure 15 modified;

Section 4.3.9 Pseudo-code description modified;

Section 5.1 Restructured and text adapted/amended

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page V

Section 5.2 Amendment of pre-processing description

Section 5.3

Adapting overview description; Figure 19 modified; Modifying Equations 4.3 – 4.7; Adapting Tables 4 and 5; Figure 21 modified;

Figure 22 introduced; Introducing processing step 5 – Spatial p_scoring; Figure 24 introduced;

UAH Section 6.1 Figure 27 adapted

Section 6.3 Figure 28 adapted

Section 6.4 Description amended; Figure 29 adapted

Section 6.5 Overview description modified;

Figures 30, 31, 32 adapated

Section 6.6 Figures 33, 34, 35 adapted

Section 6.10 Equations 6.1 – 6.3 modified and adapted

Section 6.11

Burnable tile function introduced; Build component function modified; BA detection function modified; Region growing function modified;

UL Section 7 Restructured, modified and adapted

Section 7.2 Adapted

Section 7.3.1 Figure 38 adapted

Section 7.3.2 Restructured; Figure 39 modified

Section 7.3.3 Table 6 extended

Section 7.3.4 Table 7 modified and extended

Section 7.3.5

Adapting processing steps 1 - 6, pseudo-code and functions upgraded;

Figure 40 modified;

Figure 41 modified;

Figure 42 modified;

Figure 43 modified;

Figure 44 modified;

2.1

16/01/2014

DLR

ISA

UAH

Section 1

Section 2

Section 3

Section 3.7

Section 3.8

Section 4.2

Section 4.3

Section 4.3.5

Section 4.3.6

Section 4.3.7 – 4.3.9

Section 5

Section 5.1

Section 5.2

Section 6.1

Section 6.4

Section 6.5

Section 6.6

Section 6.7

Section 6.8

Section 6.9

Section 6.11

Section 6.12

Rephrasing

Text modified as response to ESA comments

Text modified as response to ESA comments

Inserting more details in the pseudocode of masking

Changes in Figure 8

Updating Reference

Figure 11 updated; Table 2 updated;

Further remarks

Text restructured and rephrased

Rephrasing

Restructured; section on time series filtering in previous release discarded;

Rephrased

Figure 18 updated;

Restructured; Figure 20 upgraded;

Figure 21 upgraded;

Figures 22, 23 and 24 upgraded;

Figures 25, 26 and 27 upgraded;

Figure 28 upgraded;

Figure 29 upgraded;

Introducing output layers

Introducing list of variables

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page VI

UL

Section 7

Section 7.2

Section 7.3

Section 7.3.1

Section 7.3.2

Section 7.3.3

Section 7.3.5.1

Section 7.3.5.2

Section 7.3.5.3

Section 7.3.5.4

Section 7.3.5.5

Section 8

Introducing list of parameters

Rephrasing

Introducing confidence and uncertainty screening

Rephrasing and restructuring; updating index of functions;

Updating Figure 31

Updating logical flow, Updating Figure 32

Updating Table 6

Rephrasing and updating Figure 33

Updated description and restructuring processing part 1.2; Figure 34 updated; introducing Functions 7.4, 7.5, 7.7, 7.8

Updated description and restructuring processing part 1.3; Figure 35 updated; amending Function 7.11; integration of Function 7.4

Updated description and restructuring processing part 1.4; Figure 36 updated; introducing “Standard error of total burned area”; Function 7.16, 7.18, 7.19 updated

Updating description; introducing Figure 37

Updating References

2.2

08/09/2014

ISA

Section 3.1

Section 4

Section 5.1

Section 5.2

Section 5.3

Section 5.3.2

Section 5.3.3

Section 5.3.4

Section 5.4.5

Section 6.3.6

Section 6.3.7

Section 6.3.8

Section 6.3.9

Section 6.6.2

Section 6.7.2

Section 6.11

Section 7.3.1

Section 7.3.2

Section 7.3.4

Section 7.3.5

Section 7.3.5.2

Section 7.3.5.3

Section 7.3.5.

Section 7.3.5.5

Whole document

Discarding note on test sites

Note that AATSR processing is not applied in the global BA production

Text rephrasing

Upgrading text and Figure 11

Text rephrasing

Introducing Eq.4.8 Normalization of distances to ideal and anti-ideal points

Updating Table 4

Updating Table 5

Introducing – Maxima/Minima time series extraction

Upgrade of Scale Estimate

Upgraded and rephrased



Figure 26 updated

Figure 29 updated

Table 4, description of variable “Lon” updated

Updated; Figure 31 updated

Paragraph P1.5 updated

Table 7 updated

Introducing new c-shell script.

Processing updated; Function 7.5 updated

Updated

Updated; Function 7.19 updated, introducing 2nd c-shell script

Introducing 3rd c-shell script; Figure 37 updated; Cancelling layer stack

Typo / grammar correction and rephrasing; Updating references

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page VII

Table of Contents 1 Executive Summary ...................................................................................................................... 1

1.1 Scope ..................................................................................................................................................... 1 1.2 Purpose .................................................................................................................................................. 1 1.3 Organisation .......................................................................................................................................... 1

2 Processing Chain Overview .......................................................................................................... 2 3 Pre-Processing chain ..................................................................................................................... 4

3.1 Introduction ........................................................................................................................................... 4 3.2 Data preparation .................................................................................................................................... 4 3.3 Pre-processing chain overview .............................................................................................................. 4 3.4 Algorithm– Geometric Correction ......................................................................................................... 8

3.4.1 Overview .......................................................................................................................................... 8 3.4.2 Logical Flow .................................................................................................................................... 8 3.4.3 List of Variables ............................................................................................................................. 10 3.4.4 Processing step 1: Extraction of DEM ........................................................................................... 10 3.4.5 Processing step 2: Extraction of reference image ........................................................................... 10 3.4.6 Processing step 3: Transformation of A to T .................................................................................. 10 3.4.7 Processing step 4: Image-to-image matching ................................................................................. 10 3.4.8 Processing step 5: Iterative least-square-matching until sub-pixel accuracy .................................. 10 3.4.9 Processing step 6: Inverse Transformation back to A .................................................................... 10 3.4.10 Processing step 7: Selection of GCPs and ICPs ............................................................................. 10 3.4.11 Processing step 8: Estimation of affine transformation .................................................................. 10 3.4.12 Processing step 9: Calculation of deviations to measured ICPs ..................................................... 10 3.4.13 Processing step 10: Applying affine transformation to complete geolayer .................................... 10

3.5 Algorithm – ortho-rectification ............................................................................................................ 11 3.5.1 Overview ........................................................................................................................................ 11 3.5.2 Logical Flow .................................................................................................................................. 11 3.5.3 Processing Step 1: Splitting of image ............................................................................................. 11 3.5.4 Processing Step 2: Bounding Box .................................................................................................. 11 3.5.5 Processing Step 3: Resampling of image layer .............................................................................. 11 3.5.6 Processing Step 4: Resampling of flag layer .................................................................................. 12 3.5.7 Processing Step 5: Joining of Image .............................................................................................. 12

3.6 Algorithm – Water masking ................................................................................................................ 13 3.6.1 Overview ........................................................................................................................................ 13 3.6.2 Logical Flow .................................................................................................................................. 13 3.6.3 List of Variables ............................................................................................................................. 15 3.6.4 Processing steps ............................................................................................................................. 15 3.6.5 Processing step ............................................................................................................................... 15

3.7 Algorithm – Cloud, snow/ice, haze and spectral shadow masking ...................................................... 16 3.7.1 Overview ........................................................................................................................................ 16 3.7.2 Logical Flow .................................................................................................................................. 16 3.7.3 List of Variables ............................................................................................................................. 19 3.7.4 Processing step ............................................................................................................................... 19

3.8 Algorithm – atmospheric correction .................................................................................................... 25 3.8.1 Overview ........................................................................................................................................ 25 3.8.2 Logical Flow .................................................................................................................................. 27 3.8.3 Equations ........................................................................................................................................ 29 3.8.4 List of Variables ............................................................................................................................. 29 3.8.5 Processing steps ............................................................................................................................. 29

4 (A)ATSR Burnt Area Processing Chain .................................................................................... 31 4.1 Introduction ......................................................................................................................................... 31 4.2 Pre-processing ..................................................................................................................................... 31 4.3 Overview ............................................................................................................................................. 32

4.3.1 Logical Flow .................................................................................................................................. 32 4.3.2 Equations and functions ................................................................................................................. 34 4.3.3 List of variables .............................................................................................................................. 35 4.3.4 List of parameters ........................................................................................................................... 35 4.3.5 Step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and observational

statistics .......................................................................................................................................... 36 4.3.6 Step 2 - Time-series filtering .......................................................................................................... 37

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page VIII

4.3.7 Step 3 - Change point detection and ranking .................................................................................. 38 4.3.8 Step 4 - Spatial p_scoring ............................................................................................................... 40 4.3.9 Step 5 - Spatial probability revision - MRF segmentation ............................................................. 41

4.4 Post-processing .................................................................................................................................... 42 5 SPOT-VEGETATION Burnt Area Processing Chain ............................................................. 43

5.1 Introduction ......................................................................................................................................... 43 5.2 Pre-processing ..................................................................................................................................... 43 5.3 Overview ............................................................................................................................................. 44 5.4 Logical Flow ........................................................................................................................................ 44

5.4.1 Equations and functions ................................................................................................................. 46 5.4.2 List of variables .............................................................................................................................. 47 5.4.3 List of parameters ........................................................................................................................... 47 5.4.4 Step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and observational

statistics .......................................................................................................................................... 48 5.4.5 Step 2- Maxima/Minima time series extraction ............................................................................. 49 5.4.6 Step 3 - Time-series filtering .......................................................................................................... 50 5.4.7 Step 3 - Change point detection and ranking .................................................................................. 51 5.4.8 Step 4 - Spatial p_scoring ............................................................................................................... 54 5.4.9 Step 5 - Spatial probability revision - MRF segmentation ............................................................. 55

5.5 Post-processing .................................................................................................................................... 57 6 MERIS Burnt Area Processing Chain ....................................................................................... 58

6.1 Key Principles of the Algorithm .......................................................................................................... 58 6.2 Pre-processing ..................................................................................................................................... 59 6.3 Processing Chain ................................................................................................................................. 59 6.4 Data Filtering ....................................................................................................................................... 59

6.4.1 Overview ........................................................................................................................................ 59 6.4.2 Logical flow ................................................................................................................................... 59

6.5 Build Composites ................................................................................................................................ 60 6.5.1 Overview ........................................................................................................................................ 60 6.5.2 Logical flow ................................................................................................................................... 61

6.6 BA Detection ....................................................................................................................................... 62 6.6.1 Overview ........................................................................................................................................ 62 6.6.2 Logical flow ................................................................................................................................... 62

6.7 Compute Layers ................................................................................................................................... 64 6.7.1 Overview ........................................................................................................................................ 64 6.7.2 Logical flow ................................................................................................................................... 64

6.8 BA Monthly product ............................................................................................................................ 65 6.8.1 Overview ........................................................................................................................................ 65 6.8.2 Logical flow ................................................................................................................................... 65

6.9 Post Processing .................................................................................................................................... 66 6.10 Equations ............................................................................................................................................. 66 6.11 List of variables ................................................................................................................................... 67 6.12 List of parameters ................................................................................................................................ 67 6.13 Computer Programme in Pseudo-Code ............................................................................................... 68

6.13.1 Data Filtering ................................................................................................................................. 68 6.13.2 Build Components .......................................................................................................................... 69 6.13.3 BA Detection .................................................................................................................................. 70 6.13.4 Compute Layers ............................................................................................................................. 71 6.13.5 BA Monthly Product and Format Conversion ................................................................................ 72

7 Product Merging Processing Chain ........................................................................................... 73 7.1 Introduction ......................................................................................................................................... 73 7.2 Pre-processing and preparation ............................................................................................................ 73 7.3 Algorithm ............................................................................................................................................ 73

7.3.1 Overview ........................................................................................................................................ 73 7.3.2 Logical flow ................................................................................................................................... 74 7.3.3 List of variables .............................................................................................................................. 76 7.3.4 List of parameters ........................................................................................................................... 76 7.3.5 Processing steps ............................................................................................................................. 77

8 References .................................................................................................................................. 103

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page IX

List of Figures

Figure 1: Data flow and processing chains involved in fire_cci global processing ................................................ 3 Figure 2: Workflow for the test site pre-processing of (A)ATSR, MERIS and VGT data ..................................... 6 Figure 3: Workflow for the global pre-processing of (A)ATSR, MERIS and VGT data ....................................... 7 Figure 4: Geometric transformation of polygons (triangles) from the input image grid to the output image grid

using the transformation of the geo-layer. For test-site images this approach is applied for all sensor

data. For global processing, only (A)ATSR data are processed in this way. ......................................... 9 Figure 5: Geometric transformation of polygons (triangles) from the input image grid to the output image grid

using the information of the geolayer ................................................................................................... 12 Figure 6: Work flow of land-water masking process for test-site and global processing. There is no difference in

land-water masking for test-site or global processing .......................................................................... 14 Figure 7: Fuzzy operator Fdn, Fhi and Fup ........................................................................................................... 17 Figure 8: Work flow of cloud, snow/ice, haze and spectral shadow masking for test-site and global processing.

Input for the masking procedure is geo-referenced data ...................................................................... 18 Figure 9: General workflow for the atmospheric correction using ATCOR-WS .................................................. 26 Figure 10: Logical flow chart of the combined atmospheric and topographic correction ..................................... 28 Figure 11: (A)ATSR ASCII initial file example ................................................................................................... 31 Figure 12: Flow diagram of the (A)ATSR BA processing chain .......................................................................... 33 Figure 13: Flow diagram of step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and

observational statistics ......................................................................................................................... 36 Figure 14: Flow diagram of step 2 - Time-series filtering .................................................................................... 37 Figure 15: Flow diagram of step 3 - Change point detection and ranking ............................................................ 39 Figure 16: Flow diagram of step 4 –z_scoring ...................................................................................................... 40 Figure 17: DIMACS format file example ............................................................................................................. 41 Figure 18: Flow diagram of step 5 - MRF segmentation ...................................................................................... 42 Figure 19: VGT ASCII initial file example........................................................................................................... 43 Figure 20: Flow diagram of the VGT BA processing chain.................................................................................. 45 Figure 21: Flow diagram of step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and

observational statistics ......................................................................................................................... 49 Figure 22: Flow diagram of step 2 and 3 – Maxima/Minima time series extraction and filtering ........................ 50 Figure 23: Flow diagram of step 2 - Time-series filtering .................................................................................... 51 Figure 24: Flow diagram of step 3 - Change point detection and ranking ............................................................ 53 Figure 25: Flow diagram of step 4 –z_scoring ...................................................................................................... 55 Figure 26: DIMACS format file example ............................................................................................................. 56 Figure 27: Flow diagram of step 5 - MRF segmentation ...................................................................................... 57 Figure 28: Algorithm general flow........................................................................................................................ 58 Figure 29: Data filtering logical flow .................................................................................................................... 60 Figure 30: Build composites logical flow ............................................................................................................. 61 Figure 31: NIR composite logical flow ................................................................................................................. 61 Figure 32: GEMI logical flow ............................................................................................................................... 62 Figure 33: BA detection logical flow .................................................................................................................... 63 Figure 34: Identify seeds logical flow ................................................................................................................... 63 Figure 35: Region growing logical flow ............................................................................................................... 64 Figure 36: Compute layers logical flow ................................................................................................................ 65 Figure 37: BA monthly product and format conversion logical flow ................................................................... 65 Figure 38: Overview of the inputs and outputs to the data processing chain P1 ................................................... 73 Figure 39: Overview of the component parts of the processing chain P1 ............................................................. 75 Figure 40: Preparation of datasets P1.1 ................................................................................................................. 79 Figure 41: The processing steps for the primary merge at 1/120°, P1.2 ................................................................ 81 Figure 42: Processing steps for the secondary merge, P1.3 .................................................................................. 88 Figure 43: Processing steps for the generation of the pixel product, P1.4 ............................................................ 93 Figure 44: Processing steps for the grid product, P1.5 ........................................................................................ 101

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page X

List of Tables Table 1: Description of symbols applied in flow charts .......................................................................................... 2 Table 2: Variables used in the (A)ATSR BA processing chain ............................................................................ 35 Table 3: Parameters set in the (A)ATSR BA processing chain ............................................................................. 35 Table 4: Variables used in the VGT BA processing chain .................................................................................... 47 Table 5: Parameters set in the VGT BA processing chain .................................................................................... 47 Table 6: Variables in the MERIS BA processing chain ........................................................................................ 67 Table 7: Parameters set in the MERIS BA processing chain ................................................................................ 67 Table 8: Variables in the merged product chain .................................................................................................... 76 Table 9: Parameters set in the merge processing chain ......................................................................................... 77

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Models Version 2 Page XI

List of Abbreviations

AATSR

ACL

AMORGOS

Advanced Along-Track Scanning Radiometer

Average Confidence Level

Accurate MERIS Ortho-Rectified Geo-location Operational Software

AOI

AOT

ATBD

ATCOR-WS

Area of interest

Aerosol Optical Thickness

Algorithm Theoretical Basis Document

Atmospheric Correction and haze Reduction for Wide field Sensors

ASTER Advanced Spaceborne Thermal Emission and Reflection Radiometer

ASTER-GDEM ASTER Global Digital Elevation Map

BAS Burned Area Status,

BEAM Name of an open source platform for viewing, analysing and processing of

remote sensing raster data

CON Confidence Level

DEM Digital Elevation Model

DIMAP Digital Image Map (format for SPOT products, introduced for the SPOT 5)

ENVISAT ENVIronmentalSATellite

FRS Full Resolution, full Swath

GAF Name of a German company

GCP Ground Control Point

GDAL Geospatial Data Abstraction Library

GLS2000 Global Land Survey 2000

GlobCover Global land cover based on MERIS data

ICP Independent Control Point

INIA Instituto Nacional de Investigación y Tecnología Agraria y Alimentaria

IRD L'Institut de recherche pour le développement

LDC Last clear DeteCtion prior to burn

LSCE Laboratoire des Sciences du Climat et l'Environnement

lwmd1

lwmd2

lwms

dynamic land-water mask, processes in step 1

dynamic land-water mask, processes in step 2

land water mask, static

MERIS Medium Resolution Imaging Spectrometer, on board of ENVISAT

MAD Median absolute deviation.

MAP-MRF Maximum A Posteriori – Markov Random Field

MODIS MODerate Resolution Imaging Spectrometer (on board of TERRA and AQUA)

NDSI Normalised Difference Snow Index

NDVI Normalised Difference Vegetation Index

NetCDF Network Common Data Form

NIR Near Infra-Red

NOC Number of Cloud Observations

NOS Number of Sensor Observations

NOV Number of Valid Observations

PELT Pruned Exact Linear Time

RM Repeated Median

RR Reduced Resolution

SPOT

SRTM

Système Pour l’Observation de la Terre

Shuttle Radar Topography Mission

SWBD SRTM Water Body Data

SWIR Short Wave Infra-Red

UAH University of Alcalá de Henares

UL University of Leicester

VEGETATION CNES Earth observation sensor onboard SPOT-4/5 (VGT)

fire_cci


Issue/Rev-No.: 2.2

D3.7 Detailed Processing Model Version 2 Page 1

1 Executive Summary

1.1 Scope

This document is the Detailed Processing Model document for the full fire_cci burned area product

processing chain. It covers the VGT/(A)ATSR/MERIS pre-processing, as defined in ATBD I

(Bachmann et al. 2014), burned area classification algorithms for all sensors and burned area map

merging, as defined in ATBD II (Pereira et al. 2014) and ATBD III (Bradley and Tansey 2014). The

input and output products generated by the processing chain covered in this document are described in

Input Output Data Definition document (Krauß et al. 2013). All the processing is in compliance with

the System Interface Definition & Processor Guidelines Document (Krauß et al. 2014).

This document takes the elements of the ATBDs and describes

• Key principles of the algorithms.

• Top-down decomposition of its composition of the software into its components.

• Detailed list and description of the variables used in the mathematical equations.

• Computer pseudo-code

1.2 Purpose

The Detailed Processing Model shall serve as an implementation-independent description of the data

processing to be performed within the fire_cci data processing. It describes all processing steps in

terms of algorithms, functions and data structures required for the generation of the final products. As

such it is intended to serve as:

• a functional requirements specification for the data processing modules.

• a basis for the estimate of the computation resources requirements for the data processing.

1.3 Organisation

This document includes the following sections:

Section 1: Introduction gives the scope, purpose, reference and applicable documents and list of

acronyms, notations and conventions used in this document.

Section 2: Overview gives a brief overview of the processing steps described in this document.

Section 3: Pre-processing describes the processing steps required to generate the atmospheric

corrected data products which serves as input product for the Cloud and water masking and

BA classification.

Section 4: Describes in detail the processing steps required to generate the VGT based burned area

product.

Section 5: Describes in detail the processing steps required to generate the ATSR2/AATSR based

burned area product.

Section 6: Describes in detail the processing steps required to generate the MERIS based burned area

product.

Section 7: Describes in detail the processing steps for the data merging and generation of the pixels

and grid based products.

A detailed description of all algorithms developed and equations applied are documented in the

respective Algorithm Theoretical Base Documents I, II and III (Bachmann et al. 2014, Pereira et al.

2014, Bradley and Tansey 2014).

fire_cci


Issue/Rev-No.: 2.2


Table 1: Description of symbols applied in flow charts

Symbol Meaning

Something that is put in or put out, i.e. a data product, auxiliary data,

parameters, etc.

A process (a process is doing something that usually involves more

than a subroutine)

A subroutine/procedure/function (a subroutine is a simple process,

usually the implementation of an equation)

A Boolean decision

Input /Output database/repository

A line indicating the direction of flow

In the sections where pseudo-code is needed, all the code is in italic, enclosed between C-style

comment delimiters, /* */, and with shaded background.

2 Processing Chain Overview

The main purpose of fire_cci processor chain is the development of a global burned area product based

on the integration of the 3 burned area classifications based on the AATSR/ATSR2, MERIS and

SPOT-VEGETATION sensors. The processing chain, intended to be fully automatic is divided into 3

main steps (Figure 1): Data pre-processing, Burned Area (BA) classification and BA product merging.

Two end products are delivered. The first end product is based on the requirements of the climate user

modelling group. This product is a global map of each time step. The second end product is a pixel-

based BA map representing each continent.

fire_cci


Issue/Rev-No.: 2.2


Figure 1: Data flow and processing chains involved in fire_cci global processing

The pre-processing chain imports level1B MERIS, ATSR2 and AATSR data and S1 SPOT-

VEGETATION daily data, performs geometric and radiometric correction, merges images into daily

composites and produces cloud-, haze-, snow-, shadow- and water-masks and additional layers of

information to be passed on to the following step. The BA classification runs each sensor specific

burned area classification algorithm and produces burned area maps, confidence, temporal uncertainty

and observational statistics that are passed to the following step. The last step merges the MERIS,

ATSR2/AATSR and SPOT-VEGETATION BA classifications into two different output formats,

pixel-based and grid-based product.

fire_cci


Issue/Rev-No.: 2.2


3 Pre-Processing chain

3.1 Introduction

The global pre-processing chain uses original level-1B satellite data from MERIS-FRS, and

(A)ATSR) or level 2 satellite data from VGT.

The pre-processing performs following tasks:

Importing satellite data to standardised formats (including all relevant metadata)

Image-to-image correlation to a reference (Landsat) for generation of ground control points

Correction of sensor model or geo-layer included in original data using the ground control

points

Ortho-rectification of the original input imagery to the required output projection and

resolution

Derivation of a cloud mask containing fuzzy probabilities for topographic and cloud shadow,

snow/ice and haze

Derivation of a water mask containing a static and a dynamic water mask

Atmospheric correction of the ortho-rectified images

Exporting all relevant images in the requested output format

Depending on the purpose (AOI processing vs. global processing) and the input data (which sensor)

some parts are handled differently as described below in detail.

The pre-processing chain is designed for automated mass data processing without manual interaction.

For this many parameters can be tuned for optimisation of the result.

3.2 Data preparation

The data preparation for the pre-processing chain includes the provision of the required input data for

the processor. In case of the global pre-processing simply all images have to be copied as archives (zip

or tar.gz) to a directory accessible by all processing nodes.

In case of the test site processing the input data intended for processing has to be tailed to the area of

interest (AOI) prior ingestion. This is done by calling “cuttestsites” for each scene. This programme

uses the tools “geochildgen” and “pconvert” from Brockmann Consult for extracting the areas of

interest in turn for each test site from an original scene and converts the results to BEAM DIMAP

format which are used as input for the pre-processing chain.

For global pre-processing of (A)ATSR L1b data, full paths of are read and then processed. For global

pre-processing of MERIS-FRS data each segment of a path is read and processed. For global VGT

pre-processing, the daily global maps are tailored to 10° x 10° tiles.

3.3 Pre-processing chain overview

The pre-processing consists of the geo-correction, ortho-rectification, cloud masking, snow/ice

masking, haze masking, water masking and atmospheric correction including image and data handling.

These steps are described in detail later and in the ATBD I v2.2 (Bachmann et al. 2014). The logical

work flow of the different pre-processing steps for test site processing is illustrated in Figure 2, and for

global processing in Figure 3.

The pre-processing consists of the geo-correction, ortho-rectification, cloud masking, snow/ice

masking, haze masking, water masking and atmospheric correction including image and data handling.

These steps are described in detail later and in the ATBD I v2.2 (Bachmann et al. 2014). The logical

work flow of the different pre-processing steps for test site processing is illustrated in Figure 2, and for

global processing in Figure 3.

fire_cci


Issue/Rev-No.: 2.2


The following abbreviations apply for special branches:

Processing type:

G=global processing

T=AOI/test site processing

Sensor:

A=ATSR2/AATSR

M=MERIS FRS and RR

V=SPOT Vegetation

Import to standardized format (all sensors)

if G and M: call AMORGOS (PO ID ACR GS 0003, 2007) for correction of the

original data prior to import

if G: split to small, overlapping tiles (M: 1,500 px by 1,500 px, A: 500 px by 800 px,

V: 1,100 px by 1,100 px)

for all tiles do:

Retrieve DEM data from database (SRTM, >60°/<-60°: ASTER-GDEM V2 for test

site processing; GETASSE for global processing)

if T or A:

Retrieve reference data from database

Project imported image coarse to reference using corners

Apply matching

Correct geolayer

Otherwise:

(if G and M: geo-correction already done using AMORGOS)

if G and V: no geo-correction performed since input is already worldwide

daily composite

Ortho-rectify image using DEM and geolayer

Generate static and dynamic water-mask

Generate cloud/shadow/snow/haze mask

if A or M: apply atmospheric correction using ATCOR-WS

Move sun and scan-angle layers to images

if T: cut all results to requested AOI

if G: grid all results to requested worldwide gridding

collect and export all needed data

fire_cci


Issue/Rev-No.: 2.2


Figure 2: Workflow for the test site pre-processing of (A)ATSR, MERIS and VGT data

fire_cci


Issue/Rev-No.: 2.2


Figure 3: Workflow for the global pre-processing of (A)ATSR, MERIS and VGT data

fire_cci


Issue/Rev-No.: 2.2


3.4 Algorithm– Geometric Correction

3.4.1 Overview

In the geometric correction step the coarse geo-referenced scenes given by a geolayer are corrected

using existing ground control points. In fire_cci the geometric correction is applied in case of test site

processing for all sensors while in case of global processing it is applied for ATSR2 and AATSR data.

For global processing MERIS data is already geometrically corrected during import using the

Accurate MERIS Ortho-Rectified Geo-location Operational Software (AMORGOS; PO ID ACR GS

0003, 2007). AMORGOS carries out geometric correction based on orbit and attitude data as well as a

Digital Elevation Model (GETASSE) without the need for ground control points. According to the

AMORGOS Software user manual (PO ID ACR GS 0003, 2007) the following calculation is

performed. “For each MERIS sample, an ortho-geolocation algorithm computes the first intersection

between the pixel’s line of sight and the Earth surface, represented by interpolation of the

GETASSE30 high resolution Digital Elevation Model (DEM) cells elevations on top of the reference

ellipsoid. Line of sight is determined using its pointing vector expressed relative to the satellite, the

satellite location, and attitude that are in turn determined from appropriate Orbit and Attitude files

using the appropriate CFI routines. Location of the intersection is expressed as longitude, geodetic

latitude, and geodetic attitude.”

The accuracy of this approach for geo-location is better than 80 m RMSE and better than 52 m for co-

registration (Arino et al. 2007)

3.4.2 Logical Flow

The work flow for the geometrical correction is shown in Figure 4.

fire_cci


Issue/Rev-No.: 2.2


Figure 4: Geometric transformation of polygons (triangles) from the input image grid to the output image

grid using the transformation of the geo-layer. For test-site images this approach is applied for all sensor

data. For global processing, only (A)ATSR data are processed in this way.

fire_cci


Issue/Rev-No.: 2.2


3.4.3 List of Variables

Bold names (e.g. A) denoting images in capitals, point lists as p, transformations as a or c and other

individual variables as lower case letters.

3.4.4 Processing step 1: Extraction of DEM

Extract for each imported image/tile A a digital elevation model D fitting the data and including an

additional extent of about 100 km from a database containing the SRTM DEM for latitudes between -

60°N and 60°S degree and the ASTER GDEM V2 for higher/lower latitudes. For global pre-

processing GETASSE is used as DEM.

3.4.5 Processing step 2: Extraction of reference image

Extract for each imported image/tile A a reference image R fitting the data and including an additional

extent of about 100 km from a database containing a worldwide Landsat reference of panchromatic

images allowing an absolute accuracy of CE90 below 50 m. The additional extent of about 100 km is

selected as typical margin in order to allow inaccuracies in geo-location of A.

3.4.6 Processing step 3: Transformation of A to T

Do a coarse affine transformation a of the imported image A onto R as T.

3.4.7 Processing step 4: Image-to-image matching

Create a window based image-to-image-matching T and R providing matching points p of accuracy of

about one pixel.

3.4.8 Processing step 5: Iterative least-square-matching until sub-pixel accuracy

Do a local least squares matching for each of the points p on T and R providing p' with a sub pixel

accuracy of about 0.1 pixels.

3.4.9 Processing step 6: Inverse Transformation back to A

Apply the inverse affine transformation aT to the R-pixel-coordinates in points p' transposing them

back to A.

3.4.10 Processing step 7: Selection of GCPs and ICPs

Grid points p' over a 25x25 grid over A selecting the best matched point for each grid cell as ground

control point (GCP) and all others as independent control points (ICP). Select for each GCP/ICP the

height from D at its geo position; now each GCP/ICP consists of a set of (x, y, h, lon, lat) denoting x,

y as pixel coordinates in A, h the height at lon, lat extracted from D and lon,lat from the reference

image R.

3.4.11 Processing step 8: Estimation of affine transformation

Estimate a local affine transformation c of the geolayer in A to fit on all GCPs.

3.4.12 Processing step 9: Calculation of deviations to measured ICPs

Calculate the deviations of the affine transformed geolayer to the measured ICPs as RMSE values.

3.4.13 Processing step 10: Applying affine transformation to complete geolayer

Apply the affine transformation c to all coordinates in the geolayer.

fire_cci


Issue/Rev-No.: 2.2


3.5 Algorithm – ortho-rectification

A detailed description of the algorithms developed and applied is documented in the ATBD I v2.2

(Bachmann et al. 2014).

3.5.1 Overview

The ortho-rectification creates from an original image containing a geolayer an image in a requested

geographic projection and resolution by projecting each pixel of the original image to the position

given in the geolayer and interpolating correctly.

Images consist of optical data and congruent flag layers, which have to be resampled in a different

way. The optical image data are resampled using bilinear interpolation, which has to be shown as a

well resampling technique with spectral error minimizing properties (Schläpfer et al. 2007). For the

flag layers the Nearest Neighbour resampling has to be applied in order not to destroy the semantic

pixel values.

3.5.2 Logical Flow

The logical flow of the pre-processing step ortho-rectification is described in bullets instead of a flow

diagram.

Split image to image-layers and flag-layers

For each image layer:

◦ Estimate bounding box of resulting image in requested projection

◦ for each pixel at requested resolution in this bounding box:

▪ search neighbours to this coordinate in x and y direction (x should be interpreted as

column and y as line) and interpolate bilinear the corresponding values in the original

image

For each flag layer:

◦ Estimate bounding box of resulting image in requested projection

◦ for each pixel at requested resolution in this bounding box:

▪ search nearest point in geolayer and take the corresponding value from the original

image

Join ortho-rectified image-layers and flag-layers again in same order

3.5.3 Processing Step 1: Splitting of image

Split the image into sensed image layers and the flag layers, because of different processing

afterwards.

3.5.4 Processing Step 2: Bounding Box

Calculate bounding box based on the geolayer information for the output image (valid for sensed

image and flag image layer).

3.5.5 Processing Step 3: Resampling of image layer

Transform triangles from the input image grid to the output image grid according to the values given

in the geolayer. Fill resulting triangle in the output grid with bilinear interpolated values from the three

grey values at the position of the input image. The output image grid is defined by the requested pixel

size given in metre or degree depending on the used map projection. Only locations inside the

bounding box are processed.

fire_cci


Issue/Rev-No.: 2.2


3.5.6 Processing Step 4: Resampling of flag layer

Transform triangles from the input image grid to the output image grid according to the values given

in the geolayer. Fill resulting triangle in the output grid with nearest neighbour values from the three

grey values at the position of the input image. The output image grid is defined by the requested pixel

size given in metre or degree depending on the used map projection. Only locations inside the

bounding box are processed.

Figure 5: Geometric transformation of polygons (triangles) from the input image grid to the output image

grid using the information of the geolayer

3.5.7 Processing Step 5: Joining of Image

Join image layer and flag layer of the output image according to the input image layout.

PixelPixel

Input Pixel GridOutput Pixel Grid

fire_cci


Issue/Rev-No.: 2.2


3.6 Algorithm – Water masking

A detailed description of the water masking algorithms developed and applied is documented in the

ATBD I v2.2 (Bachmann et al. 2014).

3.6.1 Overview

For generating the final water mask intermediate water masks are calculated based on the ortho-

rectified images: first a static water mask and second two dynamic water masks are developed. For the

static water mask the SRTM Water Body Data (SWBD) is used which can be applied for all areas

from 60°N to 60°S. For all other regions the GSHHS water mask is applied. Both data sets are

imported as shape files. A precise geo-referencing of both the ortho-rectified images and the imported

shape files is mandatory for generating the final water mask.

3.6.2 Logical Flow

The work flow of the water masking is presented in Figure 6.

The following steps are performed:

For each ortho-rectified image O:

create an empty image fitting to the ortho-rectified image but with higher resolution s

For all water polygons touching within bounding box of the image

write a binary mask for all pixels inside

scale resulting image down by s averaging all s x s pixels as static water mask WS

Feed O and WS to generation of dynamic water mask WD

Join WS and WD as channel 1 and 2 to water mask WM

fire_cci


Issue/Rev-No.: 2.2


Figure 6: Work flow of land-water masking process for test-site and global processing. There is no

difference in land-water masking for test-site or global processing

fire_cci


Issue/Rev-No.: 2.2



The reflectances of the different sensors are input for the water masking algorithms. In addition, the

coordinates of each pixel (lat/long) are used for the co-registration of the satellite data with the static

masks.

The variables for the water masking are:

lwms: land water mask static as intermediate output

lwmd1: land water mask dynamic 1

lwmd2: land water mask dynamic 2

lwmCAT: final land water mask as output

3.6.4 Processing steps

The processing steps are visualised in Figure 6. Our new approach is based on the determination and

use of mean values representative for regional water bodies. Therefore, the pre-processing of a scene

or a full path needs a sub setting of the complete input-data set in well-defined frames. Three

processing steps can be defined as:

First, a static land-water mask is overlaid to each frame. A precise geo-location of each frame is

mandatory. The result of step 1 is a mask of the percentage water content of each pixel. With regard to

the dynamical behaviour of the water body the identified water pixels are used only as static

candidates for water.

The second step is the classification of water pixels using the static water pixels a training set. The

result of the second step is the identification of water pixels or in principle all dark pixels which are

assigned as dynamic candidates for water pixels.

The third and final processing step of the land-water masking algorithm is the data fusion of the results

of the first and second steps. The data fusion is based at first on the reliability of the identified water

objects in the static as well as in the dynamic masks on regional level. Corresponding water pixels are

assigned as stable water pixels within a new resulting mask of the Fusion Processor. Using the spectral

characteristics of these stable water pixels result in the mean water spectrum representative for the

frame under investigation (DySLEM). In the next step of the Fusion Processor the spectrum of all

static water pixels not identified as stable is tested using the mean spectrum including an off-set. When

the pixels of static mask are accepted as water pixel based on the mean spectrum as reference they are

included in the final water mask. In the last step of the Fusion Processor all pixels assigned as dynamic

candidates are investigated applying thresholds for single bands, band ratios and band differences. The

dynamic candidates are a result of the spectral classification of the second step and not identified in

the static mask. The accepted dynamic candidates are finally assigned as water pixels in the Fusion

Processor and represent temporal water bodies.

3.6.5 Processing step

The processing steps for the land-water masking are performed as shown in the pseudo code:

/*

# cmmakewatermask creates an watermask layer OWM containing 8 bit values denoting

# band 1: amount of water mask in the corresponding pixel of ortho image

# band 2: different water classes (dynamic water classes)

# get water subpixel values from 0 (no water) to 255 (100 % water

Import ortho image

call csmakewatermask

#csmakewatermask generates a water density mask from a given image using different approaches:

# - first all SRTM watermask tiles between -60...60 degree latitude will be sampled

# -second below -60 and above 60 degree latitude GSHHS watermask tiles will be sampled

if input file latitude < 60°N and > 60°S then

fire_cci


Issue/Rev-No.: 2.2


searching SRTM tiles for the input image

creating list of available and needed .shp-files

joining all .shp-files

rasterizing shapes to image

calculating stable watermask for ortho image and SRTM coverage

writing watermask also as vector layer

else

getting GSHHS coverage

rasterizing shapes to image

calculating stable watermask for ortho image and GSHHS coverage

writing watermask also as vector layer

endif

generate stable water pixel mask (lwms) for input image

call csmakedynwatermask

generate dynamic water mask 1 (lwmd1)

generate dynamic water mask 2 (lwmd2)

calculating stable water pixels (lwmd1 && lwmd2 &&lwms>60% == class.1)

calculating mean spectrum for stable water pixels

exclude class.1 pixels from static water mask (reduced static mask)

look for pixels with mean spectrum in reduced static mask == class.2

exclude class.1 pixel from dynamic water mask 1 and 2 (reduced dynamic mask)

look for pixels with mean spectrum in reduced dynamic mask == class.3

look for pixels which fulfil additional spectral criteria in reduced dynamic mask

== class.4

combining static and dynamic watermask to two layers product

*/

3.7 Algorithm – Cloud, snow/ice, haze and spectral shadow masking

A detailed description of the cloud, snow/ice, haze and spectral shadow masking algorithms developed

and applied is documented in the ATBD I 2.2 (Bachmann et al. 2014).

3.7.1 Overview

In this step the cloud, snow/ice, haze and spectral shadow masks are calculated for the ortho-rectified

image. The masking of topographic shadow is performed later in the ATCOR-WS processing step.

3.7.2 Logical Flow

The logical flow of the cloud, snow/ice, haze and spectral shadow masking is shown in Figure 8. Each

mask contains a continuous value from 0 to 100, indicating a confidence value as described in ATBD-

I-pre-processing.

The logical flow of the pre-processing steps “masking” is described in bullets of a flow diagram.


◦ Select for each sensor the appropriate wavelengths for the cloud, snow/ice, shadow and haze

masking algorithms and calculate the new, additional bands as e.g. “bright” or “flat”

◦ According to the sensor choose appropriate cloud, snow/ice, haze and shadow masking

algorithm

fire_cci


Issue/Rev-No.: 2.2


◦ For each pixel:

▪ Calculate fuzzy properties for cloud, snow/ice, shadow and haze

▪ In the case of SPOT-VEGETATION: copy flag geometrical shadow from flag layer of

ortho-rectified image as fifth channel to cloud mask

▪ Write mask image CM with the bands “cloud”, “snow/ice”, “shadow” and “haze”

The fuzzy based rules used for the masking algorithms are defined as (see Figure 7):

◦ Fuzzy_up: Fup 𝑐,𝑑(𝑥) = {

0 𝑖𝑓 𝑥 < 𝑐𝑥−𝑐

𝑑−𝑐 𝑖𝑓 𝑐 ≤ 𝑥 ≤ 𝑑

1 𝑖𝑓 𝑥 > 𝑑

◦ Fuzzy down: Fdn 𝑎,𝑏(𝑥) = {

0 𝑖𝑓 𝑥 < 𝑎𝑏−𝑥

𝑏−𝑎 𝑖𝑓 𝑎 ≤ 𝑥 ≤ 𝑏

1 𝑖𝑓 𝑥 > 𝑏

◦ Fuzzy high: Fhi 𝑎,𝑏,𝑐,𝑑(𝑥) =

{

0 𝑖𝑓 𝑥 < 𝑎𝑥−𝑎

𝑏−𝑎 𝑖𝑓 𝑎 ≤ 𝑥 ≤ 𝑏

1 𝑖𝑓 𝑏 ≤ 𝑥 ≤ 𝑐 𝑑−𝑥

𝑑−𝑐 𝑖𝑓 𝑐 ≤ 𝑥 ≤ 𝑑

0 𝑖𝑓 𝑥 > 𝑑

Fuzzy low: Flo 𝑎,𝑏,𝑐,𝑑(𝑥) = 1 − Fhi 𝑎,𝑏,𝑐,𝑑(𝑥)

◦ Fuzzy and: Fand(a,b,…) = min(a,b,…)

Figure 7: Fuzzy operator Fdn, Fhi and Fup

fire_cci


Issue/Rev-No.: 2.2


Figure 8: Work flow of cloud, snow/ice, haze and spectral shadow masking for test-site and global

processing. Input for the masking procedure is geo-referenced data

fire_cci


Issue/Rev-No.: 2.2



The variables for masking cloud, snow/ice and haze are the reflectances of the different sensors as

well as the thermal channel for (A)ATSR.

Variables which are used for the different masks in intermediate steps are:

NDSI: Normalised difference snow index (all sensors)

Mean: The mean value for the visible and NIR bands (all sensors)

Std: The standard deviation of the mean (all sensors)

Bright: Ratio of the square of the mean and the standard deviation (all sensors)

Flat: Ratio of standard deviation to mean (only for MERIS)

Fuzzy_bright: Fuzzy value based on the variable bright (all sensors)

Fuzzy_flat: Fuzzy value based on the variable flat (only for MERIS)

Fuzzy_ndsi: Fuzzy value based on the variable ndsi (all sensors)

Fuzzy_temp: Fuzzy value based on the variable temperature (only for (A)ATSR)

Fuzzy_mean4dark: Fuzzy value based on the variable mean (all sensors)

Fuzzy_bright4haze: Fuzzy value based on the variable bright (all sensors)

The output variables for the masks are:

Fuzzy_snow: Fuzzy value based on the variables fuzzy_bright, fuzzy_ndsi and fuzzy_temp

(only for (A)ATSR)

Fuzzy_snow: Fuzzy value based on the variables fuzzy_bright and fuzzy_ndsi (for VGT and

MERIS))

Fuzzy_cloud: Fuzzy value based on the variables fuzzy_mean4dark and fuzzy_bright (for

(A)ATSR and VGT)

Fuzzy_cloud: Fuzzy value based on the variables fuzzy_bright and fuzzy_flat (only for

MERIS)

Fuzzy_haze: Fuzzy value based on the variables fuzzy_snow, fuzzy_bright4haze,

fuzzy_mean4dark (for all sensors)

3.7.4 Processing step

The generation of the different masks is performed on a pixel by pixel step. All calculations are done

in matrix operation for reducing computation time.

For (A)ATSR the following calculations are necessary, shown as an example of pixel by pixel

calculation:

/*

For all pixel do

calculate mean

Input: TOA reflectances at 555nm, 659 nm and 865nm

- Calculate the mean value of the input reflectances

Output: mean

calculate std

Input: TOA reflectances at 555nm, 659nm and 865nm, mean

- Calculate the standard deviation of the input reflectances

Output: std

calculate bright

Input: mean, std

- Calculate the ratio of pow(mean,2) to std

Output: bright

calculate ndsi

fire_cci


Issue/Rev-No.: 2.2


Input: TOA reflectances at 555nm and 1610nm

- Calculate the difference of TOA reflectances at 555nm minus TOA reflectances at

1610nm

- Calculate the sum of TOA reflectances at 555nm and TOA reflectances at 1610nm

- Divide the difference by the sum

Output: ndsi

calculate fuzzy_bright

Input: bright,

threshold_low=0.1,

threshold_high=2.0

- Apply Fup to bright using threshold_low and threshold_high

Output: fuzzy_bright

calculate fuzzy_ndsi

Input : ndsi

threshold_low=0.3

threshold_high=0.6

- Apply Fup to ndsi using threshold_low and threshold_high

Output: fuzzy_ndsi

calculate fuzzy_temp

Input: Brightness temperature at 11µm

threshold_low=250

threshold_high=300

- Apply Fup to brightness temperature using threshold_low and threshold_high

Output: fuzzy_temp

calculate fuzzy_snow

Input: fuzzy_bright

fuzzy_temp

fuzzy_ndsi

land_flag

std

- Calculate the mean of fuzzy_bright, fuzzy_temp and fuzzy_ndsi if pixel is( land_flag

& std <0.13 & std > 0.005)

Output: fuzzy_snow

calculate fuzzy_mean4dark

Input: mean

threshold_low=0.15

threshold_high=0.25

- Apply Fdn to mean using threshold_low and threshold_high

Output: fuzzy_mean4dark

calculate fuzzy_cloud

Input: fuzzy_bright

fuzzy_mean4dark

fuzzy_snow

- Calculate !fuzzy_mean4dark

- Calculate mean of !fuzzy_mean4dark and fuzzy_bright if pixel is (fuzzy_snow < 0.9)

Output: fuzzy_cloud

calculate bright4haze

Input: bright

threshold_low1=0.7

threshold_high1=1.0

threshold_low2=2.0

threshold_high2=2.3

- Apply Fhi to bright using threshold_low1, threshold_high1, threshold_low2 and

threshold_high2

Output: fuzzy_bright4haze

fire_cci


Issue/Rev-No.: 2.2


calculate fuzzy_haze

Input: fuzzy_snow

fuzzy_bright4haze

fuzzy_mean4dark

soil_flag

- Calculate !fuzzy_snow

- Calculate !mean4dark

- Calculate mean of !fuzzy_snow, !fuzzy_mean4dark and Fuzzy_bright4haze if pixel is

!soil_flag

Output: fuzzy_haze

calculate shadow

Input: TOA reflectances at 865nm

TOA reflectances at 1610nm

threshold_865_low1=0.035

threshold_865_high1 = 0.045

threshold_865_high2=0.1


threshold_1610_low= 017

threshold_1610_high=0.23

land_flag

- Calculate temp1 = Fhi of TOA reflectances at 865nm

- Calculate temp2 = Fdn of TOA reflectances at 1610nm

- Calculate Fand of temp1 and temp2 if pixel is land_flag

Output: shadow

end d

*/

For MERIS the following calculations are necessary:

/*

For all pixels do

calculate mean

Input: All TOA reflectances except band 11 and band 15


Output: mean

calculate std

Input: All TOA reflectances except band 11 and band 15, mean


Output: std

calculate bright

Input: mean

std

- Calculate the ratio of pow(mean,2) to std

Output: bright

calculate ndsi

Input: TOA reflectances at 865nm and 890nm

- Calculate the difference of TOA reflectances at 865nm minus TOA reflectances at

890nm

- Calculate the sum of TOA reflectances at 865nm and TOA reflectances at 890nm


Output: ndsi

calculate flat

fire_cci


Issue/Rev-No.: 2.2


Input: mean

std

- Calculate the ratio of std to mean

Output: flat


Input: bright

threshold_low=1.0

threshold_high=2.0




Input : ndsi

threshold_low=0.005

threshold_high=0.01


Output: fuzzy_ndsi

calculate fuzzy_flat

Input: flat

threshold_low=0.1

threshold_high=0.2

- Apply Fdn to flat using threshold_low and threshold_high

Output: fuzzy_flat


Input: fuzzy_bright

fuzzy_flat

fuzzy_ndsi

- Calculate the mean of the input data

Output: fuzzy_snow


Input: mean

threshold_low=0.15

threshold_high=0.25



calculate soil

Input: TOA reflectances at 510nm, 560nm 665nm, 680nm, 753nm and 778nm

- Check if TOA reflectances increase monotonically towards the red

- If TOA reflectances increase monotonically then 1 else 0

Output: soil_flag


Input: fuzzy_bright

fuzzy_ndsi

fuzzy_flat

- Calculate mean of fuzzy_flat and fuzzy_bright if pixel is (fuzzy_snow < 0.9 & !soil)

Output: fuzzy_cloud


Input: bright

threshold_low1=0.2

threshold_high1=0.8

threshold_low2=1.0

threshold_high2=2.0


threshold_high2


fire_cci


Issue/Rev-No.: 2.2



Input: fuzzy_snow

fuzzy_bright4haze

fuzzy_mean4dark

soil_flag



- Calculate mean of !fuzzy_snow, !fuzzy_mean4dark and fuzzy_bright4haze if pixel is

!soil_flag

Output: fuzzy_haze

calculate shadow






land_flag

- Calculate Fhi of TOA reflectances at 865nm if pixel is land_flag

Output: shadow

end do

*/

For VGT the following calculations are necessary:

/*

For all pixels do

calculate mean

Input: BOA reflectances at 450nm, 645nm and 835nm


Output: mean

calculate std

Input: BOA reflectances at 450nm, 645nm and 835nm, mean


Output: std

calculate bright

Input: mean

std

- Calculate the ratio of pow(mean,2) and std

Output: bright

calculate ndsi

Input: BOA reflectances at 450nm and 1665nm

- Calculate the difference of BOA reflectances at 450nm minus BOA reflectances at

1665nm

- Calculate the sum of BOA reflectances at 450nm and BOA reflectances at 1665nm


Output: ndsi

calculate ndvi

Input: BOA reflectances at 645nm and 835nm

- Calculate difference between BOA reflectances at 835nm and 645nm

- Calculate sum of BOA reflectances at 835nm and 645nm

- Calculated ratio of difference to sum

Output: ndvi

fire_cci


Issue/Rev-No.: 2.2



Input: bright

threshold_low=0.1

threshold_high=2.0




Input : ndsi

threshold_low=0.3

threshold_high=0.6


Output: fuzzy_ndsi

calculate fuzzy_ndvi

Input: ndvi

threshold_low=-0.01

threshold_high=0.0

- Apply Ddn to ndvi using threshold_low and threshold_high

Output: fuzzy_ndvi


Input: fuzzy_bright

fuzzy_ndsi

fuzzy_ndvi

- Calculate the mean of fuzzy_bright and three times fuzzy_ndsi if pixel is fuzzy_ndvi

Output: fuzzy_snow


Input: mean

threshold_low=0.15

threshold_high=0.25




Input: fuzzy_bright

fuzzy_snow

fuzzy_mean4dark



- Calculate mean of !fuzzy_snow, !fluzzy_mean4dark and 4 times fuzzy_bright if pixel

is (fuzzy_snow < 0.9)

Output: fuzzy_cloud


Input: bright

threshold_low1=0.7

threshold_high1=1.0

threshold_low2=2.0

threshold_high2=2.3


threshold_high2



Input: fuzzy_snow

fuzzy_bright4haze

fuzzy_mean4dark



fire_cci


Issue/Rev-No.: 2.2


- Calculate mean of !fuzzy_snow, !fuzzy_mean4dark and fuzzy_bright4haze if pixel is

(fuzzy_cloud < 0.9)

Output: fuzzy_haze

calculate shadow


TOA reflectances at 1665nm





threshold_1665_low= 017

threshold_1665_high=0.23

land_flag

- Calculate temp1 = Fhi of TOA reflectances at 835nm

- Calculate temp2 = Fdn of TOA reflectances at 1665nm

- Calculate Fand of temp1 and temp2 if pixel is land_flag

Output: shadow

end for j

end for i

*/

3.8 Algorithm – atmospheric correction

3.8.1 Overview

In the case of ATSR2, AATSR or MERIS the atmospheric correction using ATCOR-WS is applied to

the ortho-rectified images O creating the atmospheric corrected images Q. In the case of SPOT

VEGETATION already atmospheric corrected images serve as input and so the images are already on

the same processing level after ortho-rectification.

fire_cci


Issue/Rev-No.: 2.2


Figure 9: General workflow for the atmospheric correction using ATCOR-WS

fire_cci


Issue/Rev-No.: 2.2


3.8.2 Logical Flow


o Transform DEM D to OD fitting exactly on O

o Import OWM and OCM to combine them a haze-cloud-water-mask HCW

o Convert O, OD and HCW to format required by ATCOR-WS (BSQ conversion)

o Call ATCOR-WS modules:

aspslp: This module is a necessary precursor for the topographic correction part of

ATCOR-WS where aspect/slope are calculated based on the DEM D used. For global

processing, GETASSE is used as DEM.

shadow: This module is also a necessary precursor for the topographic correction part

of ATCOR-WS where the shadow is calculated based on the DEM D used. A ray

tracing method is applied using the DEM D and the solar zenith and azimuth angles of

each pixel.

atcor_wfov: This module includes the atmospheric and topographic correction (see the

logical flow chart in the next figure).

o Reconvert the atmospherically corrected image of ATCOR-WS to the standard format as

Q.

First, some consistency checks are performed on metadata. The valid range of the sun zenith angle

must be greater than or equal 0 and less than 70 degrees. The range of the view angle must be greater

or equal than 0 and less than 50 degrees. Finally it is checked if for the sensor under investigation a

spectral band is available to derive the water vapour content (as e.g. for MERIS). In the next step the

scene is partitioned into small cells of 30 km x 60 km (across-track / along-track) to interpolate the

radiative transfer functions from the off-line calculated LUTs on a per-cell basis. The LUTs are

calculated using MODTRAN. Next, the cloud, snow/ice, haze and spectral shadow mask and the water

masks are read, followed by the aerosol optical thickness (AOT at 550 nm) retrieval based on dark

reference areas (dense, dark vegetation DDV and water). For MERIS the water vapour map is

calculated, for (A)ATSR a fixed water vapour column is assumed as specified in the input metadata

(for more information please refer to the ATBD-I-Pre-processing (Bachmann et al. 2014)).

The reflectance retrieval consists of two steps: the first one calculates the surface reflectance including

topographic compensation on the basis of the small cells, the second step accounts for the adjacency

effect. In the latter case, the radiative transfer functions are evaluated for nadir to reduce the execution

time. This is justified because the adjacency effect is a small second-order effect for coarse spatial

resolution imagery.

fire_cci


Issue/Rev-No.: 2.2


Figure 10: Logical flow chart of the combined atmospheric and topographic correction

fire_cci


Issue/Rev-No.: 2.2


3.8.3 Equations

A detailed description of the equations used in ATCOR-WS can be found in Richter (2010) and in the

ATBD-I (Bachmann et al. 2014).


The list of variables used in ATCOR-WS can be found in Richter (2010) and in the ATBD-I-pre-

processing (Bachmann et al. 2014).


The different processing steps for running ATCOR-WS can be found in Richter (2010) and in the

ATBD-I (Bachmann et al. 2014).

The processing of the atmospheric correction is performed on a pixel by pixel step. All calculations

are done as matrix operations to reduce computation time.

The following processing steps are necessary, shown as an example of pixel by pixel calculation:

/*

# Algorithm AEROSOL

# input: pixel p with b bands

# output: pixel q containing the aerosol parameter as

# aerosol type

# aerosol optical thickness (AOT)

if p is valid and ddv : calculate AOT@550

if p is valid and not-ddv : calculate <AOT@550>

if p is valid : smooth AOT@550 using 3 km x 3 km window

return AOT@550

*/

/*

# Algorithm WATER VAPOUR

# input: pixel p with b bands (14 and 15 for MERIS)

# output: pixel q containing the water vapour content (scale factor: 1000)

if p is valid : calculate WV

return WV

*/

fire_cci


Issue/Rev-No.: 2.2


/*

# Algorithm ATMOSPHERIC CORRECTION

# input: pixel p with b bands

# aerosol model

# AOT@550

# DEM (slope and aspect)

# Water vapour

# output: pixel q containing surface reflectance for each band

if p is valid :

for each band b do

read AOT@550

if water vapour band available read water vapour

read DEM related files

read AOT@550- and WV-dependent LUT

calculate surface reflectance

return surface reflectance

else

return surface reflectance ==NaN

endif

*/

fire_cci


Issue/Rev-No.: 2.2


4 (A)ATSR Burnt Area Processing Chain

4.1 Introduction

This section describes in detail the processing steps to generate the burned area (BA) product based on

the (A)ATSR sensor imagery. Our version of the BA processing algorithm was mainly written in the

Python 2.7 (http://www.python.org) language calling several open-source libraries, which are

specified in the next sections, but it can be written in any programming language has long as the

library requirements are met. Due to its high processing speed, the processing chain also includes the

use of a compiled C++ programme, also calling open source libraries, to evaluate the computational

demanding spatial probability revision. The (A)ATSR processor has been implemented and applied

for specific areas within selected regions, but is not implemented in the production of the global BA

products.

4.2 Pre-processing

The initial step of the burned area processing chain is data uploading. Extraction of data from the daily

files and upload info into memory was designed to limit the frequency of I/O operations and occupy

the least possible RAM memory, to allow multi-core processing on single machine. A single ASCII

file (Figure 11) is required for each run, containing all the necessary parameters and the directory

names of the sequential daily files (produced in the data pre-processing, section 3), and the auxiliary

fire-season reference cell map and the corresponding Look Up Table (LUT) weights.

These fire-season related files were based on results obtained, externally to the project, by Benali et al.

(2013) by adjusting modal and bimodal circular frequency distributions to 1 degree spatial and 10 day

temporal aggregations of the screened MODIS thermal anomalies product (MCD14ML) by Oom and

Pereira, (2013). The reference file consists on a global map at 1km spatial resolution where each pixel

contains a reference number corresponding to an individual 1 degree global cell. The map is in Plate-

Carré projection, deserts and water bodies pixels are classified by in GlobCover 2005

(http://due.esrin.esa.int/globcover/) are set to 0, so that they are recognized by the algorithm as pixels

not to process. For each reference number in the map there is a corresponding line in the LUT file with

the corresponding fire season temporal weighs. In total, this look up table has 360 x 180 lines,

corresponding to the possible 1 degree cells and 37 columns, related to the 10 day composite period,

containing the seasonal weights that were obtained by the normalised adjustments performed by

Benali et al. (2013).

2008 - year

8 - line

19 - column

351 - number Observations

/home/bmota/Datasets/Satelite/ATS/L8C19/2008/ - path

/home/bmota/CCI_fire/Fire_Season/ - path

ATS_20080102_0000

ATS_20080103_0000

ATS_20080104_0000

ATS_20080105_0000

ATS_20080106_0000

ATS_20080107_0000

ATS_20080108_0000

ATS_20080109_0000

ATS_20080110_0000

ATS_20080111_0000

ATS_20080112_0000

ATS_20080113_0000

ATS_20080114_0000

ATS_20080115_0000

ATS_20080116_0000

ATS_20080117_0000

ATS_20080118_0000

ATS_20080119_0000

ATS_20080120_0000

..

.

Figure 11: (A)ATSR ASCII initial file example

http://www.python.org/

fire_cci


Issue/Rev-No.: 2.2


4.3 Overview

The BA algorithm loads all the available annual reflectance images of the NIR channel, applies the

water and cloud masks to filter invalid surface observations (all derived from the pre-processing

stage), for each pixel extracts the time series and applies a filter to remove spikes. It then detects

significant changes in mean NIR time-series and scores them according to a calculated distance based

on its attributes to an idealized potential fire event. For each pixel, the change point with the lowest

rank is then re-scaled according to the spatial context and used to build a graph. This graph is then

segmented between the burned and unburned. The last post-processing step is information aggregation

into monthly composites.

4.3.1 Logical Flow

The logical flow of the (A)ATSR burned area processing steps is illustrated in Figure 12.

fire_cci


Issue/Rev-No.: 2.2


Figure 12: Flow diagram of the (A)ATSR BA processing chain

fire_cci


Issue/Rev-No.: 2.2


4.3.2 Equations and functions

The equations applied in the burned area processing chain are listed below.

Eq.4.1-Robust filter function (R function)

Yt_fil ←robust.filter(Yt, trend, width, scale, outlier, shiftd, p, adapt, max. width, extrapolate)

Eq.4.2-Change point detection function (R function)

Y, Yt← cpt.mean(Yt_fil,penalty,method)

Eq.4.3-Neighbour pixel connection flow velocity function

diffDdiff

Dv

diffD

=xxvmax_)

max_1(*0

max_0

),( 21

Eq.4.4-Density of observations

tt

t

tj j

SS

y=tDensity

1

1

Eq.4.5-Logit function

Logit(p) = | 100*p / (1-p) |

Eq.4.6-Normalization based on the maximum

minmax

max

xx

xx=xNorm i

i

Eq.4.7-Normalization based on the minimum

minmax

min

xx

xx=xNorm i

i

fire_cci


Issue/Rev-No.: 2.2


4.3.3 List of variables

Table 2: Variables used in the (A)ATSR BA processing chain

Variables

y1:n = (y1, . . . , yn) Time series of NIR reflectance

ytm = Time series of local minima NIR reflectance

ytM = Time series of local maxima NIR reflectance

ytmin = Time series of local minima NIR channels reflectance without spurious turning points

ytmin_fil = Time series of local minima NIR channels reflectance despiked

daytm = Time series of associated Julian dates for ytm

daytM= Time series of associated Julian dates for ytM

daytmin = Time series of associated Julian dates for ytmin

daytmin_fil = Time series of associated Julian dates for ytmin_fil

NIRi,j,t=NIR channel observations, where i and j represent the spatial location and temporal location

yt_fil = Time series of local minima despiked (NIR reflectance)

dayt_fil = Time series of associated Julian dates for yt_fil

S= Change point segments of mean NIR reflectance

decres= NIR reflectance decrease between consecutive segments

po_ref= Post change point NIR mean reflectance segment

sea_w= Change point fire-season weight

decres= NIR reflectance decrease between consecutive segments

po_dst_min= Post change point segment reflectance difference to the time-series minimum

dst= Euclidian distance to an idealised fire event point

4.3.4 List of parameters

Table 3: Parameters set in the (A)ATSR BA processing chain

Parameter value

min_diff_cutoff 0.02

max_refl_cutoff 0.4

mask_cl_thres 75

mask_wt_thres 75

min_obs_req 25

min_score 0.2

max_score 0.6

max_diff 20

den_thres 0.2

po_thres 5

v0 0.9

mean_p_score=(max_p_score-min_p_score)/2 0.5

fire_cci


Issue/Rev-No.: 2.2


4.3.5 Step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and

observational statistics

4.3.5.1 Overview

The processing chain is design to minimise I/O operations and memory usage. To achieve this, data

are read and simultaneously validated to be stored into memory. For each daily atmosphere corrected

reflectance Q.1: NIR channel and the Quality control layers, both the OWM: static and dynamic layers

and the OWM: cloud, cloud shadow, haze and snow are read using the python GDAL library and pixel

validation is performed by flagging invalid pixels where cloud/haze/ice/shadow and water masks

exceed the corresponding mask_cl_thres and mask_wt_thres levels, and where the quality layer

identifies pixels with any radiometric problems. For each day the NIR channel and flags are stored in

memory in a 3D matrix, NIRi,j,t, where i,j represent the location within the tile/scene and t is the

seasonal observation. Simultaneously, imagery dates are extracted from the directory names and

converted into Julian dates.

In addition, for each valid pixel, cloudy and the total observation frequencies are calculated and split

into monthly layers and saved into a temporary binary file to be read at the post-processing stage

(section 4.10). The total number of observations is provided by the frequency a pixel is observed with

surface reflectance, the number of valid observations is determined by the number of untagged

observations and the cloudy observations is provided by the frequency of pixels flagged as cloud.

In order to save processing time, pixels located over water bodies or desert land-surface with no fuel

are flagged and not processed. This information is provided by the auxiliary reference file (section 4.2)

identified by the value 0 and pixels are assigned a value of 0. In order to proceed to the next steps a

minimum number of observations is required (min_obs_req). If this number is not satisfied the pixel is

flagged and not processed, and information is passed on to the post-processing stage.

4.3.5.2 Logical flow

Figure 13: Flow diagram of step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and


fire_cci


Issue/Rev-No.: 2.2


4.3.6 Step 2 - Time-series filtering

4.3.6.1 Overview

The Yt time series may also contain negative outliers, usually caused by cloud shadowing, and

occasionally due to unscreened flooding. To remove these “spikes”, a robust filtering approach is

implemented based on the R “robfilter” package (Fried et al. 2011) ported to Python code with the

Rpy library (http://rpy.sourceforge.net/rpy2.html). A detailed mathematical background can be found

on section 4.1.1.1.2 of the ATBD II version 2.2 (Pereira et al. 2014). This approach requires the

following parameters for model fitting:

Robust Trend (trend) Approximation is set to repeated median regression (RM).

Initial window width (width) is set to a width of 3.

Scale estimate (scale) is set to median absolute deviation about the median (MAD).

Outlier detection (outlier) is by winsorization (W) approach.

Shift detection (shiftd) is set to 2.

Maximal window width (max.width) is set to 7.

Window width adaptation (adapt) is set to 0.8.

Extrapolate defines the rule for data extrapolation to the edges of the time series. As

implemented, it consists of the fitted values within the first half of the first window and within

the last half of the last window.

/*

library(robfilter)

Yt_fil, dayt_fil← robust.filter(Yt, “RM”, 3, “MAD”, “W”, 2, p, 0.8, 7, extrapolate=TRUE)

*/

Robust filtering of the time series of surface reflectance minima removed most of these negative

outliers, yielding time series suitable for the next step, change point detections.


Logical flow of the compute layers process is shown in Figure 14.

Figure 14: Flow diagram of step 2 - Time-series filtering

fire_cci


Issue/Rev-No.: 2.2


4.3.7 Step 3 - Change point detection and ranking

4.3.7.1 Overview

Detections of points where a change in mean occurred depends on two options, a choice of criteria to

optimise and an optimisation algorithm. The Pruned Exact Linear Time (PELT) algorithm proposed by

Killick et al. (2012) produces exact optimisation of change-point segmentations in linear

computational time. The mathematical description and background of PELT is given in ATBD II v2.2

(Pereira et al. 2014), and the function is available in the R “changepoint” package (Killick and Eckley,

2010) which can easily be ported to Python code with the Rpy library

(http://rpy.sourceforge.net/rpy2.html). The function is parameterised by the following settings:

Change point method (method) is set to PELT.

Penalty (penalty) used is set to SIC.

/*

library(changepoint)

S, St ← cpt.mean(Ytmin_fil,”SIC”,”PELT”)

*/

This approach may yield several change points per pixel / time series, and per season, since the mean

value of NIR reflectance data varies not only in response to burning, but also due to other events. Each

change point is scored according to their likelihood of representing a burned area. Compromise

programming is the technique chosen to perform temporal change point selection of the potential burn

event. For all change points, only the ones representing reflectance decrease and pre and post change

point segment observation density above den_thres are considered. Density is determined by dividing

the number of observations by the temporal extent in each segment (eq. 4.4). For each considered

change point its attributes are determined. These are the mean segment reflectance decrease, the post-

change point segment reflectance, the difference between the post-change point segment reflectance

and time-series minima and the associated seasonal weight for the corresponding 10-day time step

(column) obtained from the auxiliary LUT.

/*

for i=1 to Si

if Si+1 –Si >0 and Density(Si ) >den_thresand Density(Si+1 ) >den_thres

decresi=Si+1 –Si

po_refi=Si+1

sea_wi=LUT(pixel_reference(line,column), St)

po_dst_min=Si+1- n=jjymin:1

*/

For the seasonal weight attribute, the mean segment reflectance decrease attribute and the difference

between the post-change point segment reflectance and time-series minima, normalization in relation

to the maximum observed value is performed according to eq. 4.6and for the post change mean

reflectance attribute, based on its minimum observed value according to eq. 4.7.

Change point ranking is determined by the Euclidian distance to an idealised point determined by the

attribute values that best describe a possible burn event, i.e., the biggest reflectance decrease, the

highest seasonal weight, the lowest difference of the post segment reflectance to the minimum and the

lowest post segment reflectance.

fire_cci


Issue/Rev-No.: 2.2


/*

for i=1 to decres

dsti=sqrt((Norm(po_refi)-min(Norm(po_refi)))2+

(Norm(sea_wi)-max(Norm(sea_wi)))2+

(Norm(po_dst_mini)-min(Norm((po_dst_mini)))2+

(Norm(decresi)-min(Norm(decresi)))2)

*/

The change point with the highest rank, i.e., the lowest distance is selected to be included in a four

layer composite containing the change point post reflectance, seasonal weight, reflectance decrease,

associated dates and the difference in days to the previous valid observation. The first three layers are

passed on to the spatial rescaling and z-score determination (next section) and the last layer is passed

on to the post-processing (section 4.4) to be included in the final product format.

/*

po_refmin_dist,sea_wmin_dst, decresmin_dst,DOYdst_min, DOY_lagdst_min= min_select(dst, St)

*/



Figure 15: Flow diagram of step 3 - Change point detection and ranking

fire_cci


Issue/Rev-No.: 2.2


4.3.8 Step 4 - Spatial p_scoring

4.3.8.1 Overview

Spatial attribute rescaling serves to determine each pixel z_score based on its attributes. As for the

previous section compromise programming is the technique chosen to perform this operation but now

in a spatial context. For all the pixels, only the ones representing reflectance decrease below max_diff

and post reflectance above po_thres are considered to avoid residual atmospheric contaminated pixels

and water bodies. For each considered pixel, the attributes are: the mean segment reflectance decrease,

the post-change point segment reflectance, and the associated seasonal weight. These are normalised

following the same rule as in the previous section.

The scores are determined by the Euclidian distance to an idealised point determined by the attribute

values found in the spatial frame that best describe a possible burn event, i.e., the biggest reflectance

decrease, the highest seasonal weight, the lowest post segment reflectance.

/*

for i=1 to number of considered pixels

if decresi<max_diff and po_refi>po_thres

dsti=sqrt((Norm(po_refi)-min(Norm(po_refi)))2+

(Norm(sea_wi)-max(Norm(sea_wi)))2+

(Norm(decresi)-min(Norm(decresi)))2)

z_score=1-dsti

*/

The z_scores with the highest and lowest values are selected to rescale the z_scores into p_scores, by

normalizing between the observed range. The p_scores and associated dates, for the considered pixels,

are passed on to the next section. All pixels that are not considered are flagged and are classified as

unburned in the post-processing phase



Figure 16: Flow diagram of step 4 –z_scoring

fire_cci


Issue/Rev-No.: 2.2


4.3.9 Step 5 - Spatial probability revision - MRF segmentation

4.3.9.1 Overview

The previous steps of the algorithm lead to a characterization of each pixel of the image through two

variables: p_score and detection dates. The high p-scores detection dates are used in the Markov

random field image segmentation. The algorithm is a standard image processing algorithm that solves

the maximum a posteriori – Markov random field (MAP-MRF) problem, which arises in image

segmentation or image restoration. The MAP-MRF problem can be converted into finding a minimum

cut in a certain graph. In this case there are only two classes (burned and unburned) and the Boykov

Kolmogorov algorithm (Boykov and Kolmogorov, 2004) is used. For more mathematical background

and details refer to ATBD II v2.2 (Pereira et al. 2014). Pixels with no computed p_score and

associated dates and pixels flagged as “unburned” by the fire season mask (defined in previous

section) are not revised. The remaining pixels are classified according to their relation to their 4

neighbour pixels (by dates) and to their relation to an ideal “burned” or “unburned” pixel (by z-

scores).

To build the graph, each pixel (graph vertex) and its relation with other pixels (graph edges) has to be

set in a table of adjacencies in DIMACS format and saved on in an ASCII file temp_flow.dat.

// c Use a DIMACS network flow file example.

//

// c The total flow:

// s 0 t 2013

// c flow values:

// f 0 6 3

// f 0 1 6

// f 0 2 4

// f 1 5 1

// f 1 0 0

// f 1 3 5

// f 2 4 4

// f 2 3 0

// ...

Figure 17: DIMACS format file example

Each line of the file contains a pixel connection (edges) and its flow capacity. The capacities pixels are

calculated by applying the Logit function to v(x1,x2) calculated between two neighbour pixels. The

connection of each pixel with the ideal unburned (vertex 0) or burned (vertex n+1) is determined by

the p_score. If p_scoremax is lower/higher than the mean_score, the pixel is connected to the

“unburned”/”burned” ideal pixel and its capacity is determined by applying the Logit function to the p-

scores.

/*

for unflagged pixels

edge connections to ideal vertexes

if p_scoremax<mean_zscore

edge source = vertex 0

edge target = vertex(pixels)

edge capacity = Logit(p-score(p_score))

if p_scoremax>=mean_zscore

edge source = vertex(pixels)

edge target = vertex(N+1)

edge capacity = Logit(p-score(z-score))

edge connections between pixels

edge source = vertex(pixel A)

edge target = vertex(pixel B)

edge capacity = Logit(v(A,B))

fire_cci


Issue/Rev-No.: 2.2


*/

This temporary ASCII file is read by a C++ programme (Min_Cut.cpp) that solves the minimum

source/sink-cut problem from the unburned source (vertex 0) and the burned sink (vertex N+1) by

applying the Boykov Kolmogorov algorithm available in the Boost C++ libraries

(http://www.boost.org/). The assigned class of each pixel (vertex) is stored in a binary file where

unburned pixels take value 0 and burned pixels take value 1.



Figure 18: Flow diagram of step 5 - MRF segmentation

4.4 Post-processing

The final step is set to combine all the layer information necessary as stated in the IODD. The first

layer - burned Julian date - is determined by the date layer and contains only for pixels classified as

burned from the previous section (section 4.3.9). Water or unprocessed pixels are attributed the date -1

and for the unburned pixels the date 0. For layer two - the confidence level - the p-scores calculated in

the previous section are used and flagged with -1 for the water and unprocessed pixels. In layer three -

time since last clear detection - the information is passed from section 4.3.7 and set only for the pixels

classified as burned. The remaining unburned pixels have value 0 and the water and unprocessed

pixels have value -1. Layers 1-3 are splinted into monthly time frames. Water or unprocessed pixels

are replicated on all the monthly files and the burned pixels are distributed according to the date of

occurrence.

In the remaining three layers (4-6) - number of valid observations to classify, number of observations

covered by the sensor and the number of observations with cloud cover – contain information passed

http://www.boost.org/

fire_cci


Issue/Rev-No.: 2.2


from section 4.3.5. Water and unprocessed pixels are flagged with -1. For months where no validated

observation occurs, layer 1-3 are flagged with -1. The final step is to record the monthly files into

GeoTiff files, using the GDAL python library, according to the naming convention established in the

IODD (Krauß et al. 2014).

5 SPOT-VEGETATION Burnt Area Processing Chain

5.1 Introduction

This section describes in detail the processing steps to generate the burned area (BA) product based on

the SPOT-VEGETATION sensor imagery. Our version of the BA processing algorithm was mainly

written in the Python 2.7 (http://www.python.org) language calling several open-source libraries that

are specified in the following sections, but it can be written in any programming language, as long as

the library requirements are met. Due to its high processing speed, the processing chain also includes

the use of a compiled C++ programme, also calling open source libraries, to evaluate the

computationally demanding spatial probability revision.

5.2 Pre-processing

The initial step of the burned area processing chain is data uploading. Extraction of data from the daily

files and upload info into memory was designed to limit the frequency of I/O operations and occupy

the least possible RAM memory, to allow multi-core processing on single machine. A single ASCII

file (Figure 11) is required for each run, containing all the necessary parameters and the directory

names of the sequential daily files (produced in the data pre-processing, section 3), and the auxiliary

fire-season reference cell map and the corresponding Look Up Table (LUT) weights.

These fire-season related files were based on results obtained, externally to the project, by Benali et al.

(2013) by adjusting modal and bimodal circular frequency distributions to 0.01 degree spatial and 10

day temporal aggregations of the screened MODIS thermal anomalies product (MCD14ML) by Oom

and Pereira (2014). The reference file consists on a global map at 0.00833 degree spatial resolution

where each pixel contains a reference number corresponding to an individual 0.01 degree global cell

Geographic coordination system. Deserts and water bodies pixels are classified in GlobCover 2005

(http://due.esrin.esa.int/globcover/) are set to 0, so that they are recognized by the algorithm as pixels

not to process. For each reference number in the map there is a corresponding line in the LUT file with

the corresponding fire season temporal weighs. In total, this look up table has 480 000 lines,

corresponding to the possible 0.01 degree cells and 37 columns, related to the 10 day composite

period, containing the seasonal weights that were obtained by the normalised adjustments performed

by Benali et al. (2013).

2008 - year

3 - line

7 - column

/media/HardDisk/CCI_fire/Fire_Season6/

/media/HardDisk/Datasets/Satelite/VGT/Tiles/Data/L3C7/2008/20080101















..

.

Figure 19: VGT ASCII initial file example

http://www.python.org/

fire_cci


Issue/Rev-No.: 2.2


5.3 Overview

The BA algorithm loads all the available annual reflectance images of the NIR channel, plus the last

month of the previous year and the first month of the following year. After loading the data, the

algorithm applies the water and cloud masks to each pixel, masking out invalid surface observations

(all derived from the pre-processing stage), extracts the time series and applies a filter to remove

spikes. It then detects significant changes in mean NIR time-series and scores them according to a

calculated distance based on its attributes to an idealized potential fire event. For each pixel, the

change point with the lowest rank is then re-scaled according to the spatial context and used to build a

graph. This graph is then segmented between the burned and unburned. The last post-processing step

is information aggregation into monthly composites.

5.4 Logical Flow

The logical flow of the VGT burned area processing steps is illustrated in Figure 20.

fire_cci


Issue/Rev-No.: 2.2


Figure 20: Flow diagram of the VGT BA processing chain

fire_cci


Issue/Rev-No.: 2.2


5.4.1 Equations and functions

The equations applied in the burned area processing chain are listed below.

Eq.4.1-Robust filter function (R function)

Yt_fil ←robust.filter(Yt, trend, width, scale, outlier, shiftd, p, adapt, max.width, extrapolate)

Eq.4.2-Change point detection function (R function)

Y, Yt← cpt.mean(Yt_fil,penalty,method)

Eq.4.3-Neighbour pixel connection flow velocity function

diffDdiff

Dv

diffD

=xxvmax_)

max_1(*0

max_0

),( 21

Eq.4.4-Density of observations

tt

t

tj j

SS

y=tDensity

1

1

Eq.4.5-Logit function

Logit(p) = | 100*p / (1-p) |

Eq.4.6-Normalization based on the maximum

minmax

max

xx

xx=xNorm i

i

Eq.4.7-Normalization based on the minimum

minmax

min

xx

xx=xNorm i

i

Eq.4.8-Normalization of distances to ideal and anti-ideal points

Di+DaiDi=pscore /

fire_cci


Issue/Rev-No.: 2.2



Table 4: Variables used in the VGT BA processing chain

Variables

y1:n = (y1, . . . , yn) Time series of NIR reflectance

ytm = Time series of local minima NIR reflectance

ytM = Time series of local maxima NIR reflectance

ytmin = Time series of local minima NIR channels reflectance without spurious turning points

ytmin_fil = Time series of local minima NIR channels reflectance despiked

daytm = Time series of associated Julian dates for ytm

daytM= Time series of associated Julian dates for ytM

daytmin = Time series of associated Julian dates for ytmin

daytmin_fil = Time series of associated Julian dates for ytmin_fil

NIRi,j,t=NIR channel observations, where i and j represent the spatial location and temporal location

yt_fil = Time series of local minima despiked (NIR reflectance)

dayt_fil = Time series of associated Julian dates for yt_fil

S= Change point segments of mean NIR reflectance

day S= Time series of associated Julian dates for S

decres= NIR reflectance decrease between consecutive segments (𝑆𝑖+1̅̅ ̅̅ ̅ − 𝑆�̅�)

po_ref= Post change point NIR mean reflectance segment (𝑆𝑖+1̅̅ ̅̅ ̅)

pre_NIR= Reflectance observations in the pre change point

po_NIR= Reflectance observations in the post change point

half_temp_range=half of the time-lag in time series ((lastdate-1st date)/2)

sea_w= Change point fire-season weight

dst_time= Euclidean distance to an idealised fire event point (time series stage)

dst_space= Euclidean distance to an idealised fire event point (space stage)

dst_space_ideal= Euclidean distance to an idealised regional group of points (space stage)

dst_space_a-ideal= orthogonal distances to a constant vale (space stage)


Table 5: Parameters set in the VGT BA processing chain

Parameter value

min_diff_cutoff 0.02

max_refl_cutoff 0.4

mask_cl_thres 75

mask_wt_thres 75

min_obs_req 60

min_obs_min 35

fire_cci


Issue/Rev-No.: 2.2


Parameter value

max_slope 0.4

min_score 0.2

max_score 0.6

max_diff 0.2

den_thres 0.1

decres_min 0.02

po_max 0.2

num_observ 3

v0 0.9

mean_p_score = (max_p_score-min_p_score)/2 0.5

5.4.4 Step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and


5.4.4.1 Overview

The processing chain is design to minimise I/O operations and memory usage. To achieve this, data

are read and simultaneously validated to be stored into memory. For each daily atmosphere corrected

reflectance Q.1: NIR channel and the Quality control layers, both the OWM: static and dynamic layers

and the OWM: cloud, cloud shadow, haze and snow are read using the python GDAL library and pixel

validation is performed by flagging invalid pixels where cloud/haze/ice/shadow and water masks

exceed the corresponding mask_cl_thres and mask_wt_thres levels, and where the quality layer

identifies pixels with any radiometric problems. For each day the NIR channel and flags are stored in

memory in a 3D matrix, NIRi,j,t, where i,j represent the location within the tile/scene and t is the

seasonal observation. Simultaneously, imagery dates are extracted from the directory names and

converted into Julian dates.

In addition, for each valid pixel, cloudy and the total observation frequencies are calculated and split

into monthly layers and saved into a temporary binary file to be read at the post-processing stage. The

total number of observations is provided by the frequency a pixel is observed with surface reflectance,

the number of valid observations is determined by the number of untagged observations and the

cloudy observations is provided by the frequency of pixels flagged as cloud.

In order to save processing time, pixels located over water bodies or desert land-surface with no fuel

are flagged and not processed. This information is provided by the auxiliary reference file identified

by the value 0 and pixels are assigned a value of 0. In order to proceed to the next steps a minimum

number of observations is required (min_obs_req). If this number is not satisfied the pixel is flagged

and not processed, and information is passed on to the post-processing stage.

fire_cci


Issue/Rev-No.: 2.2



Figure 21: Flow diagram of step 1 - Data extraction, Cloud, Water mask and Quality Control flagging and


5.4.5 Step 2- Maxima/Minima time series extraction

5.4.5.1.1 Overview

Extracting local minima from a time series of observations y1:n of reflectance data entails a series of

steps:

1 Identify of “turning points” i.e. the local minimum, ytm, and local maximum, ytM if its two

adjacent neighbours are smaller or larger (Kugiumtzis and Tsimpiris, 2010). Time series edges

are extended by replicating the second value prior to the first and performing the same

corresponding operation for opposite edge, the n observation.

2 Remove from the series all yt that are not turning points.

3 Calculate a time series of first-order differences and remove all turning points that differ from

one of its immediate neighbours less than the min_diff_cutoff reflectance units. This process

need to consider which of the series, ytm or ytM, starts and which ends.

4 Retain only the local minima and remove from the series those ytm values above the

max_refl_cutoff reflectance units.

fire_cci


Issue/Rev-No.: 2.2




Figure 22: Flow diagram of step 2 and 3 – Maxima/Minima time series extraction and filtering

/*

for i=1 to lines

for j=1 to columns

Yt, dayt = validate_observations(NIRi,j,t)

ytm,ytM,daytm,daytM= find_envelopes(Yt, dayt )

ytmin,daytmin= remove_barbs( ytm,ytM,daytm,daytM, min_diff_cutoff)

end for

end for

*/

5.4.6 Step 3 - Time-series filtering

5.4.6.1 Overview

The Yt time series may also contain negative outliers, usually caused by cloud shadowing, and

occasionally due to unscreened flooding. To remove these “spikes”, a robust filtering approach is

implemented based on the R “robfilter” package (Fried et al. 2011) ported to Python code with the

Rpy library (http://rpy.sourceforge.net/rpy2.html). A detailed mathematical background can be found

on section 4.2.1.1.2 of the ATBD II version 2.2 (Pereira et al. 2014). This approach requires the

following parameters for model fitting:

Robust Trend (trend) Approximation is set to repeated median regression (RM).

Initial window width (width) is set to a width of 3.

Scale estimate (scale) is set to to Rousseeuw’s and Croux’ (1993) on scale estimator (QN) .

Outlier detection (outlier) is by winsorization (W) approach.

Shift detection (shiftd) is set to 2.

Maximal window width (max.width) is set to 7.

Window width adaptation (adapt) is set to 0.8.

Extrapolate defines the rule for data extrapolation to the edges of the time series. As

implemented, it consists of the fitted values within the first half of the first window and within

the last half of the last window.

fire_cci


Issue/Rev-No.: 2.2


/*

# robfilter R package

library(robfilter)

Yt_fil, dayt_fil← robust.filter(Yt, “RM”, 3, “QN”, “W”, 2, p, 0.8, 7, extrapolate=TRUE)

*/

Robust filtering of the time series of surface reflectance minima removed most of these negative

outliers, yielding time series suitable for the next step, change point detection.



Figure 23: Flow diagram of step 2 - Time-series filtering

5.4.7 Step 3 - Change point detection and ranking

5.4.7.1 Overview

Detections of points where a change in mean occurred depends on two options, a choice of criteria to

optimise and an optimisation algorithm. The Pruned Exact Linear Time (PELT) algorithm proposed by

Killick et al. (2012) produces exact optimisation of change-point segmentations in linear

computational time. The mathematical description and background of PELT is given in ATBD II v2.2

(Pereira et al. 2014), and the function is available in the R “changepoint” package (Killick and Eckley,

2010) which can easily be ported to Python code with the Rpy library

(http://rpy.sourceforge.net/rpy2.html). The function is parameterised by the following settings:

Change point method (method) is set to PELT

Penalty (penalty) used is set to SIC

fire_cci


Issue/Rev-No.: 2.2


/*

# changepoint R package

library(changepoint)

S, St ← cpt.mean(Ytmin_fil,”SIC”,”PELT”)

*/

This approach may yield several change points per pixel / time series, and per season, since the mean

value of NIR reflectance data varies not only in response to burning, but also due to other events. Each

change point is scored according to their likelihood of representing a burned area. Compromise

programming is the technique chosen to perform temporal change point selection of the potential burn

event. Each change point detected by PELT has to comply with each and every of the following

criteria, before it is considered apt for scoring (see Table 4):

- decres < 0

- decres < | max_diff |

- po_ref < po_thres

- Density(Si ) > den_thres , Density is determined by dividing the number of observations by the

temporal extent in each segment (eq. 4.4).

- Density(Si+1 ) > den_thres

- po_ref < min{po_NIR} +0.005

- Slope of the linear fit to the po_NIR observations in the post CP segment < max_slope

- Time lag between the CP dates of two lowest po_ref values in time series < half_temp_range.

- Number of observations prior to first CP and/or after the last CP > num_observ.

The attributes mean segment reflectance decrease decres=(𝑆𝑖+1̅̅ ̅̅ ̅ − 𝑆�̅�), the post-change point segment

reflectance po_ref=(𝑆𝑖+1̅̅ ̅̅ ̅), and the associated seasonal weight for the corresponding 10-day time step

(column) obtained from the auxiliary LUT (sea_w). Each change point satisfying these requirements is

calculated and used for scoring and selecting the best (most likely to represent a burn event) CP in

each time series

/*

for i=1 to Si # for all the change points, selectif Si+1 –Si <0 and Si+1 –Si <abs(max_diff) and po_ref < po_thres

and Density(Si ) >den_thres and Density(Si+1 ) ..>den_thres

and 𝑆𝑖+1̅̅ ̅̅ ̅ < min{po_NIR} +0.005 and 0<Slope (po_NIR (Si)) <max_slope

and t < half_temp_range

and Number of observations prior to first CP and/or after the last CP > num_observ

endif

end for

decresi=𝑆𝑖+1̅̅ ̅̅ ̅ − 𝑆�̅�

po_refi=𝑆𝑖+1̅̅ ̅̅ ̅

sea_wi=LUT(pixel_reference(line,column), St)

*/

For the seasonal weight attribute and the mean segment reflectance decrease attribute, normalization is

performed according to eq. 4.7 and for the post change mean reflectance attribute, according to eq. 4.6.

Change point ranking is determined by the Euclidean distance to an ideal point determined by the best

attribute values that characterise a possible burn event, i.e., the largest reflectance decrease, the highest

seasonal weight, and the lowest post fire segment reflectance. All data are rescaled such that the most

desirable score of each variable is assigned a value of 1, i.e. the ideal point is a vector [1, 1, 1]

/*

for i=1 to all the change point selected calculate the Euclidean distance

fire_cci


Issue/Rev-No.: 2.2


dst_po_ref= 0.332*(1-(Norm(po_refi)))2

dst_sea_w= 0.332*(1-(Norm(sea_wi)))2

dst_decres= 0.332*(1-(Norm(decresi))))2

# calculate Euclidean distance

dst_tempi=sqrt(dst_po_refi+ dst_sea_wi+ dst_decresi)

end for

*/

The change point with the highest rank, i.e., the lowest distance to the ideal is selected to be included

in a four layer composite containing the change point post reflectance, seasonal weight, reflectance

decrease, all the associated dates and the difference in days to the previous valid observation. The first

three layers are passed on to the spatial rescaling and z-score determination (next section) and the last

layer is passed on to the post-processing (section 4.4) to be included in the final product format.

/*

po_refmin_dist_temp,sea_wmin_dst_temp, decresmin_dst_temp,DOYdst_temp__min, DOY_lagdst_temp_min= min_select(dst_temp, St)

*/



Figure 24: Flow diagram of step 3 - Change point detection and ranking

fire_cci


Issue/Rev-No.: 2.2


5.4.8 Step 4 - Spatial p_scoring

5.4.8.1 Overview

Spatial attribute rescaling serves to determine each pixel pscore (burning probability) based on its

attributes. As for the previous section compromise programming is the technique chosen to perform

this operation but now in a spatial context. For each change point out of each time series (only one per

pixel) the scoring attributes are: mean segment reflectance decrease, post-change point segment

reflectance, and associated seasonal weight.

Global-scale scoring of the candidate change points from each time series/pixel relies on the

calculation of Euclidean distances to an ideal and to an anti-ideal, which are represented by sets of

points, rather than by single points. The ideal set contains one point/site/year from 9 of the 10 project

regional study sites (Borneo was excluded due to lack of appropriate data), representing the most

paradigmatic examples of fire-affected pixels, in terms of the non-rescaled variables mean segment

reflectance decrease, the post-change point segment reflectance, and the associated seasonal weight.

The convex hull of this set of ideal points was determined and distances between the change point

selected from each time series/pixel and the ideal set are calculated. The distance to the nearest point

in the convex hull is selected (minimum distance) as the distance to the ideal set, according to a

mathematical definition of distance between a point and a set of points. Distances to an anti-ideal set

were also calculated for each pixel as the shortest of the orthogonal distances to a constant vale of

decres_min and po_max, respectively for mean segment reflectance decrease and post-change point

segment reflectance. Finally, for each timeseries/pixel, the distances to the ideal (Dii) and anti-ideal

(Daii) sets were combined as eq. 4.8 to achieve the p-scores.

/* for each of considered change point selected CP i

#distance to ideal

for each ideal point j (1 to 11)

#distance of each pixel to the eleven(j) points dst_po_refij=po_ref(CPi)-pointj

dst_sea_wij=dst_sea_w(CPi)-pointj

dst_decresij=decres(CPi)-pointj

dst_space_ideali=sqrt((0.332 *(dst_po_refij)2)+( 0.332 *(dst_sea_wij)2)+ ( 0.342

*(dst_decresij)2) Dii=min {dst_space_ideal}

end for

#distance to anti-ideal

for each anti-ideal constant value max_diff and po_thres

#distance of each pixel to the max_diff

dst_decresij=decres(CPi)- max_diff

dst_sea_wij=dst_sea_w(CPi)-0 #anti-ideal for seasonal weight is zero

dst_po_refij=po_ref(CPi) #keeps the value

#distance of each pixel to the po_thres

dst_decresij=decres(CPi) #keeps the value

dst_sea_wij=dst_sea_w(CPi) -0 #anti-ideal for seasonal weight is zero

dst_po_refij=po_ref(CPi) -po_thres

dst_space_a-ideali=sqrt((0.332 *(dst_po_refij)2)+( 0.332 *(dst_sea_wij)2)+ ( 0.342

*(dst_decresij)2)

Daii=min {dst_space_a-ideal}

*/

fire_cci


Issue/Rev-No.: 2.2


The p_scores and associated dates, for the considered pixels, are passed on to the next section. All

pixels that are not considered are flagged and are classified as unburned in the post-processing phase



Figure 25: Flow diagram of step 4 –z_scoring

5.4.9 Step 5 - Spatial probability revision - MRF segmentation

5.4.9.1 Overview

The previous steps of the algorithm lead to a characterization of each pixel of the image through two

variables: p_score and detection dates. The high p-scores detection dates are used in the Markov

random field image segmentation. The algorithm is a standard image processing algorithm that solves

the maximum a posteriori – Markov random field (MAP-MRF) problem, which arises in image

segmentation or image restoration. The MAP-MRF problem can be converted into finding a minimum

cut in a certain graph. In this case there are only two classes (burned and unburned) and the Boykov

Kolmogorov algorithm (Boykov and Kolmogorov, 2004) is used. For more mathematical background

and details refer to ATBD II v2.2 (Pereira et al. 2014). Pixels with no computed p_score and

associated dates and pixels flagged as “unburned” by the fire season mask (defined in previous

section) are not revised. The remaining pixels are classified according to their relation to their 4

neighbour pixels (by dates) and to their relation to an ideal “burned” or “unburned” pixel (by z-

scores).

To build the graph, each pixel (graph vertex) and its relation with other pixels (graph edges) has to be

set in a table of adjacencies in DIMACS format and saved in an ASCII file temp_flow.dat.

fire_cci


Issue/Rev-No.: 2.2


// c Use a DIMACS network flow file example.

//

// c The total flow:

// s 0 t 2013

// c flow values:

// f 0 6 3

// f 0 1 6

// f 0 2 4

// f 1 5 1

// f 1 0 0

// f 1 3 5

// f 2 4 4

// f 2 3 0

// ...

Figure 26: DIMACS format file example

Each line of the file contains a pixel connection (edges) and its flow capacity. The capacities pixels are

calculated by applying the Logit function to v(x1,x2) calculated between two neighbour pixels. The

connection of each pixel with the ideal unburned (vertex 0) or burned (vertex n+1) is determined by

the p_score. If p_score is lower/higher than the mean_score, the pixel is connected to the

“unburned”/”burned” ideal pixel and its capacity is determined by applying the Logit function to the p-

scores.

/*

for unflagged pixels

edge connections to ideal vertexes

if p_score<mean_zscore

edge source = vertex 0

edge target = vertex(pixels)

edge capacity = Logit (p_score)

if p_score>=mean_zscore

edge source = vertex(pixels)

edge target = vertex(N+1)

edge capacity = Logit( (p_score)

edge connections between pixels

edge source = vertex(pixel A)

edge target = vertex(pixel B)

edge capacity = Logit(v(A,B))

*/

This temporary ASCII file is read by a C++ programme (max_flow.cpp) that solves the minimum

source/sink-cut problem from the unburned source (vertex 0) and the burned sink (vertex N+1) by

applying the Boykov Kolmogorov algorithm available in the Boost C++ libraries

(http://www.boost.org/). The assigned class of each pixel (vertex) is stored in a binary file where

unburned pixels take value 0 and burned pixels take value 1.



http://www.boost.org/

fire_cci


Issue/Rev-No.: 2.2


Figure 27: Flow diagram of step 5 - MRF segmentation

5.5 Post-processing

The final step is set to combine all the layer information necessary as stated in the IODD (Krauß et al.

2014). The first layer - burned Julian date - is determined by the date layer and contains only pixels

classified as burned from the previous section. Water or unprocessed pixels are attributed the date -1

and for the unburned pixels the date 0. For layer two - the confidence level - the p-scores calculated in

the previous section are used and flagged with -1 for the water and unprocessed pixels. In layer three -

time since last clear detection - the information is passed from section 5.3.7 and set only for the pixels

classified as burned. The remaining unburned pixels have value 0 and the water and unprocessed

pixels have value -1. Layers 1-3 are splinted into monthly time frames. Water or unprocessed pixels

are replicated on all the monthly files and the burned pixels are distributed according to the date of

occurrence.

The remaining three layers (4-6) - number of valid observations to classify, number of observations

covered by the sensor and the number of observations with cloud cover – contain information passed

from section 5.3.5. Water and unprocessed pixels are flagged with -1. For months where no validated

observation occurs, layer 1-3 are flagged with -1. The final step is to record the monthly files into

GeoTiff files, using the GDAL python library, according to the naming convention established in the

IODD (Krauß et al. 2014).

fire_cci


Issue/Rev-No.: 2.2


6 MERIS Burnt Area Processing Chain

6.1 Key Principles of the Algorithm

The MERIS BA algorithm detects burned areas on a global scale, based on the use of MERIS

reflectance bands, spectral indices calculation, post-fire and multi-temporal discrimination, in synergy

with hotspot locations (as determined using MODIS middle- and thermal-infrared bands). A general

flow of the algorithm can be seen in Figure 28.

The BA algorithm for MERIS has been tailored according to the data characteristics (lack of SWIR,

temporal noise…). Spectral reflectance and mask auxiliary data are used to obtain a series of monthly

spectral index composites. BA detection is performed in a two phase process: the first one detects seed

pixels, while the second one applies a contextual criterion around these seed pixels.

Input data to the algorithm is: spatial distribution of active fires for burned area detection, MERIS

reflectance bands 8 and 10, together with their auxiliary information (water, cloud, shadow, haze and

snow masks) and information on burnable/not burnable areas extracted from a land cover map

(GlobCover 2005). Details on the BA algorithm can be found in the ATBD II v2.2 (Pereira et al.

2014).

Figure 28: Algorithm general flow

fire_cci


Issue/Rev-No.: 2.2


6.2 Pre-processing

Before running the MERIS BA algorithm MERIS data must have been pre-processed as described in

section 3.

6.3 Processing Chain

The 5 main processes of the algorithm are:

- Data filtering: information needed for the specific tile in process is extracted from all

input data.

- Build composites: a set of variables and matrices needed to perform the proper BA

detection is computed within this process.

- BA detection: seed identification and region growing, together with confidence levels are

obtained in this process.

- Compute layers: separation of yearly product obtained from the BA detection process into

monthly products and calculation of other variables needed for the merged product are

performed here.

- BA monthly product: data obtained on the previous steps is assembled to obtain the BA

monthly product 6 layers.

Each one of these 5 main processes is further detailed in the following sections.

6.4 Data Filtering

6.4.1 Overview

This step of the algorithm is in charge of selecting all relevant data that are needed in the BA

algorithm. There are 2 main tasks included in this process. First step is to look for the burnable file

that corresponds to the tile that is being processed. If the tile is not burnable the process stops here and

continues with the next tile.

- Valid images: this task looks for the reflectances GeoTiff file. If the file is found, the

corresponding filename (that contains the date of the image) is saved in the variable

valid_images.

- HS: selects from all Hot Spots available in the MOD 014 product the ones in the

geographical tile and year that is being processed, saving the information in the variable

HS.

6.4.2 Logical flow

The logical flow for the data filtering is shown in Figure 29.

fire_cci


Issue/Rev-No.: 2.2


Figure 29: Data filtering logical flow

6.5 Build Composites

6.5.1 Overview

In this process the different matrices that are the input to the BA algorithm are obtained. It is

performed in 5 steps:

- HS-matrix: this task computes the distance from each pixel to a HS. The value of the HS

closest to the pixel is assigned to the pixel, obtaining with this procedure the Thiessen

polygons.

- NIR composite: this process looks for the lowest 3 NIR values every 2 months and assigns

to each pixel the NIR value that is closest in date to the HS matrix, from the set of 3

lowest NIR values.

- HS_density_monthly: obtains HS density matrices (3x3 and 9x9) for each month.

- GEMI: computes the GEMI monthly composites and finds the annual maximum value of

this index.

- Minimum NIR_HS: obtains the lowest NIR value around the HS pixels.

fire_cci


Issue/Rev-No.: 2.2


6.5.2 Logical flow

The logical flows for the process build composites, NIR composite and GEMI are shown in Figure 30,

Figure 31 and Figure 32, respectively.

Figure 30: Build composites logical flow

Figure 31: NIR composite logical flow

fire_cci


Issue/Rev-No.: 2.2


Figure 32: GEMI logical flow

6.6 BA Detection

6.6.1 Overview

In this process, the estimation of burned areas is performed. There are 3 main steps:

- Identify seeds: seed pixels are obtained from the HS information previously filtered.

Percentile curves of burned and unburned pixels are built and thresholds defined to

discriminate between burned and unburned pixels.

- Region growing: new thresholds are used (see ATBD II) for the NIR and differenceGEMI

composites to identify burned and unburned pixels. Pixels are classified as burned if the

neighbours to a seed pixel have higher values than the threshold. The threshold is defined

taking into account the value found in the seed pixel.

- Confidence levels: probability rules are applied to compute confidence levels on the burned

pixels. Probabilities of burning and not burning are set (0.5). Conditional probabilities are

computed. Confidence level is obtained with the probability of a pixel having burned given

the fact that the pixel has a particular NIR value (obtained from the composite) ((P(B/NIR)

see Equation 6.2 in section 6.9)

6.6.2 Logical flow

The Logical flow for the overall BA detection process is shown in Figure 33. The Logical flow of the

seed and region growing process are shown in Figure 34 and Figure 35.

fire_cci


Issue/Rev-No.: 2.2


Figure 33: BA detection logical flow

Figure 34: Identify seeds logical flow

fire_cci


Issue/Rev-No.: 2.2


Figure 35: Region growing logical flow

6.7 Compute Layers

6.7.1 Overview

In this process, the layers needed to obtain the final product are computed. It is performed in 3 steps:

- Convert_BA_ product:in order to obtain the BA product a yearly composite is built, with the

BA detected for each month. In this process, yearly values are separated into 12 months to

prepare the product to be delivered to the merging algorithm chain.

- Previous_observation: this process looks for the last valid image before the burnt, and counts

the number of days in between.

- Valid_all_cloud observations: this process counts the number of valid observations per pixel

and month, considering a 30 % threshold for cloud, shadow and snow, and 40 % for haze. It

also counts the number of images that were acquired by the sensor per pixel per month and the

number of images that were covered by more than 30 % of cloud and 40 % of haze per pixel

per month.

6.7.2 Logical flow


fire_cci


Issue/Rev-No.: 2.2


Figure 36: Compute layers logical flow

6.8 BA Monthly product

6.8.1 Overview

In the last process of the algorithm the variables obtained in the previous tasks are assembled. This

process is performed in 1 step:

- Group_layers: all the information obtained from the previous block is grouped by month and the 6

layers of the product are created.

6.8.2 Logical flow

Figure 37: BA monthly product and format conversion logical flow

fire_cci


Issue/Rev-No.: 2.2


6.9 Post Processing

In this step the BA monthly product in .mat format is converted to the GeoTiff output format.The

output product is divided in 6 different layers:

- Layer 1: BA state

o Burned = Julian date of burn (1-365)

o Water mask = -1

o Not burned = 0

o Non–observed = -4

- Layer 2: Confidence level: 0-100%

- Layer 3: Time since last clear detection: number of days between the detection and the

previous valid image.

- Layer 4: Number of valid observations to classify (days)

- Layer 5: Number of observations covered by the sensor (days)

- Layer 6: Number of observations with cloud cover (days)

6.10 Equations

The equations applied through the BA algorithm are listed below.

Eq.6.1 - GEMI index

R

RGEMI

1

125,0)25,01(*

5,0

)*5,0()*5,1()(*2 22

RIR

RIRRIR

Eq.6.1 - Confidence level

Confidence level calculation is based on the Bayes’ theorem:

PBcondBmax=100 * (PBmaxcondB * PB) / (PBmaxcondB * PB + PBmaxcondU * PU);

Where:

- P(B/NIR) = probability of having burned given the fact that it has a particular NIR value

- P(NIR/B) = probability of observing a certain value of NIR knowing that it has burned

- P(NIR/U) = probability of observing a certain NIR value knowing that it has not burned

- PB = 0.5 = a priori probability of burning

- PU = 0.5 = a priori probability of not burning

Eq.6.3 - Thiessen polygons

Where:

- is a space with a distance function d,

- K is a set of indices

- (Pk)k ϵK is a tuple of non-empty subsets in the space

- Rk is the region associated to the Pk set of points in X whose distance to Pk is not greater

than the distance to the other Pj sites

- (Rk)k ϵK is the Thiessen diagram.

fire_cci


Issue/Rev-No.: 2.2


6.11 List of variables

A list of variables used in the BA MERIS processing chain is detailed in Table 6.

Table 6: Variables in the MERIS BA processing chain

Variable Description

Year Time series of years to process

Path 1 Root path to data directories

Path 2 Root path intermediate variables

Path 3 Root path to HS auxiliary files

Path 4 Root path to Land cover auxiliary files

Path 5 Root path to output file

HS MOD14 product information

Lat Latitude of the tile to process

Lon Longitude of the tile year to process

Valid_images Valid images for the specific tile and year

Thiessen Matrix obtained with HS information

NIR-HS GEMI-HS, jul Composites containing the MERIS reflectances and Julian dates

associated

HS dens HS Density matrices (3 x 3, 9 x 9)

Seeds Pixels identified as burned in the seed phase

BA Pixels identified as burned after the growing phase

CL Confidence level relative to the BA pixels

Previous Previous valid observation before the burn detection date

Valid Number of valid observations over a pixel

All Total number of observations over a pixel

Cloud Number of cloudy observations over a pixel

Threshold 1 Threshold identified to filter the seeds

Threshold 2 Threshold identified to filter the pixels in the growing phase

6.12 List of parameters

A list of parameters is used through the BA MERIS processing. They are detailed in Table 7.

Table 7: Parameters set in the MERIS BA processing chain

Parameter value

Cloud threshold 30

Haze threshold 40

Snow threshold 30

Shadow threshold 30

Growing GEMI 0.9

fire_cci


Issue/Rev-No.: 2.2


6.13 Computer Programme in Pseudo-Code

The main script is the only one to interact with the user. Its pseudo code is as follows: /*

begin

Year = ask user for the year to process

for i = 1:number of tiles

call data filtering

call build composites

call BA detection

call compute layers

call monthly product

end for

end

*/

6.13.1 Data Filtering

Input: MERIS data, MOD 14, latitude, longitude of the tile and year to process

Output: valid images, HS

- Valid_images function: /*

Begin

Create path to folder with year indicated, latitude and longitude of the file being processed

for i=1:files in the folder year/tile

open file

if result open = -1

next file

else

valid_images = file_id

endif

end for

End

*/

- Burnable tile function: /*

Begin


open file

if result open = -1

go to the next tile

endif

End

*/

- HS function: /*

Begin


load MODIS_014 for the specified year

select lat long information

HS = selected information

End

*/

fire_cci


Issue/Rev-No.: 2.2


6.13.2 Build Components

Input: valid_images, MERIS bands 8, 10 and masks, HS

Output: Thiessen matrix, NIR-HS, DAY, GEMI-HS, GEMImax, HS density matrices,

minimum NIR.

- HS_matrix function: */

Begin

for i = 1:raws

for j=1:columns

dist = Euclidean distance from pixel to HS

pix = mindistHS

end for

end for:

End

*/

- NIR composite: */

Begin

for i = 1:365

find NIRmin1NIRmin2NIRmin3, jul1, jul2,jul3

end

find closest juliandate to the Thiessen matrix

End

*/

- GEMI: /*

Begin

for i = 1: valid_images

GEMI = call compute-GEMI

ifGEMI>GEMImax

GEMImax = GEMI

end if

end for

End

*/

- HS density monthly function: /*

Begin

for i=1:12

identify monthly HS

build 3x3 and 9x9 density matrices

end for

End

*/

fire_cci


Issue/Rev-No.: 2.2


- Minimum NIR HS: /*

Begin

for i=1:12

for j = 1:8

if NIR value<NIR-HS

NIR-HS = NIRvalue;

End if

End for

End for

End

*/

6.13.3 BA Detection

Input: HS, NIR-HS, DAY, GEMI, GEMImax, minimumNIR, HS densitymatrices

Output: yearly BA, CL

- Identify_seeds function: /*

Begin

burn = minimimum NIR

unburn = HSdensitymatrices==0;

build percentile curves

identify threshold

Verify conditions:

NIR < threshold1

1 burned pixel in 9x9 matrix

NIRmonth-NIRmonth-2 >0

If conditions verified

Add neigbourpixel(i,j) to burn

end if

seeds = burn*DAY

End

*/

- Region growing function: /*

Begin

burn = seed

while (length (burn)>0)

for i = -1:1

for j = -1:1

Verify conditions:

NIR-HSneighbour<threshold2

DiffGEMIneighbour< 0.9*DifGEMIseed

If conditions verified

Add neigbourpixel(i,j) to burn

end if

end for

end for

end while

Jul_burn = burn*DAY

End

*/

fire_cci


Issue/Rev-No.: 2.2


- Confidence_level function: /*

Begin

PB = PU = 0.5

PB = PU = 0.5

Compute PB(NIR/B), PB(NIR/U)

PB(B/NIR)=100 * (PB(NIR/B)* PB) ./

(PB(NIR/B)* PB + PB(NIR/U) * PU);

CL(jul_burn) = PB(B/NIR)

End

*/

6.13.4 Compute Layers

Input: yearly BA, CL, valid_images

Output: monthly BA, monthly CL, previous, valid, all, cloud

- Convert function: /*

Begin

for i = 1:12

month = find dates in BA related to the i month

CL_month = save values for pixels found on line 2

end for

End

*/

- Previous function: /*

Begin

found = 0

counter = 0

for i = 1:12

valid_images

dates =BA(i)

for i = 1:length (dates)

date = dates(i)

position = find image related to date in valid_images

while ( found ==0 & counter <31)

Open the previous image to the one indicated by position in valid_images

Counter = date of the image opened - date

if (cloud<30 & shadow<30 &snow<30 & haze<40% &counter<31)

Datefound = date of the image opened

found=1

else

Position = position -1

previous = date-datefound

End

*/

fire_cci


Issue/Rev-No.: 2.2


- Valid_all_cloud function: /*

Begin

valid =0

all = 0

cloud = 0

for i = 1:12

for j = 1: valid_images(month(i))

all = all+1

Masks related to each image will be checked

if (cloud<30 & shadow<30 &snow<30 & haze<40%)

valid = valid+1

end if

if (cloud>=30&haze>=40)

cloud = cloud+1

end if

end for

end for

End

*/

6.13.5 BA Monthly Product and Format Conversion

Input: monthly BA, monthly CL, previous, valid, all, cloud

Output: BA monthly product

- Group layers function: /*

Begin

for i = 1:12

BA_layer1 = monthly BA(i)

BA_layer2= monthly CL(i)

BA_layer3 = previous (i)

BA_layer4 = valid (i)

BA_layer5= all(i)

BA_layer6 = cloud (i)

end for

End

*/

- Format conversion function: /*

Begin

for i = 1:12

BA_monthly(i) = call geotiff (BA(i))

end for

End

*/

fire_cci


Issue/Rev-No.: 2.2


7 Product Merging Processing Chain

7.1 Introduction

This section describes the processing chain for the BA merging. The merging chain ingests all the BA

products produced as 10 by 10 degree tiles in sections 4, 5 and 6, and produces the global grid product

at 0.5 degree resolution with 22 attributes (or layers) and the pixel product at 1/120 degree prior to the

availability of MERIS data and 1/360 degree with MERIS data mosaicked into six global sub sets

containing 4 attributes (or layers).

7.2 Pre-processing and preparation

BA data described in sections 4.4, 5.5 and 6.2 must be available as 10 by 10 degree tiles.

Each 10 by 10 degree tile must have a land cover map at 1/120 and 1/360 degree resolution.

A formatted text file indicating years and months to be processed must be completed.

Thresholds for confidence and uncertainty screening for each land cover must be entered into the

processing script.

7.3 Algorithm

7.3.1 Overview

The processing chain P1 (Figure 38) uses three c-shell scripts. The first calls a series of C programmes

to process each 10 by 10 degree global tile of monthly burned area data from each of the BA products

from section 4, 5, and 6. The second and third scripts create the final global grid and pixel products.

There is also a separate reference text file read by the c-shell showing which years and months to

process. The c script loads several (free) software modules, Python, R, GDAL and NetCDF and

defines the parameters and variables required to process a 10 by 10 degree tile. The first c shell script

has 5 stages (Figure 39). The second and third c shell script mosaics the tiles to form the pixel and grid

products to the product specification format. Details of these stages are described in the logical flow

section (Section 7.3.2).

Figure 38: Overview of the inputs and outputs to the data processing chain P1

fire_cci


Issue/Rev-No.: 2.2


7.3.2 Logical flow

The processing chain (Figure 39) inside the merge processor follows this sequence:

1. Data preparation, extraction of the BA datasets P1.1 (section 7.3.5.1);

2. The first merge 1/120°, resolution screened when no MERIS data P1.2 (section 7.3.5.2);

3. The second merge 1/360°,resolution always screened P1.3 (section 7.3.5.3);

4. Aggregation of the pixel product to the final grid product, P1.4 (section 7.3.5.4).

5. Update and mosaicking the global sub tiles, P1.5 (section 7.3.5.5)

P1.1: The BA products are first split into their constituent layers, burned area status (BAS), confidence

level (CON) and number of valid observations (NOV).

P1.2: The 1/120 degree AATSR, ATSR and SPOT VGT data are merged. Data is processed to show

sensor combinations detecting the burn (SC), earliest Julian Date of detection (JD), highest confidence

level (CL) and land cover burned (LC). If there is no MERIS data single sensor observations are

performance screened using the confidence level thresholds where one sensor detects a burned area

(based on BA algorithm performance per land cover). After the merge a further screening is applied

using thresholds derived from a moving window that calculates the probability (uncertainty) that a

pixel detected by one sensor burned within its neighbourhood.

P1.3: The second merge operates when 1/360 degree MERIS data are available. Data from P1.2 are

resampled to 1/360 degree and data are processed to show sensor combinations detecting the burn

(SC), earliest Julian Date of detection (JD), highest confidence level (CL) and land cover burned (LC).

As with P1.2 sensor observations are performance screened using the confidence level thresholds

where one sensor detects a burned area (based on BA algorithm performance per land cover). After the

merge a further screening is applied using thresholds derived from a moving window that calculates

the probability (uncertainty) that a pixel detected by one sensor burned within its neighbourhood.

P1.4: The pixel level product is aggregated to the 0.5 degree cell calculating the sum of burned area

(ASB), the standard error of burned area (SEB), the percentage of clear observations (PCO), the

number of patches (NOP) and the area of each of 18 land covers that was burned (LCn).

P1.5: In the second and their c-shell scripts, the 10 by 10 degree tiles are mosaicked into the product

specification.

fire_cci


Issue/Rev-No.: 2.2


Figure 39: Overview of the component parts of the processing chain P1

fire_cci


Issue/Rev-No.: 2.2



A list of variables is used in the merge processing chain:

Dates to process in the monthly time series;

The root directories to the source code and the processing directories.

Screening flags for single sensor burn detection

Table 8: Variables in the merged product chain

Variable Description

Year time series of years to process

Month time series of months to process

root_path root path to data directories

bin_path root path to executable files

python_path Root path to Python executable and reference files

lut_conf_ats[n] ATSR confidence threshold (1 value for [n] land cover type)

lut_conf_at2[n] ATSR2 confidence threshold (1 value for [n] land cover type)

lut_conf_vgt[n] VGT confidence threshold (1 value for [n] land cover type)

lut_conf_mer[n] MERIS confidence threshold (1 value for [n] land cover type)

lut_unc_ats[n] ATSR uncertainty threshold (1 value for [n] land cover type)

lut_unc_at2[n] ATSR2 uncertainty threshold (1 value for [n] land cover type)

lut_unc_vgt[n] VGT uncertainty threshold (1 value for [n] land cover type)

lut_unc_mer[n] MERIS uncertainty threshold (1 value for [n] land cover type)


A list of parameters for each global tile is shown in Table 7, these parameters define:

Dimensions of the 1/120, 1/360 and 0.5 degree 10 by 10 degree tile

The coordinate system and top left coordinates of the 10 by 10 degree tile and global subset

Dimensions of the global subset at 1/120, 1/360 and 0.5 degree;

Position of 10 by 10 degree tiles in the global grid at 1/120, 1/360 and 0.5 degree resolutions

Dates to process

fire_cci


Issue/Rev-No.: 2.2


Table 9: Parameters set in the merge processing chain

Parameters Source Represents

pixels Fixed 1200 Number of pixels at 1/120° resolution

lines Fixed 1200 Number of lines at 1/120° resolution

pixels_meris Fixed 3600 Number of pixels at 1/360° resolution

lines_meris Fixed 3600 Number of lines at 1/360° resolution

Grid_pixels_02* Fixed 20 Number of pixels at 0.5° resolution

Grid_lines_02* Fixed 20 Number of lines at 0.5° resolution

dd_tl_lat ‘L’ ‘C’ ref from tile name Top left latitude of tile

dd_tl_lon ‘L’ ‘C’ ref from tile name Top left longitude of tile

ga_y ‘L’ ‘C’ ref from tile name Y position in global grid at 0.5° resolution

ga_x ‘L’ ‘C’ ref from tile name X position in global grid at 0.5° resolution

sub_y ‘L’ ‘C’ ref from tile name Y position in global grid at 1/120° resolution

sub_x ‘L’ ‘C’ ref from tile name X position in global grid at 1/120° resolution

subm_y ‘L’ ‘C’ ref from tile name Y position in global grid at 1/360° resolution

subm_x ‘L’ ‘C’ ref from tile name X position in global grid at 1/360° resolution

y_proc Set within the script Year being processed

m_proc Set within the script Year being processed

*file naming convention _02 = reciprocal of 0.5


The c shell script has hardcoded pre-set root paths to the processing and executable directories. There

are lines of code to load the Python, R, GDAL and NetCDF modules (specific commands may vary

according to the platform being used) some parameters are also hard coded into the script and some

parameters are calculated from a c script that uses the reference of the tile in the file name. A text file

is also read which provides the years /months to be processed.

The c shell script is called with the following syntax:

#######################################################"

echo " ESA CCI Fire ECV shell script: cci_global_merge_processor.csh"

echo " Copyright University of Leicester 2014"

echo " Code built by Andrew Bradley / Kevin Tansey"

echo " Contact person: [email protected]"

echo " #######################################################"

echo ""

echo " Syntax: cci_global_merge_processor.csh <site_y>,<site_x>,<Processor flag (0 = Merge, 1 =

MERIS ONLY, 2 = VGT ONLY)>"

echo " <Site_y e.g. L11C30 = 11>"

echo " <Site_x e.g. L11C30 = 30>"

echo " <Generate which product (0 = Merge, 1 = MERIS ONLY, 2 = VGT ONLY)>"

echo ""

exit(0)

endif

fire_cci


Issue/Rev-No.: 2.2


The full time series must be processed completely for each of the first three parts before moving onto

the next part, e.g. the data extraction, primary merge and secondary merge (<3>, <4> and <5>). If this

procedure is not followed then double counts between products are not identified correctly. After

completing the processing the second c script is run to complete the pixel product (one for each global

sub set) which mosaics the 10 by 10 degree tiles into the global sub sets.

Each of the parts are described in the following sections using these abbreviations:

GRID = Grid product

PIXEL = Pixel product

SC = Sensor Combinations

JD = Julian Day

CL= Confidence Level

LC = Land cover

NOV = Number of Valid Observations

ASB = Aggregate Sum Burned

ASE = Aggregate Standard Error

NOP = Number of patches

FOP = Fraction of Observed Pixels

ASLn = Aggregate Sum Land cover n burned

t - 0 = current month

t - 1 = previous month

7.3.5.1 Processing part 1.1: Data preparation

The first step is to extract location and position details of the tile being processed using Function 7.1

cci_tile_attributes.c. This function reads the reference of the tile name e.g. L10 and C20 from the c

shell command: >cci_global_tiles.csh 10 20 1 0 0 0 0 0. The function cci_tile_attributes contains an

array of look up values for L and C corresponding to latitude and longitude. The upper left latitude and

longitude are calculated along with the corresponding x and y, position of the tile as if it was part of

the final 0.5 degree global grid. Each value is written to a text file which is then read and the value is

assigned to a variable in the c shell script. This function also calculates the corresponding x, y position

for a 1/120 and 1/360 degree resolution global pixel product. These latter values are not used in the

first shell script and assist in the mosaicking of the global sub tiles Part 1.5.

Function 7.1 cci_tile_attributes

Input value: L_coord

Input value: C_coord

Output file: top left lat

Output file: top left long

Output file: Coordinate of y in global grid product @ 0.5 degree resolution

Output file: Coordinate of x in global grid product @ 0.5 degree resolution

Output file: Coordinate of y in a global pixel grid @ 1/120 degree resolution

Output file: Coordinate of x in a global pixel grid @ 1/120 degree resolution



The script loops through the years and months provided in the date text file and searches for the

corresponding GeoTiffs in the data archive. When found the data is reformatted into separate layers in

using a GDAL function, creating the layers Burned area state (BAS), Confidence level (CON),

Number of valid pixels (NOV) and Number of surface observations (NOS).If there are no files for a

particular year then the processor creates blank files that pass through the processing chain without

influencing the merging (Function 7.2 cci_create_raster). At this point the variable meris_switch is set

from 0 to 1 if the year is greater than 2002 when MERIS data are available. This switch determines

whether to skip or apply the secondary merge later in the processing. If the meris_switch is on and

fire_cci


Issue/Rev-No.: 2.2


there are no MERIS data e.g. a missing month of data then blank MERIS files will be created. Land

cover maps are supplied at 1/120° to 1/360° resolution in the .hdr format (Figure 40).

Function 7.2 cci_ create_raster

output file: new raster (byte/integer/float)

input value: pixels (integer)

input value: lines (integer)

input value: bytes per pixel e.g. 1, 2 or 4 (integer)

input value: value for the raster e.g. 0-255 (integer)

Figure 40: Preparation of datasets P1.1

P1.1 Extraction of data

/*

gdal_translate -b x -of ENVI<filename.tif><filename_layer>

*/

Where:

x = band in the GeoTiff

layer= BAS-Burned area state, CON-Confidence level, NOV- Number of valid pixels, NOS- Number of surface

observations

The files are then written to monthly directories.

fire_cci


Issue/Rev-No.: 2.2


7.3.5.2 Processing part 1.2: 1/120° resolution merge

In this stage, P1.2, the AATSR, ATSR and SPOT vegetation data are merged as they are 1/120° resolution datasets. The processing steps (Figure 41) are described. When processing begins for the

first time a blank data set is created for the previous month (function 7.1). A land cover map, currently

GlobCover (which will eventually be land_cover_cci) is used for the landcover map. During the

merging double counts between the previous month merge and current month data are tested for

(function 7.4). This may be when the data in the current month double counts a late burn from the

previous month which has a lagged date because of obscured observation of the surface or long

overpass intervals. The check involves comparing the results of the merging of the previous month (t –

1) to data of the current month (t + 0). If a duplicate burn is found, the burn is deleted from the current

month (t + 0) and the sensor combinations, and confidence levels are revised in the previous month (t -

1).

At this stage if there is no MERIS (1/360°) BA product a screening of occasions where single sensor

BA observations occur is made using different thresholds for different sensors for different land

covers. A threshold (TH) of CL (CLTH) and PB (PBTH) was identified, globally and for each land cover

(LC), for each fire_cci intermediate product and the final Pixel Product such that TH minimized the

bias (B) of the BA estimates. B ( 11 pp ) is computed from the error matrix. THs were found by

iteration on the calibration dataset. For example, for CL, all product pixels with a value higher than a

potential CL were classified as burned and unburned otherwise, and from that reclassification B was

computed. B was computed for all possible CL and the CLTH selected was the one that produced the

lowest B. This was done in a similar way for PB. CL is not available and PB is constant for pixels

classified as unburned. Hence, the identification of a TH is only applicable when the product itself

(without applying any TH) overestimates. If the product overestimates, there are too many pixels

classified as burned. If the product underestimates, there are too many pixels classified as unburned.

As pixels classified as unburned have no CL values and have a constant PB value, no potential pixels

can be identified for reclassifying them, from unburned to burned. A temporary data set is then created

which ‘maps’ the thresholds according to the land cover. This map is compared to the merged data set

and if the confidence level is below the calculated threshold all BA data (JD, SC, CL and LC) are

excluded from the merge.

A second contextual screen is then applied again looking at single sensor detections but this time

examining the probability (uncertainty) that the pixel burned given that the surrounding pixels burned.

The BA and CL are stacked and passed through a python programme to calculate BA uncertainty. The

uncertainty of the BA estimates on the Pixel Product is expressed in probabilistic terms, as the

probability that a pixel is really burnt (PB). Errors were measured using reference data derived at the

10 Study Sites and regression analysis was used to generate the uncertainty quantification models. For

the Pixel Product a logistic regression was calibrated for pixels classified as burned. The explicative

variables were the Confidence Level (CL) and the number of neighbour pixels classified as burned, in

a 9x9 moving window (NEI). CL and NEI values are not available for pixels classified as unburned,

for those, the PB was estimated from the ratio between the number of pixels classified as unburned by

the product but burned by the reference data and the number of pixels classified as unburned by the

product. The uncertainty is then compared to the uncertainty thresholds and the JD, CL, SC and LC is

removed (screened) if the uncertainty is below the threshold.

The 10 by 10 degree tiles are then passed on to the global sub tile processing P1.5 (section 7.3.5.5)

The resolution 1/120°, or 1/360°depends on the meris_switch.

fire_cci


Issue/Rev-No.: 2.2


Figure 41: The processing steps for the primary merge at 1/120°, P1.2

fire_cci


Issue/Rev-No.: 2.2


Firstly the script checks for the existence of AATSR, ATSR or SPOT data for the previous month (t-1)

If NO:

A blank dummy file is created (function 7.2) where BAS=0, CON=0, LDC=0, NOV=0, NOS=0,

NOC=0. This ensures that data files are available to pass through the merging function. The missing

data is reported to a text file. The YES condition is now satisfied.







If YES

Merge the burned area data sets

If there is MERIS data then call cci_primary_merge_check (function 7.3).

If there is no MERIS data then create the confidence level maps cci_create_data_screen (function 7.4)

then call cci_primary_merge_check_conf2 (function 7.5).

Function 7.3 cci_primary_merge_check

input file: sensor x JD (integer)

input file: sensor y JD (integer)

input file: sensor z JD (integer)

input file: sensor x CL (integer)

input file: sensor y CL (integer)

input file: sensor z CL (integer)

input file: LC map 1km (byte)

input file: primary merged JD t - 1 (integer)

input file:primary merged CL t - 1 (integer)

input file: primary merged SC t - 1 (integer)

input file: primary merged LC t - 1 (byte)

output file: merged output JD t + 0 (integer)

output file: merged output CL t + 0 (integer)

output file: merged output SC t + 0 (integer)

output file: Land cover of burned area t + 0 (byte)

output file: primary merged output JD revised t - 1 (integer)

output file: primary merged output CL revised t - 1 (integer)

output file: primary merged output SC revised t - 1 (integer)

output file: primary merged Land cover of burned area t - 1 (byte)

output file: check for overlap (integer)



Function 7.3 does the following:

/*

If JD > 0 for all sensors

Then: SC = 12, JD = earliest JD, CL = highest confidence, LC = Globcover class

If JD > 0 for SPOT/ATSR sensors


If JD > 0 for SPOT/ATSR-2 sensors


If JD > 0 for ATSR/ATSR-2 sensors


fire_cci


Issue/Rev-No.: 2.2


If JD > 0 for ATSR (or SPOT or ATSR-2)

Then: SC = 1 (2 or 8), JD = ATSR (SPOT or ATSR-2) JD, CL = ATSR (SPOTor ATSR-2) confidence, LC =

Globcover class

Check for double counts:

If JD > 0 for month t-1 and t+0

Then:

t+0, SC = 0, JD = 0, CL = 0, LC = 0

t-1, SC = revised, CL = revised

Write out t+0 and t-1, SC, JD, CL, LC

*/

Where SC= sensor combinations, JD = Julian day, CL = confidence level, LC =land cover

Function 7.4 cci_create_screen(confidence or uncertainty)

input file: land cover map (byte)

input data: pixels (integer)

input data: lines (integer)

output file: screen for sensor (w OR x OR y OR z)

input data: threshold for sensor (w OR x OR y OR z) for land cover type 1


input data: thresholdfor sensor (w OR x OR y OR z) for land cover type 3






















/*

When LC = n

Confidence(uncertainty) screen = lut_confidence(uncertainty)_sensor[n]

*/

fire_cci


Issue/Rev-No.: 2.2


Function 7.5 cci_primary_merge_check_conf2

input file: sensor x JD (integer)

input file: sensor y JD (integer)

input file: sensor z JD (integer)

input file: sensor x CL (integer)

input file: sensor y CL (integer)

input file: sensor z CL (integer)

input file: LC map 1km (byte)

input file: primary merged JD t - 1 (integer)

input file:primary merged CL t - 1 (integer)

input file: primary merged SC t - 1 (integer)

input file: primary merged LC t - 1 (byte)

input file: confidence threshold mask sensor x (integer)

input file:confidence threshold mask sensor y (integer)

input file: confidence threshold mask sensor z (integer)

output file: merged output JD t + 0 (integer)

output file: merged output CL t + 0 (integer)

output file: merged output SC t + 0 (integer)

output file: Land cover of burned area t + 0 (byte)

output file: primary merged output JD revised t - 1 (integer)

output file: primary merged output CL revised t - 1 (integer)

output file: primary merged output SC revised t - 1 (integer)

output file: primary merged Land cover of burned area t - 1 (byte)





/*

If JD > 0 for all sensors


If JD > 0 for SPOT/ATSR sensors


If JD > 0 for SPOT/ATSR-2 sensors


If JD > 0 for ATSR/ATSR-2 sensors


If JD > 0 for ATSR (or SPOT or ATSR-2) AND CL > CL threshold

Then: SC = 1 (2 or 8), JD = ATSR (SPOT or ATSR-2) JD, CL = ATSR (SPOTor ATSR-2) confidence, LC =

Globcover class



Then:

t+0, SC = 0, JD = 0, CL = 0, LC = 0



*/


The land cover map is changed from byte to integer so it can be stacked by GDAL (Function 7.6)

fire_cci


Issue/Rev-No.: 2.2


Function 7.6 cci_char2int

input file: LC product (byte)

output file: LC product (integer)

input value: pixels

inputvalue:lines

These data files are then stacked to a .vrt file and converted to the GeoTiff format with GDAL, P1.4.2.

Prior to the stacking a header file must be created using cci_create_envi_header (function 7.7).

Function 7.7cci_make_envi_header



input value: data type (integer)

input value: top_left_lon (integer)

input value: top_left_lat (integer)

input value: pixel_size (integer)

input value: UTM zone(integer)

input value: Hemisphere(integer)

input value: image name (string)

Once JD, SC, CL and LC are stacked into a GeoTiff the file is called by the python code (Function

7.8) and the uncertainty layer is calculated. The uncertainty layer is then converted from a GeoTiff to

ENVI format using a GDAL function.

Function 7.8PredictUnc_pixelproduct.py

input data: path to code

input data: path to input data directory

input data: data input flag (e.g. PixelProduct)

output data: path to output file

input data: optional land cover

/*: BAp = f(gpcl,gpnei) if pixel is classified as “burned”

BAp = Uaunburned if pixel is classified as “unburned”

Where: f Logistic Generalized Linear Model

BAp burned area proportion in the reference data

gpcl confidence level of the merged product

gpnei number of neighbour burnt pixels in a 9x9 window

landcover Globcover ID

Uaunburned User accuracy for the unburned category

*/

A mask indicating where single sensor detections should be revised is created using the performance

look up and the land cover map (function 7.4 cci_create_screen).










fire_cci


Issue/Rev-No.: 2.2






















/*

When LC = n


*/

The mask and uncertainty later are then used to remove data and revise the JD, SC, CL and LC

(function 7.9 cci_pix_unc_screen).

Function 7.9cci_pix_unc_screen2

input file: merged SC

input file: merged JD

input file: merged CL

input file: merged uncertainty

input file: merged LC

input file: ATSR screen

input file: ATSR2 screen

input file: VGT screen

input file: MERIS screen

ouput file: revised CS

ouput file: revised JD

ouput file: Revised CL

ouput file: Revised LC

input data: pixels

input data: lines

The revised JD, SC, CL and LC are then passed to P1.5, (section 7.3.5.5) and mosaicked into the

global subtiles. The results are also passed to the GRID merge procedure which includes calculation of

standard error of burned area Part 1.4 (section 7.3.5.4).

fire_cci


Issue/Rev-No.: 2.2


7.3.5.3 Processing part 1.3: The 1/360 high resolution merge

If the time series dates precede MERIS data the processing moves onto part 1.4 (section 7.3.5.4),

otherwise the higher resolution merge combines the MERIS data with the merge results of stage 1.2.1

if they exist, no final products are produced at the end of this part. The processing steps are as follows

(Figure 42). When processing begins a blank data set is created for the previous month (function 7.1).

A land cover map, currently GlobCover (which will eventually be land_cover_cci) is used for the

landcover map. During the merging as with the primary merge double counts between the previous

month merge and current month data are tested for. This may be when the data in the current month

double counts a late burn from the previous month which has a lagged date because of obscured

observation of the surface or long overpass intervals. The check involves comparing the results of the

merging of the previous month (t – 1) to data of the current month (t + 0). If a duplicate burn is found,

the burn is deleted from the current month (t + 0) and the sensor combinations, and confidence levels

are revised in the previous month (t - 1). At the beginning of the processing chain a blank data set is

created for the previous month (t - 1).

A second contextual screen is then applied again looking at single sensor detections but this time

examining the probability (uncertainty) that the pixel burned given that the surrounding pixels burned.

The BA and CL are stacked and passed through a python programme to calculate BA uncertainty. The

uncertainty of the BA estimates on the Pixel Product is expressed in probabilistic terms, as the

probability that a pixel is really burnt (PB). Errors were measured using reference data derived at the

10 Study Sites and regression analysis was used to generate the uncertainty quantification models. For

the Pixel Product a logistic regression was calibrated for pixels classified as burned. The explicative

variables were the Confidence Level (CL) and the number of neighbour pixels classified as burned, in

a 9x9 moving window (NEI). CL and NEI values are not available for pixels classified as unburned,

for those, the PB was estimated from the ratio between the number of pixels classified as unburned by

the product but burned by the reference data and the number of pixels classified as unburned by the

product. The uncertainty is then compared to the uncertainty thresholds and the JD, CL, SC and LC is

removed (screened) if the uncertainty is below the threshold. The 10 by 10 degree tiles are then passed

on to the global sub tile processing P1.5 (section 7.3.5.5) The resolution is 1/360° and the results are

passed into stages 1.4 and 1.5.

fire_cci


Issue/Rev-No.: 2.2


Figure 42: Processing steps for the secondary merge, P1.3

fire_cci


Issue/Rev-No.: 2.2


The module begins by checking the meris_switch, 1 = YES, 0 = NO.

If NO:

The processing continues to stage 4.

If YES:

Continue

Then the script checks for the existence of MERIS data from the previous month (t-1)

If NO:

A blank dummy file is created (function 7.2) where BAS=0, CON=0, NOV=0, this ensures that data

files are available to pass through the merging function. The missing data are reported to a text file.

The YES condition is now satisfied.







If YES:

The 1/120° data sets are resampled to the equivalent MERIS resolution 1/360°. This is done with

cci_dissaggregation(function 7.10).

Function 7.10 cci_dis-aggregation / cci_dis-aggregation_char*

input file: data to resample (short integer / character*)

output file: data at higher resolution (short integer / character*)



input value: number of pixels to disaggregate to (integer)

Then the data files are passed into the second merge function, cci_secondary_merge_check (function

7.11).

Function 7.11 cci_secondary_merge_check_conf

input file: primary merged product 1/360 SC (integer)

input file: primary merged product 1/360 JD (integer)

input file: primary merged product 1/360 CL (integer)

input file: MERIS product 1/360 JD (integer)

input file: MERIS product 1/360 CL (integer)

input file: LC product 1/360 (char)

input file: secondary merged SC t - 1 (integer)

input file: secondary merged JD t - 1 (integer)

input file: secondary merged CL t - 1 (integer)

input file: secondary merged LC t - 1 (byte)

input file: confidence threshold mask sensor w (integer)

input file:confidence threshold mask sensor x (integer)

input file: confidence threshold mask sensor y (integer)

input file: confidence threshold mask sensor z (integer)

output file: secondary merged SC t + 0 (integer)

output file: secondary merged JD t + 0 (integer)

output file: secondary merged CL t + 0 (integer)

output file: secondary merged LC t + 0 (char)

output file: secondary merged SC revised t - 1 (integer)

output file: secondary merged JD revised t - 1 (integer)

output file: secondary merged CL revised t - 1(integer)

fire_cci


Issue/Rev-No.: 2.2


output file: secondary merged LC revised t - 1 (char)




The function then revises the merged data:

/*

IF JD = 0 for MERIS and SC > 0 for first merge

THEN SC = SC for first merge (1 OR 2 OR 4 OR 8 OR 9 OR 10 OR 12), JD = JD for first merge, CL = CL for

first merge, LC = globcover class for first merge

IF JD > 0 for MERIS AND SC = 1 OR 2 OR 4 OR 8 OR 9 OR 10 OR 12 for first merge

THEN SC = 5 OR 6 OR 7 OR 11 OR 13 OR 14 OR 15, JD = earliest JD, CL = highest confidence, LC =

globcover class

IF JD > 0 for MERIS AND SC = 0 for first merge

THEN SC = 3, JD = MERIS JD, CL = MERIS confidence, LC = globcover class



Then:

t+0, SC = 0, JD = 0, CL = 0, LC = 0



*/


The land cover map is changed from character to byte so it can be stacked by GDAL (Function 7.6)

Function 7.6 cci_char2int

input file: LC product (byte)

output file: LC product (integer)

input value: pixels

inputvalue:lines

These data files are then stacked to a .vrt file and converted to the GeoTiff format with GDAL, P1.4.2.

Prior to the stacking a header file must be created using cci_create_envi_header (function 7.7).











Once JD, SC, CL and LC are stacked into a GeoTiff the file is called by the python code (Function

7.8) and the uncertainty layer is calculated. The uncertainty layer is then converted from a GeoTiff to

ENVI format using a GDAL function.

fire_cci


Issue/Rev-No.: 2.2


Function 7.8PredictUnc_pixelproduct.py





input data: optional land cover

/* BAp = f(gpcl,gpnei) if pixel is classified as “burned”

BAp = Uaunburned if pixel is classified as “unburned”

Where: f Logistic Generalized Linear Model

BAp burned area proportion in the reference data

gpcl confidence level of the merged product

gpnei number of neighbor burnt pixels in a 9x9 window

landcover Globcover ID

Uaunburned User accuracy for the unburned category

*/

A mask indicating where single sensor detections should be revised is created using the performance

look up and the land cover map (function 7.4 cci_create_screen).





























fire_cci


Issue/Rev-No.: 2.2



/*

When LC = n


*/

Where the uncertainty value is below the threshold the JD, SC, CL and LC are reset to zero as they are

no longer considered as burned area (function 7.9 cci_pix_unc_screen3).

Function 7.9cci_pix_unc_screen3

input file: merged SC

input file: merged JD

input file: merged CL

input file: merged uncertainty

input file: merged LC

input file: ATSR screen

input file: ATSR2 screen

input file: VGT screen

input file: MERIS screen

ouput file: revised CS

ouput file: revised JD

ouput file: Revised CL

ouput file: Revised LC

output file: screened data check

input data: pixels

input data: lines

7.3.5.4 Processing part 1.4: Aggregation and generation of the tiled GRID product

In this part P1.4 the 10 by 10 degree merged tiles are used to calculate the final GRID product

attributes and are aggregated and written to a global file in the NetCDF format (Figure 43). The

meris_switch determines if the final product is 1/120 degree or 1/360 degree. All input data are split

into half months prior to calculations (function 7.12 cci_split_month) except for the NOV data which

is received as half monthly summaries directly from the BA algorithm.

fire_cci


Issue/Rev-No.: 2.2


Figure 43: Processing steps for the generation of the pixel product, P1.4

fire_cci


Issue/Rev-No.: 2.2


Function 7.12cci_split_month

input file: merged JD (integer)

input file:merged CL (integer)

input file: merged LC (integer)

input file:merged CL (integer)

output file: JD first half of month (integer)

output file: CL first half of month (integer)

output file: LC first half of month (integer)

output file: JD second half of month (integer)

output file: CL second half of month (integer)

output file: LC second half of month (integer)

Input data: pixels

Input data: lines

Sum of total burned area and burned area per and cover (P1.4.1)

The total area burned and the total area burned per land cover are calculated (Function 7.13

cci_aggregation_grid_PSD), these are the source for layers 1 and 5-22 of the final grid product. A

reference file with the total area in m2 is also created and used along with the burned area sum to

calculate the standard error (see standard error explanation).

Function 7.13 cci_aggregation_grid_PSD

input file: merged JD 1/120 or 1/360 (integer)

input file: merged CL 1/120 or 1/360 (integer)

input file: merged LC 1/120 or 1/360 (byte)

output file: layer 1 aggregated ASB file (m2), GRID product 0.5 deg (float)

output file: aggregated CL - currently not required 0.5 deg (float)

output file: aggregated CL - standard deviation file, currently not required 0.5 deg (float)

output file: layer 5 aggregated ASL1 GRID product (float)


















output file: total cell area (m2) 1/120 or 1/360 0.5 degree cell (float)



input value: sensor resolution, metres (integer)

input value: window size, pixels (integer)

fire_cci


Issue/Rev-No.: 2.2


/*

FOR k

IF JD > 0 (i.e. burned pixel)

THEN

ASB = total pixels burned (equation 7.1)

Equation 7.1 Sum of burned area:

SBk= ∑ 𝐵𝐴𝑖𝑗

Where, BA = area of burned pixel, k = grid cell and i j are coordinates of individual pixels.

*/

Standard error, of total burned area (P1.4.2)

For the Grid Product, uncertainty is expressed as a standard error, similarly as in the GFED product.

The standard error is calculated using the sum of the burned area per 0.5 degree cell and the total area

of each 0.5 degree cell. Errors were measured using reference data derived at the 10 Study Sites and

regression analysis was used to generate the uncertainty quantification models. The uncertainty of the

Grid Product was expressed as the standard error of the BA proportion estimated by the product (SEp

and EBAp respectively) in each grid cell i. A linear regression model was calibrated to model SEp:

EBApEpS i 10ˆˆˆ

whereβ0 and β1 are the parameters estimated through maximum likelihood methods.

First header files are created for these layers (Function 7.7 cci_create_envi_header) and these two files

are stacked using GDAL (see function explanation) to provide the correct format for a python

programme to ingest the data files (function 7.14 PredictUnc_gridproduct_py).

Function 7.14PredictUnc_gridproduct.py





/*

Prediction functions

AEO = f(BApG)

Where: AEO = Absolute error observed

f = linear model

BApG = burned area proportion in the grid product

*/

The output file is then converted from GeoTiff to ENVI format using GDAL functions. This output

forms the basis of layer 2, standard error, in the final GRID product.

Number of patches (F1.4.3)

This is layer 4 of the GRID product. Three sequential programmes are run to create and count burned

area patches within each 0.5 degree cell. The procedure is as follows:

1) Vertical segmentation (function 7.15 cci_tile_seg). In this function the adjacent JD detections

are coalesced into larger burned area segments to reduce the number of JD patches into fewer

burned area segments. If adjacent to a burned pixel the right hand pixel and/or the pixel in the

row above are assigned the same reference value taken as the Julian Day value. The process

creates vertically striped patches.

fire_cci


Issue/Rev-No.: 2.2


Function 7.15 cci_tile_seg

input file: primary / secondary merged JD (integer)

output file: segmented patches (integer)

input value: window size, pixels e.g. 56 / 560 (integer)



/*

//loop around each cell

FOR k

FOR i

FOR j

//Assign same value to right adjacent pixels

If (JD > 0 pixelij AND JD > 0 in pixeli+1j) THEN value pixeli+1j = value pixelij

//Assign same value as row above adjacent pixels

If (JD > 0 pixelij and JD > 0 pixelij-1) THEN value pixelij = pixelij-1

*/

Where k = grid cell, I = columns, j = rows

2) Merging of the segments. Each of the segments are then coalesced and numbered into distinct

and discrete burned (function 7.16 cci_tile_morph). An iterative search and merge of adjacent

segments (equivalent to 4 neighbours) is applied. The programme works by searching for

adjacent segments, when adjacent segments are found all pixels in both segments are assigned

the same id and creates one patch. The programme then checks if other segments are adjacent

to this one, if not then the programme searches for the next segment id and repeats the

process. Each new patch is tagged with a different numerical id. When all the segments have

been assigned to patches the results are passed to step 3.

Function 7.16 cci_tile_morph

input file: segmented patches (integer)

output file: morphed patches (integer)

inputvalue: window size, pixels e.g. 56 / 560 (integer)

input value: pixels

input value: lines

/*

FOR k

// Find maximum (max) and minimum (min) values per cell k assigned in cci_seg (function 7.8),

FOR i

FOR j

//e.g. for max:

If value pixelij> max, max = value pixelij

If value pixelij< min, min = value pixelij (when value pixelij> 0)

//Find the adjacent segments

//Cover all possible adjacent segment values by iterating between the max and min values

FOR min to max

FOR i

FOR j

//Find pixels that are in adjacent segments, then assign the first segment value to the adjacent segment

if value pixelij> 0 and pixeli+1j > 0, value pixeli+1j = value pixelij

// then assign all pixels in the adjacent segment with the first segment value

IF valulepixeli+nj+nEQ value pixelij THEN

valuepixeli+nj+n= valuepixelij

//Now check if the merged segment is adjacent to another segment (min->max)

FALSE

Look for next case of two adjacent segments.

*/

Where k = grid cell, I = columns, j = rows

fire_cci


Issue/Rev-No.: 2.2


3) Calculation of number of patches, cci_patch_stat_area (function 7.17). This function searches

for unique patch reference numbers, when a new reference number is found a value of 1 is

added to a counter.

Function 7.17 cci_tile_patch_stat_area

input file: morphed patches (integer)

output file: number of patches (float)

output file: number of patches (integer)

input value: resolution 1/120 1/360 (integer)



input value: resolution 1/120 1/360 (integer)

input value: grid dims (60 /180) (integer)

input value: top left latitude (integer)

/*

//loop around each cell

FOR k

//Loop through possible patch values (min to max)

FOR m = min to max

FOR i

FOR j

IF value pixel = m

Patch count = patch count + 1

Fraction of observed area (P1.4.4)

This is the source of layer 3 of the GRID product. For this layer the algorithm works out the

proportion of area in a 0.5 degree grid cell that has never been observed during the month or discarded

from the burned area processing due to contamination by cloud, haze, snow, topographic shadow or if

data was flagged as unsuitable for burned area detection. It considers data from all sensors available in

each time step. This information extracted from the layer containing the number of valid observations

(NOV). There are two steps, firstly identify unobserved pixels and water then secondly aggregate to

the 0.5 degree cell.

In the first step, the NOV layer for each BA product is read. Each pixel indicates how many times the

pixel was observed in the time period. If the number is greater than 1 in any NOV data set then the

pixel is flagged as being observed. If the number is less than 1 in all of the NOV data sets then the

pixel is flagged as not being observed. Water pixels are flagged as -1 and not considered as land

surface to observe (function 7.18 cci_merge_nsno). If there is no BA data for any one of the four

sensors, a blank files is created and passed thorugh the algorithm. The meris_switch determines which

resolution the data set is created at.

Function 7.18cci_merge_nsno

input file: Sensor w NOV 1/120 or 1/360 (integer)

input file: Sensor x NOV 1/120 or 1/360 (integer)

input file: Sensor y NOV 1/120 or 1/360 (integer)

input file: Sensor z NOV 1/120 or 1/360 (integer)

input file: Sensor w BS 1/120 or 1/360 (integer)

input file: Sensor x BS 1/120 or 1/360 (integer)

input file: Sensor y BS 1/120 or 1/360 (integer)

input file: Sensor z BS 1/120 or 1/360 (integer)

output file: NSNO t + 0 all 1/120 or 1/360 (integer)



fire_cci


Issue/Rev-No.: 2.2


/*

FOR i

FOR j

IF sensor x OR sensor y OR sensor x > 0, then surface = observed

IF sensor x OR sensor y OR sensor x = -1, then surface = water

*/

Where, i = columns, j = rows

In the second step, the results of Step 1 are corrected for area and aggregated to 0.5 degrees and the

proportion of valid observations is calculated from the NOV layer, whilst allowing for areas that are

taken up as water. This accounts for land area where pixels were either not observed or deemed

unsuitable for processing (i.e. data discarded as unsuitable due to clouds, bad pixel data). The value is

calculated as percentage, function 7.19 cci_aggregation_nsno_area_16.

Function 7.19 cci_aggregation_nsno_area_16

input file: NOV t + 0 (integer)

output file: ANS GRID product (integer)



input value: sensor_res, 1/120 or 1/360 (integer)

input value: window_size, pixels e.g. 60 / 180 (integer)

/*

FOR k

FOR i

FOR j

%valid surface observationsk = (NOV_countk/(pixel area in gridk - water_area in gridk))*100)

Equation 7.4 Percentage of valid observations

VOk = (𝑣k/(𝑐k−𝑤k))∗100

Where VO = number of valid observations, k = grid cell, v = total area of valid pixels, c = total area of pixels in

cell, w = total area of water pixels in cell

*/

Where i = columns, j = rows, k = grid cell

Create the output format (F1.4.5)

The final step in part 5 is to combine the results of the 10 by 10 degree tiles into a global file. This is

undertaken in the second c-shell script.

#######################################################"

echo " ESA CCI Fire ECV shell script: cci_global_grid_processor.csh"




echo " #######################################################"

echo ""

echo " Syntax: cci_global_grid_processor.csh <year>,<month>,<Processor flag (0 = Merge, 1 =

MERIS ONLY, 2 = VGT ONLY)>"

echo " set year"

echo " set month"

echo " Generate which product (0 = Merge, 1 = MERIS ONLY, 2 = VGT ONLY)"

fire_cci


Issue/Rev-No.: 2.2


echo ""

exit(0)

endif

This is done by creating a master global tile for each layer for each half month (function 7.2 cci_create

raster). The 10 by 10 degree tiles are then mosaicked into a temporary global layer (function 7.20

cci_global_mosaic_float) and the global layers are then combined into a single net CDF file (function

7.21 cci_create_NetCDF_grid). The temporary global layer replaces the master global file and the next

10 by 10 degree tile is mosaicked into it.







Function 7.20 cci_ global mosaic float /cci_ global mosaic int*

input file: global raster for one layer of GRID product (float/integer)

input value: pixel dims for global 0.5 degree data set (integer)

input value: lines dims for global 0.5 degree data set (integer)



input value: x position of 10 by 10 degree tile in the global 0.5 degree data set (integer)

input value: y position of 10 by 10 degree tile in the global 0.5 degree data set (integer)

output file: temporary mosaicked raster for one layer of GRID product (float/integer)

*for number of patches cci_global_mosaic_int is used.

Function 7.21cci_create_NetCDF

input file: GRID layer 1 ASB (float)

input file: GRID layer 2 ASE (float)

input file: GRID layer 3 FOB (float)

input file: GRID layer 4 NOP (integer)

input file: GRID layer 5 ASL1 (float)


















input character: output file name (char)

input value: x dims for global 0.5 degree data set (integer)

input value: y dims for global 0.5 degree data set (integer)

fire_cci


Issue/Rev-No.: 2.2


input value: top left longitude *100 of global 0.5 degree data set (integer)

input value: top left latitude * 100of global 0.5 degree data set (integer)

input value: cell size *100 (integer)

Metadata for the final function is written into the attributes of the NetCDF files.

7.3.5.5 Processing part 1.5: Mosaicking the global sub tiles into the pixel product

The third c-shell produces the global sub tiles for the pixel product.

#######################################################"

echo " ESA CCI Fire ECV shell script: cci_global_pixel_processor.csh"




echo " #######################################################"

echo ""

echo " Syntax: cci_global_pixel_processor.csh <zone>,<year>,<month>,<Processor flag (0 =

Merge, 1 = MERIS ONLY, 2 = VGT ONLY)>"

echo " zone N America = 1, S America = 2, Europe = 4, Asia = 5,Africa = 6, Australia = 8"

echo " set year"

echo " set month"

echo " Generate which product (0 = Merge, 1 = MERIS ONLY, 2 = VGT ONLY)"

Each of the 6 sub areas has a separate script with the same generic format. These scripts also require

the input dates text file and variable values for root_path and bin_path. In each script dimensions for

1/120 and 1/360 are included and selected according to the value of the meris_switch, these values

represent:

Number of pixels / lines in a global data set made of 10 degree tiles

Number of pixels / lines in the global subset with whole 10 degree tiles

x, y coordinates of the 10 degree sub tile in a global data set of 10 degree tiles

x, y coordinates of the 10 degree tile in the global sub set with 10 degree tiles

Number of pixels / lines to clip from sub set with whole 10 degree tiles to match the exact

dimensions of the subsets defined in the PSD (sub tile dimensions are not multiples of 10 degrees

and need to be clipped)

Resolution of cells (10 -8)

Using results of section 7.3.5.2 or section 7.3.5.3 the revised JD, SC, UNC_CL and LC are mosaicked

into the global subtile areas and stacked into the final tiff format (Figure 44).

fire_cci


Issue/Rev-No.: 2.2


Figure 44: Processing steps for the grid product, P1.5

The script is hard code to loop round the tiles that fall within the global subset using the ‘L’ and ‘C’

references in the filename. For each tile location and position details are calculated (Function 7.1

cci_tile_attributes.c).

Function 7.1 cci_tile_attributes

Input value: L_coord

Input value: C_coord

Output file: top left lat

Output file: top left long

Output file: Coordinate of y in global grid product @ 0.5 degree resolution

Output file: Coordinate of x in global grid product @ 0.5 degree resolution





First for each layer JD, SC, CL and LC a raster the size of the global subset is created (Function 7.2

cci_create_raster).







fire_cci


Issue/Rev-No.: 2.2


Each 10 by 10 tile is then mosaicked into the global subset (Function 7.22)

Function 7.22cci_global_mosaic_shortint

input file: global sub tile (integer)

input value: Pixels of global sub tile (integer)

input value: Lines of global sub tile (integer)

input value: Offset of pixels in global sub tile relative to global coordinates (integer)

input value: Offset of lines in global sub tile relative to global coordinates (integer)

input file:10 by 10 degree tile mosaic into global sub tile

input value: Pixels of 10 by 10 degree sub tile

input value: Lines of 10 by 10 degree sub tile

input value:Offset of pixels in 10 by 10 tile relative to global coordinates (integer)

input value:Offset of lines in 10 by 10 tile relative to global coordinates (integer)

output file: global sub tile mosaicked with 10 by 10 degree tile (integer)

When the mosaic has been completed the global sub tile is clipped to the dimension stated in the PSD

(Function 7.23) for each of the JD, SC, CL and LC.

Function 7.23 cci_snip

input file: file to resize (char/integer/float)

output file: resized file (char/integer/float)



input value: x_start (integer)

input value: y_start (integer)

input value: x_size (integer)

input value: y_size (integer)

input value: bytes_to_skip (integer)

These data files (JD, SC, CL and LC) are then converted to a .vrt file and then to aGeoTiff format with

GDAL, P1.4.2. Prior to the stacking a header file must be created using cci_create_envi_header

(function 7.7).











/*

gdalbuildvrt -separate –overwrite<SC><JD><CL ><LC><filename.vrt>

gdal_translate -ot Int16 -of GTiff<filename.vrt><filename.tiff>*/

Where:


A generic metadata file is then copied from a source directory into the same directory as the stack. The

PIXEL product is completed.

fire_cci


Issue/Rev-No.: 2.2


8 References Bachmann, M., Borg, E., Fichtelmann, B., Günther, K., Krauß, T., Müller, A., Müller, R., Richter, R.

(2014). ESA CCI ECV Fire Disturbance, Algorithm Theoretical Basis Document – Volume I– Pre-

processing, Fire_cci_Ph2_DLR_D3_6_1_ATBD_I_v2_2.pdf, (https://www.esa-fire-

cci.org/webfm_send/734)

Benali A., Mota B., Pereira J.M.C., Oom D., Carvalhais N. (2013), ’Global patterns of vegetation fire

seasonality’. (EGU2013-11632) EGU General Assembly, Vienna, April 7-12, 2013

Bradley, A. & Tansey, K. (2014). ESA CCI ECV Fire Disturbance, Algorithm Theoretical Basis

Document Volume III BA Merging: Fire_cci_Ph3_UL_D3_6_3_ATBD_III_v2_3.pdf,

https://www.esa-fire-cci.org/

Boykov, Y., Kolmogorov,V. (2004). An experimental comparison of min-cut/max-flow algorithms for

energy minimization in vision.IEEE Transactions on Pattern Analysis and Machine Intelligence,

26(9): 1124 -1137.

Chuvieco, E., Calado T., Oliva P., (2014). ESA CCI ECV Fire Disturbance - Product Specification

Document, Fire_cci_Ph2_UAH_D1_2_PSD_v4_3.pdf, https://www.esa-fire-cci.org/

Fried, R., Schettlingerand, K.,Borowski, M. (2011). Robust Time Series Filters. R packageversion 3.0,

http://CRAN.R-project.org/package=robfilter.

Killick, R.,Eckley, I. A. (2010). R package- changepoint: Analysis of Changepoint Models. Lancaster

University, Lancaster, UK.

Killick, R., Fearnhead, P., Eckley, I. A. (2012). Optimal detection of changepoints with a linear

computational cost.In Submission.

Krauß, T., Günther, K., Bachmann, M. Alonso, I., Calado, T., Bradley, A., Pereira, J.M., Mota, B.,

Gstaiger, V. (2012). ESA CCI ECV Fire Disturbance, System Interface Definition and Processor

Guidelines, Fire_cci_Ph2_DLR_D3_1_1_SIDPG_v1_2.pdf, https://www.esa-fire-cci.org

Krauß, T., Gstaiger, V., Günther, K., Bradley, A. Alonso, I., Mota, B. (2014). ESA CCI ECV Fire

Disturbance, Input-Output Data Definition, Version 1. Fire_cci_Ph3_DLR_D3_8_IODD_v2_2.pdf,

https://www.esa-fire-cci.org

Kugiumtzis, D., Tsimpiris, A. (2010). Measures of Analysis of Time Series (MATS): A MATLAB

toolkit for computation of multiple measures on time series data bases. Journal of Statistical Software,

33(5): 1-30.

Oom, D., J.M.C. Pereira (2013). Spatial autocorrelation in global vegetation fires: exploratory analysis

of screened MODIS hotspot data. International Journal of Earth Observation and Geoinformation. 21:

326–340.

Pereira, J.M., Mota, B.,Alonso, I., Calado, T.,Oliva, P., Gonzalez-Alonso, F., (2014). ESA CCI ECV

Fire Disturbance, Algorithm Theoretical Basis Document – Volume II,

Fire_cci_Ph3_ISA_D3_6_2_ATBD_II_v2_2.pdf, https://www.esa-fire-cci.org/

PO-ID-ACR-GS-0003, “The AMORGOS MERIS CFI Software User Manual and Interface Control

Document”, prepared by L. Bourg and F. Etanchaud, (2007). Available online (Last access:

24/02/2012): http://earth.esa.int/envisat/services/amorgos/download/Amorgos_ICD-SUM_3.0a.pdf

Richter, R. (2010). Atmospheric/Topographic Correction for Satellite Imagery, User Guide, Version

7.1, January 2010, DLR, Remote Sensing Data Sensor, DLR-IB 565-01 /10

Schläpfer, D., Nieke, J., Itten, K.I. (2007). Spatial PSF Nonuniformity Effects in Airborne Pushbroom

Imaging Spectrometry Data.IEEE Transactions on Geoscience and Remote Sensing, 45(2): 458-468.



http://cran.r-project.org/package=robfilter




http://earth.esa.int/envisat/services/amorgos/download/Amorgos_ICD-SUM_3.0a.pdf

D3.7 Detailed Processing Model FIRE CCI

Documents

Transcript of D3.7 Detailed Processing Model FIRE CCI