As

Xa

b

c

a

ARRA

KCCGDC

1

pistatdtl(snaptJ

0d

Computers and Electronics in Agriculture 73 (2010) 74–83

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journa l homepage: www.e lsev ier .com/ locate /compag

utomated canopy temperature estimation via infrared thermography: A firsttep towards automated plant water stress monitoring

uezhi Wanga,∗, Weiping Yangb, Ashley Wheatonc, Nicola Cooleyc, Bill Morana

Dept. Electrical and Electronic Engineering, University of Melbourne, Gratten Street, Melbourne, VIC 3010, AustraliaSchool of Electronic Science and Engineering, National University of Defense Technology, Hunan 410073, PR ChinaMelbourne School of Land and Environment, University of Melbourne, VIC 3010, Australia

r t i c l e i n f o

rticle history:eceived 8 November 2009eceived in revised form 16 March 2010ccepted 16 April 2010

eywords:anopy temperature estimationrop water stress index (CWSI)aussian mixture clustering

a b s t r a c t

Canopy temperature estimation is an important process for plant water status monitoring. In this paperthe problem for measurement data of the scene acquired via an infrared (IR) thermography system isconsidered. An optical image taken from the plant canopy is aligned with the underlying IR image, sothat plant leaf area can be extracted via simple colour identification techniques and then the tempera-ture distribution of the leaf area is obtained. The success of this procedure relies on the assumption thatboth optical–IR image alignment and leaf area extraction are perfect. In practice, such assumptions arerarely justifiable and the computed result can often be found undesirable. Particularly, the simple colouridentification technique fails when temperatures of reference leaves, which are embedded in the canopy

ata fusionolour identification

and provide known conductance to water vapour, are required to be estimated. In this paper, we addressthis issue and propose a novel algorithm to solve the problem. The underlying plant leaf temperaturedistribution is considered to be the fusion of two temperature densities separable via a combinationof colour identification and Gaussian mixture distribution extraction techniques. A N-average methodis tested with moderate success to estimate reference leaf temperatures from the estimated leaf tem-perature distribution. Our experimental results demonstrate the effectiveness and consistency of the

proposed algorithm.

. Introduction

Plant canopy temperature acts as a good indicator of thelant water status. When plants are undergoing water stress they

ncrease their temperature. The conductance of water throughtomata on leaves decreases when water supply becomes limited tohe plant roots. Canopy temperature, relative to ambient temper-ture, changes as a result of stomatal conductance control of plantranspiration. Thus, if plant water stress increases, transpirationecreases and plant temperature may exceed air temperature. Onhe other hand, non-stressed plants will have canopy temperaturesess than air temperature, particularly when vapour pressure deficitVPD) is not greater than 4 kPa (Olivo et al., 2009). The crop watertress index (CWSI) relates canopy–air temperature difference toet radiation, wind speed and vapour pressure deficit (Jackson et

l., 1981). However, a surrogate measure is calculable from the tem-eratures of the canopy and reference leaf surfaces correspondingo fully transpiring and non-transpiring canopies (Jones, 1999a;ones et al., 2002; Moller et al., 2007). Thus, by monitoring plant

∗ Corresponding author. Tel: +61 3 8344 0373; fax: +61 3 8344 6678.E-mail address: xwang@unimelb.edu.au (X. Wang).

168-1699/$ – see front matter © 2010 Elsevier B.V. All rights reserved.oi:10.1016/j.compag.2010.04.007

© 2010 Elsevier B.V. All rights reserved.

canopy temperature and the temperatures of wet and dry leaves,it is possible to estimate the underlying plant water stress statusand therefore, intelligently control the related irrigation process.Fig. 1 illustrates a typical plant irrigation strategy, where the plantwater status information acquisition (inside the red dotted box)plays a critical role in the optimization of plant productivity andwater usage.

The Melbourne School of Land and Environment at the Uni-versity of Melbourne is currently conducting a research programto estimate spatial and temporal variation in water status ofgrapevines using data collected from remotely sensed infrared (IR)and optical digital images (Wheaton et al., 2007). The ultimateresearch goal is to design an automatically controlled irrigation sys-tem using CWSI via non-destructive IR thermography sensing andautomated measurement processing (Jones, 1999b, 2004).

Typically, measurement data of the IR thermography sensingsystem consists of a reference optical image and an IR image. Theoptical image, taken at the same location as the IR image using a

digital camera, provides a visible light view of the IR image scene. Inthe process of plant water stress analysis, the value of CWSI is cal-culated based on the knowledge of the underlying plant canopytemperature, and the dry and wet reference leaf temperatures.These temperatures can be estimated once the temperature distri-

X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83 75

he sensor here being canopy temperature detection via thermal imagery.

bTarpiatm(ep

catiaoed

dGcictioairtlttaifcatGdtpmeot

Table 1Notations.

Fo Plant (reference) optical image.FIR IR image within the scene of Fo .FOIR Optical image registered with FIR . Note that FOIR ⊂ Fo .FL Plant leaf area image. FL ⊂ FOIR .C(FL)Colour values of image FL in HSV colour space.T Temperature distribution of an image scene.TIR Pixel-wise temperature distribution (matrix) of the scene in FIR .

Fig. 1. Illustration of irrigation actuation via a plant based sensor. T

ution relative to leaf area is obtained from the measurement data.he acquisition of the temperature distribution of plant leaf area ismajor component in the measurement data processing. With the

eference optical image, the area of interest (e.g., the plant canopy,articularly the transpiring leaves other than ground or sky) may be

dentified in the IR image. Subsequently, temperature data associ-ted with the area of interest is extracted and statistical analysis canhen be performed. The effectiveness and robustness of the analysis

ethod have been reported on several occasions in the literatureMoller et al., 2007; Jones, 2004; Jones and Leinonen, 2003). How-ver, automating this process for automatic controlled irrigationrogram poses major challenges.

In previous work (Yang et al., 2009), an automated cross-orrelation (ACC) based image registration method was developednd described. The ACC algorithm is able to effectively determinehe overlapping area between the IR image and reference opticalmage without human intervention and without the assistance ofrtifacts placed in the scene to facilitate registration. The registeredptical image can then be used to identify plant canopy area andxtract the associated temperature distribution from measurementata.

In this paper, our focus is to estimate the plant temperatureistribution from the IR image via the registered optical image.iven a perfect match between the optical and IR image, a simpleolour identification technique, based on a single Gaussian modeln colour space, can be utilized to extract leaf area from the opti-al image. The temperature distribution of the plant leaf area canhen be found from the temperature data associated with the IRmage. While this simple method is effective and robust, it reliesn the assumption that both optical–IR image alignment and leafrea extraction are perfect. In practice, when such an assumptionnvariably fails to hold, the estimate can differ considerably fromesults obtained manually. In particular, the simple colour iden-ification technique fails when the temperatures of the referenceeaves, which are embedded in the scene and usually deemed to behe highest and lowest temperatures in the leaf area, are requiredo be estimated. Here we address this issue and propose a novellgorithm to solve the problem. In this approach, the underly-ng plant canopy temperature distribution is considered to be theusion of two temperature densities separable via colour identifi-ation and Gaussian mixture distribution extraction techniques. Inddition to estimating leaf temperature via colour identification,he canopy temperature distribution, also assumed to be a singleaussian, is estimated directly from the associated temperatureata. The final leaf temperature distribution is obtained by fusinghe two estimates. The ideas behind, and the operation of, the pro-

osed algorithm are illustrated in Fig. 2. Furthermore, a N-averageethod is tested to estimate reference leaf temperatures from the

stimated leaf temperature distribution. Results from the analysisf 10 image pairs demonstrate the effectiveness and consistency ofhe proposed algorithm.

TL Pixel-wise temperature distribution (matrix) of plant leaf area FL . TL ⊂ TIR .Tc Average leaf (canopy) temperature.Td Dry (non-transpiring) leaf reference temperature.Tw Wet (full-transpiring) leaf reference temperature.

This paper is organized as follows: Section 2 describes the prob-lem of plant water stress status monitoring via IR thermographyand the issues of automating the process; Section 3 presents asolution for the estimation of plant leaf temperature distribution;Section 4 demonstrates the estimation of the embedded referenceleaf temperatures via a N-average method from leaf temperaturedistribution; Section 5 presents the results of image analysis andalgorithm performance and Section 6 presents the conclusions.

2. Problem of canopy temperature estimation

A key procedure for the evaluation of crop water stress fromplant canopy temperature is to calculate CWSI based on the datacollected from IR thermography systems. The CWSI described by(Jones et al., 2002) is of the following generic form

CWSI = Tc − Tw

Td − Tw(1)

where Td and Tw represent the reference temperatures for dry (non-transpiring) and wet (fully transpiring) leaf surfaces respectively.Tc is the temperature of the transpiring surface, i.e., the actual mea-sured temperature of all sunlit leaves to represent the sunlit portionof the canopy. Although alternative methods for estimating refer-ence temperatures may be found (Guilioni et al., 2008), referenceleaves, which are artificially treated real leaves with known con-ductance to water vapour, can be physically embedded in the scene(Jones, 1999b) and so the reference temperatures Td and Tw can beestimated from the leaf temperature distribution.

It is thus clear that CWSI can be determined once the tem-peratures Td, Tw and Tc are identified using the pixel-by-pixeltemperature data associated with the IR image.

Let FOIR and TIR denote, respectively, the registered opticalimage and the pixel-by-pixel temperature matrix associated withthe IR image FIR; FL , the leaf area in FOIR, and TL the leaf tem-

perature distribution (see Table 1). An estimate of the desired leaftemperature distribution can be derived from:

TL =: {TIR(x), ∀ x ∈FL ∩ TIR} (2)

where x is the pixel index.

76 X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83

F ned in

tos

T

TdTS

3

tHmtfeit

3

i

ig. 2. Automated canopy temperature estimation process. Notations used are defi

Once the temperature distribution of the leaf area TL is found,he canopy temperature Tc can be estimated as the average valuef TL . When the reference dry and wet leaves are embedded in thecene, Td and Tw in Eq. (1) can also be found from TL . Intuitively,

d = max(TL) and Tw = min(TL) (3)

he proposed algorithm for estimating TL is presented andescribed in Section 3 and a graphical summary is shown in Fig. 2.he technique proposed for estimating Td and Tw is presented inection 4.

. Canopy temperature estimation

A straightforward estimator for TL uses leaf temperature extrac-ion via colour identification and is described in Section 3.1.owever, if the optical and IR images are misaligned, this simpleethod is often prone to errors, and the CWSI is very sensi-

ive to these kinds of errors. Therefore, an alternative methodor leaf temperature extraction via temperature data is consid-red in Section 3.2. The ultimate leaf temperature distributions then obtained by fusing these two temperature distribu-ions.

.1. Leaf temperature extraction via colour identification

Plant leaf temperature extraction via colour identificationnvolves the following three steps:

Step 1: Plant leaf extraction: In this step, the image FL (FL ⊂ FOIR),which contains only plant leaves of interest, is obtained via colour

identification. Plant leaf colour, denoted by C(FL), is assumed tofollow a Gaussian distribution, i.e., C(FL)∼N(�L, �L) in HSV (hue,saturation and value) colour space, whereN(x, �) signifies a Gaus-sian distribution with mean x and covariance �. Fig. 3(B) illustratesthe desired leaf area FL ⊂ FOIR which is extracted from Fig. 3(A)

Table 1 and the work inside the bold block can be found in (Yang et al., 2009).

with �L = [100, 0.6, 0.6]′,√

�L = diag[30, 0.5, 0.25]′, and clus-tering probability 0.9.Step 2: Leaf temperature extraction: In this step, the pixel-by-pixelplant leaf temperature distribution TL is extracted by fusing FL

with TIR. Fig. 3(C) gives the plot of the extracted temperature dis-tribution of plant leaf area of Fig. 3(A).Step 3: Canopy temperature statistics: The plant canopy tempera-ture Tc can be estimated by taking the average of TL . Temperaturesof both dry and wet reference leaves Td and Tw if embedded in thescene are calculated using Eq. (3).

While leaf area extraction via colour identification provides asimple and effective solution to leaf temperature estimation, it suf-fers from two shortcomings:

1. Non-leaf objects within the specified colour may be includedas “leaves” and the temperature of non-leaf object can departsignificantly from the leaf temperature range.

2. Unwanted temperature values can be introduced because ofimperfect image registration. Image registration error arisesbecause (a) registration algorithm accuracy is limited; (b) theoptical image detail is not matched exactly with that of the IRimage. Examples of imperfect registration are shown in Fig. 4. Inthese examples, the sky and the metal cross, at which tempera-ture values measured via IR cameras are negative, were includedin the extracted leaf area because of a registration error betweenthe IR and optical image pair.

While only slight changes can be observed in the estimatedcanopy temperature Tc , the inclusion of a non-leaf object may lead

to significant errors for the estimation of reference leaf temper-atures Td and Tw . To address this shortcoming an optimizationmethod is described in Section 3.2 which attempts to removenon-leaf objects from consideration using a Gaussian mixture tem-perature distribution approach.

X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83 77

Fig. 3. Illustration of typical plant canopy temperature via identification of green leaf area. (A) Original optical image. (B) Leaf area image extracted using a single Gaussianmodel with mean HSV value [100, 0.6, 0.6]′ , standard deviation [30, 0.5, 0.25]′ and clustering probability 0.9. (C) Temperature distribution of leaf area with colour bar oftemperature value in degree (◦C).

F ge is ei the rt to colo

3

dsG

i

p

w

�

iAtttrm

�

Lti

�

ig. 4. Examples of imperfect image registration. (A) Sky area painted in light orant is not fully overlapped with the sky (in light blue) in the optical image. (B) Insidehe white one in the associated optical image. (For interpretation of the references

.2. Leaf temperature extraction from temperature data

This approach is based on the assumption that the temperatureistribution over image scene follows a mixture of M Gaussian den-ities and the underlying sunlit leaf temperature follows a singleaussian distribution.

Denoted by T, the temperature distribution over the scene of FIR

s assumed to be a mixture of M Gaussian densities, i.e.,

(T) = p(T|�)∼M∑

i=1

˛iN(Ti, �i) (4)

here

={

˛1, . . . , ˛M; T1, . . . , TM; �1, . . . , �M

}(5)

s a set of parameters of Gaussian distributions and∑M

i=1˛i = 1.s in a general expectation maximization (EM) problem formula-

ion (Dempster et al., 1997), the parameter set is unobservable inhe parameter space. Therefore, finding the temperature distribu-ion associated with an IR image FIR for given temperature data TIR

equires determination of the set of Gaussian parameters �∗ whichaximizes the (log) likelihood∗ = argmax� ln(p(TIR|�)) (6)

et tj, j = 1, . . . , N, tj ∈ TIR denote the temperature correspondingo the jth pixel of the image FIR of total N pixels. The log-likelihood

n Eq. (6) may be written as

(�) = ln(p(TIR|�)) = 1N

N∑j=1

ln

(M∑

i=1

˛iN(tj − Ti, �i)

)(7)

stimated using the temperature matrix associated with IR image. In this example,ed circle, the black cross extracted from the IR image does not exactly match withr in the figure caption, the reader is referred to the web version of the article.)

where

N(tj − Ti, �i) = 1√2��i

exp{

− 12 �i

(tj − Ti)2}

(8)

In this work it is assumed that the underlying leaf temperature dis-tribution is according to a single Gaussian centered at the mean ofthe leaf temperature distribution T′

L obtained via the colour identi-fication procedure mentioned in Section 3.1. It is also assumed thattwo more single Gaussian components exist in the lower and highersides of the leaf area temperature distribution T′′

L , and that thesetwo components are used to model those possible temperatureswhich are physically above or below the leaf temperature range.Therefore, TIR consists of three Gaussian components with param-eters, most of which are unknown (missing data in the context ofthe EM algorithm). Then the expectation maximization algorithmiteratively calculates the expected value of the complete data log-likelihood:

Q (�, �k−1) = E{�(�)|TIR, �k−1} (9)

which is computed in alternating steps starting from an initial guessof the parameters, �1 = {T1

i, �1

i}3i=1

, where a fixed value for T11 =

mean(T′L) is used in the whole computation as illustrated in Fig. 2.

E step: Evaluation of the conditional expectation over parameter

space. It can be shown that at each iteration k, this step evaluatesthe weights {˛i}3

i=1 for each pixel given �k−1.M step: Maximization of the expectation computed in the E step.In this case, step M computes �k|TIR, {˛i}3

i=1 which maximizes (9).

78 X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83

F ns from an aerial shot of grapevines (distance between rows=3.4 m), as separated by 3G A) Original image FOIR . (B) Image extracted by the leaf temperature component centereda .00◦ . (D) Image extracted by the higher temperature component centered at 56.96◦ . (Fori to the web version of the article.)

ikdtwttrlFvttpGboti

pcinos

4

lrtoaarmiit

Table 2Auto canopy temperature TL estimation using data FOIR and TIR .

1. Leaf area extraction:FL = {FOIR(x), ∀ x satisfy C(FOIR(x))∼(N(�L, �L), �0)}where x is the pixel index, C(FOIR(x)) is the pixel-wise HSV value of image FOIR ,

�L and �L are mean and variance of Gaussian threshold for leaf colouridentification, and �0 is the given Gaussian clustering probability.

2. Find T′L from TIR , which is given by

T′L = {TIR(x), ∀ x that satisfy FL(x) ∩ TIR(x)}

3. EM iterations:Set T1

1 = mean(T′L), T1

2 = −30, T13 = 90, �1

1 = �12 = �1

3 = 102, and k = 2.For given data {tj} = TIR

Do while k < kmax and ε ≤ ε0

(a) E step: Compute ˛i|Tk−1i

, �k−1i

, i = 1, 2, 3 for each pixel j, j = 1, . . . , N

˛ji=

N(tj − Tk−1i

, �k−1i

)

M∑l=1

N(tj − Tk−1l

, �k−1l

)

, i = 1, 2, 3.

(b) M step: Compute Tki

, �ki|˛i, i = 1, 2, 3

Tki

=

N∑j=1

˛jitj

N∑l=1

˛li

, i = 2, 3.

�ki

=

N∑j=1

˛ji(tj − Tk

i)2

N∑l=1

˛li

, i = 1, 2, 3.

(c) Convergence check

If ε =√∑

i

(Tki

− Tk−1i

)2 ≤ ε0,Stop iteration.

otherwise, k = k + 1. Go to E step.

ig. 5. Segmented images and corresponding normalised temperature distributioaussian mixture components. The light orange colour covers the excluded areas. (t 43.65◦ . (C) Image extracted by the lower temperature component centered at −8nterpretation of the references to color in the figure caption, the reader is referred

The iteration is terminated after either a maximum number ofterations are reached or a cost is below a given threshold. It isnown (Dempster et al., 1997) that each E–M step in this proce-ure does not decrease the log-likelihood and typically convergeso the maximum likelihood estimate. Those temperatures in TIR

hich satisfy T′′L∼N(T1, �1) are then extracted via Gaussian clus-

ering and are fused with T′L , the output of the colour based leaf

emperature distribution described in Section 3.1, which gives theequired leaf temperature distribution TL (see Fig. 2). The completeeaf temperature estimation algorithm is summarized in Table 2.ig. 5 shows a typical example of temperature data segmentationia the Gaussian mixture of 3 models. In this example, the leaf areaemperature distribution is successfully extracted from tempera-ure data as in Fig. 5(B). Finally, it should be noted that in therocessing of images for this experiment, the lower temperatureaussian component {T2, �2} contains only the pixels of the testoard while the higher temperature component {T3, �3} containsnly pixels of the ground between leaves. This indirectly verifieshat the Gaussian assumption on the leaf temperature distributions reasonable.

We should point out that the Gaussian assumption for leaf tem-erature distribution in this work is only for temperature datalustering under a Gaussian mixture of 3 models. As demonstratedn the second row in Fig. 5, those clustered components may notecessarily be Gaussian. The final leaf temperature distribution isbtained by fusing the outcomes of both colour and temperatureegmentations.

. Computing temperature of reference leaves

Once the plant leaf temperature distribution TL is computed, theeaf canopy temperature Tc is estimated by Tc = mean(TL). Wheneference leaves are embedded in the scene of image, the estima-ion of these reference temperatures (Td and Tw) are required inrder to compute the plant CWSI. Intuitively, both temperaturesre extreme values in TL and can be simply found as Td = max(TL)nd Tw = min(TL), respectively. However, the Td and Tw are typically

epresented by multiple pixels and the CWSI is sensitive to the esti-ation error on these temperatures. Therefore, a robust algorithm

s needed to estimate the Td and Tw . In an attempt to address thisssue an N-average algorithm is designed to compute these extremeemperatures. This is both statistically appropriate and appealing.

4. Fusing temperature distributions:T′′

L = {TIR, ∀ max{˛j1, ˛j

2, ˛j3} = ˛j

1, j = 1, . . . , N}TL = T′

L ∩ T′′L

The proposed method is motivated by the fact that the refer-

ence dry and wet leaves not only have extreme temperature valuesbut also a significant number of pixels to support these values. Thealgorithm proceeds as follows. Initially, for the calculation of Tw ,a minimum value is found from TL . The average value a1 over asmall region A1 centered at the point where this minimum value is

X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83 79

Table 3N-average method for computing extreme temperature values for reference leavesfrom leaf temperature distribution.

N-average algorithm for computing Tw and Td

Given leaf temperature distribution TL and a small integer r.for j = 1, 2, . . . , N

find location (xj, yj) at which the minimum/maximum value of TL

is found.Let A ⊂ TL be the area centered at (xj, yj) with radius r, andaj = mean(A ⊂ TL).

aaTApAatsra

aTnoleitT

5

12

3

fdwie1ini(w

io2Rma

Table 4Statistical results obtained via “Auto” method (result of Auto 1).

No. File name Td (dry) Tw (wet) Tc (average) Std. Dev. CWSI

1 I0117-01 52.2 32.5 40.40 3.67 0.36222 I0117-02 47.9 30.6 39.66 3.67 0.52373 I0117-03 52.9 34.7 41.24 3.56 0.36054 I0117-04 49.6 35.1 40.96 3.59 0.40485 I0117-05 46.4 29.5 40.65 3.52 0.66006 I0117-06 47.3 28.6 40.72 3.64 0.64877 I0117-07 49.2 27.9 37.68 3.07 0.46118 I0117-08 50.7 28.7 38.96 3.22 0.46579 I0117-09 42.9 28.9 38.31 2.57 0.6703

10 I0117-10 46.1 26.4 40.06 3.13 0.6941

Table 5Results calculated via “manual” method.

No. File name Td (dry) Tw (wet) Tc (average) Std. Dev. CWSI

1 I0117-01 48.2 33.1 42.52 1.78 0.62362 I0117-02 47.2 30.5 41.65 1.78 0.66773 I0117-03 47.9 32.3 42.32 1.48 0.64214 I0117-04 48.5 32.1 44.33 1.45 0.74605 I0117-05 47.1 29.3 44.03 1.67 0.82776 I0117-06 48.7 29.7 44.42 1.62 0.77467 I0117-07 45.2 28.0 41.83 1.18 0.8043

TL = TL \ A (excluding A from TL)endTw/Td = min/max{a1, . . . , aN }

ttained and containing up to r × r pixels is computed. The temper-ture values that belong to this region A1 are then removed fromL and the minimum is found. The average a2 over a small region2 around the point where this minimum is attained is found. Therocess continues in this way—deleting the temperature values in2 and finding the next minimum. The procedure is repeated untilll N averaged minimum values a1, . . . , aN are found. The wet leafemperature is then derived as Tw = min(a1, . . . , aN). Clearly, theame principle can be applied for the calculation of Td. The algo-ithm is given in Table 3. For the test images in Section 5, N = 10nd r = 10.

For the experiment described in Section 5, additional conditionsre considered for the collection of pixel points in calculating Td andw . That is, these points must satisfy two conditions: (1) at least Np

eighbouring leaf pixels are of valid leaf temperature values (pixelsf non-leaf areas are not allowed); (2) the standard deviation of ateast N0 of these valid temperature values is less than T0 (In ourxperiment, Np = 40, N0 = 20 and T0 = 15◦). This leads to the elim-nation of those areas where fewer support pixels are found and/oremperature value fluctuations are too great for the calculation ofd and Tw .

. Experiment results and discussions

The procedure described in Sections 3 and 4 with the aim of

. Automation of all processes (see Fig. 2).

. Checking consistency with results obtained using an alternativemanual image processing method.

. Examining the robustness of the algorithm. For example, wecheck CWSI performance when leaf colour is calculated fromdifferent thresholds.

Test computations were performed using measurement datarom 10 optical and IR image pairs and associated temperatureata. These image pairs were collected as part of an experimenthere two irrigation treatments, each replicated 4 times, were

mplemented from veraison until harvest in season 2008. Thexperimental site located at Dookie College Estate vineyard (36.36S,45.70E) was in a block of Shiraz on Richter 99 rootstock planted

n 1994. Grapevine rows were spaced at 3.4 m and orientatedorth-south containing grapevines spaced at 1.8 m. One treatment

rrigated grapevines according to estimated evapotranspirationETc) and the other treatment withheld irrigation. Treatmentshere implemented from 14 January 2008.

All images were collected using a FLIR PM570 320 × 240 thermalmager and a standard Fujitsu S5600 (1120 × 750) digital camera

n all replicate grapevines between 15.00 and 16.00 on 17 January008. Images were taken of both east and west sides of the canopy.eference surfaces were also included in the image, specifically, (a)att green cardboard sheet, (b) a non-transpiring grapevine leaf,

chieved by covering top and bottom surfaces with petroleum jelly,

8 I0117-08 44.9 28.9 42.28 1.43 0.83649 I0117-09 44.3 28.6 40.95 1.30 0.7866

10 I0117-10 47.4 26.1 42.75 1.57 0.7817

and (c) a wet leaf, achieved by spraying with a low concentrationof soapy water (Jones, 1999a). Images used here are those from thewest side of the canopy and are thus, fully sunlit.

To provide a comparison with CWSI values, plant stem waterpotential (SWP) data which serves as an alternative plant waterstress indicator (Choné et al., 2001) was also collected. Pre-dawnSWP was measured on 1 leaf on each measurement grapevine on16 January. Leaves were wrapped in plastic wrap, severed from thegrapevine, placed in an airtight container and transported on iceto the laboratory for measurement by Schollander Pressure Bomb,model 3000 (Soil Moisture Equipment Corp., CA, USA).

Measurement data were processed automatically as outlined inFig. 2. The registration of optical and IR image pairs was obtainedusing the algorithm developed in (Yang et al., 2009). The algorithmproposed in this paper is utilized to estimate canopy tempera-ture Tc , reference leaf temperatures Td and Tw , and thus CWSI.Figs. 6 and 7 present the registered optical images, IR images andextracted leaf area images. In the leaf area images, the estimatedlocations of “dry” and “wet” reference leaves were highlightedusing red and blue coloured squares, respectively. Table 4 lists aset of statistical results in which a Gaussian threshold of mean[90, 0.7, 0.5]′ (in HSV) with standard deviation [30, 0.5, 0.25]′ wasselected for leaf colour identification with clustering probability0.9.

For a comparison, the statistical results processed manually forimage pairs 1–10 are given in Table 5. Six sunlit leaves per IR imagewere selected by manual comparison with the optical image of thesame scene. ThermaCAM Research Pro (version 2.8 SR-1) softwarewas used to freehand select six sunlit leaves in the IR image (Fig. 8).Leaves were selected to represent a range of leaf angles. The wetleaf and dry leaf were also selected from each image using freehandselection (Table 5). Fig. 8 illustrates how CWSI is calculated frommeasurement data using the “manual” method. Obviously, this kindof manual calculation is labour intensive and requires the skills ofa highly trained person.

To examine robustness of the proposed algorithm (termed the“auto” method), CWSI is computed by using four different HSV(threshold) values in leaf area extraction as shown in Fig. 9. Thefour cases are labelled as “auto 1”–“auto 4” in the order that leaf

80 X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83

Fig. 6. Rows 1 and 2: registered optical and IR image pairs 01– 05. Row 3: extracted leaf area used for leaf temperature distribution. The red square is the computed areafor Td and the blue square is the computed area for Tw . For all cases, the actual reference leaf for Td is at the up right corner of the tag box and the reference leaf for Tw isrepresented by the darkest area in each of the five thermal images.

Fig. 7. Rows 1 and 2: registered optical and IR image pairs 06– 10. Row 3: extracted leaf area used for leaf temperature distribution. The red square is the computed area forTd and blue square is the computed area for Tw . For all cases, the actual reference leaf for Td is at the up right corner of the tag box and the reference leaf for Tw is located atthe darkest area on the second row of the thermal pictures.

Fig. 8. Manual method for CWSI calculation. Areas AR01 and AR02 are selected for calculating Td and Tw and all other six areas are used to calculate Tc . Selection of leaveswere preformed using the freehand tool in ThermaCAM Researcher software. (A) Leaf area selection in image 01. (B) Leaf area selection in image 02.

X. Wang et al. / Computers and Electronics in Agriculture 73 (2010) 74–83 81

F n HSV3

cvl

mdliam

Fr

ig. 9. Illustration of extracted leaf areas using different mean HSV values. The mea; [70, 0.8, 0.8] for auto 4. Standard deviations for all cases are [30, 0.5, 0.25].

olour changes from “dark” to “light”. The histograms (frequencyersus temperature) of leaf temperature in Fig. 9 indicate that fewereaves were included when leaf colour becomes lighter.

The estimated Td, Tw , Tc and CWSI via “auto” and “manual”ethods are compared in Fig. 10. Fig. 11 shows the plot of a CWSI

ata linear regression versus the corresponding SWP values col-ected when obtaining these vine images. This relationship providesndirect confirmation that the proposed temperature estimationlgorithm is calculating crop stress correctly relative to provenethods (SWP).

ig. 10. Comparison of Td , Tw , Tc and CWSI estimated via “manual” and “auto” methods feference. (C) Average leaf temperature. (D) CWSI.

values are [90, 0.7, 0.5] for auto 1; [90, 0.5, 0.5] for auto 2; [90, 0.6, 0.6] for auto

5.1. Results and discussion

The effectiveness of the proposed “auto” method is evidencedby the following observations:

• The linear regression of CWSI results versus SWP data plotted inFig. 11 shows that the values of CWSI estimated by “auto” changesacross the SWP range and are consistent with that by the “man-ual” method in the sense that similar slopes are observed thougha “near constant” bias exists. The slopes presented in Table 6 are

or image pairs 1–10. (A) Temperature of “dry” reference. (B) Temperature of “wet”

82 X. Wang et al. / Computers and Electroni

Fig. 11. Comparison of linear regression of CWSI calculated using different meth-ods and conditions versus stem water potential (SWP). Each data represents onegrapevine. See Table 6 for slope values and average residual of the line fit.

Table 6Statistics of the linear regression results in Fig. 11

•

•

lAwaprcwtIf“

trtset

Method Auto 1 Auto 2 Auto 3 Auto 4 Manual

Fitted line slope −0.2574 −0.3375 −0.4189 −0.2389 −0.1747Average residual 0.0965 0.0927 0.0793 0.0710 0.0518

in the range for those found in other grape varieties (Moller etal., 2007). In Table 6, the average residual, which describe theaverage distance between data and line fit, is also provided.

In addition, curves of quite similar shapes were observed fromthe estimation of Tc and CWSI as shown in Fig. 10.All locations of reference “dry” and “wet” leaves were correctlyfound. This evidence implicitly verifies that the underlying leaftemperature distributions were all correctly estimated by theproposed algorithm.The algorithms developed in this work and in (Yang et al., 2009) asshown in Fig. 2 were implemented in C++ and can automaticallyrun on a Pentium 4, 2.8 GHz desktop PC for an input set of opticaland IR image pairs. A typical processing time for a single opticaland IR image pair is under 5 s.

From Fig. 10(C) it is observed from the “auto” results that theighter the leaf colour, the higher the average leaf temperature Tc .

higher Tc is indicating that the leaves are radiating more energyhich is most likely the result of greater exposure to sunlight. Leaf

rea extracted with a darker green threshold includes a greaterortion of shaded leaves that are not exposed to the same incomingadiation. These leaves are detected as cooler by the IR sensor. Inontrast, the estimated values of Td and Tw for auto 1–auto 4 varyithin a narrow range (see Fig. 10(A) and (B)). The variations of

he estimated CWSI are insignificant between auto 1 and auto 4.n addition, the robustness of the “auto” method is reflected in theact that all reference leave locations were corrected located in allauto” cases.

The performance of “auto” and “manual” departs mainly fromhe calculation of Tc and therefore the values of CWSI. In these two

esults, a large bias can be observed. One reason for this is the facthat the manual method is more selective and can choose the sunliturfaces of leaves only. This will yield a higher Tc . However, morexperiments will be needed to confirm this. One possibility is thathe manual method may in fact be too subjective and over estimate

cs in Agriculture 73 (2010) 74–83

Tc . Tc derived from auto method 3 or 4 (Fig. 9) may better representthe sunlit leaves that collectively represent the vine canopy. Thestandard deviation of Tc for the “manual” method was smaller thanthat of the “auto” method.

In a separate experiment, where 48 IR and optical image pairstaken from a different vineyard were processed using the proposedalgorithms (data not shown), it was found that the estimated Tc ’swere consistently close to those values calculated manually (thedifference is always less than 1 ◦C). However most of the estimatedTd’s and Tw ’s differ in the range of 2–10 ◦ C from that calculated man-ually. One of the reasons is that the estimated leaf area does includenon-leaf objects for which temperatures can be either higher thanthe dry reference leaves or lower than the wet reference leaves.This is largely due to a poor match between IR image and thealigned optical image. This permits non-leaf objects staying in thefused image and their temperatures are such that they cannot beseparated from leaf temperature by using the proposed Gaussianmixture clustering technique. In such a case Td and Tw can no longerbe considered as the extreme temperatures in the estimated leaftemperature distribution. Therefore, an alternative method will berequired to obtain both Td and Tw . This issue will be addressed infuture work. However, we may design a better data collection pro-cedure, such as synchronized and combined photo taking, to avoidthis problem.

6. Conclusions

Following the work initiated in (Yang et al., 2009), we havedeveloped an automatic canopy temperature estimation method.It attempts to automate the sensing and information processing forplant water status monitoring with the aim of providing the sens-ing for an automatic controlled irrigation system. In the proposedalgorithm, the underlying plant leaf temperature distribution isfused from two temperature densities obtained via colour iden-tification and Gaussian mixture distribution extraction techniques,and based on measurement data acquired from an IR thermographysensing system. Moreover, a N-average method is tested to estimatereference leaf temperatures from the estimated leaf temperaturedistribution, which achieves some success when the underlyingleaf area in the IR image can be reasonably identified. Our initialexperiment results presented in this paper verify that the trend ofthe computed CWSI is consistent with an alternative plant waterstress indicator, the SWP data. In particular, canopy temperaturesestimated using our proposed method are in agreement with thosecalculated manually by an expert operator.

References

Choné, X., van Leeuwen, C., Dubourdieu, D., Gaudillère, J., 2001. Stem water poten-tial is a sensitive indicator of grapevine water status. Annals of Botany 87 (4),477–483.

Dempster, A.P., Laird, N.M., Rubin, D.B., 1997. Maximum likelihood from incompletedata via the EM algorithm. Journal of Royal Statistical Society 39 (1), 1–38.

Guilioni, L., Jones, H.G., Leinonen, I., Lhomme, J.P., 2008. One the relationshipsbetween stomatal resistance and leaf temperatures in thermography. Journalof Agricultural and Forest Meteorology 148, 1908–1912.

Jackson, R.D., Idso, S.B., Reginato, R.J., Pinter, P.J., 1981. Canopy temperature as a cropwater-stress indicator. Journal of Water Resources Research 17, 1133–1138.

Jones, H.G., 1999a. Use of infrared thermometry for estimation of stomatal conduc-tance as a possible aid to irrigation scheduling. Journal of Agricultural and ForestMeteorology 95, 139–149.

Jones, H.G., 1999b. Use of thermography for quantitative studies of spatial and tem-poral variation of stomatal conductance over leaf surfaces. Journal of Plant Celland Environment 22 (9), 1043–1055.

Jones, H.G., 2004. Irrigation scheduling: advantages and pitfalls of plant-based meth-

ods. Journal of Experimental Botany, Water-Saving Agriculture Special Issue 55(407), 2427–2436.

Jones, H.G., Leinonen, I., 2003. Thermal imaging for the study of plant water relations.Journal of Agricultural Meteorology 59 (3), 205–217.

Jones, H.G., Stoll, M., Santos, T., de Sousa, C., Chaves, M.M., Grant, O.M., 2002.Use of infrared thermography for monitoring stomatal closure in the fiekd:

ctroni

M

O

X. Wang et al. / Computers and Ele

application to grapevine. Journal of Experimental Botany 53 (378), 2249–2260.

oller, M., Alchanatis, V., Cohen, Y., Meron, M., Tsipris, J., Naor, A., Ostrovsky, V.,Sprintsin, M., Cohen, S., 2007. Use of thermal and visible imagery for estimatingcrop water status of irrigated grapevine. Journal of Experimental Botany 58 (4),827–838.

livo, N., Girona, J., Marsal, J., 2009. Seasonal sensitivity of stem water potential tovapour pressure deficit in grapevine. Irrigation Science 27, 175–182.

cs in Agriculture 73 (2010) 74–83 83

Wheaton, A.D., Cooley, N., Dunn, G., Goodwin, I., Needs, S., 2007. Evaluation ofinfrared thermography to determine the crop water status of Cabernet Sauvi-

gnon grapevines. In: Poster Paper of 13th Australian Wine Industry TechnicalConference, Adelaide.

Yang, W., Wang, X., Wheaton, A., Cooley, N., Moran, B., 2009. Automatic opticaland IR image fusion for plant water stress analysis. In: Proceedings of the 12thInternational Conference on Information Fusion, Seattle, Washington, USA, pp.1053–1059.