An Experimental Study of the Relationship Between Online Engagement and Advertising Effectiveness...

An Experimental Study of the Relationship Between Online Engagement and Advertising Effectiveness

Bobby J. Calder1

Marketing Department Kellogg School of Management

Northwestern University

Edward C. Malthouse2*

Department of Integrated Marketing Communications Medill School of Journalism

Northwestern University

Ute Schaedel3

Department of Media Management Hamburg Media School University of Hamburg

1 Charles H. Kellstadt Distinguished Professor of Marketing and Psychology, Kellogg School of Management, Northwestern University, [email protected] 2 Theordore and Annie Sills Associate Professor of Integrated Marketing Communications, Medill School of Journalism, Northwestern University, [email protected] 3 Doctoral student, Hamburg Media School and University of Hamburg, [email protected] *Corresponding author

Engagement with Online Media and Advertising Effectiveness

ABSTRACT

This research defines engagement with a website, provides a systematic approach to

examining the types of engagement produced by specific experiences, and shows that this

overall engagement with the media context increases advertising effectiveness.

Measurement scales for eight different online experiences are shown to form two kinds of

overall engagement with online media – Personal and Social-Interactive Engagement.

Both of these types of engagement are shown to increase advertising effectiveness

relative to a context-free control group. Moreover, Social-Interactive Engagement, which

is more uniquely characteristic of the web as a medium, is shown to affect advertising

after controlling for Personal Engagement. Taken together these results offer online

companies and advertisers new metrics and advertising strategies.

Key Words: Engagement, Advertising Effectiveness, Context Effects, Online Media

Engagement with Online Media and Advertising Effectiveness 1

INTRODUCTION

Media provide a context for advertising that may affect consumer responses to advertising.

Many studies have investigated possible media context effects. The most general conclusion

is that when consumers are highly “engaged” with a media vehicle they can be more

responsive to advertising (e.g., Aaker & Brown, 1972; Feltham & Arnold, 1994; Coulter,

1998; Gallagher, Foster, & Parsons, 2001; DePelsmacker, Geuens, & Anckaert, 2002;

Nicovich, 2005; Bronner & Neijens, 2006; Cunningham, Hall, & Young, 2006; Wang, 2006).

While this conclusion is not surprising, media buyers do not consider consumer “engagement”

with a media vehicle in their decisions, except in secondary, ad-hoc ways. For example, the

price of print advertising is determined by circulation, the location of the ad within the

publication and characteristics of the ad such as the number of colors; the algorithms used to

place banner and sidebar ads do not consider consumer “engagement” with the hosting site.

There are many explanations for why consumer “engagement” with the surrounding media

context is not considered when making advertising decisions. One reason, as we will

demonstrate in the next section, is that no one, including both practitioners and academics,

agrees on what engagement is. Making matters worse, related terms such as “involvement”

and “experience” are also used in the academic and trade literatures without any consensus

over whether or how they are different from “engagement.” If we cannot define “engagement”

then we cannot measure it, and without systematic and comparable measurements, media

planners cannot incorporate it into their models and algorithms.

At the same time, advertisers are searching for ways to overcome the problems of ad

clutter and avoidance (Cho and Cheon 2004). Leveraging the media context is a potential


solution, since advertisers have (at least some) control over where their ads appear and we

know that context can affect reactions to ads.

The contribution of this paper is twofold. First, we define consumer engagement with a

website and its relationship to online experiences. As summarized below, other work has

explored distinct online experiences and related concepts. This article conceptualizes

engagement as a second-order construct that is manifested in various first-order “experience”

constructs. We theorize that our engagement construct is causally related to consumer

responses to online advertising. Second, we develop measures of engagement and test our

theory by evaluating whether these measures are associated with consumer evaluations of a

banner advertisement. We close with discussion on how understanding engagement can help

the online firms manage their sites and advertisers improve the effectiveness of their ads.

ENGAGEMENT, EXPERIENCES, AND ADVERTISING EFFECTIVENESS

What is Engagement?

Most people know what “engagement” with media feels like. Those who are “engaged”

with, for example, a television program or web site have a certain connection with it and

probably view or visit it often. But it is difficult to define the concept of engagement beyond

loose descriptions such as feeling a connection and using it often.

We begin with what engagement is not. Our conceptualization of engagement is different

from others who have characterized it in ways that we regard as consequences of engagement.

Marc (1966), for example, defines engagement as “how disappointed someone would be if a

magazine were no longer published.” Syndicated market research often asks whether a

publication is “one of my favorites,” whether a respondent would “recommend it to a friend”

or is “attentive.” Many equate engagement exclusively with behavioral usage. That is, they


define “engaged” people as those who visit the site often, spend substantial time on the site, or

have many page views. The Advertising Research Foundation (ARF) gives the definition

“media engagement is turning on a prospect to a brand idea enhanced by the surrounding

context” (ARF, 2006). Clearly “engagement” has many different meanings.

We argue that all of the meanings discussed above are consequences of engagement rather

than engagement itself. It is engagement with a web site that causes someone to want to visit

it, download its pages, be attentive to it, recommend it to a friend, or be disappointed if it were

no longer available. Likewise, researchers have known for years (see citations in introduction)

that the media context can “turn on” a prospect to some advertised brand, but again, this is a

consequence of engagement. Engagement is antecedent to outcomes such as usage, affect and

responses to advertising.

To think about what engagement really means, return to the basic notion of a sense of

being connected with something. We feel this intuition is essentially correct, but needs

elaboration to be useful. The fundamental insight is that engagement comes from

experiencing a website in a certain way. To understand engagement we need to understand

the different experiences that consumers have in connecting with the site (see Figure 1).

Consumer engagement with a website is a collection of experiences with the site.

Figure 1 about here

We define an experience as a consumer’s beliefs about how a site fits into his/her life. For

example, content can be engaging because users have a utilitarian experience with it. That is,

they believe that the site provides information to help them make important decisions and

accomplish something in their lives. Other content can be engaging because it provides users


with an intrinsically enjoyable experience, enabling them to unwind and escape from the

pressures of daily life.

To be engaging, different sites need not deliver the same experiences. Some sites could be

engaging because they provide high levels of a utilitarian experience while other sites could be

engaging because they are intrinsically enjoyable. Experiences are not necessarily mutually

exclusive and some content could engender high levels of multiple experiences. It is

necessary to realize that there is more than one path to engagement and that the different paths

are realized by offering different experiences. Consider, for example, the travel section of

www.nytimes.com. Some articles could engage readers by creating a utilitarian experience,

where the reader believes the articles give useful advice about what to do and where to stay at

certain destinations. Other articles could be engaging because they offer intrinsic enjoyment.

A narrative story about some travel adventure could relax readers and “transport” them to a

different place and not provide utilitarian “how-to” detail. Similarly, different consumers

could have different experiences with the same content.

In the language of measurement models, experiences are first-order constructs while

engagement is a second-order construct. We shall use the term experience whenever we refer

to a specific set of consumer beliefs about a vehicle such as utilitarian or intrinsic enjoyment,

and the term engagement whenever we refer to the overall experiences of a vehicle.

Online Experiences

It follows from the above discussion that we need to determine the first-order experiences

before we can measure this second-order construct of engagement. There are many

independent streams of research examining consumers’ experiences online and with media in

general. While there is substantial overlap between the experiences posited by the different

http://www.nytimes.com/


streams, unfortunately they are not entirely consistent. Certain experiences exist in some

frameworks but not others. Among the experiences that consistently exist in multiple

frameworks, there are often subtle differences in the way in which they are conceptualized. In

some cases, multiple experiences under one framework are subsumed by a single experience

of another.

Uses and gratifications (U&G) theory (sometimes called an “approach” rather than a

theory) provides a functionalist explanation of why people use media and has been an active

area of research within communications since the 1940s (e.g., see Ruggiero 2000 for a recent

survey). The U&G literature is vast; McQuail (1983, pp. 82-3) gives a concise

summarization that is often cited:

• “Information – finding out about relevant events and conditions in immediate

surroundings, society and the world; seeking advice on practical matters or opinion and

decision choices; satisfying curiosity and general interest; learning, self-education;

gaining a sense of security through knowledge.

• Personal Identity – finding reinforcement for personal values; finding models of

behaviour; identifying with valued others (in the media); gaining insight into one’s

self.

• Integration and Social Interaction – gaining insight into the circumstances of others:

social empathy; identifying with others and gaining a sense of belonging; finding a

basis for conversation and social interaction; having a substitute for real-life

companionship; helping to carry out social roles; enabling one to connect with family,

friends and society.


• Entertainment – escaping, or being diverted, from problems; relaxing; getting intrinsic

cultural or aesthetic enjoyment; filling time; emotional release; sexual arousal.”

The utilitarian experience discussed above is an example of information in the U&G

framework and the intrinsic enjoyment experience is an example of entertainment.

U&G approaches have been used in interactive marketing. For example, Nambisan and

Baron (2007) applied a variation of the U&G constructs to explain virtual customer

environments with four experiences: cognitive, social integrative, personal integrative and

hedonic. Bronner and Neijens (2006) measure eight experiences that are consistent with the

U&G approach: practical use, social, identification, pastime, transformation, stimulation,

information, and negative emotion. Childers, et al. (2001) discuss utilitarian and hedonic (a

type of “entertainment” in the U&G approach) experiences as explanations of online shopping

behavior. The same approach is also followed by Fiore et al. (2005) and Cotte, et al. (2006).

Flow is another construct that has received substantial attention (e.g., see Novak and Hoffman

2009) and is consistent with the U&G approach of understanding the consumer experience

with media.

Media engagement is particularly interesting in the case of websites. It is commonly

thought that online media are experienced differently than more traditional media such as

television and print. This difference is often described as “leaning forward” versus “leaning

backward.” The online experience is thought to be more active, participatory and interactive.

The internet is also thought to be more social in nature because it can be used for sharing and

communicating and it therefore breeds social engagement (Mathwick, 2002; Rappaport,

2007). Ruggiero (2000, p. 15) highlights the need to include “interactivity” in U&G

framework. Previous studies have tended to focus on this experience at a high-level or for


specific applications. For example Thorbjørnsen, et al. (2002) examined the overall amount of

experience people have with the web but deal only with the level of experience and not the

nature of that experience. Nambisan and Baron (2007) discuss an “interaction experience” in

virtual customer environments. Tremayne (2005) addresses the meaning of “interactivity” and

concludes that it can be viewed either as a process of message exchange or as a perceptual

variable. Others have studied interactivity in the form of word of mouth (e.g., Brown et al.

2007; Dwyer 2007; Sen and Lerman 2007). Prahalad and Ramaswamy (2004) and Sawhney

et al. (2005) discuss the co-creation experience.

It is unnecessary for purposes of this article to sort out differences in the ways that various

frameworks have conceptualized experiences because, for the purpose of measuring

engagement, all we need is a set of experiences that can serve as indicators of the engagement

construct domain. No set of indicators would be exhaustive of this domain but this is not

required from a measurement point of view.1 Our approach is to develop scales for a

representative set experiences that parallel those noted in the literature. We shall then factor

analyze the experience measures a test whether they could plausibly be manifestations of a

second-order engagement construct or constructs. The above discussion indicates that

websites may deliver different types of experiences than traditional media, as characterized by

the four McQuail (1983) U&G aspects.

Engagement and Advertising Effectiveness

1 The question arises of whether to treat experiences and engagement as formative or reflective. We follow Jarvis, Mackenzie and Podsakoff’s (2003) criteria for making the decision. We treat both as reflective (a Type I second-order factor specification in the language of Jarvis et al.). In the case of experiences, the items are manifestations of some experience, are interchangeable, and should covary. The items we have used represent a sample from the respective construct domains, e.g., there are many ways that a person can have a utilitarian experience and different items could represent the construct domain equally well. Thus, experiences are reflective according to the Jarvis et al framework. We also think of engagement as a reflective construct because we view experiences as manifestations of engagement (reflective) rather than as “defining characteristics” (formative).


The conceptual framework (Figure 1) posits that engagement and experiences are

antecedent to reactions to ads. We seek to test this relationship as an indicator of the

predictive validity of our measures. There has been relatively little previous research on the

impact of the online media context on advertising. Existing studies have approached this at

either a very high level or very specifically. To wit Bronner and Neijens (2006) compare the

experiences of different types of media with the experiences of advertising content. They

find, for instance, that the experience of usefulness with a site is related to the ads on that site

being experienced as useful. And Wang (2006) finds in the context of an online game that an

online ad inviting users to play a game was more effective than an ad that did not, suggesting

that the game-ad might have benefited from the game context. Previous work has also

focused specifically on the use of interactivity in online ads (e.g., Ariely, 2000; Chatterjee,

Hoffman, & Novak, 2003; Pavlou & Stewart, 2000). Hupfer and Grey (2005) test the effect of

the offer (e.g., whether there is a free sample) and user mode (e.g., goal-directed) on attitudes

toward the brand and ad.

There are several theoretical explanations for why engagement should affect reactions to

advertising including affect transfer (e.g., Broniarczyk and Alba 1991, p. 215) and

categorization theory (Cohen and Basu 1987). Dahlén (2005) does a literature review of

media context effects and summarizes three possible theoretical rationales for why context

should affect reactions to ads. The first is the mood congruency-accessibility hypothesis:

“The ad context makes a certain mood or affect more accessible and relieves the processing of

stimuli with similar moods or affects (p. 90).” The second is the congruity principle: “the

medium and the advertised brand converge and become more similar in consumers’ minds (p.

90).” The third is that the context serves as a cognitive prime that “activates a semantic


network of related material that guides attention and determines the interpretation of the ad (p.

90).” It should be noted that these explanations are not alternative explanations but rather all

of them are plausible mechanisms for how media context can affect advertising. They lead us

to formally hypothesize:

Hypothesis 1: Engagement with the surrounding online media vehicle context increases

advertising effectiveness.

METHODS AND RESULTS

Our methodology consisted of several steps, each of which will be discussed in this

section. The first step was to select scales to measure experiences that span the construct

domain and provide indicators of engagement. Next we executed a survey that employed the

scaling measures of experiences and a quasi-experimental design to evaluate advertising

effectiveness. The survey data allowed us to evaluate the psychometric properties of our

experience scales and engagement by estimating a confirmatory factor analysis measurement

model for the experience scales and then a second-order factor model for engagement. The

final step was to test the research hypothesis that engagement increases ad effectiveness.

Selecting Experience Scales

As indicated, to accomplish the objectives of this study, we needed measurement scales for

a set of online experiences that could be used as indicators of engagement. Ideally these

scales should produce an acceptable fit in a measurement model and have good psychometric

properties such as acceptable reliability and convergent and discriminant validity. We are

unaware of any previous studies of online experiences that measure such a broad range of

experiences with these high standards.


The present study uses the Calder-Malthouse (CM) set of media experiences (Calder and

Malthouse 2004, 2005; Malthouse, Calder and Tamhane 2007). We briefly summarize their

methodology and argue that these experiences span the engagement domain. CM conducted

over 400 hour-long, in-depth interviews with consumers about the role that specific web sites,

newspapers, magazines and TV news programs play in their lives. They analyzed the

transcripts for common themes and created hundreds of Likert-scale items. The items were

included on surveys of website visitors, newspaper and magazine readers, and TV news

viewers. Exploratory factor analysis identified 22 online experiences, 44 newspaper

experiences, 39 magazine experiences and 12 TV news experiences. The values of coefficient

alpha suggested that most of the scales were reliable (some had weak reliability because of too

few items). None of the CM studies estimate confirmatory factor analysis models. Some

experiences are common across media, while others are specific to a particular medium (e.g.,

media web sites). CM also showed that their experiences are associated with usage (site

usage, readership, viewership) and, in the case of magazines, reactions to advertising.

In this research we had to select eight experiences from the 22 CM online ones, due to

constraints on survey length and respondent fatigue. Requiring our experience measures to

have an acceptable fit in a measurement model also limits the number of experiences that we

can include.2 In reviewing the original 22 “experiences,” we decided that some did not fit in

the construct domain because they describe the site itself rather than how the site fits into the

consumer’s life. For example, one of the “experiences” was about credibility of the site and

another was about the site being easy to use. Several experiences were also dropped because

they were specifically about the advertising on the site.

2 Hatcher (1994, p. 260) recommends using “a maximum of 20-30 indicator variables” in a measurement model and we will use 37.


The eight experiences and their items are displayed in Table 1. They were selected with a

stratified sampling procedure from the remaining experiences so that there would be at least

one from each of the four McQuail U&G categories (the strata) that characterize more

traditional media, and others, such “community” and “participation & socializing,” that are

particularly relevant to online media. We tried to avoid picking too many experiences from

any single McQuail U&G category. For example, two of the remaining experiences fit under

McQuail’s information category: “makes me smarter,” which is about keeping people up-to-

date on issues that concern them, and “utilitarian,” which is more about advice and “how-to”

information. Using the flip of a coin we decided to include utilitarian. Likewise, the original

CM experiences “intrinsic enjoyment,” “entertains and absorbs me” and “a way to fill my

time” all fit under McQuail’s entertainment category and we selected the first (at random).

The “social facilitation” experience was selected as a representative of McQuail’s integration

and social interaction category. The “self-esteem and civic mindedness” experience

represents the personal identity category.

Table 1 about here

We claim that these eight experiences are representative of the engagement construct

domain. Of course, other sets could also represent the domain, but our approach is entirely

consistent with our objective of developing indicators of engagement. For example, we would

not expect our engagement measure to change in a substantive way if we had used “makes me

smarter” rather than “utilitarian” from the information category.

Survey Methodology


The second step was to sample users of media web sites. Eleven online media web sites

were used in the present confirmatory study.3 These sites represent a convenience sample, but

include broad range of different types of media sites including those with national reputations

(e.g., Reuters.com and Washingtonpost.com), special interest sites (e.g., about.com) and local

sites (e.g., king5.com). The target population, identified with a screening question, was

people who used the site at least once a month. Subjects were recruited from the visitors on

the particular sites, who were redirected to an online survey. The sample sizes for the 11 sites

ranged from n = 203 to n = 2,006, with a median sample size of n=1,141 and a total sample

size of n=11,541. Respondents were asked about their usage and experiences with the

particular site.

Measurement Models for Experiences and Engagement

We develop measures of online engagement using a two-step process. First, we estimate a

confirmatory factor analysis measurement model to study the psychometric properties of our

experience measures. Second, we develop second-order engagement factors by applying

exploratory factor analysis to the eight experiences and then fitting a second-order

confirmatory factor analysis model.

The first step in developing the online engagement measures is to estimate a measurement

model for the experiences, allowing each possible pair of experiences to be correlated. Fit

statistics are provided in Table 2. Question wording, factor loadings, and the values of

coefficient alpha are provided in Table 1. There were 37 items used to measure the 8

experiences. All eight scales are highly reliable, with coefficient alpha ranging from 0.87 to

0.91. In the measurement model, each of the 37 items had a parameter for the loading and

3 The sites are About.com, Washingtonpost.com, PalmBeachPost.com, Reuters.com, DallasNews.com, Projo.com, King5.com, AZFamily.com, WFAA.com, KHOU.com, and PE.com.


error variance (37+37 = 74), and there were parameters for the covariances between

every pair of experiences, giving a total of 102 parameters. GFI, CFI, and NNFI all exceed

0.90, indicating an acceptable fit.

2828

=⎟⎟⎠

⎞⎜⎜⎝

⎛

Put table 2 about here

Convergent validity was assessed with the t-values of the factor loadings, computed as the

ratio of the loading to the standard error of the item. Convergent validity is supported when t-

values reach an absolute value greater than 2. The minimum t-value was 48.2, providing

evidence in support of the convergent validity of the indicators. We assess discriminant

validity with the chi-square difference test. For each of the 28 pairs of experiences we

estimated a separate measurement model identical to the one shown in Table 2, except that the

covariance between the pair is fixed at 1. The chi-square statistics between the models were

computed, and range from 4,132 to 12,073. The differences have chi-square distributions with

1df, and are very highly significant, supporting discriminant validity.

Pearson correlations between the experiences are provided in Table 3. Note that the

correlations follow a pattern that suggests the possibility of second-order factors. The first six

experiences are moderately correlated with each other, with values between .42 and .72.

Participation and Socializing (7) is substantially less correlated with the first six, but

moderately correlated with the Community experience (8). Community is somewhat less

correlated with the first six experiences. This correlation structure suggests that there is a

higher-order factor structure generating the data.

Table 3 about here


Therefore the second step in developing the measurement model is to identify the second-

order engagement factors. To do this we did both an exploratory and confirmatory factor

analysis. We performed an exploratory factor analysis with a varimax rotation on the first-

order experiences and found two eigenvalues greater than 1. The rotated factor loadings are

provided in Table 4 and show two interpretable factors, hereafter called Personal Engagement

and Social-Interactive Engagement. The first six experiences from the correlation matrix have

the largest loadings on Personal Engagement, although Community also has a cross-loading

greater than .3. Participation and Socializing as well as Community have the largest loadings

on Social-Interactive Engagement, but several other experiences have sizable cross-loadings.

The Utilitarian experience likely cross-loads on Social-Interactive Engagement because much

of the advice and tips could be coming from the community of users rather than from content

created by employees of the site itself. Self-esteem likely cross-loads because contributing to

an online conversation could contribute to one’s self-esteem.

Table 4 about here

We then estimated a second-order confirmatory factor model, which is a more

parsimonious model for the 37×37 covariance matrix than the measurement model for

experiences. The objective was to test whether it is plausible that the Personal and Social-

Interactive Engagement latent variables generate the observed correlation structure between

the experiences and items. Personal and Social-Interactive Engagement will be used in the

subsequent analyses of advertising effectiveness. Instead of having 28 covariances between

the experiences, we assume that correlations between the experiences are due to two second-

order factors. This model can represent the correlations between the experiences with only 12

factor loadings shown in Table 5 above, and one additional term for the covariance between


the second-order factors. Fit statistics are also shown in Table 1 above, with CFI, GFI, and

NNFI all greater than .9 suggesting a good fit. Figure 2 shows the parameter estimates of the

second-order factor structure. The loadings for the 37 items were very similar to those from

the measurement model above and have been omitted. Note that the second-order factor

model finds a significant correlation between the two engagement latent variables. In the

analyses that follow, we estimate the two engagement factors using a weighted average of the

experiences, with the factor loadings as weights.

Figure 2 about here

Personal Engagement is manifested in experiences that are similar to those that people

have with newspapers and magazines. For example, experience items such as “This site

makes me think of things in new ways” or “This site often gives me something to talk about”

could also apply to a newspaper or magazine. Social-Interactive Engagement, however, is

more specific to web sites. Items such as “I do quite a bit of socializing on this site” and “I

contribute to the conversation on this site” would not characterize a newspaper or magazine,

and we did not hear such statements in our qualitative interviews for these media. While

Social-Interactive Engagement is more closely associated with the web, aspects of it can be

found for other media. For example, “A big reason I like this site is what I get from other

users” could also apply to the letters-to-the-editor page of a daily newspaper. The Utilitarian

experience is a manifestation of both forms of engagement. Service oriented websites (e.g.,

bhg.com – Better Homes and Gardens) will have a prominent utilitarian component as will

user-contributed advice sites (e.g., Yahoo!Answers or chowhound.com).

In sum, the measurement model and values of coefficient alpha have shown that the eight

experiences have been measured reliably and support the convergent and discriminant validity


of the scales. The second-order analysis shows two engagement factors, Personal Engagement

and Social-Interactive Engagement. Personal Engagement is manifested in experiences that

have counterparts in magazines and newspapers while Social-Interactive Engagement is more

specific to web sites. As reflected the loadings in Figure 2, with Personal Engagement users

seek stimulation and inspiration from the site, they want to use the site to facilitate their

interactions with other people, they feel the site affirms their self-worth, they get a sense of

intrinsic enjoyment in using the site itself, they feel it is useful for achieving goals, and they

value input from other users. With Social-Interactive Engagement, users experience some of

the same things in terms of intrinsic enjoyment, utilitarian worth, and valuing the input from

the larger community of users but in a way that links to a sense of participating with others

and socializing on the site. Thus Social-Interactive Engagement is motivated both intrinsically

and extrinsically but in this case it is the social relevance of these, rather than their personal or

individual quality, that is associated with the larger engagement experience. And it is the

valuing of input from the community and sense of participating with others and socializing

that gives Social-Interactive Engagement its dominant character.

The Relationship Between Engagement and Advertising Effectiveness

We now test the hypothesis (H1) that engagement predicts ad effectiveness. Users of the

11 media web sites were intercepted during their visit to the site and asked to complete a

survey. Participants answered questions about their use of, and experiences with, this web

site. They were then shown an ad for orbitz.com (an online travel agency) and asked to rate it

using standard copy-testing measures and their intention to click on the ad. A travel agency

was used because travel is potentially relevant to most internet users and this category often


advertises with banner ads. We shall relate engagement and experiences with the media to

these ad ratings as a test of predictive validity.

Note that participants were intercepted while actually visiting the site, though they did not

actually see the ad on the site. This manipulation of media context is not the same as

encountering the ad while actually on the site but actually provides a strong test of the

hypothesis. If the site experiences affect reactions to the ad in this test, the effect would be

expected to be, if anything, smaller than in the case of actually seeing the ad on the site.

One threat to validity is that the mere measurement of the experiences of a given site might

itself affect reactions to the ad. Whereas this would imply that all experiences would affect

the ad equally, it is at least possible that some of the experiences could be differentially

sensitive to measurement (measurement × scale interaction). In this way, merely thinking

about how a site gives advice and tips could have produced a higher rating of the ad. Another

threat is that any effect on advertising is not due to experiences with a particular site context

but to experiences with sites in general (which are correlated with the particular site

participants are told the ad is on). Alternatively, the different experiences individuals had with

their site and the responses to the ads in general could be construed as an individual difference

not dependent per se on using any particular sites. To assess these threats we used a context-

free control group design. The most important thing about the control group is that the ad was

identified only as a banner and not linked to any particular site.

Of the 11,536 intercepted on the 11 sites, 1,502 were randomly assigned to the context-

free control group, which was asked about their experiences with reading news sites in general

and told only that the ad was a banner. If any effects of the experiences on the ad are due to

simply rating the experiences and/or thinking about sites in general while taking a survey, then


the control group should respond in a similar way to those asked about a specific site. The

treatment group being different from the control group indicates that the results do not reflect

mere measurement or experiences with sites in general but rather measure the effects of

experiences with specific sites.

We have two measures of “reactions to an ad.” First, we developed a multi-item scale to

measure attitude toward the ad. Respondents were asked “How well does each of the

following words describe the ad in the [site name]?” The study included the items “interesting,

lively, helpful, believable, attractive, imaginative, and soothing” (7-point scale from “Does not

describe the ad at all” to “Describes the ad very well”). These items were selected to be

typical of those that are commonly used to test reactions to advertising stimuli (see Bearden &

Netemeyer, 1999, Chapter 5) and to fit the ad tested here. The value of coefficient alpha was

.93, indicating a reliable scale. As a second, complementary measure of reactions to the ad,

respondents were asked: “How likely are you to click on this ad?’’

Correlations between the experience, the engagement factors, and the advertising variables

are also provided in Table 3 above. All correlations are positive and highly significant,

indicating that higher experience and engagement levels are associated with more ad

effectiveness, supporting H1.

Correlations, however, do not account for the different sites, control for confounding

factors such as use of on-line travel sites, or rule out measurement effects. We now conduct a

more stringent test of the relationship between the ad ratings and experiences/engagement by

comparing the slopes of the treatment group (those who were told that the ad appeared on a

specific site) and the context-free control group (those told the ad was not linked to a site)


using an ANCOVA model. The model includes a different, fixed-effect4 intercepts for each

site (αj for site j=1, …, 11), a dummy x1=1 indicating the respondent was in the control group,

a measure for the use of on-line travel agents in general x2, the engagement rating x3 (as a

continuous variable on a 5-point scale), and an interaction term between experience rating and

the control group dummy (x1x3):

y = αj + β1 x1 + β2x2 + β3x3 + γ x1x3.

The parameter β3 is the slope for engagement in the treatment group, γ indicates how much

larger or smaller the engagement slope is in the control group compared with the treatment

group, and β3+γ gives the slope for the control group. We can test whether the slopes in the

treatment and control groups are different with H0: γ = 0.

The model is estimated separately for each of the 8 experience and 2 engagement

measures, with the results summarized in Table 5 below. Parameter estimates for the intercept

terms α1, …, α11, β1, and the slope for product usage β2 are omitted in Table 6 for clarity.5

All of the treatment-group experience slopes β3 are positive and highly significant, consistent

with the conclusions from the correlation matrix above supporting H1.6

Testing whether these results are due to measurement effects, the γ-values of both

dependent variables and for both Personal and Social-Interactive Engagement are highly

significant, indicating that engagement has a stronger effect on ad ratings when the respondent

4 Fixed effects are used rather than random since we have a convenience sample of sites. 5 In all models, the product usage variables have very highly significant positive effects. Likewise, across models the extra sums of squares are large and highly significant for the site intercepts, allowing us to reject the null hypothesis that all 11 sites have the same intercept. The control group dummy shifting the intercept (β2) is occasionally significant, but the signs change across models suggesting that the significant results could be type I errors. 6 The experience slopes for the control group (β3+γ) are also significantly different from 0, which could be due to any of the threats to internal validity mentioned above or to the method of recruiting subjects used in this study (members of the control group were also intercepted from the sites under study and some may not have completely understood that they were to answer questions about sites in general rather than the one from which they were recruited).


associated an ad with a particular site, and supporting H1 from above. The γ-values for most,

but not all, of the individual experiences are also significant. As we indicated above, the

manipulation of media context is relatively weak, and the effect sizes γ might well be larger if

respondents were actually experiencing the particular site when they were exposed to the ad.

Table 5 about here

Having established that both types of engagement are associated with advertising

effectiveness, we now examine whether Social-Interactive Engagement affects reactions to ads

after controlling for Personal Engagement by including both in the model, as well as use of

online travel agents in general (x2) and separate, fixed-effect intercepts for individual sites.7

We shall use only the treatment group in this analysis. The results are summarized in Table 6.

The coefficient for both types of engagement are highly significant and roughly of comparable

size, indicating that both forms of engagement are important in predicting advertising

effectiveness.

Table 6 about here

CONCLUSION It is commonly believed that the web is different from other media in terms of leaning

forward instead of backward, being more interactive, more social, and so forth. This research

attempted to understand and measure the actual experience of websites. Specifically, we 7 It could also be tempting to include all 8 experiences in a single regression model, but such a model is theoretically questionable because there is no “correct” model (i.e., set of experiences included as predictor variables, which are sampled from the construct domain) and all inference on slope coefficients will be suspect. If experiences are manifestations of high-order engagement constructs the experience measures will be correlated (creating multicollinearity), and must share in explaining the dependent variables. Thus the experience effect sizes will depend on the size of the sample from the construct domain. For example, with a sample of 4 experiences, each experience will have the opportunity to explain more of the dependent variable than with a sample of 8. Multicollinearity suggests that the magnitude of the slopes will also be highly sensitive to the particular sample of experiences drawn.


identified and measured eight experiences with online news web sites and showed that the

measures are reliable with discriminant, convergent and predictive validity. By factoring the

eight experiences, we found that these experiences, as anticipated, are manifestations of two

different kinds of overall engagement, i.e., two second-order factors exist. One factor,

Personal Engagement, is manifested in experiences that are very similar to those that people

have with newspapers and magazines. For example, people have social experiences with both

print and online content by bringing up an article they read; just as reading a newspaper at the

breakfast table can be habitual, so can reading a web site. The other second-order factor,

Social-Interactive Engagement, is weighted more to experiences that are more unique to the

web, such as participating in discussions and socializing with others through a site. It is these

experiences that give Social-Interactive Engagement its dominant social character. This

finding gives empirical support and specificity to the idea that the Internet is a different kind

of medium.

This work set the stage for examining the effect of online media engagement on

advertising. We related experiences and engagement to the ratings of a banner ad using a

quasi-experimental design. The results show that both Personal and Social-Interactive

Engagement affect reactions to the banner ad. So, in additional to the Personal Engagement

context effects that have been demonstrated previously for traditional media, the interactive

component of a user’s experience with a web site is also shown to affect advertising. A

context-free control group was used to strengthen internal validity by ruling out measurement

effects. The results of a regression model including both types of engagement indicate that

Social-Interactive engagement affects reactions to ads after controlling for Personal


engagement. We thus conclude that online media does involve a distinct form of engagement

and that this engagement has its own impact on advertising effectiveness.

Our conclusions are of course subject at this point to the limitations of the methodology of

this study. Three points should be kept in mind. First, no matter how “representative” the ad

used in this study might be, further research is called for to examine different product

categories and types of advertising execution. It is possible to formulate many hypotheses in

this regard. For example, ads that are more interactive may have even stronger relationships

(i.e., greater slopes) with Social-Interactive Engagement. Second, it would also seem

desirable to conduct future research with actual insertion of ads on web sites rather than only

intercepting users on the sites. This might have some value in being a more “realistic”

methodology with potentially better external validity. We note, however, that at best

achieving external validity through matching a research setting with some “real” context is

always fraught with difficulty (Calder, Phillips, & Tybout, 1983; Sternthal, Tybout, & Calder,

1987). It is never possible to duplicate the exact context, or even to know what key variable

might be missing. In our view additional work with ads varying along theoretically motivated

dimensions would be valuable. Third, we have tested the relationship between engagement ad

effectiveness for 11 websites. It is desirable to test this relationship with more sites.

Taking into consideration the limitations of this study, we believe that the effects of online

media experiences on advertising are potentially pervasive and in great need of further

investigation. While previous research has suggested the importance of online experiences

and the possibility of context effects on advertising, the present study provides a systematic

approach to examining the types of engagement produced by specific experiences with online

sites and shows that it is engagement that produces the context effect on online advertising.


Further, the distinctive Social-Interactive Engagement associated with the web not only

increases advertising effectiveness but does so independently of the type of engagement

usually associated with more traditional media. This implies that interactive marketers may

find online media to have added potential as a marketing tool.

MANAGERIAL IMPLICATIONS This research has many applications to both the managers at online companies that host

ads and to those making advertising decisions. First, managing a website involves engineering

a set of experiences for the visitors, and then measuring the extent to which the visitors have

the intended experiences. The scales presented in this paper enable a website to track both

experiences and higher-level engagement. Such measurements could provide an early

warning that the intended experiences are not being created. Likewise, advertisers and online

companies that produce websites are searching for media-neural metrics for the purpose of

common-currency comparisons, e.g., a website with a print vehicle (Winer, 2009).

Engagement and experience metrics could serve this purpose.

Second, managers of advertising vehicles are attempting to use engagement as a way to

differentiate themselves from competitors and retain advertisers. Their basic argument is as

follows: highly engaged readers are more likely to be exposed to ads; ads carried by vehicles

with more engaged readers will therefore be more effective; and a vehicle with highly engaged

readers should command a premium price for advertising space, or at least have an advantage

in retaining advertisers. Our research supports this argument.

The development of experience and engagement metrics of the kind investigated here is

important from both a media and advertiser perspective.


REFERNCES

Aaker, D. A., & Brown, P. K. (1972). Evaluating Vehicle Source Effects. Journal of Advertising Research, 12(4), 11.

ARF. (2006). Engagement. Retrieved 23 July, 2007, www.thearf.org/research/engagement.html

Ariely, D. (2000). Controlling the Information Flow: Effects on Consumers' Decision Making and Preferences. Journal of Consumer Research, 27(2), 233.

Bearden, W. O., & Netemeyer, R. G. (1999). Handbook of Marketing Scales: Multi-Item Measures for Marketing and Consumer Behaviour Research. Thousand Oaks, CA: Sage Publications.

Broniarczyk, Susan and Joseph Alba (1991), The Importance of the Brand in Brand Extension, Journal of Marketing Research, XXXI, 214-228.

Bronner, F., & Neijens, P. (2006). Audience Experiences of Media Context and Embedded Advertising: A Comparison of Eight Media. International Journal of Market Research, 48(1), 81.

Brown, Jo, Amanda Broderick and Nick Lee (2007), Word of Mouth Communication Within Online Communities: Conceptualizing the Online Social Network, Journal of Interactive Marketing, 21(3), 2-20.

Calder and Malthouse (2005), Experiential Engagement with Online Content Web Sites and the Impact of Cross-Media Usage, Proceedings of 12th Worldwide Readership Research Symposium, Prague, October.

Calder, B. J., & Malthouse, E. C. (2004). Qualitative Media Measures: Newspaper Experiences. The International Journal on Media Management, 6(1/2), 123.

Calder, B. J., Phillips, L., & Tybout, A. (1983). Beyond External Validity. Journal of Consumer Research, 10(1), 112.

Chatterjee, P., Hoffman, D. L., & Novak, T. P. (2003). Modeling the Clickstream: Implications for Web-Based Advertising Efforts. Marketing Science, 22(4), 520.

Childers, Terry, Christopher Carr, Joann Peck and Stephen Carson (2001), Hedonic and Utilitarian Motivations for Online Retailing Shopping Behavior, Journal of Retailing, 77, 511-535.

Cho, Chang-Hoan and Hongsik John Cheon (2004), Why do People Avoid Advertising on the Internet? Journal of Advertising, 33(4), 89-97.

Cohen, Joel and Kunal Basu (1987), Alternative Models of Categorization: Toward a Contingent Processing Framework, Journal of Consumer Research, 13, 455-472.

Cotte, June, Tilottama Chowdhury, S. Ratneshwar and Lisa Ricci (2006), Pleasure or Utility? Time Planning Style and Web Usage Behaviors, Journal of Interactive Marketing, 20(1), 45-57.

Coulter, K. S. (1998). The Effects of Affective Responses to Media Context on Advertising Evaluations. Journal of Advertising, 27(4), 41.

Cunningham, T., Hall, A. S., & Young, C. (2006). The Advertising Magnifier Effect: An Mtv Study. Journal of Advertising Research, 4, 46.

Dahlén, M. (2005). The Medium as a Contextual Cue: Effects of Creative Media Choice. Journal of Advertising, 34(3), 89.


DePelsmacker, P., Geuens, M., & Anckaert, P. (2002). Media Context and Advertising Effectiveness: The Role of Context Appreciation and Context/Ad Similarity. Journal of Advertising, 31(2), 49.

Dwyer, Paul (2007), Measuring the Value of Electronic Word of Mouth and its Impact in Consumer Communities, Journal of Interactive Marketing, 21(2), 63-79.

Feltham, T. S., & Arnold, S. J. (1994). Program Involvement and Ad/Program Consistency as Moderators of Program Context Effects. Journal of Consumer Psychology, 3(1), 51.

Fiore, Ann Marie, Jihyun Kim and Hyun-Hwa Lee (2005), Effect of Image Interactivity Technology on Consumer Responses Toward the Online Retailer, Journal of Interactive Marketing, 19(3), 38-53.

Gallagher, K., Foster, K. D., & Parsons, J. (2001). The Medium Is Not the Message: Advertising Effectiveness and Content Evaluation in Print and on the Web. Journal of Advertising Research, 41(4), 57.

Hatcher, Larry (1994), A Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling, Cary, NC: SAS Institute.

Hupfer, Maureen and Alex Grey (2005), Getting Something for Nothing: The Impact of a Sample Offer and User Mode on Banner Ad Response, Journal of Interactive Advertising, 6(1), http://jiad.org/article74.

Jarvis, Cheryl Burke, Scott Mackenzie and Philip Podsakoff (2003), A Critical Review of Construct Indicators and Measurement Model Misspecification in Marketing and Consumer Research, Journal of Consumer Research, 30, 199-218.

Malthouse, E. C., Calder, B. J., & Tamhane, A. C. (2007). The Effects of Media Context Experience on Advertising Effectiveness. Journal of Advertising, 36(6), 7.

Marc, Marcel (1966), “Using Readership Quality in Magazine Selection,” Journal of Advertising Research, 6(4), 9-13.

Mathwick, C. (2002). Understanding the Online Consumer: A Typology of Online Relational Norms and Behavior. Journal of Interactive Marketing, 16(1), 40.

McQuail, Denis (1983), Mass Communication Theory, an Introduction, Sage Publications, London.

Nambisan, Ratish and Robert Baron (2007), Interactions in Virtual Customer Environments: Implications for Product Support and Customer Relationship Management, Journal of Interactive Marketing, 21(2), 42-62.

Nicovich, S. G. (2005). The Effect of Involvement on Ad Judgment in a Video Game Environment: The Mediating Role of Presence. Journal of Interactive Advertising.

Novak, Thomas and Donna Hoffman (2009), Flow Online: Lessons Learned and Future Prospects, Journal of Interactive Marketing, 23(1).

Pavlou, P. A., & Steward, D. W. (2000). Measuring the Effects and Effectiveness of Interactive Advertising: A Research Agenda. Journal of Interactive Advertising, 1(1), 5.

Prahalad, C.K. and Venkat Ramaswamy (2004), Co-Creation Experiences: The Next Practice in Value Creation, Journal of Interactive Marketing, 18(3) 5-14.

Rappaport, S. D. (2007). Lessons from Online Practice: New Advertising Models. Journal of Advertising Research, 47(2), 135.

Ruggiero, Thomas (2000), Uses and Gratifications Theory in the 21st Century, Mass Communication & Society, 3(1), 3-37.

http://jiad.org/article74


Sawhney, Mohanbir, Gianmario Verona and Emanuela Prandelli (2005), Collaborating to Create: the Internet as a Platform for Customer Engagement in Product Innovation, Journal of Interactive Marketing, 19(4), 4-17.

Sen, Shahana and Dawn Lerman (2007), Why are you telling me this? An Examination into Negative Consumer Reviews on the Web, Journal of Interactive Marketing, 21(4), 76-94.

Sternthal, B., Tybout, A., & Calder, B. J. (1987). Confirmatory Versus Comparative Approaches to Theory Testing. Journal of Consumer Research, 14(114).

Thorbjørnsen, H., Supphellen, M., Nysveen, H., & Pederson, P. E. (2002). Building Brand Relationships Online: A Comparison of Two Interactive Applications. Journal of Interactive Marketing, 16(3), 17.

Tremayne, Mark (2005), Lessons Learned from Experiments with Interactivity on the Web, Journal of Interactive Advertising, 5(2), http://www.jiad.org/article62.

Wang, A. (2006). Advertising Engagement: A Driver of Message Involvement on Message Effects. Journal of Advertising Research, 46(4), 355.

Winer, Russell S. and Venkatesh Shankar (2003), “A New Vision for the Journal,” Journal of Interactive Marketing, 17(1), 2-4.


Table 1: Question Wording and Parameter Estimtaes from Confirmatory Factor Analysis Measurement Model

Experience

Item

Stand. Loading

Stimulation & Inspiration (α=0.88)

It inspires me in my own life. 0.85 This site makes me think of things in new ways. 0.84 This site stimulates my thinking about lots of different topics. 0.78 This site makes me a more interesting person. 0.79 Some stories on this site touch me deep down. 0.71

Social Facilitation (α=0.88)

I bring up things I have seen on this site in conversations with many other people. 0.85

This site often gives me something to talk about. 0.85 I use things from this site in discussions or arguments with people I know. 0.81

Temporal (α=0.90)

It's part of my routine. 0.85 This is one of the sites I always go to anytime I am surfing the web. 0.83 I use it as a big part of getting my news for the day. 0.84 It helps me to get my day started in the morning. 0.80

Self-Esteem & Civic-Mindedness (α=0.91)

Using this site makes me feel like a better citizen. 0.86 Using this site makes a difference in my life. 0.85 This site reflects my values. 0.76 It makes me more a part of my community. 0.75 I am a better person for using this site. 0.88

Intrinsic Enjoyment (α=0.87)

It's a treat for me. 0.83 Going to this site improves my mood, makes me happier. 0.85 I like to kick back and wind down with it. 0.82 I like to go to this site when I am eating or taking a break. 0.65 While I am on this site, I don't think about other sites I might go to. 0.71

Utilitarian (α=0.88)

This site helps me make good purchase decisions. 0.81 You learn how to improve yourself from this site. 0.83 This site provides information that helps me make important decisions. 0.76 This site helps me better manage my money. 0.81 I give advice and tips to people I know based on things I've read on this site. 0.74

Participation & Socializing (α=0.88)

I do quite a bit of socializing on this site. 0.86 I contribute to the conversation on this site. 0.77 I often feel guilty about the amount of time I spend on this site socializing. 0.82 I should probably cut back on the amount of time I spend on this site socializing. 0.78

Community (α=0.88)

I'm as interested in input from other users as I am in the regular content on this site. 0.84

A big reason I like this site is what I get from other users. 0.85 This site does a good job of getting its visitors to contribute or provide feedback. 0.59

I'd like to meet other people who regularly visit this site. 0.80 I've gotten interested in things I otherwise wouldn't have because of others on this site. 0.73

Overall, the visitors to this site are pretty knowledgeable about the topics it covers so you can learn from them. 0.66


Table 2: Summary of Confirmatory Factor Analysis Model

Measurement Model

Second-Order CFA Model

Parameters 102 87 GFI .9155 .9029 CFI .9482 .9392 NNFI .9426 .9343 RMSEA .0472 .0505 Note. n=5942 with 37 items.

Table 3: Correlation Matrix (Treatment Group Only) Pearson Correlation

Experience 1 2 3 4 5 6 7 8 9 10

1 Stimulation & Inspiration 2 Social Facilitation .56 3 Temporal .51 .55 4 Self-Esteem & Civic-Mindedness .65 .57 .47 5 Intrinsic Enjoyment .65 .52 .62 .63 6 Utilitarian .62 .52 .42 .72 .58 7 Participation & Socializing .24 .19 .19 .29 .33 .35 8 Community .51 .41 .32 .53 .53 .59 .56

Engagement

9 Personal Engagement .79 .75 .78 .82 .81 .71 .32 .51 10 Interactive Engagement .52 .43 .43 .69 .61 .67 .77 .77 .74 Advertising

11 Click Intention .24 .19 .15 .25 .23 .27 .12 .23 .27 .26 12 Attitude Towards Ad .30 .23 .19 .31 .29 .31 .14 .27 .34 .32 Note. All correlations are significantly different from 0 at the .0001 level.


Table 4: Exploratory Factor Analysis Loadings of First-Order Experiences.

Experience

Factor 1 Personal

Engagement

Factor 2 Social-

Interactive

Social Facilitation .768 Temporal .753 Stimulation & Inspiration .744 Self-Esteem & Civic Mindedness .710 .375 Intrinsic Enjoyment .701 .366 Utilitarian .612 .472 Participation & Socializing .881 Community .361 .755

Note. Loadings less than .3 were omitted. Table 5: Estimates from Separate Models Including Context-Free Control Group Experience Attitude Toward Ad Intention to Click

β3 γ β3 γ

Stimulation & Inspiration 0.63 −0.16 0.67 −0.19 Social Facilitation 0.42 −0.11 0.44 −0.12 Temporal 0.32 −0.09 0.32 −0.08 Self-Esteem & Civic-Mindedness 0.61 −0.27 0.62 −0.27 Intrinsic Enjoyment 0.56 −0.06 0.60 −0.04 Utilitarian 0.62 −0.15 0.68 −0.16 Participation & Socializing 0.60 −0.12 0.67 −0.15 Community 0.29 −0.18 0.35 −0.21 Personal Engagement 0.81 −0.20 0.85 −0.20 Social-Interactive Engagement 0.90 −0.23 0.99 −0.27 Note. p < 0.05 marked in italic. p < 0.01 marked in bold. Table 6: Estimates for the Model with Both Types of Engagement as Predictors Dependent Variable Personal

Engagement Social-Interactive Engagement

Online travel agency use

Attitude Towards Ad 0.536 (0.033) 0.443 (0.039) 0.106 (0.006) Intention to Click 0.479 (0.043) 0.590 (0.050) 0.244 (0.008) Note. p < 0.05 marked in italic. p < 0.01 marked in bold.


Figure 1: Engagement and its consequences

Intrinsic Enjoyment

Temporal

Self-Esteem & Civic-Mindedness

Social-Interactive Engagement

Utilitarian

Social Facilitation

Stimulation & Inspiration

.85

.73

.68

.79

.78

.69

.14

.25

.76

.74

.29

Personal Engagement

.42

Participation & Socializing

Community

Figure 2: Second-Order Factor Structure

An Experimental Study of the Relationship Between Online Engagement and Advertising Effectiveness...

Documents

Transcript of An Experimental Study of the Relationship Between Online Engagement and Advertising Effectiveness...