Learned Helplessness and Learned Prevalence: Exploring the causal relations of perceived...

Running head: THE EFFECT OF REWARD PREVALENCE ON LEARNED HELPLESSNESS

Notice: this draft is not the final version of the paper. Amongother changes, the final version of the paper includes moredetailed methods and supplementary materials. Please contact

Kinneret at [email protected] for further details.

Learned Helplessness and Learned Prevalence:

Exploring the causal relations of perceived controllability,reward prevalence and exploration

Kinneret Teodorescu* and Ido Erev

Max Wertheimer Minerva Center for Cognitive Studies

The Technion- Israel Institute of Technology

*corresponding author, contact information:

William Davidson Faculty of Industrial Engineering and Management Technion - Israel Institute of TechnologyTechnion City, Haifa 32000, IsraelPhone: +972-(0)4-829-2448Email: [email protected]

mailto:[email protected]

THE EFFECT OF REWARD PREVALENCE ON LEARNED HELPLESSNESS

Keywords: Learned helplessness, decisions from experience,

exploration, feeling of control, learning, sampling, predictions.

Abstract:

Exposure to uncontrollable outcomes was found to trigger

learned helplessness: a state in which the agent fails to take

advantage of regained control due to lack of exploration. While

the importance of this phenomenon has been widely studied, its

underlying cause remains undetermined. One can learn not to

explore because the environment is uncontrollable, because the

average reinforcement is low or because rewards are rare. The

current paper presents a simple experimental paradigm that

contrasts the predictions of the three contributors and offers a

unified psychological mechanism that underlies the empirical

phenomena. Our results demonstrate that learned helplessness is

not correlated with either perceived controllability or the

average reward, suggesting that reward prevalence is a better

predictor of exploration behavior. A simple computational model

where exploration decisions are based on small samples of past


experiences captures the empirical phenomena while also providing

a cognitive basis for feelings of uncontrollability.


Learned Helplessness and Learned Prevalence:

Exploring the causal relations of perceived controllability,reward prevalence and exploration

Seligman and Maier (1967) established that exposure to

uncontrollable events impairs future exploration and learning.

This "learned helplessness" (LH) phenomenon was originally

documented in the study of dogs escape from electric shocks (see

left-hand column in Figure 1), and was later replicated in

studies of human problem solving (e.g., second column in Figure

1). Subsequent research shows that the understanding of LH can

shed light on many important behavioral and social problems

including depression (e.g. Alloy et al., 1999, Clack & Back,

1999), failure in school (Diener & Dweck, 1978), burnout (Maslach

& Jackson, 1982), staying with abusing spouse (Walker, 1977),

maladaptive behavior in organizations (Martinko & Gardner, 1982),

poverty (Evans, Gonnella, Marcynyszyn, Gentile & Salpekar, ,2005)

and many others (for a review see Evans & Stecker, 2004).

The common explanation to the empirical LH effect focuses on

perceived uncontrollability. According to this explanation, when


experiencing absence of control the organism perceives

independence between actions and outcomes, becomes helpless and

ceases exploration of the environment. This behavior is later

overgeneralized to other situations in which exploration is

beneficial.

An alternative account for the empirical LH effect involves

reward-based considerations: Since absence of control

manipulations reduces the average reward from exploration, it

might be that agents give up too early simply because they

learned that exploration is, on average, not rewarding enough and

regardless of controllability (Cohen, Rothbart & Phillips, 1976;

Alloy & Abramson, 1979; Jardine & Winefield, 1981).

However, recent studies suggest that people (and other

animals) are more sensitive to the frequent rather than the

average reinforcement. Demonstrated by Teodorescu & Erev (2013),

people explore insufficiently when the prevalence of desirable

outcomes from exploration is low, even if the expected benefit

from exploration is positive (i.e. the average reinforcement

favors exploration). Thus, the critical factor in LH experiments


might be the frequent outcome resulting from exploration, rather

than either the average outcome or feeling of uncontrollability.

To clarify the differences between the above three

explanations, consider a bachelor who experienced unsuccessful

dates and decided to quit actively searching for true love.

According to the perceived uncontrollability explanation, the

bachelor reduced his exploration efforts because he perceived

independence between his efforts to find the one and the quality

of his dates. In contrast, it is possible that the bachelor does

in fact perceive contingency between his actions and outcomes

(e.g. he noticed that when he invest more efforts the rejection

is more tough), but has learned that exploration is too costly.

According to the average reinforcement explanation, insufficient

exploration will then be evident if the environment has changed

so that exploration became beneficial, but the bachelor who

ceased engaging in exploration will be oblivious to this change.

Last, according to the reward frequency explanation, frequent

disappointments can lead the bachelor to reduce his exploration

efforts even if it is reinforcing on average (due to extremely

rewarding but rare dates), if he feels in control or if the


environment was not changed. Importantly, the three accounts

also posit different strategies for effective intervention. The

bachelor could be driven to resume exploration by making evident

the direct effects of his actions, by ensuring he will experience

an extraordinary date (e.g. paying someone for doing so) which

will change his average experience with exploration or by

instructing him how to reduce the frequent cost of searching

(e.g. using the internet).

To determine what led the disappointed bachelor to give up,

the influence of each of the above factors must be separated. The

problem is that in previous studies of LH (left hand columns in

Figure 1), the three explanations (perceived uncontrollability,

average reinforcement and frequent reinforcement) are confounded:

Absence of control might simultaneously reduce the average and

frequent rewards from exploration, and perception of control.

Thus, all three explanations predict insufficient exploration

following an absence of control manipulation. In addition, many

LH experiments did not measure perception of control, therefore

avoiding an explicit examination of the common perceived

uncontrollability explanation (Costello, 1978). Moreover,


previous LH experiments measured only the outcome of exploration

(see left columns of Figure 1) and a direct measure of

exploration was unavailable.

The purpose of the remainder of this paper is twofold. First

we develop an experimental exploration paradigm that can

reproduce the classic LH effect while also allowing for

independent manipulations of the average and frequent payoffs and

simultaneous measurement of perceived controllability as

manifested in a novel indirect measure based on a prediction

task. We will show that our results strongly support the notion

that exploratory behavior is sensitive to reward prevalence above

and beyond both perceived controllability and the average

outcome. Last we will demonstrate that a simple sampling

mechanism can account for exploratory and perceived

controllability findings, providing a causal basis for both

phenomena.

Method


The Main Task. Participants were asked to select one

unmarked key out of 120 in each of 100 trials. After selecting a

key, the trial’s payoff was presented on the selected key. The

first time a key was selected, its color changed from light to

dark gray and remained so until the end of the task (see Fig.1).

This information was provided before the beginning of the task,

but no further instructions were given. The exact payoff after

each trial was the sum of two components: Cost and reward. The

cost was a loss of 1 point if the agent explored a new (bright)

key, and 0 if the agent selected an old (dark) key. The reward

was 11 with probability p if the agent explored a new key, and 0

otherwise.

The classic dogs experiment

A typical LH experiment inhumans

The current experiment

General Dogs select between Participants select Participants select


task possible movements in an effort to escape electric shocks

between possible strategies/solutions in aneffort to solve a problem (e.g. an anagram)

between unmarked keys in an effort to maximize monetary payment

The cost ofexploration

The dog’s physical effort to change movements (subjective and unmeasurable)

The participant’s mental effort to try new strategies or think on a different solution (subjective and unmeasurable)

A small monetary loss accompanies each selection of a new key (objective and measurable)

“With Control” setting

Shock termination depends on the dog’s movements (a specific movement terminates theshock)

The problem is solvable The monetary payment is depend on the participant’s selection of keys

“Without Control” setting

Shocks termination is independent of the dog’s movements

The problem is unsolvable The monetary payment (excluding exploration cost) is independent of the participant’s selection of keys

The second stage

Everyone experience a “with control” setting

The dependent variable

The outcome of successful exploration:The proportion of dogs who succeeded in terminating the shock in the second stage

The outcome of successful exploration: The proportion of participantswho succeeded in solving the problem in the second stage

A direct measure of exploration: exploration rates in the second stage(the percentage of new keys selected by each participant).

Figure 1. A comparison between the current paradigm and previous LH experiments.The keys matrix presented to the right is an example of a screen in the current task, at trial t=11: Dark keys represent keys that have been selected at least once in previous trials and light keys indicate unselected keys. Notethat the number of dark keys is always smaller or equal to t.


The experimental conditions. Three frequencies of rewards

from exploration were examined: p=0.1 representing an extremely

low frequency, p=0.2 for moderate frequency1 and p=1 for

extremely high frequency. In each probability condition there

were two groups of participants: Participants “WithControl”, who

played the entire game with the above probabilities, and

participants that were “Yoked”, to the rewards of a “WithControl”

participant for the first 50 trials and had regular control for

the rest of the game. Exploration was costly for all participants

on all explorative trials. Note that this incentive structure

implies that, for all p values, exploration is optimal under the

control setting but not under absence of control.

Measures of perceived controllability. The perceived

dependency between outcomes and behavior in the exploration task

1 A frequency of 0.2 was chosen based on simulations of the model from Teodorescu & Erev (2014). We looked for frequencies that predict the emergenceof the full LH pattern: WithControl agents explore in more than 50% of the last 50 trials (i.e. after regaining control), while Yoked agents explore in less than 50% of the trials and the gap between the groups does not diminish over time (i.e. Yoked agents do not show any indication for recovery). Simulations of the model with frequencies in discrete increments of 0.1 on theinterval [0, 1] predicted 0.2 as the only frequency for which the full LH pattern should emerge. The results of the model simulations for the whole range of reward frequencies are available online in the Supplemental Material.The results of the model simulations for the actual frequencies used in the experiment are available in Figure 3 bellow.


was measured retrospectively both directly and indirectly. The

direct measure was participants’ rating of the degree of dependence

they felt between their actions and the outcomes they received

(once for their feeling during the first half of the task and

again for their feeling in the second half). However, such direct

measures have been shown to result in weaker effect than measures

based on an indirect prediction task (Presson & Benassi, 1996).

Therefore, to generate the indirect measure, participants

completed 16 prediction trials were they were shown the screen of

another player, t trials into the game, and were asked to predict

the probabilities of each one of the possible outcomes, given a

particular choice on trial t+1. Half of the trials were sampled

from early (t<50), and half from late (t>50) trials. Perception

of control was calculated using the variance between predictions

of reward frequency (the factor over which yoked participants did

not have control) over different actions. Low variance

corresponds to low perceived contingency between action and

rewards. Further details about the prediction task and the

indirect measure are available online in the Supplemental

Material.


Participants and procedure. 120 Technion students (52

female, 68 male; average age= 24) participated in the experiment

in return for a performance based payment2. The experiment lasted

about 15 minutes and the average final payoff was 21.5 NIS (~6

USD). All participants completed the exploration task and the

predictions task one after the other, followed by self-reports of

controllability feelings. The manipulations of reward frequency

and control setting were performed between participants,

resulting in 6 groups of 20 participants.

Results

The exploration task. The upper panel of Figure 2 presents

exploration rates (the percentage of trials in which participants

tried new keys) throughout the task, divided into 4 blocks of 25

trials. A repeated measures ANOVA revealed a significant 3-way

interaction (Block*Reward Frequency* Control Setting:

F(6,342)=3.35, p<0.01, =0.05). Using a Tukey’s post hoc test,

comparison of exploration rates in the two control settings

(WithControl vs. Yoked) revealed the following results: When the

2 The exact incentives are detailed in the Supplemental Material available online.


frequency of rewards from exploration was 0.1, exploration rates

decreased from about 70% in the first block to approximately 40%

in the last block for both groups (p<0.01 for both). However,

when reward frequency was 0.2, participants WithControl continued

to explore in about 70% of the trials, while yoked participants

reduced their exploration rates to approximately 40%. This

difference between the groups was evident in the second block

(p<0.01) and remained after participants regained control (p<0.01

and p=0.09 for the third and fourth blocks respectively). Last,

when reward frequency was 1, yoked participants explored less

than participants WithControl in the second block (p<0.01),

however, this gap disappeared immediately after participants

regained control (p>0.9 for the third and fourth blocks). In

summary, only under moderate reward frequency (P(reward)=0.2) was

the classical LH pattern observed.


00.050.10.150.20.250.30.35

Half 1 Half 2

Variance

W ithControlYoked

00.050.10.150.20.250.30.35

Half 1 Half 20

0.050.10.150.20.250.30.35

Half 1 Half 2

0%

20%

40%

60%

80%

100%

Block1 Block2 Block3 Block4

Explo

ration R

ates

P(reward)=0.1

W ithControlYoked

0%

20%

40%

60%

80%

100%


P(reward)=0.2

0%

20%

40%

60%

80%

100%


P(reward)=1

0

2

4

6

8

10

Half 1 Half 2

RatingW ithControl

Yoked

0

2

4

6

8

10

Half 1 Half 20

2

4

6

8

10

Half 1 Half 2

Indirect M easure of Perceived Controllability

Exploration Rates

Direct M easure of Perceived Controllability

Figuer 2. The upper panel presents mean exploration rates in all groups (with SE) over 4 blocks of 25 trials. The vertical dashed line represents the transition of the yoked groups from an absence of control to a control setting. The middle panel presents Variance scores (with SE) over the two halves of the task. The lower panel presents participants’ ratings of the dependency they felt between actions and outcomes (with SE) over the two halves of the task.

The indirect measure of perceived controllability. Repeated

measures ANOVA revealed that the influence of absence of control

on perceived controllability was not equal across different

reward frequencies (Control setting * Reward frequency:

F(2,114)=10.77, p<0.0001, = 0.16). As can be seen in middle


panel of figure 2 and was evident in Tukey’s test, there was no

difference between the two control groups in the P(reward)=0.1

and P(reward)=0.2 conditions. However, in the P(reward)=1

condition, perceived controllability was significantly higher

among the WithControl group (p=0.0001). Additionally, perceived

controllability increased from the first to the last half of the

task in all groups (F(1,114)=6.88, p<0.01, =0.06).

The direct measure of perceived controllability. Results of

this measure are presented in the lower panel of Figure 2.

Participants’ ratings of the dependency between actions and

outcomes increased from the first to the second halves of the

task in all conditions (main effect of block: F(1,144)=15.63,

p=0.0001, =0.12), however, all other effects were

insignificant, suggesting that self-reports were not a sensitive

enough measure to account for observed behavioral effects.

Discussion

Previous research demonstrated that absence of control leads

to insufficient exploration. This LH pattern has been


traditionally attributed to perceived uncontrollability. However,

our results reveal a causal role for the prevalence of rewards

from exploration. When exploratory efforts were rarely rewarded,

insufficient exploration was observed regardless of the absence

of control manipulation and despite advantageous average

reinforcement. Conversely, when rewards were highly prevalent,

recovery from the absence of control manipulation was immediate

and suboptimal behaviors were not observed. LH patterns emerged

only under moderate prevalence of rewards from exploration.

Moreover, the observed differences in exploration behavior

across conditions were minimally reflected in two perceived

controllability measures. Self-reports showed similar feelings of

control across all conditions. A prediction task revealed some

sensitivity to the control manipulation, which surprisingly

manifested mostly in the high frequency condition, where the LH

effect was not observed, but not in the moderate frequency

condition, where LH did emerged. Therefore, it seems that in the

current context LH is not causally related to perceived

controllability and can emerge even when the average

reinforcement is controlled for.


A sampling account of exploration

Previous studies show that people tend to rely on small

samples of past experiences (e.g. Kareev, 2000; Hertwig, Barron,

Weber & Erev, 2004). In line with the current results, this

mechanism implies great sensitivity to the frequency of

reinforcement since rare outcomes are underrepresented in a

typical small sample. For example, assume that every time the

above mentioned bachelor deliberates whether or not to go on a

first date, he tries to recall outcomes of past first dates but

stops after accessing only three cases. Such small samples will

most often not include amazing but rare dates. Thus, his ongoing

behavior will be guided by the more frequent yet unrewarding

outcome which includes the additional cost (both monetarily and

emotionally) of dating someone new.

To examine whether a simple sampling mechanism is sufficient

to account for our results, we used the explorative sampler model

from Teodorescu & Erev (2013) to derive numerical predictions for

the current setting. The model assumes that after some initial,

random data collection which decreases over time, decisions are


increasingly based on small samples of previous experiences with

exploration (Full description of the model is available online in

the Supplemental Material).

The upper panel in Figure 3 presents the model predictions

with the parameters estimated by Teodorescu & Erev (2013). The

model provides sufficient conditions for the main results: (1)

insufficient exploration in both groups when the frequency of

rewards from exploration is rare (2) exploration rates above 50%

for the WithControl group and below 50% for the Yoked group in

the moderate condition and (3) increasing exploration rates

within the Yoked group after regaining control when exploration

is highly rewarding.


00.050.10.150.20.250.30.35

Half 1 Half 2

Variance

W ithControlYoked

00.050.10.150.20.250.30.35

Half 1 Half 20

0.050.10.150.20.250.30.35

Half 1 Half 2

0%

20%

40%

60%

80%

100%


Explo

ration R

ates

P(reward)=0.1

W ithControlYoked

0%

20%

40%

60%

80%

100%


P(reward)=0.2

0%

20%

40%

60%

80%

100%


P(reward)=1

M odelpredicitons:Indirect M easure of Perceived Controllability

M odel predictions:Exploration Rates

Figuer 3. The upper panel presents exploration rates predicted by theexplorative sampler model from Teodorescu & Erev (2013). The vertical dashedline represents the transition of the yoked groups from an absence of controlsetting to a control setting. The lower panel presents the correspondingVariance scores predicted by the model (the indirect measure of perceivedcontrollability) over the two halves of the task.

The model is also consistent with the finding that

individuals with lower levels of anxiety exhibit less LH (Lavelle

et al., 1979; Coyne et al., 1980). This effect has been suggested

to result from the higher attentional resources available to low

anxiety individuals. From the point of view of the model, more

attention to the task should manifest in larger, more

representative samples leading to improved performance in general

and less LH in particular.


A sampling account of perceived controllability

The model also allowed us to derive predictions for the

indirect measure of perceived controllability as in the actual

prediction task. This was accomplished by drawing one small

sample from experiences preceding each prediction trial and

reporting the distribution of outcomes according to this sample.

The results (Figure 3, lower panel) capture the pattern observed

in the experiment well: Similar perceptions of control among the

two groups in low/moderate difficulties which significantly

increase in the easy condition for both groups, but substantially

more for the Control group. Most importantly, the perceived

controllability results emerged as a property of the reliance on

small samples mechanism, and no assumptions regarding the

dependency between actions and outcome were used (for usage of

such priors in learning models see Huys & Dayan, 2009). This

result is in line with early findings which showed that


contingency judgments were influenced by reward frequency

(Jenkins & Wards, 1965; Alloy & Abramson, 1979).

Summary and implications

The observation that exposure to uncontrollable outcomes can

impair future learning has led to the assumption that feeling of

control is a necessary condition for effective learning. This

study questions this popular assumption. It shows that the effect

of exposure to uncontrollable outcomes depends on reward

prevalence, which like many other learning phenomena can be the

result of a tendency to rely on small samples of past

experiences.

From a practical point of view, reliance on small samples

implies that to enhance exploration, efforts should be made to

change the frequent experience with exploration. For example,

imagine a professor who tries to encourage creative thinking in

his lab. When his students raise new ideas, he can either get

very excited only if it is a great idea (a rare occurrence), or

to get mildly excited from every innovative idea regardless of

its quality. While the former strategy enhances contingency


between actions and the rewards, and could involve higher average

rewards, this study suggests that the latter strategy will be

more effective in generating new ideas.

Acknowledgment

This paper was supported by grants from the Israel Science

Foundation. We thank Eitan Man for his useful comments.

References

Alloy, L. B., & Abramson, L. Y. (1979). Judgment of contingency

in depressed and nondepressed students: Sadder but

wiser?. Journal of experimental psychology: General, 108(4), 441-485.

Alloy, L. B., Abramson, L. Y., Whitehouse, W. G., Hogan, M. E.,

Tashman, N. A., Steinberg, D. L., et al (1999).

Depressogenic cognitive styles: Predictive validity,

information processing and personality characteristics, and

developmental origins. Behaviour Research and Therapy, 37(6), 503–

531.


Clak, D. A., & Beck, A. T. (1999). Scientific foundations of cognitive

theory and therapy of depression. Wiley.

Cohen, S., Rothbart, M., & Phillips, S. (1976). Locus of control

and the generality of learned helplessness in humans. Journal

of Personality and Social Psychology, 34(6), 1049-1056.

Coyne, J. C., Metalsky, G. I., & Lavelle, T. L. (1980). Learned

helplessness as experimenter-induced failure and its

alleviation with attentional redeployment.Journal of Abnormal

Psychology, 89(3), 350-357.

Diener, C. I., & Dweck, C. S. (1978). An analysis of learned

helplessness: Continuous changes in performance, strategy,

and achievement cognitions following failure. Journal of

Personality and Social Psychology, 36, 451-462.

Evans, G. W., Gonnella, C., Marcynyszyn, L. A., Gentile, L., &

Salpekar, N. (2005). The role of chaos in poverty and

children's socioemotional adjustment. Psychological Science, 16(7),

560-565.

Evans, G. W., & Stecker, R. (2004). Motivational consequences of

environmental stress. Journal of Environmental Psychology, 24(2), 143-

165.


Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004).

Decisions from experience and the effect of rare events in

risky choice. Psychological Science, 15, 534–539.

Huys, Q. J., & Dayan, P. (2009). A Bayesian formulation of

behavioral control. Cognition, 113(3), 314-328.

Jardine, E., & Winefield, A. H. (1981). Achievement motivation,

psychological reactance, and learned helplessness. Motivation

and Emotion, 5(2), 99-113.

Jenkins, H. M., & Ward, W. C. (1965). Judgment of contingency

between responses and outcomes. Psychological Monographs: General

and Applied,79(1), 1-17.

Kareev, Y. (2000). Seven (indeed, plus or minus two) and the

detection of correlations. Psychological Review, 107, 397–402.

Lavelle, T. L., Metalsky, G. I., & Coyne, J. C. (1979). Learned

helplessness, test anxiety, and acknowledgment of

contingencies. Journal of Abnormal Psychology, 88(4), 381.

Maier, S. F., & Seligman, M. E. (1976). Learned helplessness:

Theory and evidence. Journal of experimental psychology:

general, 105(1), 3-46.


Martinko, M. J., & Gardner, W. L. (1982). Learned helplessness:

An alternative explanation for performance deficits. Academy

of Management Review, 7(2), 195-204.

Maslach, C., & Jackson, S. E. (1982). Burnout in health

professions: A social psychological analysis. In G. Sanders

& J. Suls (Eds.), Socialpsychology of health and illness. Hillsdale,

N.J.: Erlbaum, 227-251.

Presson, P. K., & Benassi, V. A. (1996). Illusion of control: A

metaanalytic review. Journal of Social Behavior and Personality, 11,

493-510.

Seligman, M. E., & Maier, S. F. (1967). Failure to escape

traumatic shock. Journal of experimental psychology, 74(1), 1-9.

Teodorescu, K., & Erev, I. (2014). On the decision to explore new

alternatives: The coexistence of under- and over-

exploration. Journal of Behavioral Decision Making, 27, 109-123.

Learned Helplessness and Learned Prevalence: Exploring the causal relations of perceived...

Documents

Transcript of Learned Helplessness and Learned Prevalence: Exploring the causal relations of perceived...