Mental Models: An Alternative Evaluation of a Sensemaking Approach to Ethics Instruction
-
Upload
independent -
Category
Documents
-
view
5 -
download
0
Transcript of Mental Models: An Alternative Evaluation of a Sensemaking Approach to Ethics Instruction
ORI GIN AL PA PER
Mental Models: An Alternative Evaluationof a Sensemaking Approach to Ethics Instruction
Meagan E. Brock Æ Andrew Vert Æ Vykinta Kligyte ÆEthan P. Waples Æ Sydney T. Sevier ÆMichael D. Mumford
Received: 23 January 2008 / Accepted: 28 May 2008 / Published online: 21 June 2008
� Springer Science+Business Media B.V. 2008
Abstract In spite of the wide variety of approaches to ethics training it is still
debatable which approach has the highest potential to enhance professionals’
integrity. The current effort assesses a novel curriculum that focuses on metacog-
nitive reasoning strategies researchers use when making sense of day-to-day
professional practices that have ethical implications. The evaluated trainings
effectiveness was assessed by examining five key sensemaking processes, such as
framing, emotion regulation, forecasting, self-reflection, and information integration
that experts and novices apply in ethical decision-making. Mental models of trained
and untrained graduate students, as well as faculty, working in the field of physical
sciences were compared using a think-aloud protocol 6 months following the ethics
M. E. Brock (&) � A. Vert � V. Kligyte � M. D. Mumford
Department of Psychology, University of Oklahoma, 455 W. Lindsey Street, Dale Hall Tower,
Room 705, Norman, OK 73019-2007, USA
e-mail: [email protected]
A. Vert
e-mail: [email protected]
V. Kligyte
e-mail: [email protected]
M. D. Mumford
e-mail: [email protected]
E. P. Waples
Department of Management and Marketing, Louisiana State University in Shreveport, One
University Place, Shreveport, LA 71115, USA
e-mail: [email protected]
S. T. Sevier
Department of Anthropology, University of Oklahoma, 455 West Lindsey, Dale Hall Tower, Room
521, Norman, OK 73019-2007, USA
e-mail: [email protected]
123
Sci Eng Ethics (2008) 14:449–472
DOI 10.1007/s11948-008-9076-3
training. Evaluation and comparison of the mental models of participants provided
further validation evidence for sensemaking training. Specifically, it was found that
trained students applied metacognitive reasoning strategies learned during training
in their ethical decision-making that resulted in complex mental models focused on
the objective assessment of the situation. Mental models of faculty and untrained
students were externally-driven with a heavy focus on autobiographical processes.
The study shows that sensemaking training has a potential to induce shifts in
researchers’ mental models by making them more cognitively complex via the use
of metacognitive reasoning strategies. Furthermore, field experts may benefit from
sensemaking training to improve their ethical decision-making framework in highly
complex, novel, and ambiguous situations.
Keywords Integrity � Ethics � Training � Evaluation � Sensemaking �Mental models � Think-aloud
Introduction
To improve ethical decision-making, one must be able to implement and
subsequently evaluate training, something that is the subject of much debate [55].
Currently, there are several approaches aimed at the enhancement of ethical
decision-making. Such approaches include, but are not limited to, the development
and introduction of ethical codes and policies [1, 7, 38, 64], organizational climate
interventions and assessments [11, 61, 65], and mentoring programs [66].
However, the most notable intervention has been the use of ethics education, also
known as Responsible Conduct of Research (RCR) training [8, 51, 62]. Two
frameworks are most widely utilized in the development of ethics education
training—those with a focus on moral reasoning development [46, 57] and those
that focus on cognitive processes (i.e., sensemaking) underlying ethical decision-
making [45, 53]. The present study will evaluate the sensemaking approach to
training.
Sensemaking involves the generation of a mental model in response to a complex
ethical problem [14, 52]. Prior research has shown that individuals vary in the way
they make sense of different aspects of a problem, and in particular, the
metacognitive reasoning strategies they apply in relation to ethical situations [3,
60]. Metacognition refers to the processes of thinking about one’s own thinking [6].
Researchers suggest that differences in cognition result in the generation and
application of different mental models [26, 50, 68]. To this end, a training program
targeted at enhancing ethical decision-making through engagement in sensemaking,
and metacognitive reasoning strategies in particular, was developed and tested
across the sciences [45, 53]. The assessment (i.e., learning gains) of this
sensemaking ethics training program with regards to immediate knowledge
acquisition and retention has been fruitful in terms of providing validation evidence
for the program [45, 53]. However, a more fundamental question yet to be
answered. Specifically, does sensemaking training actually work to alter the
underlying mental models of researchers?
450 M. E. Brock et al.
123
Thus, the purpose of the present study is twofold. The first goal is to assess a
novel curriculum that focuses on metacognitive reasoning strategies professionals
use when making sense of day-to-day professional issues that have ethical
implications for science. The second goal is to assess mental models of field experts,
or faculty, and graduate students trained and untrained in sensemaking. Assessment
of mental models will be accomplished through an in-depth analysis of underlying
sensemaking processes that scientists use in ethical decision-making and mental
model structure obtained using a think-aloud protocol [19].
Sensemaking Model
Research in sensemaking contends that when people are faced with a novel,
complex, and ambiguous problem having ethical implications for self and others,
individuals tend to derive solutions by actively engaging in sensemaking [15, 68].
More specifically, it is believed that people construct a mental model for
understanding an ethical problem based on personal and professional frames of
reference, when trying to make sense of an ambiguous situation [53]. Sensemaking
refers to building a schematic mental model for decision-making by integrating
different pieces of information in terms of individual emotions, projected outcomes,
individual goals, and social expectations [14]. This information integration, or
sensemaking process, gives rise to a final decision or action. Ethical decision-
making is held to depend on the underlying processes of sensemaking—framing,
emotion regulation, forecasting, self-reflection, and information integration—thus
making it an important framework to examine within ethical decision-making [52,
68]. Figure 1 illustrates the sensemaking model [53].
Framing Emotions
Forecasting
Self-reflection
Sensemaking
Decision
Prior Professional Experience
Prior Personal Experience
Professional & Personal
Goals
Perceived Requirements for Goal Attainment
Responsible Conduct of Research Principles
Perceived Causes of Situation
Fig. 1 Sensemaking model of ethical decision-making [53]
Mental Models 451
123
Sensemaking Training
Mumford and colleagues [53] developed a two-day sensemaking ethics curriculum
for scientists in the social, health, and biological sciences, with the goal of
enhancing ethical decision-making through engagement in sensemaking and
application of seven metacognitive reasoning strategies. Definitions of these
reasoning strategies are provided in Table 1.
The original RCR training curriculum was adapted for professionals specializing
in the physical sciences and engineering by emphasizing the interactive nature of
the intervention through the combination of lecture, discussion, and experiential
exercises [45]. The training consisted of ten modules primarily focused on
discussing the complexity of ethical decision-making and the limitations of a rule-
based approach in highly complex and ambiguous ethical situations; common
reasoning errors in decision-making (e.g., personal biases); and metacognitive
reasoning strategies that help address typical reasoning errors [45]. For a more
detailed description of the RCR training content see Kligyte et al. [45].
Training Assessment
The effectiveness of the training intervention was assessed using an Ethical
Decision-Making (EDM) measure, developed by Mumford and colleagues [54].
This measure assesses four broad domains of ethical behavior, namely (1) data
management, (2) study conduct, (3) professional practices, and (4) business
practices. The measure contains six broad scenarios, followed by three stems, or
subscenarios. The stems represent a wide range of day-to-day ethical events
within the specific professional field being assessed (e.g., data management in
engineering). For each item, eight potential response options are presented, and
Table 1 Metacognitive reasoning strategies
Strategy Operational definition
1 Recognizing one’s
circumstances
Thinking about origins of problems, individuals involved, and
relevant principles, goals and values
2 Seeking outside help Talking with a supervisor, peer, or institutional resource, or
learning from others’ behaviors in similar situations
3 Questioning one’s own and
others’ judgment
Considering reasoning errors that people often make when making
ethical decisions, remembering that decisions are seldom perfect
4 Dealing with emotions Assessing and regulating emotional reactions to situations
5 Anticipating consequences of
actions
Thinking about many possible outcomes such as consequences for
others, and short-term and long-term outcomes based upon possible
decision alternatives
6 Looking within by analyzing
personal motivations
Considering one’s own biases, effects of one’s values and goals,
how to explain/justify one’s actions to others, and questioning
ability to make ethical decisions
7 Considering others’ perspectives Being mindful of others’ perceptions, concerns, and the impact of
personal actions on others, socially and professionally
452 M. E. Brock et al.
123
the participant chooses two most appropriate actions. The responses provided
represent a variety of potential actions ranging from low to high ethicality as well
as application of a particular metacognitive reasoning strategy. Further, the pre–
post assessment of training effectiveness includes the measurement of change in
both ethicality of decision-making and application of metacognitive reasoning
strategies.
Training Results
The sensemaking training resulted in effect sizes (Cohen’s d) ranging from .53 to
1.82 [45]. Furthermore, training also led to a preference for decisions involving the
application of the afore-mentioned metacognitive reasoning strategies, which
facilitate performance on the five major sensemaking processes. More specifically,
significant metacognitive reasoning strategy gains ranged from .60 to 1.51. Based on
the sizeable increase from pre to post-training, the following hypothesis is proposed:
Hypothesis 1 Individuals trained in application of metacognitive reasoning
strategies will be more proficient in sensemaking as compared to untrained
individuals.
While prior research has shown that evaluating the amount of learning
(declarative knowledge) is the most common means of assessing training effective-
ness [13, 41, 42], some researchers propose that the examination of changes in
knowledge structures or mental models holds considerable promise in assessment of
learning in a training context [13, 47]. A mental model is different than declarative
knowledge in that it represents the schematic organization of knowledge (i.e.,
structure), rather than the amount of acquired knowledge [35, 47, 58].
According to Stout and colleagues [63], measures of knowledge structure may be
more likely to uncover effects of learning, as they are less susceptible to attenuation
and are sensitive enough to capture differences even when all participants have
some familiarity with the training concepts [63]. Therefore, as sensemaking training
effectively enhances ethical decision-making through the use of metacognitive
reasoning strategies, it is legitimate to expect that it will also introduce a shift in
novices’ mental models [29, 63]. On the basis of the afore-mentioned assumptions,
the following hypothesis is proposed:
Hypothesis 2 Individuals trained in the application of metacognitive reasoning
strategies during sensemaking will experience a shift in mental model structure as
compared to untrained individuals.
Sensemaking and Expertise
While differences in the sensemaking processes between trained and untrained
novices are important, another area in need of investigation is the influence of
expertise on application of sensemaking processes. When people make decisions
they often refer to their prior personal and professional experiences that facilitate
decision-making in novel situations. Consequently, as individuals gain experience
Mental Models 453
123
they are also acquiring knowledge about influential factors that need to be
considered within specific situations as well as improving upon their metacognitive
reasoning strategies for ethical problem resolution [18].
As experts (i.e., faculty) are more familiar with written and unwritten ethical
guidelines and acceptable professional practices in the field, they likely become
more accurate in identifying ethical implications of the situation [9, 43]. Moreover,
prior exposure to ethical or conflict situations provides individuals with the
framework for dealing with the emotional aspects of a situation [5, 30, 40, 49]. Prior
research has demonstrated that experts rely on case-based knowledge in decision-
making. Case-based knowledge allows experts to access and interpret relevant and
salient information more effectively, primarily due to their effectiveness in
organizing and integrating schemata [16, 32, 34], which facilitates decision-making
when choosing between alternatives [5]. Based on the above discussion of
sensemaking and expertise, the following hypothesis is proposed:
Hypothesis 3 Field experts will be more proficient in the sensemaking process
than either trained or untrained novices.
Following an expertise-focused approach, in theory field experts should have a
superior mental model due to their ability to integrate and structure information
more effectively and efficiently [24, 48]. On the other hand, while experts typically
exhibit superior decision-making, the mental models used across situations tend to
be similar. More specifically, experts tend to apply heuristics in decision-making
across situations [36]. Thus, in novel situations, experts are more likely to default to
an existing mental model instead of constructing a new one or expanding on the old
one based on the demands of a new situation [56].
Due to rich prior experience experts are likely to make an appropriate final
decision. Given this rationale, experts may generate a more ethical final solution
than trained and untrained novices by focusing on the extensive knowledge of
common rules and norms in the field [9, 10, 28]. However, the mental model applied
in ethical decision-making may be similar to an untrained novices’ model in relation
to highly ambiguous and novel ethical problems due to a lack of experience with the
variety of ethical situations outside their major area of expertise. Based on the
above-provided rationale, the following hypothesis is proposed:
Hypothesis 4 Field experts will resemble untrained novices in terms of mental
model structure when solving a novel, highly complex, and ambiguous ethical
problem.
Further, it is possible to predict that individuals trained in sensemaking will
develop superior mental models to experts’ models. Experts rely on their prior
experience in a particular domain, often overlooking the current situational needs,
which limit their exposure to particular issues in an academic environment [44].
Sensemaking training, on the other hand, introduces individuals to a variety of novel
and highly realistic ethical situations that are complex and ambiguous in nature.
Training also emphasizes the application of metacognitive reasoning strategies in
relation to ethical problems that enable an individual to assess the situation from
different perspectives by considering both personal and contextual factors when
454 M. E. Brock et al.
123
solving ethical problems [45, 53]. The focus on strategy application in a myriad of
novel and realistic situations may aid in reconstructing trainees’ initial novice
mental model by increasing its complexity and adaptability to changing situational
demands. Based on the afore-mentioned arguments, the following hypothesis is
proposed:
Hypothesis 5 Individuals trained in sensemaking will exhibit a superior mental
model structure as compared to untrained field experts and novices.
As the present study attempts to assess the effectiveness of the sensemaking
training by examining it from two different perspectives—learning gains and shifts
in mental model structure—a think-aloud protocol was chosen as the most suitable
method for extracting information on cognitive processes underlying ethical
decision-making.
Method
Sample
Twenty-eight faculty, research scientists, and graduate students in a top-ranked
program, from a large Southwestern university volunteered to take part in the 2-hour
study. All participants were members of a multi-disciplinary, multi-university
research center involved in the development of weather sensing technology. At the
time of this study, the branch of research center at the focal university had 17
graduate students, and 21 faculty and research scientists specializing in the fields of
electrical engineering, computer science, and meteorology. The population was
chosen because the research center members were offered the opportunity to
voluntarily participate in the RCR sensemaking training approximately 6 months
prior to the study. The total sample for the present study included 15 faculty and
research scientists (Mtenure = 16.5, SD = 2.47) and 13 graduate students (Mten-ure = 3.5, SD = 1.27). The sample included 64% males and 71% Caucasians.
Specifically, six of the 28 participants (all graduate students) had attended the
sensemaking-oriented RCR training. Thus, the final sample included 13 faculty and
research scientists (hereafter field experts), six RCR trained graduate students, and
seven untrained graduate students (hereafter trained and untrained novices). Due to
poor audio quality two field expert interviews were not included in the final
analyses.
Procedure
Participants were recruited for the study via e-mail. Upon arrival, participants were
asked to sign an informed consent form and were provided with detailed
instructions. Participants were asked to think-aloud in relation to four scenarios,
each addressing one of four areas of ethical conduct, namely (1) data management,
(2) study conduct, (3) professional practices, and (4) business practices [31] to
examine the processes that people engage in when making decisions in relation to
complex and ambiguous ethical situations.
Mental Models 455
123
Think-aloud Procedure
As mental models represent how people develop a solution and what processes are
salient in decision-making, their examination requires the use of a verbalization
technique. Therefore, a think-aloud (TA) method was selected for assessing the
implicit schematic mental models of people engaged in ethical decision-making.
Think-aloud protocol involves a participant solving a problem and making a
decision by verbalizing their thoughts aloud while performing a task [20]. This
method is particularly helpful in assessing implicit mental processes, because it (1)
enables the collection of great quantities of uncensored, spontaneous statements by
a participant that are more reflective of the actual thinking process, and (2) does not
interfere with the decision-making task [21, 23].
The TA protocol consisted of two stages, first (1) unstructured and then (2)
structured (control) verbalization. In an unstructured protocol, while discussing
each scenario the participant was instructed to TA in a stream of consciousness
while generating a solution to the scenario. The experimenter did not prompt the
participant unless the individual stopped speaking. In such cases scripted prompts
were applied (e.g., ‘‘Where did you get that idea from?’’). Of note is that the
interviewer was not blind to the level of the participants’ expertise; however only
the predetermined prompts were applied across participant groups. The structured
protocol was included for control purposes to see if participants’ mental models
would replicate the order and structure of questions asked. In the structured
protocol, the participants were provided with a series of specific questions
targeted at extracting information in relation to the specific sensemaking
processes. For example, to address situation appraisal the following question
was asked: ‘‘What do you see as the primary problem in this situation?’’, whereas
to address environmental monitoring the participants had to TA in response to the
following question: ‘‘What kinds of factors in the scenario are important to
consider in solving the problem?’’ In line with previous suggestions for enhancing
the TA process, participants were provided with an opportunity to practice
thinking-aloud with the experimenter prior to the actual session to an unrelated
issue [4].
Following the TA practice session, the same procedure, as described above, was
performed in relation to all four scenarios. Upon completion of the study
participants were debriefed and thanked for participation. Participants’ responses
were recorded via digital voice recorder and transcribed for coding purposes.
Scenario Development and Content
As previously discussed, the participants responded to four scenarios. These
scenarios pertained to one of the following four areas of ethical behavior discussed
previously. The underlying ethical problems discussed in the scenarios were
identified on the basis of ethnographic observations conducted at the research center
and information provided by two Subject Matter Experts (SMEs) in the fields of
meteorology and computer science. The scenarios were generated by a panel of
three psychologists familiar with the ethics literature and reviewed by the SMEs to
456 M. E. Brock et al.
123
increase psychological fidelity of the scenarios by focusing on the relevancy and
saliency of issues to the field.1
Coding
The transcribed unstructured and structured verbalizations were coded indepen-
dently for each scenario. Consequently, eight responses were coded in relation to
each participant. To analyze the TA transcriptions a coding system was developed
following the sensemaking framework (see Fig. 1). The 15 processes underlying
five broad sensemaking constructs were identified theoretically by a panel of five I/
O psychologists using existing literature on sensemaking and ethical decision-
making [53]. The examined processes were as follows: (1) framing (problem
appraisal, goal assessment, environmental monitoring, norm-based framing, value-
based framing, perceived threat and opportunity), (2) affect (emotion regulation),
(3) forecasting (autobiographical extraction, outcome assessment, solution revision,
contingency planning), (4) self-reflection (other-based perceptions, self-based
perceptions), (5) sensemaking (information integration, solution appraisal)—and
the overall ethicality of the final decision. For each of the 15 sensemaking processes,
anywhere from 6 to 19 behavioral cues were developed by the same panel of
psychologists using existing literature in cognition and sensemaking processes,
resulting in 210 items to be rated, including the overall ethical decision. Table 2
provides examples of constructs and specific items subsumed under them.
Participants’ responses were rated by three expert judges, which consisted of two
I/O psychologists and an anthropologist familiar with sensemaking and ethical
decision-making literature. As the judgments were being made on 15 complex
sensemaking processes these judges were more appropriate for rating purposes than
scientists, who are less familiar and thus less comfortable with sensemaking
concepts [2]. Judges were blind to the group membership. Prior to rating, the judges
received 30 h of frame-of-reference training in relation to each of the sensemaking
constructs. Training consisted of (1) familiarizing with the definitions and examples
of processes; (2) individual practice of rating responses, and (3) group discussions.
Inter-rater agreements were calculated for each construct, and averaged to create a
single composite agreement index, ICC = .92.
Manipulation Check
The relevancy of scenarios was evaluated by three blind raters, I/O psychologists,
who determined if a typical individual would identify the constructs, or sensemak-
ing processes, as salient to the situation at hand and would address them in their TA
process while generating a decision. The ratings were performed using a 5-point
Likert scale (1-not at all, 5—very much so). Items were considered irrelevant when
at least two out of three raters assigned a value of 1 to a specific item. However, as
the ratings were consistently scored higher than 1, no items were excluded from the
analyses.
1 The scenarios are available from the authors upon the request.
Mental Models 457
123
Table 2 Sensemaking model processes and underlying dimensions
Sensemaking
process
Dimension Dimension definition Item example
Framing Problem
Appraisal
Assessing the primary causes of
the issue at hand
• The primary cause of the
problem is clearly identified
• The cause of the problem is
believed to be environmental
factors
Goal Assessment Discussing the goals and
expectations of all parties
involved in the situation
• The goals of the organization
are clearly stated
• The individual expectations of
the sponsor/consumer are
clearly stated
Environmental
Monitoring
Discussion of the current political,
financial and social climate
• The current political climate is
discussed
• The current economic climate
is discussed
Norm-based
Framing
Discussion of social and
professional rules, codes of
conduct
• Social values considered in
solving the problem are
discussed
• Professional conduct rules
considered in solving the
problem are discussed
Perceived Threat
and Opportunity
Individual discussion of the
threats and opportunities of a
potential solution, for all parties
• Direct threats to others are
discussed
• Indirect threats to the
individual are discussed
Value-based
Framing
Individual consideration of
beliefs, values and personal
motivations
• Decision consistency with
values are discussed
• Personally oriented motivations
are discussed
Affect Emotion
Regulation
Assessing and regulating
emotional reactions
to situations
• Emotional responses are
discussed
• Weighing emotional versus
rational factors is discussed
Forecasting Autobiographical
Extraction
Discussion of personal and
professional experiences and
outcomes
• Prior professional experiences
are described in reference to
the problem
• Professional feedback received
in the past is discussed
Solution Revision Individual discussion of
alternative solutions and
outcomes given the situation at
hand
• Alternative outcomes are
generally discussed
• Alternative outcomes for the
team are discussed
Contingency
Planning
Development of back-up plans to
address potential issues in the
future
• Alternate plans are discussed in
terms of something going
wrong in the future
• The adaptability of the idea is
discussed
458 M. E. Brock et al.
123
Analysis
In order to further investigate the validity of the sensemaking training, two types of
analyses were conducted. First, a one-way analysis of variance was conducted to
evaluate differences between field experts, trained, and untrained individuals on the 15
constructs related to the overarching sensemaking processes examined in the present
study. Second, Pathfinder [59] was utilized to examine the mental model structure of
participants as represented by the 15 constructs related to sensemaking (see Table 3).
Pathfinder generates a graphical representation of the relatedness of concepts
using a network scaling procedure [59]. The linkages represent relationships
between concepts, and the length of the linkage represents the strength of the
relationship. The clustering of the concepts indicates that multiple concepts are
more related to each other than to concepts in a different cluster. For a complete
description of Pathfinder refer to Schvaneveldt [59]. Pathfinder networks have been
used in the past to evaluate training and have been reported as a significant source of
additional validation information [27].
Computing Distance Scores
In the present study, a distance metric was utilized to determine the interrelatedness
of the 15 sensemaking constructs. The first step in generating the distances was to
Table 2 continued
Sensemaking
process
Dimension Dimension definition Item example
Self-
reflection
Self-based
Perceptions
Consideration and justification of
individual ideas and opinions
• The individual justifies the
decision to himself/herself
• The individual acknowledges
self-biases
Others-based
Perceptions
Situational perceptions of friends,
colleagues, advisors and sponsors
regarding the situation
• Asking an advisor for their
input is discussed
• Directly asking for other
people’s opinion if they cannot
forecast their reactions is
discussed
Sensemaking Information
Integration
Consideration of the advantages
and disadvantages for all parties
given a certain course of action
• Personal advantage of the
situation is discussed
• The severity of consequences
of alternatives is discussed
Solution
Appraisal
Assessment of solution
effectiveness and ethicality
• The strategy for problem
resolution is logical
• The strategy for problem
resolution is ethical
Decision Final Decision Ethicality of final solution • The final idea proposed is
ethical
• Outcomes associated with the
final solution are ethical
Mental Models 459
123
determine difference scores using the absolute value of the differences for all
possible comparisons between the concepts for the mean of each group. Next,
Mahalanobis distances were computed using the differences scores. The distance
metric was computed in this manner because Mahalanobis distances are sensitive to
sample size. Thus by using difference scores sample size was consistent across all
groups. Finally, the distances were entered into Pathfinder for analysis. The
resulting representations refer to the similarities of the coded response scores of the
groups involved in ethical decision-making. Although the representation is not
directional, in that it does not demonstrate exactly the processes in order through
which the participants made their decisions, the similarities of processes demon-
strate the extent that processes are discussed in relation to each other.
Results
In the present study, each participant’s TA protocol included both an unstructured
and structured component. The outcomes of the structured protocol were used for
control purposes as a check to see how participants would respond if they were
guided through the think-aloud process with structured questions. Results of the
analyses on the structured protocol revealed that participants’ mental model
structure adhered to the order in which the questions were asked and differed from
Table 3 Results of one-way ANOVA with post-hoc comparisons for underlying sensemaking processesa
One-way ANOVA Post-hoc comparisons
Untrained
novice (1)
Trained
novice (2)
Field
expert (3)
1 vs. 2 1 vs. 3 2 vs. 3
M SD M SD M SD F p p p p
Problem Appraisal 1.63 0.16 1.83 0.11 1.94 0.18 11.81 0.00 0.01 0.00 0.09
Emotion Regulation 1.01 0.01 1.03 0.03 1.23 0.18 18.67 0.00 0.64 0.00 0.00
Goal Assessment 1.54 0.18 1.57 0.12 1.93 0.18 29.48 0.00 0.59 0.00 0.00
Environmental Monitoring 1.72 0.25 1.70 0.23 2.11 0.19 21.18 0.00 0.80 0.00 0.00
Autobiographical Extraction 1.07 0.07 1.03 0.02 1.39 0.27 18.52 0.00 0.56 0.00 0.00
Norm-based Framing 1.54 0.26 1.69 0.19 1.82 0.20 7.85 0.00 0.08 0.00 0.08
Others-based Perceptions 1.25 0.16 1.27 0.08 1.32 0.12 0.61 0.55 0.77 0.30 0.50
Information Integration 1.59 0.16 1.68 0.19 1.83 0.17 7.02 0.00 0.24 0.00 0.04
Solution Appraisal 2.57 0.29 2.71 0.27 3.20 0.37 17.47 0.00 0.32 0.00 0.00
Threat and Opportunity 1.47 0.22 1.56 0.21 1.83 0.23 12.54 0.00 0.33 0.00 0.00
Value-based Framing 1.28 0.17 1.38 0.13 1.60 0.22 14.35 0.00 0.17 0.00 0.00
Outcome Assessment 1.33 0.15 1.39 0.15 1.53 0.19 5.53 0.01 0.41 0.00 0.04
Solution Revision 1.23 0.14 1.32 0.18 1.38 0.21 1.65 0.20 0.40 0.07 0.45
Self-based Perceptions 1.40 0.16 1.43 0.08 1.70 0.16 30.03 0.00 0.54 0.00 0.00
Contingency Planning 1.22 0.11 1.26 0.05 1.28 0.18 0.51 0.60 0.62 0.32 0.69
Ethical Decision 2.63 0.28 2.91 0.23 3.07 0.43 4.64 0.01 0.10 0.00 0.30
a F(2,101); p \ .05
460 M. E. Brock et al.
123
the findings of the unstructured protocol.2 Prior to discussing mental model results
(e.g., Hypotheses 2, 4, & 5), ANOVA results will be presented for the unstructured
protocol.
Assessment of Individuals’ Proficiency in Sensemaking
A one-way ANOVA was conducted to identify whether untrained novices, trained
novices, and field experts differed on the 15 sensemaking processes, and overall
ethical decisions, that were coded by the expert judges. Results indicated significant
group differences on 12 of the 15 sensemaking processes, namely problem
appraisal, emotion regulation, goal assessment, environmental monitoring, auto-
biographical extraction, norm-based framing, information integration, solution
appraisal, perceptions of threat and opportunity, value-based framing, outcome
assessment and self-based perceptions, as well as overall ethical decision made. To
further assess the differences between groups, a series of post-hoc pairwise
comparisons, using Tukey’s Honestly Significant Difference (HSD) test, were
conducted. Full means, standard deviations, ANOVA and post-hoc test results for
the unstructured condition may be found in Table 3.
The first comparison was done to examine Hypothesis 1, which stated that those
individuals who had been trained in the application of metacognitive reasoning
strategies would be more proficient in sensemaking than untrained novices. Trained
novices scored significantly higher one of framing processes, problem appraisal. No
other significant differences emerged. The results of the mean differences in
sensemaking processes between the trained novices and untrained novices yield
weak support for Hypothesis 1.
The second and third comparisons examined Hypothesis 3, which stated that field
experts would be more proficient than both untrained and trained novices in
sensemaking. Experts were found to be more proficient in sensemaking processes
than untrained novices. More specifically, experts’ performance was superior in
problem appraisal, goal assessment, environmental monitoring, norm-based fram-
ing, value-based framing, and perceived threat and opportunity—all components of
framing. Further, experts were more proficient in emotion regulation, and also in the
markers of forecasting, namely autobiographical extraction, and outcome assess-
ment. Interestingly, there were no differences between experts and untrained
novices for the contingency planning. Similarly, in terms of self-reflection there
were no differences observed in relation to other-based perceptions; however
experts significantly focused on self-based perceptions. Marked by differences in
information integration and solution appraisal, experts outperformed novices in the
integrative sensemaking process. Further, there was a significant difference between
experts and untrained novices on overall ethical decision, with experts outperform-
ing untrained novices. Thus, for the comparison of experts against untrained
novices, this hypothesis was supported.
Regarding the comparison of field experts to trained novices different findings
emerged. Here, experts were more proficient in the framing processes of goal
2 The results of the structured protocol are available from the authors upon the request.
Mental Models 461
123
assessment, environmental monitoring, value-based framing, and perceived threat or
opportunity. No differences were found for problem appraisal or norm-based
framing. Experts also surpassed trained novices in emotion regulation. However,
experts were superior to trained novices in only two of four forecasting components,
autobiographical extraction and outcome assessment, and only one self-reflectionmarker, self-based perceptions. Finally, experts outperformed trained novices in
both sensemaking components, information integrations and solution appraisal.
Despite outperforming trained novices on several of the sensemaking processes,
experts and trained novices did not differ in their overall ethical decisions made.
Thus, for the comparison of experts against trained novices, this hypothesis was
partially supported.
Assessment of Mental Models
To help determine the extent of mental model shifts, both the structured and
unstructured TA protocols were assessed. This was done because implicit mental
models obtained via the structured, or prompted protocol (control) should be more
similar to each other than those extracted via the unstructured protocol (see footnote
2). As the ultimate goal of this study was to examine the unprompted cognitive
processes of individuals, only the unstructured protocol analyses are presented. The
mental models represent the interrelationships between constructs, or patterns of
constructs. Thus, schematic mental model processes should be used to interpret the
structure of the sensemaking process depicted.
Hypothesis 2 predicted that the mental models of individuals trained in the
application of metacognitive reasoning strategies would evidence a shift from those
not trained in the application of strategies. In the untrained novice’s mental model
the core component is environmental monitoring. This initial monitoring is
influenced by others’ perceptions and value-based framing. Ultimately, the appraisal
and revision of the solution is also centered on environmental monitoring. The
second core component is autobiographical extraction. Thus, past experience
essentially drives norm-based framing, goal assessment, and information integration
for untrained novices. Of note is that perceived threat or opportunity is directly
linked to emotion regulation, which is related most strongly to goal assessment and
self-based perceptions.
The mental model of the trained novices is considerably different conceptually.
First, the core component in the trained novices’ mental model is the appraisal of the
problem. This is linked to both forecasting (i.e., outcome assessment and solution
revision), self-reflection (other-based perceptions), and sensemaking (solution
appraisal). Thus, for the trained novices, framing, in the form of a problem-based
assessment, is essential. This assessment is also linked to contingency planning,
which incorporates additional elements of framing (i.e., norm-based framing and
environmental monitoring). Information integration is then directly linked to
contingency planning. Of note is that the most distal influences on information
integration, and ultimately the problem solution, are autobiographical extraction,
goal assessment, and emotion regulation, respectively. From the examination of the
mental models of trained and untrained novices is apparent that considerable
462 M. E. Brock et al.
123
conceptual differences exist between how these two groups sensemake during
ethical decision-making, providing support for Hypothesis 2. These differences will
be more clearly delineated in the discussion. Figures 2 and 3 illustrate the mental
models for untrained and trained novices in the unstructured condition. Of note is
that these mental models are schematic in nature. They outline the relationships
between 15 sensemaking processes and illustrate their significance in researchers’
ethical decision-making examined via unstructured think-aloud protocols.
Hypothesis 4 stated that experts would resemble untrained novices in their mental
model structure. The key similarity for the unstructured models is that both groups
rely on environmental monitoring. Both groups utilize autobiographical extraction
to inform the initial appraisal of the problem and the overall framing. Similarly,
both groups utilize emotion regulation in conjunction with self-based perceptions
and perceived threat or opportunity. This evidence provides strong support for
Hypothesis 4. The schematic mental model for field experts in the unstructured
condition can be found in Fig. 4.
Finally, Hypothesis 5 stated that individuals trained in the application of
metacognitive reasoning strategies would exhibit a superior mental model structure
when compared to either experts or untrained novices. A central piece of trained
Emotion Regulation
Problem Appraisal
Goal Assessment
Env. Monitoring
Autobiographical
Norm-based Framing
Ethical Decision
Contingency Planning
Other-based Perceptions
Self-based Perceptions
Info Integration
Solution Appraisal
Perceived Tht/Opp
Value-based Framing
Outcome Assessment Solution Revision
Fig. 2 Mental models for untrained novices (N = 7) based on the 15 sensemaking processes asdiscussed in the unstructured protocol condition
Mental Models 463
123
individuals’ mental model structure is problem appraisal, a key framing construct.
Thus, the entire mental model of sensemaking, for trained individuals, is built around
active assessment of the ethical problem. On the contrary, untrained novices and
experts rely more centrally on environmental monitoring, which may be defined as a
more passive approach to information gathering—a decision is based on what is
superficially recognized and observed in the environment. Thus, for field experts, and
particularly for untrained novices, problem appraisal appears to be a much less central
process in ethical decision-making. That being said, the mental models produced by
trained novices are, in fact, superior in nature to those produced by either field experts
or untrained novices, providing support for Hypothesis 5. The reason for this is that
sensemaking involves processes that are meant to gather a variety of information
about a situation. Within sensemaking training the major focus is the application of
metacognitive reasoning strategies. Thus, we would expect that trained individuals
would take a more active information seeking approach to situational assessment, or
framing and forecasting within the situation than untrained individuals.
Discussion
Prior to discussing the findings and implications of the present study, certain
limitations should be noted. In particular, this study is limited by sample
Emotion Regulation
Problem Appraisal
Goal Assessment
Env. Monitoring
Autobiographical
Norm-based Framing
Ethical Decision
Contingency Planning
Other-based Perceptions
Self-based Perceptions
Info Integration
Solution Appraisal
Perceived Tht/Opp
Value-based Framing
Outcome Assessment Solution Revision
Fig. 3 Mental models for trained novices (N = 6) based on the 15 sensemaking processes as discussedin the unstructured protocol condition
464 M. E. Brock et al.
123
characteristics. First, the small sample size in the present study limits the strength of
inferences drawn as well as the generalizability of the findings. That being said, the
sample size for the present study is fairly typical for think-aloud protocols, which
normally utilize 10–15 participants, as such methods are relatively time consuming
and costly [20].
Second, the samples sizes for individual groups, particularly the trained and
untrained groups, are small. However, since the numbers of trained individuals were
limited and all participants were from one institution, it is difficult to obtain more
participants, as departments are limited in size and this was a voluntary effort.
Further, based on the population of interest, over 70% of that population
participated in the study.
Third, individual differences were not assessed. Prior research has shown that
qualities such as verbal intelligence [67] and personality traits such as extraversion
[33] may impact the nature of think-aloud protocol results. More importantly,
previous research has shown that individual differences such as emotional
intelligence [22, 25, 69] and neuroticism and openness [3] impact receptivity and
application of training efforts [45]. However, due to concerns for anonymity by the
sample population data collection on individual differences was restricted.
Fourth, the study was limited by the application of a non-crossed design. A
crossed design would have provided additional insight into differences between the
Emotion Regulation
Problem Appraisal
Goal Assessment
Env. Monitoring
Autobiographical
Norm-based Framing
Ethical Decision
Contingency Planning
Other-based Perceptions
Self-based Perceptions
Info Integration
Solution Appraisal
Perceived Tht/Opp
Value-based Framing
Outcome Assessment
Solution Revision
Fig. 4 Mental model for field experts (N = 13) based on the 15 sensemaking processes as discussed inthe unstructured protocol condition
Mental Models 465
123
groups; however this project was a follow-up study meant to evaluate the
effectiveness of ethics training using an alternative assessment approach 6 months
following the training. As the training was conducted in an actual organization—a
multi-disciplinary and multi-university research center—and the participation was
voluntary, we did not have control over who received training. As no faculty or
research scientists took part in the training it was unfeasible to cross faculty versus
graduate students with trained versus untrained conditions. Future research should
investigate the differences between field experts trained and untrained in research
ethics to validate the currently obtained findings.
Finally, the present study is limited by the fact that the differences between
groups examined in the present study were only examined within a sensemaking
training framework. Thus, we were not able to examine differences in sensemaking
processes from individuals who had received a different form of ethics training (i.e.,
professional associations or codes of ethics). The evaluation methodology proposed
in the present study could be potentially applied to examine alternative methods for
ethics training.
Bearing these limitations in mind, the present study provides several compelling
and thought provoking findings. First, the study provides additional validation
evidence for sensemaking training by identifying the differences in key sensemak-
ing processes applied by field experts, untrained novices, and trained novices in
complex and ambiguous ethical situations. This is notable as interviews were
conducted 6 months following the training providing evidence for knowledge
transfer and retention. Second, considerable structural differences in mental models
of field experts, untrained, and trained novices were found, both in terms of the
number and types of relationships between key sensemaking processes. More
specifically, trained novices exhibited a shift in mental model, generating a more
cognitively complex model than untrained novices and experts. Further, field
experts demonstrated mental model structures that were similar to untrained
novices’, in that they were routine in nature. Consistent with prior research on
expertise [32, 43], it is not surprising that field experts were more proficient than
untrained and trained novices in sensemaking.
It is also not surprising that trained novices were more proficient than untrained
novices in application of sensemaking processes, as trainees were exposed to a
number of metacognitive reasoning strategies practiced in a variety of novel and
complex situations during ethics training. However, interestingly, this did not result
in decisions higher in ethicality than of trained novices’. Such results further
confirm that training should not only be examined in terms of group mean
differences in the application of sensemaking processes but also in terms of
differences in mental model structure [13].
When comparing the mental model structures of trained novices versus untrained
novices and field experts it is evident that those who have been trained exhibit a
model that is more cognitively complex. While experts and untrained novices
develop solutions to ethical problems by focusing on environmental influences,
trained novices tend to take a more in-depth analysis of the ethical situation by
assessing specific situation needs and considering themselves and their role, or
position, in the context of the problem. Trained individuals look within by assessing
466 M. E. Brock et al.
123
personal as well as other people’s motivations in the situation and by questioning
potential biases in their assessment. Greater awareness of the self in context of
ethical problem represents a more internal focus in ethical decision-making as
compared to untrained novices and experts. Therefore, the trained individuals are
more likely to identify potential ethical implications in relations to complex and
ambiguous situations, which may not always be the case with untrained novices and
experts. A clear understanding of the problem and consideration of ideas and
opinions in context results in a decision that reflects one’s perception of the correct
course of action and leads to better-informed decisions.
Conversely, experts take a more pragmatic world view. On the basis of
organizational experience, experts have learned to assess problems in terms of
potential risks and benefits within a given situation, especially when the situation
has implications for the self. In highly ambiguous situations experts may become
defensive in situations of perceived threat; resulting in less ethical decisions if the
perceived threats or benefits of an ethical decision are outweighed by the need for
self-protection.
When developing solutions to problems, experts, like trained novices, search
their memory for prior personal and professional experiences that facilitate the
interpretation of current environmental cues. Doing so provides a framework for
forecasting potential outcomes. Further, examination of their mental models
indicate that untrained novices and experts are solving ethical problems by
monitoring the environment and collecting additional information necessary to
adjust their schemata to a new situation. However, they are not accurately
appraising the problem via the assessment of specific situational and personal needs,
or unique circumstances, of the situation. Overall, it has been shown that untrained
novices and experts were quite similar when generating solutions to highly complex
and ambiguous ethical problems—despite the experts’ greater skill in applying this
model.
While environmental monitoring is salient in ethical decision-making, one has to
assess both external and internal (i.e., personal) factors in the situation to obtain a
broad picture of a problem. Furthermore, given that untrained novices do not have a
considerable amount of prior experience with ethical issues, they are developing
solutions to highly complex and ambiguous situations on the basis of their limited
and poorly integrated schemata as well as unidimensional assessment of environ-
mental cues, which naturally result in poorly-informed decisions [37].
Poor decisions often stem from people’s tendencies to over-rely on their previous
experiences, which do not always generalize to new situations. Prior research has
shown that experts tend to apply heuristics, which result in fast and often highly
effective solutions due to rich, flexible, and well-integrated schemata [9]. However,
this strategy may not be effective in novel and highly ambiguous situations, in
which experts may have no prior exposure. For example, field experts are highly
familiar with ethical violations inherent in research practices (e.g., data hoarding,
falsification, etc.). However, they may not be familiar with ethical considerations
within the domain of professional practices (e.g., moral leadership, effective
collaboration and communication). Thus applying heuristics from one ethical
situation to the next may not always lead to an optimal ethical action. Overall, the
Mental Models 467
123
differences between experts and untrained novices are consistent with a more
personalized outcome focus versus a strategic focus—a difference that often
emerges in novice and expert comparisons [39]. Nonetheless, both models represent
externally driven decision-making systems, based in perceived treat versus
opportunity (experts) or environmental demands (novices).
Potential Explanations
There are several potential explanations for the significant differences in mental
model of trained and untrained individuals. Sensemaking training results in an
alteration of how an individual appraises and understands the problem, providing a
framework for working through the decision-making process. By introducing
trainees to a myriad of real world problems followed by a set of metacognitive
reasoning strategies necessary to make sense of these problems and extensive
practice in application of the strategies to actual and highly realistic cases provided
trainees with the necessary tools to appropriately respond to ethical issues in novel
situations. While this is a plausible explanation for the shift in mental models, the
underlying processes and actual mechanisms driving the mental model shifts, are
not known. We can only assume, based on the results, that sensemaking training
facilitates changes in mental models by increasing their structural complexity and
flexibility. Therefore, extensive research is needed in trying to understand the
relationship between sensemaking processes and mental models, which may result
in more effective and efficient future ethics training efforts.
Implications
Bearing limitations in mind, there are several theoretical and practical implications
are of note based on the findings of this investigation. First, the examination of
training effectiveness is enhanced via investigation of differences of groups’ mental
model structure. Consistent with Day et al.’s [13] research, the evaluation of the
mental models, and more specifically the shift in mental models of trainees,
provided greater insight into the sensemaking process, as wells as the effects of
sensemaking training on ethical decision-making.
Second, the use of multiple methods in evaluating the training provides
additional training validation evidence. The examination of implicit mental models
and significant group differences found between untrained novices and experts and
trained individuals provide further validation evidence that training in sensemaking
can improve one’s ability to make ethical decisions across novel situations. The use
of the think-aloud protocol approximately 6 months after training demonstrated that
the training is effective and that the effects are retained over time. Thus by training
individuals on a variety of metacognitive reasoning strategies and introducing them
to a variety of novel real world cases may enhance professionals’ ethical decision-
making process, which will result in better-informed decisions.
Third, training benefits novices significantly as it improves their sensemaking
process, and, consequently, results in the generation of a more complex and
468 M. E. Brock et al.
123
effective mental model. Furthermore, since the structural changes in mental models
for ethical decision-making are not determined by expertise per se, exposing experts
to sensemaking training may enhance their mental model as well. Otherwise,
untrained experts and novices will maintain and apply an outcome-based model
rather than an analytic model, which can result in poor decision-making, especially
in novel and highly complex and ambiguous situations, which are characteristic of
the situations that those in the sciences and engineering face [12, 17]. Therefore,
professionals, in spite of their expertise in any given field, should consider
improving their mental models and ethical decision-making via education in
sensemaking.
Conclusion
Overall, the results of the present study indicate that sensemaking training can lead
to a shift in mental model structure. Specifically, models become more complex and
information becomes highly integrated. Although the sensemaking training was not
specifically designed to alter mental models of trainees, the results are not
necessarily surprising as sensemaking training introduces a complex way of
thinking about novel, highly complex and ambiguous ethical situations that
researchers face in their professional lives. The study demonstrates that training can
significantly benefit both field experts and novices by improving their sensemaking
process as well as introducing complexity in their mental models. The shift in
mental models is facilitated by the introduction of trainees to a diversity of novel
and realistic ethical situations that can be resolved by applying a combination of
metacognitive reasoning strategies. Such strategies are helpful in dealing with the
complexity and ambiguity posed by ethical problems. Consequently, field experts
may also significantly benefit from sensemaking training to improve their ethical
decision-making framework in novel and highly complex and ambiguous ethical
situations.
Acknowledgements We thank Elaine S. Godfrey and Richard T. Marcy for their assistance in
developing theoretical framework and data collection materials for the project. We also thank Dr. Dean F.
Hougen for sharing his expertise in physical sciences which was beneficial in contextualizing the obtained
information. We would also like to acknowledge the National Science Foundation (NSF), contract No.
SES 0529910. The resarch was funded by the Council of Graduate Students Grant, contract No. LTR
090506.
References
1. Adams, J. S., Tashchian, A., & Shore, T. (2001). Codes of ethics as signals for ethical behavior.
Journal of Business Ethics, 29, 199–211.
2. Anderson, M. S. (2007). Collective openness and other recommendations for the promotion of
research integrity. Science and Engineering Ethics, 13, 387–394.
3. Antes, A. L., Brown, R. P., Murphy, S. T., Hill, J. H., Waples, E. P., Mumford, M. D., et al. (2007).
Personality and ethical decision-making in research: The role of perceptions of self and others.
Journal of Empirical Research on Human Research Ethics, 2, 15–34.
Mental Models 469
123
4. Austin, J., & Delaney, P. F. (1998). Protocol analysis as a tool for behavior analysis. Analysis ofVerbal Behavior, 15, 42–56.
5. Baumeister, R. F., Vohs, K. D., DeWall, C. N., & Zhang, L. (2007). How emotion shapes behavior:
Feedback, anticipation, and reflection, rather than direct causation. Personality and Social Psy-chology Review, 11(2), 167–203.
6. Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995). Metacognition and
problem solving: A process-oriented approach. Journal of Experimental Psychology, 21, 205–223.
7. Bullock, M., & Panicker, S. (2003). Ethics for all: Differences across scientific society codes. Scienceand Engineering Ethics, 9, 159–170.
8. Chen, F. W. (2003). A study of the adjustment of ethical recognition and ethical decision-making of
managers-to-be across the Taiwan Strait before and after receiving a business ethics education.
Journal of Business Ethics, 45, 291–307.
9. Chermack, T. J. (2003). Mental models in decision making and implications for human resource
development. Advances in Developing Human Resources, 5, 408–422.
10. Chi, M. T. H., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics
problems by experts and novices. Cognitive Science, 5, 121–152.
11. Davis, M. (2003). The role of culture in research misconduct. Accountability in Research, 10, 189–201.
12. Davis, M. (2006). Engineering ethics, individuals, and organizations. Science and EngineeringEthics, 12, 223–231.
13. Day, E. A., Arthur, W., & Gettman, D. (2001). Knowledge structures and the acquisition of a
complex skill. Journal of Applied Psychology, 86(5), 1022–1033.
14. Dorner, D., & Schaub, H. (1994). Errors in planning and decision-making and the nature of human
information processing. Applied Psychology: An International Review, 43, 433–453.
15. Drazin, R., Glynn, M. A., & Kazarjain, R. K. (1999). Multi-level theorizing about creativity in
organizations. Academy of Management Review, 24, 286–329.
16. Eisenhardt, K. M. (1989). Making fast strategic decision in high-velocity environments. Academy ofManagement, 32(3), 543–576.
17. Emison, G. A. (2006). The complex challenges of ethical choices by engineers in public service.
Science and Engineering Ethics, 12, 233–244.
18. Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. AmericanPsychologist, 49(8), 725–747.
19. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–
251.
20. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA:
The MIT press.
21. Fidler, E. J. (1983). The reliability and validity of concurrent, retrospective, and interpretative verbal
reports: An experimental study. In P. H. O. Svenson & A. Vari (Eds.), Analysing and aiding decisionprocesses (pp. 429–440). Amsterdam: North-Holland.
22. Fisher, C. D., & Ashkanasy, N. M. (2000). The emerging role of emotions in work life: An intro-
duction. Journal of Organizational Behavior, 21, 123–129.
23. Fonteyn, M. E., Kuipers, B., & Grobe, S. J. (1993). A description of Think Aloud method and
protocol analysis. Qualitative Health Research, 3(4), 430–441.
24. Gary, M. S., & Wood, R. E. (2007). Mental models, decision making and performance in complex,
dynamic environments. Social Science Research Network (SSRN). Working Paper Series.
25. George, J. M. (2000). Emotions and leadership: The role of emotional intelligence. Human Relations,53, 1027–1041.
26. Glaser, R., & Bassok, M. (1989). Learning theory and the study of instruction. Annual Review ofPsychology, 40, 631–666.
27. Goldsmith, T. E., & Davenport, D. M. (1990). Assessing structural similarity of graphs. In R. W.
Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 75–87).
Westport, CT: Ablex Publishing.
28. Goldstone, R. L., & Sakamoto, Y. (2003). The transfer of abstract principles governing complex
adaptive systems. Cognitive Psychology, 46, 414–466.
29. Gott, S. P. (1989). Apprenticeship instruction for real-world tasks: The coordination of procedures,
mental models, and strategies. Review of Research in Education, 15, 97–169.
30. Gross, J. J. (1998). Antecedent- and response-focused emotion regulation: Divergent consequences
for experience, expression, and physiology. Journal of Personality and Social Psychology, 74(1),
224–237.
470 M. E. Brock et al.
123
31. Helton-Fauth, W. B., Gaddis, B., Scott, G., Mumford, M. D., Devenport, L. D., Connelly, S., et al.
(2003). A new approach to assessing ethical conduct in scientific work. Accountability in Research,10, 205–228.
32. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a
complex system from the perspective of structures, behaviors and functions. Cognitive Science, 28,
127–138.
33. Howard, P. J., & Howard, J. M. (2001). The owner’s manual for personality at work. Austin, TX:
Bard Press.
34. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between
experts and novices. Complexity, 6(2), 1–9.
35. Johnson-Laird, P. N. (1983). Mental models: Toward a cognitive science of language, inference, andconsciousness. Cambridge, MA: Harvard University Press.
36. Johnson-Laird, P. N. (2004). The history of mental models. In K. Manktelow & M. C. Chung (Eds.),
Psychology of reasoning: Theoretical and historical perspectives (pp. 179–212). Hove, Sussex:
Psychology Press.
37. Jonassen, D. H. (1997). Instructional design model for well-structured and ill-structured problem-
solving learning outcomes. Educational Technology: Research and Development, 45(1), 65–95.
38. Jones, N. L. (2007). A code of ethics for the life sciences. Science and Engineering Ethics, 13, 25–43.
39. Kazier, C., & Shore, B. M. (1995). Strategy flexibility in more and less competent students on
mathematical work problems. Creativity Research Journal, 8, 77–82.
40. Kennedy-Moore, E., & Watson, J. C. (2001). How and when does emotional expression help? Reviewof General Psychology, 5(3), 187–212.
41. Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and developmenthandbook: A guide to human resource development (2nd ed., pp. 18–27). New York: McGraw-Hill.
42. Kirkpatrick, D. L. (1996). Great ideas revisited. Training and Development Journal, 50, 54–59.
43. Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking: Alternative per-
spectives. Available at http://doi.ieeecomputersociety.org/10.1109/MIS.2006.75.
44. Kligyte, V., Marcy, R. T., Sevier, S. T., Godfrey, E. S., & Mumford, M. D. (2008). A qualitative
approach to Responsible Conduct of Research (RCR) training development: Identification of meta-
cognitive strategies. Science and Engineering Ethics, 14(1), 3–31.
45. Kligyte, V., Marcy, R. T., Waples, E., Sevier, S. T., Godfrey, E. S., & Mumford, M. D. (2007).
Application of a sensemaking approach to ethics training in the physical sciences and engineering.
Science and Engineering Ethics. doi: 10.1007/s11948-007-9048-z.
46. Kohlberg, L. (1984). Essays on moral development: The psychology of moral development: Thenature and validity of moral stages. San Francisco: Harper & Row.
47. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective
theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology,78, 311–328.
48. Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in
solving physics problems. Cognitive Science, 12, 101–138.
49. Larsen, R. J. (2000). Toward a science of mood regulation. Psychological Inquiry, 11, 129–141.
50. Mason-Mason, S. D., & Tessmer, M. A. (2000). Expert systems as a mind tool to facilitate mental
model learning. Educational Technology Research and Development, 48(4), 43–62.
51. Minkes, A. L., Small, M. W., & Chatterjee, S. R. (1999). Leadership and business ethics: Does it
matter? Implications for management. Journal of Business Ethics, 20(4), 327–335.
52. Mumford, M. D. (2006). Pathways to outstanding leadership: A comparative analysis of charismatic,ideological, and pragmatic leaders. Mahweh, New Jersey: Erlbaum.
53. Mumford, M. D., Connelly, S., Brown, R. P., Murphy, S. T., Hill, J. H., Antes, A. A., Waples, E. P.,
& Devenport, L. D. (2008). A sensemaking approach to ethics training for scientists: Preliminary
evidence of training effectiveness. Ethics and Behavior, 18(2).
54. Mumford, M. D., Devenport, L. D., Brown, R. P., Connelly, M. S., Murphy, S. T., Hill, J. H., et al.
(2006). Validation of ethical decision-making measures: Internal and external validity. Ethics andBehavior, 16(4), 319–345.
55. Plemmons, D. K., Brody, S. A., & Kalichman, M. W. (2006). Student perceptions of the effectiveness
of education in the responsible conduct of research. Science and Engineering Ethics, 12, 571–582.
56. Pinkus, R. L. B., Shuman, L. J., Hummon, N. P., & Wolf, H. (1997). Engineering ethics, balancingcost, schedule and risk—lessons learned from the space shuttle. Cambridge: Cambridge University
Press.
Mental Models 471
123
57. Rest, J. R. (1986). An overview of the psychology of morality. In J. R. Rest (Ed.), Moral developmentand behavior: Theory, research, and social issues (pp. 133–175). New York: Praeger.
58. Rouse, W. B., & Morris, N. M. (1986). On looking into the black box: Prospects and limits in the
search for mental models. Psychological Bulletin, 100(3), 349–363.
59. Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization.
Westport, CT: Ablex Publishing.
60. Scott, G. M., Leritz, L. E., & Mumford, M. D. (2004). The effectiveness of creativity training: A
meta-analysis. Creativity Research Journal, 16, 361–388.
61. Sims, R. R. (2000). Changing an organization’s culture under new leadership. Journal of BusinessEthics, 19, 393–403.
62. Sims, R. L., & Keon, T. L. (1999). Determinants of ethical decision-making: The relationship of
perceived organizational environment. Journal of Business Ethics, 19, 393–404.
63. Stout, R. J., Salas, E., & Kraiger, K. (1997). Role of trainee knowledge structures in aviation team
environments. International Journal of Aviation Psychology, 7(3), 235–250.
64. Tereskerz, P. M. (2003). Research accountability and financial conflicts of interest in industry:
Sponsored clinical research. Accountability in Research, 10, 137–158.
65. The National Academies Press. (2002). Integrity in scientific research: Creating an environment thatpromotes responsible conduct. Available at http://books.nap.edu/catalog.php?record_id=10430.
66. Titus, S. (2004). Descriptive study: Research integrity measures used in biomedical laboratories.
Paper presented at the 3rd Research on Research Integrity Conference, San Diego, CA.
67. Walczyk, J. J., Roper, K. S., Seemann, E., & Humphrey, A. M. (2003). Cognitive mechanisms
underlying lying to questions: Response time as a cue to deception. Applied Cognitive Psychology,17, 755–774.
68. Weick, K. (1995). Sensemaking in organizations. Beverly Hills, CA: Sage.
69. Wilson, T. D., & Gilbert, D. T. (2005). Affective forecasting: Knowing what to want. CurrentDirections in Psychological Science, 14, 131–134.
472 M. E. Brock et al.
123