Abstract: Judicial opinions often incorporate empirical claims to support their reasoning. However, for some issues, individuals perceive risks and weigh factual claims in a manner that affirms their own cultural identities. These motivated reactions threaten public acceptance of judicial rulings, and, potentially, the courts’ institutional legitimacy. One strategy for ameliorating the effects of motivated reasoning is aporetic reasoning, or explicitly acknowledging an issue’s complexity and uncertainty as well as the policymaker’s own doubt. I test the impact of this tactic with a survey experiment, examining reactions to a federal judge’s ruling on the immutability of sexual preferences. I create an “objection precondition” where subjects are presented with a vignette describing a judicial ruling with which they will presumably disagree. I then randomly vary the vignette to test the effect of aporetic reasoning. I find that at least for some subjects, aporetic reasoning meaningfully reduces negative reactions to the judge’s decision.
2
In risk assessment, law-making, or constitutional adjudication, an idealized view of
policymaking envisions reliance on the best available information, as determined through the
scientific method. Reality, of course, often falls short of this ideal. Judges, for example, might be
led astray by motivated reasoning, their preexisting attitudes towards law and legal problems
biasing their assessment of relevant empirical claims (Braman 2009). Motivated reasoning,
however, occurs not only among policymakers, but also when policy decisions are received by
the public. Once empirical or scientific claims become imbued with cultural or ideological
meanings, citizens may refuse to accept valid empirical claims, instead interpreting evidence and
weighing risks in accordance with their own worldviews (Kahan, Jenkins-Smith, and Braman
2011; Taber and Lodge 2006). In 1982, for example, 98% of nuclear energy experts rated nuclear
power plants as safe (Rothman and Lichter 1987). This high level of consensus among experts,
however, did little to hamper the growth of the antinuclear movement, which played an
important (albeit not determinative) role in the fall of the nuclear power industry (Campbell
1988; Stoler 1985). In a similar vein, a 2009 survey found that while 97.5% of scientists who
published in the field of climate science believed that human activity drove global warming, only
58% of the public agreed (Doran and Zimmerman 2009), a proportion that has almost certainly
declined in subsequent years. Clearly, for some subset of issues, motivated reasoning makes it
difficult to converge on not only a reasonable solution to a policy problem, but on whether a
problem exists in the first place.
On issues where ideological or cultural attachments bias perception and risk
assessment—a process generally referred to as motivated reasoning (e.g. Lodge and Taber
2000)—public polarization on empirical questions may not only affect how policy proposals or
decisions are received, but also threaten policymakers’ legitimacy. Importantly, individuals
3
opposed to the claim that that nuclear power plants are safe or that human activity is a cause of
global warming do not view themselves as anti-science; instead, they believe that science is on
their side and view the expert consensus as mistaken or corrupt (Kahan, Jenkins-Smith, and
Braman 2011). In this manner, motivated reasoning undermines respect for policymakers and,
potentially, institutions. The federal courts are not immune from this dynamic, as judicial
decisions on culturally polarized issues often contain a significant empirical component. When
they do, motivated reasoning may bias how these empirical claims are assessed (Kahan,
Hoffman, and Braman 2009; Kahan, Hoffman, et al. 2012). Given that federal courts rely both on
the perception of legitimacy as the basis for their power and on other actors to implement their
decisions, motivated reasoning may have a corrosive effect on the courts’ power over time.
In this study, I examine the viability of aporetic reasoning as a strategy for ameliorating
culturally and ideologically driven reactions to empirical claims. Aporia is a mode of
justification in which the decision-maker draws attention to the complexity and uncertainty of
the issue before him, admitting that he is not without doubt but must make a decision regardless
(Kahan 2011). Aporetic legal reasoning may defuse motivated reasoning by affirming the value
of opposing arguments, making it easier for an objector to consider the merits of the judge’s
argument without feeling as if he or she has betrayed their own commitments. While aporia has
been mentioned as a strategy for diffusing cultural bias in more than one analysis (e.g. Secunda
2012; Kahan 2011), its potential ameliorative effect has not been empirically tested. I test for the
impact of aporetic reasoning using experimental vignettes with an objection precondition
(Gibson, Caldeira, and Spence 2002), in which subjects are systematically assigned to a class of
vignettes with which they are expected to disagree. Specifically, I design vignettes which present
a federal ruling on whether homosexuality is an immutable characteristic, varying which
4
vignettes subjects receive (based on a brief initial assessment of their views on gay marriage) so
that all subjects encounter a ruling with which they likely disagree. At this point, subjects within
both precondition subgroups can be randomly assigned the treatment variable—exposure to legal
reasoning with aporetic characteristics—to see if doing so reduces expected disagreement. I find
that, at least for those subjects opposed to gay marriage, aporetic reasoning meaningfully
decreases disagreement with the decision and its reasoning.
Motivated Reasoning, Cultural Cognition, and Judicial Legitimacy
Like all policymakers, judges often rely on empirical claims to justify their decision-
making. Even in constitutional law, a field often dominated by competing moral and philosophic
visions, empirical claims are frequently critical to the outcome. Such claims can run from
specific factual inferences about the case at hand, as seen in the analysis of search and seizure
cases, to social or legislative facts, or broader claims about the social, political, or economic
environment relevant to legal adjudication (Borgmann 2013; Kahn 1994). The analysis of
whether violent video games lead to aggression in children (Brown v. Entertainment Merchants
Association, 2011), of whether a particular criminal sentence falls afoul of “evolving standards
of [societal] decency” (e.g. Atkins v. Virginia, 2002), or whether racial segregation generates a
sense of inferiority in African-American children (Brown v. Board of Education, 1954) are all
examples of how constitutional decision-making has incorporated empirical evidence. In the
distant past, justices could (and did) simply rely on their own intuition of social reality in making
such claims, as seen in Justice Marshall’s claim in Gibbons v. Ogden (1824) that “[a]ll America
understands, and has uniformly understood, the word 'commerce,' to comprehend navigation”
(Faigman 1991, 545). In recent decades, however, prompted by the increase in amici
5
submissions and a growing incorporation of empiricism in legal arguments, courts have
increasingly relied on outside research.
Under an idealized version of scientific and legal communication, the reception to
judicial use of science would rest on either the validity of the claim made or whether a particular
claim is being appropriately applied. In many cases this is true. For issues that have become
laden with sociopolitical meaning, however, validity is often perceived through the lens of
motivated reasoning (Lodge and Taber 2005; Taber and Lodge 2006), in which an individual’s
cognition labors to avoid dissonance with his currently held beliefs. Under this framework, the
balanced presentation of empirical arguments do not lead to a Bayesian process of updating
one’s priors, as individuals instead ignore, downplay, or reject empirical information that does
not comport with their ideological or cultural beliefs. Indeed, in some situations, the mental
effort employed to attack new and dissonant information may actually strengthen “attitudes in
ways not warranted by the evidence” (Taber and Lodge 2006, 756). Attempts to persuade
individuals skeptical of well-founded empirical claims may thus “backfire,” leaving them more
skeptical than before communication began (Nyhan and Reifler 2010).
In the domain of courts and judicial decision-making, the study of motivated reaction has
been advanced by cultural cognition (CC) theory, a variant of motivated reasoning theory in
which cultural attachments shape perception (Dake 1991; Jenkins-Smith and Smith 1994; Kahan
et al. 2007) and affect how individuals weigh risks and evaluate empirical evidence (Kahan and
Braman 2006). Under CC, (which has its roots in Douglas and Wildavsky’s cultural theory
(Douglas and Wildavsky 1983)), individuals have stable cultural attachments that generate
expectations about how social, political, and economic relations should be constructed. These
attachments are reinforced by association with like-minded people and biased perception of
6
evidence, creating a positive feedback loop. In line with broader theories of motivated reasoning,
CC drives individuals to down-weight empirical claims that challenge or threaten these
expectations, given that accepting them as true would create cognitive dissonance and threaten
one’s status within a cultural community.
CC operationalizes cultural worldviews along two axes: a “group” dimension, which
measures the degree to which individual preferences are subjugated to communal needs, and a
“grid” dimension, which assesses the degree of structure one prefers in society and politics. At
each end of these axes lies a different cultural type: communitarian for high group, hierarchical
for high grid, individualist for low group, and egalitarian for low grid (Kahan, Hoffman, and
Braman 2009, 859). Specific cultural and hybrid types (such as “egalitarian-communitarian”)
hold divergent expectations about how the world works, about the true causes of harm, and the
assessment of what risks are truly dangerous. For example, individualists believe that individuals
are largely responsible for their own success and failure, favor freedom over equality or order,
are optimistic that problems can be solved through markets and technological progress, and
worry about the risks created by excessive regulation or government interference (Kahan,
Hoffman, and Braman 2009; Swedlow 2011). Given this, we would expect individualists to
downplay, for example, the risks posed by environmental problems that require limiting
individual choice, as well as be less willing to tolerate environmental restrictions that might
impede economic growth. Survey research has repeatedly confirmed these hypotheses, finding
that individualists—in particular hierarchical-individualists—are less concerned, for example,
about the risks posed by nuclear power or global warming than egalitarian-communitarians
(Kahan, Jenkins-Smith, and Braman 2011). Similarly, egalitarian-individualists are less
7
concerned about the risks posed by the legalization of marijuana than hierarchical-
communitarians (Kahan 2013).
As a series of CC survey experiments demonstrate, empirical claims made by judges are
no less susceptible to motivated reaction than claims by other policy makers. In Scott v. Harris
(2007), the Court weighed the claim of an inmate who had been paralyzed after being run off the
road by a sheriff’s deputy during a high-speed pursuit. The Court’s ruling that the deputy’s
actions had been justified by the threat that Harris posed to the public turned largely on a video
recording of the incident. Remarking on the video in his majority opinion, Justice Scalia argued
that the plaintiff’s driving “posed a high likelihood of serious injury or death to respondent,”
making the deputy’s decision “objectively reasonable” (696). Going further, he remarked that
“no reasonable jury could believe” the plaintiff’s argument that the risks posed by his driving did
not merit being run off the road (694). Armed with the video footage, Dan Kahan, David
Hoffman, and Donald Braman tested whether survey subjects agreed with these perceptions.
They found that assessments of the risk Harris posed varied alongside the aforementioned
measures of cultural worldviews, with hierarchs being particularly likely to support the police
(Kahan, Hoffman, and Braman 2009).
In at least two other survey experiments involving law and courts—one dealing with
perceptions of whether consent was given in a case of acquaintance rape, and the other dealing
with whether protestors in a political demonstration were engaged in protected speech or
unprotected conduct—cultural worldviews had substantial explanatory power (Kahan 2009;
Kahan, Hoffman, et al. 2012). It seems reasonable that there are many more situations where CC
affects legal issues, for while the absolute number of culturally polarized empirical issues is
8
relatively small, federal appellate courts should attract a disproportionate number of such
conflicts, given that they normally hear the most difficult and complex cases.
When legal issues are polarized in this manner, deliberations over the quality of empirical
claims will degenerate, as individuals come to interpret evidence in ways that do the least
damage to their culturally-generated expectations. In Kahan’s words, once cultural polarization
takes root, taking culturally consistent positions become a “badge of loyalty,” biasing one’s
perception of evidence and risk (Kahan 2014). Importantly, the bias that CC creates does not
lead to individuals to see themselves as “anti-science,” as they instead remain confident that
“science” is on their side (Kahan, Jenkins-Smith, and Braman 2011). The egalitarian antinuclear
movement of the 1970s, for example, differed sharply with the large majority of both nuclear and
energy scientists regarding the dangers posed by nuclear power plants, but antinuclear advocates
did not see this as evidence that their opposition was unscientific (Rothman and Lichter 1982).
Instead, relying on support from a small but influential set of dissenting experts, such as
members of the Union of Concerned Scientists, they discounted the views of establishment
scientists as biased or even corrupted by money and power (Pollock, Lilie, and Vittes 1993).
Finally, and worse yet, studies indicate such biases increase, rather than decrease, with
increasing scientific knowledge (Kahan, Peters, et al. 2012). Greater education does not
ameliorate biased perception; instead, more knowledgeable individuals are better armed in
justifying their biases.
CC may therefore lead individuals to view policymakers who make pronouncements
contrary to their existing attachments as at best incompetent, and at worst corrupt or results-
driven. While this reaction remains a problem for any policy-making institution, it may be
particularly toxic for federal courts, whose power rests largely on a perception of legitimacy.
9
While negative reactions to judicial decisions are unlikely to decrease “diffuse support,” or
overall institutional legitimacy, in the short term (Nicholson and Howard 2003), they may
nevertheless weaken specific support both for the decision at hand and the sitting judges, and
more speculatively, corrode perceptions of legitimacy in the long run.1
Debiasing through Aporetic Reasoning
What is needed, then, are evidence-based strategies designed to reduce the pull of
motivated reasoning. First, it’s worth noting one tactic that is not effective: stating that the
existing scientific consensus supports one’s position and that opposition is unreasonable. In the
context of a contemporary scientific debate, this might be akin to telling a global warming
skeptic that ninety-eight percent of climate scientists agree that human activity drives climate
change. Rather than persuade, this type of justification has been shown to increase motivated
reasoning, making individuals less receptive to empirical arguments which contradict their
cultural expectations (Kahan, Peters, et al. 2012). Given that the drivers of motivated cognition
are the avoidance of cognitive dissonance and the maintenance of existing attachments, it isn’t
surprising that simply telling someone they are wrong—at least on a culturally polarizing issue—
leads them to double down on their existing views. As can be seen above in Scalia’s “no
reasonable jury could believe [otherwise]” comment in Scott v. Harris, judges do engage in this
sort of counterproductive rhetoric.
1 One need not specifically subscribe to CC theory to recognize the dangers that motivated
reasoning might pose to courts, or care about courts in particular to recognize the dangers
motivated reasoning creates in general.
10
Creating strategies for ameliorating motivated reasoning faces similar challenges to those
in the literature on factual misperceptions (e.g. Nyhan and Reifler 2010). In short, factual
misperceptions within divisive sociopolitical issues are extremely difficult to correct. One
experiment in this vein tested four communication strategies for correcting misperceptions about
vaccine safety and promoting vaccination (Nyhan et al. 2014).2 For the most part, these
strategies were ineffective, and in some cases even led to backfire effects (Lodge and Taber
2000; Nyhan and Reifler 2010), in which some subjects became more, rather than less skeptical
of vaccines following the experimental treatments.
Despite these setbacks, both the cultural cognition and misperception correction literature
have converged on one promising hypothesis: the communicator’s (i.e. a judge’s) attempt to
affirm the viewpoint of those with whom she does not agree (Kahan 2011, Nyhan and Reifler
2013). If overbearing certainty exacerbates motivated reasoning and cultural cognition, then
perhaps an admission of uncertainty can ameliorate it. In one experiment, Nyhan and Reifler
(2013) randomly assigned some of their subjects a free-writing affirmation task (specifically, to
recall an experience where they felt good about themselves). They found that subjects presented
with this treatment were less likely to hold onto factual misperceptions. However, while an
important finding on the value of affirmance, it is hard to see how this particular device (e.g. free
writing) could be meaningfully operationalized in political life.
The cultural cognition literature suggests a more feasible tactic: policymakers’ use of
aporia (Kahan 2011; Secunda 2012), a term which has its roots in the ancient Greek word for
2 These strategies included textual corrections, drawing attention to disease risks, using dramatic
narratives, and using visual displays to increase the salience and accessibility of the risks of
refusing vaccination.
11
puzzlement or uncertainty. The key features of aporetic reasoning include an explicit admission
that the issue in question is complex and lacks a simple resolution, recognition that the
policymaker is not without doubt, and conceding the strongest points of the losing side while
nevertheless justifying the decision made. In theory, this admission of doubt and complexity
should diffuse conflict between the decision-maker and listeners who oppose the decision,
showing the listener that the policy-maker takes their position seriously even though he does not
agree with it. Moreover, aporetic reasoning might show the listener that the decision-maker is not
blindly interpreting evidence according to his own cultural worldviews, potentially evoking a
similar response. By conceding that one’s own position is not obviously correct, the decision-
maker lowers the emotional stakes of the conflict, making it easier for the objector to consider
the merits of the decision-maker’s argument. As an example, Kahan notes the majority opinion
of Justice Kennedy in Kennedy v. Louisiana (2008), in which Kennedy expressed ambivalence
towards his own ruling that the Eighth Amendment forbade the execution of an individual who
raped his eight-year-old stepdaughter (Kahan 2010). Kennedy conceded, for example, that a
belief that the rape of a child merited death was a valid one which “must be acknowledged”
(435).
While prior research supports the general merits of affirmation as a debiasing approach
(Cohen 2000 and Sherman, Nelson, and Steele 2000), aporetic reasoning has not been
systematically tested. That said, two prior studies on reactions to judicial decision-making
suggest that aporetic reasoning is worth testing. In their survey experiment on the legitimacy of
the Court’s abortion rulings, Tom Tyler and Gregory Mitchell found that subject willingness to
“empower” the Court to make decisions on abortion strongly correlated with their perception of
whether the decision had been made fairly (Tyler and Mitchell 1994). This finding held true even
12
among the subpopulation of survey subjects opposed to the decision in question. More recently,
Dan Simon and Nicholas Surich found that survey subjects were less likely to find judicial
opinions acceptable when they employed one-sided or “monolithic” reasoning, and more likely
to find them acceptable when they considered both sides of an issue (Simon and Scurich 2011).
Monolithic reasoning, at least in this study, appears loosely comparable to the expressions of
certainty—as seen in Scott v Harris—that may exacerbate culturally motivated perception.
On both theoretical grounds and prior research, then, the hypothesis that aporetic
reasoning may ameliorate motivated reasoning merits systematic investigation.
Methods
Research Design
In line with many studies that examine the legitimacy of court decisions, I test this
hypothesis using a survey experiment, in which the treatment consists of manipulating a vignette
describing a legal decision (e.g. Gibson 2008; Gibson, Caldeira, and Spence 2005; Nicholson
and Howard 2003; Simon and Scurich 2011). In order to properly test the effects of aporetic
reasoning on reducing disagreement, my survey experiment must 1) present a legal issue that is
culturally or ideologically polarizing, 2) contain a controversial empirical element which is
integral to the court’s decision, 3) generate disagreement between the subject and the decision
made, and 4) vary the legal reasoning to test the impact of aporia.
For the first and second requirement, I choose the issue of gay marriage, specifically the
determination of whether homosexuals should be considered what U.S. constitutional law refers
to as a “protected class.” Laws that discriminate against such classes face greater scrutiny than
laws that do not, making this threshold determination a critical and often dispositive factor in
13
judicial review. The constitutional status of gay marriage, for example, may in part depend on
whether the federal courts view homosexuality as a protected class akin to race and gender.3
The decision to create a protected class rests on many factors, including whether the class
has experienced a history of discrimination or is politically powerless (Windsor v. United States,
2012). The factor I address here, however, rests on the question of “immutability,” or whether
the defining characteristic of the class is fixed and unchanging. Immutability is considered
relevant to protected class analysis because it seems particularly unfair, in a Rawlsian sense, to
discriminate against people based on facts established at birth and beyond their control (Hoffman
2011, 1519). Whether sexual orientation is a choice or an immutable characteristic, then, can be
an important determinant of what constitutional protections homosexuals will receive.
With this in mind, I create a vignette template that emulates a journalistic account of a
United States court of appeals decision that bases its protected class analysis on an empirical
determination of the immutability of sexual orientation. I choose this research design for several
reasons. First, a newspaper (or online journalistic account) seems the most likely manner by
which a member of the public would hear about a court decision. Second, it seems reasonable
that a journalistic account would include a short quote from the ruling itself, providing a sample
of judicial reasoning that can be manipulated. Third, using an appellate court ruling should
3 Gender is a better analogy, since sexual orientation would likely be accorded “semi-protected”
status alongside gender, i.e. Craig v. Boren (1976). In striking down the Defense Against
Marriage Act in United States v. Windsor (2013), the Supreme Court did not explicitly elevate
homosexuality to protected class status, although at least one circuit panel has inferred that in its
methods, if not its words, the Windsor decision applied intermediate scrutiny (SmithKline
Beecham v. Abbott Laboratories, 2014).
14
increase the seeming authenticity of the vignette: while some subjects would see through an
attempt to create a fake Supreme Court case, generic recreations of recently decided appellate
cases (e.g. Windsor v. United States (2012), Gill v. Office of Personnel Management (2012),
SmithKline Beecham v. Abbott Labs (2014)) are more likely to be accepted as authentic. Fourth,
while constitutional analysis per se is beyond the ken of most individuals, the question of
whether homosexuality is immutable is readily comprehensible, and appears in everyday
discussions about gay rights (absent the term “immutable, of course). Fifth, it seems highly likely
that both views on gay marriage and the immutability of sexual orientation are polarized and
divisive, with egalitarians or liberals being more likely to favor gay marriage and view
homosexuality as a fixed characteristic. Finally, the issue features an empirical claim—what
“science” has to say about sexual orientation—which remains controversial and likely to invite
motivated reasoning.
Having fulfilled my first and second criteria, I now turn to the third, or what James
Gibson, Gregory Caldeira, and Lester Spence refer to as an “objection precondition” (2002). In
essence, my research design requires disagreement in order to test whether aporetic reasoning
reduces that disagreement. Doing so requires a theory-driven deviation from initial random
assignment, as it makes little sense to test the impact of the treatment if there is no
disagreement.4 Instead, I assess (on a four-point scale) whether survey subjects support or
4 Gibson, Caldeira, and Spence (2002) similarly argue that it makes little sense to randomly sort
subjects into four categories and then discard half the data or engage in a series of cumbersome
interactions. While a deviation from classic random assignment, such a decision is necessary as
disagreement is the dependent variable one wishes to study. “A fully balanced design,” they
write, “has a great deal of potential to wreck statistical havoc, because, for those agreeing with
15
oppose gay marriage. Based on this initial response, subjects are sorted into precondition groups
where they receive a set of vignettes with which they will presumably disagree. In other words,
supporters of gay marriage read a newspaper account of a court which rules that homosexuals are
not a protected class because sexual orientation is not fixed, while opponents of gay marriage
read the converse scenario. Gibson, Caldeira, and Spence concede that this strategy might lead to
biased estimation (although they feel the subsequent random assignment generally resolves these
concerns), specifically if unmeasured variables are related to the causes of both the sorting
decision and the dependent variable (372-3). They offer a statistical method by which such bias
can be detected, which I discuss below.
After this initial sorting, subjects in both groups are randomly assigned to one of three
different versions of the vignette. The first version, the control, presents a generic defense of the
ruling. The second version, or the main treatment, includes an aporetic justification, where the
judge admits that the question is complex and the court’s decision is not obviously correct or
beyond reproach. I also include a third version, an alternate treatment, which emulates the tone
of Scalia’s opinion in Scott v. Harris (2007), including language such as “clear scientific
consensus” and “an objective reading of the evidence leads to no other conclusion.” I test the
impact of this reasoning style—which given Simon and Surich’s (2011) framework I label
“monolithic”—primarily because adding an additional treatment helps allay concern that the
aporetic treatment reduces disagreement simply because it includes a more detailed justification
(Cobb and Kuklinski 1997). We can have increased confidence that the aporetic treatment has a
real effect if it reduces negative reactions to the ruling compared to both the control and the other
the decision, the dependent variable is not measuring what it is supposed to measure” (367-68).
Moreover, this design makes more efficient use of a smaller sample.
16
treatment. Moreover, given previous findings that expressions of certainty may heighten cultural
resistance (Kahan, Peters, et al. 2012), it seems reasonable to test this additional hypothesis. The
vignette manipulations for the protected class precondition can be seen in Table 1, while those
for the unprotected class precondition can be seen in Table 2.5
[Table 1 and Table 2 about here]
Given that the manipulations are randomly assigned within the two precondition groups,
the treatment effect can be assessed through a simple comparison of means for the dependent
variable, the disagreement measure. Following standard practice, I also assess the effect of the
manipulations within a regression model which controls for other variables that might impact
disagreement.6 I estimate two models, one for each precondition group.
Data and Variables
Study subjects were drawn from a convenience sample of university students (n = 327)
and a work request on Amazon’s MTurk marketplace (n = 305). The two samples were
combined prior to precondition assignment. The Qualtrics online platform was used to generate
the survey and collect the results.7 Power analyses suggest that the model would be adequate for
detecting effect sizes of .5, but might have some difficulty detecting effect sizes smaller than .3. I
5 The common text for all vignettes can be found in the appendix.
6 As a robustness check, the models were also run using ordinal logistic regression. The model
coefficients did not vary in their sign or significance, and their effect sizes did not meaningfully
differ from the standard regression models.
7 Sixteen responses had missing data (in most cases for a single question) for one or more of the
model variables. Employing multiple imputation to address these missing responses did not lead
to noteworthy changes in means or model outcomes.
17
proffered four questions to serve as dependent variables, all of which used a six point scale
which measured agreement on whether 1) the court correctly ruled on the question of
immutability, 2) the court’s decision was in line with scientific consensus, 3) the court’s
argument was persuasive, and 4) the decision caused the subject to lose faith in the courts. Factor
analysis revealed that the first three measures had one primary dimension and formed a reliable
six point disagreement scale (alpha = .825 for the protected class subset and .797 for the
unprotected class one) to serve as the dependent variable; the fourth did not scale with the others
and was dropped from the analysis. For the independent variables, I include dummy variables for
the aporetic and monolithic treatments, with the control vignette as the reference category.
For control variables, I included dummy variables for race and gender. I also asked
subjects a series of questions designed to measure their cultural worldviews (Dake 1991; Kahan
et al. 2007); in particular, I was interested whether subjects identified with an egalitarian or a
hierarchal worldview, as this is the cultural dimension most likely to be implicated in the debate
over gay marriage. These six questions were collapsed to form a scale that I term “hierarchical”
(alpha = .717 and .770, in each model), with increasing numbers illustrating increased support
for a hierarchical worldview. As alternate measures of attitudes, I also included seven-point
measures of ideology (liberal to conservative) as well as a measure of partisanship (strong
Democrat to strong Republican). These two variables were collapsed into a scale I label
“Conservative-Republican” (alpha = .791 and .773, respectively). I also ascertained—on a five
point scale—how frequently the subjects attended church or religious services, hypothesizing
that more frequent attendance would correlate with the view that sexual orientation is not
immutable, and vice-versa.
18
Finally, I included a series of checks designed to measure whether the survey subjects
correctly understood the vignette. This is particularly important given that the broader legal
issue, protected class analysis, is relatively complex. These checks include asking whether the
decision favored homosexuals, whether the decision involved protected class analysis or
legalizing gay marriage, and a true-false assessment of an accurate statement about the
relationship between immutability and constitutional protections.
Results
First, I examine the descriptive data, seen in Table 3.
[Table 3 about here]
By almost a three to one margin, survey subjects signaled support for gay marriage and sorted
themselves into the “unprotected class” precondition. This is unsurprising, given that the survey
experiment drew in roughly equal parts from a university campus convenience sample and a
MTurk work request, two subpopulations one might expect to be more egalitarian then the
general public (Berinsky, Huber, and Lenz 2012). The precondition assignment also seems to
have created the desired disagreement, given that the mean of the dependent variable for each
precondition is higher than the midpoint of the scale. Once divided, the demographics of the two
samples break down as one would expect: those who sorted themselves into the “protected class”
precondition were more conservative, Republican, and hierarchical than subjects in the other
precondition, and also reported higher rates of church attendance. By contrast, the two samples
did not differ meaningfully along race or gender lines. Finally, subjects in the unprotected class
precondition had a more negative reaction across all three vignette types than did those in the
protected class precondition (with a 95% degree of confidence). This distinction could point to
19
the vignettes not being perfectly balanced, or result from those who favor gay marriage simply
reacting more negatively to contrary empirical claims regarding immutability than those who are
opposed.
Next, I examine the manipulation checks, as shown in Table 4.
[Table 4 about here]
The success rate for these checks is high and in line with similar studies (e.g. Gibson 2008).
Moreover, for every manipulation check, subjects who selected the correct answer were, on
average, substantially and significantly more certain of their choice than those who did not.
I now move to the heart of the study, which is to assess whether decisions using aporetic
justification generate less disagreement than the control or monolithic treatment. First I present
the mean disagreement score for each of the six experimental conditions in Figure 1.
[Figure 1 about here]
For subjects assigned to the protected class precondition, aporetic reasoning had the intended
effect, substantially reducing the mean negative reaction relative to the control and monolithic
treatment.8 For the unprotected class precondition, by contrast, the mean for the aporetic
treatment is lower than the control and monolithic treatments, but the reduction is smaller and
not statistically significant. The fact that a larger and significant effect was found in the much
smaller protected class subgroup also suggests a meaningful difference between the two groups.
It may be the case that the immutability of sexual orientation is less important to egalitarians in
their determination of gay rights than to hierarchs, as some egalitarians may believe that
8 A one-way ANOVA and Bonferroni test confirms this assertion, presenting an F-statistic of
4.25 and confirming that the mean for the aporetic condition is significantly different compared
to the other categories.
20
homosexual behavior should be protected regardless of natural proclivities. If true, this might
create greater resistance to any sort of justification—aporetic or otherwise—for a ruling denying
protected class status based on an immutability determination.
Also of interest is the finding that, at least in this study, expressions of greater certainty
did not catalyze greater disagreement relative to the control. Indeed, the monolithic vignettes had
a slightly lower mean disagreement score than the control in both preconditions, although neither
difference is statistically significant. This result suggests the ameliorative effects seen here may
result in part because the treatments contain more information and are more specific relative to
the control (justifying the initial decision to include an alternate treatment).
I now present a regression analysis of the disagreement scale, transforming the treatment
conditions into dummy variables, with the control omitted as the reference category. Doing so
allows me to more precisely estimate the treatment effect while controlling for other variables
that might impact disagreement.
[Table 5 about here]
Overall, the other model variables perform as expected. Increasing scores on the Conservative-
Republican scale lead to increasing disagreement with the protected class ruling but decreasing
disagreement with the unprotected class ruling. The egalitarian-hierarchy scale also presents
coefficients with the expected signs, although the effect is much stronger, and only statistically
significant, in the unprotected class precondition. As noted above, this difference may be
meaningful, in that egalitarians react more negatively than hierarchs to an immutability ruling
they disagree with. Church attendance has strong and predictable effects in both preconditions,
and gender has little impact in either. Race, interestingly, has little effect within the protected
class precondition, but a strong impact in the unprotected class precondition. Specifically, white
21
subjects were more likely to disagree with a claim that sexual orientation is chosen compared to
racial and ethnic minorities, even controlling for other factors. The theoretical implications of
this finding are not entirely clear.
Regression coefficients are difficult to interpret, particularly when employing scales. To
aid in interpretation, I use the CLARIFY package in Stata (Tomz, Wittenberg, and King 2001) to
simulate how changes in model variables impact disagreement with the court’s ruling. Here I
simulate changes in the statistically significant model variables in each precondition, holding all
other variables at their means or modes (for dummy variables). Figure 2 presents the percent
change in disagreement given these changes.
[Figure 2 about here]
In the protected class precondition, aporetic reasoning decreases the negative reaction
score by almost 15% relative to the other vignettes. At least in this precondition, the manner in
which a judge justifies his decision has a meaningful impact on how his arguments are received
by those inclined to disagree. A one point increase in the Conservative-Republican scale
(meaning more conservative and Republican) increases disagreement by about 4%, while a one
point increase on the church attendance scale (e.g. shifting from monthly attendance to weekly
attendance) increases disagreement by about 6%. In the unprotected class precondition,
increasing church attendance and Conservative-Republican scores by one point have similar
effect sizes, albeit (as expected) in the opposite direction. Here also the cultural scale has a
significant impact, with a one standard-deviation increase in hierarchical beliefs leading to about
a 5% decrease in disagreement. Finally, within the unprotected class precondition, a white
subject was about 13% more likely to disagree with the court’s ruling relative to subjects of other
demographic backgrounds.
22
Simulation can also be employed to test different archetypes, in order to give the reader a
sense of how individual disagreement might vary given their relevant characteristics. For
example, in the protected class model, an ideological moderate who does not attend church and
is exposed to the aporetic vignette will have a predicted disagreement score of 2.79 (with 6 being
“extremely disagree” and 1 being “extremely agree”), while a strong conservative and
Republican who attends church more than once a week and is exposed to the control vignette
will have a predicted disagreement score of 5.08. In the unprotected class model, a moderate
member of a demographic minority who is highly hierarchical and attends church on a weekly
basis will have a predicted disagreement score of 2.49, while a very liberal, highly egalitarian,
white individual who does not attend church will have a predicted disagreement score of 5.49.
I also investigated interactions between the model variables. In general, there were no
statistically significant interaction coefficients or patterns of interest in the marginal effects, with
one exception in the protected class category (where the aporia treatment had its theorized
impact). Though the initial interaction coefficient did not reach statistical significance, an
examination of marginal effects revealed that the treatment’s impact meaningfully varied
alongside the subjects’ cultural beliefs. I present this effect in Figure 3.
[Figure 3 about here]
As the figure illustrates, the effect of the aporetic treatment was most effective for individuals
who were opposed to gay marriage but did not score highly on the hierarchical cultural measure.
This finding fits well with other work on motivated reasoning, suggesting that those with the
strongest beliefs are least likely to change their minds or be impacted by debiasing techniques.
Aporetic reasoning, then, may have its greatest impact on cultural moderates or those whose
23
general beliefs and specific issue positions (e.g. the smaller number of egalitarians opposed to
gay marriage) are not in sync.
Finally, I address concerns regarding selection bias and the initial non-random
assignment, as discussed in Gibson, Caldeira, and Spence (2002). While the authors of that study
were not convinced such corrections were needed, they did offer a method to check for bias.
They suggest modeling the initial assignment decision—here support for gay marriage—and
using the residuals from that estimation as an instrument in a two-stage regression equation. I
model the decision to support gay marriage as a function of ideology, partisanship, and church
attendance, running a probit equation on the sorting decision.9 After calculating the residuals
from this probit equation, I use Stata’s treatreg command to estimate a two-stage equation. The
results of this comparison raise two points of interest. First, the rho statistic from the two-stage
equation provides information about potential bias. The negative figure for rho generated here
suggests that the OLS model coefficients underestimate the real impact of the model variables
(Gibson, Caldeira, and Spence 2002, 373-74). Second, comparing the OLS and corrected models
shows that there is no substantive difference in outcomes between the two, with the size and
valence of each model variable being roughly equal, and no meaningful changes in statistical
significance. Full comparisons can be seen in the appendix.
9 The survey question provides four options in the assignment question on gay marriage: strongly
support, somewhat support, somewhat oppose, and strongly oppose. I collapsed these choices
into a dummy variable for the precondition assignment, and use this dummy variable here. All of
the model variables were highly statistically significant.
24
Caveats and Conclusion
Cultural cognition, and indeed all forms of motivated reasoning, threatens to undermine
public convergence on the best available scientific evidence, as well as undercut the legitimacy
of political institutions that must inevitably address polarizing issues. Motivated reasoning is
rightly viewed as a type of commons problem, in which policy entrepreneurs can dramatically
raise the salience of a policy issue by imbuing it with cultural or ideological meaning,
“polluting” the information commons (Kahan, Peters, et al. 2012). In a political system
characterized by separation of powers and multiple veto points, there are many scenarios in
which transforming a low-salience empirical question into a high-salience cultural conflict might
pay substantial dividends. Given increasing ideological, partisan, and cultural polarization
among elites, this dynamic is likely to accelerate over time. Federal courts, in my view, face a
special danger, as their power rests on their legitimacy, which these dynamics might erode over
time.
Researchers who study motivated reasoning sometimes highlight the need for debiasing
techniques that might ameliorate its negative effects. In this study, I test the impact of one
suggestion from the cultural cognition literature: the adoption of aporetic reasoning, or an
explicit admission of uncertainty and complexity, often combined with explicit recognition of the
tradeoffs or even tragic dimension of policy choices. Specifically, I employ an experimental
vignette design with an objection precondition, sorting subjects into one of two groups based on
their preexisting views on gay marriage. Each group is presented with a story about a court
ruling on the immutability of sexual orientation with which they are expected to disagree. I then
vary the judicial justification for the decision reported in the story, using aporetic reasoning as
one of three manipulations. I find that in the protected class precondition, aporetic reasoning
25
substantially reduced disagreement with the court’s decision. In the unprotected class
precondition, by contrast, the aporetic treatment had a smaller effect that, while in the expected
direction, was not statistically distinguishable from the other vignette versions.
As a first step in investigating strategies for ameliorating cultural cognition and motivated
reasoning, I find these results encouraging. There seems little doubt that authoritative
pronouncements on the immutability of sexual orientation divide people along cultural and
ideological lines, and yet, for subjects in one of the two preconditions, simply using more
aporetic justifications reduced negative reactions to the ruling presented. These findings merit
further investigation. Aporia is a feasible and relatively low-cost tactic—it does not require
judges, substantively, to lie (since most, though not all, empirical issues of interest before the
appellate courts are complex and can be authentically addressed in aporetic terms) or change
their ultimate conclusion. To be sure, altering modes of justification will be no panacea, and may
not alter the mindsets of the most polarized citizens. If repeated over time and across multiple
rulings, however, a commitment to more effective justification strategies might reduce
disagreement and polarization, potentially increasing support for judicial institutions.
The study’s findings also speak to a broader concern: the importance of paying attention
to how citizens feel when policymakers wish them to reason. There is little evidence that
straightforward attempts to present valid scientific information or correct misperceptions will
succeed on their own merits. If negative reactions to empirical information are driven by a
(rational) desire to avoid cognitive dissonance, successful communication by policymakers must
defuse these cognitive and emotional defenses before meaningful give and take can occur. This
study of aporetic reasoning breaks important ground by testing the feasibility of one such
26
strategy, providing a model for future work on debiasing techniques in courts and other
policymaking domains.
Of course, these findings must be replicated, using different issue areas and perhaps
decisions by figures in different policymaking institutions. At the Supreme Court, at least, there
is no dearth of controversial and affect-laden constitutional issues with empirical foundations for
further testing. Moreover, like any survey experiment, this study has weaknesses worth noting.
First, neither the convenience sample nor the MTurk sample can be deemed a true random
sample of the population, which might affect the validity of the inferences drawn. The university
sample drew primarily on young adults, and given that age inversely correlates with support for
gay marriage, it may be that even young conservatives or hierarchs process claims about sexual
orientation differently than older individuals. MTurkers also tend to be experienced survey
takers, which raises concerns when using their responses to draw inferences about the general
population.
Other caveats on the findings are driven by more general limitations of survey
experiments (Gaines, Kuklinski, and Quirk 2007). A second caveat involves duration effects—
how long, exactly, might this reduction of disagreement last? Would it meaningfully affect the
subject’s evaluation of the judge’s decision at a later time? Perhaps the most important caveat is
one of external validity: in this study, subjects were exposed to a single message, the judge’s. In
real life, of course, individuals may receive multiple messages in response to a salient
policymaking decision, such as communications from interest groups which aim to maintain the
battle lines of polarization. There are clear incentives for organized interests to encourage
motivated reasoning on a topic they care about, both to increase the issue’s salience and grow the
ranks of their supporters. Future work might test how the inclusion of interest group messages—
27
particularly messages that seek to stoke the fires of motivated reasoning—does or does not alter
the impact of aporetic reasoning or other potential debiasing techniques.
While these caveats are serious, I believe them to be challenges to be met in future work,
rather than insuperable objects to the evidence-based study of how judges—and policymakers in
general—should justify decisions within affect-laden policy areas. Motivated reasoning is
endemic to the political process. Investigating strategies for ameliorating its worst effects, then,
must remain an important goal if our policies are to be based on the best available science.
Cases Referenced
Atkins v. Virginia, 536 U.S. 304 (2002) Brown v. Board of Education, 347 U.S. 483 (1954) Brown v. Entertainment Merchant’s Association, 564 U.S. 08-1448 (2011) Craig v. Boren, 429 U.S. 190 (1976) Gibbons v. Ogden, 22 U.S. 1 (1824) Gill v. Office of Personnel Management, 682 F.3d 1 (1st Cir. 2012) Scott v. Harris, 550 U.S. 372 (2007) Kennedy v. Louisiana, 554 U.S. 407 (2008) SmithKline Beecham v. Abbott Labs, No. 11-17357 (9th Cir. 2014) United States v. Windsor, 570 U.S. 12 (2013) United States v. Carolene Products, 304 U.S. 144 (1938) Windsor v. United States, 699 F. 3d 169 (2nd Cir. 2012)
References
Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. 2012. "Evaluating Online Labor
Markets for Experimental Research: Amazon.com's Mechanical Turk." Political Analysis
no. 20 (3):351-368.
Borgmann, Caitlin E. 2013. "Appellate Review of Social Facts in Constitutional Rights Cases."
California Law Review no. 101 (5):1185-1248.
Braman, Eileen. 2009. Law, Politics, and Perception. Charlottesville, VA: University of Virginia
Press.
28
Campbell, John L. 1988. Collapse of an Industry: Nuclear Power and the Contradictions of U.S.
Policy. Ithaca, NY: Cornell University Press.
Cobb, Michael D., and James H. Kuklinski. 1997. "Changing Minds: Political Arguments and
Political Persuasion." American Journal of Political Science no. 41 (1):88-121.
Dake, Karl. 1991. "Orienting Dispositions in the Perception of Risk: an Analysis of
Contemporary Worldviews and Cultural Biases." Journal of Cross-Cultural Psychology
no. 22 (1):61-82.
Doran, Peter T., and Maggie Kendall Zimmerman. 2009. "Examining the Scientific Consensus
on Climate Change." Eos, Transactions American Geophysical Union no. 90 (3):22-23.
Douglas, Mary, and Aaron Wildavsky. 1983. Risk and Culture: An Essay on the Selection of
Technological and Environmental Dangers. Berkeley, CA: University of California
Press.
Faigman, David L. 1991. ""Normative Constitutional Fact-Finding": Exploring the Empirical
Component of Constitutional Interpretation." University of Pennsylvania Law Review no.
139 (3):541-613.
Gaines, Brian J., James H. Kuklinski, and Paul J. Quirk. 2007. "The Logic of the Survey
Experiment Reexamined." Political Analysis no. 15 (1):1-20.
Gibson, James L. 2008. "Challenges to the Impartiality of State Supreme Courts: Legitimacy
Theory and "New Style'' Judicial Campaigns." American Political Science Review no.
102 (1):59.
Gibson, James L., Gregory A. Caldeira, and Lester Kenyatta Spence. 2002. "The Role of Theory
in Experimental Design: Experiments Without Randomization." Political Analysis no. 10
(4):362-375.
———. 2005. "Why do People Accept Public Policies They Oppose? Testing Legitimacy
Theory with a Survey-Based Experiment." Political Research Quarterly no. 58 (2):187-
201.
Hoffman, Sharona. 2011. "The Importance of Immutability in Employment Discrimination
Law." William & Mary Law Review no. 52 (5):1483-1546.
Jenkins-Smith, Hank, and Walter Smith. 1994. "Ideology, Culture, and Risk Perception." In
Politics, Policy, and Culture, edited by Dennis Coyle and Richard Ellis, 17-32. Boulder,
CO: Westview Press.
29
Kahan, Dan. "Science Literacy & Cultural Polarization: It Doesn't Happen 'Just' With Global
Warming, but it also Doesn't Happen for 'All' Risks. Why?" The Cultural Cognition
Project. June 13, 2013. http://www.culturalcognition.net/blog/2013/6/13/science-literacy-
cultural-polarization-it-doesnt-happen-just.html.
Kahan, Dan M. 2009. "Culture, Cognition, and Consent: Who Perceives What, and Why, in
Acquaintance-Rape Cases." University of Pennsylvania Law Review no. 158:729-813.
———. 2011. "Neutral Principles, Motivated Cognition, and Some Problems for Constitutional
Law." Harvard Law Review no. 125:1-77.
———. 2014. "Making Climate-Science Communication Evidence-Based—All the Way
Down." In Culture, Politics and Climate Change, edited by Deserai Crow and Maxwell
T. Boykoff. London: Routledge.
Kahan, Dan M., and Donald Braman. 2006. "Cultural Cognition and Public Policy." Yale Law &
Policy Review no. 24:147.
Kahan, Dan M., Donald Braman, John Gastil, Paul Slovic, and CK Mertz. 2007. "Culture and
Identity-Protective Cognition: Explaining the White-Male Effect in Risk Perception."
Journal of Empirical Legal Studies no. 4 (3):465-505.
Kahan, Dan M., David A. Hoffman, Donald Braman, and Danieli Evans. 2012. "They Saw a
Protest: Cognitive Illiberalism and the Speech-Conduct Distinction." Stanford Law
Review no. 64:851-906.
Kahan, Dan M., David Hoffman, and Donald Braman. 2009. "Whose Eyes Are You Going to
Believe? Scott v. Harris and the Perils of Cognitive Illiberalism." Harvard Law Review
no. 122:837-906.
Kahan, Dan M., Hank Jenkins-Smith, and Donald Braman. 2011. "Cultural Cognition of
Scientific Consensus." Journal of Risk Research no. 14 (2):147-174.
Kahan, Dan M., Ellen Peters, Maggie Wittlin, Paul Slovic, Lisa Larrimore Ouellette, Donald
Braman, and Gregory Mandel. 2012. "The Polarizing Impact of Science Literacy and
Numeracy on Perceived Climate Change Risks." Nature Climate Change no. 2 (10):732-
735.
Kahn, Ronald. 1994. The Supreme Court and Constitutional Theory, 1953-1993. Lawrence, KS:
University Press of Kansas.
30
Lodge, Milton, and Charles Taber. 2000. "Three Steps Toward a Theory of Motivated Political
Reasoning." In Elements of Reason: Cognition, Choice, and the Bounds of Rationality,
edited by Arthur Lupia, Mathew D. McCubbins and Samuel L. Popkin, 183-213. New
York, NY: Cambridge University Press.
Lodge, Milton, and Charles S. Taber. 2005. "The Automaticity of Affect for Political Leaders,
Groups, and Issues: An Experimental Test of the Hot Cognition Hypothesis." Political
Psychology no. 26 (3):455-482.
Nicholson, Stephen P., and Robert M. Howard. 2003. "Framing Support for the Supreme Court
in the Aftermath of Bush v. Gore." Journal of Politics no. 65 (3):676-695.
Nyhan, Brendan, and Jason Reifler. 2010. "When Corrections Fail: The Persistence of Political
Misperceptions." Political Behavior no. 32 (2):303-330.
Nyhan, Brendan, Jason Reifler, Sean Richey, and Gary L Freed. 2014. "Effective Messages in
Vaccine Promotion: A Randomized Trial." Pediatrics no. 133 (4):e835-e842.
Pollock, Philip H., Stuart A. Lilie, and M. Elliot Vittes. 1993. "Hard Issues, Core Values and
Vertical Constraint: The Case of Nuclear Power." British Journal of Political Science no.
23 (1):29-50.
Rothman, Stanley, and S. Robert Lichter. 1982. "The Nuclear Energy Debate: Scientists, the
Media and the Public." Public Opinion no. 5 (4):47-52.
———. 1987. "Elite Ideology and Risk Perception in Nuclear Energy Policy." The American
Political Science Review no. 81 (2):383-404.
Secunda, Paul. 2012. "Cognitive Illiberalism and Institutional Debiasing Strategies." San Diego
Law Review no. 49 (2):373-414.
Simon, Dan, and Nicholas Scurich. 2011. "Lay Judgments of Judicial Decision Making." Journal
of Empirical Legal Studies no. 8 (4):709-727.
Stoler, Peter. 1985. Decline and Fail: The Ailing Nuclear Power Industry. New York, NY:
Dodd, Mead & Company.
Swedlow, Brendon. 2011. "Cultural Surprises as Sources of Sudden, Big Policy Change." PS:
Political Science & Politics no. 44 (4):736-39.
Taber, Charles S., and Milton Lodge. 2006. "Motivated Skepticism in the Evaluation of Political
Beliefs." American Journal of Political Science no. 50 (3):755-769.
31
Tomz, Michael, Jason Wittenberg, and Gary King. "CLARIFY: Software for Interpreting and
Presenting Statistical Results, 2.0." Harvard University, Cambridge, MA, 2001.
Tyler, Tom R, and Gregory Mitchell. 1994. "Legitimacy and the Empowerment of Discretionary
Legal Authority: The United States Supreme Court and Abortion Rights." Duke Law
Journal no. 43 (4):703-815.
32
Table 1: Vignette Manipulations, Homosexuals are a Protected Class
Treatment
Control Aporetic Monolithic
In their opinion, the judges indicated that scientific evidence showed sexual orientation to be innate and fixed, rather than a choice. The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is among these factors.
In their opinion, the judges indicated that scientific evidence showed sexual orientation to be innate and fixed, rather than a choice. “Those who argue that sexual orientation is fixed or unchanging sometimes overstate their case,” said Judge Lynch, writing for the majority. “There is a great deal that science doesn't know about sexual orientation—whether, for example, genetics or the early developmental environment plays a greater role in its formation. However, looking at the evidence before us today, the argument that sexual orientation is a fixed characteristic rather than a choice seems the stronger one.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is among these factors.
In their opinion, the judges indicated that scientific evidence showed sexual orientation to be innate and fixed, rather than a choice. “The notion that sexual orientation is a choice has been repeatedly debunked,” said Judge Lynch, writing for the majority. “The scientific consensus clearly indicates that sexual orientation is fixed by birth or at a very young age, and is not subject to meaningful change. An objective reading of the evidence leads to no other conclusion.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is among these factors.
Table 1 presents the manipulations in the Protected Class precondition. All three vignettes share common text, which can be seen in the appendix. Bold-faced text highlights quoted judicial reasoning.
33
Table 2: Vignette Manipulations, Homosexuals are an Unprotected Class
Treatment
Control Aporetic Monolithic
In their opinion, the judges indicated that there was no consensus in the scientific community as to whether homosexual preferences were innate and fixed, rather than voluntary or freely chosen. The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is not among these factors.
In their opinion, the judges indicated that there was no consensus in the scientific community as to whether homosexual preferences were innate and fixed, rather than voluntary or freely chosen. “Those who believe that homosexuality is a choice often ignore evidence suggesting sexual orientation may have biological roots,” said Judge Lynch, writing for the majority. “It may be that as science advances, we will uncover clearer evidence regarding potential biological or genetic causes of homosexuality. However, looking at the evidence before us today, we cannot endorse the conclusion that homosexual orientation is an unchanging characteristic akin to race or gender.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is not among these factors.
In their opinion, the judges indicated that there was no consensus in the scientific community as to whether homosexual preferences were innate and fixed, rather than voluntary or freely chosen. “There is no clear scientific consensus that sexual orientation is fixed or unchanging,” said Judge Lynch, writing for the majority. “There is no agreement in the literature on what might ‘fix’ sexual preferences, and thus there can be no agreement as to whether such preferences change. An objective reading of the evidence leads to no other conclusion.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is not among these factors.
Table 2 presents the manipulations in the Unprotected Class precondition. All three vignettes share common text, which can be seen in the appendix. Bold-faced text highlights quoted judicial reasoning.
34
Table 3: Means and Standard Errors for Survey Variables
Protected Class (n = 161) Unprotected Class (n = 469)
Monolithic Control Aporetic Total Monolithic Control Aporetic Total
Reaction (Negative)
4.10 (.15)
4.16 (.17)
3.53 (.17)
3.95 (.09)
4.46 (.09)
4.58 (.10)
4.36 (.09)
4.46 (.05)
Conservative 4.67
(.18) 4.67 (.21)
5.02 (.15)
4.77 (.11)
3.17 (.11)
2.92 (.11)
3.08 (.10)
3.06 (.06)
Republican 4.57
(.18) 4.6
(.23) 4.76 (.19)
4.64 (.11)
3.23 (.10)
3.30 (.11)
3.23 (.09)
3.25 (.06)
Hierarchical 0.01
(.04) 0.01 (.05)
0.00 (.05)
0.01 (.03)
-0.31 (.02)
-0.33 (.02)
-0.28 (.02)
-0.30 (.01)
Male (percent) .54
(.06) .42
(.07) .45
(.07) .55
(.04) .51
(.04) .54
(.04) .47
(.04) .50
(.02) White (percent) .68
(.06) .64
(.07) .70
(.07) .67
(.04) .67
(.04) .70
(.04) .67
(.04) .68
(.02) Church Att. 3.3
(.16) 3.12 (.18)
3.04 (.18)
3.17 (.10)
1.75 (.08)
1.87 (.10)
1.81 (.08)
1.81 (.05)
Table 3 presents the means and standard errors for the model variables, for each treatment and across each precondition.
35
Table 4: Manipulation Checks
Protected Class Vignettes Option 1 Option 2 Option 3 Option 4 Favored Homosexuals 82%
1.71 (.06) 18% 2.57 (.20)
Described Ruling 8.18%
2.07 (.21) 81.13% 1.67 (.06)
6.29% 2.4 (.31)
4.40% 2.43 (.57)
Relationship 78.13% 21.88% 1.79 (.06) 2.17 (.112)
Unprotected Class Vignettes Option 1 Option 2 Option 3 Option 4 Favored Homosexuals 10.26%
2.02 (.13) 89.74% 1.61 (.03)
Described Ruling 3.23%
2.13 (.26) 3.88% 2.44 (.19)
8.62% 2.35 (.12)
84.37% 1.29 (.03)
Relationship 81.62% 18.38% 1.74 (.03) 2.01 (.08)
Table 4 presents summary data for the survey’s manipulation checks, including the frequency for each choice, the mean certainty rating for each check (ranging from “very certain” (1) to “not at all certain” (4), and the standard error of the mean in parentheses. These checks include (1) whether the decision favored homosexuals, (2) whether the ruling was on protected class analysis or the legalization of gay marriage, and in which direction, and (3) whether the subject agreed or disagreed with an accurate statement about the relationship between immutability and constitutional protections, (as described in the vignette). Bold-faced text indicates the correct option.
36
Table 5: Multivariate Analysis of Disagreement with the Court’s Ruling
Protected Class Unprotected Class
Aporetic Treatment -0.592** -0.150 (0.223) (0.116) Monolithic Treatment -0.162 -0.124 (0.209) (0.120) Conservative-Republican 0.165* -0.171*** (0.080) (0.047) Hierarchal 0.286 -0.654** (0.326) (0.203) Male 0.101 -0.0185 (0.185) (0.097) White -0.031 0.631*** (0.218) (0.107) Church Attendance 0.298*** -0.300*** (0.070) (0.049) Constant 2.462*** 5.024*** (0.434) (0.244)
N 157 459 adj. R2 0.180 0.255
*p < 0.05, **p < 0.01, ***p < 0.001.
Table 5 presents the OLS coefficients and standard errors for the models’ variables. The dependent variable is a scale variable constructed from three survey questions which assessed the degree to which the subject disagreed with the court’s ruling on the immutability of homosexuality. The control condition serves as the reference category.
37
Figure 1: Mean Disagreement with Court Ruling, By Precondition and Vignette
Figure 1 presents the means (point) and standard errors (bars) for the disagreement or negative response scale variable (ranging from “extremely agree” (1) to “extremely disagree” (6)) for each experimental condition in both the protected and unprotected class preconditions.
38
Figure 2: Predicted Disagreement, Accounting for Changes in Other Model Variables
Figure 2 presents simulation results generated using Stata’s CLARIFY package, which estimates point predictions for the dependent variable based on simulated values of the independent and control variables. Here I present the percent change in the disagreement scale based on manipulations of the statistically significant OLS model variables. “Aporetic” and “White” are dummy variables, where a value of 1 is the simulated category. “Church” illustrates the change for a one point increase on a five-point scale which ranges from “no church attendance” (1) to “more than weekly attendance” (5). “Conservative-Republican” illustrates the percent change for a one point increase on a seven-point scale, with higher scores being more conservative and more Republican. “Hierarchical” illustrates the change for a one standard deviation increase on a measure of hierarchical worldviews.
39
Figure 3: Marginal Effects of Treatment by Cultural Scale on Disagreement with the Judicial Ruling, Protected Class Precondition
Figure 3 presents the marginal effects on disagreement with the judicial decision, specifically for the interaction between the treatment and cultural beliefs for subjects assigned to the protected class precondition. Cultural beliefs were scaled from -.5 to .5, with higher scores indicating more hierarchal beliefs. The grey shaded area illustrates the 95% confidence interval.
40
Appendix
Poll Questions and Vignettes
Precondition question: The issue of gay marriage has been in the news a great deal over the past year. In general, how would you describe your own views on gay marriage? Do you strongly support it, somewhat support it, somewhat oppose it, or strongly oppose it?
1 Strongly support 2 Somewhat support 3 Somewhat oppose 4 Strongly oppose
The vignettes for the protected class precondition had the following text in common: Please carefully read the following narrative: Court Rules Same-Sex Couples are a Protected Class By John Devita August 7, 2013 NEW YORK — A federal appeals court ruled on Thursday that gay Americans deserve constitutional protections similar to those subject to racial or gender discrimination. The ruling by the Court of Appeals for the Second Circuit came as the panel assessed the legal status of gay marriage in New Jersey. The case may have important ramifications for other states which do not currently have gay marriage. Importantly, the court’s ruling rested on a finding that because homosexuality is a fixed characteristic like race or sex, homosexuality should be considered a "protected class," a legal term which indicates that a group is likely to face to unconstitutional discrimination. This means that laws that impact homosexuals as a class will have to meet higher legal standards in order to be constitutional. [Manipulations go here; see Table 1 in manuscript] It is unclear at this point whether the case will come before the Supreme Court.
41
The vignettes for the unprotected class precondition had the following text in common: Please carefully read the following narrative: Court Rules Same-Sex Couples are not a Protected Class By John Devita August 7, 2013 NEW YORK — A federal appeals court ruled on Thursday that gay Americans do not merit the same level of constitutional protections accorded to individuals subject to racial or gender discrimination. The ruling by the Court of Appeals for the Second Circuit came as the panel assessed the legal status of gay marriage in New Jersey. The ruling may have important ramifications for other states which do not currently have gay marriage. Importantly, the court’s ruling rested on a finding that because homosexuality is not a fixed characteristic like race or sex, homosexuality should not be considered a "protected class," a legal term which indicates that a group is likely to face to unconstitutional discrimination. This means that laws that impact homosexuals do not have to meet higher legal standards in order to be constitutional. [Manipulations go here; see Table 2 in manuscript] It is unclear at this point whether the case will come before the Supreme Court. The questions for constructing the disagreement scale: The court correctly found that sexual orientation is a choice/is a fixed characteristic. The court’s finding that sexual orientation is a choice/is a fixed characteristic accurately reflects scientific and expert consensus. The court’s argument about sexual orientation is persuasive. 1 Extremely agree 2 Strongly agree 3 Somewhat agree 4 Somewhat disagree 5 Strongly disagree 6 Extremely disagree The questions for constructing the hierarchy scale: We have gone too far in pushing equal rights in this country. (Hierarchy) Our society would be better off if the distribution of wealth was more equal. (Egalitarian) We need to dramatically reduce inequalities between the rich and the poor, whites and people of color, and men and women. (Egalitarian)
42
Discrimination against minorities is still a very serious problem in our society. (Egalitarian) It seems like blacks, women, homosexuals and other groups don’t want equal rights, they want special rights just for them. (Hierarchy) Society as a whole has become too soft and feminine. (Hierarchy)
1 Agree 2 Disagree
Other control variable questions:
Generally speaking, how would you describe your politics? 1 Extremely liberal 2 Liberal 3 Slightly liberal 4 Moderate or middle of the road 5 Slightly conservative 6 Conservative 7 Extremely conservative
Generally speaking, do you consider yourself a Democrat, Republican, or Independent?
1 Strong Democrat 2 Democrat 3 Lean Democrat 4 Independent 5 Lean Republican 6 Republican 7 Strong Republican
Are you male or female?
1 Male 2 Female
Which of these racial or ethnic categories best describes you?
1 White 2 Black or African-American 3 Hispanic, Latino, or of Spanish Origin 4 Other
How often do you attend church or religious services?
1 Never 2 A few times a year 3 Monthly 4 Weekly 5 More than once a week
43
Manipulation checks: Now I would like to ask you some questions about what you can remember from the narrative you read. Who won the case, the side supporting gay marriage, or the side opposing it?
1 The side supporting gay marriage 2 The side opposing gay marriage
In the narrative that you read, which of the following statements accurately describes the judge’s ruling: a U.S. Court of Appeals ruled gay marriage constitutional; a U.S. Court of Appeals ruled homosexuals were a protected class; a U.S. Court of Appeals ruled homosexuals were not a protected class; a U.S. Court of Appeals ruled gay marriage unconstitutional.
1 a U.S. Court of Appeals ruled gay marriage constitutional 2 a U.S. Court of Appeals ruled homosexuals were a protected class 3 a U.S. Court of Appeals ruled gay marriage unconstitutional 4 a U.S. Court of Appeals ruled homosexuals were not a protected class
Please indicate whether you think the following statement is true: “if the court finds that sexual orientation is a choice, homosexuals may receive less constitutional protections”/ “if the court finds that sexual orientation is a fixed characteristic, homosexuals may be entitled to greater constitutional protections.”
1 Accurately describes (correct) 2 Does not describe (incorrect)
[For each check] How certain are you of this?
1 Very certain 2 Somewhat certain 3 Not very certain 4 Not at all certain
44
Test for Selection Bias
To test for selection bias which might result from the initial non-random assignment,
Gibson, Caldeira, and Spence (2002) suggest estimating a model for the selection variable (here
whether or not the subject supports gay marriage), using the residuals from this estimation as an
instrument in a two-stage regression. I estimate the assignment variable using a probit model
with the survey measures of ideology, partisanship, and church attendance (all variables are
significant at p < .01). I then calculate the model residuals, and use them in a two-step
regression, specifically using Stata’s treatreg function.
In the table below, I present the uncorrected OLS estimates for the combined sample side
by side with the estimates generated from the two-stage model. As the table shows, the
correction did not meaningfully affect the size of the model coefficients, and did not alter the
significance or valence of any model variable. The negative value for rho suggests that the OLS
model underestimates the true effect of the coefficients.
45
Comparison of Original and Two-stage Regression Models
Original Regression Two-stage Regression
Aporetic Treatment -.265* -.314** (.111) (.106) Monolithic Treatment -.094 -.096 (.111) (.106) Conservative-Republican -.109** -.118** (.04) (.04) Hierarchal .590** .419* (.172) (.170) Male .025 .015 (.093) (.088) White .691*** .588*** (.101) (.098) Church Attendance -.091* -.090* (.04) (.04) Constant 4.246*** 3.625*** (.251) (.283) Rho -.677
N 619 619
Top Related