Abstract: Judicial opinions often incorporate empirical claims to support their reasoning. However, for some issues, individuals perceive risks and weigh factual claims in a manner that affirms their own cultural identities. These motivated reactions threaten public acceptance of judicial rulings, and, potentially, the courts’ institutional legitimacy. One strategy for ameliorating the effects of motivated reasoning is aporetic reasoning, or explicitly acknowledging an issue’s complexity and uncertainty as well as the policymaker’s own doubt. I test the impact of this tactic with a survey experiment, examining reactions to a federal judge’s ruling on the immutability of sexual preferences. I create an “objection precondition” where subjects are presented with a vignette describing a judicial ruling with which they will presumably disagree. I then randomly vary the vignette to test the effect of aporetic reasoning. I find that at least for some subjects, aporetic reasoning meaningfully reduces negative reactions to the judge’s decision.

2

In risk assessment, law-making, or constitutional adjudication, an idealized view of

policymaking envisions reliance on the best available information, as determined through the

scientific method. Reality, of course, often falls short of this ideal. Judges, for example, might be

led astray by motivated reasoning, their preexisting attitudes towards law and legal problems

biasing their assessment of relevant empirical claims (Braman 2009). Motivated reasoning,

however, occurs not only among policymakers, but also when policy decisions are received by

the public. Once empirical or scientific claims become imbued with cultural or ideological

meanings, citizens may refuse to accept valid empirical claims, instead interpreting evidence and

weighing risks in accordance with their own worldviews (Kahan, Jenkins-Smith, and Braman

2011; Taber and Lodge 2006). In 1982, for example, 98% of nuclear energy experts rated nuclear

power plants as safe (Rothman and Lichter 1987). This high level of consensus among experts,

however, did little to hamper the growth of the antinuclear movement, which played an

important (albeit not determinative) role in the fall of the nuclear power industry (Campbell

1988; Stoler 1985). In a similar vein, a 2009 survey found that while 97.5% of scientists who

published in the field of climate science believed that human activity drove global warming, only

58% of the public agreed (Doran and Zimmerman 2009), a proportion that has almost certainly

declined in subsequent years. Clearly, for some subset of issues, motivated reasoning makes it

difficult to converge on not only a reasonable solution to a policy problem, but on whether a

problem exists in the first place.

On issues where ideological or cultural attachments bias perception and risk

assessment—a process generally referred to as motivated reasoning (e.g. Lodge and Taber

2000)—public polarization on empirical questions may not only affect how policy proposals or

decisions are received, but also threaten policymakers’ legitimacy. Importantly, individuals

3

opposed to the claim that that nuclear power plants are safe or that human activity is a cause of

global warming do not view themselves as anti-science; instead, they believe that science is on

their side and view the expert consensus as mistaken or corrupt (Kahan, Jenkins-Smith, and

Braman 2011). In this manner, motivated reasoning undermines respect for policymakers and,

potentially, institutions. The federal courts are not immune from this dynamic, as judicial

decisions on culturally polarized issues often contain a significant empirical component. When

they do, motivated reasoning may bias how these empirical claims are assessed (Kahan,

Hoffman, and Braman 2009; Kahan, Hoffman, et al. 2012). Given that federal courts rely both on

the perception of legitimacy as the basis for their power and on other actors to implement their

decisions, motivated reasoning may have a corrosive effect on the courts’ power over time.

In this study, I examine the viability of aporetic reasoning as a strategy for ameliorating

culturally and ideologically driven reactions to empirical claims. Aporia is a mode of

justification in which the decision-maker draws attention to the complexity and uncertainty of

the issue before him, admitting that he is not without doubt but must make a decision regardless

(Kahan 2011). Aporetic legal reasoning may defuse motivated reasoning by affirming the value

of opposing arguments, making it easier for an objector to consider the merits of the judge’s

argument without feeling as if he or she has betrayed their own commitments. While aporia has

been mentioned as a strategy for diffusing cultural bias in more than one analysis (e.g. Secunda

2012; Kahan 2011), its potential ameliorative effect has not been empirically tested. I test for the

impact of aporetic reasoning using experimental vignettes with an objection precondition

(Gibson, Caldeira, and Spence 2002), in which subjects are systematically assigned to a class of

vignettes with which they are expected to disagree. Specifically, I design vignettes which present

a federal ruling on whether homosexuality is an immutable characteristic, varying which

4

vignettes subjects receive (based on a brief initial assessment of their views on gay marriage) so

that all subjects encounter a ruling with which they likely disagree. At this point, subjects within

both precondition subgroups can be randomly assigned the treatment variable—exposure to legal

reasoning with aporetic characteristics—to see if doing so reduces expected disagreement. I find

that, at least for those subjects opposed to gay marriage, aporetic reasoning meaningfully

decreases disagreement with the decision and its reasoning.

Motivated Reasoning, Cultural Cognition, and Judicial Legitimacy

Like all policymakers, judges often rely on empirical claims to justify their decision-

making. Even in constitutional law, a field often dominated by competing moral and philosophic

visions, empirical claims are frequently critical to the outcome. Such claims can run from

specific factual inferences about the case at hand, as seen in the analysis of search and seizure

cases, to social or legislative facts, or broader claims about the social, political, or economic

environment relevant to legal adjudication (Borgmann 2013; Kahn 1994). The analysis of

whether violent video games lead to aggression in children (Brown v. Entertainment Merchants

Association, 2011), of whether a particular criminal sentence falls afoul of “evolving standards

of [societal] decency” (e.g. Atkins v. Virginia, 2002), or whether racial segregation generates a

sense of inferiority in African-American children (Brown v. Board of Education, 1954) are all

examples of how constitutional decision-making has incorporated empirical evidence. In the

distant past, justices could (and did) simply rely on their own intuition of social reality in making

such claims, as seen in Justice Marshall’s claim in Gibbons v. Ogden (1824) that “[a]ll America

understands, and has uniformly understood, the word 'commerce,' to comprehend navigation”

(Faigman 1991, 545). In recent decades, however, prompted by the increase in amici

5

submissions and a growing incorporation of empiricism in legal arguments, courts have

increasingly relied on outside research.

Under an idealized version of scientific and legal communication, the reception to

judicial use of science would rest on either the validity of the claim made or whether a particular

claim is being appropriately applied. In many cases this is true. For issues that have become

laden with sociopolitical meaning, however, validity is often perceived through the lens of

motivated reasoning (Lodge and Taber 2005; Taber and Lodge 2006), in which an individual’s

cognition labors to avoid dissonance with his currently held beliefs. Under this framework, the

balanced presentation of empirical arguments do not lead to a Bayesian process of updating

one’s priors, as individuals instead ignore, downplay, or reject empirical information that does

not comport with their ideological or cultural beliefs. Indeed, in some situations, the mental

effort employed to attack new and dissonant information may actually strengthen “attitudes in

ways not warranted by the evidence” (Taber and Lodge 2006, 756). Attempts to persuade

individuals skeptical of well-founded empirical claims may thus “backfire,” leaving them more

skeptical than before communication began (Nyhan and Reifler 2010).

In the domain of courts and judicial decision-making, the study of motivated reaction has

been advanced by cultural cognition (CC) theory, a variant of motivated reasoning theory in

which cultural attachments shape perception (Dake 1991; Jenkins-Smith and Smith 1994; Kahan

et al. 2007) and affect how individuals weigh risks and evaluate empirical evidence (Kahan and

Braman 2006). Under CC, (which has its roots in Douglas and Wildavsky’s cultural theory

(Douglas and Wildavsky 1983)), individuals have stable cultural attachments that generate

expectations about how social, political, and economic relations should be constructed. These

attachments are reinforced by association with like-minded people and biased perception of

6

evidence, creating a positive feedback loop. In line with broader theories of motivated reasoning,

CC drives individuals to down-weight empirical claims that challenge or threaten these

expectations, given that accepting them as true would create cognitive dissonance and threaten

one’s status within a cultural community.

CC operationalizes cultural worldviews along two axes: a “group” dimension, which

measures the degree to which individual preferences are subjugated to communal needs, and a

“grid” dimension, which assesses the degree of structure one prefers in society and politics. At

each end of these axes lies a different cultural type: communitarian for high group, hierarchical

for high grid, individualist for low group, and egalitarian for low grid (Kahan, Hoffman, and

Braman 2009, 859). Specific cultural and hybrid types (such as “egalitarian-communitarian”)

hold divergent expectations about how the world works, about the true causes of harm, and the

assessment of what risks are truly dangerous. For example, individualists believe that individuals

are largely responsible for their own success and failure, favor freedom over equality or order,

are optimistic that problems can be solved through markets and technological progress, and

worry about the risks created by excessive regulation or government interference (Kahan,

Hoffman, and Braman 2009; Swedlow 2011). Given this, we would expect individualists to

downplay, for example, the risks posed by environmental problems that require limiting

individual choice, as well as be less willing to tolerate environmental restrictions that might

impede economic growth. Survey research has repeatedly confirmed these hypotheses, finding

that individualists—in particular hierarchical-individualists—are less concerned, for example,

about the risks posed by nuclear power or global warming than egalitarian-communitarians

(Kahan, Jenkins-Smith, and Braman 2011). Similarly, egalitarian-individualists are less

7

concerned about the risks posed by the legalization of marijuana than hierarchical-

communitarians (Kahan 2013).

As a series of CC survey experiments demonstrate, empirical claims made by judges are

no less susceptible to motivated reaction than claims by other policy makers. In Scott v. Harris

(2007), the Court weighed the claim of an inmate who had been paralyzed after being run off the

road by a sheriff’s deputy during a high-speed pursuit. The Court’s ruling that the deputy’s

actions had been justified by the threat that Harris posed to the public turned largely on a video

recording of the incident. Remarking on the video in his majority opinion, Justice Scalia argued

that the plaintiff’s driving “posed a high likelihood of serious injury or death to respondent,”

making the deputy’s decision “objectively reasonable” (696). Going further, he remarked that

“no reasonable jury could believe” the plaintiff’s argument that the risks posed by his driving did

not merit being run off the road (694). Armed with the video footage, Dan Kahan, David

Hoffman, and Donald Braman tested whether survey subjects agreed with these perceptions.

They found that assessments of the risk Harris posed varied alongside the aforementioned

measures of cultural worldviews, with hierarchs being particularly likely to support the police

(Kahan, Hoffman, and Braman 2009).

In at least two other survey experiments involving law and courts—one dealing with

perceptions of whether consent was given in a case of acquaintance rape, and the other dealing

with whether protestors in a political demonstration were engaged in protected speech or

unprotected conduct—cultural worldviews had substantial explanatory power (Kahan 2009;

Kahan, Hoffman, et al. 2012). It seems reasonable that there are many more situations where CC

affects legal issues, for while the absolute number of culturally polarized empirical issues is

8

relatively small, federal appellate courts should attract a disproportionate number of such

conflicts, given that they normally hear the most difficult and complex cases.

When legal issues are polarized in this manner, deliberations over the quality of empirical

claims will degenerate, as individuals come to interpret evidence in ways that do the least

damage to their culturally-generated expectations. In Kahan’s words, once cultural polarization

takes root, taking culturally consistent positions become a “badge of loyalty,” biasing one’s

perception of evidence and risk (Kahan 2014). Importantly, the bias that CC creates does not

lead to individuals to see themselves as “anti-science,” as they instead remain confident that

“science” is on their side (Kahan, Jenkins-Smith, and Braman 2011). The egalitarian antinuclear

movement of the 1970s, for example, differed sharply with the large majority of both nuclear and

energy scientists regarding the dangers posed by nuclear power plants, but antinuclear advocates

did not see this as evidence that their opposition was unscientific (Rothman and Lichter 1982).

Instead, relying on support from a small but influential set of dissenting experts, such as

members of the Union of Concerned Scientists, they discounted the views of establishment

scientists as biased or even corrupted by money and power (Pollock, Lilie, and Vittes 1993).

Finally, and worse yet, studies indicate such biases increase, rather than decrease, with

increasing scientific knowledge (Kahan, Peters, et al. 2012). Greater education does not

ameliorate biased perception; instead, more knowledgeable individuals are better armed in

justifying their biases.

CC may therefore lead individuals to view policymakers who make pronouncements

contrary to their existing attachments as at best incompetent, and at worst corrupt or results-

driven. While this reaction remains a problem for any policy-making institution, it may be

particularly toxic for federal courts, whose power rests largely on a perception of legitimacy.

9

While negative reactions to judicial decisions are unlikely to decrease “diffuse support,” or

overall institutional legitimacy, in the short term (Nicholson and Howard 2003), they may

nevertheless weaken specific support both for the decision at hand and the sitting judges, and

more speculatively, corrode perceptions of legitimacy in the long run.1

Debiasing through Aporetic Reasoning

What is needed, then, are evidence-based strategies designed to reduce the pull of

motivated reasoning. First, it’s worth noting one tactic that is not effective: stating that the

existing scientific consensus supports one’s position and that opposition is unreasonable. In the

context of a contemporary scientific debate, this might be akin to telling a global warming

skeptic that ninety-eight percent of climate scientists agree that human activity drives climate

change. Rather than persuade, this type of justification has been shown to increase motivated

reasoning, making individuals less receptive to empirical arguments which contradict their

cultural expectations (Kahan, Peters, et al. 2012). Given that the drivers of motivated cognition

are the avoidance of cognitive dissonance and the maintenance of existing attachments, it isn’t

surprising that simply telling someone they are wrong—at least on a culturally polarizing issue—

leads them to double down on their existing views. As can be seen above in Scalia’s “no

reasonable jury could believe [otherwise]” comment in Scott v. Harris, judges do engage in this

sort of counterproductive rhetoric.

1 One need not specifically subscribe to CC theory to recognize the dangers that motivated

reasoning might pose to courts, or care about courts in particular to recognize the dangers

motivated reasoning creates in general.

10

Creating strategies for ameliorating motivated reasoning faces similar challenges to those

in the literature on factual misperceptions (e.g. Nyhan and Reifler 2010). In short, factual

misperceptions within divisive sociopolitical issues are extremely difficult to correct. One

experiment in this vein tested four communication strategies for correcting misperceptions about

vaccine safety and promoting vaccination (Nyhan et al. 2014).2 For the most part, these

strategies were ineffective, and in some cases even led to backfire effects (Lodge and Taber

2000; Nyhan and Reifler 2010), in which some subjects became more, rather than less skeptical

of vaccines following the experimental treatments.

Despite these setbacks, both the cultural cognition and misperception correction literature

have converged on one promising hypothesis: the communicator’s (i.e. a judge’s) attempt to

affirm the viewpoint of those with whom she does not agree (Kahan 2011, Nyhan and Reifler

2013). If overbearing certainty exacerbates motivated reasoning and cultural cognition, then

perhaps an admission of uncertainty can ameliorate it. In one experiment, Nyhan and Reifler

(2013) randomly assigned some of their subjects a free-writing affirmation task (specifically, to

recall an experience where they felt good about themselves). They found that subjects presented

with this treatment were less likely to hold onto factual misperceptions. However, while an

important finding on the value of affirmance, it is hard to see how this particular device (e.g. free

writing) could be meaningfully operationalized in political life.

The cultural cognition literature suggests a more feasible tactic: policymakers’ use of

aporia (Kahan 2011; Secunda 2012), a term which has its roots in the ancient Greek word for

2 These strategies included textual corrections, drawing attention to disease risks, using dramatic

narratives, and using visual displays to increase the salience and accessibility of the risks of

refusing vaccination.

11

puzzlement or uncertainty. The key features of aporetic reasoning include an explicit admission

that the issue in question is complex and lacks a simple resolution, recognition that the

policymaker is not without doubt, and conceding the strongest points of the losing side while

nevertheless justifying the decision made. In theory, this admission of doubt and complexity

should diffuse conflict between the decision-maker and listeners who oppose the decision,

showing the listener that the policy-maker takes their position seriously even though he does not

agree with it. Moreover, aporetic reasoning might show the listener that the decision-maker is not

blindly interpreting evidence according to his own cultural worldviews, potentially evoking a

similar response. By conceding that one’s own position is not obviously correct, the decision-

maker lowers the emotional stakes of the conflict, making it easier for the objector to consider

the merits of the decision-maker’s argument. As an example, Kahan notes the majority opinion

of Justice Kennedy in Kennedy v. Louisiana (2008), in which Kennedy expressed ambivalence

towards his own ruling that the Eighth Amendment forbade the execution of an individual who

raped his eight-year-old stepdaughter (Kahan 2010). Kennedy conceded, for example, that a

belief that the rape of a child merited death was a valid one which “must be acknowledged”

(435).

While prior research supports the general merits of affirmation as a debiasing approach

(Cohen 2000 and Sherman, Nelson, and Steele 2000), aporetic reasoning has not been

systematically tested. That said, two prior studies on reactions to judicial decision-making

suggest that aporetic reasoning is worth testing. In their survey experiment on the legitimacy of

the Court’s abortion rulings, Tom Tyler and Gregory Mitchell found that subject willingness to

“empower” the Court to make decisions on abortion strongly correlated with their perception of

whether the decision had been made fairly (Tyler and Mitchell 1994). This finding held true even

12

among the subpopulation of survey subjects opposed to the decision in question. More recently,

Dan Simon and Nicholas Surich found that survey subjects were less likely to find judicial

opinions acceptable when they employed one-sided or “monolithic” reasoning, and more likely

to find them acceptable when they considered both sides of an issue (Simon and Scurich 2011).

Monolithic reasoning, at least in this study, appears loosely comparable to the expressions of

certainty—as seen in Scott v Harris—that may exacerbate culturally motivated perception.

On both theoretical grounds and prior research, then, the hypothesis that aporetic

reasoning may ameliorate motivated reasoning merits systematic investigation.

Methods

Research Design

In line with many studies that examine the legitimacy of court decisions, I test this

hypothesis using a survey experiment, in which the treatment consists of manipulating a vignette

describing a legal decision (e.g. Gibson 2008; Gibson, Caldeira, and Spence 2005; Nicholson

and Howard 2003; Simon and Scurich 2011). In order to properly test the effects of aporetic

reasoning on reducing disagreement, my survey experiment must 1) present a legal issue that is

culturally or ideologically polarizing, 2) contain a controversial empirical element which is

integral to the court’s decision, 3) generate disagreement between the subject and the decision

made, and 4) vary the legal reasoning to test the impact of aporia.

For the first and second requirement, I choose the issue of gay marriage, specifically the

determination of whether homosexuals should be considered what U.S. constitutional law refers

to as a “protected class.” Laws that discriminate against such classes face greater scrutiny than

laws that do not, making this threshold determination a critical and often dispositive factor in

13

judicial review. The constitutional status of gay marriage, for example, may in part depend on

whether the federal courts view homosexuality as a protected class akin to race and gender.3

The decision to create a protected class rests on many factors, including whether the class

has experienced a history of discrimination or is politically powerless (Windsor v. United States,

2012). The factor I address here, however, rests on the question of “immutability,” or whether

the defining characteristic of the class is fixed and unchanging. Immutability is considered

relevant to protected class analysis because it seems particularly unfair, in a Rawlsian sense, to

discriminate against people based on facts established at birth and beyond their control (Hoffman

2011, 1519). Whether sexual orientation is a choice or an immutable characteristic, then, can be

an important determinant of what constitutional protections homosexuals will receive.

With this in mind, I create a vignette template that emulates a journalistic account of a

United States court of appeals decision that bases its protected class analysis on an empirical

determination of the immutability of sexual orientation. I choose this research design for several

reasons. First, a newspaper (or online journalistic account) seems the most likely manner by

which a member of the public would hear about a court decision. Second, it seems reasonable

that a journalistic account would include a short quote from the ruling itself, providing a sample

of judicial reasoning that can be manipulated. Third, using an appellate court ruling should

3 Gender is a better analogy, since sexual orientation would likely be accorded “semi-protected”

status alongside gender, i.e. Craig v. Boren (1976). In striking down the Defense Against

Marriage Act in United States v. Windsor (2013), the Supreme Court did not explicitly elevate

homosexuality to protected class status, although at least one circuit panel has inferred that in its

methods, if not its words, the Windsor decision applied intermediate scrutiny (SmithKline

Beecham v. Abbott Laboratories, 2014).

14

increase the seeming authenticity of the vignette: while some subjects would see through an

attempt to create a fake Supreme Court case, generic recreations of recently decided appellate

cases (e.g. Windsor v. United States (2012), Gill v. Office of Personnel Management (2012),

SmithKline Beecham v. Abbott Labs (2014)) are more likely to be accepted as authentic. Fourth,

while constitutional analysis per se is beyond the ken of most individuals, the question of

whether homosexuality is immutable is readily comprehensible, and appears in everyday

discussions about gay rights (absent the term “immutable, of course). Fifth, it seems highly likely

that both views on gay marriage and the immutability of sexual orientation are polarized and

divisive, with egalitarians or liberals being more likely to favor gay marriage and view

homosexuality as a fixed characteristic. Finally, the issue features an empirical claim—what

“science” has to say about sexual orientation—which remains controversial and likely to invite

motivated reasoning.

Having fulfilled my first and second criteria, I now turn to the third, or what James

Gibson, Gregory Caldeira, and Lester Spence refer to as an “objection precondition” (2002). In

essence, my research design requires disagreement in order to test whether aporetic reasoning

reduces that disagreement. Doing so requires a theory-driven deviation from initial random

assignment, as it makes little sense to test the impact of the treatment if there is no

disagreement.4 Instead, I assess (on a four-point scale) whether survey subjects support or

4 Gibson, Caldeira, and Spence (2002) similarly argue that it makes little sense to randomly sort

subjects into four categories and then discard half the data or engage in a series of cumbersome

interactions. While a deviation from classic random assignment, such a decision is necessary as

disagreement is the dependent variable one wishes to study. “A fully balanced design,” they

write, “has a great deal of potential to wreck statistical havoc, because, for those agreeing with

15

oppose gay marriage. Based on this initial response, subjects are sorted into precondition groups

where they receive a set of vignettes with which they will presumably disagree. In other words,

supporters of gay marriage read a newspaper account of a court which rules that homosexuals are

not a protected class because sexual orientation is not fixed, while opponents of gay marriage

read the converse scenario. Gibson, Caldeira, and Spence concede that this strategy might lead to

biased estimation (although they feel the subsequent random assignment generally resolves these

concerns), specifically if unmeasured variables are related to the causes of both the sorting

decision and the dependent variable (372-3). They offer a statistical method by which such bias

can be detected, which I discuss below.

After this initial sorting, subjects in both groups are randomly assigned to one of three

different versions of the vignette. The first version, the control, presents a generic defense of the

ruling. The second version, or the main treatment, includes an aporetic justification, where the

judge admits that the question is complex and the court’s decision is not obviously correct or

beyond reproach. I also include a third version, an alternate treatment, which emulates the tone

of Scalia’s opinion in Scott v. Harris (2007), including language such as “clear scientific

consensus” and “an objective reading of the evidence leads to no other conclusion.” I test the

impact of this reasoning style—which given Simon and Surich’s (2011) framework I label

“monolithic”—primarily because adding an additional treatment helps allay concern that the

aporetic treatment reduces disagreement simply because it includes a more detailed justification

(Cobb and Kuklinski 1997). We can have increased confidence that the aporetic treatment has a

real effect if it reduces negative reactions to the ruling compared to both the control and the other

the decision, the dependent variable is not measuring what it is supposed to measure” (367-68).

Moreover, this design makes more efficient use of a smaller sample.

16

treatment. Moreover, given previous findings that expressions of certainty may heighten cultural

resistance (Kahan, Peters, et al. 2012), it seems reasonable to test this additional hypothesis. The

vignette manipulations for the protected class precondition can be seen in Table 1, while those

for the unprotected class precondition can be seen in Table 2.5

[Table 1 and Table 2 about here]

Given that the manipulations are randomly assigned within the two precondition groups,

the treatment effect can be assessed through a simple comparison of means for the dependent

variable, the disagreement measure. Following standard practice, I also assess the effect of the

manipulations within a regression model which controls for other variables that might impact

disagreement.6 I estimate two models, one for each precondition group.

Data and Variables

Study subjects were drawn from a convenience sample of university students (n = 327)

and a work request on Amazon’s MTurk marketplace (n = 305). The two samples were

combined prior to precondition assignment. The Qualtrics online platform was used to generate

the survey and collect the results.7 Power analyses suggest that the model would be adequate for

detecting effect sizes of .5, but might have some difficulty detecting effect sizes smaller than .3. I

5 The common text for all vignettes can be found in the appendix.

6 As a robustness check, the models were also run using ordinal logistic regression. The model

coefficients did not vary in their sign or significance, and their effect sizes did not meaningfully

differ from the standard regression models.

7 Sixteen responses had missing data (in most cases for a single question) for one or more of the

model variables. Employing multiple imputation to address these missing responses did not lead

to noteworthy changes in means or model outcomes.

17

proffered four questions to serve as dependent variables, all of which used a six point scale

which measured agreement on whether 1) the court correctly ruled on the question of

immutability, 2) the court’s decision was in line with scientific consensus, 3) the court’s

argument was persuasive, and 4) the decision caused the subject to lose faith in the courts. Factor

analysis revealed that the first three measures had one primary dimension and formed a reliable

six point disagreement scale (alpha = .825 for the protected class subset and .797 for the

unprotected class one) to serve as the dependent variable; the fourth did not scale with the others

and was dropped from the analysis. For the independent variables, I include dummy variables for

the aporetic and monolithic treatments, with the control vignette as the reference category.

For control variables, I included dummy variables for race and gender. I also asked

subjects a series of questions designed to measure their cultural worldviews (Dake 1991; Kahan

et al. 2007); in particular, I was interested whether subjects identified with an egalitarian or a

hierarchal worldview, as this is the cultural dimension most likely to be implicated in the debate

over gay marriage. These six questions were collapsed to form a scale that I term “hierarchical”

(alpha = .717 and .770, in each model), with increasing numbers illustrating increased support

for a hierarchical worldview. As alternate measures of attitudes, I also included seven-point

measures of ideology (liberal to conservative) as well as a measure of partisanship (strong

Democrat to strong Republican). These two variables were collapsed into a scale I label

“Conservative-Republican” (alpha = .791 and .773, respectively). I also ascertained—on a five

point scale—how frequently the subjects attended church or religious services, hypothesizing

that more frequent attendance would correlate with the view that sexual orientation is not

immutable, and vice-versa.

18

Finally, I included a series of checks designed to measure whether the survey subjects

correctly understood the vignette. This is particularly important given that the broader legal

issue, protected class analysis, is relatively complex. These checks include asking whether the

decision favored homosexuals, whether the decision involved protected class analysis or

legalizing gay marriage, and a true-false assessment of an accurate statement about the

relationship between immutability and constitutional protections.

Results

First, I examine the descriptive data, seen in Table 3.

[Table 3 about here]

By almost a three to one margin, survey subjects signaled support for gay marriage and sorted

themselves into the “unprotected class” precondition. This is unsurprising, given that the survey

experiment drew in roughly equal parts from a university campus convenience sample and a

MTurk work request, two subpopulations one might expect to be more egalitarian then the

general public (Berinsky, Huber, and Lenz 2012). The precondition assignment also seems to

have created the desired disagreement, given that the mean of the dependent variable for each

precondition is higher than the midpoint of the scale. Once divided, the demographics of the two

samples break down as one would expect: those who sorted themselves into the “protected class”

precondition were more conservative, Republican, and hierarchical than subjects in the other

precondition, and also reported higher rates of church attendance. By contrast, the two samples

did not differ meaningfully along race or gender lines. Finally, subjects in the unprotected class

precondition had a more negative reaction across all three vignette types than did those in the

protected class precondition (with a 95% degree of confidence). This distinction could point to

19

the vignettes not being perfectly balanced, or result from those who favor gay marriage simply

reacting more negatively to contrary empirical claims regarding immutability than those who are

opposed.

Next, I examine the manipulation checks, as shown in Table 4.

[Table 4 about here]

The success rate for these checks is high and in line with similar studies (e.g. Gibson 2008).

Moreover, for every manipulation check, subjects who selected the correct answer were, on

average, substantially and significantly more certain of their choice than those who did not.

I now move to the heart of the study, which is to assess whether decisions using aporetic

justification generate less disagreement than the control or monolithic treatment. First I present

the mean disagreement score for each of the six experimental conditions in Figure 1.

[Figure 1 about here]

For subjects assigned to the protected class precondition, aporetic reasoning had the intended

effect, substantially reducing the mean negative reaction relative to the control and monolithic

treatment.8 For the unprotected class precondition, by contrast, the mean for the aporetic

treatment is lower than the control and monolithic treatments, but the reduction is smaller and

not statistically significant. The fact that a larger and significant effect was found in the much

smaller protected class subgroup also suggests a meaningful difference between the two groups.

It may be the case that the immutability of sexual orientation is less important to egalitarians in

their determination of gay rights than to hierarchs, as some egalitarians may believe that

8 A one-way ANOVA and Bonferroni test confirms this assertion, presenting an F-statistic of

4.25 and confirming that the mean for the aporetic condition is significantly different compared

to the other categories.

20

homosexual behavior should be protected regardless of natural proclivities. If true, this might

create greater resistance to any sort of justification—aporetic or otherwise—for a ruling denying

protected class status based on an immutability determination.

Also of interest is the finding that, at least in this study, expressions of greater certainty

did not catalyze greater disagreement relative to the control. Indeed, the monolithic vignettes had

a slightly lower mean disagreement score than the control in both preconditions, although neither

difference is statistically significant. This result suggests the ameliorative effects seen here may

result in part because the treatments contain more information and are more specific relative to

the control (justifying the initial decision to include an alternate treatment).

I now present a regression analysis of the disagreement scale, transforming the treatment

conditions into dummy variables, with the control omitted as the reference category. Doing so

allows me to more precisely estimate the treatment effect while controlling for other variables

that might impact disagreement.

[Table 5 about here]

Overall, the other model variables perform as expected. Increasing scores on the Conservative-

Republican scale lead to increasing disagreement with the protected class ruling but decreasing

disagreement with the unprotected class ruling. The egalitarian-hierarchy scale also presents

coefficients with the expected signs, although the effect is much stronger, and only statistically

significant, in the unprotected class precondition. As noted above, this difference may be

meaningful, in that egalitarians react more negatively than hierarchs to an immutability ruling

they disagree with. Church attendance has strong and predictable effects in both preconditions,

and gender has little impact in either. Race, interestingly, has little effect within the protected

class precondition, but a strong impact in the unprotected class precondition. Specifically, white

21

subjects were more likely to disagree with a claim that sexual orientation is chosen compared to

racial and ethnic minorities, even controlling for other factors. The theoretical implications of

this finding are not entirely clear.

Regression coefficients are difficult to interpret, particularly when employing scales. To

aid in interpretation, I use the CLARIFY package in Stata (Tomz, Wittenberg, and King 2001) to

simulate how changes in model variables impact disagreement with the court’s ruling. Here I

simulate changes in the statistically significant model variables in each precondition, holding all

other variables at their means or modes (for dummy variables). Figure 2 presents the percent

change in disagreement given these changes.

[Figure 2 about here]

In the protected class precondition, aporetic reasoning decreases the negative reaction

score by almost 15% relative to the other vignettes. At least in this precondition, the manner in

which a judge justifies his decision has a meaningful impact on how his arguments are received

by those inclined to disagree. A one point increase in the Conservative-Republican scale

(meaning more conservative and Republican) increases disagreement by about 4%, while a one

point increase on the church attendance scale (e.g. shifting from monthly attendance to weekly

attendance) increases disagreement by about 6%. In the unprotected class precondition,

increasing church attendance and Conservative-Republican scores by one point have similar

effect sizes, albeit (as expected) in the opposite direction. Here also the cultural scale has a

significant impact, with a one standard-deviation increase in hierarchical beliefs leading to about

a 5% decrease in disagreement. Finally, within the unprotected class precondition, a white

subject was about 13% more likely to disagree with the court’s ruling relative to subjects of other

demographic backgrounds.

22

Simulation can also be employed to test different archetypes, in order to give the reader a

sense of how individual disagreement might vary given their relevant characteristics. For

example, in the protected class model, an ideological moderate who does not attend church and

is exposed to the aporetic vignette will have a predicted disagreement score of 2.79 (with 6 being

“extremely disagree” and 1 being “extremely agree”), while a strong conservative and

Republican who attends church more than once a week and is exposed to the control vignette

will have a predicted disagreement score of 5.08. In the unprotected class model, a moderate

member of a demographic minority who is highly hierarchical and attends church on a weekly

basis will have a predicted disagreement score of 2.49, while a very liberal, highly egalitarian,

white individual who does not attend church will have a predicted disagreement score of 5.49.

I also investigated interactions between the model variables. In general, there were no

statistically significant interaction coefficients or patterns of interest in the marginal effects, with

one exception in the protected class category (where the aporia treatment had its theorized

impact). Though the initial interaction coefficient did not reach statistical significance, an

examination of marginal effects revealed that the treatment’s impact meaningfully varied

alongside the subjects’ cultural beliefs. I present this effect in Figure 3.

[Figure 3 about here]

As the figure illustrates, the effect of the aporetic treatment was most effective for individuals

who were opposed to gay marriage but did not score highly on the hierarchical cultural measure.

This finding fits well with other work on motivated reasoning, suggesting that those with the

strongest beliefs are least likely to change their minds or be impacted by debiasing techniques.

Aporetic reasoning, then, may have its greatest impact on cultural moderates or those whose

23

general beliefs and specific issue positions (e.g. the smaller number of egalitarians opposed to

gay marriage) are not in sync.

Finally, I address concerns regarding selection bias and the initial non-random

assignment, as discussed in Gibson, Caldeira, and Spence (2002). While the authors of that study

were not convinced such corrections were needed, they did offer a method to check for bias.

They suggest modeling the initial assignment decision—here support for gay marriage—and

using the residuals from that estimation as an instrument in a two-stage regression equation. I

model the decision to support gay marriage as a function of ideology, partisanship, and church

attendance, running a probit equation on the sorting decision.9 After calculating the residuals

from this probit equation, I use Stata’s treatreg command to estimate a two-stage equation. The

results of this comparison raise two points of interest. First, the rho statistic from the two-stage

equation provides information about potential bias. The negative figure for rho generated here

suggests that the OLS model coefficients underestimate the real impact of the model variables

(Gibson, Caldeira, and Spence 2002, 373-74). Second, comparing the OLS and corrected models

shows that there is no substantive difference in outcomes between the two, with the size and

valence of each model variable being roughly equal, and no meaningful changes in statistical

significance. Full comparisons can be seen in the appendix.

9 The survey question provides four options in the assignment question on gay marriage: strongly

support, somewhat support, somewhat oppose, and strongly oppose. I collapsed these choices

into a dummy variable for the precondition assignment, and use this dummy variable here. All of

the model variables were highly statistically significant.

24

Caveats and Conclusion

Cultural cognition, and indeed all forms of motivated reasoning, threatens to undermine

public convergence on the best available scientific evidence, as well as undercut the legitimacy

of political institutions that must inevitably address polarizing issues. Motivated reasoning is

rightly viewed as a type of commons problem, in which policy entrepreneurs can dramatically

raise the salience of a policy issue by imbuing it with cultural or ideological meaning,

“polluting” the information commons (Kahan, Peters, et al. 2012). In a political system

characterized by separation of powers and multiple veto points, there are many scenarios in

which transforming a low-salience empirical question into a high-salience cultural conflict might

pay substantial dividends. Given increasing ideological, partisan, and cultural polarization

among elites, this dynamic is likely to accelerate over time. Federal courts, in my view, face a

special danger, as their power rests on their legitimacy, which these dynamics might erode over

time.

Researchers who study motivated reasoning sometimes highlight the need for debiasing

techniques that might ameliorate its negative effects. In this study, I test the impact of one

suggestion from the cultural cognition literature: the adoption of aporetic reasoning, or an

explicit admission of uncertainty and complexity, often combined with explicit recognition of the

tradeoffs or even tragic dimension of policy choices. Specifically, I employ an experimental

vignette design with an objection precondition, sorting subjects into one of two groups based on

their preexisting views on gay marriage. Each group is presented with a story about a court

ruling on the immutability of sexual orientation with which they are expected to disagree. I then

vary the judicial justification for the decision reported in the story, using aporetic reasoning as

one of three manipulations. I find that in the protected class precondition, aporetic reasoning

25

substantially reduced disagreement with the court’s decision. In the unprotected class

precondition, by contrast, the aporetic treatment had a smaller effect that, while in the expected

direction, was not statistically distinguishable from the other vignette versions.

As a first step in investigating strategies for ameliorating cultural cognition and motivated

reasoning, I find these results encouraging. There seems little doubt that authoritative

pronouncements on the immutability of sexual orientation divide people along cultural and

ideological lines, and yet, for subjects in one of the two preconditions, simply using more

aporetic justifications reduced negative reactions to the ruling presented. These findings merit

further investigation. Aporia is a feasible and relatively low-cost tactic—it does not require

judges, substantively, to lie (since most, though not all, empirical issues of interest before the

appellate courts are complex and can be authentically addressed in aporetic terms) or change

their ultimate conclusion. To be sure, altering modes of justification will be no panacea, and may

not alter the mindsets of the most polarized citizens. If repeated over time and across multiple

rulings, however, a commitment to more effective justification strategies might reduce

disagreement and polarization, potentially increasing support for judicial institutions.

The study’s findings also speak to a broader concern: the importance of paying attention

to how citizens feel when policymakers wish them to reason. There is little evidence that

straightforward attempts to present valid scientific information or correct misperceptions will

succeed on their own merits. If negative reactions to empirical information are driven by a

(rational) desire to avoid cognitive dissonance, successful communication by policymakers must

defuse these cognitive and emotional defenses before meaningful give and take can occur. This

study of aporetic reasoning breaks important ground by testing the feasibility of one such

26

strategy, providing a model for future work on debiasing techniques in courts and other

policymaking domains.

Of course, these findings must be replicated, using different issue areas and perhaps

decisions by figures in different policymaking institutions. At the Supreme Court, at least, there

is no dearth of controversial and affect-laden constitutional issues with empirical foundations for

further testing. Moreover, like any survey experiment, this study has weaknesses worth noting.

First, neither the convenience sample nor the MTurk sample can be deemed a true random

sample of the population, which might affect the validity of the inferences drawn. The university

sample drew primarily on young adults, and given that age inversely correlates with support for

gay marriage, it may be that even young conservatives or hierarchs process claims about sexual

orientation differently than older individuals. MTurkers also tend to be experienced survey

takers, which raises concerns when using their responses to draw inferences about the general

population.

Other caveats on the findings are driven by more general limitations of survey

experiments (Gaines, Kuklinski, and Quirk 2007). A second caveat involves duration effects—

how long, exactly, might this reduction of disagreement last? Would it meaningfully affect the

subject’s evaluation of the judge’s decision at a later time? Perhaps the most important caveat is

one of external validity: in this study, subjects were exposed to a single message, the judge’s. In

real life, of course, individuals may receive multiple messages in response to a salient

policymaking decision, such as communications from interest groups which aim to maintain the

battle lines of polarization. There are clear incentives for organized interests to encourage

motivated reasoning on a topic they care about, both to increase the issue’s salience and grow the

ranks of their supporters. Future work might test how the inclusion of interest group messages—

27

particularly messages that seek to stoke the fires of motivated reasoning—does or does not alter

the impact of aporetic reasoning or other potential debiasing techniques.

While these caveats are serious, I believe them to be challenges to be met in future work,

rather than insuperable objects to the evidence-based study of how judges—and policymakers in

general—should justify decisions within affect-laden policy areas. Motivated reasoning is

endemic to the political process. Investigating strategies for ameliorating its worst effects, then,

must remain an important goal if our policies are to be based on the best available science.

Cases Referenced

Atkins v. Virginia, 536 U.S. 304 (2002) Brown v. Board of Education, 347 U.S. 483 (1954) Brown v. Entertainment Merchant’s Association, 564 U.S. 08-1448 (2011) Craig v. Boren, 429 U.S. 190 (1976) Gibbons v. Ogden, 22 U.S. 1 (1824) Gill v. Office of Personnel Management, 682 F.3d 1 (1st Cir. 2012) Scott v. Harris, 550 U.S. 372 (2007) Kennedy v. Louisiana, 554 U.S. 407 (2008) SmithKline Beecham v. Abbott Labs, No. 11-17357 (9th Cir. 2014) United States v. Windsor, 570 U.S. 12 (2013) United States v. Carolene Products, 304 U.S. 144 (1938) Windsor v. United States, 699 F. 3d 169 (2nd Cir. 2012)

References

Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. 2012. "Evaluating Online Labor

Markets for Experimental Research: Amazon.com's Mechanical Turk." Political Analysis

no. 20 (3):351-368.

Borgmann, Caitlin E. 2013. "Appellate Review of Social Facts in Constitutional Rights Cases."

California Law Review no. 101 (5):1185-1248.

Braman, Eileen. 2009. Law, Politics, and Perception. Charlottesville, VA: University of Virginia

Press.

28

Campbell, John L. 1988. Collapse of an Industry: Nuclear Power and the Contradictions of U.S.

Policy. Ithaca, NY: Cornell University Press.

Cobb, Michael D., and James H. Kuklinski. 1997. "Changing Minds: Political Arguments and

Political Persuasion." American Journal of Political Science no. 41 (1):88-121.

Dake, Karl. 1991. "Orienting Dispositions in the Perception of Risk: an Analysis of

Contemporary Worldviews and Cultural Biases." Journal of Cross-Cultural Psychology

no. 22 (1):61-82.

Doran, Peter T., and Maggie Kendall Zimmerman. 2009. "Examining the Scientific Consensus

on Climate Change." Eos, Transactions American Geophysical Union no. 90 (3):22-23.

Douglas, Mary, and Aaron Wildavsky. 1983. Risk and Culture: An Essay on the Selection of

Technological and Environmental Dangers. Berkeley, CA: University of California

Press.

Faigman, David L. 1991. ""Normative Constitutional Fact-Finding": Exploring the Empirical

Component of Constitutional Interpretation." University of Pennsylvania Law Review no.

139 (3):541-613.

Gaines, Brian J., James H. Kuklinski, and Paul J. Quirk. 2007. "The Logic of the Survey

Experiment Reexamined." Political Analysis no. 15 (1):1-20.

Gibson, James L. 2008. "Challenges to the Impartiality of State Supreme Courts: Legitimacy

Theory and "New Style'' Judicial Campaigns." American Political Science Review no.

102 (1):59.

Gibson, James L., Gregory A. Caldeira, and Lester Kenyatta Spence. 2002. "The Role of Theory

in Experimental Design: Experiments Without Randomization." Political Analysis no. 10

(4):362-375.

———. 2005. "Why do People Accept Public Policies They Oppose? Testing Legitimacy

Theory with a Survey-Based Experiment." Political Research Quarterly no. 58 (2):187-

201.

Hoffman, Sharona. 2011. "The Importance of Immutability in Employment Discrimination

Law." William & Mary Law Review no. 52 (5):1483-1546.

Jenkins-Smith, Hank, and Walter Smith. 1994. "Ideology, Culture, and Risk Perception." In

Politics, Policy, and Culture, edited by Dennis Coyle and Richard Ellis, 17-32. Boulder,

CO: Westview Press.

29

Kahan, Dan. "Science Literacy & Cultural Polarization: It Doesn't Happen 'Just' With Global

Warming, but it also Doesn't Happen for 'All' Risks. Why?" The Cultural Cognition

Project. June 13, 2013. http://www.culturalcognition.net/blog/2013/6/13/science-literacy-

cultural-polarization-it-doesnt-happen-just.html.

Kahan, Dan M. 2009. "Culture, Cognition, and Consent: Who Perceives What, and Why, in

Acquaintance-Rape Cases." University of Pennsylvania Law Review no. 158:729-813.

———. 2011. "Neutral Principles, Motivated Cognition, and Some Problems for Constitutional

Law." Harvard Law Review no. 125:1-77.

———. 2014. "Making Climate-Science Communication Evidence-Based—All the Way

Down." In Culture, Politics and Climate Change, edited by Deserai Crow and Maxwell

T. Boykoff. London: Routledge.

Kahan, Dan M., and Donald Braman. 2006. "Cultural Cognition and Public Policy." Yale Law &

Policy Review no. 24:147.

Kahan, Dan M., Donald Braman, John Gastil, Paul Slovic, and CK Mertz. 2007. "Culture and

Identity-Protective Cognition: Explaining the White-Male Effect in Risk Perception."

Journal of Empirical Legal Studies no. 4 (3):465-505.

Kahan, Dan M., David A. Hoffman, Donald Braman, and Danieli Evans. 2012. "They Saw a

Protest: Cognitive Illiberalism and the Speech-Conduct Distinction." Stanford Law

Review no. 64:851-906.

Kahan, Dan M., David Hoffman, and Donald Braman. 2009. "Whose Eyes Are You Going to

Believe? Scott v. Harris and the Perils of Cognitive Illiberalism." Harvard Law Review

no. 122:837-906.

Kahan, Dan M., Hank Jenkins-Smith, and Donald Braman. 2011. "Cultural Cognition of

Scientific Consensus." Journal of Risk Research no. 14 (2):147-174.

Kahan, Dan M., Ellen Peters, Maggie Wittlin, Paul Slovic, Lisa Larrimore Ouellette, Donald

Braman, and Gregory Mandel. 2012. "The Polarizing Impact of Science Literacy and

Numeracy on Perceived Climate Change Risks." Nature Climate Change no. 2 (10):732-

735.

Kahn, Ronald. 1994. The Supreme Court and Constitutional Theory, 1953-1993. Lawrence, KS:

University Press of Kansas.

30

Lodge, Milton, and Charles Taber. 2000. "Three Steps Toward a Theory of Motivated Political

Reasoning." In Elements of Reason: Cognition, Choice, and the Bounds of Rationality,

edited by Arthur Lupia, Mathew D. McCubbins and Samuel L. Popkin, 183-213. New

York, NY: Cambridge University Press.

Lodge, Milton, and Charles S. Taber. 2005. "The Automaticity of Affect for Political Leaders,

Groups, and Issues: An Experimental Test of the Hot Cognition Hypothesis." Political

Psychology no. 26 (3):455-482.

Nicholson, Stephen P., and Robert M. Howard. 2003. "Framing Support for the Supreme Court

in the Aftermath of Bush v. Gore." Journal of Politics no. 65 (3):676-695.

Nyhan, Brendan, and Jason Reifler. 2010. "When Corrections Fail: The Persistence of Political

Misperceptions." Political Behavior no. 32 (2):303-330.

Nyhan, Brendan, Jason Reifler, Sean Richey, and Gary L Freed. 2014. "Effective Messages in

Vaccine Promotion: A Randomized Trial." Pediatrics no. 133 (4):e835-e842.

Pollock, Philip H., Stuart A. Lilie, and M. Elliot Vittes. 1993. "Hard Issues, Core Values and

Vertical Constraint: The Case of Nuclear Power." British Journal of Political Science no.

23 (1):29-50.

Rothman, Stanley, and S. Robert Lichter. 1982. "The Nuclear Energy Debate: Scientists, the

Media and the Public." Public Opinion no. 5 (4):47-52.

———. 1987. "Elite Ideology and Risk Perception in Nuclear Energy Policy." The American

Political Science Review no. 81 (2):383-404.

Secunda, Paul. 2012. "Cognitive Illiberalism and Institutional Debiasing Strategies." San Diego

Law Review no. 49 (2):373-414.

Simon, Dan, and Nicholas Scurich. 2011. "Lay Judgments of Judicial Decision Making." Journal

of Empirical Legal Studies no. 8 (4):709-727.

Stoler, Peter. 1985. Decline and Fail: The Ailing Nuclear Power Industry. New York, NY:

Dodd, Mead & Company.

Swedlow, Brendon. 2011. "Cultural Surprises as Sources of Sudden, Big Policy Change." PS:

Political Science & Politics no. 44 (4):736-39.

Taber, Charles S., and Milton Lodge. 2006. "Motivated Skepticism in the Evaluation of Political

Beliefs." American Journal of Political Science no. 50 (3):755-769.

31

Tomz, Michael, Jason Wittenberg, and Gary King. "CLARIFY: Software for Interpreting and

Presenting Statistical Results, 2.0." Harvard University, Cambridge, MA, 2001.

Tyler, Tom R, and Gregory Mitchell. 1994. "Legitimacy and the Empowerment of Discretionary

Legal Authority: The United States Supreme Court and Abortion Rights." Duke Law

Journal no. 43 (4):703-815.

32

Table 1: Vignette Manipulations, Homosexuals are a Protected Class

Treatment

Control Aporetic Monolithic

In their opinion, the judges indicated that scientific evidence showed sexual orientation to be innate and fixed, rather than a choice. The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is among these factors.

In their opinion, the judges indicated that scientific evidence showed sexual orientation to be innate and fixed, rather than a choice. “Those who argue that sexual orientation is fixed or unchanging sometimes overstate their case,” said Judge Lynch, writing for the majority. “There is a great deal that science doesn't know about sexual orientation—whether, for example, genetics or the early developmental environment plays a greater role in its formation. However, looking at the evidence before us today, the argument that sexual orientation is a fixed characteristic rather than a choice seems the stronger one.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is among these factors.

In their opinion, the judges indicated that scientific evidence showed sexual orientation to be innate and fixed, rather than a choice. “The notion that sexual orientation is a choice has been repeatedly debunked,” said Judge Lynch, writing for the majority. “The scientific consensus clearly indicates that sexual orientation is fixed by birth or at a very young age, and is not subject to meaningful change. An objective reading of the evidence leads to no other conclusion.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is among these factors.

Table 1 presents the manipulations in the Protected Class precondition. All three vignettes share common text, which can be seen in the appendix. Bold-faced text highlights quoted judicial reasoning.

33

Table 2: Vignette Manipulations, Homosexuals are an Unprotected Class

Treatment

Control Aporetic Monolithic

In their opinion, the judges indicated that there was no consensus in the scientific community as to whether homosexual preferences were innate and fixed, rather than voluntary or freely chosen. The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is not among these factors.

In their opinion, the judges indicated that there was no consensus in the scientific community as to whether homosexual preferences were innate and fixed, rather than voluntary or freely chosen. “Those who believe that homosexuality is a choice often ignore evidence suggesting sexual orientation may have biological roots,” said Judge Lynch, writing for the majority. “It may be that as science advances, we will uncover clearer evidence regarding potential biological or genetic causes of homosexuality. However, looking at the evidence before us today, we cannot endorse the conclusion that homosexual orientation is an unchanging characteristic akin to race or gender.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is not among these factors.

In their opinion, the judges indicated that there was no consensus in the scientific community as to whether homosexual preferences were innate and fixed, rather than voluntary or freely chosen. “There is no clear scientific consensus that sexual orientation is fixed or unchanging,” said Judge Lynch, writing for the majority. “There is no agreement in the literature on what might ‘fix’ sexual preferences, and thus there can be no agreement as to whether such preferences change. An objective reading of the evidence leads to no other conclusion.” The Constitution requires special protections, the court noted, when legal discrimination is based on factors that cannot be changed, such as race or gender. Homosexual orientation, the court said, is not among these factors.

Table 2 presents the manipulations in the Unprotected Class precondition. All three vignettes share common text, which can be seen in the appendix. Bold-faced text highlights quoted judicial reasoning.

34

Table 3: Means and Standard Errors for Survey Variables

Protected Class (n = 161) Unprotected Class (n = 469)

Monolithic Control Aporetic Total Monolithic Control Aporetic Total

Reaction (Negative)

4.10 (.15)

4.16 (.17)

3.53 (.17)

3.95 (.09)

4.46 (.09)

4.58 (.10)

4.36 (.09)

4.46 (.05)

Conservative 4.67

(.18) 4.67 (.21)

5.02 (.15)

4.77 (.11)

3.17 (.11)

2.92 (.11)

3.08 (.10)

3.06 (.06)

Republican 4.57

(.18) 4.6

(.23) 4.76 (.19)

4.64 (.11)

3.23 (.10)

3.30 (.11)

3.23 (.09)

3.25 (.06)

Hierarchical 0.01

(.04) 0.01 (.05)

0.00 (.05)

0.01 (.03)

-0.31 (.02)

-0.33 (.02)

-0.28 (.02)

-0.30 (.01)

Male (percent) .54

(.06) .42

(.07) .45

(.07) .55

(.04) .51

(.04) .54

(.04) .47

(.04) .50

(.02) White (percent) .68

(.06) .64

(.07) .70

(.07) .67

(.04) .67

(.04) .70

(.04) .67

(.04) .68

(.02) Church Att. 3.3

(.16) 3.12 (.18)

3.04 (.18)

3.17 (.10)

1.75 (.08)

1.87 (.10)

1.81 (.08)

1.81 (.05)

Table 3 presents the means and standard errors for the model variables, for each treatment and across each precondition.

35

Table 4: Manipulation Checks

Protected Class Vignettes Option 1 Option 2 Option 3 Option 4 Favored Homosexuals 82%

1.71 (.06) 18% 2.57 (.20)

Described Ruling 8.18%

2.07 (.21) 81.13% 1.67 (.06)

6.29% 2.4 (.31)

4.40% 2.43 (.57)

Relationship 78.13% 21.88% 1.79 (.06) 2.17 (.112)

Unprotected Class Vignettes Option 1 Option 2 Option 3 Option 4 Favored Homosexuals 10.26%

2.02 (.13) 89.74% 1.61 (.03)

Described Ruling 3.23%

2.13 (.26) 3.88% 2.44 (.19)

8.62% 2.35 (.12)

84.37% 1.29 (.03)

Relationship 81.62% 18.38% 1.74 (.03) 2.01 (.08)

Table 4 presents summary data for the survey’s manipulation checks, including the frequency for each choice, the mean certainty rating for each check (ranging from “very certain” (1) to “not at all certain” (4), and the standard error of the mean in parentheses. These checks include (1) whether the decision favored homosexuals, (2) whether the ruling was on protected class analysis or the legalization of gay marriage, and in which direction, and (3) whether the subject agreed or disagreed with an accurate statement about the relationship between immutability and constitutional protections, (as described in the vignette). Bold-faced text indicates the correct option.

36

Table 5: Multivariate Analysis of Disagreement with the Court’s Ruling

Protected Class Unprotected Class

Aporetic Treatment -0.592** -0.150 (0.223) (0.116) Monolithic Treatment -0.162 -0.124 (0.209) (0.120) Conservative-Republican 0.165* -0.171*** (0.080) (0.047) Hierarchal 0.286 -0.654** (0.326) (0.203) Male 0.101 -0.0185 (0.185) (0.097) White -0.031 0.631*** (0.218) (0.107) Church Attendance 0.298*** -0.300*** (0.070) (0.049) Constant 2.462*** 5.024*** (0.434) (0.244)

N 157 459 adj. R2 0.180 0.255

*p < 0.05, **p < 0.01, ***p < 0.001.

Table 5 presents the OLS coefficients and standard errors for the models’ variables. The dependent variable is a scale variable constructed from three survey questions which assessed the degree to which the subject disagreed with the court’s ruling on the immutability of homosexuality. The control condition serves as the reference category.

37

Figure 1: Mean Disagreement with Court Ruling, By Precondition and Vignette

Figure 1 presents the means (point) and standard errors (bars) for the disagreement or negative response scale variable (ranging from “extremely agree” (1) to “extremely disagree” (6)) for each experimental condition in both the protected and unprotected class preconditions.

38

Figure 2: Predicted Disagreement, Accounting for Changes in Other Model Variables

Figure 2 presents simulation results generated using Stata’s CLARIFY package, which estimates point predictions for the dependent variable based on simulated values of the independent and control variables. Here I present the percent change in the disagreement scale based on manipulations of the statistically significant OLS model variables. “Aporetic” and “White” are dummy variables, where a value of 1 is the simulated category. “Church” illustrates the change for a one point increase on a five-point scale which ranges from “no church attendance” (1) to “more than weekly attendance” (5). “Conservative-Republican” illustrates the percent change for a one point increase on a seven-point scale, with higher scores being more conservative and more Republican. “Hierarchical” illustrates the change for a one standard deviation increase on a measure of hierarchical worldviews.

39

Figure 3: Marginal Effects of Treatment by Cultural Scale on Disagreement with the Judicial Ruling, Protected Class Precondition

Figure 3 presents the marginal effects on disagreement with the judicial decision, specifically for the interaction between the treatment and cultural beliefs for subjects assigned to the protected class precondition. Cultural beliefs were scaled from -.5 to .5, with higher scores indicating more hierarchal beliefs. The grey shaded area illustrates the 95% confidence interval.

40

Appendix

Poll Questions and Vignettes

Precondition question: The issue of gay marriage has been in the news a great deal over the past year. In general, how would you describe your own views on gay marriage? Do you strongly support it, somewhat support it, somewhat oppose it, or strongly oppose it?

1 Strongly support 2 Somewhat support 3 Somewhat oppose 4 Strongly oppose

The vignettes for the protected class precondition had the following text in common: Please carefully read the following narrative: Court Rules Same-Sex Couples are a Protected Class By John Devita August 7, 2013 NEW YORK — A federal appeals court ruled on Thursday that gay Americans deserve constitutional protections similar to those subject to racial or gender discrimination. The ruling by the Court of Appeals for the Second Circuit came as the panel assessed the legal status of gay marriage in New Jersey. The case may have important ramifications for other states which do not currently have gay marriage. Importantly, the court’s ruling rested on a finding that because homosexuality is a fixed characteristic like race or sex, homosexuality should be considered a "protected class," a legal term which indicates that a group is likely to face to unconstitutional discrimination. This means that laws that impact homosexuals as a class will have to meet higher legal standards in order to be constitutional. [Manipulations go here; see Table 1 in manuscript] It is unclear at this point whether the case will come before the Supreme Court.

41

The vignettes for the unprotected class precondition had the following text in common: Please carefully read the following narrative: Court Rules Same-Sex Couples are not a Protected Class By John Devita August 7, 2013 NEW YORK — A federal appeals court ruled on Thursday that gay Americans do not merit the same level of constitutional protections accorded to individuals subject to racial or gender discrimination. The ruling by the Court of Appeals for the Second Circuit came as the panel assessed the legal status of gay marriage in New Jersey. The ruling may have important ramifications for other states which do not currently have gay marriage. Importantly, the court’s ruling rested on a finding that because homosexuality is not a fixed characteristic like race or sex, homosexuality should not be considered a "protected class," a legal term which indicates that a group is likely to face to unconstitutional discrimination. This means that laws that impact homosexuals do not have to meet higher legal standards in order to be constitutional. [Manipulations go here; see Table 2 in manuscript] It is unclear at this point whether the case will come before the Supreme Court. The questions for constructing the disagreement scale: The court correctly found that sexual orientation is a choice/is a fixed characteristic. The court’s finding that sexual orientation is a choice/is a fixed characteristic accurately reflects scientific and expert consensus. The court’s argument about sexual orientation is persuasive. 1 Extremely agree 2 Strongly agree 3 Somewhat agree 4 Somewhat disagree 5 Strongly disagree 6 Extremely disagree The questions for constructing the hierarchy scale: We have gone too far in pushing equal rights in this country. (Hierarchy) Our society would be better off if the distribution of wealth was more equal. (Egalitarian) We need to dramatically reduce inequalities between the rich and the poor, whites and people of color, and men and women. (Egalitarian)

42

Discrimination against minorities is still a very serious problem in our society. (Egalitarian) It seems like blacks, women, homosexuals and other groups don’t want equal rights, they want special rights just for them. (Hierarchy) Society as a whole has become too soft and feminine. (Hierarchy)

1 Agree 2 Disagree

Other control variable questions:

Generally speaking, how would you describe your politics? 1 Extremely liberal 2 Liberal 3 Slightly liberal 4 Moderate or middle of the road 5 Slightly conservative 6 Conservative 7 Extremely conservative

Generally speaking, do you consider yourself a Democrat, Republican, or Independent?

1 Strong Democrat 2 Democrat 3 Lean Democrat 4 Independent 5 Lean Republican 6 Republican 7 Strong Republican

Are you male or female?

1 Male 2 Female

Which of these racial or ethnic categories best describes you?

1 White 2 Black or African-American 3 Hispanic, Latino, or of Spanish Origin 4 Other

How often do you attend church or religious services?

1 Never 2 A few times a year 3 Monthly 4 Weekly 5 More than once a week

43

Manipulation checks: Now I would like to ask you some questions about what you can remember from the narrative you read. Who won the case, the side supporting gay marriage, or the side opposing it?

1 The side supporting gay marriage 2 The side opposing gay marriage

In the narrative that you read, which of the following statements accurately describes the judge’s ruling: a U.S. Court of Appeals ruled gay marriage constitutional; a U.S. Court of Appeals ruled homosexuals were a protected class; a U.S. Court of Appeals ruled homosexuals were not a protected class; a U.S. Court of Appeals ruled gay marriage unconstitutional.

1 a U.S. Court of Appeals ruled gay marriage constitutional 2 a U.S. Court of Appeals ruled homosexuals were a protected class 3 a U.S. Court of Appeals ruled gay marriage unconstitutional 4 a U.S. Court of Appeals ruled homosexuals were not a protected class

Please indicate whether you think the following statement is true: “if the court finds that sexual orientation is a choice, homosexuals may receive less constitutional protections”/ “if the court finds that sexual orientation is a fixed characteristic, homosexuals may be entitled to greater constitutional protections.”

1 Accurately describes (correct) 2 Does not describe (incorrect)

[For each check] How certain are you of this?

1 Very certain 2 Somewhat certain 3 Not very certain 4 Not at all certain

44

Test for Selection Bias

To test for selection bias which might result from the initial non-random assignment,

Gibson, Caldeira, and Spence (2002) suggest estimating a model for the selection variable (here

whether or not the subject supports gay marriage), using the residuals from this estimation as an

instrument in a two-stage regression. I estimate the assignment variable using a probit model

with the survey measures of ideology, partisanship, and church attendance (all variables are

significant at p < .01). I then calculate the model residuals, and use them in a two-step

regression, specifically using Stata’s treatreg function.

In the table below, I present the uncorrected OLS estimates for the combined sample side

by side with the estimates generated from the two-stage model. As the table shows, the

correction did not meaningfully affect the size of the model coefficients, and did not alter the

significance or valence of any model variable. The negative value for rho suggests that the OLS

model underestimates the true effect of the coefficients.

45

Comparison of Original and Two-stage Regression Models

Original Regression Two-stage Regression

Aporetic Treatment -.265* -.314** (.111) (.106) Monolithic Treatment -.094 -.096 (.111) (.106) Conservative-Republican -.109** -.118** (.04) (.04) Hierarchal .590** .419* (.172) (.170) Male .025 .015 (.093) (.088) White .691*** .588*** (.101) (.098) Church Attendance -.091* -.090* (.04) (.04) Constant 4.246*** 3.625*** (.251) (.283) Rho -.677

N 619 619