Argument Repertoire as a Reliable and Valid Measure of Opinion Quality: Electronic Dialogue During...

73

Political Communication 1973ndash93 2002Copyright atilde 2002 Taylor amp Francis1058-4609 02 $1200 + 00

Argument Repertoire as a Reliableand Valid Measure of Opinion Quality

Electronic Dialogue During Campaign 2000

JOSEPH N CAPPELLA VINCENT PRICE and LILACH NIR

A new measure of opinion quality that we name ldquoargument repertoirerdquo (AR) is intro-duced and evaluated AR refers to the relevant reasons that one has for onersquos ownopinions and the relevant reasons that others with opposite opinions might have Themeasure is shown to be reliable and to have construct validity Those with elevatedAR also were more likely to attend on-line deliberative groups during the presiden-tial election and to contribute to those conversations Those who participated in on-line deliberations tended to have higher AR scores on particular issues that werediscussed The role of AR in deliberative political groups is explored

Keywords campaign deliberation electronic discussion opinion quality

This article is concerned with the development of a valid and reliable measure of opin-ion quality Studies of opinion quality have taken two paths The first focuses on inter-ventions that have the potential to improve the quality of opinions held by members ofthe public These have included deliberative polls (Fishkin 1991 1995) issue forumsinformation-oriented polling and civic journalism (Rosen 1991) among others Thesecond focuses on methods of measuring existing opinion that in principle rank opinionfrom higher to lower on some aspect of quality These techniques have included filter-ing consistency coherence and alternative forms such as consideredness The researchwe report here focuses on the latter class of methods We introduce a new and particularmethod for measuring opinion quality that we call argument repertoire (see also Lusticamp Miodownik 2000) The reliability and validity of the argument repertoire measure areevaluated in the context of a discussion-based intervention during the 2000 presidentialelection that we have dubbed Electronic Dialogue 2000

Joseph N Cappella is Professor Vincent Price is Associate Professor and Lilach Nir is adoctoral candidate all in the Annenberg School for Communication at the University of Pennsyl-vania This research is supported by grants to the authors from The Pew Charitable Trusts and theAnnenberg Public Policy Center of the University of Pennsylvania The authors thank Yariv TsfatiJenny Stromer-Galley Danna Goldthwaite Tresa Undem Son-Ho Kim Clarissa David EmilyWest Masaki Hidaka Anca Romantan Eun-Kyung Na Roselyn Lee and Lisa Rand for theirassistance Views expressed are those of the authors alone and do not necessarily reflect opinionsof the sponsoring agencies

Address correspondence to Joseph N Cappella Annenberg School for Communication Uni-versity of Pennsylvania 3620 Walnut Street Philadelphia PA 19104-6220 USA E-mailjcappellaascupennedu

74 Joseph N Cappella et al

Improving the Quality of Public Opinion

Democratic theory assumes that the public responsible for decisions about its electedleaders is well informed (Delli Carpini amp Keeter 1996 Habermas 19811984) If col-lective decisions are not well informed then these decisions are based on other consid-erations perhaps ones that weigh less rational factors prominently The implication ifnot the reality is that uninformed decisions will be inefficacious at least and harmfulat worst to the public at large Comprehensive studies of the publicrsquos knowledge ofpolitics and social affairs suggest that the level of political knowledge is inadequate(Delli Carpini amp Keeter 1996) and that the quality of opinion is unorganized incoher-ent and uninformed (Bishop Oldendick Tuchfarber amp Bennett 1980 Neuman 1986)

In response a number of scholars and activists have sought to improve the qualityof public opinion by providing information to those being polled either as a part of thepolling process itself (Kay et al 1994 Neijens 1987 Neijens de Ridder amp Saris1992) or as a part of a larger process of providing extensive information and the oppor-tunity to deliberate about that information (Fishkin 1991 1995) The fundamental aimof both approaches is to determine what public opinion would be in a full (or nearlyfull) information environment This environment includes not only information gener-ally not possessed by the participants but also the opportunity to deliberate about evalu-ate and question this information in the presence of others with equal access to thesame informational resources

High-quality opinions emerge through free and equal exchange inviting argumentsfor all sides and granting to reasoned argument the power to shape collective choices(Arendt 1958 Habermas 19811984 19621989) Discussion is seen as vital becauseit forces more careful consideration by challenging points of viewmdashhence those whodeliberate form better reasoned opinions Moreover discussion expands the repertoire ofconsiderations and arguments and thus it fosters understanding of multiple points ofview (Gutmann amp Thompson 1996)

What is missing from these important and provocative innovations is any clear theoryabout what constitutes an effective intervention One goal of deliberative polls is toimprove the quality of participantsrsquo opinions What constitutes a higher quality opinionis subject to some debate Price and Neijens (1997 1998) proposed a series of possiblecriteria that varied by outcome (eg stability) and process (eg extensiveness of infor-mation search) and by level (individual and collective) We propose ldquoargument reper-toirerdquo as one measure of opinion quality that is an outcome-oriented individual-levelcriterion especially relevant to deliberative polling interventions

Assessing Opinions for Their ldquoQualityrdquo

A number of researchers have criticized standard polling techniques for producing re-sponses that are ill-informed pseudo-opinions (Price amp Neijens 1997) Much mass opinionis thought to be disorganized (Converse 1964) and without a basis in belief or knowl-edge (Herbst 1993 Neuman 1986 Zaller 1992) A variety of methodological tech-niques have been proposed to minimize the influence of such unanchored opinionsThey include the use of question filters probes ldquodonrsquot knowrdquo alternatives intensityratings and other techniques (Converse amp Presser 1986 Schuman amp Presser 1981)These procedures are aimed at minimizing the overselection of weakly formed opinionsgenerated off ldquothe top of the headrdquo by respondents

An alternative to eliminating weakly formed opinions from the response base is to

Argument Repertoire 75

measure the degree to which opinion statements are anchored in beliefsmdashboth support-ive and unsupportive of the focal attitude This view assumes that opinions are closelyrelated to attitudes as affective responses and that these attitudes in turn are connectedin a cognitive network to other attitudes and concepts through beliefs (Cappella amp Folger1980 Fishbein amp Ajzen 1975 Pratkanis 1989)

Fishbein and Ajzen (1975) suggest the use of open-ended elicitation procedures toextract salient beliefs (usually in the form of positive and negative consequences) thatare the basis of attitudes toward behaviors Wyatt Katz and Kim (2000) have usedldquoconsiderednessrdquo as a measure of opinion quality based on judgments by interviewers ofthe reasons provided by interviewees for their opinions Of course the NES surveyshave included open questions about the positive and negative ldquoconsiderationsrdquo that re-spondents have about political candidates (Zaller 1992)

Argument repertoire is a measure of opinion quality that functions much like con-siderations and beliefs It provides a way to rank stated opinions in terms of how wellanchored they are in reasons for onersquos own opinion and for opinions that are opposed toonersquos own

Argument Repertoire as Opinion Quality

Our measure of argument repertoire is derived from the work of Deanna Kuhn (1991)on the nature of reasoning in daily life She focuses less on what people think than whythey think what they do In a detailed conceptual and empirical analysis Kuhn main-tains the following (a) argumentation is nothing more than reasoning to show the truthor falsity of some claim (b) argumentation does not need to be dialogic to be studiedand evaluated and (c) expressed argumentation is indicative of the structure of knowl-edge (ie attitudes and beliefs)

Kuhn carries out a deep and broad description of the nature of naive causal argu-mentation including the kinds and qualities of evidence in support of onersquos claims Shealso applies this description to in-depth interviews of 160 people on three different top-ics (crime unemployment and schools) For example in the case of crime the issue wasldquoWhat causes prisoners to return to crime after theyrsquore releasedrdquo These explanationsmdashor theories as she calls themmdashare coded for the number of causes cited and whetherthere is genuine evidence in support

Kuhnrsquos real innovation is coding for and directly eliciting counterarguments Counter-arguments suggest that people can envision the conditions that would falsify their expla-nations This level of reasoning especially if accompanied by genuine counterevidencesuggests a sophisticated knowledge of the topic well beyond that represented by reasonsand evidence for onersquos own position It indicates knowledge of alternative positions atodds with onersquos own

Kuhnrsquos interview protocol even moves to a third level of argumentative complexitymdashthe ability to refute counterarguments to the original theory Referring to the hypo-thetical person who offers a counterargument to the respondentrsquos original theory theinterviewer asks ldquoWould you be able to prove this person wrongrdquo and ldquoWhat couldyou say to show that your own view is the correct onerdquo (Kuhn 1991 p 300)

The result of questions about arguments on an issue counterarguments and refuta-tions is a measure of the depth of knowledge held Those who can identify multipleexplanations with genuine evidence for them counterarguments to their own explana-tions and a resolution in favor of their own explanation are at the highest levels ofknowledge about the issue under discussion


In our opinion the importance of Kuhnrsquos work is not in its empirical findings butin its conceptual developments Her theory offers a representation of knowledge that isat least a precondition for rational argument in face-to-face contexts People with higherlevels of knowledge are contemplating the truth of their positions rather than simplyassuming the truth of their theories that is they are engaged in meta-cognition Incontemplating the accuracy of onersquos position one is allowing the possibility that it couldbe wrong As Kuhn (1991) notes

paradoxically this recognition is essential if one is to argue that theory iscorrect It is not primarily supporting evidence that shows a theory tobe correct One must also understand what evidence would show it to beuntrue and then seek out and confirm the absence of such evidence Theability to conceive of counterarguments is thus fully as critical as the abilityto conceive of alternative theories (p 266)

Kuhnrsquos analysis of the nature of naive argument provides the conceptual outlines for anoperational indicator of opinion quality that (a) will be capable of ranking opinionsoffered by respondents in terms of their quality (b) will be able to identify persons withthe tools to engage in deliberative discussion and (c) can provide an indicator of theconsequences of deliberative discussion

Kuhnrsquos conceptualization of argumentation dovetails with Parkrsquos (2000) analysisof opinion in terms of components he names individuality and civility Individualityrelates to the sophistication consistency and certainty of onersquos personal views andbehaviorally to onersquos ability to argue those preferences assertively Civility relates notto onersquos own views but instead to onersquos understanding of others Both components heargues are necessary for effective democratic deliberation

He interprets Habermasrsquos (19621989) theory of communicative action as empha-sizing mutual understanding and civility A basic democratic trait Park submits is aware-ness of what other people think coupled with some understanding of why others thinkthe way they do Whereas speaking develops strength and individuality in opinion it ishearing others speak that develops civility Deliberation requires both It is a ldquodual pro-cess of speaking and listeningrdquo (Park 2000 p 5) Put another way what makes opiniondeliberative is not merely that it has been built upon careful contemplation evidenceand supportive arguments but also that it has grasped and taken into consideration theopposing view of others Park identifies a range of cognitive attitudinal and behavioralaspects of civility The cognitive elements of civility include breadth of understandingperspective taking and understanding othersrsquo views Attitudinal components includeempathy tolerance trust in others and reciprocity Behavioral components include speakingldquowithrdquo (as opposed to speaking ldquoforrdquo oneself or ldquoagainstrdquo others) compromise andconsensus building

We focus here on what we take to be one of the core cognitive componentsof civilitymdashnamely the consideredness of onersquos opinion both in the sense of havinganchored a viewpoint in argument and in the sense of having considered other opposingviews We propose ldquoargument repertoiresrdquo (see Lustick amp Miodownik 2000) definedas the range of arguments people hold both in support of and against their favoredposition on a particular political issue or toward some political object Our conceptualizationand measurement approach follows in some respects the consideredness measureused by Kim and colleagues (Kim Wyatt amp Katz 1999 Wyatt Katz amp Kim 2000)However while their measure basically assesses the ability of respondents to argue their


views our assessment takes direct empirical account of their understanding of othersrsquopoints of view

Previous research has employed versions of argument repertoire in the political arenaWoodard (1995) developed a measure of ldquoargumentative depthrdquo from Kuhnrsquos interviewprotocol using three levels argument counterargument and refutation He found themeasure to be reliable and to discriminate between those exposed and not exposed tonews reports about Clintonrsquos health care reform proposal Rhee and Cappella (1997)found that political sophisticates exhibited higher quality arguments using a measureanalogous to Woodardrsquos In both applications relevant refutationsmdashthe third levelmdashwere very infrequent

Defining Argument Repertoire

For any given stated opinion on an issue argument repertoire includes the number ofrelevant reasons for the stated opinion and the number of relevant reasons for the oppo-site opinion For example if people indicate that they are favorable toward the Republi-can party the number of relevant reasons in support of their opinion gives the first partof their repertoire score They are then asked the reasons why someone might be un-favorable toward the Republican party The number of such relevant reasons providesthe second part of their argument repertoire score

The reasons provided must be relevant By relevant we mean reasons that areacknowledged in public discourse as plausible reasons (eg for their support of theRepublican party) Plausible reasons might include the Republicansrsquo policies regardingsmaller federal government and lowered taxes Irrelevant reasons could include a personrsquosstatement that he or she liked Republicans or knew some Republicans If irrelevantreasons were included then argument repertoire would be little more than a word countand no differentiation between genuine and bogus reasons and between reasons andmere preferences would be made

Unlike Kuhn (1991) we do not try to include genuine evidence in support of rea-sons as a part of the evaluation of argument repertoire nor do we attempt to ascertainthe coherence among multiple reasons (eg Tetlock 1989) Both dimensions are plau-sible components of argument repertoire However we want to employ our measure in asurvey context Genuine evidence is relatively infrequent and Kuhn required a series ofquestions and probes in face-to-face interviews to elicit the little evidence that peopleoffered Coherence (or integrative complexity) requires at least two arguments plus elaborationIn the current application we have ignored integrative complexity

Finally our analytic system does not attempt to distinguish accurate from inaccu-rate reasons For example suppose a person favored the Republican party and did soldquobecause the partyrsquos policies favored the poor over the richrdquo We do not judge thisclaim as inaccurate primarily because it is extremely difficult to know what reasons andevidence support the claim A person could believe in trickle-down economics and sosupport policies that enrich the wealthy with the expectation of improving the plight ofthe poor We leave factually accurate and inaccurate responses to the realm of standardclosed-ended knowledge questions

Hypotheses

If argument repertoire is to be an effective measure of opinion quality then it must beboth reliable and valid Reliability is evaluated in terms of intercoder agreement (Krippendorff


1980) The validity of argument repertoire should be established in terms of convergentdiscriminate and predictive validity Tests of predictive validity depend on the theoreti-cal accounts within which argument repertoire participates

Two tests of predictive validity are offered The first hypothesizes that those high inargument repertoire will also be more likely to participate in subsequent group delibera-tions about the presidential campaign The second hypothesizes that those participatingin group deliberations about the presidential campaign will show increases in subse-quent assessments of argument repertoire These two hypotheses together describe aspiral of effects from argument repertoire to deliberation and from deliberation to argu-ment repertoire representative of the underlying conception that has given rise to theargument repertoire concept

Method

Sample

Data are taken from parts of a multiwave survey of 1684 adult Americans conductedduring the 2000 presidential election campaign The study was a year-long panel consistingof surveys and discussions on-line Respondents came from a random sample of Ameri-can citizens age 18 and older drawn from a nationally representative panel of surveyrespondents maintained by Knowledge Networks Inc of Menlo Park California TheKnowledge Networks panel includes a large number of households (in the tens of thou-sands) that have been selected through random digit dialing (RDD) and agreed to acceptfree WebTV equipment and service in exchange for completing periodic surveys on-line1

The Electronic Dialogue 2000 project is distinguished from other Internet-basedstudies in a number of ways While most studies examine asynchronous message boardsor less formal and happenstance ldquochatrdquo experiences on the Web this project createdsynchronous real-time moderated group discussions that were designed specifically toproduce useful citizen deliberation Also the project did not rely upon a conveniencesample of Internet users as is common in Web-based studies instead it began with abroadly representative sample of Americans and attempted to recruit from that sample aset of 60 discussion groups that would be in their entirety as representative as possibleof US citizens

The project involved a multiwave multigroup panel design lasting roughly one yearAll data gathering was conducted over the World Wide Web The core of the projectconsisted of 60 groups of citizens who engaged in a series of monthly real-time electronicdiscussions about issues facing the country and the unfolding presidential campaignA set of baseline surveys in February and March 2000 assessed participantsrsquo opinionscommunication behaviors knowledge of public affairs and of the presidential candi-dates and a variety of demographic personality and background variables Subsequentmonthly group deliberations began in April and generally included pre- and postdiscussionsurveys The full text of all group discussions which lasted an hour apiece was recordedA series of end-of-project surveys (and a final discussion) were then conducted in Januaryand February 2001 Further detail on the project including a chronology of surveysdiscussions topics and participation rates is available in Price and Cappella (2001)

Measures

Argument repertoire Argument repertoire was assessed for five separate topics duringthe year-long study Here we report on three testsmdashone focused on political parties one


on the candidatesrsquo tax plans and the third on the Supreme Courtrsquos decision in Decem-ber 2000 in favor of George W Bushrsquos petition to overturn the judgment of the FloridaSupreme Court Analyses of the other topics (electoral college and choice for president)are in process

At the baseline surveys in March 2000 all respondents were asked whether theywere favorable or unfavorable toward each of the two parties on a 4-point ordinal scaleFollowing these questions we asked them in four open-ended questions to write reasonswhy they were favorable or unfavorable toward the party and why other people wouldbe unfavorable or favorable toward it respectively (See the Appendix for exact ques-tion wording)

Coding proceeded as follows2 If the respondent left the question blank he or shereceived a score of -99 If the respondent skipped the question his or her answer wascoded as ndash2 If his or her answer was otherwise missing the score was ndash1 When theanswer was irrelevant did not make sense merely restated the opinion indicated thatthe person does not know why he or she holds that opinion alluded in a vague way tothe partiesrsquo positions or was a statement about party membership only it was coded aszero For example a statement such as ldquoI like the Democratic partyrdquo or ldquoThe Democratssmellrdquo was coded as zero Except for those who registered no opinion on political partyall other skipped or blank responses were treated as zero

For each substantive answer one point was given for every reason the respondentwrote For example

Question What are the reasons you have for feeling very unfavorable towardthe Republican party

Answer Views on abortion (1) too close ties to business interests (2) fightagainst raising minimum wage (3) and other ways to help the poor andworking class Americans (4) insistence on tax cuts (5) fight against makinghealth care benefits more available and affordable (6)

This answer received a score of 6

Reasons for own opinion In the example above a person who is unfavorable toward theRepublican party is stating six different reasons why he or she is unfavorable and there-fore that statement was coded as reasons for ldquoown opinionrdquo Statements of both reasonswhy one favors the party with which one affiliates and reasons why one does not favorthe opposite party comprised this category We constructed a combined index of thecoded responses designating the sum of respondentsrsquo reasons for holding their ownopinion ( a = 77 one dimension accounting for 81 of the variance) The number ofreasons for own opinion index ranged from 0 to 20 with about 16 giving no reasonsand about 46 giving 2 to 5 reasons (Mdn = 3 M = 394 SD = 325 N = 1686)

Reasons why others might disagree Following the questions about reasons for onersquosown opinion in an additional open-ended question we asked the respondents to reasonwhy other people hold the opposite opinion Someone who indicated that he or shewas favorable toward the Republican party was asked to name reasons why others mightbe unfavorable toward the Republican party Similarly a person stating she or he areunfavorable toward the Democratic party was asked to state reasons why others mightbe favorable toward the Democratic party We constructed a combined index of the


coded responses designating the sum of respondentsrsquo reasons for the opinion of otherswho might disagree with their own position ( a = 80 one dimension accounting for83 of the variance) The index for the number of reasons for othersrsquo opposite opinionranged from 0 to 16 with about 28 giving no reasons and slightly more than 40giving 1 to 3 reasons (Mdn = 2 M = 266 SD = 264 N = 1685)

Intercoder reliability was assessed on a subsample of 50 open-ended responses Cohenrsquoskappa values ranged between 70 and 100 (mostly in the 80 range)

Similar procedures were followed for Bush and Gore tax proposals assessed duringOctober 2000 Exact wording for these questions is included in the Appendix Ownreasons summed across the two proposals ranged from 0 to 13 (Mdn = 20 M = 26SD = 251 N = 566) Opponentsrsquo reasons ranged from 0 to 8 (Mdn = 1 M = 16 SD =182 N = 566) Coding did not go forward until intercoder reliability produced kappasabove 75 Wording for the Supreme Court decision is included in the Appendix Ownreasons summed ranged from 0 to 5 (Mdn = 10 M = 152 SD = 096 N = 967)Opponentsrsquo reasons ranged from 0 to 7 (Mdn = 1 M = 113 SD = 091 N = 965 )Intercoder reliability was above 70

Political Knowledge

Various dimensions of political knowledge were combined to form a single scale mea-sure Items included 10 general political and civics knowledge questions (eg Who hasthe final responsibility to decide if a law is constitutional or not) 7 questions about thepersonal backgrounds of the presidential candidates (eg Which one of the democraticcandidates was a professional basketball player Which one of the GOP candidates is aformer POW) and an additional 7 questions about issue positions of candidates in theDemocratic and Republican presidential primaries (eg Which one of the Democraticcandidates supports universal health care Which of the Republican candidates supportsvouchers) All 24 items were scored 1 for correct answers and 0 for incorrect Theitems were averaged to create a scale (Cronbach alpha = 82 M = 62 SD = 19)

Political Interest

We used two different items to construct a political interest scale The questions mea-sured on a 4-point ordinal scale inquired about habitual following of public affairs andcaring which party wins in the 2000 elections The majority of respondents (79) re-ported that they followed public affairs either ldquomostrdquo or ldquosomerdquo of the time About 50of the respondents replied they cared ldquoa great dealrdquo which party wins the elections Bothitems loaded on a single factor that explained 73 of the variance A scale averagingthe two responses ( a = 62) was computed for further analyses (M = 320 SD = 071)

Mass Media Use

Exposure Exposure to mass-mediated current events content was measured on five dif-ferent items inquiring about the respondentsrsquo self-reported media use in days during thepast week (0 to 7) Newspaper reading and political talk radio exposure were measuredas single items Three itemsmdashexposure to television national network news cable newsand local newsmdashwere scaled together A factor analysis of the three items yielded asingle factor explaining 59 of the variance in responses A scale averaging the scores( a = 66) was computed (M = 346 SD = 193)


Attention Two questions asked respondents to self-report their attention to articles aboutthe presidential campaign in newspapers and to reports about the campaign on televisionduring the past week For each medium responses were measured on a 5-point scaleranging from ldquoa great dealrdquo to ldquononerdquo Of the people who completed both waves of thebaseline survey about 18 (n = 294) did not report any exposure to newspapers 72(n = 122) did not report exposure to television news Newspaper attention ranged from0 to 5 (M = 249 SD = 152) as did television news attention (M = 304 SD = 134)

Political Conversation

Respondents were asked to name by initials up to four people with whom they regularlydiscussed politics whether family members friends coworkers acquaintances or otherrelationships In addition respondents were asked to report several features of thesediscussionsmdashamong them the frequency of talking about politics with these four per-sons and the extent of disagreement the main respondent perceived Slightly less than11 did not name any discussion partners and about 56 (n = 940) named four dis-cussion partners

Frequency of discussion The respondents reported how many days in the past week (0to 7) they discussed political issues with each of the four discussion partners The re-sponses of those who did not name any discussants were coded as zero days theseindividuals discussed politics as well An additive scale of the respondentsrsquo answers tothe four questions was computed ( a = 74 M = 594 SD = 493)

Disagreement For each of the four discussion partners the respondents reported theextent to which they perceive their discussants to disagree with their own point of viewDisagreement was measured on a 5-point ordinal scale ranging from ldquoneverrdquo to ldquoalmostall the timerdquo Survey respondents who did not have any discussion partners at all re-ceived a zero An additive scale of the four disagreement scores ( a = 72) was com-puted for further analyses (M = 734 SD = 416)

Demographics

Respondents reported a median age of 43 years (M = 445 SD = 158) 46 werewomen 54 men 83 identified themselves as Caucasian 7 as Black or AfricanAmerican 2 as Asian and 16 as Native American Average education level was135 years (SD = 181) while 64 reported either full- or part-time employment

Results

Descriptive (Baseline)

The questions assessing argument repertoire (AR) clearly are meaningful to respondentsOn familiar topics such as reasons for favoring a political party 5 of 6 people are ableto cite relevant reasons while 3 of 4 can cite relevant reasons for why others are op-posed On less familiar topics such as candidatesrsquo tax plans 7 in 10 can cite at least onerelevant reason for their own opinion and 6 in 10 can cite a reason to have the oppositeopinion On less familiar but extremely important topics such as the Supreme Courtrsquoselection decision 9 of 10 offered at least one reason for their opinion and 3 of 4 areason for opponentsrsquo positions


Procedures to assess reasons are reasonably well defined Coders develop accept-able levels of reliability with relative ease (kappas above 75) and maintain them throughouttheir coding

The two aspects of argument repertoire are not independent with own and othersrsquoreasons correlating at 70 74 and 46 for party taxes and Supreme Court ARs re-spectively These correlations suggest that those able to cite their own reasons are alsoable to cite opponentsrsquo reasons Yet the two are not identical with at least 50 of thevariance in othersrsquo reasons not explained by own reasons

Table 1 summarizes the associations between demographic characteristics and argu-ment repertoire Across the three topics those with the higher argument repertoire scoreshave higher educational achievement and more extreme scores on an ideology-partyindex and they are more likely to be older White retired and without children under18 The relationships to gender ideology and party identification are inconsistent Thestrongest and most consistent findings are that those with anchored opinions are bettereducated and more committed to party and ideology

Convergent Validity

One aspect of the validity of a measure is its correlation with variables that are sup-posed to tap into similar constructs Table 2 presents correlations between argumentrepertoire for three issues and several constructs measuring aspects of the same latentdomain The variables listed as rows are measured at the baseline survey Argumentrepertoire for party is also measured at baseline while argument repertoires for taxesand court are measured later Thus the correlations between subsequent argument reper-toire measures and earlier transient variables such as attention to news network sizeand disagreement with discussion partners are not included in the table3

Table 1Correlations between argument repertoire scales and demographic variables

Party Tax Court

Own Other Own Other Own Other(N = 1628) (N = 1627) (N = 563) (N = 563) (N = 921) (N = 920)

Othersrsquo reasons 70 74 46More educated 24 29 16 20 15 21Conservative ideology 10 04 03 03 07dagger 07More Republican 07 02 06dagger 05 02 02Extremisma 24 14 18 17 08 10Male 08 05 ndash02 03 ndash04 ndash01White 09 10 07dagger 05 05 10Older 16 14 18 12 08 09Retired 07 09 11 07dagger 02 03No children under 18 06 06 13 11 ndash02 -05

Note Own refers to number of reasons for onersquos own position Other refers to reasons for those withopposing position

aAbsolute value of a ten-point scale (ndash5 to 5) of party affiliation and ideologydaggerp lt 10 p lt 05 p lt 01 p lt 001


Those higher in argument repertoire also tend to be more interested in politicsmore knowledgeable about politics more attentive to political news and more likely toread newspapers for news Moreover they have larger political discussion networks andperceive these groups to have more opposed positions

Also shown in Table 2 is a correlation with ldquoflexibilityrdquo a variable that calculates aparticipantrsquos availability to join proposed discussion groups at scheduled times The higherthe flexibility the more free time available to the respondent One possibility is thatthose with a great deal of free time might also be those who are able to write longerresponses to our open questions about their own and othersrsquo reasons The results inTable 2 are mixed on this question showing 4 of 6 significant positive correlations Weinvestigated this relationship more carefully in a series of regressions not reported hereWith argument repertoire as the outcome variable and flexibility age retirement statuschildren in the home political interest education and political knowledge in the equa-tion flexibility drops out as a significant predictor while political knowledge interestand education remain This suggests that simple availability of time is less a reason forthe higher argument repertoire scores than is motivation to answer

The correlations in Table 2 are in line with our expectations about how argumentrepertoire should function Those with the capacity to write out reasons for their opinionsand to identify relevant reasons for opposed opinions also express interest in politicsare more accurate in their factual political knowledge and use the print and broadcastmedia as sources of their political news Even their personal communication is more

Table 2Convergent validity argument repertoire scales Bivariate correlations

Party Tax Court


Political knowledge 48 46 29 35 21 27Political interesta 35 30 27 24 12 13Flexibility 13 14 13 09 05 03Newspaper exposure 12 12 15 13 08 12Television news 05 02 15 02 00 02

exposurePolitical talk radio 14 14 03 04 05 02

exposureAttention to campaign

Newspapers 23 22Television news 24 20

Network size 30 30Perceived disagreement 22 25

of discussion partners


aMean interest scale averaging interest in public affairs and how much the individual cares whichparty wins the 2000 election (1ndash4 scale)

p lt 05 p lt 01 p lt 001


political and more diverse (see Price Cappella amp Nir 2002) Coupled with the datafrom Table 1 indicating higher argument repertoire for those with more education andmore commitment to their ideology and party we have good evidence of convergentvalidity

Consistency

A personrsquos ability to provide reasons for his or her own and opponentsrsquo positions mightvery well change as issues change With different issues familiarity exposure to newsand interest may also differ However some stability in argument repertoire scores shouldcharacterize this form of political knowledge In Table 3 we offer the correlations be-tween points in time for three different measures of argument repertoire Argument rep-ertoire at baseline (political parties) is compared with AR (for tax proposals and theSupreme Court decision) The AR for tax was taken 7 months later and that for thecourt decision 10 months later

Despite these lengthy elapsed times all correlations are significant and of moderatesize The correlations between AR party and AR court are lower than those for partyand taxes One possible reason is that the party and taxes measures are actually sumsacross two different questions (ie both political parties and both candidatesrsquo tax plans)AR court is based on only one question regarding reasons for favoring or opposing theSupreme Courtrsquos decision and one question for opponentsrsquo reasons for the oppositeposition

Nevertheless despite the elapse of long periods of time slightly different measure-ment procedures and very different issues those with higher AR scores tend to havehigher scores on other issues at later points in time

Predictive Validity (Consequences of Argument Repertoire)

Those with elevated AR scores should be people with the capacity to participate inpolitical deliberation Their argument repertoires give them not just factual knowledgebut knowledge that can be used in deliberative activity (ie knowledge of their ownopinions and of the opinions of their interlocutors) Thus AR should predict willingness

Table 3Consistency of argument repertoire scores over time and topic Correlations

Tax proposals Supreme Court decision(October 7ndash18 2000) (December 4ndash15 2000)

(N = 563) (N = 920)

Own Opponentsrsquo Own Opponentsrsquoreasons reasons reasons reasons

Own reasons party 42 43 29 31(March 10ndash23 2000)

Opponentsrsquo reasons party 39 41 28 34(March 10ndash23 2000)

Note All correlations are significant at p lt 001


to participate in deliberative group discussions about politics and once there delibera-tive behaviors contributing to group exchange

In Table 4 we present results from multivariate logistic regressions predicting par-ticipation in any of the events 4 Of the demographic variables only age predicts partici-pation after controls such that older respondents are more likely to attend5 Time con-straints appear to influence participation in a number of ways Having children at homebeing employed full time and being a full-time student were each negatively associatedwith participation The effect of time constraints is further demonstrated by the measureof schedule flexibility The more flexible respondents (those who said they were avail-able for participation in more time slots during our preparation of group assignments)were considerably more likely to attend the projectrsquos electronic events

Political knowledge political discussion community participation and interpersonaltrust also affected participation significantly (and positively as expected) controllingfor time constraints ideological tendencies and demographic factors The higher peoplescored on each of these measures the higher the estimated odds that they would partici-pate in at least one of the electronic discussions Overall the pattern of coefficientsstrongly supports the view that ldquosocial capitalrdquo goes hand in hand with political partici-pation (eg Putnam 2000) Meanwhile mass media use had no such effect In factafter application of these extensive controls it is actually the case that people who

Table 4Logistic regressions predicting discussion attendance from argument repertoire

Model 1 Model 2

B eb B eb

Sex (male = 1) ndash17 84 ndash16 86Race (White = 1) ndash02 94 ndash05 95Years of education 03 103 02 102Age 03 103 03 103Party-ideology index ndash01 99 ndash02 99News media exposure ndash15 86 ndash14 87Political knowledge 110 300 15 115Political discussion 04 104 02 101Interpersonal trust 62 186 47dagger 161Community participation 123 341 105 285Schedule flexibility 18 120 17 118Employed full time ndash53 59 ndash48 62Number of children ndash14dagger 87 ndash16dagger 86Full time student ndash134 26 ndash144 24Argumentation 11 114

R2 21 22

Cases correctly classified () 75 75

N 810 778

Note The dependent variable is attendance in at least one online discussiondaggerp lt 1 p lt 05 p lt 01 p lt 001


report higher levels of exposure to newspapers TV and talk radio were somewhat lesslikely to turn out for their on-line discussions

Model 2 in Table 4 adds to the equation argument repertoire This measure capturesthe effects of political knowledge and political conversation which no longer producesignificant coefficients and weakens somewhat the impact of interpersonal trust Evenafter extensive controls AR for political parties measured at baseline accounts forattendance in subsequent on-line deliberations Consistent with our expectations thosewith elevated AR scores tend to be more willing to participate in subsequent politicalconversations

Willingness to attend on-line group discussions is not the same as contributing tothose discussions In Table 5 we present correlations between AR for political partiesassessed at baseline and behavioral participation in a discussion of education two monthslater In all of our discussions an initial period was devoted to welcoming participantsas they logged in This short period included general chitchat The remainder of theconversation was directed toward substantive issues determined a priori and led by amoderator In this particular event the preponderance of arguments went against vouch-ers as a solution to problems in the schools (Price amp Cappella 2001) Those higher inAR generated more words more substantive words on the central topics of the conver-sation and most important more arguments against vouchers

AR shows predictive validity in two important senses Those with greater ARs tendto be more willing to attend group deliberations about political topics and they tend toparticipate in the substantive exchanges while there They put their argumentative skillto use

Predicting Argument Repertoire

In October our participants discussed the tax proposals advanced by candidates Bushand Gore In the survey completed in the weeks following respondents were asked theirviews of both candidatesrsquo tax plan If they stated an opinion argument repertoire scoreswere obtained for each candidatersquos plan

As shown in the top panel of Table 6 the number of discussions attended by projectparticipants significantly predicts scores on this argument repertoire measure controlling

Table 5Correlations between argument repertoire at baselineand subsequent communication during deliberation

Communication duringeducation discussion Own reasons party Opponentsrsquos reasons party(May 5ndash21 2000) (March 10ndash23 2000) (March 10ndash23 2000)

Total words 14 22Words vouchers 16 20Words character education 21 25Pro arguments vouchers 09 07Con arguments vouchers 18 18

Note p lt 05 p lt 01 p lt 001


for argument repertoire assessed on the baseline survey and for propensity to attendFurthermore it appears to be attendance at the October event in particularmdashwhen taxeswere discussedmdashthat is primarily responsible for this effect As shown in the lowerpanel of Table 6 an equation that includes dummies for participation at multiple events(eg the May event when education was discussed as well as the October event) pro-duced significant coefficients only for the most relevant event when the candidatesrsquo taxproposals were discussed

Argument repertoire is affected by deliberative group interactions Both the act ofdeliberating and the substance of conversations can be efficacious in elevating argumentrepertoire The details of how this effect is produced remain to be uncovered However

Table 6Regressions predicting tax proposal argument

repertoire from participation in discussions

Own reasons Othersrsquo reasons Total reasons

B b B b B b

Regressions using number of events attended as predictor

Baseline Argument 27 37 24 36 30 43repertoire

Number of discussions 11 09 09 10 18 09attended (throughOctober)

Propensity to attend 208 12 140 11 292 10Intercept ndash27 ndash18 ndash41

R2 21 18 24

N 527 525 525

Regressions using specific events attended as predictors

Baseline argument 27 37 23 36 30 43repertoire

Attended in May ndash02 00 22 06 18 02(education discussed)

Attended in October 56 11 14 04 66 08(taxes discussed)


R2 21 19 26

N 527 525 525

Notes Entries are ordinary least squares regression coefficients Dependent variables are countsof the number of reasons coded as relevant to opinions on the candidatesrsquo tax proposals

p lt 05 p lt 01 p lt 001


the AR measure itself is sensitive enough to pick up differences resulting from topicaldeliberations

Discussion

The research reported in this paper offers a new measure tapping one component ofopinion quality a component representing the degree of anchoring of onersquos own opinionand knowledge of the bases for opinions held by others These two aspects are pre-sumed to tap into the ability to express onersquos own viewpoints and to be open to alterna-tive viewpoints In short those with more extensive argument repertoires should also becapable of participating effectively in deliberative discussions and opinion quality shouldbe a sensitive measure of the impact of substantive give-and-take about politics fromdeliberative groups These claims have been tested in the research reported here

Argument repertoire is a reliable measure of opinion quality In three applicationscoders were able to make reliable assessments of the relevant reasons people have fortheir opinions and reasons that others have for holding opposed opinions Coding pro-cedures are relatively straightforward and do not require extensive training time Re-spondents treat the questions assessing AR as meaningful with substantial proportionsgiving at least one relevant reason for their own and for opposed opinions AR hassome stability across time and across topics While topical variation is expected ARwould be a less effective measure if different topics produced totally different AR scores

AR is also a valid measure of opinion quality Those with higher scores on AR arepeople with expected political and communication profiles Those with higher AR aremore educated have greater political knowledge more interest in politics more expo-sure and attention to news more interpersonal communication about politics more com-mitment to their political parties and are older People we expect to have higher levelsof AR have higher levels of AR

AR shows itself to be an effective indicator of participation in political discussionRespondents with higher ARs were more likely to participate in our on-line discussiongroups and once there talked more on topic including offering arguments about the issueunder consideration The AR measure has proved to be sensitive to the effects of delib-eration adding to participantsrsquo prior levels of reasons for their own and othersrsquo opinions

Together these two effects suggest a spiral between deliberative discussion andAR with each being a causal force in the otherrsquos growth at a later point in time

The research we have completed so far has not yet addressed two key questionsthe bases of argument repertoire and argument quality Although we have seen thatdeliberation can increase AR the mechanism for this change is not yet well understoodThe availability of divergent opinion within the grouprsquos discussion is a plausible candi-date (Price Cappella amp Nir 2002) Onersquos own reasons may grow as one searches forresponses to challenges from others Knowledge of othersrsquo opinions may increase asmore diverse reasons are heard from others On the negative side disagreement mayproduce polarization and clique formation within heterogeneous groups Understandingthe processes through which ARs develop will help us understand the role of delibera-tion in developing informed citizens

We have been careful not to label our measure argument quality Quality impliesthat the reasons coded are somehow superior to other reasons offered or that evidence ismore serious than other evidence considered The quality of arguments that people haveat their disposal at a minimum should take into account the relevance number integra-tion (or coherence) and accuracy of reasons offered Our measure of AR includes only


the first two components of argument quality Although we have presented consistentand solid correlations between AR and factual political knowledge accuracy and coher-ence remain challenges for future research with AR

AR holds considerable promise for future research It can be used to distinguishdeeper from more shallow opinions and in this sense it is a component of opinionquality Moreover the concept is built around the idea that information in the form ofreasons for onersquos own and for othersrsquo positions is central to opinions that are deliberativemdashopinions that can be the focus of communicative give-and-take Without a sense ofonersquos own reasons and without a sense of the audiencersquos political communication asdeliberation with others is undermined

Notes

1 The Knowledge Networks panel sample begins with a list-assisted RDD sampleprovided by Survey Sampling Inc (SSI) Samples are acquired approximately once amonth to ensure that they are drawn from up-to-date databases Numbers in the SSIsample are then matched against a database of numbers known to be in the WebTVnetwork These numbers are then contacted and households are asked to participate asmembers of the Knowledge Networks panel In exchange for completing surveys (ap-proximately 40 minutes of cumulative survey time per household per month) panelistsreceive WebTV equipment and access free of charge The recruitment process results ina response rate of approximately 55 to 60 It produces a sample of American house-holds that closely approximates the population at large with a very slight underrepresen-tation of minorities and the elderly (Knowledge Networks 2000) In February 2000 arandom sample of American citizens age 18 and older (N = 3967) was drawn from theKnowledge Networks panel with the intention of recruiting participants for the Elec-tronic Dialogue 2000 project The initial recruitment survey provided a brief descriptionof the project emphasized the need for a representative sample and included a formindicating a respondentrsquos consent to participate Overall 51 of those recruited agreedto participate Those who consented were then sent two baseline surveys the first fromFebruary 8 to March 10 and the second from March 10 to March 23 The surveysincluded extensive measures of media use interest in the presidential campaign generalpolitical knowledge and knowledge of the campaign political discussion and a widevariety of political attitudes and opinions Response rates to each of the baselines wereapproximately 90

2 A detailed coding manual is available from the authors upon request3 When these correlations are calculated they exhibit the same pattern of results

only slightly weaker in magnitude4 Other analyses (not shown) examined predictors of the number of discussions

attended rather than the odds of attending any single event Results are quite similar5 Several of the political variables were not significantly associated with on-line

attendance and hence are not presented in Table 4 These include political efficacypolitical interest and political participation These variables do not significantly predictparticipation in the electronic discussion after controlling for demographics and politicalmeasures in Table 4

References

Arendt H (1958) The human condition Chicago University of Chicago Press


Bishop G F Oldendick R W Tuchfarber A J amp Bennett S E (1980) Pseudo-opinions onpublic affairs Public Opinion Quarterly 44 198ndash209

Cappella J N amp Folger J P (1980) An information processing explanation of the attitude-behavior inconsistency In D P Cushman amp R McPhee (Eds) Message-attitude-behaviorrelationship Theory methodology and application (pp 149ndash194) New York Academic Press

Converse J M amp Presser S (1986) Survey questions Handcrafting the standardized question-naire Beverly Hills CA Sage

Converse P E (1964) The nature of belief systems in mass publics In D E Apter (Ed) Ideol-ogy and discontent (pp 206ndash261) New York Free Press

Delli Carpini M X amp Keeter S (1996) What Americans know about politics and why it mat-ters New Haven CT Yale University Press

Fishbein M amp Ajzen I (1975) Belief attitude intention and behavior An introduction to theoryand research Boston Addison-Wesley

Fishkin J S (1991) Democracy and deliberation New directions for democratic reform NewHaven CT Yale University Press

Fishkin J S (1995) The voice of the people Public opinion and democracy New Haven CTYale University Press

Gutmann A amp Thompson D (1996) Democracy and disagreement Cambridge MA HarvardUniversity Press

Habermas J (1984) The theory of communicative action Vol 1 (T McCarthy Trans) BostonBeacon (Original work published 1981)

Habermas J (1989) The structural transformation of the public sphere An inquiry into a cat-egory of the bourgeois society (T Burger Trans) Cambridge MA MIT Press (Originalwork published 1962)

Herbst S (1993) Numbered voices How opinion polling has shaped American politics ChicagoUniversity of Chicago Press

Kay A F Henderson H Steeper F amp Lake C (1994) Interviews with the public guide us on the road to consensus St Augustine FL Americans Talk Issues Forum

Kim J Wyatt R O amp Katz E (1999) News talk opinion participation The part played byconversation in deliberative democracy Political Communication 16 361ndash385

Knowledge Networks (2000) Sampling report Annenberg survey Unpublished reportKrippendorff K (1980) Content analysis An introduction to its methodology Beverly Hills CA

SageKuhn D (1991) The skills of argument New York Cambridge University PressLustick I amp Miodownik D (2000) Deliberative democracy and public discourse The agent-

based argument repertoire model Complexity 5(4) 13ndash30Neijens P (1987) The choice questionnaire Design and evaluation of an instrument for collect-

ing informed opinions of a population Amsterdam Free University PressNeijens P de Ridder J A amp Saris W E (1992) An instrument for collecting informed opin-

ions Quality and Quantity 26 245ndash258Neuman W R (1986) The paradox of mass politics Knowledge and opinion in the American

electorate Cambridge MA Harvard University PressPark S G (2000 October) The significance of civility in deliberative democracy Paper pre-

sented at the conference of the Public Opinion Research in the Digital Age (PORDA) projectSeoul Korea

Pratkanis A (1989) The cognitive representation of attitudes In A Pratkanis S Breckler amp AGreenwald (Eds) Attitude structure and function Hillsdale NJ Erlbaum

Price V amp Cappella J N (2001 May) Online deliberation and its influence The ElectronicDialogue project in campaign 2000 Paper presented at the meeting of the American Asso-ciation for Public Opinion Research Montreal Quebec Canada

Price V Cappella J N amp Nir L (2002) Does disagreement contribute to more deliberativeopinion Political Communication 19 97ndash114

Price V amp Neijens P (1997) Opinion quality in public opinion research International Journalof Public Opinion Research 9 336ndash360


Price V amp Neijens P C (1998) Deliberative polls Toward improved measures of informedpublic opinion International Journal of Public Opinion Research 10 145ndash176

Putnam R D (2000) Bowling alone The collapse and revival of American community NewYork Simon amp Schuster

Rhee J W amp Cappella J N (1997) The role of political sophistication in learning from newsMeasuring schema development Communication Research 24 197ndash233

Rosen J (1991) Making journalism more public Communication 12 267ndash284Schuman H amp Presser S (1981) Questions and answers in attitude surveys Experiments on

question form wording and context New York Academic PressTetlock P (1989) Structure and function in political belief systems In A Pratkanis S Breckler

amp A Greenwald (Eds) Attitude structure and function Hillsdale NJ ErlbaumWoodard E (1995 May) Argumentative skill A measure of schema development Paper pre-

sented at the meeting of the International Communication Association Albuquerque NMWyatt R O Katz E amp Kim J (2000) Bridging the spheres Political and personal conversa-

tion in public and private spaces Journal of Communication 50 71ndash92Zaller J R (1992) The nature and origins of mass opinion Cambridge England Cambridge

University Press

Appendix Question Wording for Argument Repertoire

Argument Repertoire Baseline (March 10ndash23 2000)

Reasons for Own Opinion and Reasons Why Others Might DisagreeR1 How favorable in general are you toward the Democratic party

1 Very favorable2 Somewhat favorable3 Somewhat unfavorable4 Very unfavorableAsk R1a-b if favorable [R1(1-2)]

R1a What are the reasons you have for feeling (verysomewhat) favorable toward theDemocratic party (Please list all the reasons that come to mind) [textbox]

R1b What reasons do you think other people might have for feeling unfavorable to-ward the Democratic party (Please list all the reasons that come to mind) [textbox]Ask R1c-d if unfavorable [R1(3-4)]

R1c What are the reasons you have for feeling (verysomewhat) unfavorable towardthe Democratic party (Please list all the reasons that come to mind) [textbox]

R1d What reasons do you think other people might have for feeling favorable towardthe Democratic party (Please list all the reasons that come to mind) [textbox]

R2 How favorable in general are you toward the Republican partyVery favorable1 Somewhat favorable2 Somewhat unfavorable3 Very unfavorableAsk R2a-b if favorable [R2(1-2)]

R2a What are the reasons you have for feeling (verysomewhat) favorable toward theRepublican party (Please list all the reasons that come to mind) [textbox]


R2b What reasons do you think other people might have for feeling unfavorable to-ward the Republican party (Please list all the reasons that come to mind) [textbox]

Ask R2c-d if unfavorable [R2(3-4)]

R2c What are the reasons you have for feeling (verysomewhat) unfavorable towardthe Republican party (Please list all the reasons that come to mind) [textbox]

R2d What reasons do you think other people might have for feeling favorable towardthe Republican party (Please list all the reasons that come to mind) [textbox]

Argument Repertoire Regarding Tax Proposals(October 7ndash18 2000)

In their campaigns for President the candidates have made different proposals for cut-ting federal taxes Wersquod like to know some of your reactions

[Randomize Order of C1 and C2]

C1 How favorable in general are you toward the tax proposals that have been madeby Al Gore[radio]1 Very favorable2 Somewhat favorable3 Somewhat unfavorable4 Very unfavorableAsk C1a-b if favorable [C1(1-2)]

C1a What are the reasons you have for feeling (verysomewhat) favorable towardGorersquos tax proposals (Please list all the reasons that come to mind) [textbox]

C1b What reasons do you think other people might have for feeling unfavorabletoward Gorersquos tax proposals (Please list all the reasons that come to mind)[textbox]

Ask C1c-d if unfavorable [C1(3-4)]

C1c What are the reasons you have for feeling (verysomewhat) unfavorable towardGorersquos tax proposals (Please list all the reasons that come to mind) [textbox]

C1d What reasons do you think other people might have for feeling favorable towardGorersquos tax proposals (Please list all the reasons that come to mind) [textbox]

C2 How favorable in general are you toward the tax proposals that have been madeby George W Bush[radio]1 Very favorable2 Somewhat favorable3 Somewhat unfavorable4 Very unfavorableAsk C2a-b if favorable [C2(1-2)]

C2a What are the reasons you have for feeling (verysomewhat) favorable towardGeorge W Bushrsquos tax proposals (Please list all the reasons that come to mind)[textbox]


C2b What reasons do you think other people might have for feeling unfavorable to-ward George W Bushrsquos tax proposals (Please list all the reasons that come tomind) [textbox]


C2c What are the reasons you have for feeling (verysomewhat) unfavorable towardGeorge W Bushrsquos tax proposals (Please list all the reasons that come to mind)[textbox]

C2d What reasons do you think other people might have for feeling favorable towardGeorge W Bushrsquos tax proposals (Please list all the reasons that come to mind)[textbox]

Argument Repertoire Questions for Supreme CourtDecision (January 4ndash18 2001)

C1 As you may be aware in December the US Supreme Court ruled by a 5ndash4majority in favor of George W Bush and overturned the Florida Supreme Courtrsquoscall for a statewide manual recount of disputed election ballots in Florida Didyou support or oppose the US Supreme Court decision that ended the recount[radio]1 Support strongly2 Support somewhat3 Oppose somewhat4 Oppose strongly

If C1 = 1 or 2

C1a What are your reasons for supporting the Courtrsquos decision (Please list all thereasons that come to mind) [textbox]

If C1 = 1 or 2

C1b What reasons do you think other people might have for opposing the Courtrsquosdecision (Please list all the reasons that come to mind) [textbox]

If C1 = 3 or 4

C1c What are your reasons for opposing the Courtrsquos decision (Please list all the rea-sons that come to mind) [textbox]

If C1 = 3 or 4

C1d What reasons do you think other people might have for supporting the Courtrsquosdecision (Please list all the reasons that come to mind) [textbox]


Improving the Quality of Public Opinion

Democratic theory assumes that the public responsible for decisions about its electedleaders is well informed (Delli Carpini amp Keeter 1996 Habermas 19811984) If col-lective decisions are not well informed then these decisions are based on other consid-erations perhaps ones that weigh less rational factors prominently The implication ifnot the reality is that uninformed decisions will be inefficacious at least and harmfulat worst to the public at large Comprehensive studies of the publicrsquos knowledge ofpolitics and social affairs suggest that the level of political knowledge is inadequate(Delli Carpini amp Keeter 1996) and that the quality of opinion is unorganized incoher-ent and uninformed (Bishop Oldendick Tuchfarber amp Bennett 1980 Neuman 1986)

In response a number of scholars and activists have sought to improve the qualityof public opinion by providing information to those being polled either as a part of thepolling process itself (Kay et al 1994 Neijens 1987 Neijens de Ridder amp Saris1992) or as a part of a larger process of providing extensive information and the oppor-tunity to deliberate about that information (Fishkin 1991 1995) The fundamental aimof both approaches is to determine what public opinion would be in a full (or nearlyfull) information environment This environment includes not only information gener-ally not possessed by the participants but also the opportunity to deliberate about evalu-ate and question this information in the presence of others with equal access to thesame informational resources

High-quality opinions emerge through free and equal exchange inviting argumentsfor all sides and granting to reasoned argument the power to shape collective choices(Arendt 1958 Habermas 19811984 19621989) Discussion is seen as vital becauseit forces more careful consideration by challenging points of viewmdashhence those whodeliberate form better reasoned opinions Moreover discussion expands the repertoire ofconsiderations and arguments and thus it fosters understanding of multiple points ofview (Gutmann amp Thompson 1996)

What is missing from these important and provocative innovations is any clear theoryabout what constitutes an effective intervention One goal of deliberative polls is toimprove the quality of participantsrsquo opinions What constitutes a higher quality opinionis subject to some debate Price and Neijens (1997 1998) proposed a series of possiblecriteria that varied by outcome (eg stability) and process (eg extensiveness of infor-mation search) and by level (individual and collective) We propose ldquoargument reper-toirerdquo as one measure of opinion quality that is an outcome-oriented individual-levelcriterion especially relevant to deliberative polling interventions

Assessing Opinions for Their ldquoQualityrdquo

A number of researchers have criticized standard polling techniques for producing re-sponses that are ill-informed pseudo-opinions (Price amp Neijens 1997) Much mass opinionis thought to be disorganized (Converse 1964) and without a basis in belief or knowl-edge (Herbst 1993 Neuman 1986 Zaller 1992) A variety of methodological tech-niques have been proposed to minimize the influence of such unanchored opinionsThey include the use of question filters probes ldquodonrsquot knowrdquo alternatives intensityratings and other techniques (Converse amp Presser 1986 Schuman amp Presser 1981)These procedures are aimed at minimizing the overselection of weakly formed opinionsgenerated off ldquothe top of the headrdquo by respondents

An alternative to eliminating weakly formed opinions from the response base is to


























Hypotheses





Method

Sample




Measures
















Political Knowledge


Political Interest


Mass Media Use








Demographics


Results







Convergent Validity



Party Tax Court










Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4



























Hypotheses





Method

Sample




Measures
















Political Knowledge


Political Interest


Mass Media Use








Demographics


Results







Convergent Validity



Party Tax Court










Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4





Method

Sample




Measures
















Political Knowledge


Political Interest


Mass Media Use








Demographics


Results







Convergent Validity



Party Tax Court










Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4
















Political Knowledge


Political Interest


Mass Media Use








Demographics


Results







Convergent Validity



Party Tax Court










Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4








Demographics


Results







Convergent Validity



Party Tax Court










Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4






Convergent Validity



Party Tax Court










Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4







Party Tax Court










p lt 05 p lt 01 p lt 001



Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4




Consistency








(N = 563) (N = 920)










Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4







Model 1 Model 2

B eb B eb


R2 21 22


N 810 778




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4




















B b B b B b





R2 21 18 24

N 527 525 525






R2 21 19 26

N 527 525 525


p lt 05 p lt 01 p lt 001



Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4




Discussion











Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4





Notes






References





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4





































University Press


































If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4

























If C1 = 1 or 2


If C1 = 1 or 2


If C1 = 3 or 4


If C1 = 3 or 4


Argument Repertoire as a Reliable and Valid Measure of Opinion Quality: Electronic Dialogue During...

Documents

Transcript of Argument Repertoire as a Reliable and Valid Measure of Opinion Quality: Electronic Dialogue During...