1

“Peer review is melting our glaciers”: What led the Intergovernmental Panel on Climate Change (IPCC) to go astray?

---Please do not quote without permission---

Abstract:

An error in the Fourth Assessment Report (2007) of the Intergovernmental Panel on Climate Change (IPCC), which wrongly predicted the disappearance of Himalayan glaciers by 2035, fueled doubts about the authority, honesty and rigor of the IPCC as a leading institution in climate science and, correspondingly, raised questions about whether global warming is anything more than a hoax put forward by environmentalists. The late and confusing reaction of the IPCC to these allegations only worsened the matter. By comparing assessment reports issued by the IPCC, this paper asks the question: ‘Why, despite extensive peer review, did the Himalayan glacier melting rate error get published?’ I chronicle exactly what happened (section 3), consider why it happened as it did (section 4), and show how these answers require us to rethink the concept of ‘peer review’ in scientific practice. I also identify several future directions for peer review (section 5) if it wants to stay ahead of the game, bearing in mind the forthcoming IPCC Assessment Reports.

2

Between November 2009 and January 2010, climate skeptics managed to dominate the worldwide media after it came to light that the widely-acknowledged leading authority on climate science, the Intergovernmental Panel on Climate Change (IPCC), had made a mistakenly endorsed an ill-supported estimation in its Fourth Assessment Report (2007) of the disappearance of Himalayan glaciers by the year 2035. This incident was quickly recognized as one of the darkest pages in IPCC history. In the commotion, many questions arose: ‘Is the IPCC trustworthy?’, ‘Were these mistakes published on purpose just to convince everyone of the urgency to respond to global warming?’, and ‘Do we need to reconsider climate scientists’ consensus that the earth is warming and that it is largely due to anthropogenic factors?’ Apart from climate skeptics’ previous attacks on the organization, the obvious error in its report baffled even the IPCC itself, and it took them a while to give an official response. My aim in this paper will be, first, to show what happened and address why the IPCC blundered on its estimation of the melting rate of the Himalayan glaciers, and then, second, to use this case study to shed new light on ‘peer review’ in organizational scientific practice and set out its future directions. In section 1, I present a brief overview of the goals and principles of the IPCC, then, consider in detail its peer review process for establishing Assessment Reports. In section 2, I compare the IPCC’s Second (1995) and Fourth Assessment Reports (2007) to expose a problem in its review procedure, and so consider ‘What happened that the Himalayan glacier melting rate mistake got through the IPCC’s extensive peer review?’ This is an important question and must be answered if one wants to put into perspective the worldwide commotion that arose in reaction to the discovery of the mistake. In section 3, I answer the question by assessing the multitude of review comments. In section 4, I investigate why flaws like this could occur and present some mitigating circumstances in coming to a possible answer. This detailed analysis allows me, in section 5, both to rethink the concept of ‘peer review’ in scientific practice and to set out future directions for ‘peer review’ if we want to safeguard its current role in science and society. Section 6 brings all these claims together and presents some suggestions for how future research could be conducted.

1. The IPCC as an organization: goals, principles and structure On their website, the Intergovernmental Panel on Climate Change (IPCC) characterizes itself as the leading international body for the assessment of climate change. The United Nations Environment Program (UNEP) and the World Meteorological Organization (WMO) established the IPCC to provide the world with a clear scientific view on the current state of knowledge in climate change and its potential environmental and socio-economic impacts. (IPCC, official website) As a scientific body, the IPCC reviews and assesses the most recent scientific, technical and socio-economic information produced worldwide relevant to the understanding of climate change. It does not conduct any of its own research nor does it monitor climate related data or parameters. As an intergovernmental organization, it invites the participation of all member countries of the United Nations (UN) and WMO. Currently 194 countries are members of the IPCC. Governments participate in the review process and the plenary

3

sessions, where main decisions about the IPCC work program are taken and reports are accepted, adopted and approved. Thousands of scientists from all over the world contribute to the work of the IPCC on a voluntary basis. The IPCC aims to reflect a wide range of views and expertise and, through its ‘peer review’ process, tries to ensure an objective and complete assessment of current information, which is bundled into Assessment Reports. Because of its scientific and intergovernmental nature, the IPCC is a unique source of rigorous and balanced scientific information for decision makers. By endorsing the IPCC Assessment Reports, governments acknowledge the authority of their scientific content. The work of the organization is therefore considered policy-relevant and yet policy-neutral, never policy-prescriptive. (IPCC, official website) In dividing up the work of writing an Assessment Report, the organization splits itself into three Working Groups. These three Groups have carried out the scientific and technical assessment of climate change:

“Working Group I assesses the physical scientific aspects of the climate system and climate change, including attribution of past change and projections of future change.

Working Group II assesses the vulnerability of socioeconomic and natural systems to climate change, negative and positive consequences of climate change, and options for adapting to it.

Working Group III assesses policy and technology options for mitigating climate change through, for example, limiting and preventing greenhouse gas emissions and enhancing activities that remove them from the atmosphere”. (IAC, 2010: 12)

To ensure proper preparation and analysis, the IPCC has devised a peer review procedure consisting of the following chronological steps:

1. “Compilation of lists of Coordinating Lead Authors, Lead Authors, Contributing Authors, Expert Reviewers, Review Editors and Government Focal Points

2. Selection of Lead Authors 3. Preparation of draft report 4. Review

a. First review (by experts) b. Second review (by governments and experts)

5. Preparation of final draft report 6. Acceptance of report at a session of the Working Group(s)” (IPCC, 2008a: 3-8)1

In step 1, a compilation of lists of Coordinating Lead Authors, Lead Authors, Contributing Authors, Expert Reviewers, Review Editors and Government Focal Points is made. At this stage, it is crucial that researchers from developed countries, developing countries and countries in economic transition are all represented. In step 2, a selection of Lead Authors is made. Amongst them ought to be at least one author from a developing country. In step 3, these authors produce a draft report. This procedure is governed by three principles: First, the best available scientific and technical advice must be included in the report, assuring that the IPCC-report is up to date. This advice must be transmitted as comprehensibly as possible. Second, the procedures have to ensure that participating countries are fairly represented as far as possible. Third, the Review-process has to be objective, open, and transparent; it has 1 For a detailed description of the tasks and responsibilities for Lead Authors, Coordinating Lead Authors,

Contributing Authors, Expert Reviewers, Review Editors and Government Focal Points see (IPCC, 2008b).

4

to be understandable to anyone. In step 4, the involved experts and governments analyze the draft report by investigating how well the authors have acted in accordance with the three conditions spelled out above. To help ensure a balanced and complete assessment of current information, Review Editors are appointed. These editors are not involved in the preparation or review of material for which they serve as editor. The first review circulation should include (1) experts who have significant expertise and/or publications in particular areas covered by the Report, (2) experts nominated by governments as Coordinating Lead Authors, Lead Authors, Contributing Authors or Expert Reviewers. The second review circulation takes place only once comments of the first period have been considered and the draft is revised accordingly. In step 5, the final draft of the report is made, taking into account any criticisms that arose in step 4. In step 6, the report is accepted at a session of the Working Group(s). (IPCC, 2008a: 4-7) I will refer again to this procedure, and mainly to its fourth step (peer review), later on, but first I will draw a comparison between two published Assessment Reports of the IPCC.

2. The societal impact of IPCC Assessment Reports In this section, I examine the Second (IPCC, 1995) and Fourth (IPCC, 2007) Assessment Reports issued by the IPCC. The comparison will (1) contribute to an understanding of what happened and understand why the mistaken Himalayan glacier melting rate estimate got into the Fourth Climate Change Assessment Report (2007) (sections 3 and 4) and (2) allow me to shed new light on peer review in scientific practice (section 5). In their recent book, “Merchants of Doubt” (2010), Naomi Oreskes and Erik M. Conway describe how a handful of scientists have obscured the truth on issues ranging from tobacco smoke to acid rain, global warming, and the hole in the ozone layer. In the case of global warming, they recount how respected scientist Ben Santer, one of the lead authors of the IPCC’s Second Assessment Report (1995), was accused by several scientists of ‘scientific cleansing’ (expunging the views of dissenting opinions) and ‘deceiving policy makers and the public’, all because he had allegedly changed parts of the report behind the backs of his fellow authors. In Santer’s case and others like it, more or less the same group of skeptics, driven by scientists like Fred Singer and Frederick Seitz, among others, and conservative think tanks as the Alexis de Tocqueville Institution, was behind this type of attack on mainstream science. According to Oreskes and Conway, the skeptics’ goal was not to uncover the truth of the matter, but mainly to put in question facts that contradicted their personal interests. By thus fighting the scientific evidence and spreading confusion, they were able to merchandise doubt and promote a state of mind according to which ‘more evidence is needed before action can be undertaken’. In taking up the defense of science and of Santer in particular, Oreskes and Conway touch upon the role peer review played in both formulating and refuting the alleged charges. It turned out that Ben Santer was simply doing his job by granting peer review its rightful place in science. The extensive peer review process of the IPCC (see section 1) required that Santer involve both scientific experts and representatives of the governments of the participating nations in order to ensure not only that factual errors were caught and corrected, but that all judgments and interpretations were adequately documented and supported and that all interested parties had an equal chance of being heard. It entails a responsibility to revise the

5

report based on valuable review remarks (see the allegation). In Oreskes’ and Conway’s words, Santer was attacked for being a ‘good scientist’. So, if these skeptics had simply taken the time to familiarize themselves with IPCC rules and procedures, they would have discovered that no rules had been violated and no procedures transgressed, and thus that no wrongdoing had occurred. The changes made were made in accordance with valuable review comments. (Oreskes and Conway, 2010: 2-9) According to Oreskes and Conway, peer review is a helpful tool in distinguishing ‘the good science from the junk’. Exactly this peer review procedure, they hold, contributed to making the IPCC into the respected organization it is today. Through the four assessment reports the organization has released to date,2 it has proven its scientific worth in the debate on global warming. More recent attacks by skeptics, however, which are widely referred to under the general term ‘climategate’, have been directed against the conclusions and procedures of the IPCC’s Fourth Assessment Report (2007)3. These critical voices reached a peak when the IPCC-board was forced to acknowledge that the report contained a false projection of the disappearance of Himalayan glaciers. The fact that the IPCC waited more than 2 months to release its official apology4 (Carrington, 2010) only fueled the doubts that had arisen concerning the motives, diligence and trustworthiness of IPCC climate scientists (and of the IPCC in general). Now if we compare both cases, i.e. the skeptical voices raised against the Second (1995) and Fourth Assessment Reports (2007), we find the term ‘doubt’ constantly re-emerging. In the second report, no actual mistakes were publicly attributed to scientists (like Ben Santer), but

2 Note that at the time of publication, only the Working Group I report of the Fifth Assessment Report was

released. 3 The most well-known dispute here was the unauthorized release of e-mail exchanges, which were seen to

discredit the University of East Anglia and its scientists. This leaked e-mail traffic gave rise to serious skeptical doubts concerning the honesty of climate scientists. The question of whether (some of) the doubts raised were grounded is beyond our scope here and seem to remain an open question. In any case, the credibility of the overall conclusion of the Fourth Assessment Report (2007), namely that global warming is happening and is due mainly to anthropogenic factors, was not affected by this incident in any respect. 4 Since the Indian government pointed out the melting error in early November 2010, the IPCC took more than

two full months to release a statement, made public on 20 January 2010, containing the IPCC board’s apologies. The apology read as follows: “It has, however, recently come to our attention that a paragraph in the 938-page Working Group II contribution to the underlying assessment refers to poorly substantiated estimates of rate of recession and date for the disappearance of Himalayan glaciers. In drafting the paragraph in question, the clear and well-established standards of evidence, required by the IPCC procedures were not applied properly. The Chair, Vice-Chairs, and Co-Chairs of the IPCC regret the poor application of well-established IPCC procedures in this instance. This episode demonstrates that the quality of the assessment depends on absolute adherence to the IPCC standards, including thorough review of the quality and validity of each source before incorporating results from the source into an IPCC Report. We reaffirm our strong commitment to ensuring the level of performance” (IPCC, 2010a). Follow-up letters explaining the principles, procedures, role and key elements of the IPCC were released on 2 and 4

February 2010 (IPCC, 2010b&c). The

Chairman of the IPCC, Rajendra Pachauri, made no personal comment at the time, but in response to criticism concerning his position as Chairman he responded as follows to the Guardian "They can't attack the science so they attack the chairman. But they won't sink me. I am the unsinkable Molly Brown. In fact, I will float much higher." (Carrington, 2010) Even more striking is the first (and emotional) response made by the Chairman in response to the geologist Vijay Kumar Raina, who issued a report in name of the Indian government pointing out the error. Pachauri dismissed the report as not peer-reviewed and said: “With the greatest of respect this guy retired years ago and I find it totally baffling that he comes out and throws out everything that has been established years ago." (Carrington, 2010)

6

public ignorance concerning the existing procedures of peer review paved the way for skeptics to fuel doubt on global warming. In the Fourth report, despite peer review, an error was made, according to which the likelihood of the Himalayan glaciers disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate. (IAC, 2010: 22) The IPCC’s delayed response to this problem served only to further stoke doubts about global warming. What has so far never been sufficiently addressed in the academic literature on the IPCC is the question of what exactly happened and why this mistake got past the review procedure. Maybe skeptics are right when they say that scientists who study the purported effects of global warming too often let their ethical and political viewpoints take over? Whatever the answer might be, it seems obvious that something more is at stake here. I tackle these concerns by answering the question of what exactly happened and why it happened (section 3 and section 4).

3. What happened? Obviously, this mistake opens the door to skeptical criticism: pointing out flaws in the report could initiate a chain reaction resulting in a dispute over whether global warming is actually happening and, even if it is, whether mankind plays a role in it5. Shortly after releasing its letter of apology, and mainly in response to sustained criticism and increased public scrutiny, the United Nations and the IPCC asked the Inter Academy Council (IAC) to assemble a professional committee to review the processes and procedures of the IPCC and make recommendations for change, which could confirm and enhance the authoritative nature of IPCC reports. The IAC is a multinational organization of scientific academies created to produce reports on scientific, technological, and health issues related to the great global challenges of our time, providing knowledge and advice to national governments and international organizations. (IAC, 2010: ii) As promised, the IAC issued a critical study, entitled ‘Climate Change Assessments: Review of the Processes and Procedures of the IPCC’ (2010), in which, amongst other things, they briefly referred to the error of the Himalayan glacier melting rates. (IAC, 2010: 22-24) The disputed paragraph addressing the melting rates, reads as follows: “Glaciers in the Himalaya are receding faster than in any other part of the world and, if the present rate continues, the likelihood of them disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate. Its total area will likely shrink from the present 500,000 to 100,000 km2 by the year 2035 (WWF, 2005).” (IAC, 2010: 22)6 As can be seen here, the data in this paragraph was attributed to a source issued by the campaign group ‘World Wide Fund for Nature’ (WWF), although the claim originated from an interview

5 I recognize this to be an ill-supported slippery slope and should hereby acknowledge the IPCC’s statement of

its overall conclusion, that “Widespread mass losses from glaciers and reductions in snow cover over recent decades are projected to accelerate throughout the 21

st century, reducing water availability and hydropower

potential” (IPCC, 2007: 49), still holds. Regardless, however, of whether one finds this claim and critique convincing, one has to recognize the impact that a situation like this could have upon overall public opinion and media reporting, bearing in mind the halting effect of propagating doubt on public policy making and how even a very small hunch can successfully merchandise doubt (Oreskes & Conway, 2010). 6 It is important to keep in mind, as suggested by a reviewer of this paper, that since the claim made is only

hypothetical, no deliberate falsification of data is involved here.

7

with the Indian glaciologist Syed Hasnain for the ‘New Scientist’ on June 5, 1999. (Pearce, 2010) In order to determine the extent to which the inclusion of this error might reflect weaknesses in the IPCC review process, I call upon the IAC’s detailed examination of the draft text and its reviewer comments. This analysis was possible due to the detailed record the IPCC maintains of all review comments and author responses. As noted in section 1, there are two rounds for ‘peer review’. In the first round, relating to the section containing the paragraph above, the experts raised six comments, of which none could be regarded as critical. However, of the twelve expert reviewer comments on the second draft, two were related to the erroneous estimation (see Table). (IAC, 2010: 23)

Comment Number Comment Writing Team Notes

E10-466 100,000? You just said it will disappear. (David Saltz, Desert Research Institute, Ben Gurion University)

Missed to clarify this one.

E10-468 I am not sure that this is true for the very large Karakoram glaciers in the western Himalaya. Hewitt (2005) suggests from measurements that these are expanding – and this would certainly be explained by climatic change in precipitation and temperature trends seen in the Karakoram region. (Fowler & Archer, J Climate in press; Archer & Fowler, 2004) You need to quote Barnett et al.’s 2005 Nature paper here – this seems very similar to what they said. (Hayley Fowler, Newcastle University)

Was unable to get hold of the suggested references, will consider in the final version.

Table: Expert Reviewer Comments on the Second Draft (Second period of peer review)7 Comment E10-466 identifies a contradiction between the first and second sentence of the paragraph. However, the text was not changed by the authors. Comment E10-468 disputed the statement by providing references that had different conclusions. Had the Writing Team and/or Review Editors checked these references, they would have stumbled upon peer reviewed papers which were, at the very least, more cautious in making such sweeping claims:

- Hewitt stated: “In the late 1990s widespread evidence of glacier expansion was found in the central Karakoram, in contrast to a worldwide decline of mountain glaciers. The expansions were almost exclusively in glacier basins from the highest parts of the

7 This table is a simplified, easily assessable version of the one that can be found in the official IAC-report (IAC,

2010: 24).

8

range and developed quickly after decades of decline. Exceptional numbers of glacier surges were also reported.” (Hewitt, 2005: 332)

- Fowler & Hewitt stated: “The observed downward trend in summer temperature and runoff is consistent with the observed thickening and expansion of Karakoram glaciers, in contrast to widespread decay and retreat in the eastern Himalayas. This suggests that the western Himalayas are showing a different response to global warming than other parts of the globe.” (Fowler & Hewitt, 2006: 4276)

What the table shows is that the claim was published without being revised in accord with relevant criticism. The procedure did work in that it enabled reviewers to pinpoint the error, but it did not work in that it failed to appropriately consider the remarks made8 and thus let the error9 slip through. The error was published due to the authors’ failure to appropriately consider the peer reviewers’ comments, as they were supposed to do, and the Review Editors’ failure to ensure that the comments had in fact been addressed. Thus, false, insufficient and inconsistent information was included from an unreliable source.10 In this section, I have shown what exactly happened behind the scenes of the IPCC, allowing us to get a clearer picture of the matter and the inner working of the IPCC review process. In the following section, I try to understand why the error got published, not simply by blaming the responsible authors, who failed to take the review comments into account, but by mapping out some of the pressures in play during the IPCC review process.

4. (Possible) mitigating circumstances In this section, I want to dig more deeply into the context in which reviewers, authors and Review Editors have to perform their duties in order to understand why the error could slip through the review process. The Fourth Assessment Report (2007) consisted of 20 chapters, plus an introduction. In the first round for peer review by Experts, each of these chapters received around 1,000 comments that needed to be reviewed (22,205 in total). During the second round of peer review, the experts raised 10,826 comments, whereas the government representatives added another 4,047, bringing the total to 37,078 comments. (IAC, 2010: 22) Clearly, such an overwhelming number places a heavy burden on the Writing Team and Review Editors in charge. As formulated in their job description, they are supposed to consider all review comments, but are required to provide written responses only to the most significant issues raised by reviewers. (IPCC, 2008b) Obviously, in the case under consideration, this built-in filter did not function appropriately, because (a) an obvious contradiction (E10-466), (b) potential errors (E10-466 & E10-468), and (c) the citation of dissenting literature (E10-468) are all significant remarks. Given this failure, and in light of the astonishing number of

8 The standpoint of the IAC-report on the matter reads as follows: “In this example, IPCC’s report review process

failed in two ways: (1) Failure of the authors to carefully consider thoughtful review comments (E10-466 and E10-468), which would have improved the quality of the report, and (2) Failure of the Review Editors to ensure that reviewer comments were adequately addressed and that controversies are reflected adequately in the text of the report (E10-468).” (IAC, 2010: 24) 9 The error mentioned here is ‘that the likelihood of Himalayan glaciers disappearing by the year 2035 and

perhaps sooner is very high if the Earth keeps warming at the current rate’. 10

It is not my intention here to scapegoat the Review Editors and/or Writing Team members. My aim is simply to address what happened.

9

comments received in this case, the current peer review procedure must be reevaluated in order to achieve some reliable trade-off between open review (resulting in more remarks being raised) and limited available resources (to deal appropriately with these remarks). I return to this question of reevaluation in the next section. Glaciologist Syed Hasnain, reacting to allegations that his faulty results has set off the entire chain reaction, responded that he is not to blame, since (1) "the magic number of 2035 has not [been] mentioned in any research papers written by me, as no peer-reviewed journal will accept speculative figures" (Pearce, 2010), and (2) "it is not proper for IPCC to include references from popular magazines or newspapers." (Pearce, 2010) Yet here the question arises: ‘Does the IPCC really have to rely upon the use of grey literature, such as the WWF report, and run the risk that its results were not properly peer reviewed?’ ‘Grey literature’ is literature in which the extent of peer review varies a great deal, as does its quality. Examples include “technical reports, working papers, presentations and conference proceedings, fact sheets, bulletins, statistics, observational data sets, and model output produced by government agencies, international organizations, universities, research centers, nongovernmental organizations, corporations, professional societies, and other groups”. (IAC, 2010: 19) If the IPCC can leave grey literature aside, the reliability of the information used could perhaps increase and, in its turn, could result in fewer review comments11, thus largely dissolving the trade-off spelled out above. The answer, however, is that the IPCC does rely and indeed must rely upon grey literature if it wants to reach the goals it has set for itself (section 1). Although some have suggested that only peer-reviewed literature be used in the Assessment Reports, this would require the IPCC to disregard much valuable information, such as insights from very large data sets and detailed model results (Working Group I), reports of farmer cooperatives, government agencies, nongovernmental organizations, the World Bank, and United Nations Bodies (Working Group II), and company reports, industry journals, and information from the International Energy Agency (Working Group III). Peer-reviewed papers do not always contain all the useful information about a topic, such as vulnerabilities and strategies for adaptation and mitigation in particular sections and regions, which are an important part of the reports issued by Working Groups II and III. (IAC, 2010: 19) Bjurström and Polk, in an analysis of the Third Assessment Report (2001), demonstrated the presence and importance of grey literature in the Working Group reports. Based on an analysis of the 14,000 articles cited, the authors reveal that peer-reviewed papers comprised 84 % of references in Working Group I, but only 59 % of references in Working Group II and 36 % of references in Working Group III. (Bjurström and Polk, 2011) The IPCC issued a document stating the procedure for using non-published/non-peer-reviewed sources in the Assessment Reports (IPCC, 2008c), which requires authors to critically assess grey literature and review their quality and validity before incorporating them. These objectives are to ensure that all information used has had some kind of critical evaluation, that all references are easily able to be checked, and that the use of information is open and transparent. However, as we saw in the case of a a ‘grey’ source (WWF, 2005) being used to support the claim concerning Himalayan glacier melting, the procedure is not always followed to the letter.

11

As the IAC-study suggests, most of the review comments were related to problems with grey literature.

10

5. What do we learn from this?12 In this section, I show in what respects the ‘Himalayan glacier melting rate’ debacle can inform how we conceive of, and ought to conceive of, the role of ‘peer review’ in scientific practice. First, I recall the insights in section 4 to show that ‘peer review’ in scientific practice has certain limitations, with which we must fully acquaint ourselves if we want to continue using it as a guiding mechanism in the ‘advancement of science’. Second, this case study will be taken to support an argument that reviewers’ names ought to be made public. Though this would not guarantee that errors like this could always be avoided13, it would help to ensure that appropriate credit is accorded and that accountability/responsibility is promoted. Third, the pressure of social media and time restrictions has led academics to defend a ‘Publish-then-filter’-attitude towards publishing. Although this idea (could) work(s) in scientific academia, the case under consideration here shows that it cannot easily be extended to science outside academia, where a notion of ‘filter-then-include’ is more appropriate.

5.1 Restrictions in play Although procedures for dealing with grey literature do exist (see section 4), they are not always put into practice. The IAC-report notes that a search through the Working Group reports of the Fourth Assessment report found only a few instances of information being flagged for being based on grey literature, though the procedure requires that this is to be done at all times. (IAC, 2010: 20) The authors of the IAC-report suggest that clearer guidelines and stronger mechanisms be instituted to enforce existing rules14. To stimulate an open review process, some governments made the second draft available for review by national experts and other interested parties, resulting of course in a higher number of remarks to be reviewed. On the one hand, having to go through thousands of review comments does not seem to weigh up to the benefits of an open extensive review: (1) if the IPCC wants to be the leading scientific organization on climate issues and makes it one of their goals to transmit this information to the public in a transparent manner, an open review process will increase their credibility and reliability as a spokesperson or advisor, and (2) it increases the range of viewpoints offered and (potentially) improves the quality of the report. On the other hand, this does not imply that we could ignore the challenges for open review, i.e. (1) of finding an appropriate way to deal with multiple remarks in an environment of scarce time and resources (trade-off)15, (2) of managing the possibility that

12

‘Peer review’ in the IPCC review process relates only to the second goal, identified by Steve Fuller, that peer review could have in the pursuit of knowledge. The three goals are the following: “(1) A mechanism for certifying research activity as knowledge and crediting researchers with having produced knowledge (e.g. journal editorial policies). (2) A mechanism for protecting the knowledge base from error contamination and the public from the application of unsound research (e.g. state regulatory bodies, ethics review panels). (3) A mechanism for the efficient and equitable allocation of the scarce resources available and needed for conducting research (e.g. funding agencies).” (Kihara, 2003: 66; Fuller, 2002) 13

In our example, as the Table shows, the reviewers’ names were made public and yet the error was published anyway. I return to this point later in the paper. 14

A possible suggestion is to install Review Editors with more power to ensure that grey literature is selected appropriately and used consistently (IAC, 2010: 20). 15

Although the IAC does not suggest any concrete way for tackling this problem, it proposes that the IPCC adopt a more targeted and effective process for responding to reviewer comments: “In such a process, Review Editors would prepare a written summary of the most significant issues raised by reviewers shortly after review

11

the press will release premature/flawed/insufficiently grounded conclusions and (3) of addressing the possibility that people with a strong bias towards climate change will attempt to dominate the issue. When we speak of climate science and of science in general, scientists, academics, and the media often portray it as a process governed by mere progress. Luckily, philosophical work has shed much needed light on scientific practice and shown that it is often clouded by failures and setbacks. As Oreskes manages to put it: “In general, scientific research does not give us logically firm evidence about the natural world. And, thus, the corresponding idea of science being able to deliver undisputable evidence upon which policy decisions could be made is flawed. In a scientific community, different individuals can weigh evidence in different manners through the use of different standards. In the best case, science puts forward a robust consensus based on a research process that allows continued scrutiny, re-examination, and revision.” (Oreskes, 2004: 369-370) Recognizing the limits of scientific research and characterizing it as, at best, a robust consensus puts a major burden on the process, allowing for continued scrutiny, re-examination and revision; namely, allowing for ‘peer review’. Although the idea of peer review in science is simple, i.e. that no scientific claim can be considered legitimate until it has passed the critical scrutiny of other experts (Oreskes & Conway, 2010: 15), we should not overstress the capacity of ‘peer review’ to serve as an ultimate means of distinguishing between good and bad science. Peer review can never tell one whether a claim is right, but at best only whether, based on current knowledge, it is considered reliable by the community of experts. Peer review generally ensures only that a text makes no unacceptable claims, rather than ensuring that it makes good claims. And that, indeed, is the rub: society demands that science present true claims, whereas science, through peer review, aims only to rule out unacceptable claims. In dealing with examples where right-wing scientists and pressure groups forge their own (pseudo-)scientific claims (section 2), one might easily come to expect too much of ‘peer review’, and in doing so, minimize and underestimate the many restrictions in play. However, acknowledging the limitations set out above should not be taken to imply that ‘peer review’ (and by extension science) is wholly prey to relativism; it implies only that we need to devise ways to master and localize these restrictions, or, at the very least, recognize their presence. Although peer review is not perfect, it is, if done correctly, the best means to ensure that a study has had the benefit of independent scrutiny and quality control before being cited by others, as the case studied shows.

5.2 Remain anonymous? This case study touches upon an interesting topic of constant debate in academia16, i.e. whether or not journal editors should make the names of referees public. Possible arguments in favor are that (1) peer reviewers ought to receive public recognition for the important service they provide, (2) anonymous review permits reviewers to be unduly scathing and publicly identifying reviewers might help to improve accountability, (3) anonymous review turns much of the process of peer-review into a “black box”, in that

comments have been received. Authors would be required to provide detailed written responses to the most significant review issues identified by the Review Editors, abbreviated responses to all non-editorial comments, and no written responses to editorial comments” (IAC, 2010: 21). 16

Such as seen on the academic blog http://www.newappsblog.com/

12

authors have no way of discerning whether a reviewer possesses appropriate expertise and are left wholly dependent upon the unsubstantiated claims of editors that their papers are under consideration by appropriate experts, (4) anonymous reviewing conceals the shortcomings of editors, in that authors have no assurance that the body of reviewers a journal has at its disposal possess the necessary expertise to assess the worth of their paper, and thus are left in the dark concerning whether papers are being forwarded to reviewers who might not have the best interest of the authors in mind (Roy & Ashburn, 2001), (5) reviewers may have a personal interest in whether or not a paper is accepted for publication, neglected, or delayed in publication (Dalton, 2001; Turner, 2003: 184) Possible arguments against making names public are that (1) anonymity protects reviewers from personal attacks by displeased authors, (2) anonymity dispels any pressure to give approval for publication, and (3) scientists might not want to make full-fledged criticisms if they have to make their names explicit, as they may be afraid of repercussions. I would like to introduce into this debate a distinction between two ways in which ‘peer review’ can be understood in relation to organizational scientific practice: first, as reviewing the reliability of information, and second, as reviewing the originality and publication-worthiness of the tenets in a publication. This distinction is too often overlooked in the current debate. Our example is useful in evaluating the first type of peer review. Let me explain this further. Refereeing is still often done anonymously, both in as well as outside of academia. If referees were no longer anonymous, it would be possible to grant them credit for checking the information included in the reports. It is hard to see why a referee who checks the correctness of claims should be anonymous. Judgment on the originality and publication-worthiness of a paper can be another case, but perhaps these tasks should therefore be separated in the refereeing process.17 If we remember the review comments made (section 3), we notice that comment E10-466 pointed out a (rather obvious) contradiction and that comment E10-468 identified opposing viewpoints. As the example mainly illustrates a failure of the authors to follow up the references, I do not and cannot claim here that making reviewers’ names public would have resolved this, as the names were already made public. I would only claim, rather, that the fault is attributable (partly) to the fact that the authors failed to follow up on the reviewers’ comments, presumably due to mitigating time and resource restrictions, although correct procedures requiring them to do so were in place. The example does reveal the potential benefit of opening up peer review in this manner, namely (1) that reviewers would receive credit for the mistakes they pointed out, (2) that an analysis of these types of mistakes can be undertaken, and (3) that responsibility/blame can be attributed accordingly. Making the names public would allow other scientists, and the wider public, to see who did and did not do their job appropriately. It would give complex situations such as this more visibility and, accordingly, accountability.18 David Saltz and Hayley Fowler, who raised these remarks, wholly deserve

17

We do not address this question here, as our example merely engages with reviewers checking the reliability of information, as can be seen in the Table and in the stated values of the IPCC (section 1). For further information on how this distinction relates to peer review in academic journals, where both ways of assessing a paper come into play, see AUTHOR (xxx). 18

One could raise the objection that ‘If comments are overlooked, having a name attached to them might make no difference.’ Whereas it is often true that if no one responds to remarks, nothing will happen (as was the case here), attaching a name to comments does make a difference in another respect. As the example shows, it becomes easier for insiders and outsiders to see what exactly happened, to allocate credit to those who managed to see the flaws, and assign responsibility accordingly. Not only do the reviewers benefit from

13

the credit assigned to them; in fact, it has even made them rather famous. This example illustrates how opening up peer review’s “black box” in science by making the names of reviewers explicit could be favorable for the career of a scientist who dedicates the same amount of time to peer review as (s)he used to, but now is able both to, receive credit for it and, as well, to be held accountable in an appropriate manner.19

5.3 ‘Publish-then-filter’ versus ‘Filter-then-publish’ Living in an era when the internet and social media have increased the amount of information available on most any given topic through online ‘folk’ encyclopedias such as Wikipedia, as well as through blogs and open access, puts pressure on the traditional system of ‘peer review’. Moreover, an increasing scarcity of time and attention on people’s behalf and a projected increase of paper submissions from universities in developing countries require that we rethink the concept as we know it today. According to Kathleen Fitzpatrick, it is time to distance ‘peer review’ from its image as a kind of gatekeeper, as determining what should be published, and focus more on filtering the wealth of available information, or determining what, from among the vast amount of material that has been published could be of interest to people (Fitzpatrick, 2010: 172). A connected idea that could allow for such a shift is Clay Shirky’s ‘Publish-then-filter’-proposal: “Filter-then-publish, whatever its advantages, rested on a scarcity of media that is a thing of the past. The expansion of social media means that the only working system is publish-then-filter.” (Shirky, 2008: 98). This idea could function in academic scientific research and, in some respects, already does. It would in fact take a considerable load off of reviewers’ work and allow for means such as meta-analyses to rule out mistaken or disproven results later on. Besides, in academia one’s claims are very often rejected, making rejection, indeed, an intrinsic and accepted part of scientific research. Only on rare occasions are papers withdrawn from previous publication, even if they contain evident mistakes. The reasoning behind this is that these mistakes will eventually be filtered out in subsequent publications or meta-analysis. Academia functions in this manner and most everyone seems to be fine with it, making it seem clear that the ‘publish-then-filter’ idea has a certain academic ground on the basis of which it could be further developed. The IPCC’s job consists of bringing together all relevant information on climate change in one report that accurately represents current views on the matter. In the sense that it filters the multitude of available information, it fits nicely with Fitzpatrick’s claim that we would do well to distance peer review from gatekeeping. However, the ‘publish’ part of Shirky’s idea does not really apply to our example, as the papers brought together in the IPCC were already published beforehand.20 In this sense, the IPCC process does not resemble a

this (in terms of the due credit they receive for checking and questioning the reliability of information), but also the organization as a whole benefits (as its transparency is increased and responsible factors can be located, allowing it to deal with the matter more quickly). 19

One can easily imagine cases in which it would be best to conceal the name of the reviewer, such as when a paper is rejected on grounds that it lacks originality. However, this does not exclude the possibility of installing independent experts to assess whether the decision to reject was grounded or not. We will go no further into this matter here, as with the question of how to judge the originality and publication-worthiness of a paper is not at issue in our example here. As suggested, however, I do think that these two ways of understanding peer review can and should be separated. 20

Recall, here, the guidelines of the IPCC, noted in section 1.

14

traditional peer review process at an academic journal. We can, however, introduce a similar notion that does apply to our case: the distinction between ‘include-then-filter’ and ‘filter-then-include’. As opposed to Shirky’s proposal, what is new here is the inclusion of information, i.e. whether or not information should be regarded as part of the reliable body of evidence, as opposed to determining whether or not a report should be published. The follow-up question here is clear: ‘Does include-then-filter make sense for the IPCC, i.e. does the IPCC need to release the assessment report first, including all relevant information and only later deal with making up for possible mistakes, such as the Glacier melting mistake?’ If we recall our analysis and the real possibility of making human errors in view of the pressures attending the IPCC peer review procedure, we may be inclined to answer this question in the negative. In a situation where the smallest doubt is sufficient to hamper decision making, where people tend to have a completely mistaken conception of the uncertainty surrounding science and peer review (Oreskes & Conway, 2010), and where reliable sources upon which to base policy are needed more than ever, it would be best for the general public, for policy makers and for scientists if organizations like the IPCC stick to the strategy of ‘filter-then-include’, so long as they take into account the restrictions sketched out above. Given the societal impact of a topic such as global warming, one can ask for no less than an open transparent peer review procedure resulting (ideally) in a standpoint, or consensus, forged upon the best information available without disregarding valuable dissent. A combination of recognizing the limitations of peer review and devising or instituting an improved procedure accordingly will be our best available weapon in battling doubt and dealing with errors. Spreading confusion by prematurely regarding information as part of the core - i.e. by including information without sufficiently checking its reliability, then afterwards adjusting one’s claims as one goes along and throwing certain information out - is not, as a rule, compatible with the IPCC’s situation at the interface between science and society. The fact of this incompatibility illustrates an underlying discrepancy between academic scientific research and science at the border of public policy. The case considered in these pages shows that, whereas in the former case mistakes are (ideally) filtered out eventually, applying an ‘include-then-filter’ rule in the latter case could do more harm than good.21 Theoretically speaking, one can imagine an organization having to deal with flaws in their reports by acknowledging them and subsequently re-issuing a revised version of the same report, where the error is corrected. The release of this new version would imply that the organization no longer regarded the information in its original report to be part of the body of evidence, i.e. that it was no longer included. While this approach might be effective for less socially sensitive issues, it would not be so in the case of climate change, where doubt-mongers lurk around every corner and many non-epistemic factors are in play. Someone could counter this assessment, of course, by saying that this is what one should expect - flaws will eventually be made - and of course flaws will inevitably occur given the human aspect of the matter. Sadly enough, however, we cannot allow them to occur, despite the complexity of the situation. One epistemological consequence of organizational

21

The statement of Georg Kaser, an expert in tropical glaciology and lead author for the Fourth Assessment Report of the IPCC, in response to the 2035 prediction illustrates my claim nicely: “All the responsible people are aware of this weakness in the fourth assessment. All are aware of the mistakes made. If it had not been the focus of so much public opinion, we would have said ‘we will do better next time’.” (Carrington, 2010) At the boundary between science and policy, there seem to be requirements and relations at stake other than those to which scientists are accustomed in their ‘laboratory’ work.

15

scientific practice seems to be that error will often entail grave non-epistemic risks. A possible way to address this is to incorporate such risks into one’s assessment of a hypothesis, so that the requirements become even stricter than in cases of ‘laboratory science’. In other words, we may hope to avoid non-epistemic risks, such as doubt-mongering, by toughening our standards for the reliability of information even as we endorse a ‘filter-then-include’ strategy.

6. Conclusion Let me summarize briefly the path I have followed: In sections 1 and 2, I noted a discrepancy between two Assessment Reports issued by the IPCC and addressed the question ‘what happened, that the Himalayan glacier melting rate mistake got into the final report, despite the extensive peer review process?’ In section 3, I gave a detailed account of what happened. Review comments were raised but were not followed up. In section 4, I considered why this error came to be published, identifying a couple of mitigating circumstances. This detailed analysis allowed me, in section 5, to re-evaluate some aspects of peer review in scientific practice and identify some possible future routes:

- Although peer review is by no means perfect, it can ensure, if done properly, that the study being considered has had the benefit of independent scrutiny and quality control before it is published. The example we have considered shows that, even when procedures for dealing appropriately with peer review are in place, mistakes can be made: authors and editors may simply forget to incorporate insightful remarks, in part due to time and resource restrictions arising from the overwhelming amount of information and the appeal to grey literature. Peer review – rightly conceived as a means, however fallible, of capturing agreement amongst scientists rather than certainty – is our best available recourse in attempting to reduce the frequency of such errors.

- Peer review procedures would gain much in the way of credibility and reliability, as would the scientific (and broader) community, were the names of information reviewers to be public22. This is so, however, only on condition that comments are appropriately followed up, as opposed to what happened in the Himalayan glacier example.

- ‘Publish-then-filter’ and its companion ‘include-then-filter’, though an interesting route to pursue in academic research, would seem to promise more harm than good at the boundary between science and public policy. This illustrates the different relationships and requirements in play when comparing academic scientific research to research at the boundary between science and society, calling for an incorporation of non-epistemic risks in the assessment of hypotheses.

Further research on the matters treated here could investigate how the IPCC review procedure, and ‘peer review’ in general, might benefit from opening up the procedure to the public, i.e. letting stakeholders make contributions. One could ask how the underlying idea of science as a robust consensus relates to the ‘equal time’ debate in climate science. Another route would be to devise specific means and/or tools to deal with the trade-off

22

In this paper, I have argued for this only in the first sense of reviewing, i.e. of checking the reliability of information.

16

presented here. As this is work for future papers, I would end here just by citing the famous claim made by Winston Churchill, that: “Democracy is the worst form of government except for all those others that have been tried.” So unless peer review finds its match, let us make shift with what we have.

17

References

Bjurström, A., and Polk, M. (2011). Physical and economic bias in climate change research: a scientometric study of IPCC Third Assessment Report, Climatic Change, 108: 1-22.

Carrington, D. (20 January 2010), IPCC officials admit mistake over melting Himalayan glaciers, The Guardian. Available online: http://www.guardian.co.uk/environment/2010/jan/20/ipcc-himalayan-glaciers-mistake

Dalton, R. (2001). Peers under pressure, Nature, 413: 102-104. Fitzpatrick, K. (2010). Peer-to-peer review and the future of scholarly authority, Social

Epistemology, 24: 3, 161-179. Fowler, H.J., and Archer, D.R. (2006). Conflicting signals of climate change in the Upper Indus

Basin, Journal of Climate, 19: 4276. Fuller, S. (2002). Knowledge Management Foundations. Butterworth-Heinemann: Woburn,

MA. Hewitt, K. (2005). The Karakoram anomaly? Glacier expansion and the ‘elevation effect’,

Karakoram Himalaya, Mountain Research and Development, 25: 332-340. IAC (2010). Climate Change Assessments, Review of the Processes & Procedures of the IPCC.

Retrieved October 24, 2011, from: http://reviewipcc.interacademycouncil.net/report.html

IPCC, official website: http://www.ipcc.ch/index.htm IPCC (1995). IPCC Second Assessment Report. Available online:

http://www.ipcc.ch/pdf/climate-changes-1995/ipcc-2nd-assessment/2nd-assessment-en.pdf

IPCC (2007). Climate Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team, Pachauri, R.K. and Reisinger, A. (eds.)]. IPCC, Geneva, Switzerland, 104 pp.

IPCC (2008a). Procedures for the preparation, review, acceptance, adoption, approval, and publication of IPCC reports. Retrieved October 24, 2011, from IPCC: Official Website: http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf

IPCC (2008b). Annex 1: Tasks and responsibilities for lead authors, coordinating lead authors, contributing authors, expert reviewers and review editors of IPCC reports and government focal points. In IPCC, Procedures for the Preparation, review, acceptance, adoption, approval, and publication of IPCC reports. Retrieved October 24, 2011, from IPCC: Official Website: http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf

IPCC (2008c). Annex 2: Procedure for using non-published/non-peer-reviewed sources in IPCC reports. In IPCC, Procedures for the Preparation, review, acceptance, adoption, approval, and publication of IPCC reports. Retrieved October 24, 2011, from IPCC: Official Website: http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf

IPCC (2010a). IPCC statement on the melting of Himalayan glaciers. Letter put online on the official IPCC website on January 20th 2010.

IPCC (2010b). Statement on IPCC principles and procedures. Letter put online on the official IPCC website on February 2nd 2010.

IPCC (2010c). The role of the IPCC and key elements of the IPCC assessment process. Letter put online on the official IPCC website on February 4th 2010.

AUTHOR XXX

18

Kihara, H. (2003). The extension of peer review, How should it or should not be done?, Social Epistemology, 17: 1, 65-77.

Oreskes, N. (2004). Science and public policy: what’s proof got to do with it?, Environmental Science & Policy, 7: 369–383.

Oreskes, N. & Conway, E.M. (2010). Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. Bloomsbury Press: New York.

Pearce, F. (11 January 2010). Debate heats up over IPCC melting glaciers claim, The Guardian. Available online: http://www.newscientist.com/article/dn18363-debate-heats-up-over-ipcc-melting-glaciers-claim.html?DCMP=OTC-rss&nsref=online-news

Roy, R. & Ashburn, J.R. (2001). The perils of peer review, Nature, 414: 393-394. Shirky, C. (2008). Here comes everybody: The power of organizing without organizations.

Penguin Press: New York. Turner, L. (2003). Promoting F.A.I.T.H. in peer review: five core attributes of effective peer

review, Journal of Academic Ethics, 1: 181-188.