Tracking areas of strategic importance using scientometric journal mappings

217

Tracking areas of strategic importance using scientometric journal mappings *

Loet Leydesdorff ‘, Susan Cozzens b and Peter Van den Besselaar ’ a Department of Science Dynamuzs, Nleuwe Achtergracht 166, 1018 WVAmsterdam, Netherlands b Department of Scrence & Technology Studies, Rensselaer Polytechnic Institute, Troy, NY 12180-3590, USA ’ Department of Social Science Informatics, Roetersstraat 15, 1018 WVAmsterdam, Netherlands

Final version received January 1993

In science pohcy, it is often important to track emerging developments: new fields, fast-changing areas that are the

focus of special funding efforts, or areas of growth or decline. This article presents methods to produce literature-based

indicators for such areas using journal-to-journal citations.

Using case studies of AIDS, superconductivity, and onco-

genes. we posit that the inclusion of a new journal can be used

as an indicator of structural change if the addition indicates

the emergence of a new journal category. Using the cases of

robotics and artificial intelligence, we illustrate the develop-

ment of areas chosen for priority funding. Again usmg artifi-

cial intelligence, we demonstrate the importance of construct-

ing even such simple measures of scientific performance as

publication counts using dynamic rather than constant journal

sets. Change in performance within a subfield can be system-

atically distinguished from change in the delineations among subfields over time.

Most of the features of science that are important for science policy making are in flux, but most of the indicators available to help decision makers use static categories (e.g. disciplinary delineations) to organize the data. The research reported here addresses that gap. In a previous publication [20], we have described methods for analyzing the structure of science using citations among journals. We have also illustrated [9] how such analysis can follow changes in researchers’

Correspondence to: L. Leydesdorff. Department of Science

Dynamics, Nieuwe Achtergracht 166. 1018 WV Amsterdam,

Netherlands. * We gratefully acknowledge the support of the National

Science Foundation (grant SRS-8810197) for the research

reported here.

Research Policy 23 (1994) 217-229 North-Holland

active perceptions of their environments. This kind of analysis shows promise as a basis for refining and updating not only literature-based data systems, but also the categories used in gathering personnel and funding data. In this article, we present some finer-grained indicators of scientific structures with particular policy sig- nificance, namely emerging areas and priority funding areas.

The obvious indicator of new developments in the sciences using aggregated journal data is the inclusion of new journals into the database. But what do the new journals indicate? In most cases they indicate the extension or further differentiation of existing journal clusters, although some- times they may indicate qualitatively new developments. Thus, in addition to the criterion of newness, we need a second criterion to distinguish when the introduction of a journal signals the emergence of a new journal category rather than the extension of an existing one. The first part of the article provides that criterion, and shows that the indicator matches in value in a number of obvious cases of emergence during the 1980s. We propose upscaling this indicator for the purpose of systematically tracking emerging areas.

In the second part of the article, we show how shifts in the internal structure of a priority funding area can be traced with similar techniques. As in emerging areas, one expects that priority areas are changing at an above-average rate. We use robotics and artificial intelligence as examples of priority funding.

By focusing on artificial intelligence as a case, we also discuss the implications of using dynamic

0048-7333/94/$07.00 0 1994 - Elsevier Science B.V. All rights reserved SSDI 0048-7333(92)00715-6

218 L. Leydesdorff et al. / Trackmg areas of strategic importance

instead of fixed journal sets for the scientometric measurement of performance, e.g. in terms of publication counts. To do evaluation, one needs the ex post perspective. Additionally, one is able to distinguish between prospective and retrospec- tive definitions of a field or specialty; correspond- ingly, the evaluator has options to backtrack using either the current understanding of the field or the actors’ perspective.

The problem of measuring change in a non-stable environment

Studer and Chubin [31, p. 2691 have noted that with all science indicators, one needs a baseline in order to be able to measure change. Among the various techniques for mapping the sciences with scientometric methods, journal-journal citation relations are one candidate for providing such a baseline [22,14]. The use of aggregated journal-journal citation data (derived from the Science Citation Index) is appealing for its trans- parency [9]. Aggregated journal-journal citations contain information about field and subfield structures at a sufficient level of aggregation for the construction of indicators relevant to the assessment of general and global S&T priorities. The mappings exhibit clear structures which can be shown to be more or less robust by using rather different methods [10,13,33].

Other science indicators, such as co-citations [29,30], provide a finer-grained mapping of scientists’ perceptions of the literature that is relevant to their research. Because of the lower level of aggregation, this data is expected to change more rapidly [28,14]. Co-word analysis is considered a more cognitive approach which results in pictures that are then translatable among fields and over time [6,7], and therefore qualify not as a baseline, but rather as an indicator of change.

As a baseline, however, journals alone are not sufficiently stable for making comparisons over time, since there is a yearly turnover of journals in the database [ill. One needs methods for distinguishing incremental change in this dataset from change in the structure of the dataset [161. With support from the Science Indicators Unit at the U.S. National Science Foundation, we have been engaged in a project concerned in part with creating baselines in the dataset of aggregated journal-journal citations.

Methods

Our data consist of journal-journal citation counts among all journals processed in 1984,1986, and 1988 by the Institute for Scientific Informa- tion. This data can be thought of as a grand matrix of ‘citing’ versus ‘cited’ journals. Since we were interested in structure and not in incidental citations, we discarded all cell values lower than 0.5% of a journal’s total, either ‘citing’ or ‘cited’, i.e. below the minimum of 0.5% of the margin totals for its respective row and column. Note that since the 0.5% threshold is applied equally to larger and smaller journals, smaller journals processed by IS1 were not removed. But the threshold allowed us to reduce the volume of the data with approximately 70% [cf. 201, so that further analysis could be pursued by using PCs.

Since we wished to work as much as possible on the basis of the information contained in the data, we did not make any ex ante distinctions among journals (e.g. review journals), nor did we align high values on the diagonal to expected values. This latter normalization for citations within the domain of a journal (cf. ‘self-citations’) has been proposed in the literature [23,25], but in our case it would artificially highlight underlying structure in the data, since normalization im- proves the correlations among citation patterns. In these various respects, our data are similar to those in the first data column of the printed editions of the Journal Citation Reports.

Our methods, about which we have reported more extensively elsewhere [20], are based on iterative use of factor or eigenvector analysis. First, the citation environment of an entry journal is limited to 50 journals in each case by raising the relative (percentage) threshold. This matrix is factor analysed. The journal that had the highest factor loading on the factor under investigation then becomes the starting point for the delineation of the domain. This procedure is repeated until input and output coincide. The resulting factor loadings can be considered as the best representation of the factor under study, and the journal with the highest loading on that factor can be treated as a ‘central tendency journal’.

While a central tendency journal can be considered as the best indicator for the centre of a local density (i.e. a cluster of points) in the multi-dimensional space, one can also use the

L. Leydesdorff et al / Trackmg areas of strategic importance 119

whole group of journals loading on a particular In other publications [9,20], we have focused factor as an ego for the delineation of the factor on applying these methods in more or less stable domain in a next round, and iterate the analysis fields and in cases of gradual change. In this until input is equal to output in terms of this article, we build upon our previous results in group. ’ This approach is particularly useful if order to address what we regard as the central one is interested in the delineation of a cluster question from a science and technology policy (e.g. in a case study), irrespective of knowledge perspective, notably whether and to which extent about the precise position of the centre of this one can use this baseline for the systematic mea- cluster in relation to other clusters. surement of rapid change.

The central tendency analysis tells us about local densities in the overall network. Central tendencies themselves can be compared among years, even though they are not necessarily tied to the same indicator journal, and each centre can moreover be indicated by more than one journal. The other method (i.e. using the group as the ego of a network) provides us with an opportunity to compare a specific cluster over the years, both in terms of the journals that compose it, and in terms of the relations of this cluster with the relevant citation environments. The two methods are related, since one has to specify a typical journal as a point of orientation during the iterative clustering. Actually, the two methods are specific forms of multi-variate (network) analysis at different levels of aggregation, which have been so defined in order to enable us to make comparisons over the years [cf. 321.

When do newly added journals indicate structural change?

The journals in the Science Citation Index are selected by IS1 using a set of criteria which are largely independent of science policy considerations, since the database is primarily an information retrieval tool that follows the market [ll]. Can change in this dataset be used as an indicator of emerging developments in the science and technology systems? As noted above, inclusion of new journals does not seem, a sufficient criterion for significant change, since one needs to distinguish between ‘newness’ within an existing group- ing in the data and ‘newness’ with reference to the categories themselves [17].

The grand matrix of journal-journal citation data is extremely sparse. If we represent it as an n-dimensional vector-space, this space is corre- spondingly almost empty with relatively small clouds of journals in higher densities. As noted, our methods focus on the stabilization of the representation of these densities by successive approximation, using factor analysis iteratively. Furthermore, the matrix of journal-journal citations is asymmetrical. (Journal A cites JournaZ B to a different number of times than uice Llersa.) Thus, there are two structures contained in each matrix: a ‘cited’ and a ‘citing’ structure. ’ Note that being ‘cited’ will refer to the whole archive of a journal under this name, while ‘citing’ refers to the current literature in each year. The ‘citing’ patterns are thus the indicators for the specific actions in each year [19].

’ The authors are grateful to Frank Wamelink, who wrote the program FIELD to perform this kind of analysis [37].

* For a representation of the two dimensions in one mapping. see [33].

In order to assess whether the categories have changed, one needs to consider the classificatory scheme as provisional, and thus perhaps in need of an update. An update is necessary if a newly added journal does not fit into any of the previously existing structures. In many cases, a new journal will find its place within an existing structure; incorporation may already have been achieved before the journal’s inclusion in the IS1 database. In the case of a journal in an emerging area, however, one expects that the environment may have to react to this event in terms of the organization of the network, since a qualitatively new signal is received. In general, whether a new pattern emerges, is decided at the level of the (receiving) network, and remains initially latent to the (sending) actor(s). The network has to accommodate to the new element of structure. Although the newly added journal may itself draw heavily on existing fields of knowledge in its ‘citing’ pattern, the pattern of citations to it is expected to be distinctive if it represents an emerging development [cf. 101.

220 L. Leydesdorff et al. / Tracking areas of strategic importance

Usually, the ‘cited’ position of a journal in the overall network is relatively stable, since it re- fleets citations to a whole archive of what the journal has previously published. But a new jour-

nal has no archive, and the development in the ‘cited’ dimension can therefore be used as an indicator of change. If the ‘cited’ pattern is signif- icantly different, we should be able to show a new

Table 1 Factor-analysis for AIDS Research, citing patterns (1986; 43 journals included with threshold = lSO’%‘c); ten factors extracted; Kaiser normalization: Varimax converged in nine iterations

Rotated factor matrix: Factor 1

V2685 0.94732 V5583 0.92195 V6575 0.89800 V1520 0.88411 V5229 0.83916 V561 0.83707 V4563 0.69944 v574 0.69870 V4689 0.50391 V569 0.35068

V1695 0.09189 v3714 - 0.00863 Vl686 0.06403 V4018 0.21913 V3894 0.43347 V858 - 0.00071 v1717 0.39130

v4592 V562 V758 V5292 v377 V4085 V298

- 0.04600 -0.01310 - 0.02490 0.00347 - 0.16843 - 0.03135

0.03579 0.11444 -0.08451 0.06370

0.01225 -0.01140 0.23271 0.10329

Factor 2 Factor 3 Factor 4 Factor 5 -

- 0.00013 0.00889 0.14414 - 0.03120 0.14706 0.08985 0.03569 - 0.05213 0.11056 -0.01310 0.13995 - 0.06090

- 0.01601 -0.11671 0.39505 0.11325 0.39974 - 0.08778 0.14396 0.30807

0.17341 - 0.00605 0.19101 0.03062 0.19658 0.03002 0.08555 - 0.00837 0.14835 0.02105 0.15141 0.17298

- 0.02896 0.10080 0.32334 0.03808 0.15813 0.06087 0.03660 - 0.07443

0.95303 0.08740 0.08953 0.13489 0.93498 0.14961 0.11124 0.13757 0.92413 - 0.00890 0.05846 0.17569 0.89523 -0.10528 0.18432 0.17670 0.78161 - 0.09494 0.18747 0.17123 0.52808 0.17835 - 0.12385 -0.16855 0.50329 0.29170 0.06964 0.03088

0.86578 - 0.03080 0.02038 0.86408 - 0.04667 0.02932 0.85635 - 0.07955 - 0.00431 0.85158 - 0.00706 0.05245 0.82621 - 0.08828 0.00492 0.74820 - 0.02005 - 0.03297 0.68702 0.10583 0.15722

V1487 V4613 V3212 V1483

V3126 V4036 V5024

v414 v352 V392

0.30620 0.04661 - 0.02589 0.31699 0.11179 - 0.04444 0.44518 0.21049 - 0.04104 0.00819 0.61746 -0.11277

0.02946 0.19633 - 0.08305

0.03076 0.12590 0.31116 0.20285 0.50773 - 0.09447

- 0.09169 - 0.08771

0.24191

- 0.08484 0.13487 0.04290 0.31225 0.35911 -0.05191

0.88730 - 0.01849 0.87790 - 0.00576 0.80643 0.02859 0.70027 0.10900

- 0.00649 0.86965 - 0.03649 0.82745

0.14575 0.73119

0.00930 - 0.07841 - 0.02633 -0.01312

0.11856 0.06531

V298 AIDS RES; V352 AM J CLIN PATHOL; V377 AM J MED; V386 AM J OBSTET GYNECOL; V392 AM J PATHOL; V414 AM J SURG PATHOL; V441 AM REV RESPIR DIS; V561 ANN INST PASTEUR VIR; V562 ANN INTERN’MED: V569 ANN NEUROL; V574 ANN NY ACAD SCI; V758 ARCH INTERN MED; V858 ARTHRITIS RHEUM; V1243 BLOOD; V1337 BRIT MED J; V1483 CANCER IMMUNOL IMMUN; V1487 CANCER RES: V1520 CELL: V1686 CLIN EXP IMMUNOL; V1695 CLIN IMMUNOL IMMUNOP; V1717 CLIN RES; V2544 FERTIL STERIL; V2685 GENE ANAL TECH; V3126 INFECT IMMUN; V3212 INT J CANCER; V3714 J CLIN IMMUNOL; V3715 J CLIN INVEST; V3894 J EXP MED; V4018 J IMMUNOL; V4036 J INFECT DIS: V4085 J LEGAL MED; V4154 J MED VIROL, V4563 J VIROL; V4592 JAMA-J AM MED ASSOC; V4613 JPN J CANCER RES; V4689 LAB INVEST; V4694 LANCET; V5024 MICROBIOL IMMUNOL; V5229 NATURE; V5292 NEW ENGL J MED: V5583 P NATL ACAD SC1 USA; V6188 RADIOLOGY; V6575 SCIENCE.

L. Leydesdorff et al. / Tracking areas of strategic Importance 221

central tendency in this dimension by using the journal as an entry into our analysis.

Inspection of the list of journals added to the database between 1984 and 1986, and between 1986 and 1988, provided us with three obvious cases of emergence which could serve as tests of these ideas. These three areas are: (1) the study of AIDS as a disease from the mid-1980s on- wards, leading to the inclusion of one new journal in 1986, and two new journals in 1988 with the term AIDS in their titles, (2) the discovery of high-temperature superconductivity in 1986, and the subsequent inclusion of a journal entitled Superconductor Science & Technology in 1988, and (3) two new journals with the word oncogene in their titles, which were both included in the 1988 journal set only, although the discovery of the oncogene was made in 1979. Each of these cases is well known as an emerging area in science. Does the proposed indicator match that prediction?

1. Journals with ‘AIDS’ in their titles

The incidence of AIDS challenged the science system from around 1985. 3 Indeed, the IS1 dataset contained no journals with the term AIDS in their titles in 1984. In 1986, a journal, entitled AIDS Research, was added, and in 1988, two journals with such titles (AIDS and AIDS Re- search and Human Retroviruses) were covered. We used these journals as single entries into a central tendency analysis, both in the ‘citing’ and in the ‘cited’ dimensions. As noted, we predict a new pattern emerging in the ‘cited’ dimension.

In 1986, the new journal AIDS Research exhibited a citing pattern which can be classified as typical for (internal) medicine journals. The highest factor loading for this journal was on a factor for which the JAMA-Journal of the American Medical Association was the central tendency journal. (Table 1 provides the reader with an example of the factor matrices used in the analy- ses below.) Other factors in the citation environment include ‘molecular biology’ and ‘clinical im- munology’. However, when studied in terms of the structure of citations to it, AIDS Research was itself a central tendency journal in 1986,

3 AIDS was first reported to the scientific literature m 1983. See [1.271.

loading highest on a minor factor in its citation environment and sharing the factor only with the journal Annals of Neurology.

In 1988, AIDS Research had been renamed into AIDS Research and Human Retrorjiruses, and a new journals with AIDS as its title was now also included. The titles already hint that the two journals belong to different fields, clinical attention to the disease on the one hand and the laboratory search for the virus on the other. By using factor analysis in the ‘citing’ dimension, we explored this differentiation in greater detail, using the pattern of references from the two journals to display structure. In this dimension, nei- ther AIDS nor AIDS Research and Human Retro- Lliruses was a central tendency journal. AIDS Research and Human Retrociruses placed itself within a ‘molecular biology’ factor led by Science and Nature and which also included Gene Analy- sis Techniques, Cell, and Embo Journal. AIDS, however, had its major loading on a factor with journals named for infectious diseases, indicating a clinical orientation. The factor analysis of citing patterns of these journals thus suggests that the two sides of the AIDS problem, i.e. the pre-clinical question of the identification and study of the virus and the clinical problem of AIDS as a disease, have been reflected from an early stage in the journal literature.

If we pursue the ‘cited’ patterns for 1988, however, both AIDS and AIDS Research and Human Retroviruses emerge as central tendency journals. When AIDS is taken as the entry, its cited pattern exhibits the highest factor loading on a factor on which also AIDS Research and Human Retrorliruses has its primary factor loading. The New York State Journal of Medicine also loads on this factor, indicating some clinical orientation. If AIDS Research and Humanan Retro- viruses is taken as the entry journal, a factor appears which correlates highest with this journal, and also with the cited pattern of the journal AIDS. As cited journals, then, both of these are the best indicators of their own central tendencies, thus fulfilling the condition for indicators of emergence specified in the hypothesis. Remem- ber that the same inference was true for the one new journal (AIDS Research) in 1986. Since AIDS research would clearly be widely accepted as an emerging research area in the 198Os, these results help validate the criterion.

222 L. Leydesdorff et al. / Trackmg areas of strategrc importance

2. High-temperature superconductivity

A second obviously emerging area strengthens the case. In April 1986, two scientists at the IBM Laboratory in Zurich, Switzerland, discovered superconductivity at higher temperatures than was previously held to be possible [3]. In 1987, these scientists received the Nobel Prize for the discovery. Within the science system, this discovery had a major impact on physics, particularly in the areas of condensed matter physics and solid state physics [36]. One would expect journals specializ- ing in high-temperature superconductivity to emerge in the years to follow.

Indeed, in 1988, a new journal Superconductor Science & Technology was included in the SCI database. We took this journal as an entrance point, for both a citing and a cited analysis, using the central tendency method described above. Through its references, the journal placed itself into the areas of solid state and condensed matter physics, 4 and also showed connections to a group of journals on the applied side. 5 However, in the ‘cited’ journal patterns, Superconductor Science & Technology emerged as a separate factor, with negative factor loadings on all other factors. The strong deviance in its cited pattern was due to the fact that the new journal was hardly cited by the journals to which it made reference, while all the other journals intensively cited one another. This may be a typical pattern in an emerging area of applied research, which immediately draws on several basic areas but develops its own network of cross-citation only slowly.

3. Oncogene

The oncogene, discovered in 1979, is the genetic code in normal cells which is similar to the genetic code of the tumor inducing Sarcoma virus [24]. The discovery has been of paramount impor-

’ Superconductor Science & Technology has a factor loading 0.89040 on a factor with 14 physics journals. among which

six with larger factor loadings, i.e. Phystcal Reutew B-Con- densed Matter (0.95437). Physica B (0.950631, Soltd State Commumcations (0.94611), Zeitschnft fiir Phystk B-Con- densed Matter (0.94067) and Physica C (0.92286).

s Among others: Applied Physics Letters, Journal of Applied Physics, and the Japanese Journal of Applted Phystcs.

tance for the development of cancer research, and has been honoured with the Nobel Prize.

Only in 1987, however, did a new journal with Oncogene as its title come onto the market place. In 1988, two journals with the term oncogene in their titles were included in the ISI database, Oncogene and Oncogene Research. When taken as entry journals and analysed in terms of citations to them, both journals are central tendency journals, and thus indicate ‘newness’ in the structural sense.

When analysed in terms of the citing side, the journal Oncogene was one among five central tendency journals of the molecular genetics factor in 1988. These five journals referred to one another as the best representation of this density in a loop-like structure. 6 Oncogene Research points to Oncogene as a better representative of this density.

Genes & Development is another journal that was newly added in 1988 when compared with the 1986 database. It plays a role in the same citation environment, but it exhibits its main factor loading on a Llirology factor. Genes & Development was thus comparable to Oncogene in the citing dimension, but it did not show the specific kind of cited patterns which we hypothesized above as specific for a qualitatively new development. Thus, we see that the proposed method made it possible precisely to distinguish between a new journal which was added to the database and indicated a new development, and an otherwise similar journal which was added to the database without this additional feature.

4. Summary

The instrument needed to distinguish systematically between newly emerging developments and gradual extensions in the existing structure of the database seems to be the specificity of the cited pattern of a new journal. In an upscaled version of our procedure, one could routinely check all newly added journals on this specific property. This would lead to a list of journals which have a significant effect on other journals’ citation behaviour upon inclusion in the database, and which would therefore merit further analysis.

’ We discuss this case more extensively in 1201.

L. Leydesdorff et al. / Trackmg areas of strategrc importance 223

Can one trace journal-sets as indicators of selected priority areas?

The second question we raise in this study is whether one can use journal sets to describe developments in areas selected for priority funding. As in emerging areas, one may expect that these areas are changing at an above-average rate, although not necessarily in terms of the inclusion of new journals. Can we also trace this ‘turbulence’ among journals using our methods, and if so, what do the results teach us about the areas under study?

In the methods section, we distinguished between research questions that refer to a group of journals as the unit of analysis, and questions that focus on the position of individual (e.g. newly added) journals. Additionally, one can raise the question whether the extension of existing groupings in the ‘citing’ dimension with newly added journals, points to journals exhibiting new patterns when analysed individually in the ‘cited’ dimension. In this section, we focus on the group level using the ‘citing’ dimension, but we shall include the discussion of new patterns in the ‘cited’ dimension when it informs the analysis.

As examples, we present data here on two areas of computer science, robotics and artificial intelligence, which experienced stormy developments in the late 1980s and which have strategic policy relevance. At the end of the time period our data cover, the U.S. National Science Foun- dation proposed robotics as an area in which government stimulation of an advanced technology could provide benefits to both government and the private sector. The change in budget for fiscal year 1991 was a dramatic +28%, as compared with +5% for ‘high performance comput- ing’, a second priority area. The analysis we re- port here describes the background against which NSF’s recommendation for priority funding was made. Artificial intelligence has likewise been treated as a priority area in the Netherlands. 7

1. Robotics

Over the period under study, four journals in the ISI database refer in their titles to robotics as

’ One of us was recently asked to make a quantitative assess-

ment of the state of the art in this area for a Dutch governmental agency [34,35].

their main topic. These are the International Journal of Robotics Research and the Journal of Robotic Systems which were both fully processed in all three years (1984, 1986, and 19881; Zndus- trial Robotics, which appears only as a cited journal; and the IEEE Journal of Robotics and Au- tomation, which entered the database in 1986. ’ Only the last journal qualifies as a new journal, but it does not meet the criterion we introduced above for an indicator of an emerging area. In 1988, when it received enough citations to appear in our analysis, it loaded onto an existing factor on ‘robotics’.

Our analysis of this area identified a group of journals associated with the area of robotics, and used this group, rather than a single journal, as the entry point to build citation environments in 1984, 1986, and 1988. 9 Over this time period, this environment showed three factors of interest: ones on automation and robotics, which appeared in all three years; and one on pattern recognition, which appeared for the first time in 1986 and then again in 1988.

Over the time period, the journals associated with the automation factor were relatively stable. The set of journals associated with the robotics factor, however, was disturbed somewhat by the emergence of the pattern recognition factor in 1986. In 1984, four journals loaded highest on the robotics factor: IEEE Transactions on Systems Man and Cybernetics, Acta Polytechnica Scandi- nacica -Ma thema tics and Computer Science Series, Journal of Robotic Systems, and the International Journal of Robotics Research. In 1986, however, when a new ‘pattern recognition’ factor appeared with IEEE Transactions on Pattern Analysis and Machine Intelligence as its central tendency journal, IEEE Transactions on Systems Man and Cy- bernetics and IEEE Journal of Robotics and Au- tomation loaded onto this new factor while exhibiting substantial factor loadings on the robotics factor as well. These two journals, then, form a

’ Additionally. IEEE Transactions on Robotrcs and Automa- tion was listed in the journal list of 1980, but this journal was not processed.

9 The FIELD analysis (see above and [37]) began with Inter- national Journal of Robotics Research, one of the two jour-

nals with the root robot in their titles that already appeared in our data for 1984. International Journal of Robotics Re- search emerged as the central tendency journal of the ‘robotics’ factor in 1988.

224 L. Leydesdorff et al. / Trackmg areas of strategic importance

Table 2

Journals loading on the factor with Artificral Intelligence as

central tendency indtcator in 1984, 1986. and 1988, respec-

tively

1984

Artificial Intelligence

Communication & Cognition

Futures Journal of the American Society for Information Science

emphasizing the different focus of the factors. By 1988, the cross-loading of journals has disap- peared, indicating that the separation of the two areas appeared to be more complete. The distinct identity of the robotics factor also appeared better established, as the International Journal of Robotics Research became the central tendency journal.

1986

Artificial Intelligence

Thought

ACM Transactions on Office Information Systems

Computer Industry

Annual Review of Computer Science Betriebswirtschaftlichte Forschung und Praxis

I988 Artificial Intelligence

AI Magazine IEEE Circuit Device

IEEE Expert

Journal of Philosophical Logic

Acta Psychologica

Daedalus

Cognitive Science

In short, just before the time when robotics was declared a priority area by the National Sci- ence Foundation, the literature on automation was already in a state of flux, with the emergence of a journal literature on pattern recognition and the consolidation of the specialty of robotics. Data from the time period we had available do not indicate whether NSF’s priority-setting put it on a surfboard riding a powerful wave or whether the wave had already passed it by, but an extension of the analysis, with more detailed attention to what NSF had already supported and subsequently supported under the funding surge, would certainly be able to shed light on that issue.

Progress in Physical Geography

Journal of Molecular Graphics 2. Artificial intelligence

bridge between the areas. In contrast, IEEE Our second illustrative priority area is artificial

Transactions on Pattern Analysis and Machine In- intelligence. According to our indicators, this area

telligence loaded negatively on the robotics factor, is much more turbulent than robotics in the late

Table 3 The group of Journals around Artrfictal Intelltgence, stabilized in 1984, 1986 and 1988 using the program FIELD

1984 iteration 10 9 8 7 6

International Journal of Man-Machine Studies * * *

ACM Transactions on Office Information Systems *

Artificial Intelligence * * * * *

Human System Management * *

Journal of the American Society for Information Science *

1986 iteration 10 9 8 7 6

Artificial Intelligence * * * * *

Communication * * *

Comput Ind * * * * *

Thought * * * * *

ACM T Offic Inform S * 1; * * *

1988 iteration 10 9 8 7 6

Artificial Intelligence 1 * * * *

AI Magazine * * * * *

International Journal of Intelligent Systems * *

IEEE Expert * * * *

International Journal of Man-Machine Studies * * * * *

Daedalus * *

Journal of Philosophical Logic * * *

Journal of Logic Programming *

L. Leydesdorff et al. / Tracking areas of strategic mportance 225

1980s. To study this area, we used the journal Artificial Intelligence as an entry point in all three years of data, since it was the only journal with the term ‘artificial intelligence’ in its title that was included over the whole time period in the SCZ database. Experts generally confirmed that this was the leading journal for discussions on the subject.

In the citing dimension, Artificial Intelligence was a central tendency journal in each of the three years, but the set of other journals loading on its dimension was different in each of the years (see Table 2). In addition, using the FIELD program, we could not stabilize a group of journals that represented this area in either 1984 or 1988, even after ten iterations (which means some 15 or so hours of computation on a fast PC). In 1986, stabilization was achieved, but again the relevant journals did not overlap at all with those of previous years (see Table 3).

By using either the cluster method or the central tendency method, the results for 1988 pointed to two newly added journals which exhibited highest factor loadings on the dimension indicated by Artificial intelligence, i.e. AZ Maga- zine and IEEE Expert. lo AI Magazine became additionally one of the central tendency journals in the cited dimension, and thus qualifies as an indicator of an emerging area, using the criteria articulated in the previous section. The new con- stellation of three journals points to the emergence of an artificial intelligence literature in 1988.

Performance assessment in a changing journal set

As in the other cases, perhaps even more urgently so, the results for ‘artificial intelligence’ raise the question of how to measure perfor-

1” A fourth journal which joins the core group in 1988 using

either method, is Daedulus. Daedalus. however, is a general science journal of the American Academy of Arts and

Sciences. In 1988, it happened to publish a theme issue on artifictal intelligence, in which all the major figures of this

specialty published. This led to an artifactual but meaning- ful citing pattern in this area for this year only [cf. 8,191.

mance using traditional publication and citations measures in a journal set which exhibits structural instability. If the journal set were more stable, one might feel comfortable about making assumptions concerning a baseline in terms of a journal set fixed a priori, but as soon as journal structures begin to differentiate and to extend, one must increasingly question results based on this assumption.

How can one select a journal set to measure changes in performance in a changing environment? By definition, the journal set that describes such a situation cannot be chosen ahead of time, but instead must be selected with hindsight, after the change has occurred. Thus, the analyst who wishes to make a statement about performance according to the current understanding of the field should choose the content of the category according to current, i.e. a posteriori, standards, and backtrack on the basis of this hypothesis.

The above analysis of ‘artificial intelligence’ can be used to illustrate this point. On the basis of the 1988 data three journals were designated as the core of ‘artificial intelligence’. On the basis of 1984 data, either a completely different set would have been chosen or only the one journal Artificial Intelligence. But we know that papers in journals which were closely related to Artificial Intelligence in 1984 (e.g. JAILS), have to be classified with hindsight to specialties and subspecial- ties other than artificial intelligence.

Similarly, in the case of robotics discussed above, IEEE Transactions on Systems Man and Cybernetics was the central tendency journal in 1984, while it had lost this position in favour of the International Journal of Robotics Research in 1988. On the basis of an a priori delineation of the journal set (in this case, in 1984) one would therefore come to a different weighting of articles in these journals for the development of robotics than on the basis of an a posteriori delineation.

The use of a journal set that is fixed ex ante for statistical performance measurement is therefore ill advised in the case of a research question which looks for, or even wishes to take account of’ structural change over time. The a posteriori perspective enables us to interpret results based on different journal sets. It also allows us deliber- ately to choose a ‘fixed journal set’ in some research designs, but not in others.


-USA A __-- _& +

---- Japan 0 EEC 0

1963 IQ64 1965 1966 1967 1986 1989 1990

Fig. 1. Geographical distribution of publications in ‘artificial intelligence’ using a ‘fixed and a ‘dynamic’ journal set, respectively.

Markers indicate publications in Artificial Intelligence only; lines indicate values for extension with AI Magazine and IEEE Expert in 1987.

Comparisons among nations in the field of ‘artificial intelligence’

Let us apply these ideas to the delineation of ‘artificial intelligence’ as discussed above, and compare the national contributions for the United States, the EEC countries, Japan and Canada, the major national actors in this field, using the different options.

In the previous section, we defined this specialty ex post in terms of a journal group consist- ing of Artificial Intelligence, AI Magazine, and IEEE Expert. However, the latter two journals entered into the SCZ database only in 1987. If one wishes to analyse the period under study on the basis of the ex ante set based, for example, on the 1984 data, only the journal Artificial Intelli- gence would fully qualify to be in the category. As noted, the addition of other journals on the basis of the citation pattern in this year only would worsen the selection in later years, since these journals could no longer be considered as core journals for artificial intelligence in these later years on any of the criteria used in this study. Thus, we compared the choice of Artificial Intelli- gence alone, as the best choice of a fixed domain, against the domain defined a posteriori, i.e. with the addition of the two new journals for the later years. Does this choice make a difference, and if so, what?

In Fig. 1, the percentage world share for the United States, the EEC, Japan and Canada are shown for the period 1983-1990. The lines in the figure are based on the values for the changing

journal set, while the points scattered over the figure are based on the values using only Artifi- cial Zntelligence as the domain. Of course, the two representations coincide until 1987. l1

The differences among the results are not large, but they are systematic. The plot shows us first that the United States has a dominant position in this specialty. However, in the later years, the contributions of the EEC and Canada are substantial (with a maximum of 20% for the EEC in 1990). The contributions of the EEC and Canada, however, are systematically underestimated if one uses the one journal Artificial Intel- ligence in comparison with the results in the dynamic journal set. The newcomer nations do relatively better in the newly added journals. (In terms of the figure, the points representing the fixed domain appear systematically below the cor- responding values for the lines.) Initially, this effect occurs at the expense of the US contribution, but over time the larger population seems to adapt to the new relations of competition.

If we now open the black box of the EEC and distinguish among the 12 member countries, we find a similar effect repeated on a smaller scale.

*’ The percentages do not necessarily add up to 100, since there are also publications from other countries (e.g. Swe-

den). Internationally co-authored publications are counted as integer points for each nation of origin. The method of

fractional counting can be used as a further refinement, but requires inspection of the publications at a lower level

of aggregation (cf. 12,151). We followed Braun et al. [4,51 in limiting the counting to only articles, reviews, letters, and

notes.

L. Leydesdorff et al. / Tracking areas of strategic importance 227

Only seven of the 12 member countries have relevant publications, and among them Portugal has only one relevant publication. Since one publication cannot indicate much change, we limited the analysis to the United Kingdom, France, the Federal Republic of Germany, Italy, Holland, and Belgium. The graph in Fig. 2 represents again the fixed journal set assumption as points, and the dynamic extension to three journals in 1987 as lines.

In the figure, some points clearly lie above the line. These points represent the three major countries exclusively (the United Kingdom, France and Germany), although not every major country exhibits this effect in every year. Thus, using the fixed journal set, the contribution of the smaller countries is systematically underestimated when compared with using the dynamic journal set.

Of course, this empirical result may not gener- alize to other specialties. What the case illus- trates, however, is that systematic effects are likely to occur by using different definitions of domains. This would be expected based on what we know about journals, since different journals have lower thresholds for different constituencies and reach different audiences. Active scientists are thor- oughly aware that journals have specific flavours of style and nationality (or nationalities), some- times even despite international and cosmopoli- tan allure. The emergence of new journals in a field increases the choices authors have in regard to their publication profiles [21]. Thus, for policy purposes, it is always necessary to specify the

+u<- 0 Franc A FRG

relevant journals with respect to the specific evaluation question to be answered. As we argued a few paragraphs above, in the case of (statistical) performance measurement, this will usually mean that the journal set has to be chosen with hindsight.

Summary and conclusions

The common thread among the different parts of this study has been that the huge communication network of aggregated journal-journal citations changes in two respects: first, it changes in terms of primary information - i.e. the cell values of the citation matrix - and second, in terms of its latent structure [cf. 12,261. The two types of change cannot be distinguished in terms of the observed events, unless the analyst has analyti- cally distinguished them in terms of expectations. We emphasized the importance of proper attention to change in the second (structural) dimension of the information, since otherwise one may fail to understand the dynamics of the system. If one does not systematically account for redelin- eation in the groupings over time but uses ‘fixed journal sets’ instead, one risks making a prediction of performance with reference to an out- dated unit.

In the first part of the study, we focused on the inclusion of new journals as potential indicators of important developments in the scientific communication system. We hypothesized that one can use the ‘cited’ pattern of the newly added journal

1986 1987 1988 1989 1990

Fig. 2. Distribution of publication in ‘artificial intelligence’ for six EEC countries using a ‘fixed’ and a ‘dynamic’ journal set, respectively. Markers indicate publications in Artificial Intelligence only; lines indicate values for extension with AZ Magazine and

IEEE Expert in 1987.


as an indicator of structural change, since the new journal is not yet stabilized in its being ‘cited’ pattern on the one hand, and it has to be accommodated by its citation environment on the other. The ‘central tendency method’ provided us with a statistical criterion to assess whether the newly added journal was an indicator of a newly emerging pattern of citation or not in the year of its inclusion. In more abstract terms, this criterion evaluates the effect of the inclusion on the second (structural) dimension of the information contained in the network.

In the three cases that we subsequently discussed in more detail (AIDS, superconductivity, and oncogenes), we could show that the indicator journals for the underlying new developments exhibited the predicted property in the ‘cited’ dimension. Additionally, in the case of the highly comparable journals Oncogene and Genes & De- velopment, both added in 1988, the one journal could be distinguished from the other on the basis of this criterion.

In the latter part of the study, we found that AI Magazine also exhibited the hypothesized property. In this case, the inclusion of the new journal seemed to stabilize a structural element that had been unstable in the citation network (i.e. the cluster of Artificial Intelligence). Further- more, in the robotics case, we saw a much more complex pattern of change and stabilization, in which the emergence of a neighbouring field (i.e. pattern recognition) was crucial for the more pronounced differentiation. We are not allowed to reverse the reasoning in stating that structural change leads to the inclusion of new journals indicating new patterns. However, it seems worthwile to routinize our analysis method by applying it to all journals that are newly added to the database in a certain year. This method leads to a list of journals that have an impact on the communication network, and that merit further policy analysis for this reason.

In the second part of the study, we elaborated our concept of structural change for the case of priority funding areas. In both cases (robotics and artificial intelligence) relevant journal groups were initially present in the period under study. However, the areas were going through important transitions, and the study of these transitions in terms of structural positions of groups of journals

allowed us to specify why it is important in general to use dynamic journal sets.

We have shown in this study that one can replace taxonomic journal classifications with evolutionary models. If the system is not in equi- librium, but developing, any classification system remains a hypothesis for the next round(s) of operation. If the classification does not ‘learn’ from the operation of the system, it makes itself obsolete in short order [9]. This reasoning holds true for any taxonomy (e.g. a thesaurus, classification, index system, etc.) which is not defined in terms of its co-evolution with the system it is supposed to monitor reconstructively. An evolutionary perspective differs from a taxonomic perspective, in that a taxonomy is (usually) hypothesized ex ante, and not capable of signalling relevant change other than as disturbance or unspe- cific noise; the need for an update is not systematically derived from change in the data over time. In an evolutionary model, the classification remains provisional, based on the information currently available, and is reflexively updated and improved with the operation of the system under study.

The shift to an evolutionary model, however, affects not only the methodologies, by changing the perspective to an a posteriori one, but also the theoretical understanding of what one measures with scientometric indicators. As depictions of a science system under study, scientometric mappings should never be reified. They contain only a pattern which the analyst can propose hypothetically for the further analysis of, for example, performance. By using the case of artificial intelligence we have shown empirically that the choice of the a priori or the a posteriori perspective has important consequences for scientometric applications like publication counting in statistical performance measurement.

In general, then, the kind of analysis we describe here can contribute to the development of scientometric tools with policy relevance. Unlike the static categories that appear so often in national indicators reports, indicators based on this dynamic form of analysis can help science policy decision makers with the very real problems of responding at a national level to rapid change in science.

L. Leydesdorff et al. / Trackzng areas of strategic importance 229

References

[ll The Chronology of AIDS Research, Nature 326 (2 April 19871 435-436.

[2] J. Anderson, P.M.D. Collins, J. Irvine, P.A. Bard. B.R. Martin, F. Narin and K. Stevens, On-line Approaches to Measuring National Scientific Output - A Cautionary

Tale, Science and Public Pohcy 15 (1988) 153-161.

[3] G.J. Bednorz and K.A. Miiller, Possible High Tc Super- conductivity in the Ba-La-Cu-0 System, Zeitschrzft fur Physzk B 64 (19861 193-198.

[4] T. Braun, W. Glanzel and A. Schubert, Assessing Assess- ments of British Science. Some Facts and Figures to

accept or Decline, Scientometrzcs 15 (19891 165-170. [51 T. Braun. W. Glinzel and A. Schubert. The Bibliometric

Assessment of UK Scientific Performance - Some Com-

ments on Martin’s Reply, Sczentometrics 20 (19911 359- 362.

[61 M. Callon, J.-P. Courttal. W.A. Turner and S. Bauin, From Translations to Problematic Networks: An Intro-

duction to Co-word Analysis, Soczal Sczence Information 22 (1983) 191-235.

[7] M. Callon, J. Law and A. Rip (Editors), Mapping the Dynamics of Sczence and Technology (Macmillan, Lon- don, 1986).

[81 S.E. Cozzens, What do Citations count? The Rhetoric- first Model, Scientometrzcs 15 (1989) 437-447.

191 S.E. Cozzens and L. Leydesdorff. Journal Systems as Macro Indicators of Structural Change m the Sciences,

in: A.F.J. van Raan (Editor), Proceedings of the Joint EC/Leiden Workshop on Science and Technology Indzca- tars (DSWO, Leiden, 1993).

1101 P. Doreian and T.J. Fararo. Structural Equivalence in a Journal Network, Journal of the American Soczety of Informatzon Sczence 36 (1985) 28-37.

1111 E. Garfield, How IS1 selects Journals for Coverage: Quantitative and Qualitative Considerations, Current Contents (28 May 1990) 5-13.

1121 K. Krippendorff. Informatzon Theory. Structural Models for Qualitatiue Data (Sage, Beverly Hills, 1986).

1131 L. Leydesdorff, The Development of Frames of Refer- ence, Sczentometrics 9 (1986) 103-125.

1141 L. Leydesdorff. Various Methods for the Mapping of Science, Scientometrics 11 (19871 291-320.

1151 L. Leydesdorff, The Prediction of Science Indicators Using Information Theory. Scientometrics 19 (1990) 297- 324.

1161 L. Leydesdorff, The Static and Dynamic Analysis of Network Data Using Information Theory, Social Net- works 13 (19911 301-345.

[171 L. Leydesdorff, Irreversibilities in Science and Technol-

ogy Networks: An Empirical and Analytical Approach, Scientometrzcs 24 (1992) 321-357.

1181 L. Leydesdorff, The Impact of Citation Behaviour on Citation Structure, in: A.F.J. van Raan (Editor), Proceed-

zngs of the Joint EC/Lezden Workshop on Science and Technology Indicators (DSWO. Leiden, 19931.

[19] L. Leydesdorff and 0. Amsterdamska, Dimensions of

Citation Analysis, Sctence, Technology and Human Values 15 (19901305-335.

[20] L. Leydesdorff and S.E. Cozzens, The Delineation of Specialties in Terms of Journals Using the Dynamic Journal Set of the Science Citation Index, Scientometrzcs 26 (1993) 133-154.

[21] L. Leydesdorff and P. Van der Schaar, The Use of Sctentometric Methods for Evaluating National Research

Programs, Science & Technology Studzes 5 (19871 22-31. [22] F. Narin, Ez,aluatiz’e Biblzometrzcs (Computer Horizons,

Inc., Cherry Hill, NJ, 1976).

[23] E. Noma, An Improved Method for Analyzing Square Scientometric Transaction Matrices. Sczentometrics 4 (1982) 297-316.

[25] D. de Solla Price, The Analysis of Square Matrices of

Scientometric Transactions, Scientometrics 3 (19811 55- 63.

[24] H. Popperman, A.D. Levinson, H.E. Varmus, L. Levin- tow and J.M. Bishop, Uninfected Vertebrate Cells contain a Protein that is closely related to the Product of

Avian-Sarcoma Virus Transforming Gene (SRC), Pro- ceedings of the National Academy of Sciences 76 (1979)

1804-1808.

[26] C.E. Shannon, A Mathematical Theory of Communica- tion, Bell System Technzcal Journal 27 (19481 379-423. and 623-656.

1271 R. Shilts, And the Band Played On (St. Martin’s Press, New York, 19871.

[28] H. Small and E. Greenlee, Collagen Research m the 197Os, Scientometrzcs 10 (1986) 95-l 17.

[29] H. Small and E. Sweeney, Clustering the Science Citation Index Using Co-Citations I. A Comparison of Methods,

Scientometrics 7 (1985) 391-409. 1301 H. Small, E. Sweeney and E. Greenlee, Clustering the

Science Cttation Index Using Co-Citations II. Mappmg

Science, Scientometrzcs 8 (19851 321-340. 1311 K.E. Studer and D.E. Chubin, The Cancer Mission. Soczal

Contexts of Bzomedical Research (Sage, Beverly Hills, 19801.

[32] H. Theil, Statistical Decompositzon Analysis (North-Hol- land, Amsterdam/London. 1972).

[33] R. Tijssen, J. de Leeuw and A.F.J. Van Raan, Quasi- Correspondence Analysis on Square Scientometric

Transaction Matrices, Scientometrics 11 (19871 347-361.

1341 P. Van den Besselaar, Kennissystemen. Maatschappelijke kansen en rzszco’s (NOTA, The Hague, 1991).

[35] P. Van den Besselaar and L. Leydesdorff. De Neder- landse bijdrage aan de Kunstmatige Intelligentie. Infor- matie & Informatiebelezd 10 (2) (1992) 54-61.

[361 J. Vlachy, Informationsanalyse zur Supraleitungsfor- schung, Physikinformatzonen 14 (1988) 15-49.

1371 F. Wamelink, A Quantitative Strategy for the Delin-

eation of Scientific Fields (in preparatton).

Tracking areas of strategic importance using scientometric journal mappings

Documents

Transcript of Tracking areas of strategic importance using scientometric journal mappings