Home > Know how > Studies available on line > Travaux 2010 > Note de présentation : le facteur d'audience des périodiques scientifiques > Experimental measure of the Audience Factor of scientific journals

Experimental measure of the Audience Factor of scientific journals


Abstract
Contact
Background
Impact Factor and field diversity
Field-normalized Impact Factor
Influence measures
Audience Factor
Experimental version of AF
Conclusion
References

Abstract

The file presents an experimental measure of journal impact, based on a novel normalization of Garfield’s Journal Impact Factor (Thomson-Reuters). The field-dependence of impact levels is corrected on the citing side (the propensity and immediacy to cite in citation sources) rather than on the cited side as usual ex post field-normalization. The AF measure shown here does not depend on a particular field-nomenclature.

Presentation of the Audience Factor (AF) 


Tables FA v0.2 (2006) :

Journals ranked by decreasing AF (PDF)

Journals ranked by alphabetical order of journal titles (PDF)

Journals with incomplete information (PDF)



Articles

Contact

Michel Zitt : zitt(at)nantes.inra.fr or michel.zitt(at)obs-ost.fr


OST
93 rue de Vaugirard
75006 Paris
Tel: +33 (0)1 42 22 30 30
Fax: +33 (0)1 45 48 63 94

INRA- LERECO
Rue de la Geraudiere, BP 71627
F 44316 NANTES cedex 03
Tel: +33 (0)2 40 67 51 73
Fax : (+33) (0)2 40 67 50 74

 



The Audience Factor (AF) is a field-normalized, experimental variant of the Impact Factor of scientific journals.

Background

The reputation of journals is very unequal, referees and committees are the gatekeepers in charge both of scientific quality and conformity to the journal standards, with high rates of rejection in prestigious journals. A milestone of scientometrics was the quantitative measure of this reputation or notoriety, the Journal Impact Factor built by E. Garfield (1972) and issued in the Journal of Citation Reports (1975). It describes the academic visibility of a journal by the average citation of its articles.

The dissemination of an explicit hierarchy of journals among the scientific community, through the ISI’s Journal Citation Report (JCR), made clearer the terms of competition for the access to most visible media, and soon gained a great popularity in evaluation. It is not exaggerated to state that the impact factor played, and still plays, a key role in the self-organization of the scientific community. It is also the instrument that attracted the most criticism, firstly because of general issues of citation analysis it introduced on a large scale, secondly of some specific technical imperfections, but lastly and especially because of misuses in evaluation procedures, disregarding its characteristics and limitations. Some of these limitations are clear from the definition and largely documented in an abundant literature (see for example the review by W. Glänzel & H.Moed, 2002).

Let us only review the particular limitations of IF:
- IF is a citation average, which summarizes the unequal values recorded for individual papers in the journal. It is then quite dangerous, especially for evaluation purposes, to reduce the visibility of institutions or individual scientists to this average value which may strongly differ from their actual citations.  A classical reference on this point is Seglen (1997).  Among indicators directly complementary to the IF, the RCR "relative citation ratio" (A. Schubert & W. Glänzel, 1983) describes the intra-journal competition.
- IF is based on too short a window, a serious issue for "slow" disciplines. Variants of the impact factor based on a wider citation window (5 years instead of 2) are now available from Thomson-Reuters.
- IF is slightly flawed in terms of  "types of documents" present at the numerator and the denominator, a point made by H. Moed & T. Van Leeuwen (1995).
- IF does not control for differences amongst fields or types of research. Field-dependence, especially, is a recurrent question in citation analysis. Soon after impact factor was invented, scholars strove to propose a variety of “field-normalizations”, made easy if one picks the JCR field classification scheme (ca. 170 fields for hard sciences) as the reference.

To top

Impact Factor and Field Diversity

In its initial definition (E. Garfield, 1972), for a year t, the Impact Factor of a journal is the ratio of citations received to number of articles, calculated on the years t-1 and t-2. The standard impact factor of a journal in 2006, for example, is calculated as the ratio of the number of citations received by the journals' articles in 2004 and 2005, and of the number of articles in 2004-2005. It is a particular measure of average citations by article. Beyond the canonical IF published every year in the Journal of Citation Reports (ISI now Thomson-Reuters), numerous variants have been proposed, including the Thomson-Reuters' 5-years version.

The standard Impact Factor is a gross measure, sensitive to discipline effects. There are several reasons why the unevenness of the average IF is so large across scientific fields. A main reason is the difference in the propensity to cite, and to cite rapidly (within the citation window considered) across research communities. Differences in citing behavior stem from social habits of communities with respect to the degree of specialization and the structure of scientific issues within each field. A field that is generous and quick in its citations will tend to exhibit a higher impact factor than a field where the average length of bibliographies in citing articles is short and directed towards old articles, outside the citation window - for example in slowly evolving fields.

The other main cause of inequalities is the citation exchanges across domains. A field positioned as an exporter of knowledge or information will in principle gather citations from outside - and conversely for importers.

It may be noticed that the size of the field as such does not directly matter, a point made by E. Garfield in Current Contents in 1976 (see E. Garfield, 1999). If we take the limiting case of a closed field without exchanges of citation, both the numerator and the denominator of the IF are proportional to the size of literature of the field - under certain conditions of definition of citing and cited sets. The size of the fields might affect referencing behavior and thus have an indirect influence through other field features such as the degree of specialization.  In fact, at the subject category level or the aggregate discipline level, correlation between field size and aggregate field impact factor is quite low. If the field size does not influence the average impact, it affects the upper bound.

To top

Field-normalized Impact Factor

Field-normalized IF is obtained by a variety of methods, cardinal or ordinal. The simplest is the ratio to field average. These methods ex post, after calculation of field statistics, drastically reduce all causes of distortion: referencing habits, but also advantages in import-export or growth. Moreover, in the usual definitions, the normalization depends on how fields are delimited, then is classification-dependent. References on ex-post normalized IF may be found in the bibliography of articles cited below.

To top

Influence Measures

In 1976, G. Pinski and F. Narin (1976) proposed the influence weight, using an iterative weighting of sources of citation. For example a citation from Nature receives a greater weight than a citation from an average journal, because Nature is itself more cited. Similar indexes of influence (see references below) are now calculated using Google-like algorithms, for example C. Bergstrom's "eigenfactor" based on Thomson-Reuters data, Scimago index (F. de Moya-Anegon) based on Scopus data, and others such as Red Jasper index (A. Lim). In typical implementations, those algorithms also neutralized the across-fields discrepancies, but this effect is immersed in the reinforcement links of influence. Some versions of these indexes present questionable treatments of journal self-citations. However, influence measures are very promising to study knowledge flows.

To top

Audience Factor

The principle of the Audience Factor (M. Zitt & H. Small, 2008) is to correct one of the causes of inequality, the propensity to cite (and to cite rapidly) at the source, the citing side, rather than ex post on the cited side. The AF does not control other causes of inequality, differential growth amongst fields and citation transactions (import-export) between fields.

Technically, the AF normalization consists of weighting citations emitted by a journal: the weight is inversely proportional to the average length of the "active references" list in the citing community. "Active" means that only those references falling in the chosen citation window are considered. The above-mentioned article suggests options, depending on how the citing community is delineated.

One is to consider the citing journal itself as a micro-community, with the advantage of producing a classification-independent AF. The shortcoming of this approach is too narrow a view of communities and a fragile statistical basis: an ad hoc treatment of the tails of distribution is necessary to avoid on the one hand over-estimation of the references from scarcely citing journals such as trade journals, and on the other hand an under-estimation of the references from journals with very long bibliographies such as reviews journals. The examples given in the article (version 0.1 of the AF, citation window: 5 years) followed this option.

The second option mentioned consists in calculating the bibliography length on the field level rather than the journal level, a much more robust approach but missing the distinct advantage of independence regarding classifications systems.

The present version 0.2 (see below) is intended to make the best of both options.

To top

Experimental version of AF

Year 2006, 5 years citation window, excluding Social Sciences and Humanities

Compared to the two versions above, version 0.2 adopts an intermediary solution, which is likely to be rather robust, without ad hoc treatment, while retaining the advantage of a classification-free measure. The main difference between versions 0.1 and 0.2 is the basis of calculation of bibliography length. The new version estimates the propensity to cite in the community represented by the neighborhood of the citing journal A under scrutiny.  The neighborhood considered, defined in terms of journals, will be large if the domain is open and not dense, narrow if it is dense and closed. Allowing large variations of the neighborhood is a way to address, quite partially, the delicate issue of reference sets for normalization (M. Zitt, S. Ramanana, E. Bassecoulard, 2005).  A quite robust method based on length two paths was developed for defining the neighborhood in this v0.2 version.  It is ruled by the choice of a single parameter. The details of the method will be published later.

Data were provided by Thomson-Reuters and used by permission for the experiments. They bear on the year 2006.  In Table 1, journals are ranked by decreasing AF (except for journals with AF < 0.5, where the alphabetical order is used). In Table 2, rows are sorted alphabetically by title. The "active" references used in the weighting process correspond to a citation window of 5 years. Note that the 5 years window is chosen for reliability, but at the expense of a loss of journals not documented throughout the full period. The notation 'a' in 0.2a refers to an option concerning citation sources. In this option, citations to a cited journal (say B) from all available sources are counted and the weighting of citations from sources with incomplete information is extrapolated from sources with complete information citing B.  Information is typically incomplete for journals assigned both to SCI and SSCI-A&HCI (for example Econometrica) and several journals in computer science/communication. The ratio of all sources over complete sources is coded in Table 1 from *** (good) to * (fair). Data for 123 journals with a ratio superior to 2.6 are not considered significant, whatever the level of impact. The list of these is given in Table 3. The all science norm used for AF includes citations to these journals. In the option 0.2b (not shown) only citations from well-informed sources are counted, without extrapolation.

We stress the fact that this is an experimental version, subject to correction or improvement.

To top

Conclusion

As expected, AF and IF exhibit a fairly strong correlation. The across-domain part of variance is efficiently neutralized by AF, but not of course the inequality of journal visibility within their particular community. Moreover, in contrast with usual field-normalized IF, the advantage of trans-field visibility for generic journals present in the IF is kept in the AF measure.

Michel Zitt, January 2010

To top

References:

Literature on impact factor is abundant. We only mention below a few milestones and reviews where references can be found.

Impact Factor

  • Bordons M., Fernandez  M.T., Gomez I. (2004) Advantages and limitations in the use of Impact Factor measures for the assessment of research performance, Scientometrics 53, 2,  195-206 http://www.springerlink.com/content/e63nn06qd6h05p2t/
  • Dong P., Loh M., Mondry A. (2005) The impact factor revisited, Biomedical Digital Libraries, 2, 7 doi:10.1186/1742-5581-2-7 http://www.bio-diglib.com/content/2/1/7
  • Garfield E. (1972) Citation as a tool in journal evaluation, Science, 178, 471-479
  • Garfield E. (1999) Journal Impact Factors: a brief review, Canadian Medical Association Journal, Oct 19, 161 (8)
  • Impact Factor:  http://thomsonreuters.com/products_services/science/free/essays/impact_factor/
  • Glänzel W., Moed H.F. (2002) Journal impact measures in bibliometric research, state-of-the-art report, Scientometrics 53, 2, 171-193
  • Moed, H. F., van Leeuwen, TH. N. (1995), Improving the accuracy of the Institute for Scientific Information’s Journal Impact Factor, Journal of the American Society for Information Science, 46 : 461–467.
  • Seglen P.O. (1997) Why the impact factor of journals should not be used for evaluating research, British Medical Journal 314, 7079, 498–502.
  • Schubert, A., Glänzel W. (1983), Statistical reliability of comparisons based on the citation impact of scientific publications, Scientometrics, 5, 59–74.

To top

Influence Measures

To top

Audience Factor

v0.1

  • Zitt M., Small H. (2008) Modifying the Journal Impact Factor by Fractional Citation Weighting: the Audience Factor, Journal of the American Society for Information Science and Technology, vol 59, n°11, pp. 1856-1860. Article
  • Zitt M., Small H. (2008b) Field normalization of impact factors: a citing side approach, 10th International conference on Science & Technology indicators 17-20 Sept. Vienna (AUT)

v0.2

  • Zitt M., Citing-side normalization of Journal Impact: a robust variant of the Audience Factor (forthcoming in Journal of Informetrics, 2010) Article

Background

  • Zitt M., Ramanana-Rahary S., Bassecoulard E. (2005). Relativity of citation performance and excellence measures: from cross-field to cross-scale effects of field-normalisation, Scientometrics, vol 63, n°2, pp. 373-401. Article

Another method of citing-side normalization:
the SNIP (Source-Normalized Impact per Paper)

  • Moed, H.F. Measuring contextual citation impact of scientific journals, forthcoming in Journal of Informetrics, 2010 arxiv.org/pdf/0911.2632

A comparison

  • Waltman, L., van Eck, N.J., The relation between Eigenfactor, audience factor, and influence weight, forthcoming 2010 arxiv.org/pdf/1003.2198

Recent debates on impact normalization (forthcoming)

  • Opthof T., Leydesdorff L. Caveats for the journal and field normalization in the CWTS ("Leiden") evaluation of research performance, forthcoming in Journal of Informetrics, 2010 arxiv.org/pdf/1002.2769
  • Van Raan A.F.J., Van Leeuwen T., Visser M.S., Van Eck N.J., Waltman L.  Rivals to the crown: reply to Opthof and Leydesdorff, forthcoming 2010 arxiv.org/pdf/1003.2113

To top