ABSTRACT
Several studies have argued that open access (OA) journals have higher citation rate than non-OA journals (subscription-based journals). However, there are still controversial discussions regarding the OA citation advantage. Considering the difference between the diffusion factor and the journal impact factor (JIF) in measuring scientific impact and its potential for a new insight in understanding the multidimensionality of impact, this study aims to examine and compare the journal diffusion indicator values of OA and non-OA journals to test whether or not OA journals would have wider impact than non-OA journals. The data used in this study were obtained from the three-year Web of Science data (2015, 2014, and 2013) in the areas of general internal medicine. The results reveal that unlike this study’s propositions, the OA journals do not have diffusion advantage than the non-OA journals. This result contradicts several prior citation advantage studies which focus on the citation benefits of OA journals. This study contributes to the controversial OA citation advantage discussions in the prior studies.
Key words: Open access, citation advantage, journal diffusion factor, journal impact, citation benefit.
In modern society, communication system has dramatically changed with the development of internet technologies, and this change has also influenced knowledge sharing system in scholarly communication. Open access (OA) has emerged on the basis of free accessibility to enhance wide readership and thus strengthen scholarly impact. Since its implementation, several bibliometric studies have proved its strength that OA journals have higher citation rate than non-OA journals (subscription-based journals) (Antelman, 2004; Eysenbach, 2006). However, there are still critiques against these research findings with positive correlation between open accessibility of journals and their citation rate. There are also many countries that have very low usage rate of OA journals. In addition, all of the previous bibliometric studies on OA use simple citation rate or journal impact factor to test its advantage.
While the citation rate and journal impact factor could indicate a certain dimension of scholarly impact called the strength of impact, they fail to indicate the breadth of impact. Considering the fact that scholarly impact is a multi-dimensional concept, there is still a research gap in the comparison between open and non-open access. Regarding the breadth, there is a useful scientometric indicator with journal diffusion factor (Frandsen et al., 2006; Rowlands, 2002).
The journal diffusion factor was first introduced by Rowlands (2002) and revised by Frandsen (2004) as a measure for journal’s trans disciplinary reception. This is a method of representing the breadth of influence of journals throughout the literature. This indicator considers journals as a diffusion unit. Rowlands (2002) uses a ripple effect metaphor as follows:
while the journal impact facto (JIF) intends to measure the size of the splash (an impact) when pebbles (articles) are thrown into water (a scientific community), it does not capture the extent of the ripples (diffusion) flowing from a particular journal. He then presents the ripple measurement that measures the breadth of a particular journal reception. In the case of scientific influence, journal diffusion factor shows distinct examples from the JIF.
A journal with both high scores of JIF and the diffusion factor has a large number of citations received and its citation activities are more evenly distributed across the citing journals. In the same vein, a journal with low JIF and high diffusion score has a small number of citations but its citation activities are more evenly spread out. Considering the difference between the diffusion factor and the JIF in measuring scientific impact and its potential for a new insight in understanding the multidimensionality of impact, this study aims at examining and comparing the journal diffusion indicator values of OA and non-OA journals to test whether or not OA journals would have wider impact than non-OA journals. To achieve this purpose, this study has the following research questions:
RQ1. Is there a significant difference in the diffusion factor depending on whether the journal is OA or non-OA?
RQ2. Can OA journals have a higher diffusion factor than non-OA journals?
The following procedure includes literature reviews on OA and journal diffusion factor, Web of Science citation data collection process, analysis and comparison using the diffusion factor, and discussion for the analysis results.
OA and citation advantage
OA journals are online journals that are freely offered to the readers requiring nothing else but the Internet access. The full contents are open to any readers in the world and it also has peer review process to control the quality. There are different types of OA journals (for example, journals with entirely OA, journals with some OA, and journals with delayed OA) but all of these stresses open scholarly communication by making it possible for readers to access academic articles from publishers directly. OA journals have grown over the past decade. According to a representative of OA resource (directory of OA journals (DOAJ)), the number of OA journals indexed have been increasing from 1,396 to 10,074 during a decade, an increase of over 7 times.
There have been various studies on the correlation between OA and citations. Based on pilot finding of the present study, about 80% of these studies report the positive citation advantage of OA while the other 20% report shows that there are no statistically significant citation differences between open and non-open modes. However, none of these used journal diffusion factor tried to measure different dimension of journal impact. Regarding the positive citation advantage, Lawrence (2001) observed that freely accessible articles are cited more and should be used and accessed by authors and publishers for higher impact and more rapid scientific advancement.
Kurtz et al. (2005) also found this positive relation between OA journals and citations. As an early comparison between open and non-open modes, Antelman (2004) examined various OA adoption periods in the disciplines of philosophy, politics, electric and electronic engineering, and mathematics, and concludes that all of the freely accessible articles have a higher research impact and non-open-access articles. Similarly, Eysenbach (2006) also found that papers published in an immediate access article on a journal site had a higher impact than non-open-access articles that were self-archiving or publicly accessible. OA articles, in particular, are faster than non-open-access articles and are more likely to be cited by other peer researchers. More recently, the positive citation advantage was advocated by several fields.
In political science, Atchison and Bull (2015) compared mean citation rates between OA and subscription-only articles from the same journals. In the field of cytopathology, Frisch et al. (2014) compared citations to articles by the same authors (314 publications by 28 authors) published in OA and non-open-access journals. The analysis of 2,026 articles (442 OA) published in 2007 in 14 journals in the field of civil engineering also reported the advantage (Koler-Povh et al., 2014).Conversely, some studies found that there was no citation advantage for OA. Comparing 2,012 OA journals and 16,121 subscription journals longitudinally, Solomon et al. (2013) concluded that Source Normalized Impact per Paper (SNIP2): a field-difference-adjusted impact factor) averages for OA journals were roughly equal to or lower than those for subscription journals, whilst those for other OA journals were still lower.
In the field of environmental science, Vanclay (2013) indicated no significant correlation between open accessibility and citation rates. Vanclay (2013) collected 131 publications by members of the School of Environmental Science and Management at Southern Cross University during 2006 to 2007 and tested Pearson correlations between OA status and citations per year. Björk and Solomon (2012) found that average citation rates were about 30% higher for subscription journals than for OA journals. After controlling for discipline, age of journal, location of publisher, and the difference were removed for most subcategories.
Journal diffusion factor
Rowlands (2002) first introduced the concept of diffusion factor and made it possible to grasp the ripple effects and dynamic citation information of specific journals which had been neglected by the journal impact factor. He found that the diffusion and the impact of a journal contents seemed to be statistically independent and they are capturing different aspects of reception. He thinks that broad reception (diffusion) as well as fast citation uptake (impact) is an essential component of high quality research. The journal diffusion factor was defined as a standardized average number of citing journals per source; citations within a given time window (Rowlands, 2002).
Rowlands (2002) believed that the number of unique new citing journals would be a new measurement index for the extent of diffusion in the sense that the citing journals adopt knowledge from the target cited journal as the direction of knowledge flow is opposite from that of citations. The ‘unique new citing journals’ are newly-appearing citing journals to the target cited journal in a target year and they have not cited the target journal before the target year. In other terms of knowledge flow, they represent the amount of knowledge newly flowed from the target cited journal to the citing journals. The n-year journal diffusion factor of a certain target journal J is (here, only synchronous version is discussed):
where: U indicates the number of unique new citing journals published in the year Y-j to articles published in the target journal in a fixed year Y. CIT indicates the sum of citations of the target journal. As this is a synchronous version, it takes into account the multiple publication years and single citation year of CIT (Y, Y-j) that refers to the number of citations received in the single year Y-j from articles published during the multiple years Y-j. RJDF refers to Rowland’s journal diffusion factor or relative journal diffusion factor. While RJDF represents the ratio of unique citing journals to total citations of target journal to normalize the impact of the unique citing journals, Frandsen (2004) argued that the RJDF correlates negatively with the number of citations received. It means that a journal with high RJDF score might have a small number of citations received and a journal with low RJDF score might have a large number of citations. This situation is believed to be problematic in applying the RJDF for evaluation of research influences.
Solving this problem, Frandsen (2004) introduced an alternative JDF showing that it is independent of the number of citations received. This new Frandsen JDF (FJDF) uses the number of publication instead of the number of citations for the denominator of the RJDF formula. The synchronous version of the FJDF for a particular journal J for the year Y (Frandsen et al., 2006) is:
Only difference from the RJDF is PUB(Y-j) in the formula. It refers to the number of publications of a journal J in the year Y. As it is also a synchronous version, it takes into accounts multiple publication years. Frandsen (2004) also showed that this FJDF can differentiate journals with similar journal impact factor (JIF) scores although there is a strong positive correlation between FJDF and JIF. In a similar vein, Frandsen et al. (2006) also identified the relationships among FJDF, RJDF, and JIF (synchronous version) as follows:
FJDF = RJDF x JIF
The present study uses the synchronous version of FJDF, RJDF, and JIF to examine the breadth of OA journal impact.
Data collection
The data used in this study were obtained from the three-year Web of Science journal citation report (2015, 2014, and 2013) in the areas of ‘Medicine, General, and Internal.’ Data comprises of the journal name, impact factor for a specific journal for five years, impact factor for a specific journal, total citations received for a particular journal in the target year, the total number of papers published by the particular journal in the target year, and the International Standard Serial Number (ISSN) number. In addition, a search through the ISSN number was performed at Directory of Open Access Journals (DOAJ, https://doaj.org/) to check whether the journal is OA or not. Based on these data collection, two analyses were conducted.
According to the first analysis, of the 93 journals collected, journals ranging from 1st to 60th in the rank of impact factor during the previous 5 years selected. Of the 60 journals selected, a total of 12 OA journals were found, and the impacts of these 12 OA journals were analyzed based upon JIF, RJDF, and FJDF. This aims to see how OA journals would be of greater value if they applied the diffusion factors instead of the traditional impact factor.According to the second analysis, a total of 27 OA journals were further collected in the order of impact factor score. Then, non-OA journals were collected having similar impact factor score with the corresponding OA journals. These OA and non-OA journals were then statistically compared by JIF, RJDF, and FJDF. This analysis tried to test whether significant difference exists in the scores of the diffusion factors between OA and non-OA journals whose JIFs are similar.
The names of the journals are based on the abbreviations provided by the JCR. Since the 2015-year JIF data was used, the base data for each impact score was citation data in the year 2014. The journal ranks by each of the three indicators (JIF, RJDF, and FJDF) were calculated to compare the relative positions in each impact score. Figure 1 and 2 shows the ranking comparisons of the 12 OA journals by the three indicators. Each journal’s rank was transformed to rank score (max rank to actual rank) for example, PLOS MED was ranked first (the highest rank) by all the three indicators and it has the rank score of 55, 47, 55 for JIF, RJDF, and FJDF, respectively. The higher the rank scores are, the higher is the quality or the wider breadth the journals have.
The 12 journals are PLOS Medicine (PLOS MED), BMC Medicine (BMC MED), Annals of Family Medicine (ANN FAM MED), Cleveland Clinic Journal of Medicine (CLEV CLIN J MED), Deutsches Ärzteblatt International (DTSCH ARZTEBL INT), Indian Journal of Medical Research (INDIAN J MED RES), Swiss Medical Weekly (SWISS MED WKLY), Canadian Family Physician (CAN FAM PHYSICIAN), Journal of the American Board of Family Medicine (J AM BOARD FAM MED), South African Medical Journal (SAMJ AFR MED J), BMC Family Practice (BMC FAM PRACT), and Croatian Medical Journal (CROAT MED J). Each value represents the rank value of the 56 journals collected that involves both OA and non-OA journals together. The full lists of 56 journals are shown in Appendix 1. Based on the Figures 1 and 2, and Table 1, the initial result shows a visual tendency that the higher JIF scores leads to relatively lower diffusion factor scores in OA journals and vice versa. In other words, the diffusion factor scores seem to inversely proportional to the JIF scores. This result is based only on OA journals with a limited sample. The next second analysis covers the wider sample including both OA and non-OA journals.
The second analysis compared OA and non-OA journals having similar JIF values. Each group of the journals has 27 items. As the two groups (OA and non-OA groups) are independent and they are non-parametric data, this study applied the Mann–Whitney U test (a parametric version of t-test) in addition to description statistical analyses. All the statistical analyses were conducted by the IBM SPSS v23. The following boxplots (Figures 3, 4 and 5) show visual differences of the values of JIF, RJDF, and FJDF between the selected OA and non-OA journals. This study intentionally selected both-group journals with similar JIF values, and Figure 3 reflects this similarity. However, unlike the proposition of this study, Figures 4 and 5 shows no distinguishable difference between OA and non-OA journals. To further examine this issue, this study conducted the Mann-Whitney U test in order to statistically test if the measures of the three indicators (JIF, RJDF, and FJDF) are significantly different between OA and non-OA journals.
As shown in Table 2, the result of the Mann-Whitney U test also reflects similar result with prior boxplot description. It says none of the indicators has a significant difference in the measurement between OA and non-OA journals. For example, total citations and the values of JIF and RJDF have no difference between OA and non-OA journals. The significance level of FJDF was 0.054, which was slightly out of the significant range. This difference seems to lead to further analysis in the future.With this non-parametric analysis, this study also conducted t-test (counterpart of Mann-Whitney U test) to make sure the result of Mann-Whitney U test is correct. As expected, the t-test reconfirm that there is no significant difference (all the p-values > 0.05, total citations: p=0.221, JIF: p=0.833, RJDF: p=0.574, and FJDF: p=0.093).
DISCUSSION AND CONCLUSIONS
This study, with reference to previous studies, opines that OA journals tend to be cited more frequently than non-OA journals and that the higher citation counts are, the higher diffusion factor would be; which is an indicator of the degree of citation diffusion of a journal. Therefore, this study set up a proposition that OA journals would have a higher diffusion factor value than non-OA journals, and if there is a significant difference in the diffusion factor between OA and non-OA of journals, it seek to examine whether OA journals have a higher diffusion factor values than non-OA journals.However, unlike what the proposal suggested, the result reveals that the OA journals does not provide any statistical basis for the higher diffusion factor than the non-OA journals.
This result contradicts several prior citation advantage studies that focused on the benefits of OA journals. In this sense, this study contributes to the series of studies on the controversial OA citation advantage. It provides a new insight to the relationship between the journal diffusion factor and OA journals. Just like other studies, this study has some limitations. First, there were time-consuming works during the data collection of the citation data. So that, in the second analysis, only a 54 journal cases were collected to compare OA and non-OA mode. Especially, if there were more data, the difference of FJDF in OA and non-OA journals might be significant. As indicated earlier, this point is clearly necessary to examine in future study.
The author has not declared any conflict of interests.
REFERENCES
|
Antelman K (2004). Do open-access articles have a greater research impact? College Res. Libr. 65(5):372-382.
Crossref
|
|
|
|
Atchison A, Bull J (2015). Will open access get me cited? An analysis of the efficacy of open access publishing in political science PS: Political Sci. Politics 48(1):129-137.
|
|
|
|
|
Björk BC, Solomon D (2012). Open access versus subscription journals: a comparison of scientific impact. BMC Med. 10:73.
Crossref
|
|
|
|
|
Eysenbach G (2006). Citation advantage of open access articles. Plos Biology 4(5):692-698.
Crossref
|
|
|
|
|
Frandsen TF (2004). Journal diffusion factors–a measure of diffusion?Paper presented at the Aslib proceedings.
|
|
|
|
|
Frandsen TF, Rousseau R, Rowlands I (2006). Diffusion factors. J. Document 62(1):58-72.
Crossref
|
|
|
|
|
Frisch NK, Nathan R, Ahmed YK, Shidham VB (2014). Authors attain comparable or slightly higher rates of citation publishing in an open access journal (CytoJournal) compared to traditional cytopathology journals-A five year (2007-2011) experience. CytoJournal, 11. Available from:
View
Crossref
|
|
|
|
|
Koler-Povh T, Juznic P, Turk G (2014). Impact of open access on citation of scholarly publications in the field of civil engineering. Scientometrics. 98(2):1033-1045.
Crossref
|
|
|
|
|
Kurtz MJ, Eichhorn G, Accomazzi A, Grant C, Demleitner M, Henneken E, Murray SS (2005). The effect of use and access on citations. Inform. Process. Manag. 41(6):1395-1402.
Crossref
|
|
|
|
|
Lawrence S (2001). Online or invisible. Nature 411(6837):521.
Crossref
|
|
|
|
|
Rowlands I (2002). Journal diffusion factors: a new approach to measuring research influence. Aslib Proceedings. 54(2):77-84.
Crossref
|
|
|
|
|
Solomon DJ, Laakso M, Björk BC (2013). A longitudinal comparison of citation rates and growth among open access journals. J. Inform. 7(3):642-650.
Crossref
|
|
|
|
|
Vanclay JK (2013). Factors affecting citation rates in environmental science. J. Inform. 7(2):265-271.
Crossref
|
|
