Quality Measures of NCPEA Publications: Outputs or Impacts?
Theodore Creighton, Publications Director
Brad Bizzell, Publications Associate Director
Executive Summary
The purpose of this position paper is to describe in detail, the methods used by NCPEA Publications to ensure and assess the quality, significance, and impact of their publications. This paper is presented to the NCPEA Executive Board and NCPEA Membership to thoroughly define our method[s] of providing reliable data measuring the “impact” of our various publications (i.e., ELR, IJELP, Policy Briefs, and Books). This position statement provides published and potential authors, editors, reviewers, and general membership an understanding of the processes we use to measure the quality of our publications and focus on our responsibility and commitment to the profession.
Peer Review
NCPEA Publications is fortunate to have a large number of peer reviewers and editors (over 250 professors and practitioners). Each manuscript submitted for consideration follows the same 6-tier path of review:
- First, the manuscript is checked by Editors for adherence to submission guidelines, appropriateness of topic, and APA and length guidelines.
- If in alignment, the Editors delegate the manuscript to a Topic Editor (with the expertise and experience in the topic area). This Topic Editor again checks for appropriateness to make certain the topic is aligned with his/her area of expertise. If the manuscript has any weaknesses described in tier 1 above, it is returned to the author/s with thorough explanation of why we will not proceed with sending to reviewers.
- Once approved by the Topic Editor, he/she selects and assigns two reviewers from our review pool, who also have indicated a topic area of interest and expertise. Tier 2 and 3 are especially important ensuring that the manuscript is reviewed by reviewers who know and understand the topic.
- Upon completion of the reviews, the Topic Editor checks for level of agreement. If disagreement exists, a third review is requested.
- Once all the reviews are completed, the Topic Editor notifies the Editors with a recommendation. Editors and Directors meet to caucus and discuss the recommendation along with results of the reviews. Pros and cons of each of the manuscripts in the queue are discussed in detail and Editors make preliminary decisions of accept, accept with revisions, or reject.
- The Editors and Directors discuss the content of the author notifications and share in the communication of decisions to the submitting author/s.
Output vs. Impact
Outputs. First, we make a distinction between the definition of an “output” and an “impact.” We generally view outputs as an “end result of some action or procedure.” Using this definition, a few examples of outputs are: (a) graduation rates, (b) accreditation scores, (c) GPA and GRE scores, and yes, (d) attaining tenure or a promotion to the rank of full professor. These data, though significant measures, are limited to a numerical or labeled measurement. Moving directly to our publications, examples of outputs are: (a) a good review and successful publication in a journal, (b) journal acceptance rates, and (c) ISSN/ISBN or ERIC designations. An important question to ask at this point is: What do these kinds of data tell us? Or not tell us?
Impacts. Our use of the term impact, involves measuring the activity or accomplishment that occurs AFTER an output. In other words, what does a principal accomplish on the job AFTER he/she graduates and is credentialed from our programs? What accomplishments and impact does a tenured faculty have on teaching and learning? Again, moving to our publications, some important questions to ask are:
- Is your published work being read by our colleagues and/or used for educational purposes?
- Who's reading your published work and how often?
- How geographically broad is the impact of your published work?
- Is the impact and readership of your work "dynamic" and does it grow over time?
- Are educators downloading PDF copies of your work for educational purposes?
- What is the percentage of new vs. returning readers of your work?
What is THE Impact Factor?
The formal Journal Impact Factor is from Journal Citation Report (JCR), a product of Thomson ISI (Institute for Scientific Information). JCR provides quantitative tools for ranking, evaluating, categorizing, and comparing journals. The Impact Factor is a measure of the frequency with which the "average article" in a journal has been cited in a particular year or period. The annual JCR impact factor is a ratio between citations and recent citable items published. Thus, the impact factor of a journal is calculated by dividing the number of current year citations by the source items published in that journal during the previous two years. All things being equal, the larger the number of previously published articles, the more often a journal will be cited.
From the ISI founding director, Dr. Eugene Garfield, we find caution given to the inappropriate use of such data:
"Thomson Reuters does not depend on the impact factor alone in assessing the usefulness of a journal, and neither should anyone else. The impact factor should not be used without careful attention to the many phenomena that influence citation rates, as for example the average number of references cited in the average article. The impact factor should be used with informed peer review. In the case of academic evaluation for tenure it is sometimes inappropriate to use the impact of the source journal to estimate the expected frequency of a recently published article. Again, the impact factor should be used with informed peer review. Citation frequencies for individual articles are quite varied."
Garfield also warns about the “misuse in evaluating individuals because there is a wide variation from article to article in a single journal.”
NCPEA believes the "formal ISI Impact Factor" to be limited in scope and not applicable to the purposes and goals of NCPEA and NCPEA Publications. NCPEA does not define impact data as a number, ratio, or formula and therefore rejects the use of the Institute for Scientific Information (ISI) Impact Factor calculation. We believe this classification is an output measurement at best: based on the variable of citing alone and not actually a measurement of impact. Though ISI Impact Factor and a journal’s acceptance rate are significant data, neither addresses the contribution (or impact) of an individual author's published scholarly work to the journal or profession. In other words, acceptance rates and impact factors measure the standing of the journal - and often say little about an individual author's work (other than it is published).
NCPEA “chooses” to measure and assess the quality of the content of individual articles and not so much as the reputation of a journal in which it is published (measured by an arithmetic mean based on average number of cites/references found in other articles or journals). We further view an “impact factor” to be only one item on the quality measurement scale – whereas “impact data” are several items on the measurement scale.
To investigate the impact data of NCPEA Publications, we provide impact data for our authors that begin to address some of the underlying factors related to the quality and impact of an individual's research and scholarly contribution to the field:
- NCPEA Publications utilize Google Analytics to track the readership of our journals each 30-day cycle. These data are posted in 30-day increments on our site's Home Page. Included in these data are such items as: (a) total views, (b) from which specific countries, states, and cities, (c) percentage of new vs. returning readers, and (d) average time spent on individual reads. Presently, our publications are averaging over 2,000 visits each 30-day period. In addition, readership comes from over 60 countries including each of the 50 U.S. states. Sixty-percent of our readership are new visitors.
- To investigate and report further on the impact of the individual manuscript’s content, we utilize software through CyberPress Statistics. Whereas the Analytics data highlight access to and impact of the journals themselves, CyberPress Statistics data are directly linked to the individual author’s PDF copy and reveal the actual number of times readers have not only accessed a specific article, but the specific number of downloaded full-text PDF copies. This data tracking begins upon publication date and continues for all time (as all manuscripts remain accessible though archived in the system). Our individual manuscripts are downloaded in PDF and these data can be shared with individual authors.
The rigorous peer review process with the collection and analysis of impact data combine to ensure and assess the quality and value of NCPEA Publication’s published works.