Jessica Barness, Dan Wong, Aaris Sherin, Robin Landa, Alex Girard
This white paper focuses on the basics of peer review for the following types of work: abstracts, books, grant proposals, journal article manuscripts, as well as papers, creative projects, and documentation of teaching submitted for conferences and symposia. Throughout this paper, “author” refers generally to authors, creators, and researchers, while “editor” refers generally to publication editors as well as staff members that facilitate peer review evaluations for organizations or events.
What is peer review and why is it important?
Peer review is the evaluation of scholarly work by others in the same field or discipline. Peer reviewing (or refereeing) is a significant part of an author’s path to publication and other modes of dissemination. Though specifics of the process differ depending on the intended outcome of the author’s work, it is widely used for publishing scholarly journals, books, and conference proceedings, as well as for trade/commercial books. Additionally, peer review is used to select participation for academic conferences, symposia, and colloquia, and grant funding.
Variations of the peer-review process have been in use for over a thousand years, and it is rooted in the circulation of scientific texts. There is evidence that the first documented instance of peer review was in Syria, and dates to the 9th century CE; it is described in the book Ethics of the Physician by Ishap bin Ali Al Rahwi1. The peer-review process involving an editor began to take shape in the 17th century with scientific journals Journal des Sçavans and Philosophical Transactions2.
Why do we do peer review?
Peer review is a means to assure a field of study, in our case communication design, that the research that was conducted and the results are worthy of adding to the body of knowledge. It is a system of checks and balances used to confirm the work being done is original, persuasive, and currently of significance to the field. This is why citation and reference should play a key role in the communication of the research and ideas. Just as professional practice does not exist in a vacuum, research projects and the writing and presentations that result from them should build on the work of other designers, scholars, researchers, and educators.
The Purpose of Peer Review is to:
- Support the building of a discipline’s knowledge base.
- Determine whether or not the work is suited for a given publication or presentation venue.
- Provide a means to self-monitor the quality of creative work, scholarship, and research for the discipline.
- Confirm the work is original, persuasive, and significant.
- Provide a standardized method to evaluate the validity and rigor of research.
- Offer feedback on errors, highlight problems, or indicate gaps in the research.
- Engage authors and reviewers in a collaborative effort to provide feedback, to help the author strengthen their work with the aim of advancing the field.
- Offer an objective and ethical process for considering the presentation or publication of research.
Who Engages in Peer Review?
Peers are researchers and faculty who specialize in a specific area of research, creative practice, or scholarly activity similar to the work being reviewed. Depending on where the author is submitting the work, they may be asked to recommend researchers who are experts in the same field to be peer reviewers. In some cases, the author may request the exclusion of reviewers where there may be potential conflicts-of-interest. More often, the publishers will have a peer committee or invite experts. An academic conference will have a pre-determined group of peer reviewers to evaluate proposals and/or proceedings. These peers will review the work for substance, originality, creativity, context, methodology, verifiability, and whether the results and conclusions seem accurate and believable.
Types of Peer Review
Different types of peer review may be used by journals, conferences, publishers, colloquia, and so forth. Varying degrees of anonymity is one of the most substantive ways peer review differs from other forms of review. Peer-review works on the assumption that an anonymized process lessens the chance of bias in the evaluation of the work. Blind peer review aims to correct biases such as an author’s reputation, the institution where they work, their geographic location, the previous work they have published or presented, or any professional or personal bias the reviewer may have towards the reviewer. Even with a carefully anonymized peer review process, however, a reviewer may be able to guess an author’s identity based on their writing style, selected citations, or the research itself. In highly specialized research communities, for example, specific research projects may be well known to others. Standardized peer-review processes are used across disciplines to maintain an objective evaluation and help guide authors and researchers toward the goal of producing high-quality publications or presentations.
Double-blind Peer Review
- Identities of the reviewer and author are concealed from each other.
- Common review type for publications and conferences in the design, humanities, and social sciences disciplines.3
- Considered rigorous and the highest quality by tenure and promotion committees at research institutions.
Single-blind Peer Review
- Identity of the author is known to the reviewer, but not vice versa.
- Common review type for science and medical disciplines.3
- Common for textbooks and trade books in the design disciplines.
Open Peer Review
- Identities of reviewer and author are disclosed to both at some point in the process.
- Newer model that encourages cooperation, accountability, and civility by peer reviewers.
Types of scholarly work that involve peer review
- Book proposals and book manuscripts
- Grant proposals
- Journal article manuscripts
- Academic posters
- Materials submitted for conferences and symposia including papers, creative projects, documentation of teaching
Other Types of Review for Communication Design Scholarship
Academic editorial review
Academic journals and book publishers generally have a mission, focus, and readership. An editor-in-chief’s responsibility is to ensure that the content of a submission is appropriate and within the scope of the publication. The editor decides whether or not to initiate the peer review process.
Exhibition juried/curatorial review
Exhibition of creative projects through a jury-selection process, or chosen by curator/s, may be viewed as comparable to peer review in the context of tenure and promotion. Unlike the peer review process, the evaluation for exhibiting creative work is often given as a yes/no decision with little or no feedback given to the applicant.
Trade or commercial events
The organizers of trade shows, commercial exhibitions, trade conferences, and editors and publishers of trade magazines, also offer options for disseminating creative and written work produced by faculty in communication design. Though these may have merit and value for a faculty member’s creative career, evaluations for these types of venues are not the same as academic peer review. They may or may not be “counted” as scholarship for tenure and promotion at an institution. Authors should check with their department chairperson, director, and/or dean before pursuing these venues for dissemination.
Editorial review of consumer media
Trade journals, popular magazines or websites, and industry conferences, for example, do not typically make use of the peer-review process. An editor or staff member reviewing a submission may be a subject expert and provide guidance. However, the decision typically rests on the decision of one person (or a group) and is often based on anticipated sales of books, seats at a conference, and so forth. For tenure and promotion purposes, this type of review is often referred to as “accepted through the editorial process.”
Elements evaluated by peer reviewers
The specific criteria for peer review can vary widely among publishers, journals, conferences, and organizations. Common elements that peer reviewers consider will include some or all of the following:
- The work is significant to the field/discipline (the contributions relate to practice, theory, methodology, pedagogy, history, etc.)
- The title is concise, descriptive, and appropriate for the topic and venue.
- The project/topic fits within the scope of the venue and/or is appropriate for its readers.
- The topic is clearly defined and presented.
- The keywords/keyword phrases are concise, descriptive, appropriate for the topic, and venue.
- The thesis is original or unique, it clearly builds on the work of other researchers, and/or it furthers existing ideas or theories (rather than repeating established concepts).
- The approach or methodology of the research is valid.
- Literature/media reviews are provided to give context within the field/discipline.
- The conclusions are drawn from the results of the research and the assessment of the author’s outcomes is valid.
- The citations and referencing styles are correct as indicated by each organization/publication for example Chicago or APA.
- The style of writing is appropriate for the journal or venue
for dissemination. - An introduction and conclusion are included and evaluated.
Initiation of the peer review process
The specific steps an author, editor, and peer review evaluators will be asked to go through as part of the peer review process may vary by publisher, journal, conference, or organization. Note: “editor” refers generally to publication editors as well as staff members who facilitate peer-review evaluations for organizations or events. The outline below includes a basic overview of the most common steps:
- Author submits work such as a full paper, abstract, or proposal (in the case of a book project, grant, or conference presentation).
- Editor reviews the submission and determines whether or not to proceed with peer review (often called a desk or editor review).
- Editor contacts peer reviewers to request their evaluation of the submission.
- Typically, a submission undergoes evaluation by 2 or 3 peer reviewers (sometimes more). Some journals or publishers may ask the author to provide names of potential peer reviewers who align with their area of research and could provide an unbiased evaluation. Other entities will have editors identify the potential peer reviewers.
- Often, one evaluator will have expertise that is closely tied to the topic of the submission, and another reviewer will represent a more general audience.
- Peer reviewers are often provided with a rubric to guide their evaluation.
- Submission is considered to be “under peer review” when peer reviewers agree to review and receive the submission.
- This step is crucial for tenure and promotion. It can provide evidence that the submission has made it through the first step in the peer-review process.
- The editor will provide an author with information that their submission is under peer review and will give some indication of how long it may take to receive feedback.
- This could take anywhere from two weeks to six months (or beyond), depending on the publisher, journal, or organization and their schedule.
- Peer reviewer sends their evaluation back to the editor.
- Editor reads through the peer reviews and decides what outcome will be communicated to the author.
- At times, the peer reviewers will offer similar evaluations of a submission. Or, they could be wildly different. The editor compiles the evaluations and returns the recommendation to the author for next steps.
- In some instances, an editor may override the recommendation of the peer reviewers. In rare cases, the submission may be sent to a different group of reviewers. This is done when editors feel the reviewers have bias or did not adequately review the submission.
Outcomes of the peer review process
The author is given a recommendation after peer reviews are completed. The specific terminology may vary among publishers, journals, and organizations, but the outcomes listed below are commonly used.
Accept as-is with no revisions
- This is an uncommon response.
Accept with minor or major revisions
- This means the evaluations and editor are in favor of publication/presentation of the work, but some minor or major edits must be made before it is officially accepted. There may be multiple rounds of edits and further review.
- See the “Advice to Authors Navigating Peer Review” document for further information on advice for authors going through the peer-review process.
Revise and resubmit
- This means the evaluations and editor indicate that there are substantial revisions necessary before they can agree to accept the work under review. The author can choose whether or not to move forward and revise the piece.
- Once the author has revised the work, it should be resubmitted to the publisher, journal, or organization according to their submission guidelines.
- In some instances, the revised work is considered a new submission, and it may be sent back to the same reviewers. In other instances, the editor or staff member will decide whether the author has adequately incorporated feedback.
- See the Advice to “Authors Navigating Peer Review” document for further information on advice for authors going through the peer-review process.
Reject
- This means the evaluations and editor/staff are not in support of the publication/presentation.
- When rejection happens as an outcome of peer review, the author is typically given the reviewers’ comments. This differs from rejection at the time of the initial editor/desk review because peer review feedback is not provided when a submission is rejected at that stage.
- The work rejected at either initial desk review or after peer review should not be resubmitted to the same venue.
- If the researcher has submitted a grant proposal, the grantor, trust, or foundation will have guidelines on whether or not feedback will be provided for rejected proposals.
Advice for Peer Reviewers
At some point, you may be asked to serve as a peer reviewer
for a journal, conference, colloquia, or an organization awarding grant funding. Taking on the role of peer reviewer means you are dedicating time and energy to support another author’s/researcher’s work, and you are committing to helping to strengthen the research within the communication design discipline. Before you agree (or decline) to take on the task, think about the following things.
- Be familiar with the publisher, journal, or organization requesting peer review. Predatory journals and organizations exist; if the request is coming from an unfamiliar source, it’s wise to investigate. If you are unsure, ask trusted colleagues; if that is not possible, check to see if your institution has a faculty professional development office and if they can support you.
- For journals, conferences, and colloquia: Once you have received the submission, consider whether the work relates to your area of communication design expertise or knowledge. If yes, great! If not, it’s okay to decline the invitation or ask for clarification from the editor or staff person assigned to the submission. Perhaps you know a colleague who is the perfect fit; consider recommending them as a peer reviewer or as a general reader (for a book).
- Take note of the deadline and consider if you will have the necessary time and energy to review the work and provide substantive feedback by the deadline. Peer reviewers should want to review the work, and not simply take it on as an obligation. If the deadline poses a barrier, be open with the editor and consider asking if an extension is possible.
- Evaluate the work according to the journal or organization’s peer review rubric.
- Peer review rubrics vary widely. Sometimes they are surveys with a combination of yes/no, numeric ranking, space to provide comments, a spreadsheet, or guidelines that require several paragraphs of feedback.
- If the journal or organization does not provide a rubric or guidelines, ask the editor or staff member for guidance.
- One of the most helpful things a peer reviewer can do, if they recommend rejecting the submission, is to suggest other venues for the author to submit the work.
- Be kind and generous. It is appropriate to provide negative comments, but your feedback should be constructive and actionable. Spend time thinking about how the author could improve the work. Give specific and detailed suggestions and remember to note what works in the piece/submission. The goal is to provide feedback to help make the work better!
References
- Ray Spier, “The history of the peer-review process”, Trends in Biotechnology, Volume 20, Issue 8, 2002, Pages 357-358.
- “350 Years of Scientific Publication: from the Journal des Sçavans and Philosophical Transactions to SciELO” https://blog.scielo.org/en/2015/03/05/350-years-of-scientific-publication-from-the-journal-des-scavans-and-philosophical-transactions-to-scielo/#.X80Bt8tKiAl
- “Understanding Peer Review: A Guide for Authors” https://authorservices.taylorandfrancis.com/publishing-your-research/peer-review/
Bibliography
A Beginner’s Guide to the Peer Review System
https://www.insidehighered.com/blogs/gradhacker/beginner%E2%80%99s-guide-peer-review-system
Case No. 16-08: Author requests for certain experts not to be included in the editorial process
https://publicationethics.org/case/author-requests-certain-experts-not-be-included-editorial-process
CAA Statement on Exhibition Venues
https://www.collegeart.org/standards-and-guidelines/guidelines/statement-of-exhibition-venues
COPE Peer Review Process (website subsection)
https://publicationethics.org/peerreview
Editorial and Peer Review Process
https://journals.plos.org/plosone/s/editorial-and-peer-review-process#loc-editorial-decisions
How To Be a Good Peer Reviewer
https://scholarlykitchen.sspnet.org/2019/09/17/how-to-be-a-good-peer-reviewer/
Kill Peer Review or Reform It?
https://www.insidehighered.com/news/2012/01/06/humanities-scholars-consider-role-peer-review
Our Processes & Policies. Royal Society of Chemistry.
https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/
Peer Review in the Humanities and Social Sciences
https://www.wiley.com/network/researchers/being-a-peer-reviewer/peer-review-in-the-humanities-and-social-sciences
Peer Review in the Humanities and Social Sciences: If It Ain’t Broke, Don’t Fix It?
https://scholarlykitchen.sspnet.org/2016/09/21/peer-review-in-the-humanities-and-social-sciences-if-it-aint-broke-dont-fix-it/
“350 Years of Scientific Publication: from the Journal des Sçavans and Philosophical Transactions to SciELO”
https://blog.scielo.org/en/2015/03/05/350-years-of-scientific-publication-from-the-journal-des-scavans-and-philosophical-transactions-to-scielo/#.X3YB2JNKiL8
Scholarly Journals vs. Trade Journals vs. Popular Magazine
https://www.jbu.edu/assets/library/resource/Scholarly_Journals_vs_Trade__Journals_vs__Popular_Magazines.pdf
“The History of the Peer-Review Process” Spier, Ray. Trends in Biotechnology, Volume 20, Issue 8, 357 – 358
https://doi.org/10.1016/S0167-7799(02)01985-6
Types of Peer Review
https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/types-of-peer-review.html
Understanding Peer Review: A Guide for Authors
https://authorservices.taylorandfrancis.com/peer-review/
What is Peer Review?
https://www.elsevier.com/reviewers/what-is-peer-review
What To Do After the Reviews Arrive
https://howardaldrich.org/2016/07/what-to-do-after-the-reviews-arrive/
Grant Review Process
https://www.genome.gov/research-funding/apply/Grant-Review-Process#:~:text=A%20submitted%20grant%20application%20undergoes,grant%20award%20can%20be%20made.