Tag: white papers

Basics of Peer Review for Communication Design Scholarship

The evaluation of scholarly work by others in the same field or discipline.

Jessica Barness, Dan Wong, Aaris Sherin, Robin Landa, Alex Girard

Design-Incubation-White-Paper-Peer-ReviewDownload

This white paper focuses on the basics of peer review for the following types of work: abstracts, books, grant proposals, journal article manuscripts, as well as papers, creative projects, and documentation of teaching submitted for conferences and symposia. Throughout this paper, “author” refers generally to authors, creators, and researchers, while “editor” refers generally to publication editors as well as staff members that facilitate peer review evaluations for organizations or events.

What is peer review and why is it important?

Peer review is the evaluation of scholarly work by others in the same field or discipline. Peer reviewing (or refereeing) is a significant part of an author’s path to publication and other modes of dissemination. Though specifics of the process differ depending on the intended outcome of the author’s work, it is widely used for publishing scholarly journals, books, and conference proceedings, as well as for trade/commercial books. Additionally, peer review is used to select participation for academic conferences, symposia, and colloquia, and grant funding. 

Variations of the peer-review process have been in use for over a thousand years, and it is rooted in the circulation of scientific texts. There is evidence that the first documented instance of peer review was in Syria, and dates to the 9th century CE; it is described in the book Ethics of the Physician by Ishap bin Ali Al Rahwi1. The peer-review process involving an editor began to take shape in the 17th century with scientific journals Journal des Sçavans and Philosophical Transactions2. 

Why do we do peer review?

Peer review is a means to assure a field of study, in our case communication design, that the research that was conducted and the results are worthy of adding to the body of knowledge. It is a system of checks and balances used to confirm the work being done is original, persuasive, and currently of significance to the field. This is why citation and reference should play a key role in the communication of the research and ideas. Just as professional practice does not exist in a vacuum, research projects and the writing and presentations that result from them should build on the work of other designers, scholars, researchers, and educators. 

The Purpose of Peer Review is to:

  • Support the building of a discipline’s knowledge base.
  • Determine whether or not the work is suited for a given publication or presentation venue.
  • Provide a means to self-monitor the quality of creative work, scholarship, and research for the discipline.
  • Confirm the work is original, persuasive, and significant.
  • Provide a standardized method to evaluate the validity and rigor of research.
  • Offer feedback on errors, highlight problems, or indicate gaps in the research.
  • Engage authors and reviewers in a collaborative effort to provide feedback, to help the author strengthen their work with the aim of advancing the field.
  • Offer an objective and ethical process for considering the presentation or publication of research.

Who Engages in Peer Review?

Peers are researchers and faculty who specialize in a specific area of research, creative practice, or scholarly activity similar to the work being reviewed. Depending on where the author is submitting the work, they may be asked to recommend researchers who are experts in the same field to be peer reviewers. In some cases, the author may request the exclusion of reviewers where there may be potential conflicts-of-interest. More often, the publishers will have a peer committee or invite experts. An academic conference will have a pre-determined group of peer reviewers to evaluate proposals and/or proceedings. These peers will review the work for substance, originality, creativity, context, methodology, verifiability, and whether the results and conclusions seem accurate and believable.

Types of Peer Review

Different types of peer review may be used by journals, conferences, publishers, colloquia, and so forth. Varying degrees of anonymity is one of the most substantive ways peer review differs from other forms of review. Peer-review works on the assumption that an anonymized process lessens the chance of bias in the evaluation of the work. Blind peer review aims to correct biases such as an author’s reputation, the institution where they work, their geographic location, the previous work they have published or presented, or any professional or personal bias the reviewer may have towards the reviewer. Even with a carefully anonymized peer review process, however, a reviewer may be able to guess an author’s identity based on their writing style, selected citations, or the research itself. In highly specialized research communities, for example, specific research projects may be well known to others. Standardized peer-review processes are used across disciplines to maintain an objective evaluation and help guide authors and researchers toward the goal of producing high-quality publications or presentations.

Double-blind Peer Review

  • Identities of the reviewer and author are concealed from each other.
  • Common review type for publications and conferences in the design, humanities, and social sciences disciplines.3
  • Considered rigorous and the highest quality by tenure and promotion committees at research institutions.

Single-blind Peer Review

  • Identity of the author is known to the reviewer, but not vice versa.
  • Common review type for science and medical disciplines.3
  • Common for textbooks and trade books in the design disciplines.

Open Peer Review

  • Identities of reviewer and author are disclosed to both at some point in the process.
  • Newer model that encourages cooperation, accountability, and civility by peer reviewers.

Types of scholarly work that involve peer review

  • Book proposals and book manuscripts
  • Grant proposals
  • Journal article manuscripts
  • Academic posters
  • Materials submitted for conferences and symposia including papers, creative projects, documentation of teaching

Other Types of Review for Communication Design Scholarship

Academic editorial review

Academic journals and book publishers generally have a mission, focus, and readership. An editor-in-chief’s responsibility is to ensure that the content of a submission is appropriate and within the scope of the publication. The editor decides whether or not to initiate the peer review process. 

Exhibition juried/curatorial review

Exhibition of creative projects through a jury-selection process, or chosen by curator/s, may be viewed as comparable to peer review in the context of tenure and promotion. Unlike the peer review process, the evaluation for exhibiting creative work is often given as a yes/no decision with little or no feedback given to the applicant.

Trade or commercial events

The organizers of trade shows, commercial exhibitions, trade conferences, and editors and publishers of trade magazines, also offer options for disseminating creative and written work produced by faculty in communication design. Though these may have merit and value for a faculty member’s creative career, evaluations for these types of venues are not the same as academic peer review. They may or may not be “counted” as scholarship for tenure and promotion at an institution. Authors should check with their department chairperson, director, and/or dean before pursuing these venues for dissemination. 

Editorial review of consumer media

Trade journals, popular magazines or websites, and industry conferences, for example, do not typically make use of the peer-review process. An editor or staff member reviewing a submission may be a subject expert and provide guidance. However, the decision typically rests on the decision of one person (or a group) and is often based on anticipated sales of books, seats at a conference, and so forth. For tenure and promotion purposes, this type of review is often referred to as “accepted through the editorial process.”

Elements evaluated by peer reviewers

The specific criteria for peer review can vary widely among publishers, journals, conferences, and organizations. Common elements that peer reviewers consider will include some or all of the following:

  • The work is significant to the field/discipline (the contributions relate to practice, theory, methodology, pedagogy, history, etc.)
  • The title is concise, descriptive, and appropriate for the topic and venue.
  • The project/topic fits within the scope of the venue and/or is appropriate for its readers.
  • The topic is clearly defined and presented.
  • The keywords/keyword phrases are concise, descriptive, appropriate for the topic, and venue.
  • The thesis is original or unique, it clearly builds on the work of other researchers, and/or it furthers existing ideas or theories (rather than repeating established concepts).
  • The approach or methodology of the research is valid.
  • Literature/media reviews are provided to give context within the field/discipline.
  • The conclusions are drawn from the results of the research and the assessment of the author’s outcomes is valid.
  • The citations and referencing styles are correct as indicated by each organization/publication for example Chicago or APA.
  • The style of writing is appropriate for the journal or venue
    for dissemination. 
  • An introduction and conclusion are included and evaluated.

Initiation of the peer review process 

The specific steps an author, editor, and peer review evaluators will be asked to go through as part of the peer review process may vary by publisher, journal, conference, or organization. Note: “editor” refers generally to publication editors as well as staff members who facilitate peer-review evaluations for organizations or events. The outline below includes a basic overview of the most common steps:

  1. Author submits work such as a full paper, abstract, or proposal (in the case of a book project, grant, or conference presentation).
  2. Editor reviews the submission and determines whether or not to proceed with peer review (often called a desk or editor review). 
  3. Editor contacts peer reviewers to request their evaluation of the submission.
    • Typically, a submission undergoes evaluation by 2 or 3 peer reviewers (sometimes more). Some journals or publishers may ask the author to provide names of potential peer reviewers who align with their area of research and could provide an unbiased evaluation. Other entities will have editors identify the potential peer reviewers. 
    • Often, one evaluator will have expertise that is closely tied to the topic of the submission, and another reviewer will represent a more general audience.
    • Peer reviewers are often provided with a rubric to guide their evaluation.
  4. Submission is considered to be “under peer review” when peer reviewers agree to review and receive the submission.
    • This step is crucial for tenure and promotion. It can provide evidence that the submission has made it through the first step in the peer-review process.
    • The editor will provide an author with information that their submission is under peer review and will give some indication of how long it may take to receive feedback. 
    • This could take anywhere from two weeks to six months (or beyond), depending on the publisher, journal, or organization and their schedule.
  5. Peer reviewer sends their evaluation back to the editor.
  6. Editor reads through the peer reviews and decides what outcome will be communicated to the author.
    • At times, the peer reviewers will offer similar evaluations of a submission. Or, they could be wildly different. The editor compiles the evaluations and returns the recommendation to the author for next steps.  
    • In some instances, an editor may override the recommendation of the peer reviewers. In rare cases, the submission may be sent to a different group of reviewers. This is done when editors feel the reviewers have bias or did not adequately review the submission.  

Outcomes of the peer review process

The author is given a recommendation after peer reviews are completed. The specific terminology may vary among publishers, journals, and organizations, but the outcomes listed below are commonly used.

Accept as-is with no revisions 

  • This is an uncommon response.

Accept with minor or major revisions

  • This means the evaluations and editor are in favor of publication/presentation of the work, but some minor or major edits must be made before it is officially accepted. There may be multiple rounds of edits and further review.
  • See the “Advice to Authors Navigating Peer Review” document for further information on advice for authors going through the peer-review process.

Revise and resubmit

  • This means the evaluations and editor indicate that there are substantial revisions necessary before they can agree to accept the work under review. The author can choose whether or not to move forward and revise the piece. 
  • Once the author has revised the work, it should be resubmitted to the publisher, journal, or organization according to their submission guidelines.
  • In some instances, the revised work is considered a new submission, and it may be sent back to the same reviewers. In other instances, the editor or staff member will decide whether the author has adequately incorporated feedback.
  • See the Advice to “Authors Navigating Peer Review” document for further information on advice for authors going through the peer-review process.

Reject

  • This means the evaluations and editor/staff are not in support of the publication/presentation. 
  • When rejection happens as an outcome of peer review, the author is typically given the reviewers’ comments. This differs from rejection at the time of the initial editor/desk review because peer review feedback is not provided when a submission is rejected at that stage.
  • The work rejected at either initial desk review or after peer review should not be resubmitted to the same venue.
  • If the researcher has submitted a grant proposal, the grantor, trust, or foundation will have guidelines on whether or not feedback will be provided for rejected proposals. 

Advice for Peer Reviewers

At some point, you may be asked to serve as a peer reviewer
for a journal, conference, colloquia, or an organization awarding grant funding. Taking on the role of peer reviewer means you are dedicating time and energy to support another author’s/researcher’s work, and you are committing to helping to strengthen the research within the communication design discipline. Before you agree (or decline) to take on the task, think about the following things.

  • Be familiar with the publisher, journal, or organization requesting peer review. Predatory journals and organizations exist; if the request is coming from an unfamiliar source, it’s wise to investigate. If you are unsure, ask trusted colleagues; if that is not possible, check to see if your institution has a faculty professional development office and if they can support you. 
  • For journals, conferences, and colloquia: Once you have received the submission, consider whether the work relates to your area of communication design expertise or knowledge. If yes, great! If not, it’s okay to decline the invitation or ask for clarification from the editor or staff person assigned to the submission. Perhaps you know a colleague who is the perfect fit; consider recommending them as a peer reviewer or as a general reader (for a book).
  • Take note of the deadline and consider if you will have the necessary time and energy to review the work and provide substantive feedback by the deadline. Peer reviewers should want to review the work, and not simply take it on as an obligation. If the deadline poses a barrier, be open with the editor and consider asking if an extension is possible.
  • Evaluate the work according to the journal or organization’s peer review rubric.
    • Peer review rubrics vary widely. Sometimes they are surveys with a combination of yes/no, numeric ranking, space to provide comments, a spreadsheet, or guidelines that require several paragraphs of feedback.  
    • If the journal or organization does not provide a rubric or guidelines, ask the editor or staff member for guidance.
  • One of the most helpful things a peer reviewer can do, if they recommend rejecting the submission, is to suggest other venues for the author to submit the work.  
  • Be kind and generous. It is appropriate to provide negative comments, but your feedback should be constructive and actionable.  Spend time thinking about how the author could improve the work. Give specific and detailed suggestions and remember to note what works in the piece/submission. The goal is to provide feedback to help make the work better!

References

  1. Ray Spier, “The history of the peer-review process”, Trends in Biotechnology, Volume 20, Issue 8, 2002, Pages 357-358.
  2. “350 Years of Scientific Publication: from the Journal des Sçavans and Philosophical Transactions to SciELO” https://blog.scielo.org/en/2015/03/05/350-years-of-scientific-publication-from-the-journal-des-scavans-and-philosophical-transactions-to-scielo/#.X80Bt8tKiAl
  3. “Understanding Peer Review: A Guide for Authors” https://authorservices.taylorandfrancis.com/publishing-your-research/peer-review/

Bibliography

A Beginner’s Guide to the Peer Review System
https://www.insidehighered.com/blogs/gradhacker/beginner%E2%80%99s-guide-peer-review-system

Case No. 16-08: Author requests for certain experts not to be included in the editorial process
https://publicationethics.org/case/author-requests-certain-experts-not-be-included-editorial-process

CAA Statement on Exhibition Venues
https://www.collegeart.org/standards-and-guidelines/guidelines/statement-of-exhibition-venues

COPE Peer Review Process (website subsection)
https://publicationethics.org/peerreview

Editorial and Peer Review Process
https://journals.plos.org/plosone/s/editorial-and-peer-review-process#loc-editorial-decisions

How To Be a Good Peer Reviewer
https://scholarlykitchen.sspnet.org/2019/09/17/how-to-be-a-good-peer-reviewer/

Kill Peer Review or Reform It?
https://www.insidehighered.com/news/2012/01/06/humanities-scholars-consider-role-peer-review

Our Processes & Policies. Royal Society of Chemistry.
https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/

Peer Review in the Humanities and Social Sciences
https://www.wiley.com/network/researchers/being-a-peer-reviewer/peer-review-in-the-humanities-and-social-sciences

Peer Review in the Humanities and Social Sciences: If It Ain’t Broke, Don’t Fix It?
https://scholarlykitchen.sspnet.org/2016/09/21/peer-review-in-the-humanities-and-social-sciences-if-it-aint-broke-dont-fix-it/

“350 Years of Scientific Publication: from the Journal des Sçavans and Philosophical Transactions to SciELO”
https://blog.scielo.org/en/2015/03/05/350-years-of-scientific-publication-from-the-journal-des-scavans-and-philosophical-transactions-to-scielo/#.X3YB2JNKiL8

Scholarly Journals vs. Trade Journals vs. Popular Magazine
https://www.jbu.edu/assets/library/resource/Scholarly_Journals_vs_Trade__Journals_vs__Popular_Magazines.pdf

“The History of the Peer-Review Process” Spier, Ray. Trends in Biotechnology, Volume 20, Issue 8, 357 – 358
https://doi.org/10.1016/S0167-7799(02)01985-6

Types of Peer Review
https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/types-of-peer-review.html

Understanding Peer Review: A Guide for Authors
https://authorservices.taylorandfrancis.com/peer-review/

What is Peer Review?
https://www.elsevier.com/reviewers/what-is-peer-review

What To Do After the Reviews Arrive
https://howardaldrich.org/2016/07/what-to-do-after-the-reviews-arrive/

Grant Review Process
https://www.genome.gov/research-funding/apply/Grant-Review-Process#:~:text=A%20submitted%20grant%20application%20undergoes,grant%20award%20can%20be%20made.

Peer Review
https://grants.nih.gov/grants/peer-review.htm

Advice to Authors Navigating Peer Review

Planning for a successful submission can help you identify where to publish and how to set yourself up for success.

Aaris Sherin, Jessica Barness, Robin Landa

White-Paper-Peer-Review-AdviceDownload

Whether you’ve just received your first peer-review evaluation, or this is one of many, you should begin by reading through the editor’s decision letter and peer review comments. 

Then, (depending, of course, on the peer review outcome and suggestions) give yourself a high-five or cry into a pillow. Go for a walk. Remember, having something accepted with no revisions or minor revisions is rare. Order pizza. Then, no matter how you are feeling about the peer reviews, take a step back and wait a day or two before proceeding.

Major or Minor Revisions

If your work has been accepted with major or minor revisions, or you’ve been asked to resubmit, you have some decisions to make. Grants and juried exhibitions often provide a yes/no outcome, so the following may not apply. Keep in mind that the peer reviewer’s evaluation of your project is based solely on what you’ve written/created. Take a deep breath and get back to work:

  • Thank the editor. Do this even if you received a rejection—don’t burn bridges!
    • If you were given a timeline for completing revisions, let them know you will meet that deadline. 
    • If you were not given a timeline, let them know when you plan to submit revisions.
  • Remember that getting feedback from peer reviewers is valuable. Also, peer review is one of the only ways to get detailed free, unbiased feedback. Allow the process to help strengthen your work.
  • Read through the peer review feedback again. If you have difficulty understanding the comments, ask a trusted colleague to help you navigate the feedback; if that is not possible, check to see if your institution has a faculty professional development office or research officer and if they can support you. If feedback continues to be confusing you may address specific points with the editor/ staff person assigned to your submission. 
  • Authors are often asked to address all concerns noted by peer reviewers. Sometimes it’s helpful to create a spreadsheet to sort through these.
    • You might agree with the peer review comments, and you might also disagree with some of them. 
    • If you disagree with the feedback given by the reviewers, it is acceptable to submit a rebuttal to the editor and explain why. The editor will then weigh whether or not specific revisions need to be made and will advise accordingly. For example, you may realize that a suggested revision is outside the scope of the paper. If that is the case, it may be worth looking at how you are communicating your research and make sure the scope is clearly described.
  • If you receive a rejection or a substantial amount of negative feedback, it’s possible your submission may not be the best fit for your selected venue. Consider a different, more appropriate place to publish or present your research. 
    • It is considered ethical to ask for feedback and find out why the work was not accepted. Sometimes an editor may suggest an author look at other venues and at other times they may provide additional feedback. 
    • If the work is rejected during the peer review process the author is usually given feedback from the peer reviewers. 
  • When revising, be aware of word count and stay within the criteria of the publication/venue you are working with.
  • Make a writing schedule so that you meet your deadlines, and respond to peer reviewer feedback accordingly.

What happens after a submission is accepted?

  • Most of the time, this is the end of the peer-review process. At this stage, the author’s work is accepted for publication or presentation. 
  • Grants and abstract submissions often end here, and the author is awarded funds. For some grants, a report is due at the midterm and/or completion of the grant-funded research.
  • For longer forms of writing such as books, significant work may still need to be done even after a work is accepted for publication. For example, authors of books may have a proposal accepted through the peer review process but still need to write the full manuscript, or write/rewrite sample chapters. 
  • Journal article manuscripts typically go through another round of editor and peer review. Authors of journal articles and books may need to secure permissions for images and quotes beyond fair use and work with a copy editor on small text and citation changes. 

Rejections Happen

Why might a submission be rejected? There are several reasons a submission might be rejected and they include but are not limited to the following: it may not fit the scope of what the journal/venue publishes, the methodology may be flawed, there may be too few references or it may be unclear how the work contributes to current topics considered to be particularly relevant in the field, citations may be lacking, or the structure and syntax of the writing may not meet stated criteria. Regardless of why a submission is rejected, the peer review process should provide feedback. The feedback you receive can be used to help you revise your work and get it ready to submit to another venue. It can provide information that can help you decide whether you should realign your research to fit the scope of publications/venues that are more likely to want to disseminate your work.

Submitting the same work to multiple venues 

  • It is typically considered unacceptable to submit a paper to multiple journals at one time. Start with the venue that seems most fitting and appropriate; if the reviews are not helpful, or the work just doesn’t fit that venue, consider submitting elsewhere. 
  • Books proposals are an exception and may be submitted to multiple publishers at one time. State that your submission is simultaneous, if it is.
  • Typically, it is considered unacceptable and unethical to submit the same paper or presentation to multiple conferences. However, this continues to be common practice in the design disciplines as standards evolve. A best practice would be to vary the presentation each time to make each work unique from the other. Check the criteria for each conference or venue you submit to ensure you are maintaining and supporting academic standards. Note: Trade publications and conferences are primarily concerned with generating revenue and may welcome or even solicit duplicate submissions. (This would be a good time to read the fine print and understand who retains the copyright for a given text. You may inadvertently hand over the rights to your own words and ideas.) 

Self-plagiarism and self-citation

  • Self-plagiarism is unethical and occurs when an author presents/publishes their previously presented/published work as something new. 
  • You may need to secure permission from your publisher if you’re quoting yourself elsewhere beyond fair use.
  • If you must reference your previously published scholarly work (creative project, written research, etc.) self-citation is appropriate. Style guides such as Chicago, APA, and MLA provide further guidance on how to approach self-citation.

Before you submit to a publication or venue:

Unsure where to start?  Planning for a successful submission can help you identify where to publish and how to set yourself up for success. 

  • Ask colleagues/mentors about where they have published.
  • Look for venues that have published work similar to yours. 
  • Understand the publisher and type of publications they produce.
  • Understand the mission or focus of the publication, venue, or conference.
  • Learn each venue’s process for submission and review.
  • Research the biographies of the editors, and if possible, look at the criteria they use to choose who will review submissions.
  • Adjust the focus of the work you have done and adapt it to the audience you are seeking.