X:\Informing Science\ISI Governor\Editor Manual draft for final review.docx

GUIDELINES FOR EDITORS

OF INFORMING SCIENCE INSTITUTE JOURNALS AND CONFERENCES

Congratulations on your appointment to serve as an Editor on the International Review Board for an Informing Science Institute (ISI) journal. Please read and save this document because it describes your duties as Editor.

YOU ARE PART OF THE SOLUTION, FIXING THE PROBLEMS WITH PAPER REVIEWING

Have you ever sent out a paper you wrote to a journal, and you didn’t hear about the review for the better part of a year or even longer? We have.

Have you ever received a decision letter that left you wondering how to proceed? Perhaps that decision letter contained contradictory or irrational guidance. We have.

We designed the paper review system for ISI journals to overcome those and other problems with the journal peer review process. Instead of the old production model (reject defects), we at ISI journals use a continual process improvement model. While fewer than half of the papers submitted make it into print, we view our role as mentor to all authors, sharing with them our wisdom and the wisdom of our reviewers, to assist them to improve their submission. Most papers that we do publish undergo several rounds of revisions.

We also designed the system to avoid the problem of delayed reviewing of manuscripts. We endeavor to get back to authors with our initial assessment within two months of their submission. To do this, we have abandoned the model of asking Editors and Reviewers to accept assignments. Editorial board members are expected to accept their assignment. (See below for how to set periods of non-availability.)

You, as Editor, are vital to these improvements. They entail that you, the Editor, interact with authors in new and better ways. This manual is to get us all on the same page, to inform you about the ISI philosophy of the paper review process, and to share with you how to use the ISI Paper Review automated system so you can to do your part.

The ISI philosophy

As an Editor for an ISI journal, your principal role is as a mentor to the author and the reviewers; that is, you are a key person in carrying out the mentoring that makes ISI one of the most constructive forces in higher education.

Your crucial role is that of an agent for improving research and researchers. While other journals just ask for gatekeepers, your role is of a guide. While other journals ask for referees, your role is of the coach. In ISI, we all give each other a helping hand. Reviewers mentor authors, Editors mentor reviewers and authors, and Editors-in-Chief mentor their editors.

This document is organized to give you first a quick view of your role and then to provide details on how to do the tasks of an Editor.

If you have questions about editorial issues, ask the journal’s Editor-in-Chief or Associate Editor-in-Chief. If you have questions about how to use the system not covered here, ask the Managing Editor.

GETTING STARTED – THE ESSENTIALS

Your role

In all journals, peer reviewing aims to ensure the integrity and validity of research findings; the process used by all serious journals to reach those findings is rigorous. The submission review process is the main check

2

we have to ensure that new scientific knowledge being published is honest, accurate, ethical, and of value. In effect, it acts as a measure of quality assurance and ‘certifies’ the new knowledge.

The journals of the ISI go a step beyond this. We ask our reviewers and editors to be coaches and guides to the authors. We use the paper review process to support authors by providing them with mentoring suggestions on ways to improve their work. We do this by packaging constructive and informed feedback using best practices methods on managing gleaned from the literature. For example, our suggestions are do-able and concrete. We treat authors with respect.

As an Editor, you play the central role in managing the review process, using your reviewers’ comments and suggestions to synthesize and compose wording designed to guide the author on ways to improve the submission. Your task is to provide this mentorship for all authors, whether or not you recommend to the Editor-in-Chief to progress their submissions toward publication.

An overview of your duties

In your role as an ISI journal Editor, you have four primary responsibilities:

1. Manage the reviewers working on papers assigned to you. Occasionally, a reviewer may contact you with a question.

2. For each review, rate the quality of that review on our 5-point scale.

3. Make a recommendation to ‘reject’ or ‘accept subject to revisions’, and mentor the author via a ‘Development’ letter.

4. Work directly with authors of papers you ‘accept pending revision’ to get the paper into a publishable format.

You are never all by yourself. When you have questions or concerns, contact the Editor-in-Chief or Associate Editor-in-Chief who sent you your assignment letter.

Let’s now get into some more detail.

Essentials of the Editor’s role in the review process

Let’s go over how this process works, from start to finish.

1. You receive an assignment letter as Editor because the Editor-in-Chief has read the submission and thinks that it deserves to go out for external review. The Editor-in-Chief will have already ensured that the paper is blinded, line numbers added, turned it into a PDF, and assigned reviewers to the paper. You and these reviewers comprise the paper’s ad hoc review committee with you as the chair of that committee. Occasionally, a reviewer may contact you with a question. Your role includes managing the reviewers on your review committee.

2. You read the paper and develop your thoughts on how to improve it. Your reviewers do the same, and they each submit a review with their thoughts and suggestions. You rate the quality of each review on our 5-point scale.

Almost all reviews earn a ‘3’, exceptionally good reviews earn a ‘4’ or ‘5’ and those found lacking earn a ‘2’ or ‘1’. This rating is used by the system to identify which reviewers can use more training and which to commend for their excellence and possibly promote. As you rate each review, you provide the reviewer with brief feedback on the quality of the review and possibly how future reviews can be improved.

3. You make a recommendation to the Editor-in-Chief regarding acceptance and include a ‘Development Letter’ within the templated wording provided for the author. At first, the recommendation should be either to ‘reject’ or ‘accept subject to revisions.’ (The third choice, to ‘accept for publication,’ applies only for when the paper has already addressed all your suggestions, and the author has resubmitted it in near camera-ready format.) In all cases, you accompany your recommendation with comments for the author on how to improve the paper in the form of the

3

Development Letter that you compose. Creating a Development Letter (how to improve the submission) for each paper is your most important duty as editor.

Your task in creating the Development Letter is to integrate the reviewers’ comments with your own views to create a coherent, mentoring response to the author’s submission. It is a synthesis, not a summary, of the various suggestions from your review board. We term this guidance to the author a Development Letter since it outlines concrete steps needed to improve the paper. You compose a development letter for all submissions, even those that you do not recommend for publication.

ISI journals never send authors raw reviews. Instead the Editor packages and synthesizes all suggestions into wording that speaks with a single voice, pointing out those improvements needed to make the paper publishable (if you accept pending revision) or at least improved (if you reject). (See the Appendices for more information about the Development Letter.)

4. If you recommend ‘accepted subject to revisions,’ you work directly with authors to get the paper into a publishable format.

This revision process typically requires several rounds of revisions.

Here are the steps with some more details

Step 1: You receive an email notifying you that you have been assigned a paper to shepherd through as Editor

When a paper is submitted, the Editor-in-Chief (EiC) first ‘desk reviews’ it for suitability. The EiC rejects some at this point, some are sent back to the author to re-work, and some are advanced for peer review.

When the EiC selects you to be the Editor for a paper, the system sends you an assignment email that identifies which EiC or Associate Editor-in-Chief (AEiC) has assigned the new submission to you. That assignment letter contains important guidance for you. Your assignment email includes a direct link to that assigned paper, but you can always view all your assignments and copies of your assignment emails in your dashboard (Figure 1). To access your dashboard, log in at http://My.InformingScience.org . From your dashboard, you can even locate the authors’ names and email addresses

Figure 1. You can always find your assignments in your dashboard by logging in at http://My.InformingScience.org

4

Figure 2. How to obtain the authors’ names and email addresses

As the paper’s Editor, you chair the ad hoc paper review committee that also includes the paper’s reviewers. Typically, by the time you receive the assignment email, the EiC has already assigned reviewers to the paper. The EiC selects reviewers for your paper based on a large number of criteria which includes similarity of topical interest and current workload of the reviewer. Your role is to act as Chair of this paper’s ad hoc review committee. You will have a different ad hoc paper reviewer committee for each paper assigned to you.

Step 2: Read the paper and develop your own thoughts on how it might be improved

Since you will be synthesizing your first thoughts with the suggestions you obtain from your ad hoc committee of reviewers in about a month, read the paper yourself before you read the reviews.

Target dates for Reviewers and the Editor. Typically, the EiC sets the target date in the system for reviewers to complete their review for two weeks hence. We set a short target date on purpose. We don’t expect all reviewers to complete their review by this target date. We know from experience that some colleagues don’t respond until a review is well past due. We set the target date early to allow time for the system to send a reminder to reviewers. We understand human nature that some reviews will come in late, so we recommend that you start working on the development letter a couple of weeks after the reviewers’ target date; typically this is four-weeks after you received the assignment letter. This scheduling provides time for these late reviews to arrive. The EiC usually sets your target date at four weeks after the assignment for this reason.

Step 3: Make your recommendation regarding acceptance, insert the appropriate Best Practices Template, and insert your Development Letter into it

The Board of Governors has devoted much time, thought, and discussion to create some ‘Best Practice’ templates using wording that best represents the ISI philosophy. Indeed, these templates are designed to ensure consistency, completeness, and coherence with ISI practice and philosophy. We want you to use these templates as the basis for your communication with authors.

You can find these templates in our Notepad on the 5. Decision tab. Select the appropriate template and Insert it into your dialogue box. Once the template’s wording is in the comments for author field, modify it as needed, including inserting your wording on how the author needs to improve the paper; that is, copy and paste the wording of your Development Letter. Note that the system will fill in for you the placeholders; for example, [ARTICLE-ID], [ARTICLE-TITLE]. When you click the PREVIEW button, you see the message formatted with the placeholders filled in.

5

Figure 3. Select and insert the Best Practice Wording, then paste your development letter into it; the system will fill in the placeholders automatically

The Board of Governors continually improves the Best Practices templates so use only these and do not create your own local copies of letters. Since the templates fill in the form for you, why make extra work for yourself?

The Development Letter you compose for the paper goes into the specifics of what you, as Editor, suggest to or require of the author to make the paper better. So first Insert the templated wording and then either copy the contents of your Development Letter into the proper place of this wording or attach the file with your comments where the template reads == Insert Development Letter Here ==.

You also tell the system the target date for the author to submit the revision. The system uses that date to know when to send reminders. You can change that date; for example, if the author asks for an extension (Figure 4).

Figure 4. If the author asks for an extension to submit the revision, click on the pencil icon to modify the due date; the system uses this date to send reminders

For quality assurance, the Editor-in-Chief reviews letters to the author before having the system send them.

6

Step 4: Work with the author on revisions

The system sends your letter to the corresponding author only. The system does not send it to co-authors to avoid confusion and contradiction when there are multiple authors.

The system gives the author the choice to revise the paper according to your requirements or to ‘choose not to revise.’ If the author chooses the revise, s/he needs to upload two documents: the revised paper, and the ‘revision document’ in which the author tells you how the revised document addresses the concerns posed in your development letter.

Formatting for publication. The ‘Accept subject to revision’ letter recommends to the author to save time by formatting the next draft for publication. Waiting until all the revisions are completed to format just delays publication. But not all authors follow this advice and sometimes a revised formatted paper loses its formatting when the author further revises the paper. In any case, the author needs to submit a fully formatting paper before you accept it. A template is available at

http://www.informingscience.org/Uploads/JournalFormat.docx.

You accept a paper for publication when the revision meets all these conditions:

1. The revision contains all the requirements specified in your development letter. All sentences and paragraphs make sense and are free of grammatical and spelling errors.

2. The paper conforms to the formatting guidelines for the journal. If it doesn’t, use the best practice template: ‘Step 3 Acceptance Letter – BUT camera-ready formatted copy with author bios and photos still needed’ to fill in the wording and recommend ‘accept pending revisions.’

3. The references are in APA format; while they may not be perfect, they should include all bibliographic information.

4. The formatted paper contains a brief biography and a photo (head and shoulders) for each author at the end.

Step 5. Accepting a paper for publication

When the paper needs no further work, use the ‘Step 3 Final Acceptance Letter – ONLY for FINAL, camera-ready formatted copy with author bios and photos’ template. The wording in this template congratulates the author and provides information about the publishing process. Then select the ‘Accept’ button in the dialogue box as your recommendation.

NEVER accept a paper until you have read it and thoroughly checked it to ensure that it meets the four conditions above. Once you click the ‘accept for publication’ button, the system moves the paper, and the author can no longer make changes to it.

Once you accept a paper (and the Editor-in-Chief approves your recommendation), the paper moves on to the publisher’s queue. The publisher reviews the paper and checks if it is ready for publication. She may contact the authors directly to check on or ask for information, such as references. It is not her job to copyedit the paper. Usually, the paper will be online within two weeks.

7

LET GO OVER THIS AGAIN IN STILL GREATER DETAIL

Here are more specific steps you take as Chair of the ad hoc review committee for a submission:

1. Read the submission and assemble your thoughts on how the author might improve it. But don’t close your mind; wait to hear how the reviewers react to the submission.

2. Check the reviewers’ progress after a few weeks from your dashboard at http://My.InformingScience.org. Give the reviewers time to complete their review, at least until a week after their target date. Anytime after that when you have at least three reasonable quality reviews, you can start to compose your Development Letter that synthesizes your thoughts and the reviewers’ thoughts for improving the current version of the submission.

3. Four weeks after your initial assignment, you should have at least three useful reviews. What if you don’t? In that case, email the Editor-in-Chief (the EiC’s email address is in your assignment email) to follow up with the assigned reviewers or to appoint new reviewers should that be necessary. (Some Editors-in-Chief may ask you to email the non-responsive reviewers.) See the reviewers’ email addresses on your 4. Peer Review tab.

4. When you have finished composing your development letter (after the initial reviewer target date when you have sufficient feedback from reviewers), insert the Best Practice letter template ‘Accept subject to revision’ or ‘Reject’ and add your developmental comments. You can add your Development Letter either by cut and paste into the template or, if need be, by uploading your formatted file. When you upload a file for the author, the system inserts a link (at the point where you currently have your cursor) to your uploaded file. While it is more awkward for the author to download an inserted file, you may find it more useful in communicating complex thoughts.

5. The wording you use in your development letter is critical. Ensure that you write your letter with a single voice and that you share with the author practical advice for improving the paper using supportive language. To be clear, never write “reviewer 1 said this, reviewer 2 suggested that”, but instead tell the author your summary, that is what changes you recommend or require. Your development letter can use any or all of the suggestions your review board provided you as chair of the ad hoc review board. You can even use a reviewer’s exact wording, should you wish, since you are writing as chair of the committee. At other times, you will need to rephrase the views of the reviewers. It is important that comments from different reviewers do not contradict each other. A significant aspect of being an editor is synthesizing and determining the most appropriate comments or suggestions to be included.

6. Good development letters include positive comments about what the author did well. When you criticize, criticize the paper, not the author; but it is fine to praise the author. Phrases like “Your paper will be even better by doing this or that” expresses encouragement. Make your comments clear and as detailed as possible, giving examples. You are serving as the coach, not the referee. See the sample development letter in Appendix B.

7. When you click on the submit button, the system sends your recommendation and comments (the template with your Development Letter pasted in) to the Editor-in-Chief for quality assurance review. After reviewing it, the EiC ‘submits’ your letter, and the system emails a copy of it to the author.

8. If the decision is to ‘reject’ the paper, the process stops here. An ‘accept subject to revisions’ decision requires further action as indicated in your development letter. The next section describes the next steps.

8

Two Features for You: The paper’s Activity Log and the paper’s Notes for EiC and Editor towards the top of the screen (Figure 5).

1. The system keeps an activity log for each paper that records all decisions and communications made through the dialogue box. You can review all this activity by clicking on the paper’s Activity Log.

2. Use the Notes feature to post comments related to the Development Letter and the review process. If you have a private email conversation with the author, put a copy in the Notes, or perhaps just important paragraphs. These notes are shared only with the EiC or Associate EiC.

Figure 5. The activity log and the Notes features (notes for a paper are seen only by the Editor and the EiC)

Reviewer rating

You and your fellow Editors across all ISI journals help us develop our ISI reviewers. You do this by rating the quality of each review (Figures 6, 7 and 8). When you do, you also provide the reviewer with brief feedback about their review that the system emails to the reviewer. The rating you provide makes it possible for us to identify reviewers whose work needs improving and those whose work needs commendation. Our goal is to help all reviewers to develop their skills and to identify excellent reviewers for acknowledgment and eventual promotion to associate editor or editor. We rely on you and your fellow editors to provide us with quality feedback about reviewer performance.

Almost all reviews earn a ‘3’, exceptionally good reviews earn a ‘4’ or ‘5’ and those found lacking earn a ‘2’ or ‘1’. This rating is used by all ISI journals to identify which reviewers can use more training and which to commend for their excellence and possibly promote. As you rate each review, you will provide the reviewer with brief feedback on the quality of the review and possibly how future reviews can be more suitable.

9

Figure 6. Reading and rating reviews – you can view any single review by clicking the date in blue or select all reviews from the bottom of this Peer Review tab (Figure 7)

Figure 7. To read an individual review, click the date; to rate the review, click RATE

10

Figure 8. Rate each review and provide the reviewer with personal mentoring

The decision to ‘Not accept’

Before you receive a submission, the Editor-in-Chief reads the paper and believes that, with revision, it can be made publishable. We endeavor to work with authors to help them to revise their submission to a state where it deserves publication. But there are times when the Editor needs to let the author know that we will not be pursuing publication of the paper any further. In that case, you select ‘Not accepted’ (Figure 9), insert the appropriate best practice wording from your Notepad, and copy and paste or upload your development letter into the best practice wording that you just inserted. Once the Editor-in-Chief does the final quality assurance check, the system emails your wording to the author and removes the submission from the journal’s queue.

The decision to ‘Accept subject to revision’

When you select ‘accept subject to revision’ (Figure 9), the system sends an email to the corresponding author asking the author to go to the review site at http://My.InformingScience.org to let the system know whether or not the author intends to revise the submission. If the author does not intend to revise, the process stops here. If the author decides to revise and re-submit the paper, you will work directly with the author on the resubmission. For papers with co-authors, the system notifies only the corresponding author, and only that person can interact with the system to accept or upload revisions.

11

The author who accepts the ‘pending revision’ needs to submit two files. The first is the revised article. In the second, the revision document, the author describes the changes made and if other suggestions are not incorporated in the revision, why not.

The next step is for you to review the changes the author made and decide whether or not the author fulfilled all the requirements stated in your development letter to your satisfaction. It is common for a paper to go through two or three rounds of revision before it is ready for publication. As the paper gets closer to being ready for acceptance, the editor should ensure the author has formatted the paper to be near camera-ready according to ISI guidelines (a template is available at http://www.informingscience.org/Uploads/JournalFormat.docx), and the references are formatted in APA style (a guide is available at https://www.informingscience.org/Uploads/APA_2017.pdf). The template requires that the near camera-ready version includes a brief biography for each author along with a head and shoulders photo at the end of the paper. Doing these steps before the final version saves the author time in getting the paper published.

Figure 9. The Decision tab: select a recommendation, insert the appropriate template, and then insert your development letter into this wording

The decision to ‘Accept for publication as is’

You MUST carefully check the final version before selecting the ‘Accept for publication as is’ button on the system by reading through the revision to ensure the following:

1. it contains all changes you required;

2. the grammar and spelling are of a publishable standard;

3. the formatting conforms to ISI requirements, including the APA formatting of references; and

12

4. most importantly, re-read the paper to ensure that all sentences still make sense. This reading is necessary because often authors just make changes without re-reading their paper and the submitted version does not make sense.

Please understand why this is important. Once you click the ‘accept’ button, the system sends the paper to the publisher. You and the author will no longer be able to interact. Prematurely accepting a paper creates a mess for the author, for you, for the Editor-in-Chief, and for the publisher.

When the paper needs no further work, use the ‘Step 3 Final Acceptance Letter – ONLY for FINAL, camera-ready formatted copy with author bios and photos’ template. This step emails congratulates to the author and provides information about the publishing process.

Select the ‘Accept’ button in the dialog box.

Editor rating

We already mentioned how you rate each review. Our AEiCs and EiCs rate the quality of each editor’s Development Letter. We use this feedback to identify potential Associate Editors-in-Chief and to provide support for those who will benefit from it.

When the time is right, you will earn the promotion to Senior Editor, and then Associate Editor-in-Chief and help even more colleagues help us all become better and better at our professions.

Your availability

We know that there are times when you will not be available to accept assignments. To avoid being assigned a paper during these periods, just let the system know.

1. If you will be unavailable to act as Editor for an extended period, change your availability status on your ISI profile. (‘Sign On’ to your ISI account at http://My.InformingScience.org to change your availability status in your Profile and indicate the period of your unavailability). Letting the system know you are not available ensures that the EiC will not assign a paper to you during that time.

2. If you need more time to complete your Development Letter, email your Editor-in-Chief as soon as possible. The EiC will change the target date (and tell the system not to email you reminders).

Thank you

Thank you for your service to the academic community and for being a leader to improve the academic paper review process. Some journals focus on rejecting as many papers as they can, others on accepting almost all. Our journals’ mission is to help all authors and reviewers to improve their craft as researchers. We do this by getting like-minded people together to mentor each other. As an editor, you are a leader, guide, and coach for authors and your reviewers. Thank you for your service.

APPENDIX A. WRITING YOUR DEVELOPMENT LETTER

Your main intellectual achievement is to create a mentoring, ego-acceptable Development Letter for the authors of all submissions, whether or not you recommend for advancing the paper for publication. As noted above, you write your development letter as chair of the paper’s ad hoc review committee, using your own best judgment.

All Development Letters need to follow these guidelines:

1. It speaks with ‘one voice’ (rather than a range of differing views). It is important that comments from different reviewers do not contradict each other.

2. You may use words from reviewers verbatim without quotes if they are appropriate. At other times, you will need to rephrase the views of the reviewers. A significant aspect of being an editor is synthesizing and determining the most appropriate comments or suggestions to be included.

13

3. Write your development letter in supportive, constructive language that avoids harsh criticism. Think of how you would have liked to receive criticism when you were writing your first journal paper and use that tone. Use terms like “this is how you can make your paper even better…”. You had good training from your school experience on how to write journal papers, so please share some of your wisdom with the authors you mentor.

4. Criticize the paper, not the author. So, word your letter appropriately, and it will have a greater impact. For example, avoid using the word ‘you,’ and instead voicing the advice in terms of revising ‘the paper.’ For example, rather than ‘you need to update the literature review,’ write ‘the literature review will be stronger by adding more recent studies. These might include for example…’

5. Be sure to include positive comments about what the author did well.

6. Most importantly, provide feedback that is intended to help the author improve their work:

• make your comments as clear and detailed as possible and give examples;

• give feedback that identifies specific problems and is accompanied by practical suggestions for addressing the issues; and

• provide suggestions for additional relevant references and for an alternative interpretation of results, where possible.

In other words, be a coach, not a referee.

Below are snippets from a Development Letter that illustrate the mentoring approach we ask our editors to take.

Introduction: I recommend that the next revision includes a section with details on the environment that led to the research (economic, political, etc.). Without it, the reader is left to infer details and may infer incorrectly. For example, make it clear if the program was mandated, what funding was available, if and how technology was distributed to schools and students.

Literature Review: Adding a review of Change Theory literature here will provide a stronger basis for the subsequent Results and Findings. A search at http://ISJournals.org can suggest literature ISI has published related to this. Including more recent references will strengthen the literature review and demonstrate that the paper takes into consideration the most recent developments.

Method/Methodology: Consider rearranging the opening paragraph in this section so that lines 213 to 216 follow line 217. Doing this would improve flow. I recommend that your paper make further justification of the decision to use mixed methods since that would build a stronger case for accepting the results of the analysis. Have the paper explain to the reader what a mixed methods study does that quantitative method alone cannot. This justification might well fit as part of the Literature Review.

We recommend that you compose your Development Letter offline and then copy and paste it into the appropriate Best Practices template (accept or reject) in the system. If formatting is important, you can upload your Development Letter into the comments at the appropriate location into the template after you have inserted the template. But realize that doing that makes it more difficult to convey your information to authors on mobile devices and other small screens.

APPENDIX B. SAMPLE DEVELOPMENT LETTER

Above we saw snippets of wording from Development Letters. Here follows a specimen (example) development letter. It gives you an example of the wording of a real development letter that is both supportive (in wording) and touch (requiring substantial but do-able improvements).

The ad hoc board of reviewers and I enjoyed reading your submission and want to work with you to get it into a publishable state. As strong as this draft is, I need for you to improve it in several areas before acceptance for publication. Your published paper will be read and cited for decades so let’s spend a few more hours on it.

14

With this in mind, please consider the following suggestions. While I do not require that your revised paper accept all these suggestions in the next revision, I do require that you upload a ‘revisions document’ in addition to your revised paper. In the revision document, show how your revised paper addresses and responds to each of the following suggestion. Do this point by point for each of the following suggestions:

1. I recommend condensing the Abstract. The abstract is a precise summary of the paper, and its purpose is to provide the reader with a preview of the contents. Aim for 1-2 sentences for each section of the Abstract.

2. I suggest strengthening the literature review to show how this paper builds on prior research. For this and any other topic, I recommend using the ISI full-text search engine at http://ISJournals.org to see if there are other papers published in this or other ISI journals related to the topic. All papers found here are available online for free, so this is a very useful tool.

3. Here are some other papers that relate to this paper’s topic for you to consider as you strengthen the literature review:

a. [the development letter included 11 current papers on the topic, but they are omitted from this sample letter]

4. I highly suggest hiring a copyeditor to polish the grammar and spelling in your next draft. While your paper has much merit, we can publish it only after you fix the grammar and spelling. Here are examples of errors from the first couple of pages. As these are just examples, your revised paper needs to be copyedited by someone capable of catching errors throughout your next draft.

a. [the development letter included 12 examples of problems in the first few pages]

5. The copy editor can likely also help you to reduce the paper’s length. Your paper will be stronger after vigorous editing to slash unnecessary verbiage. Good research papers for this journal are concise, just long enough to make their point.

6. I recommend solving the methodological problem of multiple, and possibly, confounding variables in the study. The research would be improved by controlling some of the loose variables, such as the class size, or the time that the class was offered, and adding the format of the courses where the evaluations were conducted. Or it might be possible to state why these variables were excluded from the study.

7. Your revision should make it clear:

a. Whether the courses were online, hybrid or in person, or if just the evaluations moved from being in person to an online platform.

b. Were the students given an incentive to complete the evaluations in either format?

c. Were the students guaranteed anonymity and privacy?

d. That this is a pilot study and that the results may not be generalizable to other institutions.

8. Please consider adding graphics (with captions) since they will improve the readability of your paper.

9. The Discussion section could be expanded to include more interpretation of the results and relate the findings back to the literature discussed in the Literature Review section.

10. The conclusion of this paper would be improved by adding a Conclusion section, even if it is just 5 or 6 sentences long.

11. Check that references conform to APA style. A style guide can be downloaded from https://www.informingscience.org/Uploads/APA_2017.pdf