THE CHALLENGES AND OPPORTUNITIES OF PEER REVIEW IN HEALTH IMPACT ASSESSMENT

Background: While HIA guidelines and practice standards are used throughout the field, peer review is a potentially untapped resource for HIA practitioners in the US and potentially internationally. Peer review is thought to strengthen HIA practice, although very few guidance documents exist, and there has been little research to date on the efficacy of peer review for improving HIAs. Methods: To explore the possible value of peer review in HIA, an expert panel was convened at the 2013 HIA of the Americas Workshop, and an online survey was used to query HIA practitioners regarding their experience with and motivation for HIA peer review. Results: Most survey respondents (n=20 out of 26) indicated that peer review in HIA was helpful, and 15 respondents thought a formal peer review process would improve HIA practice. Respondents wanted peer review to be timely and the reviewer to approach the review as a mentor rather than a gatekeeper. Conclusion: This paper offers the initial development of a peer review typology based on feedback from the online survey and workshop participants. Better understanding of the potential challenges and opportunities for using peer review in HIA may help to improve HIA practice.


Introduction
In the past 30 years, health impact assessment (HIA) has developed into a tool used in many sectors all over the world (Vohra, 2007).Countries such as Australia, England, Thailand, and the Netherlands have integrated HIAs into formal decision-making processes.In the United States (US), although initiation of HIA practice occurred later than in other parts of the world, practice has grown 10-fold in the past decade from 27 completed HIAs in 2007 (Dannenberg;Dannenberg et al., 2008) to 407 completed or in progress HIAs in 2016 (The Health Impact Project, 2016).The diverse and growing practices in HIA in the US have called atten-tion to the need to improve overall HIA application (National Research Council, 2011).One potential area to advance HIA practice in the US (and potentially internationally) is through better consideration of the use of peer review.
Peer review is the evaluation of a process or product by experts in the field to maintain or enhance the quality of the process or product in that field (Smith, 2006).Peer review plays a critical role within the scientific community to improve the quality and applicability of research and evidence (Abelson, 1990).While peer review often occurs after submission of research manuscripts for publication or for proposals for funding, other types of formal and informal peer review processes are used at different stages of research (Solomon, 2007).These other types include collegial review of products before they are submitted and reviews of outlines to ensure that a proposed product is well-designed.Some institutions may require that reports and other documents undergo internal review prior to being shared externally.Though peer review is considered necessary for maintaining scientific standards and quality control, it is subject to its own set of challenges such as bias, complexity, and a lack of understanding of its overall effectiveness (Goldbeck-Wood, 1999).
Peer review has been conducted in environmental impact assessment (EIA), and though it is not required, it is a recommended practice (Office of Management and Budget, 2004).In EIA, peer review tends to focus on the technical quality of assessment standards, methods and results in order to ensure attainment of appropriate levels of scientific rigor (Beanlands et al., 1983).In some EIA processes, peer review is conducted by the contracted agency by specialists not involved in the work.The report authors respond to the reviewers' comments and make necessary changes to the scientific report.In some cases, the peer reviewers' comments and the authors response may become part of the public record (Klamathrestoration.gov).Given that the focus of peer review in EIA is to ensure scientific integ-rity, it is recommended that peer review be conducted at the inception and design stages, though given the practical complexities of environmental assessment, it is not clear to what extent this is actually completed (Beanlands et al., 1983;Chaker et al., 2006).It is also argued that as an applied science, EIA should conform to the same rules and standards that govern scientific research and therefore EIA reports should be subject to peer review (Cashmore, 2004).
Incorporating peer review in HIAs may be one strategy that can help to improve the quality and usefulness of HIAs.Some guidance exists, particularly for peer review at the end stage of the HIA (Fredsgaard et al., 2009;Vohra, 2005), however there has been little research to date on to what extent, and in what manner, peer review can be used to improve HIAs.Given that HIAs must be conducted within policymaking cycles (Harris et al., 2014), it is unclear how peer review processes can account for timing restrictions and negotiating of interests from multiple stakeholders within diverse HIA projects.Based on a practitioners workshop and subsequent survey, this paper explores the challenges and opportunities presented by using peer review to support HIAs.We examine the current use of peer review by HIA practitioners primarily in the US, the perceived value of peer review to the HIA process, and provide a typology of peer review practice in HIA.

Methods
The development of an HIA peer review typology was initiated by a working group convened at the 2013 HIA of the Americas Workshop (changed to the HIA Practitioner Workshop), a meeting for HIA practitioners to discuss the state of the field and plan steps for future improvements.Workshop participants (n=11) self-selected to attend the working group entitled "Peer Review of HIA."All participants had previously been involved in one or more HIA and had some level of experience with peer review in the HIA process.The workshop was facilitated by two of the paper authors (KR and TC) and incorporated an open format to discuss the potential role of peer review in HIA practice in the US.Most of the paper authors attended the workshop.During the conference, the working group discussed the potential range of peer review which might be applied to HIA.Additionally, participants described the positive and negative factors of applying peer review across a range of HIA typologies.Participants also looked at the HIA process (see Figure 1) and identified at which steps peer review could be beneficial.
Following the meeting, the authors designed and distributed an online survey to investigate the experiences of practitioners with peer review of HIA projects (see Box 1).The survey consisted of eight questions designed to gauge the motivations of HIA practitioners to engage in peer review, the perceived value of peer review, and the broad and multiple practices of peer review within HIA.The authors did not strictly define peer review in the survey so that a full range of experiences with peer review, both formal and informal, could be captured.The 1. SCREENING Determine whether an HIA is needed and likely to be useful.2. SCOPING Develop a plan for the HIA, including identification of potential health risks and benefits.3. ASSESSMENT Describe the baseline health of affected communities and assess the potential impacts of the decision.4. RECOMMENDATIONS Develop practical solutions that can be implemented within the political, economic or technical limitations of the project or policy being assessed.5. REPORTING Disseminate the findings to decision makers, affected communities and other stakeholders.6. MONITORING AND EVALUATION Monitor the changes in health risk factors and evaluate the efficacy of the measures that are implemented and the HIA process as a whole.

Sample
A total of 26 HIA practitioners responded to the survey.Respondents represented a variety of organizations although the majority were affiliated with governmental agencies (n=12) (see Table 1).The majority of respondents were from the US.

Respondents' experience of peer review
The roles of respondents in the peer review process varied but generally fell into three categories: • HIA project lead -oversees the project and drafting of the HIA report and may have received a peer review on their HIA; Several respondents stated that they performed multiple roles and respondents listed "other" roles, including moderator (conducts peer review process), in the peer review process (see Table 2).
Respondents described several different types of peer review: 1. Technical -review by "qualified statisticians and scientists" for "technical accuracy;" 2. Process -review by "HIA experts" to "ensure all steps of HIA were addressed;" 3. General -review for "identification of oversights, corrections," "general edits," and "flow and readability" and; 4. Political -review for "what in the document seemed biased," to avoid "hot buttons," and "unnecessarily alienating the local audience." Some of these types of review may be interpreted more as a type of technical assistance, rather than traditional peer review, however the authors have included all types of peer review in order to reflect the respondent's perception of the meaning of peer review in HIA.

Perceived value and challenges of peer review
Although the overwhelming majority (n=20) agreed or strongly agreed that peer review was helpful for improving the quality of the HIA, fewer respondents agreed (n=15) that HIAs would benefit from a more formal or standardized peer review process, and six participants disagreed that a standardized peer review process would benefit HIAs (see Table 3).
Generally, respondents described peer review as beneficial or positive.Respondents commented that peer review identified needed corrections or missing information.Respondents stated that it also validated "HIA leaders' concerns" and provided further opportunity for answering questions.In addition, participants felt that peer review helped to identify additional data sources, legal citations, and publications to consider or include, and to clarify language and framing of the report.Respondents identified peer review as helping to refine logic models and pathway diagrams, and providing useful feedback on recommendations.They highlighted that having the HIA reviewed by statisticians, scientists, and other qualified reviewers was crucial to ensuring the technical accuracy of the HIA, the alignment with methodological best practices, and the relevance of recommendations.Respondents also reported that peer review helped to produce a more credible product, to increase confidence of HIA staff in their findings, and to present the information clearly and effectively.
Respondents also reported that peer review aided incorporation of perspectives of different stakeholders within the HIA process.They stated that diverse viewpoints of people with dissimilar skill sets add value by providing input and perspectives on issues not apparent to those leading the HIA project.Respondents suggested that diversity also adds credibility to the analyses, findings, and recommendations, making the HIA stronger overall.
Respondents stated that the timing of peer review was important.One respondent commented that retrospec-tive reviews of finished HIAs do little good; the key to an effective review is to engage the reviewer in the HIA process early enough to address issues and make changes.Another respondent stated that incorporating peer review in the early stages of HIA helped to avert complications that would have been more problematic later in the process.
Respondents identified several challenges to peer review.Some found that time limitations impeded addressing and incorporating feedback.One respondent mentioned that peer review could add value, but could also increase the time and resources needed to complete an HIA and present a different set of barriers and constraints for the project.Another respondent stated that HIAs are often conducted under tight deadlines by already busy staff, and although reviewers add credibility by identifying realities about the HIA being conducted, HIA staff may not be able to adequately react to constructive comments.Respondents suggested that peer reviewers need to be matched to the work based on their own specific talents, skills, or time constraints.One respondent noted that the benefit of the review is highly dependent on the reviewer.
Inadequate communication between the reviewer and those leading the HIA was also mentioned as a barrier.
One respondent found peer review to be useful but indicated that the lack of communication between the reviewers and the HIA team can lead to misunderstandings.Peer reviewers of journal article submissions are usually seen as gatekeepers, this respondent stated, but HIA peer review needs to be less anonymous to ensure that feedback is useful.The respondent suggested that reviewers act more as mentors.
To address some of the challenges of conducting peer review of HIAs, respondents described the need to clarify the purpose of the peer review, the type of review (e.g., general or technical), the level of review, and the timeframe for comments.Respondents suggested providing parameters or instructions to reviewers, such as specific questions, concerns, or content areas for the reviewers to consider.They noted that providing such details seemed to help manage the work of the peer reviewers, the expectations of the HIA project team, and the goals of the peer review within the constraints of the overall HIA project.Varies: may be desirable in most cases Practitioners often viewed the peer review they received as helpful but were less supportive of a standardized process.Additionally, they described an application of peer review to HIA that would be different than the application to a journal article (see Table 4).For example, the review process should be sensitive to the timeliness of the HIA, the reviewer might act more as a mentor rather than a gatekeeper, and different types of review could be applied, such as a technical review or a general review.

Discussion
Based on the open-ended questions querying practitioners about their perceived motivation, value and use of peer review in HIA, we found that peer review can address many aspects of an HIA: • Process (e.g., did the HIA follow the steps of HIA, did the HIA involve significant stakeholder input throughout the process).
• Analyses (e.g., were quantitative and qualitative analyses performed according to best scientific practices in their respective fields, are the analyses transparent and replicable).
• Recommendations (e.g., are the recommendations based on the analyses, are the recommendations politically feasible).
• Reports (e.g., was the final report comprehensive, was the final product written in a format understandable to and useable by the appropriate audiences to inform the decision-making process).
Our findings highlight several key considerations for integrating peer review into HIA practice.First, almost all the respondents to the survey reported that peer review, when performed early in the process, when timely and cognizant of resource constraints, and when targeted to the particular step and needs of the HIA, is a helpful practice to improve the quality and applicability of an HIA.When feasible, peer review should begin as early as possible in the HIA process and be incorporated in all steps.
Second, our findings indicate that peer review in HIAs should be fit-for-purpose.Because the HIA process is complex and can involve different disciplines and expertise that are subject to improvement by review, peer review should be conducted in a way that is flexible and appropriate to the needs of the individual HIA.
In order to represent the different types of peer review being used in HIA practice, and when they are best applied, we propose a typology of HIA peer review (see Figure 2).The typology includes process, technical, general, and political peer review.Each of the types of peer review provide different information to improve the HIA, can be applied at different stages in the HIA process, and may require a different reviewer depending on their individual skill set.
Process review involves ensuring that the HIA performs all steps in a manner consistent with published HIA guidelines and best practices.Process review could also help to ensure the inclusion of equity considerations in each step of an HIA.A process reviewer could draw on their experience as an HIA practitioner and be well versed in the many practice guides and standards available.
Technical review ensures that the qualitative and quantitative data analyses were performed according to best practices and applicable scientific standards, and that the level of evidence and uncertainty for the predictions was stated clearly.The qualifications of the technical reviewer will vary according to the decision and health issues being assessed in the HIA.
General review includes general editing of documents and can include review related to best methods of communicating results to appropriate audiences.A Political review helps to understand and negotiate the political space in which the HIA occurs.This is especially important for politically sensitive HIAs, often the norm rather than the exception.Politics can influence whether the HIA is undertaken, which health issues are addressed (e.g., scoping and pathways), the recommendations, and the reporting of the HIA.A political reviewer has a firm understanding of the politics and context surrounding the HIA and provides insight and advice to the HIA practitioner to ensure that the HIA recommendations are salient to the decision makers.
The four types of review can be applied at various stages of the HIA process (Table 5).
A peer review typology should consider the HIA steps as well as the different typologies of practice (Harris-Roxas et al., 2011).The type of review should be expanded or minimized according to time and resources in accordance with the type of HIA (rapid, comprehensive, etc.).For example, minimally-resourced and time-constrained HIAs might only have the capacity for one reviewer, who may perform several types of review at different stages of the HIA.To facilitate and expand peer review opportunities, a pool of potential HIA reviewers could be drawn from identified authors of HIA reports or from existing HIA communities of practice, such as the Society of Practitioners of Health Impact Assessment (SOPHIA, n.d.) .
In summary, several key considerations are important for conducting peer review in HIA.
• Timing and coordination: Conducting peer review early within the appropriate stage of an HIA helps suggested changes and recommendations to be meaningfully addressed.Reviewing an HIA after it is completed may do little to improve the HIA itself.The HIA coordinator or project lead may also need to build in time to possibly respond to or address peer reviewers' recommendations.
• Reviewer fit: The value of the review is highly dependent on the reviewer.Peer reviewers should be matched with the right type of review needed based on their specific skills and availability.
For example, if a technical review is needed, the peer reviewer would be well versed or have had experience in conducting the methodology used in the analysis.
• Peer review scope: Clarifying the purpose and scope of the peer review is essential.Providing parameters or instructions for reviewers -such as specific questions to consider, areas of the HIA in need of attention, time constraints of the project, and type of review (i.e., general, technical, process, or political) -will help to increase the usefulness of the peer review and ensure that the issues of greatest concern are most likely to be responded to and addressed within the given time frame of the project.
• Communication: HIA peer review benefits from an open dialogue between the HIA lead or project team and the peer reviewers.Good communication will decrease the likelihood that peer review recommendations will be misunderstood, will increase the usefulness of the comments, and will increase the potential for the peer review to improve the HIA by clarifying any points of concern or suggestions.

Limitations
Our study and findings are subject to several limitations.Initial input from practitioners was obtained only from participants at the HIA of the Americas Workshop, and survey information from only a subset of (mostly US) HIA practitioners.Given that most respondents were from the US, the results cannot be taken to reflect the views of practitioners in other international settings.However, while our survey was relatively small, we believe it illustrates a range of useful perspectives on an evolving HIA practice.
Respondents were solicited from a pool of experienced practitioners, and the working group and authors of this article also have a diverse range of HIA experience.Still, due to the nature of the convenience sampling, and the small number of respondents, we may have excluded other experiences and perspectives on HIA peer review.Additionally, the survey was only sent out to HIA practitioners, and not more broadly to those conducting other forms of impact assessment.
While the focus on HIA practitioners helped to elucidate HIA-specific issues and opportunities for peer review, other insight related to peer review within other impact assessment may have been missed.Moreover, the term "peer review" could be interpreted by respondents differently; different potential interpretations were apparent from discussions at the initial working group meeting and survey responses.Future research might better define peer review and address similar questions in a more representative sample.Nonetheless, we believe our findings highlight several opportunities for improving the practice of HIA through peer review.

Conclusion
Peer review in the context of HIA can be both an endstage quality control measure and an iterative quality improvement process used throughout multiple steps of the HIA.HIA provides timely and valid evidence amid myriad scientific and political uncertainties.
Peer review of HIA may be an opportunity to support the legitimacy, acceptance and utility of the research findings, thereby increasing the value of HIA in decision-making.However, further incorporation of peer review in HIA will need to consider the potential pitfalls and criticisms of peer review practice (i.e.bias), the types of practitioners that can conduct peer review (who precisely is considered a "peer") and how this practice can be applicable in international settings.

Figure 1 :
Figure 1: The Steps of HIA

Figure 2 :
Figure 2: Typology of HIA Peer Review

Table 1 : Survey respondent characteristics
Based on your general experiences of using peer review in HIA, indicate how much you agree with the following statements: "I think HIAs could benefit from a more formal or standardized peer review pro-What has been your role in the peer review process?6. Briefly describe your experience overall with the HIA peer review process: Having an HIA reviewed (skip, if not applicable.)7. Briefly describe your experience overall with the HIA peer review process: Reviewing an HIA (skip, if not applicable) 8.If peer review added value to the HIAs, please specify how.
• Technical reviewer -has expertise in a given field and reviews part of the assessment; and 4.• General editor -provides non-technical revisions to the report.

Table 2 : Respondents' role in HIA peer review
aRespondents could report more than one role