Evaluating the Effectiveness of An Evidence-Based Research Study: A Pilot Study

^aMedical Librarian, Information Commons, California University of Science and Medicine, Colton, CA, 0009-0009-5459-1313, Deborah.farber@cusm.edu

Abstract

Background:

Over two semesters, California University of Science and Medicine (CUSM) medical students complete a course called Academic Research Study in which they collaborate in groups to design and conduct research studies. However, instructional sessions for this course are only scheduled during the first half of the course. The medical librarian presented a 1-hour overview of literature searching during one of these sessions, which is insufficient time to explore critical appraisal. Anecdotal evidence from subsequent research consultations indicated that students need more time to practice critical appraisal.

Experience:

The medical librarian conducted a literature review to identify effective instructional strategies and assessment tools. The medical librarian designed a workshop for medical students, using materials adapted (with instructor permission from the Community of Online Research Assignments (CORA)). The Fresno Test was also modified to align with learning outcomes. Students were given 15 minutes after the workshop to complete the 6-question case-scenario based survey which focused on developing an appropriate research question, identifying appropriate study types and sources of information, as well as criteria for determining research quality.

Discussion:

The study took place over two years, with two different cohorts of second-year medical students. Only 13 students participated in the research study between the preliminary and study phases. The small study size and cross-sectional research design makes it impossible to generalize findings. However, all study participants were successful in completing the survey and accurately identified the research question, key search terms and measures of research quality.

Takeaways:

California University of Science and Medicine utilizes an 18-month curriculum for pre-clinical training. As a result, second-year students are on campus until the end of their second fall term and have limited time for extracurricular programs. The medical librarian designed a workshop to address the need for additional critical appraisal practice. The plan was to assess students’ knowledge before the workshop, but time limitations made this difficult. Furthermore, students were not assessed before the workshop, so there is no proof that their skills changed, yet the modified Fresno Test proved to be an effective assessment tool for a short instructional session on critical appraisal.

Background

California University of Science and Medicine (CUSM) transitioned from a 24 to an 18-month curriculum when the United States Medical Licensing Exam Step 1 (USMLE) became pass/fail in 2022. The condensed curriculum enables students to take part in a six-month Professional Development Semester before or after completing core clinical rotations to distinguish themselves from other residency candidates in year four. However, it emphasizes health systems science, leaving little room for dedicated evidence-based medicine (EBM) instruction which includes critical appraisal.

CUSM medical students are required to complete two-semester course called Academic Research Study during the first 18 months of the four-year curriculum. This course incorporates some aspects of EBM. Students work in groups to collaboratively author different research proposals in the first semester of the course and conduct the studies in the second semester. Instruction sessions on bias, validity, and statistical significance are scheduled during the first semester over the first four weeks. The medical librarian only has an hour in this schedule to provide an overview of literature research in the biomedical sciences. While the medical librarian briefly examines study types during instruction on PubMed search strategies, critical appraisal is discussed in a different session by a faculty member. Instead, the medical librarian focuses on identifying appropriate search terms from a pre-determined research question. The medical librarian demonstrates how to incorporate Medical Subject Headings (MeSH) terms into the search strategy to improve search precision.

Although students demonstrated mastery of search strategies as evidenced through a post-session activity, these sessions did not provide the opportunity to appraise the evidence that students found while searching PubMed. Based on anecdotal evidence from student research consultations and a conversation with a faculty member, it became clear that students needed additional learning opportunities to practice critical appraisal. The medical librarian determined that a one-hour workshop on critical appraisal would help students gain the practice that they need to successfully complete their research projects.

The opportunity to participate in the 2023-24 (June to June) cohort of the Research Training Institute (RTI) led the medical librarian to expand the idea of a critical appraisal workshop into a research project. Originally planned as a mixed-methods study involving a survey and follow-up interviews, project deadlines and lack of research expertise led to the modification of an existing survey for data collection and a quantitative study design.

The medical librarian aimed to present an evidence-based research session to 130 first-year medical students in January 2024 during an in-person class and provide time to complete the survey before and after instruction. The medical librarian discussed the research study with two faculty members prior to completing the institutional review board (IRB) application. During the meeting, the medical librarian explained that the workshop was part of a research project for RTI and that the goal was to measure the effect of instruction on critical appraisal skills through completion of a survey. However, the faculty members expressed concern about how the workshop would fit into the established curriculum and steered the research toward an optional evidence-based workshop for second-year medical students to occur at the end of the Fall 2023 term.

In response, the medical librarian developed an evidence-based research workshop for second-year medical students and received approval from the California University of Science and Medicine IRB to conduct the study using a cross-sectional design due to the limited time available to gather pre and post-test data.

To determine the effectiveness of the survey instrument, the medical librarian distributed the study to a group of 11 second-year medical students who had previously attended an extracurricular research workshop in September 2023. Six of the 11 students participated in the preliminary phase in November 2023. Based on preliminary results, the medical librarian revised the wording of one survey question but made no other changes to the research project. The question that was reworded asked students to explain where they would search for medical research. One student responded that they do not use Medline. As a result of this response, the medical librarian added the word Medline to ask, “If you were to search Medline for original research on one of these questions, describe what your search strategy would be?”

Experience

The medical librarian searched the literature for similar studies to determine if the study methodology was replicated at other institutions. However, the studies found were distinct from this study in the length of instruction, format, and study population. Some of these studies utilized the Berlin Questionnaire, developed by Fritsche et al.¹, which was adapted from the earlier Fresno Test. At one institution, Alahdab et al.² performed a study in which 50 Syrian undergraduate medical students participated in a two-day EBM course and included pre/posttest assessment utilizing the Berlin Questionnaire. In the United Kingdom, Lee et al. conducted a two-day critical appraisal program comprised of mixed methods: lecture, workshop, viva-style examination, and a pre/post-test critical appraisal quiz based on the Berlin Questionnaire³. Fifty-nine students from 17 different medical schools participated in the study. Al Sharani et al⁴ led a study of Saudi fourth-year medical students through a required hybrid course taken after a third-year course in medical research and epidemiology. In this third study, students’ EBM skills were assessed with the Fresno Test.

As explained by Ramos et al.⁵, the Fresno Test is a validated instrument designed to evaluate evidence-based medicine instruction. Created in consultation with Fresno State University family practice residents and faculty, the questions went beyond multiple-choice and true/false statements to examine whether medical students can develop clinical questions from two different case scenarios presented and use those questions to critically appraise the literature in the form of short-answer responses. In addition, the original instrument was designed to assess graduate medical students’ ability to calculate statistical and clinical significance as part of the critical appraisal process. While the Berlin Questionnaire was designed to assess EBM skills in undergraduate students, it was validated for use after a 3-day course. Therefore, the medical librarian decided to modify the Fresno Test to match the shortened instructional format. The modified Fresno Test is shown in Image 1 and consists of two scenarios that form the basis for the six open-response questions of the assessment. The last two optional questions asked participants to identify their gender and ethnicity for statistical purposes.

Figure 1. Modified Fresno Test

In addition, the original Fresno Test included test items on measures of clinical significance such as number needed to treat, risk reduction ratio and specificity vs. sensitivity. These items were removed from the modified Fresno Test to align with the learning objectives of the workshop.

Originally, the medical librarian planned to conduct a pre and post-test study using the modified Fresno Test. However, due to logistical issues in recruiting participants and scheduling the evidence-based research workshop, the medical librarian opted for a cross-sectional study design. The workshop was initially scheduled for the week of final exams in December 2023, but only two students showed up. Therefore, the medical librarian recruited a student researcher to increase student participation and rescheduled the workshop for September 2024 with a different group of second-year medical students.

In designing the workshop, the medical librarian incorporated Kirkpatrick’s Four Levels of Evaluation which include: reaction, learning, behavior, and results. This framework was developed for adult learning programs in the corporate sector. However, aspects of the framework apply to higher education, particularly level 2: Learning which emphasizes active learning through knowledge tests and checklists⁶. Due to time constraints, the medical librarian decided to seek out existing instructional materials instead of creating them from scratch. From previous teaching experience, the medical librarian was familiar with educational online repositories and found materials on CORA. The workshop materials were adapted, with permission, from Keba Knecht⁷. The original two-hour workshop was condensed into a 60-minute session, and the pre-reading requirement was eliminated. As part of the broader presentation on literature search and measures of validity, the medical librarian utilized the CASP (Critical Appraisal Skills Programme) RCT (Randomised Controlled Trial) Checklist⁸ to guide students through critical appraisal of the following study: “The effect of vitamin D and magnesium supplementation on the mental health status of attention-deficit hyperactive children: a randomized controlled trial”⁹.

The workshop began with an overview of EBM, including the importance of critical appraisal. The medical librarian also demonstrated PubMed search strategies using MeSH terms and medical librarian discussed different types of study design. The workshop also included a broad overview of statistical significance with a discussion on p-values and confidence intervals but excluded in-depth discussion of effect size measures such as power calculation, number needed to treat, and application to patient care due to coverage of these topics in the curriculum and lack of librarian expertise. The workshop concluded with the critical appraisal activity using the CASP RCT Checklist to appraise the article referenced in the presentation. It was intended to be a small group activity, but the medical librarian facilitated the activity as a whole group to allow time for completion of the modified Fresno Test at the end.

Discussion

Eighteen students attended the September 2024 workshop. Thirty-three students had originally registered for the 2024 workshop, but a last-minute schedule change led to a drop in student attendance. As a result, only 13 students participated in the study, including six students who tested the survey during the preliminary phase in Fall 2023.

Overall, students demonstrated proficiency in literature searching and critical appraisal skills in their responses to the case-based survey questions. In question one, students constructed appropriate clinical questions based on each scenario. Student responses were scored according to the criteria in the Fresno Test of Evidence Based Medicine Grading Rubric Form A\textsuperscript{5}. Student responses are shown in Figure 2.

Figure 2. Writing a Clinically Focused Question

Similarly, student responses to questions 2-5 demonstrated their ability to identify study methodology relevant to a research question, select appropriate search terms, and effectively search the medical literature using MeSH and other advanced search options in PubMed. However, student responses illustrated a particularly strong understanding of the critical appraisal process in question six: What characteristics of the study will you consider determining if its findings are valid? as shown in Figure 3¹⁰.

Figure 3. Voyant Tools Word Cloud

The word cloud was chosen to visualize the patterns of critical thinking among students while appraising the research study.

Takeaways

The study was conducted over two academic years with two different groups of students. There were significant time constraints in conducting the study because California University of Science and Medicine offers an 18-month curriculum. As a result, second-year medical students are only on campus full time during the first semester of their second year, with limited time for extracurricular programs. The study did not assess students’ critical appraisal skills prior to the workshop, so there is no way to determine if their level of knowledge changed. If repeated in the future, the workshop will be presented during a mandatory class session with the medical librarian.

While the aims of the study were not met due to the limitations previously described, the medical librarian learned some valuable lessons. For one, the modified Fresno Test is an effective assessment tool for a one-hour evidence-based research workshop. Students successfully crafted appropriate research questions and identified key search terms and measures of research quality. To determine if students’ critical appraisal skill levels changed, the workshop should align with the curriculum and be developed in collaboration with a faculty member. A pre/posttest research design should also be used instead of a cross-sectional survey.

Acknowledgments

I would like to thank Dr. Jonathan Shaw, 2026 graduate of the California University of Science and Medicine, who took time out of his busy third-year medical school schedule to help with the workshop. I would also like to thank RTI mentors and peers for support with this research project.

References

1. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002;325(7376):1338–41. doi:10.1136/bmj.325.7376.1338
2. Alahdab F, Firwana B, Hasan R, et al. Undergraduate medical students’ perceptions, attitudes, and competencies in evidence-based medicine (EBM), and their understanding of EBM reality in Syria. BMC Res Notes. 2012;5(1):431. doi:10.1186/1756-0500-5-431
3. Lee AE, Ardissino M, Bednarczuk NF, Tennyson M, Khajuria A. Prospective assessment of a critical appraisal teaching programme on medical students' confidence and performance in appraising medical literature. J R Coll Physicians Edinb. 2020;50(1):60–66. doi:10.4997/jrcpe.2020.118
4. Al Shahrani AS. Development and evaluation of an evidence-based medicine module in the undergraduate medical curriculum. BMC Med Educ. 2020;20(1):256. doi:10.1186/s12909-020-02181-7
5. Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003;326(7384):319–21. doi:10.1136/bmj.326.7384.319
6. Kirkpatrick JD, Kirkpatrick, WK. Evaluating Level 2: Learning. In: Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick's Four Levels of Training Evaluation. Association for Talent Development; 2016:42-48
7. Keba Knecht M. Introduction to Critical Appraisal for Medical Students. Online Repository. 2023. Accessed November 20, 2023. https://projectcora.org/assignment/introduction-critical-appraisal-medical-students
8. Critical Appraisal Skills Programme. CASP Randomised Controlled Trial (RCT) Checklist. Accessed September 18, 2024. https://casp-uk.net/casp-tools-checklists/randomised-controlled-trial-rct-checklist/
9. Hemamy M, Pahlavani N, Amanollahi A, et al. The effect of vitamin D and magnesium supplementation on the mental health status of attention-deficit hyperactive children: a randomized controlled trial. BMC Pediatr. 2021;21(1):178. doi:10.1186/s12887-021-02631-1
10. Voyant Consortium. Word Cloud. Voyant-tools.org. Accessed August 8, 2025. https://voyant-tools.org/