Evaluating the Effectiveness of An Evidence-Based Research Study
A Pilot Study
DOI:
https://doi.org/10.18060/29320Keywords:
critical appraisal, medical studentsAbstract
Background: Over two semesters, California University of Science and Medicine (CUSM) medical students complete a course called Academic Research Study in which they collaborate in groups to design and conduct research studies. However, instructional sessions for this course are only scheduled during the first half of the course. The medical librarian presented a 1-hour overview of literature searching during one of these sessions, which is insufficient time to explore critical appraisal. Anecdotal evidence from subsequent research consultations indicated that students need more time to practice critical appraisal.
Experience: The medical librarian conducted a literature review to identify effective instructional strategies and assessment tools. The medical librarian designed a workshop for medical students, using materials adapted (with instructor permission from the Community of Online Research Assignments (CORA)). The Fresno Test was also modified to align with learning outcomes. Students were given 15 minutes after the workshop to complete the 6-question case-scenario based survey which focused on developing an appropriate research question, identifying appropriate study types and sources of information, as well as criteria for determining research quality.
Discussion: The study took place over two years, with two different cohorts of second-year medical students. Only 13 students participated in the research study between the preliminary and study phases. The small study size and cross-sectional research design makes it impossible to generalize findings. However, all study participants were successful in completing the survey and accurately identified the research question, key search terms and measures of research quality.
Takeaways: California University of Science and Medicine utilizes an 18-month curriculum for pre-clinical training. As a result, second-year students are on campus until the end of their second fall term and have limited time for extracurricular programs. The medical librarian designed a workshop to address the need for additional critical appraisal practice. The plan was to assess students’ knowledge before the workshop, but time limitations made this difficult. Furthermore, students were not assessed before the workshop, so there is no proof that their skills changed, yet the modified Fresno Test proved to be an effective assessment tool for a short instructional session on critical appraisal.
References
1. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002;325(7376):1338–41. doi:10.1136/bmj.325.7376.1338
2. Alahdab F, Firwana B, Hasan R, et al. Undergraduate medical students’ perceptions, attitudes, and competencies in evidence-based medicine (EBM), and their understanding of EBM reality in Syria. BMC Res Notes. 2012;5(1):431. doi:10.1186/1756-0500-5-431
3. Lee AE, Ardissino M, Bednarczuk NF, Tennyson M, Khajuria A. Prospective assessment of a critical appraisal teaching programme on medical students' confidence and performance in appraising medical literature. The Journal of the Royal College of Physicians of Edinburgh. 2020;50(1):60–66. doi:10.4997/JRCPE.2020.118
4. Al Shahrani AS. Development and evaluation of an evidence-based medicine module in the undergraduate medical curriculum. BMC Med Educ. 2020;20(1):256. doi:10.1186/s12909-020-02181-7
5. Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003;326(7384):319–21. doi:10.1136/bmj.326.7384.319
6. Kirkpatrick JD, Kirkpatrick, WK. Evaluating Level 2: Learning. In: Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick's Four Levels of Training Evaluation. Association for Talent Development; 2016:42-48
7. Keba Knecht M. Introduction to Critical Appraisal for Medical Students. Online Repository. 2023. Accessed November 20, 2023. https://projectcora.org/assignment/introduction-critical-appraisal-medical-students
8. Critical Appraisal Skills Programme. CASP Randomised Controlled Trial (RCT) Checklist. Accessed September 18, 2024. https://casp-uk.net/casp-tools-checklists/randomised-controlled-trial-rct-checklist/
9. Hemamy M, Pahlavani N, Amanollahi A, et al. The effect of vitamin D and magnesium supplementation on the mental health status of attention-deficit hyperactive children: a randomized controlled trial. BMC Pediatr. Apr 17 2021;21(1):178. doi:10.1186/s12887-021-02631-1
10. Voyant Consortium. Word Cloud. Voyant-tools.org. Accessed August 8, 2025. https://voyant-tools.org/
Published
How to Cite
License
Copyright (c) 2026 Deborah Farber
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All works in Hypothesis are licensed under a Creative Commons Attribution 4.0 International license. Authors own copyright of their articles appearing in Hypothesis. Readers may copy articles without permission of the copyright owner(s), as long as the author(s) and the Medical Library Association are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. For any other use of articles, please contact the copyright owner(s).
