Longitudinal Student Research Competency: Comparing Online and Traditional Face-to-Face Learning Platforms

This exploratory research compares longitudinal research self-efficacy and retention between a completely asynchronous Master of Social Work (MSW) online cohort and its traditional face-to-face counterpart. This study used a non-equivalent comparison groups design with two groups: online instruction only (n=16) and traditional face-to-face instruction (n=32), with pretest (Time 1), posttest (Time 2) and follow-up (Time 3) standardized measures of practice evaluation knowledge (PEKS) and research selfefficacy (RSES) in a beginning research methods course. Results indicate that students’ knowledge and research self-efficacy improved between pretest and posttest and remained significantly improved at follow-up one year later, with no significant difference between online learners and traditional face-to-face students. Students gain and maintain confidence in research methods and evaluation regardless of the learning platform utilized.

Studies comparing online, face-to-face, and hybrid (a mix of online and face-to-face meetings) program models continue to grow, but no studies to date explore differences in the retention of knowledge over time by learning platform.The current research study addresses this gap by examining the practice evaluation knowledge and research selfefficacy of online and face-to-face MSW students over an 18-month period.

Online vs. Traditional Classroom Platforms
Previous social work literature specifically comparing online vs. traditional classroom experiences in terms of research methods learning has examined differences in student satisfaction (Faul, Frey, & Barber, 2004;Ligon, Markward, & Yegidis, 1999;Westhuis et al., 2006;York, 2008) showing mixed results ranging from higher overall satisfaction with web-assisted courses, to no difference in satisfaction levels by learning platforms, to higher satisfaction in the face-to-face classroom.While student satisfaction is considered an important part of successful online education (Siebert, Siebert, & Spaulding-Givens, 2006;Stocks & Freddolino, 2000), satisfaction does not necessarily equal effectiveness.Students might be satisfied with their program, but have they learned?
Previous studies operationalize learning outcomes using course grades (Harrington, 1999;Hisle-Gorman & Zuravin, 2006;Kleinpeter & Potts, 2003;O'Neill & Jensen, 2014;York, 2008), exam scores (Cummings et al., 2015;Westhuis et al., 2006), and overall grade point average (GPA; Cummings et al., 2015).O'Neill and Jensen (2014) compared final course grades and self-reported GPA for forty-four MSW students enrolled in either a faceto-face (23 students) or an online (21 students) section of the same research course and found no significant differences between the two groups of students at the end of the course.Cummings et al. (2015) found mixed results, with advanced-standing face-to-face students having statistically significant higher GPA scores than their online counterparts, but found no significant difference in GPA between non-advanced-standing face-to-face and online students.
Few studies comparing online with face-to-face learning practices have used standardized measures with demonstrated reliability and validity.Stocks and Freddolino (2000) examined comfort with technology and classroom environment in a sample of 60 MSW students.Using standardized measures of attitudes toward computers and technology use at the beginning and end of a research methods class with two sections (one online, the other face-to-face), they found no significant difference in the computer attitude scale between pretest and posttest and marginal differences in technology use between the two groups, with online students reporting greater comfort using technology than their face-to-face counterparts.Buchanan and Mathews (2013) used the Kirk-Rosenblatt Research Inventory (1981) to assess MSW social work students' beliefs, knowledge, and opinions about research, finding no statistically significant difference between main campus and satellite MSW students' knowledge.Cummings et al. (2015) used a standardized measure of self-efficacy to explore differences between online and face-to-face students from the perspective of an overall program, finding no significant difference between online and face-to-face student outcomes (Cummings et al., 2015).These results provide additional support of no significant difference between online and face-to-face learning platforms (WCET, 2010), but do not explore the retention of knowledge over time.

Knowledge Retention
One challenge educators face is choosing a learning strategy that will result in longterm retention of knowledge (Beers & Bowden, 2005).Various theories posit strategies to improve knowledge and memory including, but not limited to, problem-based learning (Beers & Bowden, 2005;Schmidt, 1993), integrating the arts in education (Hardiman, Rinne, & Yarmolinskaya, 2014), team-based learning (Macke & Tapp, 2012), and diffuse learning (Raman et al., 2010).Outlining the myriad education mechanisms suggested to increase knowledge retention is beyond the scope of this article.The importance of examining knowledge retention, however, should not be overlooked (Raman et al., 2010;Wayne et al., 2006).If students do not retain knowledge, they may be less likely to successfully use their education and skills in the field, which is particularly important in helping professions such as social work.
MSW students are required to successfully complete at least two research courses to earn their degree, and many schools require additional research methods coursework.However, few studies have explored the degree to which students maintain their research methods knowledge over time.Using a small sample (n=25) of undergraduate social work students, Baker, Pollio, and Hudson (2011) found evidence that students maintain educational gains one year after a research methods class, but their study tested knowledge perception as opposed to knowledge gained and did not compare differences by learning platform.

The Current Study
This study builds on previous research by replicating the Baker et al. (2011), pre/post/post study of BSW students using the same measure, the Practice Evaluation Knowledge Scale (PEKS, Baker & Ritchey, 2009), with a sample of MSW students.Further, we included a measure of Research Self-Efficacy (RSES, Holden, Barker, Meenaghan, & Rosenberg, 1999) and a comparison group of online distance learning students.This study compares practice evaluation knowledge and research self-efficacy learning outcomes between a completely asynchronous online MSW research methods class and its face-to-face classroom counterpart using a quasi-experimental non-equivalent comparison groups design.Using standardized measures of students' perception and confidence, this project builds on previous social work literature and adds to the ongoing online versus face-to-face classroom debate.
Based on previous literature, we hypothesized that there would be no difference between the research knowledge and self-efficacy of online students and face-to-face classroom students a) after completing a beginning research class, and b) one year after the class.

Method Program/Class Description
In fall 2012, a large public university on the west coast introduced a two-year, degreegranting, fully asynchronous online Master in Social Work (MSW) program.Students in both the online program and face-to-face program follow a cohort model, meaning that students enter and exit the program together and take classes in a prescribed order.Online students complete the program in two years.Traditional students have the option of completing the program in either two or three years.All students in the current study were part of a two-year cohort.Online students and face-to-face students differ in that online students follow a quarter system, taking two eight-week classes per quarter totaling four classes per semester.Traditional face-to-face students take four classes over 16 weeks each semester.
In order to graduate, all students must take and successfully complete three semesters of research methods: beginning research methods, advanced research methods, and a Capstone project.The beginning research methods class focuses on problem formulation, operationalization, conceptualization, design, and measurement concepts, and students complete a single subject design over the course of the semester.The advanced research methods course focuses on sampling, ethics, program evaluation, qualitative research methods, and survey research and touches on statistical analyses.The Capstone project gives students the opportunity to design and conduct their own small research project including data analysis and presenting results.Students may choose to collect their own data, use agency secondary data previously collected for non-research purposes, use secondary data from publicly available sources (e.g., the General Social Survey or the National Health and Nutrition Examination Survey), or conduct a program evaluation.Students in all cohorts receive the same content, and classes happen in the same semester for students in a two-year cohort (i.e., everyone in the cohort has beginning research methods their second semester, advanced research methods their third semester and Capstone the semester before graduation.)

Sample
The study population included MSW students enrolled in one of three master's level beginning research methods sections.Instructor B taught one section online (n=21).Instructor B and Instructor P each taught one traditional face-to-face section (n=13 for Instructor B; n=23 for Instructor P) for a total of 57 participants.Five students (3 face-toface; 2 online class) did not complete the pretest, and three different students (1 online class; 2 face-to-face) did not complete the posttest, resulting in a valid n=49 (n=18 online; n=31 face-to-face) between Time 1 and Time 2.
Between posttest (Time 2) and one-year follow-up (Time 3), two students left the online cohort for personal reasons; the remainder of the cohort (n=16) completed the Time 3 measure.One face-to-face student who did not complete the Time 2 measure did complete the Time 3 measure resulting in a valid n=48 (n=16 online; n=32 face-to-face) between Time 1 and Time 3.

Design
This study used a non-equivalent comparison groups design with two groups: one consisting of students who received online instruction only, and one group consisting of students who received only face-to-face instruction, with pretest and posttest measures of student competency for both groups.Pretest competency scores were compared for all three sections.No significant differences in demographic variables or research competency scores were found between Instructor B's face-to-face students and Instructor P's face-toface students, so those face-to-face sections were combined and compared to the online student competencies.Pretest measures were taken at Time 1 in January 2013 prior to the first research methods class, Time 2 in May 2013 after the first research methods class, and Time 3 in May 2014 after two subsequent research courses (advanced research and Capstone).

Measurement
Two standardized measures were used to assess student achievement of research competency: the Practice Evaluation Knowledge Scale (PEKS) and the Research Self-Efficacy Scale (RSES).The PEKS was developed to measure social work practitioners' beliefs about their knowledge of practice evaluation competencies and has demonstrated internal consistency and validity (Baker et al., 2011, p. 558).Example items include "I have been adequately trained to conduct practice evaluation" and "I am familiar with issues of reliability and validity."The 8-item PEKS (α=.88) is measured on a scale from 1-5 where 1=strongly disagree and 5=strongly agree.
The RSES developed by Holden, Barker, Meenaghan, and Rosenberg (1999) has demonstrated internal consistency reliability, evidence of construct validity, and sufficient sensitivity "to detect change in students' research self-efficacy from the beginning to the end of their participation in a single-semester research course" (p.472).The 9-item RSES (α=.95) is measured on a scale from 0-10 where 0=cannot do at all, 5=moderately certain can do, and 10=certain can do.Items begin with the statement "How confident are you that you can…" and include "Do effective electronic searching of the scholarly literature"?and "Design and implement the best sampling strategy possible for your study of some aspect of practice?"

Data Collection
Together the PEKS and RSES total 17 questions.For the purpose of this study, each measure was included in an easily readable online chart where respondents were asked to click the button next to their response for each question.
After receiving approval from the University Institutional Review Board, pretest data were collected online via the class web pages.Students were directed to a link to the survey prior to the first class session via an email message from the other instructor.Students were assured that their instructor would not see their survey results until after the class ended, and then only in aggregate.Each student has a unique login, so matching pretest with posttest data occurred seamlessly.There were no duplicate entries, meaning it was unlikely that students logged in under another students' ID to complete either the pretest (Time 1) or posttest (Time 2).
Follow-up (Time 3) data collection began at the end of the third research (Capstone) class, a full 18 months after the pretest, and continued for approximately one month.Students were contacted via email, reminded about the study, and asked to complete the posttest using a class web page that was set up to collect their data with their unique login, thus allowing for pretest, posttest, and follow-up data to be matched easily.If students had difficulty navigating the webpage they were invited to return their responses via email, fax, or in-person.These responses were then entered into the database by hand by Instructor B and checked for accuracy by Instructor P. Approximately one-third of the responses were recorded in this manner.

Data Analysis
Descriptive statistics were used to summarize demographics.Paired samples t-tests were used to determine differences between pretest and posttest knowledge and selfefficacy scores.One-way between groups analysis of covariance (ANCOVA) was conducted to compare differences in learning platforms for research methods instruction for MSW students.ANCOVA tests the significance of group differences between two or more groups while controlling for one or more covariates (e.g., pretest scores) that may influence the dependent variable (Tabachnick & Fidell, 2007;Wright, 2006).For the current study, the independent variable was the type of learning platform (online vs. faceto-face classroom), and the dependent variable consisted of scores on the PEKS and RSES surveys administered at the end of the first research class (Time 2) and again at the end of the third research class (Time 3).Participants' scores on the PEKS and RSES pretest surveys (Time 1) were used as the covariate in the analysis.

Sample Demographics
Students ranged in age from 22-44 years (M=28.55;SD=5.79) and were mostly women (83%).The majority of students identified as Hispanic (n=18), followed by Caucasian (n=14).No significant differences in age, race, gender, or previous research experience were found between the online and face-to-face students (Table 1).Face-to-face students (M=2.36,SD=.96) scored significantly higher than online students (M=1.89,SD=.56) on item 1 of the PEKS (I have been adequately trained to conduct practice evaluation), t(50)=2.21,p=.03, two-tailed, d=0.59.The magnitude of the difference in the means (mean difference=.47,95% CI: .04-.89) was moderate (Cohen's d=0.59).There were no statistically significant differences in any of the remaining PEKS or RSES items for faceto-face or online students.2 and 3 for paired samples t-test results by individual items on the PEKS and RSES.There was a substantial difference in program evaluation knowledge (as measured by the PEKS) and research self-efficacy (as measured by the RSES) for all students (online and traditional face-to-face) after taking the foundation research methods course.We explored differences between Time 2 and Time 3 and found that only questions 1 (effective electronic searching of the scholarly literature) and 3 (review a particular area of social science theory and research, and write a balanced and comprehensive literature review) on the RSES were significantly different between posttest (Time 2) (Q1 M=87.60,SD=13.63;Q3 M=78.60,SD=15.52) and follow-up (Time 3) (Q1 M=92.60,SD=13.82),t (49)=2.29,p=.03 (two-tailed), d=0.32; (Q3 M=87.80,SD=14.88),t (49)=3.77,p < .001(two-tailed), d=0.53.None of the items on the PEKS, and no other items on the RSES were significantly different between Time 2 and Time 3.

Online vs. Traditional Face-to-Face Platform
Our hypothesis stated that there would be no difference between the self-reported knowledge and self-efficacy of online and face-to-face students after the beginning research methods class and one year after the beginning research methods class.Preliminary checks were conducted to ensure that there was no violation of the assumptions of normality, linearity, homogeneity of variances, homogeneity of regression slopes, and reliable measurement of the covariate.After controlling for pre-test scores, there was no significant difference between online and traditional face-to-face classroom students on posttest (Time 2) PEKS scores F(1, 46)=0.01,p=.91, and posttest (Time 2) RSES scores, F(1, 46)=0.36,p=.55 after the first research class.There was no significant difference between online and traditional face-to-face classroom students on follow-up (Time 3) PEKS scores F(1, 45)=1.51,p=.22 or follow-up (Time 3) RSES scores, F(1, 45)=0.09,p=.76 one year after the beginning research class.

Discussion
The online students in this program completed a two-year, completely asynchronous, online MSW program with the same 16 hours per week in a local field placement as students in the face-to-face cohort.The standards for the online program are the same as those for the traditional classroom, and the same faculty members teach in both programs.As such, and based on previous literature, the investigators expected to find no difference in learning outcomes between the online and traditional face-to-face classroom students.
Although not social work specific, meta-analyses comparing distance education and classroom instruction reveal somewhat mixed results with support leaning toward distance education being similar to traditional classroom instruction.Allen et al. (2004) andSitzman, Kraiger, Stewart andWisher (2006) found no differences in educational effectiveness for distance learners, with effectiveness defined as assessment of student performance (e.g., grades) or acquisition of declarative knowledge.However, Bernard et al. (2004) found wide variability and low effect sizes on various outcomes including student achievement, attitude, and retention.Note that Bernard et al. (2004) define retention as "the opposite of dropout" (p.388) as opposed to the maintenance of knowledge.Creating subsets of synchronous and asynchronous applications resulted in effect sizes for asynchronous applications favoring distance education (Bernard et al. 2004).Sitzman et al. (2006) found web-based instruction 6% more effective than classroom instruction for teaching declarative knowledge.The current study examined pre and posttest practice evaluation and research self-efficacy scores of students in an asynchronous, web-based, distance-learning classroom compared to a traditional face-toface classroom for research methods (declarative knowledge).Meaningful gains between pretest, posttest, and follow-up support the effectiveness of online and face-to-face education as seen in significant gains between the three time periods for the two groups.Further, finding no differences in the learning outcomes between the two learning platforms provides additional support that online and face-to-face learning modalities are equally effective.Finding differences in learning outcomes would have resulted in adjustments being made to either course, depending on the nature and direction of those differences.
Despite a growing body of evidence that online learning or distance education is as effective as traditional face-to-face classroom instruction, the various types of online education make comparisons difficult.Online learning or distance education ranges from in-service training on-demand via television and satellite (Williams, Nicholas, & Gunter, 2005) to asynchronous electronic software content (Harrington, 1999) to hybrid models combining face-to-face instruction with distance learning applications (Ayala, 2009;Osguthorpe & Graham, 2003;York, 2008).
The current study adds to the developing body of literature by using standardized measures of learning outcomes, a pre/post/post quasi-experimental longitudinal design, and controlling for instructor and content differences in that the same instructor taught both the asynchronous online and face-to-face beginning research classes.Knowledge retention over time is one indicator of teaching effectiveness, and our results suggest that online learning platforms are at least as effective as traditional face-to-face classroom strategies in students maintaining their practice evaluation knowledge and research self-efficacy one year after taking a beginning research methods class.

Limitations
While we were able to control for instructor and content differences between Time 1 and Time 2, we were not as fortunate between Time 2 and Time 3. Online students maintained the same professor for all three research courses (beginning, advanced, and capstone), whereas the face-to-face students had the ability to choose their instructor for the advanced and capstone research classes.While faculty work together to ensure students are receiving standard content, there were likely differences in how that content was delivered.Since there were four potential instructors for the advanced research class that occurred fall 2013 (Instructor P and three additional faculty), and six potential capstone instructors during spring 2014 (Instructors B and P and four additional faculty), there was too much variability to statistically control for potential instructor differences.
The sample size of this study was small.Although the use of small samples is common in social psychology (Johnson & Bachan, 2013) and education (Cook & Hatala, 2015), this limits the statistical power.A finding of no statistically significant difference between online and face-to-face students in this study may not indicate that there is no true difference but may also be a result of inadequate power.However, considering this study employs a theory-based prediction, strong design with longitudinal data, and standardized measures, the findings of this study may have meaningful educational implications.
Findings indicate increased student self-efficacy in research methods, which may translate into greater comfort recognizing and employing evidence-based practices in the field.However, results should be interpreted with caution considering the current study used self-reports of a non-random, convenience sample of graduate social work students from a single university.Although there were no statistically significant differences between the face-to-face and online students at pretest, participants were not randomized into experimental and control groups, and it is possible that there are inherent differences between the two groups that account for the knowledge and self-efficacy gained.Although randomization may not be realistic in this type of education research, selection bias remains a threat to internal validity with this non-randomized design.With the exception of the first item on the PEKS, there were no significant differences between the face-to-face and online students.

Conclusion
As online learning and distance education continues to develop, MSW programs have an opportunity to be on the cutting edge of this growth in the social work field.However, remaining on the cutting edge involves conducting research that goes beyond student satisfaction or course evaluations.Our results indicate that students gain and maintain confidence in research methods and evaluation regardless of the learning platform utilized.Using this model, we encourage future researchers to explore competencies and areas of social work practice beyond research methods.
Despite the limitations of the current study, our results add to the growing body of literature showing that successful student research learning and knowledge retention may occur equally well through online and traditional face-to-face learning platforms.Future studies should continue exploring longitudinal research knowledge retention since the timing of content delivery in this sample differed by eight weeks (face-to-face students had a 16-week semester; online students had an eight-week course).It is possible that the shorter learning time could negatively affect longer-term retention of knowledge.Furthermore, research about the quality of programs from the perspective of learning outcomes triangulated with faculty-measured student competency could provide useful knowledge for informed practice and policy.
A major strength of this study is the use of standardized learning outcome measures taken at three different time points over 18 months, with findings suggesting that the modality of content delivery is less important than the content itself.Implications for social work education include effectively utilizing a broad range of information and communication technologies and increasing accessibility to social work students in traditionally underserved areas.

Table 1 .
Student Demographics by Instructional Method

Table 2 .
Paired Samples T-Test Results for Online Student Responses by Item, Pretest (Time 1) and Posttest (Times 2 & 3)

Table 3 .
Paired Samples T-Test Results for Face-to-Face Student Responses by Item, Pretest (Time 1) and Posttest (Times 2 & 3)