back

Learning Effectiveness and Students’ Perceptions in a Flexible Learning Course

Claude Müller [muew@zhaw.ch], Michael Stahl [michaelstahltg@bluewin.ch], Mark Alder [alma@zhaw.ch], Maximilian Müller [mlxi@zhaw.ch], Zurich University of Applied Sciences [www.zhaw.ch], Winterthur, Switzerland

Abstract

With flexible learning, students gain access and flexibility with regard to at least one of the following dimensions: time, place, pace, learning style, content, assessment or learning path. Zurich University of Applied Sciences (ZHAW) has launched a new flexible learning study format called FLEX, a blended learning design allowing students to be more flexible as to when and where they study. It reduces classroom learning time, replacing some of it with an e‑learning environment for self-study that includes instructional videos. In a pilot phase, we conducted a semi-experimental study on the learning effectiveness of FLEX. Students’ perceptions of the new study format FLEX were found to be positive. In addition, the final test results of students in the FLEX programme were similar to those of other students, despite classroom learning time was reduced by about half.

Abstract in German

Mit flexiblem Lernen wird den Lernenden Flexibilität in Bezug auf mindestens eine der folgenden Dimensionen ermöglicht: Zeit, Ort, Tempo, Lernstil, Lerninhalt, Bewertung oder Lernpfad. Die Zürcher Hochschule für Angewandte Wissenschaften hat ein neues flexibles Studienformat FLEX eingeführt. Das Blended Learning Design FLEX ermöglicht Studierenden, zeitlich und örtlich flexibel zu studieren, da die physische Präsenzzeit an der Hochschule reduziert und durch eine E-Learning-Umgebung mit Lernvideos ersetzt wird. In einer Pilotphase des FLEX-Programmes konnte eine semi-experimentelle Studie zur Lerneffektivität von FLEX durchführt werden. Die Befunde zeigen, dass die flexible Lernumgebung FLEX von den Studierenden positiv beurteilt wurde. Zudem erzielten die Studierenden äquivalente Klausurergebnisse, obwohl die Präsenzzeit in FLEX im Vergleich mit dem konventionellen Studienformat um die Hälfte reduziert wurde.

Keywords: flexible learning, learning effectiveness, blended learning, post-only design

Introduction

Many universities are looking for a coherent response to the dynamics of higher education, such as the need for technological innovations (MOOCs), increasing competition and a highly mobile international student body. One way of responding to this is the introduction of flexible learning, which allows students to learn at a time and place that suits them and to decide their own learning path.

There is no universally accepted definition of the term flexible learning (Li & Wong, 2018). According to the often quoted definition of Chen (2003), flexibility must be present in at least one of the following learning dimensions: time, place, pace, learning style, content, assessment, or learning path. From an institutional point of view, this also implies a change in the organisation of teaching and learning. For example, the content must be made available in such a way that students can access it anytime and anywhere. This design is the most basic form of flexible learning. In this sense, flexible learning is often used synonymously with terms such as e-learning, open learning, distance learning, or blended learning (Tucker & Morris, 2012). Blended learning is commonly understood as a combination of face-to-face instruction and computer-mediated learning (Graham, 2006). Brown (2016) pointed out that although an increasing number of online tools are being used to enrich face-to-face learning, for a real blended learning setting the online elements and face-to-face elements must be combined with each other in a purposeful manner; it is not enough to simply upload documents to a learning management system (LMS). For such a blended learning setting to become a flexible learning design, not only does classroom teaching have to be enriched. The course must be restructured fundamentally to give learners higher degrees of freedom. Thus, students should be able to study more independently of time and place and/or be able to individually determine content, assessment, learning path or the pace and style of how they learn.

Recent meta-analysis on blended learning (Bernard, Borokhovski, Schmid, Tamim, & Abrami, 2014; Means, Toyama, Murphy, & Baki, 2013; Vo, Zhu, & Diep, 2017) found a moderate but significant positive effect of blended learning compared with face-to-face instruction. The problem is that these studies usually do not indicate whether conventional teaching is supplemented by e-learning or replaced altogether. Confounding factors such as additional learning resources, additional learning time, or other interactions with the instructor could thus contribute to the positive outcomes for blended learning. Furthermore, a major problem to avoid in empirical studies is the selection bias of students in the experimental (blended-learning) and the control (face-to-face learning) groups. Some studies compare learning outcomes of students of a specific educational design (e.g., blended-learning) with those of a control group, (e.g., face-to-face learning) of the same programme (eg. Oftedal, Urstad, Hvidsten, & Foss, 2015). Students who enrol in a blended learning programme are likely to be different from students in a traditional cohort; there might be differences in time constraints because of job or family obligations, self-regulated learning skills, cognitive abilities or motivation. Although studies try to include a statistical control for selection bias, it is hard to ensure that all relevant characteristics are controlled and the estimated effect of blended learning might not be biased by the selective group of blended learners (Deschacht & Goeman, 2015). The authors of the above-mentioned meta-analysis concluded that further controlled experimental research is needed to investigate the outcomes of blended learning (Bernard et al., 2014) and that tested design principles must be developed for blended learning (Means et al., 2013).

This paper analyses these issues from a learners’ perspective. The following research questions are addressed: What are the students’ perceptions of the blended learning format FLEX? Does a blended learning design with reduced face-to-face time by half influence the effectiveness of learning?

The paper is structured as follows: First, the flexible learning programme FLEX is used as an example to illustrate objectives and considerations when implementing flexible learning in a blended learning design. Then, the research design of the pilot FLEX course is introduced. Finally, the results are presented and discussed.

FLEX Study Format

The School of Management and Law (SML) at Zurich University of Applied Sciences has launched a new study format it calls FLEX as part of a comprehensive e-learning strategy for its Bachelor’s study programmes. The Bachelor’s degree programmes “Business Administration” (BA), “Business Law” (BL) and “Business Information Technology” (BIT) are already offered in a full-time and a part-time format. Accordingly, FLEX is the third study format. Full-time programmes normally take six semesters to complete, part-time and FLEX programmes two semesters more. For the part-time programmes, lessons are held on one weekday and a maximum of two evenings and/or the Saturday morning. Part-time and FLEX students are recommended to do no more than 25 hours (a 60% workload in Switzerland) of outside work (e.g., a part-time job). The first part of each study program consists of an assessment level worth 60 ECTS (European Credit Transfer System) credits. Students acquire a basic knowledge of different subjects such as business administration, economics, law, mathematics, and English (as a foreign language); many courses are identical for all study programs. In the main section of the program, students choose specializations (majors), for which they earn another 120 ECTS credits.

Before launching the FLEX format for all study programmes and their courses, the SML decided to run a pilot course to gain some experience and further insights about the FLEX blended learning design. The pilot FLEX course chosen for this purpose was “Introduction to Business Administration” (IBA). IBA is a first-semester course that is identical in every Bachelor’s study program (i.e., BA, BL and BIT). For analysis of the students’ perceptions with regard to the new format and the learning effectiveness of FLEX in a semi-experimental design, the pilot course was integrated into the school’s smallest study program, BIT (i.e., Business Information Technology). For a period of one semester, all BIT students were assigned to FLEX classes by the school’s administration, while the BA and BL students studied in the conventional study.

The main objective of the newly introduced FLEX format was to offer students the best possible conditions for combining work or private responsibilities with a flexible study programme. The key consideration in deciding on the number of face-to-face lessons and their distribution across the 14-week term of a semester was the format’s suitability for students living some distance from the university. In other words, a decision had to be made as to how many overnight stays away from home would be acceptable for students living farther away. Regular physical face-to-face meetings were nevertheless deemed necessary to foster reflection on and consolidation of course content covered during the online phases. The university has a highly qualified faculty that is effective in delivering expertise through face-to-face instruction and interacting with students in the classroom.

As a result of these deliberations, on-site classroom teaching for the FLEX format was reduced by about half compared to the part-time programme, and that half was replaced with online sessions. Accordingly, the FLEX students attended classes approximately every three weeks for two days, while the self-study phases in between allowed them to learn more flexibly. Following the dimensions of flexible learning proposed by Chen (2003), the FLEX format offered greater flexibility in terms of time, place, pace, learning style and learning path than the conventional study format, but not in terms of assessment and content, which were identical in both study formats.

After the time structure for the new study format had been determined for the whole program, the transition to the flexible learning format was carried out at course level. In so-called “scripting workshops,” the lecturers redesigned the courses according to a defined process, using a specially developed didactic visualization language that was developed by adapting other systems (e.g. Molina, Jurado, de la Cruz, Redondo, & Ortega, 2009). Visualization of the didactic design of the IBA course (see Figure 1) shows how the virtual self-study phases (yellow) are embedded in the respective phases of face-to-face classroom instruction (blue). Different shapes illustrate the various methods used for conveying information (rounded rectangle), for activation and interaction (rectangle) and for learning assessment (circle), allowing an overview of the didactic design to emerge.

Horton (2012) pointed out that cognitive activation and elaboration are crucial in online learning. Accordingly, care was taken to ensure that information on all areas of knowledge is not only delivered (by means of learning videos, learning texts, etc.), but also elaborated and reflected on by the students through exercises, case studies and other tasks.

Figure 1

Figure 1. Result of the scripting process for the pilot course, “Introduction to Business Administration”

In the online self-study phase, web-based technologies such as LMS Moodle and other tools are used, and the information is provided in digital form, especially using self-produced learning videos. Lecturers create their own videos for their courses because it is important that the content is presented at the appropriate student skill level enabling an exact fit of the self-study phase with the face-to-face phase. For this reason, a new recording studio was set up to produce high-quality educational videos. The lecturers receive didactical and technical support for the transition of their courses and the production of their learning videos by the Center for Innovative Teaching and Learning of the SML.

Research Design

The research questions were answered by analysing the pilot course “Introduction to Business Administration” (IBA). The most important aspect of this course is the absence of a self-selection bias because the students had been assigned to the FLEX class by the school’s administration. This ensured a semi-experimental setting under highly controlled conditions. The IBA course had 989 students: 140 students in the experimental FLEX group and 849 students in the control group (CONV). Students in the FLEX group had four lessons every three weeks while students in the CONV group had four lessons every week. Accordingly, the face-to-face time of the FLEX group was 57 percent less than that of the CONV group.

All students were university freshmen, and the student eligibility requirements, lecture content, exam questions and grading scale were identical for all students. Although they were all in the same IBA course, however, they belonged to different study programmes. The students in the FLEX group belonged to the study programme “Business Information Technology” (BIT, = 140, three classes), the students in the CONV group to the study programmes “Business Administration” (BA, = 587, ten classes) and “Business Law” (BL, = 262, five classes). All classes were taught by experienced lecturers, and the lecturers teaching the FLEX classes also taught at least one CONV class.

The research design was tightly controlled for a field study in an educational area, firstly, because the framework conditions are comparable due to the same learning objectives and identical assessment and, secondly, because the presence of a control group ensures a quasi-experimental design (see also Fraenkel, Wallen, & Hyun, 2015).

To analyse the effectiveness of learning, a post-test-only design was used in which the final exam scores of both groups of students were compared. To compare performance, comparative pedagogical studies usually rely on test methods checking for significant changes. The objective is to reject the H0 hypothesis (no differences between groups) and to confirm the H1 hypothesis (a difference between groups exists at a certain level of significance); in other words, the experimental group under investigation should perform significantly better than the control group, which is verified by means of a t-test or a variance analysis (ANOVA or ANCOVA). Also in this study, an independent-samples t-test was administered comparing the 2014 final exam results of the students studying under experimental flexible learning (FLEX; = 140) and conventional learning (CONV, = 849) conditions (see results in “Learning Effectiveness” section).

Some may question whether such a comparison makes sense in a study related to blended learning in which face-to-face teaching is not supplemented by e-learning in the sense of an enrichment strategy, but instead a substantial part of face-to-face teaching is replaced. In the research context studied for FLEX, for example, the primary objective is to offer students flexible learning conditions in terms of time and place, thereby improving the compatibility of studies with professional or private responsibilities. Improving students’ exam results was, explicitly, not the objective of this project, either. Rather, the intended outcome was for the two groups of students to achieve equivalent exam results despite a significant reduction in face-to-face instruction. If the p-value in a t-test or ANOVA is higher than the alpha level (e.g., p > .05), the H0 hypothesis (mean values are equal) cannot be rejected, but the H1 hypothesis (mean values are not equal) is not confirmed. Maintaining H0 does not mean that the averages are in fact the same (Schmidt & Hunter, 1997). There is no confirmation for the acceptance of H0; the test conducted might simply have too little statistical power to prove any difference. In certain contexts, however, statistical evidence of equivalence can be of great importance for decision-makers. In the case described here, proof that students in a blended learning course with reduced face-to-face classroom time achieve equivalent results may, for example, lead to the implementation of the new study format or the transition of further study programmes into a blended learning FLEX format. To provide this proof, a two-sample test for equivalence (see also Meyners, 2012; Wellek, 2010) was applied.

Students’ perceptions of the new learning design and their learning process were analysed through an online survey. At the end of term, the FLEX group completed a questionnaire consisting of seven items of different instruments: interest/enjoyment (Intrinsic Motivation Inventory, Ryan, 1982), structure (Stiller, Bachmaier, & Köster, 2013), coherence (SCEQ), usability (own item), guidance and motivation (Course Evaluation Questionnaire, Wilson, Lizzio, & Ramsden, 1997), learning outcome (HILVE, Rindermann & Amelang, 1994) and two open-ended questions.

Results

Students’ Perceptions of the FLEX Study Format

The online survey results (= 117) show positive perceptions of FLEX in all dimensions, with broad agreement (response categories 3–5) ranging from 73.0% to 84.6% (see Figure 2). The learning environment was felt to be well structured and coherent, and especially the usability was evaluated in a positive way. On the other hand, more than a quarter of the students perceived the learning environment as not motivating. This was also reflected in the answers to the open-ended questions.

Figure 2

Figure 2. Students’ perceptions of the FLEX learning design (n=117)

Table 1 gives an overview of the most frequent answers about the FLEX learning design, with positive and negative connotations in the open-ended questions. Some of the answers were not linked directly to the flexible learning design but to the learning design of the course in general; in fact, students in the control group often made similar comments, for example, regarding reference solutions, unclear assignments or reading assignments. Temporal and spatial flexibility were perceived as the biggest advantages of the new design. In addition, students found the instructional videos to be a valuable learning resource.

Table 1:  Students’ answers about the learning design in FLEX

Table 1

The negative answers given most often concerned “Problems engaging in self-regulated learning” (= 26), “Missing direct interaction with lecturers/students” (=2 0) and “Difficulty estimating the time needed for learning” (= 19). As Samarawickrema (2005) found, for some students the partial shift in responsibility for the learning process to the student leads to problems; they, he concluded, “seem to be extremely teacher reliant, a trait that is counter to flexible, off-campus learner requirements” (p.63). Planning, organising and reflecting on their individual learning process is a major challenge for students in a flexible study format. They need self-regulated learning skills in a blended-learning design such as FLEX. To address this issue, the SML introduced special courses, support and coaching to enhance students’ capability for self-regulated learning in the FLEX format. Additionally, students were provided with a task plan which can be used to guide them through each self-study phase, especially at the first-year level of the programme.

Overall, the students in the FLEX group varied in their opinion of the flexible study design. Some students would welcome even more FLEX classes: “It’s a pity we don’t have FLEX in more courses.” Others pointed out, however, that they had enrolled in a classroom course and were therefore expecting face-to-face classroom instruction: “I don’t want FLEX anymore; we’re not at a distance learning college. That’s the reason why we like to go to lectures.” These statements indicate that students’ needs and choices are highly diverse (Li, 2014), reflecting their life context and study skills, and that it may make sense to offer different study formats to cater for the different learning design preferences.

Learning Effectiveness

Table 2 is an overview of the final exam results of the different groups of the IBA course in 2012-2014. The t-test reveals no significant difference in the final exam results 2014 for the FLEX (= 35.9, SD = 8.32) and the CONV (= 37.20, SD = 8.72) groups, (987) = -1.56, = 0.10 (ns). Accordingly, the effect size (Cohen’s d = 0.15) is low. The additional equivalence test shows a value of the test statistics T (1.64) lower than the corresponding quantile of the noncentral Fisher distribution (3.83). Thus, the H0 hypothesis can be rejected, and the exam results for the FLEX and the CONV groups can be regarded as statistically equivalent.

Table 2:  Results of final exam for the course “Introduction to Business Administration”

Table 2

The results in Table 2 indicate a bias of the final exam results of the IBA course due to the study program. The group of BA students achieved the best exam results on average in all years, followed by the BIT students; the BL students had the lowest exam results on average. Considering this bias of the study programme group, it was necessary to assess whether there was a significant change in the deviation of the BIT exam results over the years. To compare the final exam results of the different years, the deviations between the BIT students’ exam results and the exam result means of each year of the BA students were calculated and z‑transformed.

The results of the t-test shown in Table 3 indicate no significant change in the deviation of the final exam results in 2014 (year with FLEX format) compared with 2012/13 (years with conventional format). The effect size (Cohen’s = 0.04) is very low. Again, with a t-test of 0.40 the equivalence test shows a lower value compared to the quantile of the non-central Fisher distribution (2.93). In other words, the values can be regarded as statistically equivalent.

Table 3:  T-test comparing BIT group in 2012/13 (conventional format) and in 2014 (FLEX format) on deviation

Table 3

Conclusion

In a pilot course (“Introduction to Business Administration”), the face-to-face instruction time for students assigned to an experimental blended learning study format (FLEX) had been reduced by half. Students’ perceptions of the FLEX learning design as reported in the survey were mainly positive. Moreover, students achieved equivalent final exam results compared with students enrolled in the control group, a conventional face-to-face format of the same pilot course. These results are especially remarkable considering that the students had not been allowed to choose whether or not to join the FLEX group. It makes this one of just a few studies with some level of randomization comparing the blended learning format with conventional face-to-face teaching given the same environment and circumstances. However, the study was conducted in one subject and for a limited period (14 weeks) only. Further research should be done over a longer period and including several subjects to arrive at generalizable conclusions about the learning effectiveness of the FLEX study format.

References

  1. Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A meta-analysis of blended learning and technology use in higher education: from the general to the applied. Journal of Computing in Higher Education, 26(1), 87-122. doi:10.1007/s12528-013-9077-3
  2. Brown, M. G. (2016). Blended instructional practice: A review of the empirical literature on instructors’ adoption and use of online tools in face-to-face teaching. The Internet and Higher Education, 31(Supplement C), 1-10. https://doi.org/10.1016/j.iheduc.2016.05.001
  3. Chen, D.-T. (2003). Uncovering the provisos behind flexible learning. Educational Technology & Society, 6(2), 25-30.
  4. Deschacht, N., & Goeman, K. (2015). The effect of blended learning on course persistence and performance of adult learners. Comput. Educ., 87(C), 83-89. doi:10.1016/j.compedu.2015.03.020
  5. Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2015). How to design and evaluate research in education (9th ed.). New York: McGraw-Hill.
  6. Graham, C. R. (2006). Blended learning systems: definition, current trends, and future directions. In C. J. Bonk & C. R. Graham (Eds.), The handbook of blended learning: global perspectives, local designs (pp. 3-21). San Francisco: Wiley & Sons.
  7. Horton, W. (2012). E-Learning by Design. San Francisco: Pfeiffer.
  8. Li, K. C. (2014). How flexible do students prefer their learning to be? Asian Association of Open Universities Journal, 9(1), 35-46.
  9. Li, K. C., & Wong, B. Y. Y. (2018). Revisiting the Definitions and Implementation of Flexible Learning. In K. C. Li, K. S. Yuen, & B. T. M. Wong (Eds.), Innovations in Open and Flexible Education (pp. 3-13). Singapore: Springer Singapore.
  10. Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The Effectiveness of Online and Blended Learning: A Meta-Analysis of the Empirical Literature. Teachers College Record, 115(3), 1-47. Retrieved from http://www.tcrecord.org/Content.asp?ContentId=16882
  11. Meyners, M. (2012). Equivalence tests – A review. Food Quality and Preference, 26(2), 231-245. https://doi.org/10.1016/j.foodqual.2012.05.003
  12. Molina, A. I., Jurado, F., de la Cruz, I., Redondo, M. Á., & Ortega, M. (2009). Tools to Support the Design, Execution and Visualization of Instructional Designs. Paper presented at the Cooperative Design, Visualization, and Engineering, Berlin, Heidelberg.
  13. Oftedal, B., Urstad, K., Hvidsten, V., & Foss, B. (2015). Blended VS On-Campus Learning: A study of Exam Results in the Bachelor Degree in Nursing. International Journal of Learning, Teaching and Educational Research, 11(3). Retrieved from https://www.ijlter.org/index.php/ijlter/article/view/319
  14. Rindermann, H., & Amelang, M. (1994). Entwicklung und Erprobung eines Fragebogens zur studentischen Veranstaltungsevaluation. Empirische Pädagogik, 8(2), 131-151.
  15. Ryan, R. M. (1982). Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43(3), 450.
  16. Samarawickrema, R. G. (2005). Determinants of student readiness for flexible learning: Some preliminary findings. Distance Education, 26(1), 49-66.
  17. Schmidt, F. L., & Hunter, J. E. (1997). Eight Common but False Objections to Discontinuation of Significance Testing in the Analysis of Research Data. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 37-64). Mahwah, N.J.: Lawrence Erlbaum Associates Publishers.
  18. Stiller, K., Bachmaier, R., & Köster, A. (2013). NiceDesign4KMU. Online-Weiterbildung „Mediengestaltung”. Evaluationsbericht.
  19. Tucker, R., & Morris, G. (2012). By Design: Negotiating Flexible Learning in the Built Environment Discipline. Research in Learning Technology, 20(1), n1. doi:10.3402/rlt.v20i0.14404
  20. Vo, H. M., Zhu, C., & Diep, N. A. (2017). The effect of blended learning on student performance at course-level in higher education: A meta-analysis. Studies in Educational Evaluation, 53(Supplement C), 17-28. https://doi.org/10.1016/j.stueduc.2017.01.002
  21. Wellek, S. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Boca Raton: CRC Press.
  22. Wilson, K., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of the course experience questionnaire. Studies in Higher Education, 22(1), 33-53.

 

 

 

Tags

e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, ICT, MOOCs, information and communication technology, collaborative learning, internet, learning management system, LMS, interaction,

Current issue on Sciendo

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.