back

E-learning measurement of the learning differences between traditional lessons and online lessons

Giuseppe Favretto, Giovanna Caramia and Matteo Guardini
Università di Verona (Italy)
giovanna.caramia@formazione.univr.it

Abstracts

English Abstract

The research aims at highlighting, through the application of appropriate statistical techniques, any possible differences in the results achieved by three distinct groups of students - attendees, e-learners and non-attendees – formed at the beginning of the course according to their own personal choice.  

Italian Abstract

L'obiettivo della ricerca consiste nell'evidenziare, attraverso l'applicazione di opportune tecniche statistiche, le eventuali differenze a livello di risultati raggiunti da tre gruppi distinti di studenti: frequentanti  in aula, frequentanti on-line e non frequentanti.

Keywords

E-learning, online learning, classroom students, pedagogy, student performance

Introduction

The exponential development of increasingly sophisticated communication technologies has prompted universities, companies and educational institutions to experiment with alternatives to the traditional classroom teaching methods, thereby leading to the evolution of a wide range of online courses. Nonetheless skepticism towards this virtual "means of communication" is still common. This conflict has probably influenced research on the subject, which mainly focused on the students' performance, reaching the conclusion that on average online students' results are at least as good as classroom students' ones. Research conducted in Denver, Colorado, showed that 85% of e-learners reached an equal or even better learning level compared to campus students (eCollege 1999). The comparison between performances is often based on differences in exam marks, assessed using the t-Student test (Rivera and Rice 2002) or through analysis of variance as, for example, in the experiments carried out by Parker, Gemino (2001) at Simon Fraser University in Canada. The authors set out to measure the students' final mark with the aim of plotting the differences between the groups formed with regard to a particular learning model (Learning Process Model).

Picciano (2002) approached the problem by considering interaction between students and teacher and between students themselves relevant for the success of web courses, and particularly emphasized involvement in the lessons and the students' sense of belonging to the same class; performance was therefore assessed as a function of these two previously defined variables.

Spiceland, Hawkins (2002) based a study on the themes of interaction, active participation in lessons, perception of the students and learning outcomes, in order to compare web-based courses with traditional courses to assess their effectiveness.

Dutton J., Dutton M., Perry J. (2002) addressed another relevant aspect of distance learning. Drawing on their previous research (1999) on students' performance, they considered the different types of students interested in starting an online course rather than a traditional one. Furthermore, they tried to highlight which factors influence performance, evaluating whether different factors apply to the two types of student. Other research demonstrated the effectiveness of online learning in company training courses (Spiceland 2002).

Sphere of reference

Online lessons seem therefore to offer an alternative for those who cannot attend classroom lessons. Many Italian universities, as well as the Consorzio Nettuno (Telematic And Television University, http://nettuno.stm.it), are contributing to the rapid spread of e-learning courses. Milan Polytechnic, in collaboration with Somedia, has also started its first online degree course in Engineering. Equally important is the ICON project (Italian Culture On the Net at http://www.italicon.it), consisting of 34 higher education institutions and designed to allow foreign students and Italians living abroad to graduate through the Internet.

All this, coupled with the interest of the Centro Docimologico1 (Docimological Centre) at the University of Verona in improving knowledge and experience, undoubtedly encouraged research, leading to the design, experimentation and development of an informatics platform for e-learning (Eletti 2002; Corbi 2002; La Noce 2001). This platform is the result of synergy and interaction between different areas of humanistic expertise, translated into the informatics and multimedia arena, the assessment of different types of users who can access it (Najjar 2001) and hypotheses for possible educational contexts, and potential group and individual activities (Garrison 1993; Hedestig and Kaptelinin 2001).

This distance learning method was tested and refined by applying web software to university teaching at the Department of Psychology and Cultural Anthropology. The online course took shape by defining the following elements:

  • Classes of 10-15 students, also interactive, each of them guided by a tutor. Students can send e-mails, exchange opinions in forums and thus have the opportunity of developing the different topics together.
  • Learning check: each student has to pass a proficiency multiple-choice random test before moving on to the next lectures. This allows a constant check and assessment of the students' progress and it is the main difference between this method and  traditional classes, since students are obliged to keep up constantly with the contents of each lesson.
  • Interactive lectures, in animation format, with a lecturer who speaks in a virtual classroom and  who can formulate questions, using the same methods as in the traditional classroom lesson.  

The continuous feedback between students and tutor also helped to verify the quality of the software used for the online lessons, the E.L.M.S. (E-learning Management System).

The web homepage supporting this software is divided into three main parts (Administrator, Tutor and User), each of them consisting of interfaces able to handle data input, classroom interaction or course lessons respectively.

To assess the effectiveness of this innovative educational tool, the traditional face-to-face formula was compared with the same lectures available online2, enjoyed by users following an "Asynchronous Model"3 of distance learning. Specifically, the aim was to describe and measure the learning levels of attendees and e-learners, in order to highlight any important differences in terms of results and level of knowledge reached; can we claim, then, that such differences do not exist? The following experiment plan was therefore designed.

The experiment plan

The experiment was carried out over the following three stages:

  1. Definition of both the experimental and the control groups;
  2. identification of the independent variable, i.e. the method of giving the lecture;
  3. calculation, in the two groups, of the average value of the dependent value read in the pre-test and post-test.

The experiment aimed at comparing the two independent samples using the t-Student test to assess the adopted methodology and check the effectiveness of distance learning. From a statistical analysis point of view, the aim was to answer the following question: can any discrepancies resulting from the test be attributed to initial differences rather than to an effect of giving the lecture?

Structure of operative stages of the experiment

Figure 1. Structure of operative stages of the experiment

In detail:

  • Control group: classroom students; experimental group: online students. The groups are not equivalent, and they not only differ in treatment; at this stage any social and cultural variables must be identified4.
  • Pre-test: aiming at supplying information on the features of the two groups (and therefore possible pre-established differences), trying to check for any prior knowledge. The causal impact of the stimulus can therefore be assessed, comparing the variation arising between the two groups after the lesson. In order to check for any post-test influence and to survey the level of preparation before the lecture, the pre-test contained general questions concerning themes not addressed directly or in a detailed way in the post-test questions.
  • Giving the experiment lecture.
  • Post-test: test containing questions on the experiment lecture, to be completed during the final classroom exam.

Briefly, the aim was to read the dependent variable, or rather the pre-test and post-test scores, in order to observe and assess post-pre differences by analyzing increases in knowledge5.

Finally, the students' preparation was assessed and compared through the exam marks, subdividing them into e-learners, attendees and non-attendees to infer any differences in the results. Can we assert that e-learners follow the trend of the previous research, conducted in different university environments?

Development of the stages in the experiment plan in order to measure the learning level

The differences in learning level were therefore obtained using two measurements:

  1. Difference in learning in a sample lecture
  2. Marks in the final exam

As regards the first experiment, an identical lesson was given to two groups of students: one group followed the content in the classroom and the other online, in order to assess and measure any important differences in their learning levels and connect them to the teaching method used (Trentin 2001; Venkatachary 2002).

As for the methodological process used, it was essential to determine the most appropriate tools for the aims of the experiment: any prior knowledge (Toyton 2002) on the subject chosen, the choice of the subject of the lecture and the method for administering the tests.

In order to check for any prior knowledge, one test was set before and one after the lesson. This immediately posed a problem: in order to yield data which can be compared, the pre and post tests would have to be very similar, creating the risk that after answering the first test, the subject might pay special attention to the elements addressed in these questions and developed during the lesson, thereby answering in a distorted way and producing better post-test results. The compromise consisted in formulating very general questions for the first test and more detailed ones for the second.

A further problem (McDonald 2002) regarded the opportunity of giving a general-type distance lecture or instead following the method developed by the Docimological Centre, i.e. online lessons providing for a test at the end of each teaching unit with precise items suggesting the points to be revised. Indeed, setting these questions could have interfered with the test to be administered after the lesson to check learning.

Pre-test and final check: some considerations for possible solutions

As previously suggested, the pre-test contained general questions, specially developed to survey the students' preparation before the lecture was given (21 E-learning students did the pre-test online, 26 students in the classroom). Multiple-choice questions were chosen since their answers could be managed more effectively; they referred to topics and concepts already developed during the course.

The subject of the experiment lecture coincided with a lecture on the course, held in the classroom and activated online.

The final check stage followed the same method for both groups and was carried out at the exam sessions in June and September, when a high turnout could be guaranteed in the classroom. For this reason, besides the exam programme, students had to answer a few questions about the experimental lecture. All subjects were in identical conditions in terms of method of answering (test paper) and motivation for answering carefully. The items contained were extracted at random from the pre-set database.

Comparison between E-learners and attendees: differences in learning level?

As regards the first experiment, the giving of the test to the two groups of E-learners and attendees inevitably led to a comparative analysis of the marks obtained (on a 1–10 scale). Descriptive indexes were then calculated for both groups to display marks distribution (see Tables 1 and 2), thereby giving a rough idea of the characteristics of the two samples.

Table 1. Statistics relating to the mark obtained in the pre-test by the E-learning and attending groups

E-learning

attendees

Lowest mark

1

Lowest mark

1

Highest mark

10

Highest mark

6

Mean

5.524

Mean

2.923

Median

6

Median

3

Mode

6*

Mode

4

Std deviation

2.294

Std deviation

1.495

*there are several modes, the lowest value is indicated

Table 1, regarding the marks obtained in the pre-test, displays some considerable differences between the indexes; Table 2, regarding the second test, does not highlight any particular differences in the relative indexes.

Table 2. Statistics relating to the marks obtained in the second test by the E-learning and attending groups

E-learning

attendees

Lowest mark

4

Lowest mark

5

Highest mark

10

Highest mark

10

Mean

7.381

Mean

8.115

Median

8

Median

8

Mode

8*

Mode

9

Std deviation

2.037

Std deviation

1.395

*there are several modes, the lo west value is indicated

Within this data, the mean and standard deviation, analyzed to identify and explain the nature of the resulting differences, should be noticed. In this respect two comparisons, between the marks obtained before and after the lecture, were carried out.

As regards the former, the t-Student test for independent samples (t = 4.68 with p < .01) confirmed the significance of the difference in marks between the two groups, in the sense that the difference in the corresponding means (about 2.6) can be attributed to systematic factors or possible so-called external variables. An influential factor could be, for example, the fact that e-learners become skilful and familiar with multiple choice tests by compiling the check questionnaire after each lecture; or the belief that E-learners are able to organize their learning programme autonomously and are therefore more motivated than attendees.

There are no meaningful differences after the lesson; the t test (t = -1.41 with p = .17) is not indicative, and the difference between the means of the marks obtained is considered merely accidental.

Non-attending students: how they fit into the picture

The second experiment involved also non-attendees, i.e. students who take exams without having taken part in lessons, with the aim of plotting the marks obtained in the exam and analyze any differences in the results. Everyone sat the exam in the classroom at the same time and under the same conditions.

In order to provide some tools for an initial reading of the salient points of the three situations outlined, and for subsequent comparisons, a table was created with some indexes clearly displaying the three subject groups of the experiment – exam marks (out of thirty) for E-learners, attendees and non-attendees.

Table 3. Statistics relating to the distribution of marks

Course Number Mean Median Mode Min mark Max mark Range (max mark)
- min mark
Variance Standard dev
E-learning 88 24.4886 25 27 13 30 17 11.5861 3.4038
Attendees 40 24.4750 25 29 9 29 20 20.3071 4.5063
Non-attendees 142 14.2535 13 11 3 29 26 34.9707 5.9136
Total 270 Descriptive indexes   Variability indexes

The first column displays the number of each distribution of marks in the exams, followed by columns indicating these indexes: mean, median and mode; the next two show the relative minimum and maximum values; the final columns contain the figures for range (i.e. the difference between the maximum and minimum marks), variance and standard deviation to measure dispersal around the mean.

E-learning, attending and non-attending students: what implications?

The resulting picture shows three well-defined situations: E-learning and attending students in contrast to non-attendees. E-learning shows a mean value slightly higher compared to attendees (analysis of variance, ANOVA, with F = .000 and p = .985, reconfirms that the difference is not statistically significant). This value is also associated with a lower standard deviation (3.40 against 4.51), indicating less dispersal of the data around the mean value and, consequently, better representativeness of marks distribution. Furthermore, in the interpretation of these statistics, the different quantities of data in single groups (88 opposed to 40) is important as it consolidates this representativeness. The non-attendees differ markedly from the other two groups, indeed displaying a very low mean value (the lowest) and a high standard deviation (the biggest). This can all be explained by observing the range of the distribution of marks, from 3 to 29: the mean is strongly affected by these extreme values. In light of these findings, ANOVA was applied as confirmation of what can be inferred from simple interpretation of the calculated indexes and of the following graphs, Figures 2 and 3, plotting the trends of marks obtained in the exam.

Figure 2 displays the absolute frequencies, with the aim of stressing further the distinctive elements, and especially of highlighting the scale of the study subject phenomenon, and the plotting of distribution as a Gaussian variable.

Distribution of frequency of marksDistribution of frequency of marksDistribution of frequency of marks

Figure 2. Distribution of frequency of marks

These considerations led to Figure 3, displaying the three situations observed in a single diagram, plotting the probabilities on the y-axis with a view to making comparisons. The curves for E-learning and attendees are quite similar: their means differ slightly (only 0.0136 difference), they have the same median, but the former has a lower mode (27 compared to 29 for attendees), and is more "pointed" as it has less dispersal. The curve for non-attendees moves further to the left compared to the other two graphs, indeed the mean of relative distribution is very low (14.25), the mode corresponds to the mark 11 and the median is 13; finally it shows the biggest dispersal (its curve is flatter) with a standard deviation of 5.91.

Distribution of exam marks: the three groups compared

Figure 3. Distribution of exam marks: the three groups compared

As already suggested, ANOVA was applied by forming a sample of non-attendees, contrasted with the E-learning and attending groups. The analysis of the results (F = 280.56 with p < .01) also underlines the separation between the groups, assimilating E-learners and attendees.

In the experiments of the Docimological Centre, distance teaching was just as effective as classroom teaching; it also allowed for constant feedback between teacher/tutor and student6, by using the test provided at the end of every teaching unit.

Concluding remarks

E-learners must be active and assimilate as much knowledge as possible to reach the necessary level of expertise to pass the exams. Moreover, they must participate in activities, thereby increasing their understanding of the concepts7. Furthermore, traditional students may easily assume a passive role during the lesson, in a form of dependence on the teacher. All this is opposed to psycho-pedagogical principles concerning active involvement of students and intrinsic motivation essential to the teacher to give them an effective, efficient and long-lasting learning experience.

In our experiment too, the comparison between the proficiency test marks of online and classroom students highlighted that, on average, the marks of E-learners are just as good, if not better, than the marks of attendees. Moreover, analysis of the data further suggested testing the hypothesis of comparing E-learners and attendees by comparing them to non-attendees. The application of the analysis of variance between groups thus defined further confirmed this supposition.

These considerations highlight the importance of the characteristics of the student, with the central role carried out by active participation in the lesson but also by personal aspects which can influence the performance of e-learners.

The method and technologies used, the type of interaction between course participants, constant feedback, together with the characteristics of the student, are all factors the Docimological Centre will further study.

1 Docimology derives from dochimoun (of Indo-European origins), which means "put to the test". This discipline, based on psychology and pedagogy, scientifically studies the methods of scholastic tests and their assessment criteria. It is fundamental in the university sector both for degree course proficiency exams and university course entrance exams. http://cd.univr.it

2 Online teaching is indeed characterized by physical separation between teachers and students, who can, however, communicate using appropriate technologies (Perrault 1997; Moore and Kearsley 1996).

3 iTechnology mediates communication between teacher and student, in the sense of time displacement between the lesson given and the participation of the student (Casarotti, Filippini, Pieti, and Sartori 2002). It is otherwise defined as the "Diachronic Model", opposed to the "Synchronous Model" based on simultaneity, or rather the teacher conducts the lesson while the students attend it (Garito 1996; Garito 1997; Garito 1998). Trentin (1998) suggests a further solution in the potential of both models, i.e. the organization of mixed environments based on the streaming effect.

4 In a future study the Docimological Centre is going to take into consideration also the social differences of the whole group (attendees, e-learners, non-attendees), as well as the reasons for the students' choice of the type of lesson.

5 Johnson (2002), for example, in a study in the field of expertise in biology, demonstrated, by ANOVA, that there are no statistically differences in the results obtained by online students compared to traditional students, with reference to pre- and/or post-attitudes towards particular abilities, such as scientific reasoning, a positive attitude towards biology, learning the contents of the course.

6 The constant feedback on the learning level reached together with teacher-student interaction prove to be fundamental elements in planning an online course (Coldeway, MacRuy, and Spencer 1980; Moore and Thompson 1990; Mason and Kaye 1989).

7 The active role of online students allows them to organize and develop their knowledge and expertise autonomously, creating a sort of "self-regulating learning" (De Jong and Simons 1990; Lowyck 1996).

References

[1] Casarotti, M., L. Filipponi, L. Pieti, and R. Sartori 2002. Educational interaction in distance learning. Analysis of a one-way video and two-way audio system. PsychNology Journal 1 (1). Available online at http://www.psychnology.org.

[2] Consorzio Nettuno. Available online at http://nettuno.stm.it.

[3] Corbi, E. 2002. La formazione a distanza di terza generazione. Nuove frontiere per l'educazione degli adulti. Napoli: Liguori.

[4] Coldeway, D. O., K. MacRuy, and R. Spencer. 1980. Distance education from the learner's perspective: the results of individual learner tracking in Athabasca University. Athabasca University, Edmont, Alberta.

[5] De Jong, F. P., and P. R. J. Simons. 1990. Cognitive and Metacognitive Processes of Self-regulated Learning, in Research in Computer Based Instruction ed. J. M. Pieters, P. R. J. Simons and L. De Leeuw, 81-100. Amsterdam: Swet & Zeit Linger.

[6] Dutton, J., M. Dutton and J. Perry. July 2002. How do online students differ from lecture students? Journal of Asynchronous Learning Networks 6 (1).

[7] Dutton, J., M. Dutton and J. Perry. 1999. Do Online Students Perform As Well As Traditional Students?, in The No Significant Difference Phenomenon ed. T. L: Russell. IDECC, Montgomery.

[8] eCollege. 1999. Survey Finds Online Education Equal to or Better than On Campus Learning. Denver. May 10, 1999. Available online at http://www.ecollege.com.

[9] Eletti, V. 2002. Che cos'è l'e-learning. Roma: Carocci.

[10] Garito M.A. (edited by) (1996). La multimedialità nell'insegnamento a distanza. Roma:Garamond.

[11] Garito, M.A. 1997. La costruzione della Società Euromediterranea dell'Informazione: comunicazione, educazione, formazione e ricerca. Firenze: Giunti.

[12] Garito, M.A. 1998. Tecnologie e processi cognitivi. Insegnare ed apprendere con la multimedialità. Roma: Garamond.

[13] Garrison, D.R. 1993. A cognitive constructvist view of distance education: an analysis of teaching-learning assumptions. In Distance education 14 (2): 199-211.

[14] Hedestig, U. and V. Kaptelinin. 2001. A conceptual model of the social context of distance learning. In The future of learning – learning for the future: shaping the transition. Abstracts and conference papers of the 20th ICDE word conference. April 1-5, 2001. Düsseldorf: Germany On CD-ROM. 10p.

[15] iCDL International Centre for Distance Learning. Available online at http://www-icdl.open.ac.uk.

[16] Johnson, M. 2002. Introductory Biology Online: Assessing Outcomes of Two Student Populations. In Journal of College Science Teaching. XXXI (5): 312-317.

[17] La Noce, F. 2001. E-learning. La nuova frontiera della formazione. Milano: Franco Angeli.

[18] Lowyck, J. 1996. Lo stato dell'arte della ricerca sui processi cognitivi in relazione all'uso delle nuove tecnologie interattive. In La multimedialità nell'insegnamento a distanza. Ed. M. A. Garito. 1996. 97-104.

[19] Mason, R., and A. Kaye. 1989. Mind weave: communication, computers and distance education. Oxford: Pergamon Press.

[20] McDonald, J. August 2002. Is "as good as face–to-face" as good as it gets? in Journal of Asynchronous Learning Networks 6 (2).

[21] Moore, M. G., and G. Kearsley. 1996. Distance education: A System view. Boston: Wadsworth Publishing Company

[22] Moore, M. G., and M. M. Thomson. 1990. The effect of distance learning: A summary of the literature. Research Monograph, 2, University Park, PA: The Pennsylvania State University, American Center for the study of Distance Education.

[23] Morgan, A. 1991. Research into student learning in distance education. Victoria, Australia: University of South Australia, Undedale.

[24] Najjar, L. J. 2001. Principle of educational multimedia user interface design. In Readings in training and simulation: A 30-year perspective.ed. R. W. Swezey and D. H. Andrews, 146-158. Santa Monica, CA: Human Factors and Ergonomics Society.

[25] Parker, D., and A. Gemino. September 2001. Inside online learning: comparing conceptual and technique learning performance in place-based and ALN formats. In Journal of Asynchronous Learning Networks 5 (2): 64-74.

[26] Perrault, J. 1997. La comunicazione del sapere a distanza. Bologna: Pitagora Editrice.

[27] Picciano, A. G. July 2002. Beyond student perceptions: issues of interaction, presence and performance in an online course. In Journal of Asynchronous Learning Networks 6 (1): 21-40.

[28] Progetto ICON. Italian Culture On the Net. Available online at http://www.italicon.it.

[29] Rivera, J. C., and M. L. Rice. 2002. A comparison of students outcomes & satisfaction between traditional & web based course offerings. In Online Journal of Distance Learning Administration V (III). Fall 2002. State University of West Georgia, Distance Education Centre.

[30] Spiceland, J. D., and C. P. Hawkins. July 2002. The impact on learning of an asynchronous active learning course format. In Journal of Asynchronous Learning Networks 6 (1): 68-75.

[31] Spiceland, J. D. April 2002. An assessment of the effectiveness of e-learning in corporate training programs. In International Review of Research in Open and Distance Learning 3 (1).

[32] Toynton, B. 2002. Perceptions of learning and perceptions of being taught: adult learner reactions to an interactive Website. In Networked learning 2002: a research based conference on e-learning in higher education and lifelong learning. Ed. S. Banks, P. Goodyear, V. Hodgson, D. McConnell. 2002. Proceedings of the third international conference jointly organized by the University of Sheffield and Lancaster University, and held at Sheffield University, United Kingdom, 26th-28 March 2002, 581-587.

[33] Trentin, G. 1998. Insegnare e apprendere in rete. Bologna: Zanichelli.

[34] Trentin, G., 2001. Dalla formazione a distanza all'apprendimento in rete. Milano: Franco Angeli.

[35] Venkatachary, R. 2002. From distance education to e-learning: philosophical and design imperatives. In Networked learning 2002: a research based conference on e-learning in higher education and lifelong learning. Proceedings of the third international conference jointly organized by the University of Sheffield and Lancaster University. Ed. S. Banks, P. Goodyear, V. Hodgson, D. McConnell. 2002. Sheffield University. United Kingdom. 26th-28 March 2002. 596-601.

 

Tags

e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, ICT, information and communication technology, internet, collaborative learning, learning management system, MOOC, interaction, LMS,

Current issue on De Gruyter Online

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.