back

Information technology in developing a meta-learning environment

Seppo Sirkemaa [seppo.sirkemaa@pori.tukkk.fi]
Turku School of Economics and Business Administration, Pori Unit
P.O.Box 300, 28101 Pori, Finland [http://www.pori.tukkk.fi]

Abstract

Advances in technology provide new tools for educators and students to be used in the learning process. Today students need to be skilled computer users and able to deal with different types of media in order to study effectively. However, technical expertise alone is not enough – one must know what to study, where to find information, what to do next etc. Therefore special attention is needed to orientate students to use technology so that they learn to study in a modern learning environment.

In this study the focus is on adult students. It is also likely that some adult students may not be familiar with computers and information technology. In addition, they have not been in school for several years and therefore may not be used to studying. Consequently, adult students need advice and help in both using modern technology in studying, and in the studying itself.

Here we argue that a meta-learning environment can help students in the learning process. A meta-learning environment is defined as a platform for studying and it should integrate technology and resources so that they empower the learning process. In this article we look at the development of a learning environment using the meta-learning environment approach. The theoretical contribution is in the recognition of key dimensions of the meta-learning environment. They provide a promising framework for developing learning environments in the digital age.

Keywords

meta-learning, modern learning environment, orientation

Introduction

Information technology (IT) has affected our life profoundly – one can work, do shopping and keep in touch with far-away relatives with an inexpensive personal computer. IT has had a major impact on education as well. Firstly, it has highlighted the importance of continuous IT education that makes it possible to use changing technology. Secondly, IT has introduced new tools for educators and students to be used in the learning process. Thirdly, there has been an explosion in teaching and learning with new technology. The pace of technological change continues as ever smaller, portable computers are being introduced that make it possible to connect anywhere and anytime. However, it can be argued whether technology is being utilized so that it facilitates learning in the best possible way.

Here we look at the role of information technology in education and continue discussion with a modern learning environment, here called a meta-learning environment. A meta-learning environment is defined as a platform for studying and it should integrate technology into the learning process (Sumner, & Taylor, 1998; Scott, & Philips, 1998).  The focus of this article is on adult education and distance learning. In this setting mastering the technology and the "media mix" are challenges for all involved in the learning process (Laurillard 1993; 1995).

Towards a learning environment

Technology can be utilized in education in several ways. This continuum starts from information delivery and ends with cognitive tools (Reeves, & Laffey, 1999). Information delivery means that technology is used primarily to access and deliver information that is categorized and stored into digital format. For example, a library system may be considered as an information delivery system. Cognitive tools refer to adaptive technology or systems that are developed to support and improve the learning process (Jonassen, Reeves, Hong, Harvey, & Peters, 1997). However, the difference between information delivery oriented systems and cognitive tools is not very clear. For example, Reeves (1996) claims that the World Wide Web (WWW) is a cognitive tool because it focuses on creating knowledge and should therefore not be considered as an information delivery system alone. Both information delivery systems and cognitive tools are part of the learning environment and the learning process.

The difference between concepts learning environment and learning process is the same as with words teaching, studying and learning (Tella, Vahtivuori, Vuorento, Wager, & Oksanen, 2001). The learning environment is defined as a place to study, an environment for the learning process (Vivet, 1996). Today many learning environments take advantage of information technology and are built to a WWW-format or they connect to the Internet using a WWW-interface. As a result, a single computer connected to the Internet and running a WWW-browser can be used for ubiquitous access.

It is clear that the learning material must be organized within a meaningful structure (McGee, & Howard, 1998). In modern, computer-based learning environments the navigational logic relies on links that bind information together so that it makes sense. This navigational support is also called logistics (Neumann, Krzyzaniak, & Lassen, 2001). Ideally, the logistics system recognizes the user and delivers only the content that the user needs. The logistics system of a computer-based learning environment may also be seen as an intelligent agent or tutoring system (Prentzas, Hatzilygeroudis, Koutsojannis, & Rigou, 2001) and in a broader sense it is a human interface to the system (Raskin, 2000). Therefore, the usefulness of the modern learning environment depends on the navigational logic and the interface to the system.     

Adaptation

It is important to be able to modify the learning environment so that it supports the learning processes in the best possible way. This ability is called adaptation. Adaptation can be divided into two categories (Brusilovsky, 2001): adaptation to the user's behaviour and adaptation to the client device.

The first type of adaptation means that the system should know what the user expects. Because this "a priori" knowledge may not be available adaptation to the user's behaviour is often based on user feedback and behaviour. In this case facts about the user are gathered and analyzed so that users can be grouped according to agreed criteria (Rich, 1998).

The second type of adaptation refers to the portability of the platform, and is manifested in the flexibility to move and produce content to different hardware platforms and user devices. For example, the same content might be accessible with a desktop computer and a personal digital assistant (PDA). It is argued that the easiest method to increase portability in WWW- based learning environments is relying on XML-standard (Wehner, & Lorz, 2001).

In the digital age technology is changing fast. The result is that compatibility and flexibility to adapt to different devices and platforms are important issues in infrastructure (David, & Greenstein, 1990). Even though flexibility and the opportunity to make modifications are important, the information technology infrastructure should be relatively stable over time (Weill, & Broadbent, 1998, 85). This is a significant IT management challenge: how should infrastructure be developed so that it provides opportunities, instead of being a hindrance to development?  Developers of the system need therefore to ensure that new devices and applications can be connected to the existing information technology infrastructure (Broadbent, Weill, O'Brien, & Neo, 1996; Byrd, & Turner, 2000). However, technical compatibility with existing hardware and software is not enough. It is also important to consider fit with the organization and users of technology (Schein, 1986; Markus, 1991). As noted by Orlikowski (1992), technology needs to work within the frameworks of both existing technology and organizational philosophies.

The greatest challenge in learning environments is to adapt the computer-based system to differently skilled learners (Raskin, 2000, 68-70). If the environment is too complex the user will be lost, confused or frustrated. On the other hand, too simple or non-systematic environments can cause motivational problems. Technology and the learning environment should adapt to different user skills and learning strategies, not vice versa. Therefore the focus in adaptation should be on how the user manages technology that is constantly changing and requires the user to learn new technical skills in order to work with technology.  

The need for a meta-learning environment

Why learning support

Generally, the role of learning support is important (Bailey, Brown, & Kelly, 1997). Students must be able to get an answer to questions like "what should I do next?" and "do I have to take this course?". The advice is precious at the beginning of the student life, but guidance is needed also later during the studies. Each course raises questions before, during and after the course, and they all need to be answered. Telling students "what" and "why" is important as student's performance can be improved by guidance (Conway, & Kahney, 1987).

Advice and guidance are important in studying, whatever the technology is and whether students are in the faculty or at distance. As Twigg (1994) argues, we must take into consideration the recent changes that have occurred in what students need to learn, when they learn, how they learn, where they can learn, and what students can access while they learn.

In traditional, paper- and lecture-based education study guides had an important role in providing advice and guidance. However, study guides may not be enough in the digital age. Knowing what courses to select and why is still important in studying, but students must also cope with technology and digital media. In addition, students need to develop new studying habits in order to learn effectively. As a result, managing the learning environment becomes increasingly important.

However, what if the student is inexperienced with computers? Most young people are skilled computer users, but there are many adult people who have little experience with information technology. When these people get interested in studying further, the inexperience with computers can suddenly become a serious barrier for learning. As a result, adult students may feel uncomfortable with the technology that is used in education.

There is only one solution to this problem: the student must learn to use information technology, as studying requires skills in using technology and media. The need for assistance and support in using technology is most important in the beginning of studies. Therefore it is important to create an environment which helps students to "learn how to learn" effectively (Sumner, & Taylor, 1998, 64).

Obviously, face-to-face guidance is required but what else can be done? Here we argue that a meta-learning environment can become a valuable source of advice and support learning during the studies.

Dimensions of a meta-learning environment

Meta-learning environments have two main functions (Sumner, & Taylor, 1998; Scott, & Phillips, 1998). Firstly, they help in learning to use traditional and digital media in learning. A meta-learning environment should make it possible to access different kinds of digital content, but act as a guide to utilization of paper-based material as well. Secondly, a meta-learning environment can provide information on the courses and studies. The environment should present the structure of the course so that the student knows what exercises etc. are coming next. However, a meta-learning environment can be more than a study guide in computer format. It can describe the relationship between different media, methods and goals in each course.  This is important because students should have general understanding of learning tools, methods and media that will be used during a course. Furthermore, students need specific support in bridging various activities to the goals of the course. In order to utilize the environment effectively students need to be skilled enough to use information technology as a tool in learning. Many younger students are familiar with computers and information technology, but for adult students computers, file formats, login procedures etc.  can become a real barrier to studying. Even among students with "similar" background there are significant differences in technology skills and study methods. As a result, special attention is needed at the start of study and each course; otherwise results can be poor if the students do not know how to use the available tools and resources in studying. Often personal hands-on advice and tutoring is needed so that the student learns to effectively use technology in studying.

In a traditional classroom education these requirements are typically addressed well by teachers and other students (Taylor, Sumner, & Law, 1997). In modern learning environments this is a challenge. Whereas the overall resources can be made explicit by displaying the structure of the course in detail, hands-on support is a more complicated issue. For example, in distance learning advice and tutoring are limited to the tools that the student is using, and these are often the ones that students have problems with.

The Meta-learning environment should also establish and maintain new study habits and ways of working (Sumner, & Taylor, 1998; Scott, & Phillips, 1998). Once the methods of studying in the modern learning environment have been learned they should be maintained. If the students notice that they pass exercises and exams without seriously utilizing the tools and resources that would be available, they might just make the minimal effort – and not use the full potential of the learning environment. Therefore the courses and exercises should be planned so that students need to get involved personally. For example, in doing exercises there might be several interactions with the system - exercises could be split into parts - and checkpoints in order to prevent the student from copying answers to exercises from others.

Other key dimensions in meta-learning environments are (Sumner, & Taylor, 1998; Scott, & Phillips, 1998)

  • Ease of use.
  • Reliability and confidence building.
  • Possibility to modify and add material.
  • Tailoring the system to fit individual preferences.
  • Feedback and increasing motivation to study.

The meta-learning environment can integrate various tools, resources and media into a common workplace. This would make it possible to easily locate, access and combine information from several sources. The learning environment should also be easy to understand and use without cumbersome procedures. Both students and teachers should find it easy to navigate and find information in the learning environment.

Confidence is a critical issue in learning. Therefore, the learning environment should be reliable and robust. The system needs to function smoothly without interruptions, otherwise confidence is rapidly lost. Consequently the IT infrastructure is the cornerstone of the operations.

Adding and modifying material should be straightforward. It should be possible to add new material, and modify existing content in the meta-learning environment. Modifications need to be easy enough so that both students and teachers can submit and share ideas, concerns and questions. The modifications should be possible without deep technical expertise or help from the IT-department.

The environment should also be customizable to fit personal preferences. Typically,  students have different learning styles: Some learn fast and advance rapidly while others prefer to learn at a slower pace and repeat. In addition, some like working alone whereas others prefer to working in groups. There are also differences in using help or feedback that might be available. Information technology allows customization of the learner's learning experience and makes it possible to accommodate different learning styles (Albright, 1999; Wild, & Quinn, 1998).

Motivation is an critical issue in computer based learning environments – and a driving force in the learning process. Feedback can increase motivation and confidence, and it is considered essential for effective learning process (Rowntree, 1992). Feedback gives also student a realistic view of skills and knowledge (Scott, & Phillips, 1998). In classroom-based education feedback can happen face-to-face, and range from one-to-one conversation to a group discussion. However, in "virtual" learning there is often a lack of immediate feedback as lecturers and students do not necessarily meet very often. Feedback can also be constrained by the media and technology that is used. It is often typical that there is lack of immediate feedback. Information technology is ideal for providing timely and individualized feedback to students. Even though the richness of face-to-face discussion may not be achieved, technology can be used in ways that give rapid feedback to the student. For example, when the student sends an exercise the system could send an automatic acknowledgement. A meta-learning environment makes it also possible to monitor the students' learning process and to identify different styles of information access, communication and learning styles resulting from using new media for education (Hummel, & Hlavacs, 2003).

Experiences and findings

Research methods

The scientific approach of this research is qualitative. Here qualitative research refers to work that mainly uses qualitative data (Tesch, 1990, 55). From the methodological viewpoint this research is a case study research (Yin, 1994; Patton, 1989). Typically, data in a case study is observational and based on researchers' interpretations from interviews and discussions (McKinnon, 1988, 46).

In this research both interviews and a questionnaire are used to gather information. When data has been gathered from many sources observations can be triangulated (Benbasat, Goldstein, & Mead, 1987). As Miles and Huberman (1994, 267) note, multiple sources and types of evidence are used (in triangulation) to increase the reliability of the research findings. This should ensure that results provide deeper and more insightful information (Benbasat, Goldstein, & Mead, 1987; Patton, 1989).

The case setting

The case setting is a learning environment that was designed and implemented in 1999. The environment was developed for adults who wanted to continue their studies and take a degree, but could not do it because of work, family etc. The only possibility to reach these potential students was to arrange courses in the evening. Information technology had a major role in this setting as most material would be in digital format, located in a common information system and be accessible with a WWW-browser.

The group of students consisted of 25 adult students, between 25 – 50 years of age. Based on initial discussions with the students they had very different skills in using computers, and some had also been away from school for long time. It was mentioned when the students were selected that they should have a computer at home. However, it took a long time to help students to setup their own learning environment so that all could access the system from their own computers.

The content of studies was information technology, but also several other disciplines like mathematics, marketing, accounting and foreign languages were included. Altogether the studies took three years to complete. The studies were divided into two parts: lectures and distance learning. Lectures took place on evenings from Monday to Thursday, four hours at a time. In addition, students were expected to study at home, and here material accessible through the computer-based learning environment would be important. Also most exercises would be available in digital format. Initially, it was estimated that one third of studies would be lectures and two thirds would be distance learning.

The goal was to develop an environment that functions as a tutor by giving information on all courses and studies. Furthermore, it should support the learning process by functioning as a centre for course material. The idea of the learning environment was to integrate traditional media and digital media. Textbooks, articles and other paper-based material, digital text and graphics would be accessible from the learning environment. The interface of choice was a WWW-browser, which would not require students to purchase software or install it.

Pedagogically, information technology would be used as a tool that empowers learning, and it should be accessible to all students. This would mean that the technology that would be used should not be too high-tech and expensive so that only few could afford it. On the contrary, the environment was developed for low-end computers. It was also important to keep in mind that course material would be accessed from a distance and the connection would be probably made with a low-bandwidth modem. Therefore the amount of audio and video remained minimal.

Lotus Domino was selected to serve as a basis for the environment. It was considered as a well-supported and reliable infrastructure. Furthermore, Lotus Domino was seen as an open platform that can easily be modified for learning purposes. One of the key issues in selecting this platform was ease of use: adding material does not require special knowledge in WWW-editing, text and files can simply be pasted from other applications to the environment. Despite this, training lecturers in using the environment was considered as a critical success factor, and therefore training and assistance was made available for people who are involved and interested in using the environment in teaching. Technical staff would assist lecturers in preparing material, and they would also maintain and develop the system. They are also responsible for the high-level architecture of the environment. Making changes that impact other areas than a given course require also technical expertise. These changes also require accounts with higher rights than "normal" user has.

In practice most courses that were available through the environment had to do something with information technology as these lecturers were able to prepare material and use the environment with confidence. It seemed that lecturers in other areas were not as eager to change their existing methods and give technology such a major role. However, today several lecturers in other disciplines have started to use the environment. Despite this the number of people who are actively involved in the development process has remained between 5 to 10 persons. In addition, there are technical staff maintaining the platform and available in technical support for lecturers and students.

Today the learning environment is in active use. The technical platform is surprisingly robust and reliable. However, the environment is not considered as "complete" or "finished". The learning environment is now being used in several courses, and not only in the adult education experiment that it was originally developed for.

The research process

The research data was gathered with qualitative methods. The qualitative aspects of the learning experience are seen as a critical part of the learning process. This prompted us to approach the research with questionnaire complemented with interviews.

The researcher conducted key person interviews during 2000 and 2001. Key persons refer to people who have been actively involved in the development of the environment. Interviews lasted typically from one to two hours. They proceeded in a semi structured way so that predefined themes were discussed in a rather open manner. Each interview started with a short presentation of the purpose of the research project. Here the objective was to make concepts and goals clear to all interviewed persons (Klein, & Myers 1996, 31). In addition to the key person interviews, the researcher discussed with students that were using the learning environment. These discussions together with semi structured key person interviews were important sources of information. Relying solely on fill-in questionnaires would not have revealed as much information of the meta-learning environment.

The questionnaires asked the students' opinion of the learning environment. The students marked their opinions on a scale from 1 (poor) to 5 (excellent). We asked both adult students (n=19) and younger students (n=16) as later the environment was also made available to other student groups than adult students. Both groups answered the questionnaire during year 2001. As the number of students was small in both groups averages are used to estimate the opinions of the students.

Results

The results from the questionnaire are presented in Table 1. Here the two bars indicate results from two research groups: the black bar refers to adult students and the lighter bar refers to younger students. Even though the answers from adult students seem to be more positive than the opinions of the younger students, the overall trends are surprisingly similar.

Table 1. The questionnaire results

The questionnaire results

The results indicate that the environment presents the course structures well with adult students especially feeling this way. On the other hand, guidance and tutoring were not rated very highly. Discussions revealed that advice and help that is available through the environment tends to be incomplete or scattered so that it is often better to ask a "real human being" as one interviewed person said. Asking questions directly from a staff member is also seen as a faster way to get advice than browsing the system.

The learning environment helps in establishing new study methods, and integrates available resources so that all information is accessible through the same environment simply by clicking from the menu. As noted in discussions advice and assistance in using the technology and the environment is needed at the beginning of study. In this case it was obvious, that these arrangements could have been more thoroughly planned as several students and also developers of the environment told us that technical problems were considerable. At this time the environment was new and incomplete. Paradoxically, the tools that were to help the students in learning were considered as obstacles to learning, or at least the focus was heavily on how to cope with technology.  Fortunately, problems were soon forgotten as the users learned to master the tools that were available.

The learning environment is built to WWW-format and so it adapts to multiple user devices. Furthermore, the interface is straightforward because it can be used with a WWW-browser. As questionnaire results indicate the environment might do better in maintaining methods of working with technology (Table 1). Even though the environment is considered easy to use, it does not allow modifications that would adapt the environment to individual preferences. However, detailed questions did reveal that the learning material is readily accessible and it is easy to both add and modify material.

Overall, the environment is considered very robust and reliable. The freedom to study without time and place restrictions is considered as a positive feature both among students and teachers. With current technology access to information is instant, whether it is an article, exercise or result from a test. Adult students answered that the feedback that was available through the environment is excellent. Especially immediate feedback and possibility to contact the lecturer instantly were appreciated. Based on interviews the students considered the system as a online tutor, and the modern learning environment increased the motivation to study. The environment helps both students and lecturers to control the learning progress. For example, it is easy to rapidly check what exercises had been completed and which were missing. However, younger students did not value the feedback and control-related features very highly, and they did not consider the system to have significant impact on their motivation. Discussions with lecturers revealed that currently feedback requires personal attention from the lecturer, for example commenting exercise results. Even though this kind of feedback would be appreciated, it is very time-consuming and most lecturers do not have enough time to give enough feedback.

Interviews revealed also several other challenges in developing a computer-based learning environment. Firstly, the setup of the environment is expensive. Infrastructure-related costs are significant as lecturers should have modern computers and multimedia peripherals. The environment also needs a server for storing learning material and exchanging information. In addition, students need computers at home and they have to be willing to pay costs from connections to the environment. Secondly, developing material to the learning environment takes far longer than expected. It takes years rather than months to prepare a course in digital format. Thirdly, technical support is important. Both teachers and students need advice in order to cope with technology and applications. The amount of problems with modem configurations, logging in and access rights is endless. Despite help from the IT centre most problems need to be solved at teachers or students home. Furthermore, it seems that most problems occur after regular working hours. Here fellow students and friends can help as extra source of advice. Interestingly, student survival in the digital age seems to depend on how well one knows people who can help with different problems.

Motivation and enthusiasm are key elements in a "computerized" learning environment. It can be a challenge to find people who are willing to prepare material to a digital format. These people need to be skilled computer users, and brave enough to pilot in an environment that requires one to "present and give feedback through a monitor". We also found that students need tutoring and advice in order to make the sometimes not-so-reliable technology work for them.

Discussion

Information technology has significant potential in education. Based on experiences from several studies (Sumner, & Taylor, 1998; Scott, & Phillips, 1998) and this case study we believe that IT can support the learning process, and especially adult students can benefit. However, lot of work needs to be done in order to realise the potential.

Students need guidance and support before and during their studies. This is especially true in the digital age when students need to be skilled computer users and able to deal with different types of media in order to study effectively. For adult students who have not had experience with computers it is a big challenge. Many students feel overwhelmed and lost when they are facing a new way of learning with information technology. Therefore special attention is needed in teaching students to use technology so that they learn to study in a modern learning environment.

Here we argue that a meta-learning environment could be a response to these challenges. A meta-learning environment functions as a tutor that is always available. It integrates media and can be a channel for feedback and sharing ideas. It can help the student to orientate as the studies proceed. Accordingly, the developers should build an environment that supports the student in the best possible way.

Results from the case study indicate that the learning environment helps in establishing new study methods, and integrates available resources so that course information and material can be found in one place. The environment presents the course structure in a way that it makes sense.  The environment was also robust and reliable, and it is easy to access, add and modify learning material. The case environment also had shortcomings. It does not adapt well to user requirements, and cannot be customized. Feedback-related features were appreciated, even though the other group of users in the case study was not impressed. The system does, however, allow for checking of which exercises have been completed and which are missing.

The results are promising, but our experiences indicate also that the role of personal advice and support remains important. No matter how sophisticated the technology, the teaching resources and the environment are, students also need to discuss their problems face-to-face. The role of a meta-learning environment is still significant as it enriches and supports "real" tutoring that may not always be available. Furthermore, the meta-learning environment can improve learning by giving the student better control of the learning process.

To summarize the experiences, the learning environment should be built to support the learning process in the best possible way. Here the key dimensions of the meta-learning environment have an important role. We suggest that the developers of modern learning environments use these dimensions as cornerstones in development work. In light of this research the meta-learning environment approach seems a promising framework for developing learning environments in the digital age. However, it is important to highlight the role of the people involved in the learning process. In the long run the motivation and dedication to use and develop the environment are the most critical success factors. Consequently, the success depends on the human element rather than technological sophistication.

References

[1] Albright, M. (1999). "Teaching in the Information Age: A New Look." In Theall, M. (Ed.) New Directions for Teaching and Learning. San Francisco: Jossey-Bass Publications,  91-98.

[2] Argyris, Chris, Putnam, Robert, & McLain-Smith, Diana (1985). Action Science - Concepts, Methods and Skills for Research and Intervention. Jossey-Bass Publishers, London.

[3] Bailey, D., Brown, J., & Kelly, P. (1997). Academic advice, personal counselling and on-programme guidance in the Open University. In Personal Tutoring and Advice in Focus, Higher Education Quality Council, London.

[4] Benbasat, Izak, Goldstein, David K., & Mead, Melissa (1987). The Case Research Strategy in Studies of Information Systems. MIS Quarterly, 11(3), 369-386.

[5] Broadbent, M., Weill, P., O'Brien, T., & Neo, B.S. (1996). Firm Context and Patterns of IT Infrastructure Capability. In Proceedings of the Seventeenth International Conference on Information Systems. New York.

[6] Brusilovsky, P. (2001). Adaptive Hypermedia. User Modelling and User-Adapted Interaction, 11, 87-110.

[7] Byrd, T.A., & Turner, D.E. (2000). Measuring the Flexibility of Information Technology Infrastructure: Exploratory Analysis of a Construct. Journal of Management Information Systems, l 17(1), Summer, 167-208.

[8] Conway, M., & Kahney, H. (1987). Transfer of learning in inference problems. John Wiley, Chichester.

[9] David, P.A., & Greenstein, S. (1990). The economics of compatible standards: An introduction to recent research. Economics of Innovation and New Technology, 1, 3-41.

[10] Jonassen, D. H., Reeves, T., Hong, N., Harvey, D., & Peters, K. (1997). Concept Mapping as Cognitive Learning and Assessment Tools. Journal of Interactive Learning Research, 8(3), 289-308.

[11] Hummel, K.A., & Hlavacs, H. (2003). Anytime, Anywhere Learning Behaviour Using a Web-Based Platform for a University Lecture. SSGRR 2003, 7th – 12th of January, l'Aquila.

[12] Jönsson, Sten (1991) Action Research. In Nissen, H-E., Klein, H. K., & Hirschheim, R. (editors) Information Systems Research: Contemporary Approaches and Emergent Traditions. Elsevier Science Publishers B. V., North-Holland.

[13] Klein, Heinz K., & Myers, Michael D. (1996). The Quality of Interpretive Research in Information Systems. Binghamton University School of Management, Working Paper Series 1:1996.

[14] Laurillard, D. (1993). Rethinking University Teaching: A Framework for the Effective Use of Educational Technology. Routledge, London.

[15] Laurillard, D. (1995). Multimedia and the changing experience of the learner. British Journal of Educational Technology, 26(3), 179-189.

[16] Markus, M. L. (1991). Systems in Organizations. Cambridge.

[17] McGee, S., & Howard, B. (1998). Evaluating educational multimedia in the context of use. Journal of Universal Computer Science, 4.

[18] McKinnon, Jill (1988). Reliability and Validity in Field Research: Some Strategies and Tactics. Accounting, Auditing and Accountability, 1, 34-54.

[19] Miles, Matthew B., & Huberman, A. Michael (1994). Qualitative Data Analysis: an expanded sourcebook. Sage Publications, Inc. California.

[20] Neumann, G., Krzyzaniak, S., & Lassen, C.C. (2001). The Logistics of Knowledge Portal: Gateway to More Individualized Learning in Logistics. In Proceedings of ED-MEDIA 2001, 1400-1405.

[21] Orlikowski, Wanda J. (1992). The Duality of Technology: Rethinking the Concept of Technology in Organizations. Organization Science, 3(3), 398-427.

[22] Patton, Michael Q. (1989). Qualitative Evaluation Methods. Sage Publications, Newbury Park, CA.

[23] Prentzas, J., Hatzilygeroudis, I., Koutsojannis, C., & Rigou, M. (2001). Architectural Aspects of a Web-Based ITS for Teaching New Information Technologies. In Proceedings of ED-MEDIA 2001, 1516-1521.

[24] Raskin, J. (2000). The Human Interface, New Directions for Designing Interactive Systems. Addison-Wesley Longman Inc, Massachusetts.

[25] Reeves, T. C. (1996). A model of the effective dimensions of interactive learning on the World Wide Web. In Proceedings of Ed-Telecom 1996, Boston, USA.

[26] Reeves, T. C., & Laffey, J. (1999). Design, Assessment, and Evaluation of a Problem-based Learning Environment in Undergraduate Engineering. Higher Education Research & Development,18(2), 219-232.

[27] Rich, E. (1998). User Modeling via Stereotypes. In Maybury, M. T., & Wahlster, W. (Ed.) Readings in Intelligent User Interfaces. Morgan Kaufmann Publishers Inc, San Francisco, California.

[28] Rowntree, D. (1992). Exploring Open and Distance Learning. Kogan Page, London.

[29] Schein, Edgar H. (1986). Organizational Culture and Leadership. Jossey-Bass Publishers.

[30] Scott, P., & Phillips, M. (1998). Developing Web-based student support systems: telling student stories on the Internet. In Eisenstadt, M., & Vincent, T. The Knowledge Web: Learning and Collaborating on the Net. Kogan Page, London.

[31] Sumner, T., & Taylor, J. (1998). Media integration through meta-learning environments. In Eisenstadt, M., & Vincent, T. The Knowledge Web: Learning and Collaborating on the Net. Kogan Page, London.

[32] Taylor, J., Sumner, T., & Law, A. (1997). Talking about Multimedia: A Layered Design Framework. The Journal of Educational Media, 23(2-3), 215-141.

[33] Tesch, Renata (1990). Qualitative research: Analysis Types and Software Tools. The Falmer Press, New York.

[34] Tella, S., Vahtivuori, S., Vuorento, A., Wager, P., & Oksanen, U. (2001). Verkko opetuksessa – opettaja verkossa. [The Net in Teaching – The Teacher on the Net]. Edita, Helsinki.

[35] Twigg, C. (1994). "The Need for a National Learning Infrastructure." Educom Review. 29,  4-6.

[36] Wehner, F., & Lorz, A. (2001). Developing Modular and Adaptable Courseware Using TeachML. In Proceedings of ED-MEDIA 2001, 2013-2018.

[37] Weill, P., & Broadbent M. (1998). Leveraging the New Infrastructure: How Market Leaders Capitalize on Information Technology. Harvard Business School Press. Boston, Massachusetts.

[38] Wild, M., & Quinn, C. (1998). "Implications of educational theory for the design of instructional multimedia." British Journal of Educational Technology, 29(1), 73-82.

[39] Van Maanen, John (1983). Qualitative Methodology. Sage Publications, Inc., Newbury Park, CA.

[40] Yin, R. K. (1994). Case Study Research, Design and Methods. Sage Publications, Beverly Hills.

[41] Vivet, M. (1996). The classroom as one learning environment of the future. Journal of Universal Computer Science, 2.

 

Tags

e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, ICT, information and communication technology, internet, collaborative learning, learning management system, MOOC, interaction, LMS,

Current issue on De Gruyter Online

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.