New Chief Editor in EURODL

The EDEN Executive Committee assigned Dr. Ulrich Bernath as Chief Editor of the journal.
 

Latest article

Introducing a Structured Format for Learner Self-Assessment: The Tutor's Perspective

15.12.2014

Latest Brief Item

Strategies for the Adoption of Open Textbooks: The Latin American Open Textbooks Initiative

02.10.2014

Archives

EURODL Mailinglist

2879 subcribers
 

EURODL Visitors

 

B-learning Quality: Dimensions, Criteria and Pedagogical Approach

Paula Peres [pperes@iscap.ipp.pt], e-IPP / CICE, Polytechnic Institute of Oporto, Portugal [www.iscap.ipp.pt]
L. Lima [llima@estgf.ipp.pt], V. Lima [vlima@estgf.ipp.pt], e-IPP / ESTGF - CIICESI, Polytechnic Institute of Oporto, Portugal [www.estgf.ipp.pt]

Abstract

Measuring the quality of a b-learning environment is critical to determine the success of a b-learning course. Several initiatives have been recently conducted on benchmarking and quality in e-learning. Despite these efforts in defining and examining quality issues concerning online courses, a defining instrument to evaluate quality is one of the key challenges for blended learning, since it incorporates both traditional and online instruction methods. For this paper, six frameworks for quality assessment of technological enhanced learning were examined and compared regarding similarities and differences. These frameworks aim at the same global objective: the quality of e-learning environment/products. They present different perspectives but also many common issues. Some of them are more specific and related to the course and others are more global and related to institutional aspects. In this work we collected and arrange all the quality criteria identified in order to get a more complete framework and determine if it fits our b-learning environment. We also included elements related to our own b-learning research and experience, acquired during more than 10 years of experience. As a result we have create a new quality reference with a set of dimensions and criteria that should be taken into account when you are analyzing, designing, developing, implementing and evaluating a b-learning environment. Besides these perspectives on what to do when you are developing a b-learning environment we have also included pedagogical issues in order to give directions on how to do it to reach the success of the learning. The information, concepts and procedures here presented give support to teachers and instructors, which intend to validate the quality of their blended learning courses.

Keywords: e-learning, blended learning, learning quality, quality models, e-learning dimension, b-learning criteria, pedagogical approach for learning

Introduction

E-learning has become widely used in every type of education (traditional and formal education, continuous education and corporate training) because of its characteristics such as flexibility, richness of materials, resource-sharing and cost-effectiveness.

In this work we paid more attention to the blended-learning (b-learning) systems, which consider systems “combining face-to-face instruction with computer-mediated instruction” (Graham, 2004). E/b-learning has been largely used in the context of higher education. It includes a wide range of learning formats including self-study and instructor-led in both an asynchronous and synchronous mode. The e/b-learning systems may represent as an alternative to traditional teaching/learning and training and, therefore, has had to battle for recognition. As a consequence of this need, procedures have been developed in order to demonstrate its quality.

Evaluating the quality of a b-learning environment is not an easy task since this concept is not objective. It depends on the students’ perceptions and there are also several multi-dimensional variables factors with factors that we have to take into account. As the number of b-learning courses is increasing, more and more, it is important to evaluate the quality offered in order to help the potential users choosing the best course.

Models to evaluate the quality in education context

There are several standards related to quality in education context. The ISO/IEC 19796 series of standards, published as ISO/IEC 19796: Information technology – Learning, education and training – Quality management, assurance and metrics, provides a framework for implementation and monitoring quality management systems in educational organizations. Currently, two standards of this series are published – ISO/IEC 19796 Part 1 – General Approach and ISO/IEC 19796 Part 3 – Reference methods and metrics – and other three are under preparation – ISO/IEC 19796 Part 2: Harmonized quality model; ISO/IEC 19796 Part 4: Best practice and implementation guide; and ISO/IEC 19796 Part 5: How to use ISO/IEC 19796-1.

The ISO/IEC 19796-1 (ISO/IEC 19796-1, 2005) is a general framework to develop and implement quality in educational organizations. This standard contains the reference process model “Reference Framework for the Description of Quality Approaches (RFDQ)”, that is divided into seven process categories: needs analysis, framework analysis, conception/design, development/production, implementation, learning process, and evaluation/optimization. RFDQ model covers the whole lifecycle of learning, education and training, including e-learning and b-learning.

The ISO/IEC 19796-3 (ISO/IEC 19796-3, 2009) extends the RFDQ model, by providing harmonized methods and metrics required to implement quality management systems.

Recently, a Portuguese standard that specifies requirements for a vocational training management system, including technology enhanced learning was published (NP-4512, 2012). This standard has its focus on ensuring that the organization processes (such as those related to the formative cycle: diagnosis of training need; design, development and innovation of training products; planning, organization and realization of training courses and assessment and/or certification of learning) are managed with the aim of increasing customer satisfaction and conformance. The standard NP 4512:2012 already considers the scenarios of e-learning, b-learning or m-learning (mobile learning). Other countries, such as the United Kingdom, Spain and Germany, also have developed standards for e-learning. The British Standard 8426:2003 (BS-8426, 2003) makes recommendations for e-support in e-learning systems independently of the pedagogical approach (e-support provided by human tutors or automated; learners’ work done individually or in groups; or the underlying pedagogy of a course involves learners in constructing their own understanding or in committing course content to memory). The Spanish standard UNE 66181:2012 (UNE-66181, 2012) specifies guidelines to identify the characteristics of virtual courses in relation to potential customers. The use of this standard is intended to increase transparency and market confidence in e-learning. The Germany standard PAS 1032-1 (PAS 1032-1, 2004) provides a reference model for quality management and quality assurance especially developed for the education and vocation training sector, with a special focus in e-learning.

Still about this topic, several initiatives have been recently conducted on benchmarking and quality in e-learning products. For this paper, six frameworks for quality assessment of technological enhanced learning were examined and compared regarding similarities and differences. These frameworks were chosen as representative of international initiatives, already used in the worldwide in higher education institutions. A brief description of these frameworks follows.

Open ECBCheck Initiative (Ehlers, 2010) from EFQUEL (European Foundation for Quality in e-Learning), aimed to offer a quality label for e-learning in Capacity Building, is rooted in four best practice labels: UNIQUe (EFQUEL, 2011), EFMD CEL (EFMD, 2010), D-ELAN DELZert (Ehlers, 2010) as well as ISO/IEC 19796-1 (Pawlowski, 2006). Open ECBCheck follows a certification process with three major steps: self-assessment, peer-review of self-assessment report, peer-review report, including evaluation results, and learning report as well as recommendations for certification. ECBCheck covers seven main areas: Information about the program and respective organization, target group and orientation, quality of the contents, programme /course design, media design, technology, evaluation & review (Ehlers, 2010).

The SEEQUEL Core Quality Framework is an outcome of the SEEQUEL (SEEQUEL, 2004) project, which was supported by the EU e-learning initiative, originated from the collaboration between the e-learning Industry Group (eLIG) with a number of European expert organizations and associations, co-ordinated by the MENON Network. The SEEQUEL Core Quality Framework is based on a matrix where a list of common quality criteria applicable to the whole e-learning experience can be weighted by the several users (people or organization), enabling any category of stakeholders to position their perception of quality with respect to the perceptions of another category of stakeholders. The SEEQUEL Core Quality Framework proposes a quality model with three main quality characteristics: the learning resources, the learning processes, and the learning context.

E-xcellence (EADTU, 2012) is a project that started in 2005, with the support of the eLearning Programme of the European Commission (DG Education and Culture), and in cooperation with 13 higher education e-learning and quality assessment and accreditation partners in Europe. An outcome of this project was an assessment tool (programme and institutional level) providing a set of benchmarks and quality criteria covering six main areas: strategic management, curriculum design, course design, course delivery, staff support and student support.

The report “Quality on the Line: Benchmarks for Success in Internet Based Distance Education” (Merisotis & Phipps, 2000) was commissioned by the National Educators Association and Blackboard, Inc., and prepared by the Institute for Higher Education Policy (IHEP). This report identifies 24 individual quality indicators, in seven main categories: institutional support, course development, teaching/learning, course structure, student support, faculty support, and evaluation and assessment. The study called each indicator a benchmark, but they are, in reality, attributes to indicate overall quality and so they are not measurable against other institutional results.

The Quality Matters Program (QM, 2011) is dedicated to quality assurance for online education, with a broad range of subscribers, including K-12 schools and higher education institutions. QM is a faculty-centered, peer review process that is designed to certify the quality of online and blended courses. There are three main components in the QM Program: The QM Rubric, the Peer Review Process and QM Professional Development. The QM Rubric framework is a set of 8 general areas and 41 specific criteria used to evaluate the design of online and blended courses. The general areas are: course overview and introduction, learning objectives, assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, accessibility. The framework is supported by a set of online tools to facilitate the evaluation by a team of reviewers.

In the UK, two bodies established quality assurance guidelines for their institutions: the Quality Assurance Agency for Higher Education and the Open and Distance Learning Quality Council (ODLQC). In the first one, e-learning guidelines are an extension of general learning guidelines. The ODLQC defines six main areas for quality assurance criteria (and possibly accreditation) as follows: outcomes, resources, support, selling, providers, collaborative provision (ODLQC, 2005).

These frameworks aim at the same global objective: the quality of e-learning environment/products. They present different perspectives but also many common issues. Some of them are more specific and related to the course and other are more global and related to institutional aspects. The EFQUEL (EFQUEL, 2011) and E-xcellence (EADTU, 2012) are the more detailed ones.

There are many researches done in the e-learning quality. Nevertheless, it is difficult to choose one model to support the development of a b-learning environment, since we needed to consider different perspectives and different levels of detail. At same time, it is also important to consider pedagogical issues that influence the success of any e-learning environment. As so, the main idea was to create a new reference that summarize the main dimensions and criteria referred in the existent quality frameworks and adds new pedagogical elements in the context of the quality in a b-learning environment.

In this work we collected and arrange all the quality criteria identified in the models referred above in order to get a more complete framework and determine if it fits our b-learning environment (blended learning in Portuguese higher education). We also included elements and pedagogical issues related to our own b-learning research and experience in Portuguese Higher Education, acquired during more than 10 years of experience.

A framework for measure the quality of a b-learning environment

There is not a consensus grouping the e-learning elements. The six models referred above present many ideas of quality grouped in different categories. Some of them referred the same element using a different label. Others grouped one or more elements in the same categories. In one model we can find elements in one category that are divided in different categories in another. Based on our own 10 years of experience and on the most common groups, in this work we considered the following categories: Institutional Aspects, Program and Course Design, Media Design, Technology and Evaluation & Review. These categories included the elements identified in each models studied as shown in the table below:

Table 1:

 

1. Institutional aspects

2. Program and course design

3. Media Design

4. Technology

5. Evaluation and review

EFQUEL – European Foundation for Quality in e-Learning[1]

X

X

X

X

X

SEEQUEL – Sustainable Environment for the Evaluation of Quality in e-learning[2]

X

X

X

X

X

E-xcellence[3]

X

X

X

X

X

IHEP – Quality on the Line[4]

X

X

X

X

X

QM – Quality Matters[5]

X

X

X

X

 

ODLQC – Open & Distance Learning Quality Council[6]

X

X

     

 

As a result of the analysis of the selected frameworks we identified the main quality areas, each one with a set of criteria, aiming the self-assessment of the b-learning courses as described below.

Institutional Aspects

The institutional aspects are related to the cultural organization and global elements that should be taken into account when we are preparing a b-learning environment/product. In different ways, all models studied refer institutional aspects. In general, present technologies should use to be innovate the learning process and to face nowadays demand.

Education and Technology Research

E-learning strategy should be a part of general educational strategy, should be embedded within teaching/learning strategy of the institution and widely understood and integrated into the overall strategies (EADTU, 2012). Faculty should provide incentives to innovative practices, to encourage development of distance, including rewards for effective teaching in distance learning (Merisotis & Phipps, 2000). Internal and external publication on teaching and learning issues related to e-learning should be encouraged and rewarded (EADTU, 2012). The institution should encourage and support participation in inter-institutional collaboration and exchange programmes related to teaching and learning development (EADTU, 2012). The institution should have an identified group of key staff responsible for formulating, evaluating and developing institutional e-learning policies. That policy should include the weights of blends and on the use of external environments and resources such as social networks. The institution should invest in the development of online assessment tools and techniques. It should have evidence of research and development of online assessment and the dissemination of these across the institution (EADTU, 2012). It is expected that learning design choices will vary with the subject and level of the course. Policies on research and scholarship in innovation in e-learning need to cover both technical and educational aspects. It should include policy for scheduling curriculum (face-to-face sessions, deadlines for assessments). It should consider the needs of the target audience. Learners should know why the physical attendance sometimes is required, for instructional reasons or identity reasons. Institution should have an effective mechanism to share knowledge and experience in the design of course content and consequent impact on students’ learning. The policy in infrastructures should include: financial, physical and technical resources; staffing and staff development; management, responsibilities and accountability (EADTU, 2012). Institution should investigate and explore emergent technology in the field of e-learning, should provide a framework of technical accessibility and presentational standards that apply to e-learning materials and systems. It should include the evaluation of the requirements needed, such as equipment purchase, software implementation, recruitment of staff, training and research needs, staff workload and technology developments. There should be an institutional plan for the provision of training in the technical aspects of e-learning. Staff development programs in online assessment should be provided. Information about how to use the institution’s e-learning system and services should be provided to all users in a logical, consistent and reliable way (EADTU, 2012). Process which has been considered successful and effective should be shared among staff and used to train new staff (SEEQUEL, 2004). Institution should provide training and support to staff as well as example materials, good practices and netiquette. Even on technical aspects in the course development (EADTU, 2012). Faculty members should be assisted in the transition from classroom teaching to distance instruction and be assessed in the process, including having training sessions before and during the online classes. Guidelines regarding minimum standards for course development, design and delivery should be provided. An analysis of training needs of the staff should be conducted in a regular basis (SEEQUEL, 2004). Documented technology plan should be in place to ensure quality standards. Faculty members should have access to written resource to deal with issues arising from students’ use of electronically-accessed data (Merisotis & Phipps, 2000). Institution should offer a e-portfolio service to assist students in recording evidence of their knowledge and skills development (EADTU, 2012) and should also provide a e-repository (repository of digital contents) (Peres & Pimenta, 2011). The institution should have a process for indexing and archiving its e-learning materials for evaluation and potential re-use (EADTU, 2012).

External providers

Medium and long term partnerships with learning and training providers should be established (SEEQUEL, 2004). Any provision delivered by two or more organizations should be covered by a written agreement which clearly specifies the respective rights and division of responsibilities. One of the organizations should be the leader and learners should know it (ODLQC, 2005; EADTU, 2012; SEEQUEL, 2004). Adherence to all relevant legal requirements, national or foreign should be ensured (SEEQUEL, 2004). Institutions should be aware of the national policies regarding recognition of qualifications in the country partners. Institutions should work closely with professional bodies in the development of online professional communities (EADTU, 2012).

Teams with peer review

The team must comprise content experts, instructional designers and technical experts (Merisotis & Phipps, 2000). e-Learning debates should be developed in order to collect and confront different points of view (SEEQUEL, 2004). Key stockholders should be involved in the program design (EFQUEL, 2011) and be consulted on a regular basis (SEEQUEL, 2004). The course should be developed by a faculty team with a peer review. The course must be approved through a broad peer review process (academic and technical aspects) (Merisotis & Phipps, 2000; EADTU, 2012).

Course design, development and evaluation should involve individual or team with expertise in both academic and technical aspects (EADTU, 2012). People responsible for the analysis, design, development, implementation and evaluation should be qualified (ODLQC, 2005) (EFQUEL, 2011). Responsibilities of different staff groups (teachers, tutors, etc.) involved should be specified and clear to learners. The roles of individuals with the project team should be well defined (EADTU, 2012). All tutors should have specific knowledge and competences to facilitate online courses/programmes, to attest this, a set of predefined tutoring skills should be used as a standard within the program (EFQUEL, 2011). A document of responsibilities of each intervenient should be produced (expertise on the subject, e-tutor, technical support, quality assurance manager, etc.) (EFQUEL, 2011; SEEQUEL, 2004). It should also include legal and ethical responsibilities (EADTU, 2012) and the information on how people and services can help students’ succeed (EFQUEL, 2011). Procedures for staff profile should be established and selection should be based on those competences. A system for recognizing the staff’s competencies should be in place. It is important to have exchange of agreements with other educational institutions for students’ virtual mobility providing e-learning programs and operability (EADTU, 2012).

Learning outcomes

The learning outcomes should be agreed between staff and learners (EFQUEL, 2011) (SEEQUEL, 2004) and written in the students’ perspective, using an action verb. They should reflect the level of performance students will achieve in a measurable form and should be related to the course program (EFQUEL, 2011). Learning outcomes should reflect both knowledge and skills to be developed (EADTU, 2012). The level of ability inherent in the outcomes should be matched to a national level of qualification (ODLQC, 2005; EADTU, 2012) and be placed in a wider educational, vocational & professional context (ODLQC, 2005). The institution should have a clear policy regarding the acquisition and assessment of core transferable skills, including e-skills. Courses, including their intended learning outcomes, should be regularly reviewed, updated and improved using feedback from stakeholders as appropriate (SEEQUEL, 2004; Merisotis & Phipps, 2000; EADTU, 2012). Beyond global learning outcomes it is also important consider the soft skills that refer to a transversal objectives such as:

  • SS1 – Learning to learn;
  • SS2 – Information processing and management;
  • SS3 – Deduction and analytical skills;
  • SS4 – Decision making skills;
  • SS5 – Communication skills, language skills;
  • SS6 – Teamwork, team based learning and teaching;
  • SS7 – Creative thinking and problem solving skills;
  • SS8 – Management and leadership, strategic thinking;
  • SS9 – Self-management and self-development (Peres & Pimenta, 2011).

Promotional and administrative activities

E-learning provider should maintain and demonstrate a strong commitment to educational value. It should adopt widely accepted norms of good ethical practices (ODLQC, 2005).

The provider should conduct all promotion activities in a fair and ethical manner, following the best practices and legislation. All promotional materials should give clear and accurate information. All enquiries from potential applicants should be handled promptly and appropriately, avoiding mis-selling (ODLQC, 2005).

Institutional policies, services and resources should be clearly stated. Students should be provided on how to access them (QM, 2011). The institution should have a credit transfer policy aligned with national system credit (EADTU, 2012). A system to recognize and accredit the learners’ prior competences and knowledge should be in place. The evaluation system should be able to measure to what extent informal learning meets the expected objectives and outcomes. A system for allowing credit transfer from informal to the formal settings should be foreseen. A process which helps learners reflect on the transferability of their acquired competences and skills to their specific context of use should be in place. The vision of the quality of learning and value system should be shared and agreed within the organization (SEEQUEL, 2004). Student interaction with faculty should be facilitated through a variety of ways (Merisotis & Phipps, 2000) even to deal with difficult situations (ODLQC, 2005). A structured system should be in place to address students’ complaints (Merisotis & Phipps, 2000). It is important to have a documented process to manage complaints. A person responsible for dealing with learners' complaints related to the programme, assessment, tutoring should be identified (EFQUEL, 2011). Enrolment when completed should be confirmed to the learner. A documented confirmation of outcomes should be available where students finish the course (ODLQC, 2005). The administrative impact of e-learning and b-learning systems on the workloads of all staff groups should be assessed and adjustments made as required (EADTU, 2012).

Information available

The potential students should have all information available (online, brochure, flyer, etc.) in order to orient their decision (EFQUEL, 2011; ODLQC, 2005). They should be introduced to the purpose and structure of the course (QM, 2011; ODLQC, 2005; EFQUEL, 2011). Students should have a clear understanding of all components and structure of the course (SEEQUEL, 2004). Students should be provided with clear course objectives, learning outcomes, concepts and ideas. Written information should be supplied to the student about the program (Merisotis & Phipps, 2000; EFQUEL, 2011). Students should be provided with a clear picture of what will be involved in using e-learning resources and the expectations that will be placed on them (ODLQC, 2005; EADTU, 2012). Beyond this, it is also important to give information related to technical requirements (minimum basic computer configuration, operating system, internet access, plug-ins, particular software, firewall access, etc.) (ODLQC, 2005; EFQUEL, 2011), pre-requisites of knowledge and competences (including technical skills) (EFQUEL, 2011; EADTU, 2012; QM, 2011; ODLQC, 2005), target group, variety of the methods, requirements of evaluation, timetable, workload, expectation about students participation in community of practice related to social and academic dimension. A document describing the characteristics of the target group should be produced (EFQUEL, 2011). Specific expectations about a minimum amount of time per week for study and homework should be in place (Merisotis & Phipps, 2000). All conditions relevant for the course should be explained to learners (ODLQC, 2005). It should be explained how the methodology approach (including technology and blended approach) leads to the achievement of the learning objectives (EADTU, 2012; EFQUEL, 2011). The contribution of e-learning components to the development of educational objectives needs should be clear (EADTU, 2012). The information of the contact of the person which responsible for the pedagogical and methodological issues, technical issues, for the tutoring and for the complaints should be given (EFQUEL, 2011). Learners should be able to discuss the suitability of the course with the responsible and know who will be responsible for the evaluation (internal or external agent) (ODLQC, 2005). If applicable, it should also refer the selection process and fees (EFQUEL, 2011), it should include the possible extensions to finish the course (ODLQC, 2005; EFQUEL, 2011). All information should be consistent and reliable. It should be also easy to access, update, coherent, consistent, etc. In addition a document to students with value system of the institution and students’ role, rights and responsibilities should be created.

Program and course design

Learning methods

Program and methodology should take into account professional context, previous experience, prior learning and allow self-directed learning. The course should allow the personalization of the learning path (EFQUEL, 2011).

The blending of learning methods (online, face-to-face, self-learning, tutor-facilitated, asynchronous, synchronous) should be appropriate and meet the needs and characteristics of learners (EADTU, 2012; EFQUEL, 2011). Should have a reasoned coherence between learning outcomes, the strategy for use e-learning, the scope of the learning material and the assessment method used (EADTU, 2012). A document which sets out the relationship between learning outcomes, learning activities and assessment should be produced. In a b-learning context there should be an explicit rationale for the use of each component in the blend (EADTU, 2012). The expectations on students regarding their participation in the on-line community of learners and for a minimum amount of time per week to study and do homework assignments should be clear both in general terms (for all applicants) and in relation to specific parts of their course or programme (EADTU, 2012; Merisotis & Phipps, 2000).

Learning objectives

Course’ learning outcomes should be clearly defined and aligned with learning objectives (QM, 2011; California State University (CSU) Chico, 2003). Each course should include a clear statement of learning outcomes in respect of both knowledge and skills. It is also important to ensure that all learning objectives are measurable and written from the students’ perspective. The learning objectives should be appropriately designed for the level of the course and students should have instruction on how to meet them (QM, 2011) the results and outcomes of the learning experience should be agreed between the staff and learners (SEEQUEL, 2004). The learning objectives specification process conducts the development of an important guide to be used both by teacher and students. The use of a taxonomy may facilitate the process such as Bloom Taxonomy (Peres & Pimenta, 2009). Bloom (Bloom & Krathwohl, 1956) suggests a taxonomy of learning objectives sorted out in six levels: Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation. The intention is to display the behaviours starting from the simplest to the more complex one.

Assessment & test

The specific and descriptive criteria of assessment should be provided for the students’ work and participation (QM, 2011; EADTU, 2012). The assessment of the course should be done in a formative and summative mode and should be appropriate to the curriculum design (EADTU, 2012) in a (face-to-face or in a distance mode). The assessment instruments selected should be varied and appropriated to the work being assessed (QM, 2011). Assignments and knowledge assessment tests should be aligned with the learning objectives, enabling the adequate measurement of their achievement (EFQUEL, 2011; ODLQC, 2005; Merisotis & Phipps, 2000; Peres & Pimenta, 2011) they should be aligned with the learning activities (California State University (CSU) Chico, 2003; Peres & Pimenta, 2011). In this way, the types of assessments selected should be able to measure the stated learning objectives and are consistent with course activities and resources (QM, 2011; Peres & Pimenta, 2011).

Ongoing multiple assessment strategies should be used to measure content knowledge, attitudes, and skills (California State University (CSU) Chico, 2003; Merisotis & Phipps, 2000). Assignments and/or knowledge assessment tests and tasks should be designed using different approaches, including multiple opportunities of self-assessment (QM, 2011; EADTU, 2012) and peer-review (EADTU, 2012; EFQUEL, 2011; California State University (CSU) Chico, 2003) and using adequate instruments (QM, 2011; EADTU, 2012). Appropriate measures should be in place to prevent impersonation and plagiarism, especially when assessment is conducted online (EADTU, 2012).

Assignments should be clearly formulated and adequately explained to learners. Learners should have a clear understanding of what they are expected to perform and how their performance will be measured.

The results of assessments should be communicated to students (ODLQC, 2005). During the formative evaluation, learners progress and achievements should be monitored and evaluated (EFQUEL, 2011; SEEQUEL, 2004; Merisotis & Phipps, 2000). The feedback should have an analytical approach and consider the way the solution was provided (EFQUEL, 2011), should be relevant and contain appropriated depth (EADTU, 2012).

Should have a specific timeframe to provide learners with feedback on assignments and knowledge assessments (EFQUEL, 2011; QM, 2011; Merisotis & Phipps, 2000). Regular feedback about student performance should be provided in a timely manner throughout the course (California State University (CSU) Chico, 2003; QM, 2011; EADTU, 2012).

It is also important to store and organize evidence and records of the results achieved within the system (ODLQC, 2005; SEEQUEL, 2004). The confidentiality of their records should be respected (ODLQC, 2005). The evaluation process should be used to improve the teaching/learning process (Merisotis & Phipps, 2000) and the evaluation system should be able to measure to what extent informal learning meets the expected objectives and outcomes (SEEQUEL, 2004).

Curriculum

Another concern should be on the curriculum design. The objectives of each module/unit should describe outcomes that should be measurable and consistent with the course-level objectives (EADTU, 2012; QM, 2011). They should be designed in order to include components that contribute to the development of outcomes. Students should be able to relate course contents to skills and learning outcomes defined to the course (EADTU, 2012). According to Peres and Pimenta (2009) after defining objectives and designing learning objectives assessment, we should establish the sequence of contents. This organization avoids the specification of learning objectives based on the contents. This scenario usually results in sentences such as “understand the content A” and in a lowest level of knowledge (first or second Bloom taxonomy level). Despite the importance of these levels, if the learning objectives consist in achieving a higher critical thinking level, it is important to explicit it on the objectives definition associated with analysis, synthesis and evaluation.

The content of the program should be organized in a logical sequence from the simpler to the most complex concepts (ODLQC, 2005; EFQUEL, 2011). The modules/lessons/units should be built progressively on each other (EFQUEL, 2011). The modules/segments should have a varying lengths determined by the complexity of the learning outcomes (Merisotis & Phipps, 2000). A written information about the program should be supplied to the student (Merisotis & Phipps, 2000). The modules should be introduced with their introductory elements such as: brief description, objectives, estimated amount of time required, eventual assessment (EFQUEL, 2011). The self-contained modules should be used to assess student mastery before moving forward in the course (Merisotis & Phipps, 2000). The curriculum should include research modules taking into account the skills and independence that will be demanded of students in conducting research remotely (EADTU, 2012).

The curricula should be designed in such a way that allows personalization for individual learning styles and needs and a flexible path for the learner (SEEQUEL, 2004; Merisotis & Phipps, 2000).

Learning influence factors (motivation)

Before starting the program, students should be advised about the program to determine if they have the self-motivation and commitment to learn at a distance. Assessment instrument should be used in order to determine the students’ learning styles (Merisotis & Phipps, 2000) and motivation. The learning methodologies should motivate learners to actively participate in the learning process (EFQUEL, 2011; SEEQUEL, 2004). Should valorize the learner’s self-esteem and competences among the learning community (SEEQUEL, 2004). The learning methods should take into account the balance between time to develop activities and complexity (Peres & Pimenta, 2011). One way to motivate students is to give class voice-mail and/or e-mail to encourage students to work with each other and their instructor(s) (Merisotis & Phipps, 2000).

Preparatory classes, additional learning materials, recommended reading as well as pedagogical guidance and other forms of support should be available during the course in order to bridge learning deficits (EFQUEL, 2011). In general the following main learning influence features could be identified: contextual analysis results; time definition; e-learning strategy complexity; nature of subject in study; personality, ways and individual learning styles; previous experiences, knowledge and culture (Peres & Pimenta, 2009).

Learning activities

The workload demanded by the course should be realistic regarding the objectives, curriculum and according to the characteristics of target group, including full time job occupation (EFQUEL, 2011). The design of an instructional strategy should conduct to the learning success, individual or in group according to pedagogical models. Many pedagogical views may be used to support the instruction planning. The selection doesn’t have to be exclusive, it is possible to use more than one pedagogical approach. The choice of the pedagogical model should consider the moment of learning. At the beginning of the subject study, it is important to make sure that students are getting the basic knowledge (behaviourist and cognitivist theory). Then, it is important to consolidate it and promote the self-learning based on previous experiences (constructivist theory). At the end, it is important to promote a deep learning by social interaction (social constructivist theory). This learning path should be aligned with learning objectives (Peres & Pimenta, 2009).

At beginning of learning activities, it is important to provide the self-introduction by the instructor and students (QM, 2011). A learner centred learning design should facilitate the development of the desired skills and expected competencies described in the learning objectives (EFQUEL, 2011; California State University (CSU) Chico, 2003; EADTU, 2012). Students should have clear instructions on how to start and where to find various information of course components (QM, 2011). Opportunities for online publications and peer review should be provided (EADTU, 2012).

Learning activities should also provide opportunities for interaction in order to support active learning (QM, 2011). Social and collaborative activities should be included in the program methodologies and contribute to the achievement of the learning objectives, it includes, for example, peer review, group work, discussion board, (EFQUEL, 2011) or problem-solving group activities (Merisotis & Phipps, 2000). At any time, student interaction with other students should be facilitated through a variety of ways (Merisotis & Phipps, 2000). Learning activities should enable participation in academic community and contact with external professionals. The institution should provide mechanisms for students to participate in active communities of professional practices in order to stimulate a critical attitude (EADTU, 2012). The requirement for student interaction should be clearly articulated. Institutions should provide an online community for student-student and student-teacher interaction and make their policies available (EADTU, 2012). Etiquette expectation for online discussion, email and other forms of communication should be clearly stated (QM, 2011). To support communities of learners the activities should be designed in order to offer an appropriate use of asynchronous tools (e.g. discussion forums, wikis, blogs, social networking sites) and synchronous tools (e.g. video-conferencing, real-time chat) (EADTU, 2012). In spite of the objectives defined, higher education courses should provide multiple activities that help students to develop critical thinking, problem-solving skills (California State University (CSU) Chico, 2003) analysis, synthesis, and evaluation (Merisotis & Phipps, 2000). The most important is to develop learning activities in order to promote the achievement of the stated learning objectives (QM, 2011; Peres & Pimenta, 2011) and to be relevant to professional practice, including case studies, practical examples, good practices and real-life examples (EFQUEL, 2011).

Courses, and learning activities, should be designed with a consistent structure, easily understandable for students from various learning styles (Merisotis & Phipps, 2000) and offer multiple visual, textual, kinaesthetic and/or auditory activities to enhance student learning and accessibility (California State University (CSU) Chico, 2003). The description of learning activity should include the objectives, pedagogical models, subject/community, title and general description, tools, e-contents, activity phases, division of labour, rules and results (Peres & Pimenta, 2011).

The learning experience should be built in a flexible manner so as to ensure its contextualization and relevance to the learner’s context (EADTU, 2012; SEEQUEL, 2004). The flexibility should be also in terms of time, place and pace (EADTU, 2012). The instruction path should be students’ centred, trying to promote an inclusive environment that explores the student’s differences and the openness of the present digital network.

The course and learning activities instructions should be linked to a description of the technical support and institutional accessibility polices and services (QM, 2011).

Learning process and eTutoring

The course should offer ample opportunities for interaction and communication: student to student, student to instructor and student to content (California State University (CSU) Chico, 2003). The tutor should maintain and demonstrate a clear commitment to help learners achieve their educational goals. Nevertheless, learners should be responsible for their own learning and be informed of the tutor support. Students should be encouraged to complete their courses (ODLQC, 2005). Access to tutors should be provided on a regular and sufficient basis, known to both tutors and students. Tutor should be able to use a variety means to interact with learners (e-mail, forum, VLE tools, etc.) (EADTU, 2012). A plan to support the interaction and collaboration within the learners community (SEEQUEL, 2004) and with other students (Merisotis & Phipps, 2000) should be in place.

During the learning process, the tutor should track and monitor the behaviour of the users (SEEQUEL, 2004) and provide guidance and accompany to the learners. Tutoring should offer opportunities for learners to determine their own learning pace. Despite the deadlines that must be established, learners should be able to control their own path throughout the program. Tutors should provide learners with timely expert advice on course issues or materials and feedback on assignments (EADTU, 2012). Learners should be also informed of the timeframe (EFQUEL, 2011), the certain period of time to grade and return all assignments (Merisotis & Phipps, 2000; QM, 2011). During the course development the tutor should provide timely, accurate, helpful feedback to learners on tasks/activities (Merisotis & Phipps, 2000; EFQUEL, 2011), non-threatening and appropriated to the level of the course (Merisotis & Phipps, 2000; ODLQC, 2005), using measures for monitoring (SEEQUEL, 2004) and the various learning styles of students should be considered (Merisotis & Phipps, 2000), whenever possible in a personal basis (ODLQC, 2005).

Through the learning activities learners should be encouraged to consider and use higher thinking skills and to view issues from different perspectives (EFQUEL, 2011).

During the learning process, students should be instructed about the proper methods of effective research, including assessment of resource validity and to help them use electronically accessed data (Merisotis & Phipps, 2000).

When there are one or more tutors, steps should be taken to ensure that tutor support is consistent (ODLQC, 2005). Directions should be provided as to how students can participate in a broader academic community (EADTU, 2012). Students should have access to support services including technical help desk, administrative support and course choice advice (EADTU, 2012).

Learning materials/resources

Regarding to materials, it is important to guarantee that each learning unit is supported on the elements needed to guide learners in achieving the learning objectives (EADTU, 2012; QM, 2011; EFQUEL, 2011). The purpose of instructional materials and how they should be used for learning activities should be clearly explained (QM, 2011). They should be structured to facilitate individual study and the development of study skills (ODLQC, 2005). They should be current, present a variety of perspectives on the course contents (QM, 2011), appropriated to the learners’ need, knowledge, and experience (ODLQC, 2005; EFQUEL, 2011). Contents should be relevant and clearly presented, build on and reinforce prerequisites concepts and skills. Introduce, assess and reinforce new concepts and skills, should be logically structured and sequenced (EADTU, 2012). The contents should be sufficient and not in excess. They should be challenged, centred on the student, be relevant for students’ life, allow the interaction, tell a story with emotion. Little pieces of content are better than a more extensive one. The language should be simple and include visual elements. Any noise should be deleted. It is important to capture students’ attention in the beginning, trying to evolve with contents.

Resources should be adequately balanced regarding cognitive load and presented in a sub-divided form, in a logical sequence, without extra information. The course material should promote collaboration among students (Merisotis & Phipps, 2000; EADTU, 2012) and students-to-content. Independent learning materials should provide learners with regular feedback through self-assessment activities or tests. The availability, function and purpose of independent learning materials should be clearly defined and communicated to students. Self-paced materials should incorporate extensive embedded testing of learning outcomes (EADTU, 2012).

It is also important to ensure that contents are provided in a flexible manner, allowing different learning paths (EFQUEL, 2011), customization and personalization to individual learning styles (SEEQUEL, 2004). Media rich contents should be utilized with specific purpose (EFQUEL, 2011). This can include videos, tutorials, interviews with specialists, scenario based learning, games, etc.

Sufficient library resources should be available to the students (Merisotis & Phipps, 2000). At the same time, students should be provided with hands-on training and information to help them in seeking material through electronic databases, interlibrary loans, government archives, news services, etc.

Concerning the modules/lessons/units it is important to provide a glossary of terms associated to the learning materials and available from any part of the course (Merisotis & Phipps, 2000). A distinction between compulsory and recommended study/reading materials should be made (QM, 2011). The bibliography should be commented.

The materials should be effective and not contain significant errors of facts, misleading or out-of-date information, concepts and approaches (ODLQC, 2005). Contents should be inclusive, respect cultural diversity and gender sensitive (EFQUEL, 2011). It is also important to produce a lesson plan, including the learning objectives, reference to the contents and learning activities (face-to-face or online) (Peres & Pimenta, 2011).

Media Design

Accessibility

Accessibility standards should be considered in the design of the course. The course should contain equivalent alternatives to auditory and visual contents. Course should employ accessible technologies and provide guidance on how to obtain accommodation (QM, 2011) (EFQUEL, 2011). The course design should accommodate the use of assistive technology that can help students with disability (QM, 2011; EADTU, 2012) e.g. with respect to keyboards, touch screens, screen-readers (and “talking books”) or speech recognition. Learning material should be accessible and usable via a variety of devices including mobile devices.

Usability

The course design should facilitate readability and minimize distractions (QM, 2011). The layout of the course should be clear and free of unnecessary elements. The size and type of font utilized should be comfortable for reading. The images, illustrations, tables and other visual elements should be easy to read (EFQUEL, 2011). Course should be designed with a consistent structure easily discernible to students from various learning styles (Merisotis & Phipps, 2000). Interfaces used in the technical design of course should conform to up-to-date usability and accessibility standards. Font, text, placement and presentations should be consistent (EADTU, 2012). All interfaces should be functional and attracted to students.

Navigation

The course should be well-organized and easy to navigate, the aesthetic design should present and communicate the course information clearly throughout the course. All web pages should be visually and functionally consistent throughout the course (SEEQUEL, 2004).

The navigation through the mandatory learning materials should allow learners to know about their progress and position in relation to the overall content, identifying the unit, module, lesson, part of a unit, etc. (EFQUEL, 2011). Navigating throughout the online course should be intuitive, consistent and easy to navigate.

Printable

All screens, tables of contents and learning materials, including additional sources should have a printable version (EFQUEL, 2011).

Cultural diversity

Materials should be neutral as to sex, ethnicity, age and related issues (EFQUEL, 2011).

Copyright

All images, graphics, illustrations should be copyright free (EFQUEL, 2011). All resources and materials used in the course should be appropriately cited (SEEQUEL, 2004; QM, 2011). Learning materials should comply with legal requirements, copyright issues should be identified and documented (EADTU, 2012).

Download

The materials available for download should take into account reasonable standards of time for download, regular formats and forms of compression (EADTU, 2012; EFQUEL, 2011).

Technology

Server and Applications

The technical infrastructure that maintains the e-learning system should fit the purpose and support both academic and administrative functions. Technical infrastructure should be well defined and support institutional e-learning objectives. Institution should set standards for the operation of its technical infrastructure that are benchmarked against other major online customer service providers (EADTU, 2012).

The tools (learning management system and other tools) used in learning strategies and collaborative learning should be in accordance with the information technology infrastructure available, with the target group equipment and connectivity, learning skills and needs, staff teaching skills, learning objectives, assignments and other activities (EFQUEL, 2011). The selection of the tools should be based on the learning outcomes and objectives (QM, 2011) (Merisotis & Phipps, 2000). A system for learners profiling should be in place. The diversity and identity of each individual learner should be guaranteed (SEEQUEL, 2004).

The course technology should be current and students should readily access the technologies required in the course (QM, 2011). Search functions should be available for forums discussions, blogs, etc. whenever such tools may be utilized (EFQUEL, 2011).

Security and performance

Appropriated operating and security standards for all aspects of provision of online services should be defined. Measures should be in place for system recovery in the event of failure or breakdown (EADTU, 2012).

Electronic security measures, such as backup procedures, should be in place to ensure the integrity and validity of information (Merisotis & Phipps, 2000; SEEQUEL, 2004). The virtual learning environment should runs on an adequate server, which guarantees its stability. The course should be tested on various browsers and operating systems before launching (EFQUEL, 2011). Learning management system should be integrated with the management information and administrative system (EADTU, 2012).

Information such as system recovery, key performance indicator of system availability, download time, queuing time for access, etc. should be provided. Monitoring the patterns of the use of the system by students and staff should be a source of information for improvement in pedagogical as well as technical issues (EFQUEL, 2011).

Support

Support for building and maintaining the distance education infrastructure should be addressed by a centralized system. Easily accessible technical assistance should be available to all students. Technical assistance in course development should be available (Merisotis & Phipps, 2000). Students should be confidante while using virtual learning environment, even people with disability (EFQUEL, 2011).

Evaluation & Review

Periodically review

A feedback procedure for assessing effectiveness should be implemented (SEEQUEL, 2004). The provider should be committed to the continuous improvement. A process for integrating the recommendations for improvement should be foreseen as a part of the programme (EFQUEL, 2011). Procedures to ensure the quality, effectiveness and relevance of the material should be developed on a regular basis. It is also important to evaluate the accessibility and effectiveness of the resources’ usage. Procedures in order to assess the effectiveness and relevance of the learning materials should be developed (SEEQUEL, 2004). Course materials, including the intended learning outcomes should be regularly reviewed, updated and improved using feedbacks from stakeholders as appropriate (Merisotis & Phipps, 2000; EADTU, 2012) to ensure clarity, utility and appropriateness (Merisotis & Phipps, 2000).

The program’s educational effectiveness should be measured using several methods. The results should be used to improve the teaching/learning process. Specific standards should be in place to compare and improve learning outcomes. Data on enrolment, cost, and successful/innovative uses of technologies should be used to evaluate the program effectiveness (Merisotis & Phipps, 2000).

The performance of the e-learning systems should be monitored and opportunities for performance improvement identified. Performance of mentors, tutors and moderators should be monitored regularly. Problems and issues should be acted upon promptly. Longer term improvements should be identified (EADTU, 2012). On the formative evaluation we should answer questions such as “do the activities cover all learning objectives?”; “are there activities that are not covering any objectives?”; “do students have all the necessary information?” etc. (Peres & Pimenta, 2011).

Collected data

A questionnaire developed specifically for the program should be used in order to assess overall quality and appropriateness of:

  1. Course design (methodology used, pedagogical approach, navigation, structure of the course)
  2. Course management (schedule and workload, grading policy, e-tutor performance, ability to engage learners, accuracy and timeliness of feedback, guidance and advice provided, collaborative activities versus individual activities, assignments)
  3. Course content (accuracy and relevance of learning content, learning objectives, knowledge assessment tests, case studies examples, relevance of discussion, additional resources, course documentation, guide, course syllabus, flyers and information provided to learner prior to delivery.
  4. Course media and technical support (registration process, access and user friendliness of learning in the platform, download time) (EFQUEL, 2011).

Data on pattern of students use on e-learning environment may be gathered and analyzed, ‘learning analytics’, in addition to evaluations information’s from survey activity (EADTU, 2012).

Final Report

To evaluate the quality and overall coherence of the course in order to develop further improvement, it is important to get learners’ feedback that could be collected through questionnaires or other means. An evaluation report should be prepared, in which feedback is analyzed and converted into clear recommendation for improvements in future programme design. The final report should include the learners’ course evaluation. The report should include the following topics:

  1. Course design
  2. Course management
  3. Course content
  4. Course media and technical support (EFQUEL, 2011).

Evidences and results achieved should be organized and stored within the system (SEEQUEL, 2004). An evaluation of cost-benefits should be made.

Conclusions and Future Work

More and more institutions are offering courses at distance. This might be a good solution to overcome some difficulties (e.g. time and distance) but it is necessary to promote a reflection about the quality of these practices as well as of the b-learning product offered. Moreover, the success of a course also depends on the expectations of students and in the ability of the educational institution to meet them. The information, concepts and procedures here presented give support to teachers and instructors, which intend to validate the quality of their blended learning courses. The framework developed helped to identify the areas to be analyzed and reflected upon. As a result of this research, the following elements arose in the analysis of the quality of a blended learning environment: Institutional Aspects (education and technology research, external providers, teams with peer review, learning outcomes, promotional and administrative activities, information available), Program and Course Design (learning methods, learning objectives, assessment & test, curriculum, learning influence factor, learning activities, learning process and e-tutoring, learning materials/resources), Media Design (accessibility, usability, navigation, printable, cultural diversity, copyright, download), Technology (server and applications, security and performance, support), Evaluation & Review (periodically review, collected data, final report).

In the future, in order to get more accurate results, we will propose an evaluation methodology based on a relative scale to weigh main areas and associated criteria, a [0..1] scale, where 0 means that area is not important for the research and 1 means of maximum importance. The total weight of all main areas must be 1. Similarly, the total weight of all the criteria of a main area must be 1. Each criterion has its own score for evaluating the quality level also in a [0..1] scale. The evaluation is organized as a tree-like structure of quality characteristics and the weights are determined using pairwise comparisons, adapted from Analytic Hierarchy Process (AHP) approach (Saaty, 2008). Unlike the way of assigning a number from a fixed scale with an arbitrary unit to the weights of each area/criterion, the measurements are not fixed but depend on each other and on the context of the course and its objectives. We think this methodology will facilitate future benchmarking within others courses. The final quality index Q is obtained by the bottom-up iterative aggregation of the scores as in AHP. First the score for each main area Ai is calculated as the sum of the products (Si * Wci), where Si is the score for criterion i and Wci is the corresponding weight. The final quality index Q is calculated as the sum of (Ai * Wai), where Wai is the weight for main area Ai.

References

  1. Blakely, G.; Skirton, H.; Cooper, S.; Allum, P. and Nelmes, P. (2009). Educational gaming in the health sciences: systematic review. In Journal of Advanced Nursing, 65(2), (pp. 259-269). doi: 10.1111/j.1365-2648.2008.04843.x
  2. Bloom, B. and Krathwohl, D. (1956). Taxonomy of Educational Objectives: The Classification of Educational goals by a Committee of College and University Examiners. New York: Longmans.
  3. BS-8426. (2003). BS 8426:2003 A code of practice for e-support in e-learning systems. BSI.
  4. California State University (CSU) Chico. (2003). Rubric for Online Instruction. Retrieved from http://www.csuchico.edu/roi
  5. EADTU (2012). Quality Assessment for E-learning: a Benchmarking Approach – Second edition. European Association of Distance Teaching Universities.
  6. EFMD (2010). EFMD CEL Introductory Guide. Retrieved from http://www.efmd.org/images/stories/efmd/downloadables/EFMD_CEL_Quality_Brochure_SinglePages.pdf
  7. EFQUEL (2011). UNIQUe guidelines. Retrieved from http://cdn.efquel.org/wp-content/blogs.dir/5/files/2012/09/UNIQUe_guidelines_2011.pdf
  8. Ehlers, U.-D. (2010). Open ECBCheck – Low cost, community based certification for E-learning in Capacity Building. InWEnt, European Foundation for Quality in E-Learning.
  9. Graham, C. (2004). Blended learning systems: Definition, current trends, and future directions. In C. Graham & J. Bonk (eds.), Handbook of blended learning: Global Perspectives, local designs. Pfeiffer.
  10. ISO/IEC 19796-1. (2005). ISO/IEC 19796-1 Information technology – Learning, education and training – Quality management, assurance and metrics – Part 1: General approach. ISO.
  11. ISO/IEC 19796-3. (2009). ISO/IEC 19796-3 Information technology – Learning, education and training – Quality management, assurance and metrics – Part 3: Reference methods and metrics. ISO.
  12. Merisotis, J.P. and Phipps, R.A. (2000). Quality on the Line: Benchmarks for Success in Internet-Based Distance Education (Policy Report). IHEP.
  13. NP-4512. (2012). NP 4512:2012 Sistema de gestão da formação profissional, incluindo aprendizagem enriquecida por tecnologia – Requisitos. IPQ.
  14. ODLQC. (2005). Open and distance learning quality council: standards in open and distance learning.
  15. PAS 1032-1. (2004). PAS 1032-1 Learning, education and training focussing on e-learning – Part 1: Reference model for quality management and quality assurance – Planning, development, realisation and evaluation of processes and offers in learning, education and training. DIN.
  16. Pawlowski, J. M. (2006). Adopting quality standards for education and e-learning. In Handbook on Quality and Standardisation in E-Learning (pp. 65–77). Springer Berlin Heidelberg. Retrieved from http://link.springer.com/chapter/10.1007/3-540-32788-6_5
  17. Peres, P. and Pimenta, P. (2009). MIPO model – A framework to help the integration of web technologies at the higher education. In K. Terry (ed.), Online Education and Adult Learning: New Frontiers for Teaching Practices. IGI Global (Information Science Publishing).
  18. Peres, P. and Pimenta, P. (2011). Teorias e práticas de b-learning. Edições Sílabo.
  19. QM. (2011). Quality Matters Rubric Standards 2011 - 2013 edition. Maryland Online, Inc. Retrieved from https://www.qualitymatters.org/layout-1/download/QM%20Standards%202011-2013-4.pdf
  20. Saaty, T. (2008). Relative measurement and its generalization in decision making. Why Pairwise Comparisons are Central in Mathematics for the Measurement of Intangible Factors The Analytic Hierarchy/Network Process. In Revista de la Real Academia de Ciencias Exactas, Físicas y Naturales. Serie A: Matemáticas (RACSAM), 102(2), (p. 251).
  21. SEEQUEL. (2004). Quality Guide to the non-formal and informal Learning Processes. Scienter-MENON Network.
  22. UNE-66181. (2012). UNE 66181:2012 Gestión de la calidad. Calidad de la formación virtual. AENOR.


[1] EFQUEL: http://efquel.org

[2] SEEQUEL: http://www.menon.org/wp-content/uploads/2012/11/SEEQUEL-eLearners-user-guide1.pdf

[3] E-XCELLENCE: http://e-xcellencelabel.eadtu.eu

[4] IHEP: http://www.ihep.org/assets/files/publications/m-r/QualityOnTheLine.pdf

[5] QM: http://www.qmprogram.org/about

[6] ODLC: http://odlqc.org.uk

 

Tags

e-learning, distance learning, distance education, online learning, DE, higher education, blended learning, ICT, information and communication technology, internet, collaborative learning, LMS, learning management system, constructivism, interaction,

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.