Evaluating e-learning courses for continuing professional development using the Conversational Model:
A Review of UNIGIS

Dr. Charles Buckley, Senior Lecturer/International Co-ordinator, Sport Studies (bucklec@hope.ac.uk) and
Karl Donert, International Fellow (donertk@hope.ac.uk) Liverpool Hope University College, England
Correspondence address: Dr C. Buckley, Sport Studies, Liverpool Hope University College,
Taggart Avenue, Liverpool L16 9JD. (http://www.hope.ac.uk)

Abstract

UNIGIS is a distance learning course designed specifically for students working in the area of geographical information and is now represented by a worldwide network of educational institutions. This study aimed to assess the pedagogical orientation of the programme, the usefulness of Laurillard's Conversational Model as an evaluative tool and, student perceptions of the course in relation to their expectations for continuing professional development (CPD). An integrated evaluation strategy was used combining analysis of 23 on-line questionnaires, interviews with 14 students and 5 tutors together with an analysis of e mail messages sent amongst UNIGIS students over a period of twelve months. The course goes some way to encouraging student discourse and interaction yet the scope and organisation of the mail-base does not develop a true feeling of community. The Conversational Model focuses on the individual learner and interaction between tutor and student. It provides a suitable framework to assess how well a system integrates and organises people involved in different roles within the learning environment. It can act as a useful tool for evaluating the process of learning, in particular the potential of course materials for encouraging a reflective and considered approach to learning. Students were very positive about most features, especially the core materials, workshops and flexibility in course structure. However, they requested smaller self-contained units of learning materials which could be integrated into their work schedules for immediate application and certification.

Keywords:

UNIGIS, e-learning, pedagogical orientation, continuing professional development, evaluation, Conversational model.

Introduction

Geographical information systems (GIS) is a computer system capable of assembling, storing, manipulating and displaying geographically referenced information (Blake et al., 2003). It has diverse applications and is used in a range of industries including facilities and land management, utility companies, transportation, farming and forestry. Whilst traditional courses in geography and GIS will continue to be a major part of university provision, the most likely growth area will be in lifelong learning courses offered through distance learning (Britain, 1999). Web-based distance and flexible learning environments are enabling universities to reach increasing numbers of postgraduate students for continuing professional development (Nunes and McPherson, 2003). Such programmes can ensure accessibility and relevance to learning needs aligned with career goals. Within the field of geography there have been efforts to increase access to computer-based teaching resources such as the Deliberations Geography Project within the JISC Electronic Libraries Programme and the Teaching and Learning Technology Programme (TLTP). The increase in use of digital resources has been accompanied by concern by government and other funding bodies both in the UK and US about the impact of such developments (Blake et al., 2003). This concern has been reflected in initiatives such as a recent programme funded by the Joint Information Systems Committee (JISC) to consider the way in which teaching and learning may be transformed using IT and digital resources and the quality of new distance learning programmes (Barbera, 2004). Evans and Nation (2000) argue that even in traditional universities, the exploitation of the Web is requiring that staff change their teaching towards more constructivist approaches: "The better versions of these approaches…", argue Evans and Nation (2000:175), "…such as that articulated by Laurillard, and based on plausible and sensible reflexive theories of human endeavour are becoming influential; as guiding lights at least". Garrison and Anderson (2003) also believe that quality should follow constructivist principles in an integrated way, teaching students and providing them with more independence in terms of learning. The constructivist approach encourages student autonomy and initiative and critical dialogue between teacher and student and amongst students themselves (Sahin, 2003).

UNIGIS

The UNIGIS course has evolved in response to changes in education and educational technology. It comprises of "a network of universities co-operating in the design and delivery of distance learning in GIS" (UNIGIS Brochure, 2004). There are currently 15 universities in 11 countries which contribute to the programme. The course emphasises student-centred learning using the WWW, downloadable study packs, computer based learning materials, core texts, workshop and help lines. Interviews with tutors involved in the design and delivery of the course revealed that the materials and philosophy were strongly influenced by initiatives at the Open University and the Conversational Model of Laurillard (1993). There were deliberate attempts to make the material adaptive and self-assessment exercises were incorporated to help students adopt a reflective approach. There is a three-year MSc programme, two year Postgraduate Diploma and one year Postgraduate Certificate. Assessment is by coursework and dissertation for students completing the MSc. Distance learning is complimented with optional residential workshops. The core units and choice of electives are designed to allow the course to be tailored to in-service needs.

Continuing professional development in geographical information systems

Peters (1998) has described distance education as the most industrialised form of learning and teaching. The vocationalist view of learning (e.g. Wain, 2000) holds that employers want higher education to more closely attend to generic skills or competencies which provide transformative potential. That is, employers want new graduates to be able to transform their organisation, not merely enhance its productivity and competitiveness along current lines. There has recently been a move towards developing a learning organisation (Pedler et al., 1997) in which training goals and education goals merge. In Western Europe, argue Jolliffe et al., (2001), industry is demanding better access to courses for professional training and continuing education (European Commission, 2004). There is also a growing trend towards the joint development by partners from the pubic and private sectors to develop training programmes using open and distance learning. The number of distance and off-shore MBA's and similar credential courses now offered by universities, colleges and commercial training bodies indicates that professional workplace learning is changing rapidly (Sacchanand, 2000; Munro and Rainbird, 2002). The role of higher education is becoming much wider and more complex and includes the need to give students the ability to apply skills in the context of the specific field or industry sector (Nunes and McPherson, 2003).

Kennie (1998:156) believes that the key features of effective CPD are that it is:

Continuous- "throughout the practitioner's working life"
Professional/organisation focused- "necessary for the execution of professional and technical duties" and related to "maintaining the quality and relevance of the professional services"
Broad based- "knowledge and skills and the development of personal qualities"
Structured-"systematic maintenance, improvement and broadening"

Kennie's (1998) explanation of CPD emphasises the diversity of the concept, the importance of the development of both skills and knowledge and the necessary links between education and training. It also reflects the current moves in higher education to competence-based education and links between universities and the private sector. Kennie and Enemark (1998), for example, have argued that there is a need for a clearer link between CPD and organisational strategy in view of recent changes in the business environment.

The Association of Geographical Information (AGI) is the principle organisation concerned with continuing professional development in geography. They have defined CPD as "the systematic maintenance, improvement and broadening of knowledge and skill, and the development of personal qualities necessary for the execution of professional and technical duties throughout one's working life" (AGI, 2001:1). The AGI argued that CPD should be a cyclical process rather than a linear one. Participants should review and plan their needs, undertake development activities, and then sign off the whole project.

Despite the emergence of distance learning courses for professionals in GIS there appear to be few which provide substantial training or education in developing personal skills. This is incongruent with trends in industry towards the desirability amongst employers for ever more flexible and self-confident individuals with skills in communication, problem analysis and problem-solving, planning and networking (Kakabadse and Korac-Kakabadse, 2000). Research by Nabi and Bagley (1999) and Lange et al., (2000) suggests that some current educational programmes are not meeting the demands of employers' expectations. Such transferable skills are an important part of CPD courses offered by the AGI. UNIGIS strives to encourage the development of transferable skills in a sector which is arguably characterised as having a highly technical and computer driven focus.

Pedagogical approach and use of learning technology

Cloete (2001) stressed that traditional learning paradigms are not necessarily suitable in e-learning environments. It is vital, emphasised Cloete (2001) that educators and institutions planning to embark on the development of e-learning systems have a clear and accurate understanding of the capabilities, limitations and influences of these technologies. She argued that creative approaches and competent strategies to manage these limitations at the instructional design and user levels, as well as integration into other systems, need to be established and understood in order to ensure a degree of quality comparable with that of distance learning.

Swayne (2001) argues that those designing courses and materials for users in the field of GIS need to develop user interfaces appropriate for that community of users as opposed to many existing systems which are designed for information technology professionals. Although uptake of new technology in the field of education has not kept pace with the commercial sector, there has been a definite shift in pedagogical approaches to learning and teaching requiring a change of paradigm in educational philosophy. Nunes and McPherson (2004) have argued that both the design team and tutors need to be aware of the appropriate pedagogical approaches in order to maximise the benefits for students in ICT based learning environments.

The Conversational Model was pioneered by Laurillard (1993) has its roots in conversation theory developed by Pask (1976). Laurillard (1993) traces the need for dialogue in the learning process back to the Socratic method of philosophical enquiry. Pask's (1976) conversation theory provides an alternative approach to education and training, which is based upon the eliciting of distinctions and agreements in formalised and exteriorised conversation. Conversation theory arises from a concern to promote enlightenment and an acceptance of the problematic nature of both language, and the notion of the human individual. Pask's (1976) work is fundamental to any discussion of dialogue and simulation environments in that he pioneered the thoughtful use of computers in education.

Britain (1999) has argued that Laurillard's (1993) model is suitable for analysing the pedagogical orientation of courses as, "her approach has led the way in looking at how learning technology could be employed to promote more effective and varied teaching styles" (p.3.) Britain (1999) argues that an effective way of using the model for evaluation is to incorporate the four central principles associated with the Conversational Model, i.e. reflective, adaptive, interactive and discursive.

Reflective: The teacher must support the process in which students link the feedback on their own actions to the topic goals for every level of description within the topic structure.
Adaptive: The teacher has the responsibility to use the relationship to determine the focus of the continuing dialogue.
Interactive: The students must act to achieve the task goal. The teacher must provide meaningful intrinsic feedback on the actions that relate to the nature of the task goal.
Discursive: Teachers' and students' conceptions should each be accessible to one another, they must agree learning goals, teachers should provide an environment in which students can act on, generate and receive feedback.

Barbera (2004) emphasises that evaluation of virtual learning has tended to focus on the use of business quality models which are not suitable for educational environments; user satisfaction which has only partial relevance for assessing quality and the multimedia systems and design of materials. More importantly, argues Barbera (2004) is the need to focus on the teaching and learning processes. Evaluation material should consider relations that are established between instructor and students, and the dialogue that comes from this practical community (Harasim et al., 1995). Laurillard's model encourages the researcher to consider the nature of the learning and teaching environment and extent to which students and tutors are involved in meaningful interaction and dialogue.

This study set out to assess the perceptions of students studying the UINGIS course, in particular, the extent to which they felt it matched the principles of the Conversational Model (Laurillard, 1993) and whether it matched their expectations for continuing professional development in the GIS industry.

Method

The use of method triangulation is desirable in assessment of organisational infrastructures (such as UNIGIS) as it provides for the analysis of processes and decision-making, implementation and change characteristic of dynamic organisations (Gummesson, 2000). Course assessment and evaluation is an area which has received very little attention and there are consequently few guidelines for the researcher. This study incorporated an integrative evaluation strategy recommended by Draper et al., (1996) and The Teaching with Independent Learning Technology Programme (2001). The guidelines contained within such an approach necessarily involve an integration of a number of data collection methods and forms of data analyses. This method provides a comprehensive coverage of features, pedagogical orientation and potential weaknesses in course design. Research consisted of:

A content analysis of questionnaire and interview data was used to identify themes, concepts and meanings in the data using code categories recommended by Burns (2000).

Student perceptions of UNIGIS

Communication and a sense of community

UNIGIS strives to achieve a sense of community amongst its user group. There is the e-mail base which allows for students to interact and form working groups for completion of assignments and sharing of ideas. Some assignments are collaborative. Communication is also facilitated through students writing their own web page which provides others with a photograph, profile of their skills, interests and brief cv. Each student has a member of staff acting as a support tutor for the intake who contacts them by e-mail. There are occasional on-line synchronous discussions although these are infrequent. Students were asked to rate the features incorporated into UNIGIS and were mostly positive rating them as at least satisfactory/good. Day (2000) points out that because computer technologies such as e-mail, synchronous chat, group discussions and class Web pages might be new and unfamiliar to some students; these students face a steeper learning curve and may encounter technological glitches. In this way, they may be harsher critics than those in face to face learning situations. Interviews with UNIGIS students revealed that for many, the use of a virtual learning environment and authoring web pages was a new skill.

It is interesting to note that opportunities for human interaction featured as some the most desirable aspects of the course. The most highly rated aspect of UNIGIS was the core materials, the support of the Administrative Office, the introductory meeting at the start of the course and Easter workshops. The Easter workshops in particular were seen as an opportunity to ask questions in a supportive environment to clarify issues and concerns that they had stored during their study. 'They are a chance to meet with people who you normally only hear about on the mail base. It gives me a chance to ask questions and share some of my ideas with other students on the course'. (UNIGIS user, 4.7.2001). This was particularly the case for students whose first language was not English. Many students thought that a combination of e learning and face-to-face teaching was the most effective way of organising courses for continuing professional development. Within GIS, with its demand on complex technical skills, having opportunity to ask questions directly to tutors to get help with practical problems was seen as being essential. Language can be perceived as a barrier to effective integration and communication within any system. It has been argued that whilst UNIGIS is successful at the institutional level, it has progressed only in a limited way in terms of the integration of departments across international boundaries (Personal communication, UNIGIS tutor, 18.5.2001). UNIGIS mailbase messages from those whose first language was not English tend to be rare and very brief; they were also confined to the interactive category and were usually responses to requests for collaborative group work on assignments.

Continuing professional development and UNIGIS

Students were positive about e learning courses as an effective means of facilitating their continuing professional development. Many of the students expressed a preference for short modules which could be accumulated over periods of time. This would better facilitate fluctuations in amount of pressure they faced at work, would make carrying their qualification from one job to another easier and provide short-term goals which were perceived as easier to achieve, whereas courses which were extended over longer periods were viewed as daunting. There is some flexibility afforded to students in relation to the deadlines for the return of assignments This was an aspect of the course where students were very positive in particular, the flexibility afforded to return of assignments, informal style of conversations with tutors and office staff through e-mail and telephone and opportunity for collaboration with peers were thought to be positive features. Users found modules relevant and felt that the course was adaptable although some asked for more choice of modules.

UNIGIS is designed as an integrated course to be studied whilst in employment, this suited students who felt that the opportunity to implement concepts learnt during the units with immediate effect at work was preferable to sabbatical courses. UNIGIS users expressed a preference for integrated rather than sabbatical courses, many were in senior positions and felt that organising time away from work to problematic for continuing development; integrated courses were seen as offering the potential for the immediate application of skills in the workplace. The perceptions of the cohort in this study fit with Shaw and Green's (1999) analysis that CPD tends to be fragmented and control lies with employers. Respondents in this study gave the impression of having limited control in how CPD was organised and felt unsure about the possibility of it being continuous in a work culture characterised by short term contracts and greater mobility within job markets. It appears that, as Kennie and Enemark (1998) suggest, there is the need for a clearer link between CPD and organisational strategy.

Students ranked communication skills as most important above personal, teamwork and analytical skills as an essential aspect of courses for continuing professional development, especially for those with a responsibility for teaching or supporting others in the process of learning GIS and the complex concepts associated with its implementation. Most students (89%) wanted to see an equal balance between the development of professional competences and more organisational and technical skills. They stressed the importance of courses providing examples of applications of skills in real life situations similar to those they would experience in their workplace.

Students felt that certification was very important for improving career opportunities and promotion, although they felt that generally employers did not rate this as highly. More than half of those studied were members of a professional organisation that had a continuous professional development scheme, the most common organisation being the AGI, 41% of respondent belonged to this institution.

Pedagogical underpinnings of UNIGIS

In this case study the conversational framework was been found to be useful for assessing the academic learning environment and interactions between the tutors and users. It provides insights into the extent and meaningfulness of interactions between tutors and students and between students themselves. Laurillard's (1993) model allows for the investigator to assess the extent to which users have control over aspects of the learning process and the responsiveness and potential for adaptation of those involved in administering and designing the course. It also provides a useful framework for evaluating the process of learning, in particular the potential of course materials for encouraging a reflective and considered approach to learning. It has great potential as an evaluative tool, particularly in the area of assessing students' thoughts on the quality of feedback and the procedures in place for developing learning. The model allows for assessment of the opportunities provided for student reflection based on feedback from self-assessment exercises and tutor feedback during the course and for assignments. One of the limitations of the conversational framework is that it only represents conversations between individuals and thus is not adapted for collaborative learning. It also fails to reflect functionality associated with managing groups of learners.

The questionnaires and interviews asked students if they believed that UNIGIS encouraged them towards the four principles contained in Laurillard's (1993) model, that is the extent to which the course was interactive, adaptable, discursive and reflective. In addition, the analysis of messages in the mailbase was used to supplement students' comments.

Laurillard (1993) points out that the teacher must support the process in which students link the feedback on their own actions to the topic goals for every level of description within the topic structure. UNIGIS incorporates self-assessment exercises as a formative activity throughout the course. The self-assessment exercises guide students to relevant resources and, in this sense also, the availability of learning material is partly dependent on students following the recommended guidance offered by course materials. Most thought that the self-assessment exercises were potentially useful for developing confidence in being able to tackle the formal assessments set during the course although most also admitted to not making effective use of the exercises, both in frequency, and the way in which they incorporated the results into future assessments. Students were critical of the delayed return of assignments and lack of structure in the feedback process. Metacognitive messages were those relating to issues based on analyses of knowledge and skills and/or elements of planning, evaluating and reflecting on the use of skills. These types of messages formed 23% of the total analysed indicating that students, for a significant proportion of their course students are required to actively reflect on learning experiences. However, the trend for metacognitive messages was greater between tutors and students than students with students. Interestingly, metacognitive messages predominated in May, June, July and November and were often focused on issues associated with assignments.

The UNIGIS brochure emphasises that students will be joining an "extensive GIS community" with representatives in over 30 countries. Rainbow and Sadler-Smith (2003) have highlighted the importance of there being ample opportunities for students to interact with fellow students in computer aided learning environments and explore learning in greater depth under the guidance of tutors and lecturers. There is a core of individuals who use the mail base frequently, the most regular users being tutors and staff in the UNIGIS UK Office. Regular users are not confined to a particular type of message and cross all categories within Henri's 1992 model. This indicates that a desire and/or confidence to use the mail base is a determining factor in frequency of use. The mailbase was found to contain many interactive and social messages, however, certain individuals tend to dominate. A total of 632 messages were sent during the twelve-month period. The busiest month for correspondence was January (151), followed by October (115), February (99) and November (77). The quietest periods were July (10), August (15) September (18) and May (18). More than half of these (61.4%) were from UK users. UK users seem to become less active in proportion to users based outside the UK in July and August. For all other months except December, they form more than half of the correspondents. There is a core of individuals who use the mail base frequently, the most regular users being tutors, and staff in the UNIGIS UK Office. Regular users are not confined to a particular type of message and cross all categories. This indicates that a desire and/or confidence to use the mail base is a determining factor in frequency of use. The following table (Table 1) shows the relative distribution of types of message sent over the twelve month period.

Table 1 Distribution of type (using Henri's, 1992 model) and number of messages sent to the UNIGIS e-mail base between May 2000 and April 2001

  May June July Aug Sept Oct Nov Dec Jan Feb Mar Apr
Total messages 18 25 10 15 18 115 77 44 151 99 25 35
Total UK messages 17 18 4 5 18 70 44 21 93 59 18 21
Social 6 8 5 0 4 40 30 33 74 41 6 13
Interactive 9 13 2 3 13 90 62 21 118 82 14 22
Cognitive 2 2 0 5 14 56 31 1 69 62 8 20
Metacognitive 6 12 4 5 6 37 37 4 58 29 9 4

Tutors were active in responding to user requests for information in both cognitive and metacognitive categories yet there were very few instances of tutor student interaction in any other category. The lack of contact with support tutor reinforced comments in responses to how discursive users thought the course had been. Students were critical of the amount of interaction with course tutors reinforcing findings by Mooney and Martin (2003) who found that learners thought that interaction with course designers was inadequate. In addition, the feeling of a lack of communication was exacerbated in that some felt that the mail base needed reorganising. The system tends to be used by students and staff as a forum for sharing of thoughts about the course, assignments, job opportunities, queries about software and applications. Some users thought that there needed to be a filter system for the e-mail base to allow for easier access to more relevant information. Some reported feeling threatened by the scope of the e mail base and which inhibited free expression and sharing of ideas because of the fear of being thought of as ignorant for example, "The idea of the UNIGIS discussion group is good, but I often feel embarrassed to ask questions, which may be very simple ones, to my group" (UNIGIS user, 30.5.2001). It is obvious however, that many communications take place outside the mail base which is used as the starting point for forming research groups. A separate discussion list for each intake was also suggested by many users. 'The mail base provides a good forum for professionals asking professional questions but it has too large a membership to make a good forum for course related queries. Perhaps a specific newsgroup for each intake should be set up'… (UNIGIS user, 25.5.2001).

The e-mail base allows students to interact and form working groups for completion of assignments and sharing of ideas. Some assignments are collaborative. A chat facility and regular opportunity for synchronous learning opportunities would raise the extent to which UNIGIS can be seen as actively monitoring students. Students are encouraged, through collaborative assignments to share learning experiences. This carries over into some voluntary organising of group learning for other aspects of the course although this is limited to individual e-mails. The design of Web pages at the beginning of the course also allows for a certain degree of self-organisation and sharing of materials. There are few opportunities for synchronous interactive chat. There is a student contribution section in the Resource Centre which can be considered as an opportunity for students to organise themselves for learning.

UNIGIS is a tailor-made course which represents one of its major strengths. This allows tutors to modify the course easily in light of student comments. There is plenty of opportunity for students to comment on the design, administration and organisational structure of the course which can be subsequently adapted.

Conclusion-the importance of evaluation

Evaluation of distance learning courses is crucial, particularly in the light of current trends towards courses which are dynamic and granulated to offer variety to cater for multiple user needs (Holly and Rainbird, 2000). The growth of various forms of online education has also raised concerns over the state of quality of virtual education environments (Barbera, 2004). Most models of instructional design include evaluation as part of their process (e.g. Dick and Carey, 1990; Jolliffe et al., 2001). There is still a relatively limited body of research evaluating the effectiveness of online learning (Petracchi and Patchner, 2001) and some contradictions have emerged (Harvey, 2002). Britain (1999) argues that amongst the factors that are slowing the uptake of learning management systems in higher education is the lack of a coherent framework within which to evaluate both the pedagogical benefits and organisational changes required to effectively implement it. Henri and Pudelko (2003) also stress the need for evaluative models to be able to take account of processes of participation (communication and action) and reification (use and production of intermediary objects). There have been several models proposed by researchers for the evaluation of programmes which are delivered electronically for example: The EMAR Model, (Nunes and McPherson, 2003); The CIAO! Framework, (Open University). Laurillard's Conversational Model (1993) encourages researchers to evaluate the properties, capabilities and pedagogical orientation of virtual learning environments. Jackson (2003) has argued that evaluation in the field of the use of IT in education is complex and that the most effective means might be a focus on user perceptions rather than on hard data. Henri and Pudelko (2003) emphasise that research on virtual learning communities and the associated evaluative criteria need to take into account their social context of emergence, their given goal, their evolutionary aspects as well as the activity they carry out. In addition they should assess the effectiveness of interactions between teacher and student. This study did not focus on either gender or cultural influences which are likely to have an impact on students' perceptions of online learning. Evaluative tools should incorporate a facility for acknowledging such differences (Murchu and Sorensen, 2004).

UNIGIS is an example of a course which has developed from a paper-based to on-line distance-learning system. It has evolved substantially and recruits large numbers of students from around the world. Whilst there is a mail base the level of communication and interaction between users is limited and students, for the most part work, independently on the materials. Users are positive about most elements within the course although a recurrent criticism is lack of communication with tutors and peers, even though most admit that they do not initiate this communication often enough. Resourcing is a crucial factor in determining the level of interaction that can take place within a learning and teaching environment and UNIGIS has a well-established and reputable infrastructure which could be enhanced greatly by seizing future opportunities in the area of e-learning.

References

Association of Geographical Information (2001) CPD (http://www.agi.org.uk/public/gi-life/frames.html). Last accessed 20/8/2001.

Barbera, E. (2004) Quality in virtual education environments British Journal of Educational Technology 35 (1): 13-20.

Benigno, V. and Trentin, G. (2000) The evaluation of online courses Journal of Computer Assisted Learning 16 (3): 259-270.

Blake, C. T., Davies, C., Jones, A., Morris, E. and Scanlon, E. (2003) Evaluating complex digital resources ALT-J 11 (1): 4-16.

Britain, S. (1999) A framework for pedagogical evaluation of virtual learning environments. JISC Technology Applications Programme. University of Manchester http://www.jtap.ac.uk (Accessed 25th June 2001).

Burns, R.B. (2000) Introduction to Research Methods London: Sage.

Cloete, E. (2001) Electronic education system model Computers and Education 36: 171-182.

Day, M. (2000) Teachers at the crossroads: Evaluating teaching in electronic environments. Computers and Composition 17: 31-40

De Vries, L., Naidu, S., Jegede, O. and Collis, B. (1995) On-line professional staff development: an evaluation study. Distance Education 16, (1): 157-173.

Dick, W. and Carey, L. (1990) The Systematic Design of Instruction (3rd Edit.) New York: Harper Collins.

Draper, S.W., Brown, M.I., Henderson, F.P. and Mcateer, E. (1996) Integrative evaluation: An emerging role for classroom studies of CAL. Computers and Education 26 1-3 17-32.

European Commission (2004) La formation professionnelle continue en Europe,
http://europa.eu.int/comm/education/programmes/leonardo/new/leonardo2/cvts/cvts_fr.pdf, Last accessed 31/3/2004.

Evans, T. and Nation, D. (2000) Understanding changes to university teaching. In: T. Evans and D. Nation. Changing University Teaching: Reflections on Creating Educational Technologies. London: Kogan Page (pp. 160-175).

Garrison, D. and Anderson, T. (2003) E-Learning in the 21st Century: A Framework for Research and Practice London: Routledge Falmer.

Gummesson, E. (2000) Qualitative Methods in Management Research (2nd Edition.) London: Sage.

Harasim, L., Roxanne, S., Teles, L. and Turoff, M. (1995) Learning networks: a field guide to teaching and learning online. Massachusetts Institute of Technology, Cambridge.

Harvey, L. (2002) The end of quality. Quality in Higher Education 8 (1): 5-22.

Henri, F. (1992) Computer conferencing and content analysis. In A.E. kaye (Ed.) Collaborative Learning Through Computer Conferencing Berlin: Springer-Verlag. (pp. 117-136)

Henri, F. and Pudelko, B. (2003) Understanding and analysing activity and learning in virtual communities Journal of Computer Assisted Learning 19: 474-487.

Holly, L. and Rainbird, H. (2000) Workplace Learning and the Limits of Evaluation. In: H. Rainbird (2000) Training in the Workplace London: Macmillan 264-282.

Jackson, P. (2003) Ten challenges for introducing Web-supported learning to overseas students in the social sciences. Active Learning in Higher Education 4, (1): 87-106.

Jolliffe, A. Ritter, J. and Stevens, D. (2001) The Online Learning Handbook: Developing and Using Web-Based Learning. London: Kogan Page.

Kakabadse, A. and Korac-Kakabadse, N (2000) Leading the pack: future role of IS/IT professionals. Journal of Management Development 19 (2): 97-155.

Kennie, T. J. M. (1998) The growing importance of CPD. Continuing Professional Development (1): 154-163.

Kennie, T.J.M. and Enemark, S. (1998) The growing importance of CPD. Continuing Professional Development 1 (4): 156-167.

Lange, T., Ottens, M. and Taylor, A. (2000) SME's and barriers to skills development: a Scottish perspective. Journal of European Industrial Training 24 (1): 5-11.

Laurillard, D. (1993) Rethinking University Teaching: A Framework for the Effective Use of Educational Technology London: Routledge.

Nabi, G. R. and Bagley, D. (1999) Graduates perceptions of transferable skills and future career preparation in the UK Education and Training 41 (4): 184-193.

Nunes, M. B. and McPherson, M. (2003) Action research in continuing professional distance education. Journal of Computer Assisted Learning 19: 429-437.

Nunes, M. B. and McPherson, M. (2004) The role of tutors as a integral part of online learning support. European Journal of Open and Distance Learning www.eurodl.org

Mooney, K. and Martin, A. (2003) The potential of distance e-learning in the spatial information sciences-An evaluation of a pilot programme at the Dublin Institute of Technology. International Conference on Network Universities and e-Learning 8-9 May 2003 Valencia, Spain. http://www.hsh.no/menu/ Last accessed 16th March 2004

Munro, A. and Rainbird, H. (2002) Job Change and Workplace Learning in the Public Sector: The Significance of New Technology for Unskilled Work, New Technology, Work and Employment, 17: 224-235.

Murchu, D, O. and Sorensen, E. K. (2004) Online master communities of practice. Collaborative learning in an intercultural perspective. European Journal of Open and Distance Learning www.euorodl.org

Pask, G. (1976) Conversation, Cognition and Learning. London: Elsevier.

Pedler, M., BurgoynE, D. and Boydell, J. (1997) The Learning Company: A Strategy for Sustainable development. (2nd Edition) Maidenhead: Mc Graw Hill.

Peters, O. (1998) Learning and Teaching in Distance Education: Analyses and Interpretations from an International Perspective. London: Kogan Page.

Petracchi, H. and Patchner, M. (2001) A comparison of life instruction and interactive televised teaching: a two-year assessment of teaching an MSW research methods course. Research on Social Work Practice 11 (1): 108-117.

Rainbow, S. W. and Sadler-Smith, E. (2003) Attitudes to computer-assisted learning amongst business students and management students British Journal of Educational Technology 34 (5): 615-624.

Sacchanand, C. (2000), Workplace learning for information professionals in a changing information environment, 66th IFLA Council and General Conference, Jerusalem 13-18 August, http://www.ifla.org/IV/ifla66/papers/109-136e.htm, Last accessed March 31 2004.

Sahin, T, Y. (2003) Student teachers' perceptions of instructional technology: developing materials based on a constructivist approach British Journal of Educational Technology 34 (1): 67-74.

Shaw, M. and Green, H. (1999) Continuous professional development: emerging trends in the UK. Quality Assurance in Education. 7 (3): 169-176.

Swayne, D. A. (2001) Design principles for environmental information systems Environmental Modelling and Software. 16 (5): 417.

Teaching with Independent Learning Technology Programme (2001) Overview of the TILT Project TILT - Teaching with Independent Learning Technologies - University of Glasgow - home page. Last accessed 29/5/2001.

UNIGIS Brochure (2004) http://unigis.org. Last accessed 20/2/2004.

Wain, K. (2000) The learning society: Postmodern politics. International Journal of Lifelong Education 19, (1): 36-53.