On-line Course: From Experimental Use to Integration into a University Programme

Lea Bregar 1, Mojca Bavdaž1, Irena Ograjenšek1,, Marko Papić2, Janez Bešter 2
CEES Project Homepage:
1 University of Ljubljana, Faculty of Economics, Kardeljeva ploščad 17, 1000 Ljubljana, Slovenia
2 University of Ljubljana, Faculty of Electrical Engineering, Tržaška 25, 1000 Ljubljana, Slovenia

© EURODL 1999

CEES Development Methodology
CEES Software support
- Integrated On-line Distance Education System (IODES)
- CEES Navigation System
- CEES Functional Tools
Pilot Course Delivery
Students’ Evaluation of the Course


The on-line Course on European Economic Statistics (CEES) is an example of the application of the modern information and communications technology (ICT) with the goal of increasing the quality and efficiency the learning process. Three strategic areas of the ICT application in the course development process are identified: extension and deepening of student’s access to information, increased flexibility of the study delivery and integration of student’s learning experiences and knowledge. The paper outlines in detail the course development methodology, both from the viewpoint of the contents and the technological support. Additionally, the results of the pilot course delivery’s evaluation are presented and commented upon.

Key words: open/distance learning, economic statistics, official statistics, Internet, web-based course, virtual class



Development of high-quality economic statistics, based on common standards that link national statistical systems, is one of the prerogatives for efficient functioning of the European Union. The survey of available courses on business and economic statistics, carried out at the University of Ljubljana’s Faculty of Economics in August 1997, showed that Eurostat has been paying considerable efforts to appropriate training of professional statisticians in the EU and CEE countries. However, the initiative of training wider (non-professional) audiences (e.g. students of business and economics, students of social sciences, government officials, researchers and analysts both in private and public sector etc.) has been left to the countries themselves. The topics dealing with official statistics have been seldom included in formal non-accredited study programmes within the formal educational system.

To fill in this gap, the Course on European Economic Statistics (CEES) was developed at the Faculty of Economics in Ljubljana/Slovenia (FE) with the help of a consortium of partners. The consortium also included Faculty of Electrical Engineering from Ljubljana/Slovenia, Faculty of Economics and Business Administration from Sofia/Bulgaria and the Training of European Statisticians (TES) Institute from Luxembourg. The main objective of the CEES project was the development of an original course module on economic statistics for non-statisticians at a higher educational level, taking into account recent developments of Eurostat statistics and deploying information and communication technologies (ICT) to increase the quality of the learning process. A specific course development methodology was applied for effective integration of ICT and course contents. Software support, developed originally by the Faculty of Electrical Engineering for open distance learning, was adapted to specific CEES course requirements.

As a result of studying the CEES course topics, students should:

  • be able to understand the role of harmonisation and standardisation of official statistics world-wide and in the EU;
  • know where to search for statistical data needed in the analysis and how to access various national and international sources on different media;
  • know how to analyse statistical data and present the results of statistical analysis;
  • be able to recognise and understand both the contents and the socio-economic implications of officially published statistical data;
  • be able to identify cases of improper use and/or misuse of official statistics in everyday life.

The course development phase of the CEES project was finished in the beginning of 1999. The pilot course delivery was carried out in February and March 1999 at the University of Ljubljana/Faculty of Economics and at the Faculty of Economics and Business Administration in Sofia/Bulgaria as a part of a regular second year undergraduate study programme.

In the paper, the methodology underlying the CEES course development and the supporting software are presented. The attention is also given to the results of the pilot course delivery’s evaluation, focusing mainly on the analysis of problems accompanying the transformation of a traditional university course into the on-line mode.


CEES Development Methodology

The course on economic statistics, focused on the appropriate use of large scattered decentralised databases (diversified by various aspects, e.g. by students, contents, type of data, media etc.), can by its very nature profit substantially from the creative use of ICT. In our view, the use of ICT can increase the quality and efficiency of the course in three strategic areas:

  • extension and deepening of students’ access to information;
  • increase in the flexibility of the access, exploration and use of official statistics;
  • integration of students’ learning experiences and knowledge.

For effective integration of ICT and CEES course contents a specific course development methodology was applied.

When embarking on development of the on-line study course, our main challenge was to design a methodology which would assure the fulfilment of study objectives of complex, comprehensive and hierarchically structured contents (inherent in the course on economic statistics) in the flexible, open and non-linear environment offered by hypermedia. It is pretty obvious that students of on-line study courses do not have to follow a course in a set manner (e.g. from page 1 to page 2 or from chapter A to chapter B etc.) as they are obliged to do in a traditional learning process. Instead, they are free to roam around the application on their own, to explore available semantic links and already established logical relationships among different topics, to try designing their own hierarchy of studied issues etc. Therefore the danger of phenomena ‘to be lost in cyberspace’ lurks all the time.

In order to keep the advantage of hyper-media non-linear study possibilities and to fulfil the requirements of the learning objectives of the CEES, a specific course development approach was developed. It is based on the course contents modularization combining linear and non-linear study paths and supported by a range of navigational and functional tools.

Course modularization approach decomposes the fundamental hierarchically built structure of a traditional course into homogeneous topics. Each topic contains several blocks. Blocks embody course contents (in our case Overview, European Standards, National Application) or pedagogic support with guidance to additional sources of information (Objectives, Activities, Resources, Links). The breakdown of the course by topics and blocks leads to a matrix presentation of the course, which helps course developers to create nodes and links in a consistent, transparent and meaningful way.


Figure 1: Matrix Presentation of the CEES Course

Each cell in the matrix represents a basic html unit. If necessary, some cells can be further divided into new html pages or can be left empty. Each html page can contain several types of links: explanations and cases within an html page (in the so-called “pop-up” window); internal links (links to pages within the course) and external links (links to other web-sites).


CEES Software Support

Adaptation of the software support according to the specific CEES course requirements necessitated thorough and deep co-operation between the course contents developers and the software experts.

To comply with the course requirements, the CEES course developers simply couldn’t use any of the commercial products already available on the market. Not only are they marked by their functional disadvantages, most of them also offer a rather low and/or expensive technical support. Therefore, the course developers welcomed the technical support offered by the Laboratory for Telecommunications at the Faculty of Electrical Engineering in Ljubljana, Slovenia. The support included:

  • provision of the specially adapted software for the WWW-based distance learning;
  • adaptation of the html course contents pages prepared by course developers to the requirements of the supporting software;
  • technical tutorship to the students during the pilot course delivery;
  • consulting and assistance to the course content developers in all phases of the course development project.


Figure 2: Technical support during all steps of course development

In the following part of the paper we will focus on the presentation of the software features which make CEES a modern educational tool, based on an up-to-date concept of distance education and efficient use of information and telecommunication technology.


Integrated On-line Distance Education System (IODES)

CEES supporting software is based on the Integrated On-line Distance Education System (IODES), which was originally developed at the University of Ljubljana, Faculty of Electrical Engineering, Laboratory for Telecommunications.

The primary objective of the system implementation was to facilitate the use of ICT in the teaching process while at the same time enable tutors support of students and guarantee mechanisms for monitoring the success of individual students and of the entire educational system.

The distinctive features of the IODES are the navigation system and the IODES functional tools.


CEES Navigation System

The efficiency of the learning process in the hyper space largely depends on the relevance of the navigation system. CEES offers the user three basic navigation tools:

  • suggested study path;
  • multilevel table of contents (ToC);
  • study map.

Suggested study path, created initially within the IODES, is the first and the most important way of navigation which offers maximal flexibility to the course developer or tutor and provides students with the guided access to course contents. Course developer or tutor suggests a certain study path through the course pages that students should follow in order to gain the knowledge they are supposed to. By clicking navigation buttons + (plus) and – (minus) they move onwards or backwards on the suggested study path. They can visit any of the existing internal or external hyper links on the path as there is a button called ‘back to the path’, which brings them back to the course page that is currently on the path. In case they were visiting external links, they have two options. Either they click ‘back to the path’ or ‘back to the course’ which brings them to the last already visited course page. This very user-friendly and effective way of navigation thus combines advantages of a traditional way of navigation (students are used to reading pages from the study book successively) and advantages of the hyper space (random access to external information).

Multilevel table of contents (ToC) is based on the matrix presentation of the course structure. It is presented on the left side of the student’s workspace. Its first level consists of topics’ titles only. By clicking on one of the topics students access its second level which comprises all blocks of the selected topic. By selecting one of the blocks, they proceed further to the selected html page. Different colours are used to inform the student about the current status of the selected page (whether it is an already visited page, a currently visited page or a page on the suggested study path).

Multilevel table of contents can be combined with the tool called the study profile which was specially created for CEES within IODES. By marking selected areas in the course matrix, study profile option allows students to choose the topics and/or blocks they would like to study. Compared to the multilevel ToC, this option gives them even more flexibility with regard to the selection of the preferred course contents.

Study map navigation is based on the graphical presentation of the hierarchical course structure. Coloured buttons indicating titles of different course levels (chapters, subchapters or topics and html pages or learning units) are used to make this navigation tool as user-friendly as possible. By clicking one of them, students directly access the selected course page.

Of course, during each study session, students have the possibility of combining all ways of navigation presented above.


CEES Functional Tools

Navigational tools facilitate students’ access to the relevant information while functional tools help to increase the delivery process flexibility in terms of place, time and pace of study. Additionally, they help integrating students’ learning experiences and knowledge with available information resources; in other words they support students’ active learning and knowledge creation.


Figure 3: Functional tools

Instant communication between a tutor and a student is one of the most important features of distance learning. IODES offers possibility of writing e mails directly to tutor (to the previously given e–mail address).
Communication between students is possible through a news group. One or more news groups can be active simultaneously. Students can contact the newsgroup via e-mail, download their assignments, exchange comments etc. If possible, the news group should be guided by the tutor to maximise its effects.
Notice board is the third way of communication. This one way communication tool is used to inform students about events concerning the study process (e.g. assignments deadlines, dates of face-to-face meetings etc.). More information about the course and other related issues can be found in the study office tool, accessible in the study tools menu.
Videoconference is an optional communication tool. Limited by the amount of data that can currently be transmitted via Internet, the so-called IP videoconference is of a very low quality and can not be of real assistance in communication or replace a face-to-face contact in the study process. However, videoconference implementation is legitimate in a corporate or in an ISDN environment.

The next group of tools offered by IODES are the study tools.
The tool called progress offers an overview of the students’ progress through the on-line course pages. Statistics such as number of pages already visited, percentage of the course still to be completed, etc. can be displayed and analysed.
After completing the course, students can evaluate the quality of the course contents, available tools, support and other important features of the distance education process through an evaluation form.
The importance of the study profile tool was already explained. Additional function of this tool is the possibility to reset parameters of the student’s progress through the course pages (resulting in a change of his/her history, e.g. in an adjustment of his/her current position on the suggested study path to reflect his/her own study or repetition needs).
With the personal profile tool, students have the option of changing their personal data.
Finally, other tools menu offers a comprehensive search engine through all CEES keywords with distinctively written search results. A calculator is also available. Technical terms and keywords used in the course are thoroughly explained in the glossary. There is a link to the FE library, giving students access to library’s catalogues. A tool called Q-mark contains a database of self-assessment questions. IODES also offers students a possibility of writing notes for each course page. Notes can be created and edited with the notes editor.

When adapting IODES for CEES, the basic objectives were to make it easy to use and give students a round-the-clock technical support. Apart from a tutor who can be contacted via e-mail at any time, a help engine exists within the IODES. Additionally, a special bug report and technical questions form was created to make the communication between users and technical experts as smooth as possible.


Pilot Course Delivery

The course development phase of the CEES project was finished in the beginning of 1999. The CEES course in its present form is an integral university degree course consisting of about 300 pages of text split into 15 topics (or 315 html pages with about 1500 external and internal hyperlinks) in three languages (English, Slovene and Bulgarian), available on different media (in print, on a CD-ROM and on-line). The course can be delivered in different formats thanks to its flexibility in terms of the media used, modularised structure of contents and different types of support embedded and thus applicable for different target groups.

During the course pilot implementation two delivery formats were tested: a traditional distance education class and a virtual class.

The CEES pilot course delivery at the University of Ljubljana/Faculty of Economics started with a joint kick-off meeting where the course objectives were presented, two different delivery formats were described, and an introductory lecture on economic statistics was given.

The students were invited to sign up for the virtual class (10 students). Consequently, a selection was made among them with the goal of working only with the best and the most motivated ones. All other 115 students were assigned to the traditional distance learning class, including those without or with problematic access to Internet, those with insufficient computer skills or simply those without courage to cope with the new, different way of studying that requires a much more independent and active learning style.


Figure 4: Pilot delivery scheme

The students in the traditional distance learning class were divided into smaller study groups. They participated in five face-to-face tutorials, the professor’s face-to-face consultations and in two computer workshops. Their main required study material was the pilot version of the textbook ‘Economic Statistics with European Standards’ (the printed version of the on-line textbook). An extensive presentation of the on-line textbook was given at the first computer workshop, where the students also learned about other Internet resources. These resources were explored at the second computer workshop in order to find statistical data needed in the assignments. The assignments consisted of four short activities and focused on the interpretation of data and ratios. At the end of the two-month course the students had to pass a written exam. Their final grade was a combination of the assignment grade (10%) and the exam grade (90%).

The ten students in the virtual class started their course with a face-to-face workshop where they were given instructions on the use of the internet resources and CEES on-line course. A timetable was proposed to students with the seminar papers submissions deadline as the only constraint. Meetings with the tutor responsible for software issues and two group consultations with the professor were their only face-to-face forms of communication. However, for issues concerning the course contents they had constant tutors’ and professor’s support via e-mail.

The students from the virtual class had to prepare three research seminars of a much larger extent compared to the assignments of the traditional distance learning class. The first assignment (2000 words) was a comparative study of web-sites of two statistical data providers. The second assignment (2500 words) was a statistical-methodological analysis and the third one (2500 words) was a statistical analysis of a selected economic problem. As a final event, the students presented their papers to and discussed them with their colleagues. The professor participated in the discussion, extending it also to relevant related topics to find out whether the students understood the underlying concepts and thus achieved the CEES learning objectives.


Students’ Evaluation of the Course

In the last week of the pilot course delivery a survey was carried out among the students of both study groups using two different questionnaires. Both questionnaires covered the same issues (study materials, support and requirements, comments, overall impression), but they are not directly comparable because of the specific features of each delivery format (e.g. printed versus on-line textbook etc.).

61 students of the traditional distance learning class were included into a survey, which is more than a half of the whole population. The survey revealed that as many as 92% of the students were satisfied with the textbook quality judged by its graphical appeal, writing style, the extent and selection of topics, the adaptation to the demands of the independent learning and the quality of activities and assignments.

Most students missed the keys to activities. They would also appreciate some additional examples with detailed explanations and exam questions with keys.

A little more than half of the students think that just the right number of tutorials and computer workshops was organised, while the others think that this number should be higher. Among the advantages the students emphasised small study groups that support the development of personal relations with the tutor and more active participation of all group members. They also believe that an important part of the course content was explained and discussed during the sessions. They were satisfied with the possibility of independent learning (which is very important for those already working). Among the disadvantages they emphasised a too fast pace of the topics delivery (explanations given in a haste) and the lack of time for the proper conduct of activities and assignments. However, they admit the lack of participation from their part which is a consequence of their being more used to the role of passive listeners.

The survey in the virtual class included eight students. They thought that the user interface did not take too much time to load, was very well designed and user friendly, allowed a simple and comprehensive navigation and offered a lot of internal and external links and an interesting set of supporting tools.

The students gave the highest grade to the selection of topics covered in the course and the external links satisfied even the most critical of them. They found the text graphically very appealing and the writing style very fluent. They also highly graded the study map with its colourful and transparent course structure (although it might take too long to load in case of a slow on-line connection) and a pre-set pop-up window for the tutor’s e-mail. Good working atmosphere finally resulted in highly graded communication with the professor and tutors.

The students were the least (but still quite) satisfied with the search and help functions which have been upgraded since pilot delivery. They evaluated the study profile concept as excellent but were not satisfied with the graphical appearance (in their opinion it is not transparent enough). They also missed more detailed initial instructions for the assignment preparation.

The students emphasised the time consumption: the study of the subject took them a lot of time. They felt that at the beginning they underestimated the extent of the course.

Nevertheless, the students found the on-line study mode very interesting, different from the traditional one, attractive, demanding active participation of students and supporting more and different ways of communication with the professor and tutors. They appreciated the interchange between theory and practice, which made the studied subject much more interesting and – as one of them declared - the knowledge lasting longer.


Pilot Course Delivery and Evaluation: Main Findings

Independent external experts (TES Institute Luxembourg, Statistical Office of the Republic of Slovenia, University of Ljubljana/Faculty of Economics) participated in the evaluation, too. They – as students and tutors - also positively evaluated the CEES course and its pilot delivery.

The main findings of the pilot course delivery and its evaluation are listed below:

  • Students from the virtual class were very motivated to embark on a new way of study, but their insufficient skills for independent self-study and use of modern information technology forced them to rely heavily on the pedagogical support. Consequently, teacher-student interaction by various communication means was improved, but teachers’ workload also increased.
  • The traditional distance learning format can be regarded as a compromise between a virtual class and lectures emphasising active self-study, but allowing more face-to-face contacts in the form of tutorials, workshops and consultations. It can be viewed as a possibility to improve the quality of learning in a mass study environment.
  • The pilot course delivery showed that learning process based on hyper-media imposes new pedagogic concepts and new operational criteria for students’ assessment.
  • Development of an on-line course is not a finite action, which is terminated by instalment of the course on Internet. The Internet dynamism itself requires continuous upgrading and updating of an on-line course. The rigidity of a traditional university heavily jeopardises these newly emerged tendencies and needs for flexibility.
  • On-line learning makes life-long learning easier and more feasible. The concept of a life-long learning is of a special importance for employed persons because of constant changes and updates of professional knowledge and expertise, induced mainly by permanent technological progress and changes in modern dynamic societies.



In the information society, knowledge is power. However, traditional teaching and learning processes are rigid, inflexible and usually very expensive. New technologies make them more productive and at the same time also more flexible with significantly different cost structure. By extending and deepening sources of information, on-line study courses open promising alternatives to traditional study courses in many areas of expertise – also in the field of official statistics.

Compared to traditional educational programmes, an on-line course is much more complex and demanding in terms of its development procedures. An appropriately designed on-line course enhances the complexity of an educational product because it integrates various educational services (e.g. self-developed study materials combined with outside on-line sources, administrative and study support services etc.) into one integral product mix. Thus, it makes the study process more flexible in terms of contents, time, pace and place. The integration of various educational functions into one product makes the quality aspects of educational process more transparent and open.

The research carried out as part of the CEES course development’s preparatory activities revealed an existence of a rather diverse approach to the use of the Internet by the main providers of official statistics. On-line dissemination of official statistics is mainly in its initial development stage. For thorough future exploitation of the ICT advantages in the field of economic statistics, systematic and co-operative research efforts of academics, professional statisticians, information technology experts and users are needed.



  1. Bregar, L., Ograjenšek, I. (1999). Impact of Internet on Official Statistics: Users’ Opportunities. 52nd Session of International Statistical Institute, Helsinki.
  2. Bregar, L., Ograjenšek, I. (1999). Improving Teaching Process by Use of Information Technology: Case of Economic Statistics. To appear in the Collection of Papers of the Second Creative Teaching Conference, organised by the World Association for Case Method Research and Application in Lucerne/Switzerland, 3.-7. January 1999.
  3. Pustišek M., Bešter J. (1998) An Integrated On-line Distance Learning System. 1998 EDEN Conference, Bologna, pp. 400-406.



e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, MOOCs, ICT, information and communication technology, collaborative learning, internet, interaction, learning management system, LMS,

Current issue on Sciendo

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.