back

Success drivers in an electronic performance support project

Giovanni Giacometti Ceroni, INSEAD, tel. +33 1 60.72.41.45, France

© EURODL 2000


Contractual context: BOPS is a project subsidised by the European Commission, MM Project N° 1031 (ET).
This paper is based on working paper D11.5 of this project.


Abstract
What is an EPSS?
Evolution of concepts
From USA to Europe
What is BOPS?
Approaches to the project
Challenges
Three pilots
Caritas Luxembourg
- Scenario
-Targeted processes
- Documents to be stored and key performance indicators
- Outcome
SIEMENS ICN
- Scenario
- Targeted processes
- Documents to be stored and key performance indicators
- Status and outcome
Chamber of Commerce and Industry of Paris (CCIP)
- Scenario
- Targeted processes
- Documents to be stored and key performance indicators
- Outcome
Conclusions
References


Abstract

Electronic Performance Support Systems (EPSS) were born in the United States in 1991 to address the problems that raising software complexity was creating to employees. It is now a well-established software niche in the US, with many case histories and success stories and a very active research community working on it. In Europe there have been only a few implementations of EPSS in the last years, and still the acronym EPSS and the approach, methodologies and technologies it implies are almost unknown. Nevertheless the European approach to performance support deserves some insights, as it is not only a late copy of American experiences, but also a new innovative way to develop highly flexible software tools to introduce, support and manage best practices inside corporations. The Back Office Performance Support (BOPS) European project is one of the most recent initiatives to investigate this field and develop methodologies and products. Carried out by a consortium of eight European firms and partially funded by the European Commission, the BOPS project goes beyond the traditional EPSS boundaries and combines net coaching with knowledge management, training management and decision support. BOPS has developed a fully integrated web-based solution for performance support in medium and large companies. The final product of the project, an intranet system, has been installed and extensively tested in 4 pilot organizations. The Centre for Advanced Learning Technologies (CALT) of the INSEAD Business School has monitored these four pilot tests, collecting data through questionnaires, interviews and analysis of the log files. These data have been used to assess the impact of the adoption of an IT system like BOPS in the target organizations. This impact analysis focuses on the following points:

  • Benefits (improvements in efficiency, training and quality of work);
  • Barriers to the adoption, both external (social and legislative) and internal (company culture);
  • Potential organizational changes due to new processes and practices introduced by BOPS;
  • Resistances to change and effectiveness of change management plans.

Keywords: EPSS, learning organisation

 


What is an epss?

"The Genie appeared when the monitor was rubbed. 'I'll grant you three wishes, Ms. Manager. What will they be?’ The manager thought hard and decided, 'Why not go for broke and ask for my real software fantasies: WISH #1: I wish I could just bring people onto the job, sit them down and have them start being productive on day one. WISH #2: I wish I didn't have to staff with one person to support three people answering questions about the work itself or helping people use related software. WISH #3: I wish anyone could perform as an expert so best practice is a way of life here, rather than the occasional star performance"

This is how Gloria Gery [1], the industry guru who first spoke of "EPSS" in 1991, introduces the concepts of Electronic Performance Support System (EPSS) and Performance-Centred Design (PCD). EPSSs are defined as any computer software program or component that improves employee performance:

  • reducing the complexity or number of steps required to perform a task;
  • providing the performance information an employee needs to perform a task;
  • providing a decision support system that enables an employee to identify the action that is appropriate for a particular set of conditions.

The objective is to deliver on-the-job, on-demand training and coaching so to reduce training costs and improve employees’ performance. Performance Centred Design (PCD) is an innovative approach to user interface design and usually goes together with EPSS. It aims at making user interfaces as intuitive as possible.

 

Evolution of concepts

With the emergence of the so-called "Knowledge Economy", the late concepts of EPSS have evolved to support the knowledge workers and encourage their shared learning. New definitions of EPSS are given:

  • "An EPSS is the electronic infrastructure that captures, stores, and distributes individual and corporate knowledge assets throughout an organization, to enable individuals to achieve required levels of performance in the fastest possible time and a minimum of support from other people." [2]
  • "Dynamic support systems are characterized by the ability to change with experience, the ability to be updated and adjusted by the performer, and by augmenting other supports found in the performer’s community". [3]

By using these new concepts, EPSS gets closer to the Knowledge Management and learning tools. To develop an EPSS, cognitive principles are now becoming useful. How do people learn? The focus moves from individual performance to knowledge management and organizational learning: EPSSs now aim at the creation of a stimulating learning environment in order to achieve performance at the level of the organisation.

 

From USA to Europe

The concept of EPSS has been spreading first throughout North America and then, to a much smaller extent, in Europe. The number of researchers and consultants working on it witnesses the success of EPSSs in the US. Leading American research centres have carried out projects to investigate the potentialities of performance support tools, from software wizards to wearable computers. Several companies have been working side-to-side with them, specializing in the development of such systems and producing a rich variety of case histories often very well documented. In Europe these applications have raised much less interest. There have been a few very interesting experiences, for instance the projects developed at the University of Helsinki [4] and at the Knowledge Media Insitute, but there has never been a real market for performance support software. This is due mainly to different industry requirements, but also to the differences in the cultural and social environment. In Europe there is a cultural (and often legal) aversion to measuring or tracking individual performance. EPSSs started raising interest only when their focus moved towards knowledge management and global performance of teams and organizations.

 

What is BOPS?

BOPS is an European Project started in September 1998 by a consortium of eight companies distributed in France, Italy, Germany, Greece and Luxembourg, which has developed an innovative type of EPSS. BOPS has taken the late definitions approach of EPSS and has enlarged and enriched it, through both a wider coverage of processes and a careful adaptation to the European context. Within BOPS the objective of the performance support has shifted even more towards the management of learning and knowledge at the corporate level, so that it is probably not correct to define BOPS as an EPSS. A more meaningful definition would be "performance centred corporate information portal".

 

Approach and objectives

BOPS starts from the "learning organization" model mentioned by Raybould and Laffey: reliability, efficiency, or performance speed in the back office can be achieved only through a continual improvement of the operating practices, based on employees training and performance support.
BOPS main objective is to offer an environment that allows workers to:

  • get trained on demand on the new best practices;
  • be assisted and monitored by a training staff that may recommend additional training courses;
  • contribute to the development of new best practices.

As an instance let's take the case of a salesman who needs to prepare the commercial presentation of a product. First, he looks for the existing documents in the corporate knowledge base through the company’s intranet. The only presentations he finds are too technical for him. Then he enters the appropriate intranet discussion forum and posts a message asking for help. The technical staff helps him to understand the existing documents. His message is also forwarded to the training manager, who uses the intranet to check the salesman's personal profile: he verifies that the salesman didn't get the necessary basic technical training and finds the right course for him in the on-line training catalogue. As a result, the salesman could make a good presentation, what he produced is stored in the base as re-usable knowledge, and he got some necessary training.
It sounds like the three wishes of Gloria Gery. But to make it real, what do we need?
BOPS gives the following answer:

  • A workflow management system to trigger and support the information flows;
  • A corporate knowledge catalogue to store, index and retrieve documents;
  • A personnel management system, to manage the data about employees skills, training needs and performance;
  • A decision support system to provide the necessary aggregated information to training and division managers.

Figure 1 and 2 illustrate the technical architecture of the system.

 fig1.gif (15550 bytes)

Figure 1

 

fig2.gif (58346 bytes)

Figure 2

 

Challenges

BOPS had to face two key challenges: flexibility of the system and motivation of the users.

Challenge1: Flexibility vs. Effectiveness

The first dilemma developers had to solve was identifying the business processes to support within BOPS so that they were generic enough to represent the processes of different business environments, but at the same time specific enough to allow a detailed design of the system functionalities. The risk was either to develop a very effective application, but limited to very specific contexts (as most American EPSSs look like), or to end up with a very generic solution which didn't really implement any essential functionality. This is really the core problem of EPSS development, similar to the one designers face since more than 20 yeras when developing Decision Support or Expert Systems [20, 21]: "ad hoc" solutions work great, but their cost is barely justifiable, as they can be used just in the context they have been built for. Research shows that attempts to transport very customized solutions into other contexts fail 80% of the times. On the contrary those who tried to build standard commercial EPSS packages ended up with generic and flexible but meaningless functionalities [4].
BOPS mediates these two approaches, providing a framework which can be adapted relatively quickly and cheaply to a given business environment. BOPS is not a standard package ready to be installed, but a set of software modules, methodologies, best practices and templates which are at the end a "recipe" to build a good customized EPSS without reinventing the wheel.

Challenge 2: Motivation

In the last 10 years a decisive percentage (estimated between 50 and 70%) of large IT projects failed. In most cases these failures are not due to technical or development mistakes, but to a lack of users motivation or to a real active resistance to innovation. Assuming that people will use a software application just because it is good and improves performances is quite ingenuous. That's why the BOPS project has addressed also what has been called the "change management dimension": convincing users to use the new system. Tools and methodologies like the "EIS Simulation" business game [20], have been used to train "change agents", to identify organizational resistances to change and to implement change management plans. Resistances may have several causes: people may think that they might lose power, influence or opportunities, or may not understand the implications of the proposed changes, or also may lack trust in the "Change Agents". These attitudes must be taken into consideration and, wherever possible, changed. In general most people don't have any preconceived aversion to innovation, as shown in figure 3, but nevertheless it ought to actively lead and control the change process so that they get involved and motivated.

fig3.gif (4025 bytes)

Figure 3: Distribution of attitudes

Supporting performance: the good, the bad and the ugly

  • " [...] companies that manage to extract significant business value from their IT investments are rare exceptions to the rule."[10]
    " Many redesign efforts do not deliver the step-function improvements in performance that, by rights, they should."[11]
  • "Major Information Technology projects incur significant risks"[12]

Assertions of this kind are quite common and as a matter of fact there is a vast literature about IT project failures. When moving from prototyping to the final deployment of a complex IT system in several sites, a failure rate above 50% shouldn't be a surprise [4]. Is this what happened in BOPS? The Back Office Performance Support (BOPS, see the first part of this article "Beyond Performance Support") European project has developed an advanced type of electronic performance support system (EPSS), which had to be installed and tested in four pilot sites. The initial idea and design have been fine-tuned and validated; the platform has been successfully developed using the latest technology and is very good from a technical perspective. But what happened when it has been installed in the pilot sites?

 

Three pilots

This article presents the experience of three of these four pilot sites, describing the scenario, the objectives and the outcome of each one. The three pilots are:

  • Caritas Luxembourg, member-organization of Caritas International, the largest Catholic charitable organization in the world;
  • Siemens Information and Communication Networks SpA. Formerly Italtel Sistemi and part of the Italtel group Siemens ICN is one of the largest European manufacturers of telecommunication equipment;
  • The Chamber of Commerce and Industry of Paris (CCIP), and in particular its department called "Direction des Formalités Internationales et des Services aux Entreprises" (DFISE).

 

Caritas Luxembourg

Scenario

Caritas Luxembourg includes 13 member-organizations employing about 850 people and working with 500 volunteers. The tasks of Foundation Caritas are: liaison/emergency relief, financial control, fund raising, technical know-how and training of operation units, systems support, counselling of the member organizations. BOPS has been installed during spring 2000 and 30 user profiles have been created for testing and validation purposes: 10 profiles with administration, secretariat skill, 10 experts in the social domain, 5 users with computer skill and 5 managers and decision-makers.

Targeted processes

The business processes implemented within BOPS are:

  • the request of training by an employee and the possible authorization/denial by her responsible. Communication between the employee and the responsible relies upon the BOPS workflow module. The responsible uses the BOPS catalogue server to check the employee's profile, the qualification of the requested training and its suitability. The system also supports the subscription and the final evaluation of the course by the employee;
  • the validation of the data stored in the catalogue server, which holds the records with the personal and professional data of the employees and of the member-organizations;
  • clients arrival at Caritas premises. After the initial contact by phone, email or meeting, the employee has to direct the clients to the organization, which might help them. The BOPS catalogue is used to understand client's needs and to identify the right service or organization to recommend.

Documents to be stored and key performance indicators

BOPS is used to manage a "Guide of resources and skills", a database of personal and professional profiles of the employees. In the system are also stored the data about the organizations and associations collaborating with Caritas.

Caritas is not interested in measuring employees’ performance; therefore the BOPS Performance Scorecard module has not been implemented. An estimate of the performance of the whole organization will be made by monitoring the system usage and on the basis of the users feedback.

Outcome

The installation of BOPS at Caritas was severely delayed by the data population of the catalogue and consequently there has not been time for any serious observation of the system usage before the end of the project. Anyway it is interesting to analyse the reasons of this delay. It must be pointed out that the management of Caritas was very committed to make the installation successful, and the new system was definitely perceived as necessary and useful. The users and the management have been well informed and motivated by the project responsible with a series of meetings and interviews. Nevertheless Caritas had a very basic IT culture: with the exception of a few Access databases all the data were not in electronic format and had to be entered manually; no central information system was used. Consequently the data population of the catalogue server, which is the key activity in the installation of information portals like BOPS, took much more than expected. The delay is relevant but it is not going to compromise the final result, as the system will be in use in a few weeks.

 

SIEMENS ICN

Scenario

Siemens ICN is a large Italian manufacturer of telecommunication equipment, employing more than 3000 people. To test BOPS the Sales Department has been chosen, and 12 pilot users have been identified as follows: 7 salespersons, 4 business product managers and 1 business development responsible. The commitment of the management and staff of Siemens to the project has been very high, as witnessed by a careful change management and by the fact that Siemens has sponsored BOPS.

Targeted processes

The business processes implemented are related to the activity of the Sales Dept.:

  • first presentation to a new customer. The salesperson uses the system to gather data about the customer, to select the right content for the presentation and to store the information acquired during the presentation itself together with its own comments;
  • follow-up meeting. The salesperson and a business product manager use the system document catalogue to prepare a more detailed presentation for the new customer. After the meeting customer requirements and comments, evaluation of the meetings, further requirements and lacks in the documentation are entered in the system;
  • proposal to customer and finalisation of the proposal. The system supports this process by providing existing documents and data and samples of past proposals. The new proposal is eventually stored in the system and possibly updated.

Documents to be stored and key performance indicators

The BOPS catalogue is the main module used at Siemens and it stores marketing studies, price schemes, product presentations and brochures, case histories, technical documents and procedures, meeting outcomes, evaluations and comments. The format of these documents might vary from MS Office to Acrobat PDF and HTML. Most documents were already digitized and additional data were available in Access databases.
Siemens is also going to use BOPS to track two types of performance indicators. The department performance is measured by ratios between the number of contacted customers and the number of successes (in approaching the customer, in making the proposal and in getting the contract signed). The quality of the catalogue content is measured by the lacks in the documentation notified by the employees and by the number of meetings/presentations not fulfilling the expectations.

Status and outcome

At Siemens BOPS installation has been completed on time and after the first 2 months of testing a questionnaire [13] has been distributed to evaluate the impact on the organisation and on the employees work. The questionnaire, composed by 16 questions, addressed four different dimensions:

  • Level of motivation and awareness. The first group of questions aimed at determining if the change management activities carried out at Siemens had been effective and if there was enough commitment to make the test valid. It emerged that the users had a good knowledge of the project and its objective, were moderately optimistic about its usefulness and sufficiently committed to try it out.
  • Level and frequency of usage. The second part of the questionnaire showed that the 12 pilot users have been using BOPS on average once every 2 days for 5-10 minutes each time. When questioned about the overall experience of using BOPS some users have rated it "a little boring", some others "quite pleasant".
  • Job enrichment. This dimension refers to the improvement in the quality and speed of the usual employees work. Already after the first 2 months most users noticed that they were able to perform certain tasks slightly faster and that in some cases the quality of their output had improved. Users have recognized in BOPS a tool to obtain training or, in the worst case, just some useful hints. These improvements have not been judged quantitatively relevant yet, but they show a positive trend and a diffuse appreciation of BOPS job support capabilities. On the other side BOPS didn't seems to boost the information flows and the communication among employees nor to have perceivable ergonomic benefits.
  • Job enlargement. The last part of the questionnaire investigated the effect of BOPS on the knowledge that employees have of their company business processes and of the tasks related to their job. This is the most interesting result of the questionnaire. Also in this dimension answers show a positive trend. Most users recognized that using BOPS they acquired a slightly better knowledge and visibility of the corporate business processes, learning about tasks which are not strictly part of their job but anyway helpful.

 

Chamber of Commerce and Industry of Paris (CCIP)

Scenario

CCIP has a public service mission: providing assistance and advice to firms, local traders and craftsmen in the Paris region. The CCIP department called "Direction des Formalités Internationales et des Service aux Entreprises" (DFISE) has been chosen as BOPS test site. DFISE is in charge of the ATA carnet, the document necessary for temporary exportation of goods. Handling the ATA carnet requires a specific training course of one day. After the course the employees have a 100 pages guide and a free access to an on-line support service. The objective of BOPS at CCIP is to give on-the-job assistance to employees, which have to manage the ATA carnet.

Targeted processes

CCIP has decided to use the workflow module and the catalogue server of BOPS. The supported workflows are:

  • ATA carnet drawing up. This is a simulation which helps the users to understand how to fill the carnet in a risk-free environment;
  • Answering questions about the ATA carnet. The catalogue server provides documents and information to help users answering questions;
  • Verification of an ATA carnet. This activity is supported by the documents stored in the BOPS catalogue server.

Documents to be stored and key performance indicators

All the documents stored in the catalogue server are related to the drawing up and the verification of the ATA carnet. CCIP has chosen not to implement any kind of performance monitoring.

Outcome

At the time of the official end of the project the installation at CCIP was not finished yet, and there were not enough data to make any forecast about its outcome. There have been major delays due to customization problems and to a lack of co-ordination with the users and the CCIP project managers, but also to a basic organizational complexity, as the project management was originally entrusted to Le Preau, an IT research center affiliate to CCIP, while the system had to be deployed at DFISE. This duplicity probably made more difficult to obtain a good co-ordination and a sufficient level of commitment. As a matter of fact the system has not been successfully deployed yet and a contingency plan still has to be defined.

 

Conclusions

The outcome of the installation has been very different in the three test sites, although the technology used was exactly the same and the objectives were very similar. The successes obtained seem to confirm the validity of the BOPS concept and of its implementation from a pure technical perspective, but the failures imply the existence of other necessary success conditions, which in some cases have not been accomplished. We have identified the following key success drivers in the deployment of an "information portal" like BOPS:

Corporate IT culture. A large diffusion of PC usage and the familiarity with a networked communication in a company obviously favour the adoption of a new IT system on the user side. In the case of Siemens, in which IT is part of the core business, a high level of IT culture also helped the developers to speed up the installation: the data population has been easier thanks to the existence of digitized documents, the server deployment has been supported by the company EDP staff, and so on. On the other hand we must say that the lack of corporate IT culture at Caritas lead to relevant delays but did not compromise a final good outcome of the project. Therefore a very basic level of IT culture is definitely a disadvantage when adopting complex information system, but does not preclude the success. Moving from typewriters to portals is possible. It just takes more time and the risk of underestimating the amount of time required is very high.

Users motivation and involvement. The BOPS project has had a special attention for the change management. The first warning about potential failures due to organizational resistances to change had been given since the very beginning of the project; afterwards, approximately two months before the installation in the test sites, a series of change management activities, like a session to train change agents in each pilot organization and the periodical drawing up of a change management report, have been launched. Results seem to be strictly related to the intensity of these activities. At CCIP they have been neglected. At Siemens and at Caritas they helped to avoid significant resistances to change. Nevertheless they came late and were not of any use to really involve users in the system deployment, taking advantage of their point of view. BOPS is a very information-intensive system; its usefulness is proportional to the quantity of data, information and documents stored in it. Employees are not passive users of this information, but actively participate in its creation. Therefore it would have been useful to involve them in the pilot system design, addressing issues like motivation and change management much earlier, at least four months before the actual installation. Probably this would have helped not to underestimate the data population at Caritas.

Responsibility and organizational framework. We already mentioned the fact that Le Preau was responsible for the installation of BOPS at DFISE (CCIP). As a matter of fact the two organizations are independent and people from the first were not in the best position to manage a project inside the second one, i.e. they had the responsibility but not the formal authority (though the most complete co-operation). Additionally we should consider that due to its size Le Preau did not have the organizational resources and control of its counterparts, Siemens and Caritas. The combination of a small and not powerful organization, a difficult organizational framework and a complex IT system prevented to achieve the expected results. Even in its simplest form BOPS impacts on several business processes and deeply affects the way in which organizations handle their knowledge and procedures. Those in charge of deploying it must have a proportionate control over the target organization and its active collaboration. Roles and responsibilities must be as clear as possible.

 

 References

[1] Gery, G. (1997) "Granting three wishes through Performance-Centered Design". Communication of the ACM , July.

[2] Raybould, B. (1995) "Performance Support Engineering: an emerging development methodology for enabling organisational learning". Performance Improvement Quarterly, 8 (1).

[3] Laffey (1995) "Dynamism in Electronic Performance Support Systems". Performance Improvement Quarterly, 8(1).

[4] Kasvi, J.J.J. and Nieminen, M. and Pulkkis A. and Vartiainen, M. (1998) "Knowledge Management on the Shop-Floor" in Manufacturing agility and hybrid automation edited by W. Karwowski and R. Goonetilleke.

[5] Clark, R.C. (1993) "EPSS Look before you leap: Some Cautions about Applications of Electronic Performance Support Systems". Performance and Instruction, May.

[6] Collins, A. and Brown, J.S. and Newman, S.E (1990) "Cognitive Apprenticeship: Teaching the crafts of reading, writing, and mathematics" in L. B. Resnick (Ed.).

[7] Duffy, T. and Palmer, J. and Mehlenbacher, B. and Norwood, NJ (1992) Online help design and evaluation. Ablex Publishing.

[8] Galagan, P. A (1994) "Think performance". Training & Development, 48(3).

[9] Gery, G. (1991) Electronic performance support systems: How and why to remake the workplace through the strategic application of technology. Boston, MA: Weingarten Publications.

[10] Battles, B. and Mark, D. and Ryan, C. (1996) "An open letter to CEOs: How otherwise good managers spend too much on information technology ". The McKinsey Quarterly, Number 3, pp. 116—127.

[11] Heygate, R.(1993) "Immoderate redesign". The McKinsey Quarterly, Number 1, pp. 73—87.

[12] Willocks, L. and Griffiths, C. (1994) "Predicting Risk of Failure in Large-Scale Information Technology Projects". Technological Forecasting and Social Change, 47, 205-228.

[13] Butani, S.J. and Phipps, P.A. and Chun, Y.I. (1995) "Research on Establishment-Survey Questionnaire Design". Journal of Business & Economics Statistics, July, Vol. 13, No. 3.

[14] Manzoni, J.F. and Angehrn, A.A. (1998) "Understanding Orgnizational Dynamics of IT-enabled Change: a Multimedia Simulation Approach". Journal of Management Information Systems, 14, 3, 109-140.

[15] Gery, G. (1995) "Attributes and behavior of performance-centered systems". Performance Improvement Quarterly, 8(1).

[16] McGraw, K. (1994) "Performance support systems: Integrating AI, Hypermedia, and CBT to enhance user performance". Journal of Artificial Intelligence in Education, 5(1).

[17] Raybould, B. (1990) "Solving human performance problems with computers". Performance & Instruction, 29(11).

[18] Reeves, T. (1995, February 7) "What is EPSS and why should instructional designers know about them?". Paper presented at the AECT Annual Conference, Anaheim, CA.

[19] Stevens, E. and G. (1996) "The truth about EPSS". Training and Development, 60(6).

[20] Angehrn, A.A. (1993) "Computers that Criticize You: Stimulus-Based Decision Support Systems". Interfaces, 23, 3, 3-16.

[21] Silver, M.S. (1990) "Decision Support Systems: Directed or Non-Directed Change". Information Systems Research, 1, 1, 47-70.

 

24/11/00 (article submitted July 2000)

 

Tags

e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, ICT, information and communication technology, internet, collaborative learning, learning management system, MOOC, interaction, LMS,

Current issue on De Gruyter Online

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.