back

Converging traditional assessment philosophies with open assessment philosophies - introducing a low tech. solution through action research

Authors
Andrea Cassidy & Lawrence Benson
School of Health Studies, University of Bradford, Bradford, West Yorkshire, BD7 1DP, United Kingdom

© 1999


Abstract
Introduction
Methodology
Results and Discussion
- question 1 & 2
- question 3
- question 4
- question 5
- question 6
- question 7
Conclusion
References
Figure 1: Action Research Plan


Abstract

This paper uses an action research cycle to implement and evaluate a new assessment strategy in the BSc in Professional Practice - Open / Distance Learning. After a short introduction on the action research cycle (given as figure 1 at the end of this article), we present the results of an evaluative questionnaire identified as an action step from the action research process.

Introduction

The Division of Health Care Studies (created April 1996) in the University of Bradford offers an Open and Distance learning degree (BSc in Professional Practice - Open / Distance Learning) with a philosophy grounded in adult learning theories (Bruner 1966, Revans 1983, Knowles 1985, Rogers 1982 ).

Traditionally the University incorporates a written invigilated examination for each component of a degree to ensure that the student’s knowledge is tested appropriately, an assessment incongruent with the adult learning philosophy of the open learning degree.

The original assessment scheme for the degree consisted of written assignments thus not adhering to the University’s requirements nor the Division’s philosophy, as it did not fully assess the application of theory to practice through reflection (Schon 1983). Health care professionals are being encouraged to use reflection to evaluate practice in the United Kingdom (UKCC and the IHSM). It was therefore decided to implement an assessment strategy that would meet the needs of both the university and the division. We consequently commenced an action research study to help us implement and monitor a new assessment strategy.

Methodology

Elliot’s (1991) Action Research cycle was used to structure the enquiry. The stages in action research can be described as a cyclical process involving the diagnosis of a problem, the development of an action plan, the implementation of the plan and the evaluation of the plans effectiveness.

Data was collected using pen and paper methods and, live methods (McNiff 1988). The pen and paper method involved notes from meetings, written feedback from tutors and questionnaires sent to students who had experienced the assessment interview. The live methods included simulating the experience for tutors and the actual interviews themselves.

The sample for the questionnaire was purposive (Frankfort-Nachmias and Nachmias 1997) and included all students who had undertaken the newley implemented assessment interview of which there were 19. The questionnaire was designed using open and closed questions.

Results and Discussion

The action research process identified a number of issues regarding the implementation of the new assessment strategy and each issue formed a new action step, which was then implemented. These issues are best described using the action research framework and this is diagramaticaly demonstrated in Figure 1. One of the major decisions to take place during the process was to evaluate the student experience with a view to further improving the assessment strategy. It was felt that the best way to tackle this was to design and send out a questionnaire. The following results are taken from the questionnaire sent to those students.

There were 13 respondents from the 19 questionnaire sent indicating a response rate of 68. Of the 13 replies 5 had experienced face to face interviews and 8 had experienced telephone interviews. The results were analysed using the student’s experiences as a whole and a comparison of the two options answers were then categorised for tabulation purposes. However the authors were interested in the qualitative answers.

Question 1 & 2

The first question was simply to determine which interview option the student had chosen, face to face or telephone, the second question however was intended to obtain a more qualitative reply and was concerned with the students thoughts about having this type of assessment. 7 of the respondents indicated levels of anxiety, this was more pronounced in the telephone assessment.

  • "I was very nervous about having an assessment interview"
  • "Apprehensive although reassured because it is using reflective practice"

6 respondents however believed it to be a good way to undertake the assessment for a number of reasons

  • "Some people can talk better than write therefore talking and writing gives everyone a fair chance"
  • "Had preference for this type of assessment as compared to a written examination"

and 2 students felt they had not been prepared adequately

  • "there were no specific guidelines for the assessment so one could have read all the books available and still been unprepared"
  • "Very anxious not knowing what questions would be asked, would there be enough time? How would I answer? Would the answer be appropriate…………."

The conclusions from this question were that students needed a great deal of reassurance prior to the interview and needed better preparation, perhaps the module leader to go through the written guidelines with the students and give example questions.

Question 3

Question 3 was concerned with the preparation for the assessments, one of the assessments aims was to encourage students to read their materials and apply the concepts to practice. There was a wide range of different preparation strategies reported by students (Fig 2) the most common ways of preparing were reading the open learning materials and taking notes before for the interview.

Fig.2
Figure 2: Answers for question 3

Students who had chosen the telephone or the face to face assessments reported that the oral assessment had stimulated them to read the open learning materials, they all utilised other sources as well, including their reflective notes. The conclusion from these answers was the assessment was meeting its aim:

  • "By reading the material and highlighting the main points and making notes on how each point related to my practice"
  • "Read course material, made notes on daily practice"

Question 4

Question 4 was designed to give the student the opportunity to describe the experience first hand so the question asked was "How did you find the interview on the day?" The comments made contained a mixture of emotions, the students were describing the anxiety they felt alongside the relief that the experience was more positive than they had expected and were therefore looking forward to the next time.

  • "Relaxing not as bad as I expected encouraged by the non verbal communication"
  • "Initially nerve racking, but as the interview progressed I became more settled and at the conclusion my feeling was of a relaxed nature. Generally I enjoyed the interview and felt really good afterwards in the knowledge that I answered the questions confidently. I was surprised that I remembered so much, and also how easy it was to relate theory to my practice. I put this down to my thorough revision of the whole module as I did not need to refer to my notes. Brilliant! Not as worried about the next one"

2 respondents however described a negative experience:

  • "Awful, I felt as if I waffled my way through it, however it has prepared me for the next one"
  • "A disaster, + stressful occasion. Felt stupid trying to answer questions so long winded. Forgot the question by the end."

The responses to this question has led us to plan a follow up study to ascertain whether the anxiety experienced at this interview was reduced at the next interview and also to repeat this study with a new set of students following the changes made to the preparation.

Question 5

We wanted to discover the student’s opinions of the benefits and weaknesses of the assessment process therefore in question 5 the students were asked to comment upon the benefits and weaknesses of the assessment process (Fig 3).

Fig.3
Figure 3: Answers for question 5a

There was a wide range of benefits reported by both groups of students.

  • The most commented upon were that the oral assessment interview:-
  • Explored the relationship between theory and practice;
  • Provided a stimulus for using learning materials;
  • Offered a way with which the student between verbal and written expression could strike a balance across the degree and its modular components.
  • Other reported benefits included that the assessment interview:-
  • Allowed an exercise of communication skills;
  • Gave an opportunity for the student to clarify issues and their reflections;
  • Acted as a medium for the synthesis of ideas from the student’s practice and their studies on the module.

There was also a wide range of commentary regarding the weaknesses of this assessment approach (Fig 4)

Fig.4
Figure 4: Answers for question 5b

The most commonly reported included having no time to think due to nervousness, the assessment design and finding the oral assessment off putting if you were not of an out going personality. It was interesting to note however that none of the face to face assessment students made a comment about the assessment design and one student assessed by telephone suggested that the telephone itself could potentially be a weakness.

The conclusions we drew from this two-part question was described eloquently by one student: "The benefits outweigh the weaknesses".

Question 6

Question 6 asked the student directly if they had been encouraged to read the open learning materials by the assessment interview (Fig 5) 11 students stated that this was the case and only 2 stated that it had not and these statements were suggesting that the materials would be read anyway.

Fig.5
Figure 5: question 6

"No – it encouraged reading around ones own chosen topic area"

"Not really, you had to read the materials to be able to instigate the written assessment"

One respondent wrote

"It encouraged me to read the materials in considerable depth and try to develop a clear understanding of complex areas.It also encouraged me to use the materials as a basis for study ie, taking each section and selecting areas which I was weak on. Then carrying out my own research & study in these areas to meet my learning needs"

This reconfirmed one of our main aims which was to make the reading of the materials more meaningful.

Question 7

Similarly question 7 asked students directly if the assessment interview encouraged them to reflect on the application of theory ( Fig 6) 12 said that it did compared to 1 that said it did not

Fig.6
Figure 6: question 7

"NO there was no time to reflect, no amount of studying prepares you for the interview. My mind went blank, couldn’t concentrate on questions, Too nervous."

Of the 12 who believed it had encouraged them to relate theory to practice, comments were made confirming our belief that application of theory to practice can be demonstrated in this way and has actually taken place, going someway to reducing the theory practice divide:

  • "Yes. It gave the opportunity to add depth and feeling, adding your values. Kept opinions into perspective. But also made you think and evaluate ourselves & enhance the care each individual client receives. Also realising our own faults and failures and finding ways of compromise"
  • "Yes encouraged you to rapidly analyse the topic in terms of applicabilityetc to your own area. One question asked said ‘how your system aides strategic planning?’ as the assessment was verbal I had the opportunity to say ‘well actually it doesn’t BUT I can tell you why that is"

The final question simply asked for any other comments of which there were a number, varying from affirmation of it’s benefits to some tips for improvement such as:

  • "a warming up, 5 min exercise before the interview starts"
  • "Interviews need to be more structured & more info about them given to students"

Conclusion

In conclusion we believe we have designed and successfully implemented an assessment strategy that enables all students of varying learning needs to achieve the same results. The assessment demonstrates the application of theory to practice and ensures that the student is practising reflectively. The end result must be a positive one not just for the students learning and grades but also for the clients and organisations who will be on the receiving end of this able and knowledgeable practitioner.

"The benefits are to give writers and talkers an equal chance, It is a communication skill that nurses and other health care professionals should have. Speaking clearly and using language appropriate to patients/clients and other people is a necessary part of nursing care"

Action research is a suitable method for carrying out this type of change as it enables close evaluation and refinement to be made constantly ensuring the change is effective and successful.

We once again find ourselves at a reconnaissance and we will be re-defining the general plan soon it is always difficult to end a cyclical process and so this paper does not end it awaits questions, ideas and further discussion.

References

Bruner, J. (1966) Toward a theory of Instruction, New York: W.W. Norton.

Elliott, J. (1991) Action research for educational change Buckingham: Open University Press

Frankfort-Nachmias, C. and Nachmias, D. (1997) Research methods in the social sciences 5th Ed London: Arnold

Knowles, M. (1985) Andragogy in Action, London: Jossey-Bass

Mc Niff, J. (1993) Teaching as learning: an action research approach London: Routledge

Revans, R. (1983) The ABC of Action Learning, Bromley, Chartwell-Bratt

Rogers, C.R. (1982) On becoming a person, London: Constable.

Schon, D.A. (1987) Educating the reflective practitioner, San Fransico: Jossey-Bass

United Kingdom Central Council for Nursing, Midwifery and Health Visiting (1992) Code of Professional Conduct, London, UKCC.

Institute of Health Services Management (1997) Continuing Professional Development - Support Book - Pilot Scheme, London: IHSM

Figure 1: Action research plan

Problem

   

Current assessment format not adhering to University requirements or Division of Health Care Studies philosophy

(This issue emerged following a curriculum development meeting and was brought back to the Division of Health Care Studies.)

 

Reconnaissance

   
The Division proposed to introduce alongside the written assignment an assessment interview, which could either be completed via the telephone or face to face. The assessment interview would test the student's presentation, comprehension and application of theory into practice against the learning outcomes of the particular component of the degree.

 

General Plan

   
To introduce this low tech solution when assessing distance and open learning students which would be completed in a non-threatening way whilst satisfying the University’s more traditional approach.

Action Step 1
To write guidelines for tutors and students

Action Step 2
Write assessment criteria

Action Step 3
Write questions to be used for assessment

Action Step 4
To offer and deliver tutor training for this type of assessment

 

Implement Action Steps 1 - 4

 

Monitor Implementation and Effects

   
  1. Guidelines were written, distributed for discussion with the tutorial team and then agreed.
  2. Assessment criteria was written, distributed for discussion with the tutorial and also to the Head of the Division to ensure it met the learning outcomes of the undergraduate assessment within the University.
  3. Questions written and distributed to module leaders for refinement and discussion and sent to Head of the Division to ensure they met the learning outcomes of the undergraduate assessment within the University.
  4. Simulation set up of assessment interview by telephone and tutors invited to attend

 

‘Reconnaissance’

Revise General Idea

 
  1. A number of tutors made suggestions for inclusion in the guidelines e.g. if a student was not present when the module tutor phoned then they would be referred, students should not use mobile phones and they should be in a room undisturbed from interruptions. The general feeling however was that the guidelines would meet the students needs following these refinements.
  2. Nobody had any adverse comments regarding the assessment criteria and the Head of Division ratified these criteria
  3. Nobody had any adverse comments regarding the questions and the Head of Division ratified these questions against the learning outcomes of the degree.
  4. The simulation of the assessment interview by telephone gave the tutorial team its best understanding of the potential strengths and problems of this assessment. There were concerns regarding the questions module leaders had set as they appeared to test the knowledge of the materials without clearly testing the learning outcomes.

It was felt important to reassure students undertaking this assessment by telephone, as they were unable to pick up the non-verbal clues. Questioning skills were found to be of particular importance as follow up questions could easily be multiple or leading resulting in confusion and ambiguity. Tutors felt that the face to face interview would be less threatening to the students than the telephone.

Strengths evident from the simulation were that the tutors strongly felt that this assessment would fully assess the application of theory to practice and would give the student the opportunity to reflect and synthesise.

This assessment gave equal weighting to the reflection and application of theory to practice orally as it did to the production of an academic essay therefore meeting everybody’s learning needs.

To implement and evaluate the revised assessment interview which would still satisfying the University’s more traditional approach.

Action Step 1
Module leaders to look at their questions and match them against the learning outcomes.

Action Step 2
The hands free speaker telephone needed to bet set up carefully to ensure the proper transmission of the interviews room.

Action Step 3
Send guidelines and assessment marking criteria to all students who will be undertaking the assessment interview.

Action Step 4
Module leaders to plan assessment interviews with student and second assessor.

Action Step 5
Assessment Interview to take place.

 

 

 

 
   

Implement Action Steps 1 - 5

 

Monitor Implementation and Effects

   
  1. Two of the module leaders refined the optimum volume for the telephone.

  2. Guidelines and assessment criteria were posted to all students.

  3. Assessment interviews were set up.

  4. 19 assessment interviews took place over three modules involving 6 assessors in total.

   

 

‘Reconnaissance’

Revise General Idea

 
The students who attended for tutorials were satisfied with the guidelines but it was felt that this feedback was not enough for the team to be re assured that the guidelines were satisfactory.

It was difficult to set up the interviews at short notice as many tutors had full diaries, it was also important that there was a constant i.e. an experienced assessor seeing more than one module leader assessing so equity and parity could be assessed.

Discussing the experience with students after the event it was apparent that the students who had telephone assessments had had a more positive experience than those who had had a face to face interview. This notion was in complete contrast with the tutors beliefs at the simulation.

It was apparent from the reconnaissance that the assessment interview needed to be evaluated. The results of such an evaluation could then be used to further improve the student experience. It was also felt that some structure to the planning of the assessments should be in place.

Evaluate the student experience to ensure the revised assessment interview was in fact meeting its aims and therefore satisfying the University’s assessment ethos

Action Step 1
Introduce an assessment interview week on a schedule so students and tutors can plan ahead to ensure they are free to organise the interview on a specific date.

Action Step 2
Write questionnaire to evaluate student experience.

Action Step 3
Post questionnaires to all students who had an assessment interview

Action Step 3
Analyse and report results

Implement Next Action Steps

1 - 3

 

Monitor Implementation and Effects

Revise General Idea

 
  1. An assessment schedule was written and included in the student handbook.
  2. A semi-structured questionnaire was devised using 8 open questions.
  3. 19 questionnaires were posted out and 13 students responded of these 5 had experienced face to face interviews and 8 had experienced telephone interviews.
  4. The face to face students was compared with telephone students to analyse comparative experiences and the responses were also analysed as a whole to interpret the full experience to enable evaluation.
To enact any results which would enhance the students experience

Action Step 1
Analyse results of questionnaire

Action Step 2
Discuss with colleagues changes needed following analysis of results

 

Tags

e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, ICT, information and communication technology, internet, collaborative learning, learning management system, MOOC, interaction, LMS,

Current issue on De Gruyter Online

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.