Wednesday, February 2, 2011
Evaluation Report - an updated version
Please click here to retrieve my updated version of my final report
Tuesday, February 1, 2011
Journey - Evaluation paper - Graduate Certificate in Applied eLearning
Hi Bronwyn and fellow class mates
Some comments on my journey through the Evaluation paper.
At first I thought this paper would be 'no sweat'. How wrong I was. First and foremost, I must confess that I am not the X or Y and contemporary generations that has been brought up in front of a computer screen. So for me accessing and using on-line forms of communication is a struggle. I am fine with everyday useage but once I have to create blogs, access eluminate, use eluminate - it is somewhat of a challenge.
I recall one night trying to get onto eluminate (without Dana) took me nearly one whole hour of the two hour session. The reason was that I was using the wrong password. I not only stayed at my work (MIT) to access it until the required 6.30 p.m. had the frustration of not being able to get into it when I knew that the discussion was occurring, but then have the 3/4 hour drive home afterwards. So for me, this aspect of the educational delivery was somewhat of a challenge.
Working with Dana was a real treat. He is amazing and has a great command of the technology. We worked very well as a partnership and it was a real delight and a privilege to have such a partner. We learnt much from each other.
The plan - was a lot of work. I finally found all the parameters were well identified in the Reeves and Hedberg (2003) book but did not discover this till I have completed it. I did not think it would be so much work but even with the guidelines and the exemplars (which I couldn't always have time to read) one still has to do the thinking and construction required to achieve this goal. I found that instructions had not only to be clear but to be understood. The comprehending of each aspect as it related to my topic took many hours to produce. The comprehensiveness of the plan was not understood at first. Having Dana as a partner was great in this process.
Eluminate
I did not always find the discussion helpful for most members had their own perspectives they wished to explore which was not necessarily what I wanted to know at the time but however, it was helpful.
It was good to hear from each member what they were doing and what approach they were taking to their evaluation.
Although e-learning is all about the 'constructivist' approach to knowledge development, I would have preferred a good guide to a reading list that would have been focused specifically on the evaluation process.
I am of the generation that still likes to explore literature rather than on-line material wholely.
The Evaluation Report took hours and hours. It did not help that I was tremendously busy at the conclusion of the year with my work and personal commitments. This report was an extremely time-consuming piece of work to produce. The writing up required to report on every facet of the evaluation process, creating and using the data collection tools, refining tools on the basis of Bronwyn's feedback and then putting these into operation, meant negotiation over time with students and lecturers and for me the Programme Leader of the Bachelor of Nursing programme at MIT. Catching the students and requesting their involvement is easy but then getting them to carry out what you intend for them to do becomes a secondary requirement when other more urgent matters are pressing for them. Additionally, any instructional content, major modifications and teaching strategies must be approved by the Programme Committee before alterations can be made. So time consumption in gathering data is a very real factor when collecting data within and around everyday commitments.
The analysis and compilation of the report to include portraying the results in table and graphic form was time consuming and challenging. Fortunately I had Dana to help me with the graphs for I use these so infrequently
I had to relearn how to do it. Then to get Bronwyn's detailed feedback requiring work still to be done after completion was daunting.
Bronwyn's teaching and support throughout the paper has been awesome even if at times, I have been somewhat disappointed with the evaluated outcome. I have appreciated Bronwyn's obvious desire for students to do well and to cope with a predominantly on-line teaching method which, we all know, is open to the students interpretation. It is in the latter that I found much ambiguity and at times confusion. This in turn has brought frustration. I can say that it has been a challenging journey but it has been good to share that journey with others and especially Dana as we have paddled our waka together to conclude the paper.
Thank you Bronwyn for your commitment to your teaching and for the patience that is required for students to achieve their educational goals. I have learnt a great deal and have appreciated your detailed comments that give very direction for correction and improvement.
Thank you fellow class mates for sharing my journey - I trust that you have all achieved good outcomes and will go well in your future educational endeavours.
Louise
Some comments on my journey through the Evaluation paper.
At first I thought this paper would be 'no sweat'. How wrong I was. First and foremost, I must confess that I am not the X or Y and contemporary generations that has been brought up in front of a computer screen. So for me accessing and using on-line forms of communication is a struggle. I am fine with everyday useage but once I have to create blogs, access eluminate, use eluminate - it is somewhat of a challenge.
I recall one night trying to get onto eluminate (without Dana) took me nearly one whole hour of the two hour session. The reason was that I was using the wrong password. I not only stayed at my work (MIT) to access it until the required 6.30 p.m. had the frustration of not being able to get into it when I knew that the discussion was occurring, but then have the 3/4 hour drive home afterwards. So for me, this aspect of the educational delivery was somewhat of a challenge.
Working with Dana was a real treat. He is amazing and has a great command of the technology. We worked very well as a partnership and it was a real delight and a privilege to have such a partner. We learnt much from each other.
The plan - was a lot of work. I finally found all the parameters were well identified in the Reeves and Hedberg (2003) book but did not discover this till I have completed it. I did not think it would be so much work but even with the guidelines and the exemplars (which I couldn't always have time to read) one still has to do the thinking and construction required to achieve this goal. I found that instructions had not only to be clear but to be understood. The comprehending of each aspect as it related to my topic took many hours to produce. The comprehensiveness of the plan was not understood at first. Having Dana as a partner was great in this process.
Eluminate
I did not always find the discussion helpful for most members had their own perspectives they wished to explore which was not necessarily what I wanted to know at the time but however, it was helpful.
It was good to hear from each member what they were doing and what approach they were taking to their evaluation.
Although e-learning is all about the 'constructivist' approach to knowledge development, I would have preferred a good guide to a reading list that would have been focused specifically on the evaluation process.
I am of the generation that still likes to explore literature rather than on-line material wholely.
The Evaluation Report took hours and hours. It did not help that I was tremendously busy at the conclusion of the year with my work and personal commitments. This report was an extremely time-consuming piece of work to produce. The writing up required to report on every facet of the evaluation process, creating and using the data collection tools, refining tools on the basis of Bronwyn's feedback and then putting these into operation, meant negotiation over time with students and lecturers and for me the Programme Leader of the Bachelor of Nursing programme at MIT. Catching the students and requesting their involvement is easy but then getting them to carry out what you intend for them to do becomes a secondary requirement when other more urgent matters are pressing for them. Additionally, any instructional content, major modifications and teaching strategies must be approved by the Programme Committee before alterations can be made. So time consumption in gathering data is a very real factor when collecting data within and around everyday commitments.
The analysis and compilation of the report to include portraying the results in table and graphic form was time consuming and challenging. Fortunately I had Dana to help me with the graphs for I use these so infrequently
I had to relearn how to do it. Then to get Bronwyn's detailed feedback requiring work still to be done after completion was daunting.
Bronwyn's teaching and support throughout the paper has been awesome even if at times, I have been somewhat disappointed with the evaluated outcome. I have appreciated Bronwyn's obvious desire for students to do well and to cope with a predominantly on-line teaching method which, we all know, is open to the students interpretation. It is in the latter that I found much ambiguity and at times confusion. This in turn has brought frustration. I can say that it has been a challenging journey but it has been good to share that journey with others and especially Dana as we have paddled our waka together to conclude the paper.
Thank you Bronwyn for your commitment to your teaching and for the patience that is required for students to achieve their educational goals. I have learnt a great deal and have appreciated your detailed comments that give very direction for correction and improvement.
Thank you fellow class mates for sharing my journey - I trust that you have all achieved good outcomes and will go well in your future educational endeavours.
Louise
A theoretical model that I have learnt about and used during my journey on the evaluation paper
Hi everybody
I am sure, like me, you have learned a lot of new ideas. I did not know that so much had been written about the term'evaluation' and now recognise that there are many interpretations of this term.
Using Reeves and Hedberg's (2003) six facets of instructional product design I learnt that all of these require different questions and approaches. The six facets, needs analysis, formative evaluation, effectiveness evaluation, the overall impact of the learning package, maintenance and then finally a review of the total instructional process. The most confusing terms for me were formative and summative evaluation as these terms I am very familiar with within a particular context. The 'effectiveness evaluation' which was the focus of my study and report required me to look at the total learning package. This I found was extremely comprehensive and complex. Reeves and Hedberg claim that evaluating on-line instructional designs is a complex and an inexact science and I would totally agree with them. Perhaps what was most significant for me was to re-orient myself to the different approaches to the word 'evaluation' and recognise that there are different orientation e.g. the orientation required for my 'effectiveness' evaluation was the methodological approach.
During the theoretical exploration I looked anew at a research term 'triangulation' that I was familiar with from another context but began to understand it differently in relation to the 'effectiveness' evaluation. I read up on and explored the electic mixed methods pragmatic approach to data collection to understand the origins of this approach and recognise that it has evolved from principally the logico-positivistic paradigm in the first instance, then the interpretive paradigm and finaly has a pragmatic dimension to it. Reeves and Hedberg (2003) consider that using the electic mixed methods pragmatic approach to collect data for analysis on the topic of issue provides a practical orientation to practical problems that instructional designers are confronted with and provides useful information in order to move forward.
My exploration into the electic mixed methods approach took me to Bronwyn's webpage (2003, (c) Otago Polytechnic "Experimental and Multiple Methods Evaluation Models' and my reading of this literature extended further. The term 'triangulation' became clearer and the techniques e.g. survey, interview, discussion board entries, as well as quantitative scales that show definitive outcomes. The upside to this method is the breadth and depth one can achieve by examining your topic from a range of different perspectives. For my 'effectiveness' evaluation I used a student survey, lecturer peers and a written comment to address the questions that I had raised to gain insight into the 'effectiveness' of my on-line 'Conception to irth' of the lifespan content that new nurses need to learn. But of course the downside to this method is that the results achieved depends on a large enough sample to gather the data, the adequacy of the techniques to establish the depth and breadth of information required and then much of it is probabalistic rather than predictable. However, the triangulation approach does give comprehensive data from which to make decisions to points ways to improving teaching/learning processes.
So as far as I am concerned, I learnt a lot but there is still more to learn no doubt.
I will be interested to hear others' points of view.
Congratulations on concluding the evaluation paper. I am sure you too have had an interesting journey.
All the best for your future studies.
Louise
I am sure, like me, you have learned a lot of new ideas. I did not know that so much had been written about the term'evaluation' and now recognise that there are many interpretations of this term.
Using Reeves and Hedberg's (2003) six facets of instructional product design I learnt that all of these require different questions and approaches. The six facets, needs analysis, formative evaluation, effectiveness evaluation, the overall impact of the learning package, maintenance and then finally a review of the total instructional process. The most confusing terms for me were formative and summative evaluation as these terms I am very familiar with within a particular context. The 'effectiveness evaluation' which was the focus of my study and report required me to look at the total learning package. This I found was extremely comprehensive and complex. Reeves and Hedberg claim that evaluating on-line instructional designs is a complex and an inexact science and I would totally agree with them. Perhaps what was most significant for me was to re-orient myself to the different approaches to the word 'evaluation' and recognise that there are different orientation e.g. the orientation required for my 'effectiveness' evaluation was the methodological approach.
During the theoretical exploration I looked anew at a research term 'triangulation' that I was familiar with from another context but began to understand it differently in relation to the 'effectiveness' evaluation. I read up on and explored the electic mixed methods pragmatic approach to data collection to understand the origins of this approach and recognise that it has evolved from principally the logico-positivistic paradigm in the first instance, then the interpretive paradigm and finaly has a pragmatic dimension to it. Reeves and Hedberg (2003) consider that using the electic mixed methods pragmatic approach to collect data for analysis on the topic of issue provides a practical orientation to practical problems that instructional designers are confronted with and provides useful information in order to move forward.
My exploration into the electic mixed methods approach took me to Bronwyn's webpage (2003, (c) Otago Polytechnic "Experimental and Multiple Methods Evaluation Models' and my reading of this literature extended further. The term 'triangulation' became clearer and the techniques e.g. survey, interview, discussion board entries, as well as quantitative scales that show definitive outcomes. The upside to this method is the breadth and depth one can achieve by examining your topic from a range of different perspectives. For my 'effectiveness' evaluation I used a student survey, lecturer peers and a written comment to address the questions that I had raised to gain insight into the 'effectiveness' of my on-line 'Conception to irth' of the lifespan content that new nurses need to learn. But of course the downside to this method is that the results achieved depends on a large enough sample to gather the data, the adequacy of the techniques to establish the depth and breadth of information required and then much of it is probabalistic rather than predictable. However, the triangulation approach does give comprehensive data from which to make decisions to points ways to improving teaching/learning processes.
So as far as I am concerned, I learnt a lot but there is still more to learn no doubt.
I will be interested to hear others' points of view.
Congratulations on concluding the evaluation paper. I am sure you too have had an interesting journey.
All the best for your future studies.
Louise
Subscribe to:
Posts (Atom)