My evaluation plan

Hi everybody

I am posting my evaluation plan for assignment two.  I have done a literature review but cannot access it so will post it later.  I am at present in London at a conference.   I would be pleased to get comments
Dana and from all the group. Hope everybody is going Ok with theirs. 

Dana I could not get our pictures to copy!!


Introduction
In 2009, the writer enrolled into the Graduate Certificate of e-learning and in the Design paper of this qualification, an on-line Conception to birth component was developed and posted on the MIT Learning Management System for student learning. The first time this component was run was in Semester one, 2010 and it became evident that there should be some modification if the students were to use this component and learn from it. In semester two and a new group of students, a modification has been carried out.

The purpose of this evaluation plan is to evaluate the modified e-learning component of the Lifespan development course “Conception to Birth” to see if it as an effective learning tool.

Background

The design of the component is directed by the Blackboard Learning Management System (LMS) and uses the format the students have been introduced to through EMIT. Students are expected to access the LMS through their User Login and password that they receive on enrolment. They are also taught as part of their orientation how to navigate around the system. They are informed that all announcements, course information, course content, and assessment information will be accessed via the LMS. Students become familiar with the LMS set up and expect to see information presented in the design of the system. They can access the system asynchronously any time day or night.

From Conception to Birth is designed around a story of a family expecting a new baby. The family consists of father, mother, an elder son and baby daughter and an expected new baby entering the family (appendix i).

Students are directed to a course text from which the story of the family acts as a background for numerous questions that trace the conception to the zygote, embryo and fetal phase of the pregnancy. They are given online resources that illustrate the development process, the timeline involved and critical phases with questions to assist the students to find out the information they need to for their learning.

Additionally, a number of technological tools are used to assist them in their learning process. They have a number of ‘matching’ exercises to learn key terms, Quizlet is used for learning new terminology, a crossword is used to link concepts to definitions and a summative multi-choice test to see how they were learning. The summative multi-choice test was worth 10% of their total mark for the paper and comprised 20 multi-choice questions. From the total enrolment only four students failed to reach a pass mark and the highest mark was 19/20 and the lowest 9/20.

A discussion board was also set up for students to discuss the component as they learnt the material. In the original exposure there were 148 students enrolled in the course. To manage the large number of students, they were placed in 16 discussion groups. Instructions were given verbally, demonstrated in class and placed within the conception to birth component with an appropriate access button. Each group was asked to choose a leader who would take the week’s discussion from the group and place the key response themes on a class wiki. The writer would then check on each of the wiki’s to gain an overall impression of how students were learning and more importantly, how students were accessing and using the systems that were available to them to assist them with their learning. The discussion board did not contribute to the overall course mark for students. The discussion activity was broken up into four weeks with questions to drive the discussion each week. The writer kept a check on the discussion activity of the students.

A summative evaluation was undertaken at the conclusion of the component. Students were asked to evaluate their experience with the component and sadly only four students completed the evaluation. These four were positive.

From the writer’s observation, the discussion area of the learning component was not utilised well with very few students using it and the leadership and class wiki was only taken up by one leader. It was decided on observations made, that the component would be modified for the new enrollees for semester two.

Modifications made have included more instructions given frequently to students through the Emit LMS. Checking in class to see how the students are managing. The leadership for group activity has been scrapped along with the wiki but the discussion groups have continued with the questions being announced and published weekly to guide the students. The writer checks on all groups twice per week to see how the activity is being managed. To date I have noticed that this monitoring is making a total difference to the uptake. The students are enthusiastic and I monitor how frequently individual students access the discussion. The very new students are beginning to ask their own questions apart from the formulated ones which are very encouraging. The summative multi-choice has been changed to a formative test which is for their own monitoring of their learning but the students have been informed that questions related to the component will appear in their summative test to contribute to 20% of their total mark for the paper. Additionally the students have been informed that the group who has the most participation on the discussion board will get chocolate fish. This really has got the students going.

Aim:

The aim of the evaluation is to see if the modified e-learning component of the Lifespan development conception to birth is an effective learning tool for new students of nursing.

The purpose of the evaluation was threefold:

(a) to determine whether e-learning can satisfactorily replace the traditional lecture method of transmitting theoretical information to new students of nursing.

(b) To determine whether the conception to birth component was designed in an effective manner for student learning?

(c) To assess the effectiveness of the component through both formative and summative evaluation to demonstrate student learning.

Methodology:

Evaluation big questions:

Matched to purpose

1. How effective is the on-line segment of the lifespan curriculum component “conception to birth” as a tool for student learning?

2. How effective was the formative learning tools to assist their learning?

3. What did their formative test results demonstrate in relation to the material to be learnt?

4. What did their summative test results demonstrate in relation to the material to be learnt?

Related questions:

1. Was the timing correct for the students to respond to the questions asked:

2. Did they find the reading material interesting?

3. Did they find the questions easy to answer from their text book reading?

4. How easy was it to access the discussion board?

5. Did they find the discussion board helpful to their learning?

6. Were the questions asked relevant to their key learning goals?

7. How did fellow students assist their learning through their participation in the discussion board?

Data collection methods:

Design of evaluation

It is proposed to use a range of data collection methods to evaluate the effectiveness of the on-line conception to birth component of the lifespan curriculum content. These are as follows:

1. An evaluation form has been developed as part of the component and students are asked to complete the evaluation form (appendix 1)

2. Lecturer assessing the discussion board regularly over a two week period to assess participation and to assess the quality of the responses to the questions asked

3. Lecturer assessing the formative test participation and results of the students who participated

4. Lecturer assessing the summative questions as part of a formal test of the lifespan component of the curriculum to include conception to birth (10 questions)

5. Students completing an evaluation of the conception to birth component of the lifespan content

Audiences

1. Students undertaking 722.520 Foundations of Nursing (Lifespan Conception to birth) component are the primary audience

2. Myself as the primary lecturer

3. Secondary sources other lecturers involved in the teaching of the Lifespan component

4. Bachelor of Nursing Teams One and Two Co-ordinator

5. Programme Leader of Bachelor of Nursing programme

Logistics and Timeline

Implementation

Instrumentation

1. Evaluation form

2. Multichoice questions – formative

3. Summative


Sample

1. 112 students

2. 6 lecturers

Timeline = one semester 20 weeks.

1. Discussion board open for four weeks – the month of August, 2010

2. Analysis of responses on discussion board for 12 groups of 10 students twice per week and responding to the students 8 hours per week = 40 hours

3. Monitoring hits and performance on formative test of participating students –August to September 3 - 4 hours per week

4. Summative test results = 8 hours

5. Evaluation of component per evaluation form yet to be determined – 4 hours

6. Approximately 56 hours estimated in the evaluation process

Budget – 56 hours at $45 per hour = $2,520

Decisions

Outcomes expected:

Positive outcomes would be:

1. Students participate fully in the discussion board

2. Students take advantage of the formative test for their own learning and their is at least a 50% participation rate as measured

3. Students demonstrate 100% achievement of the ten related questions to the conception to birth component of a forty question summative test worth 20% of their grade for the Foundation of Nursing paper

Negative Outcomes could be:

1. Poor participation (<30%) in the discussion board

2. Poor participation (.30%) in the 20 question formative test on the conception to birth component of the Life Span teaching

3. Poor pass rate (>30%) on the 10 question component of the lifespan 40 question summative test

From the total process – further modification to the component a conception to birth of the lifespan teaching according to the demonstrated performance of the students and the feedback on their evaluation of the component.

It would the same for the lecturers involved in the teaching and the summative test results.

Working with Dana

I have been working with Dana on our assignment on Friday September 3.  We have had many useful discussions one of which was to clarify the distinction between an aim and a purpose.  This seems a grey area but I believe it is that an aim is a more broader approacvh to evaluation in relation to our topic and that the purpose is more specific.  That is the approach that we have taken. 

Unfortunately in the LSC interruptions occur with students wanting assistance and making appointments for that is the focus of the centre so we get quite a few interruptions.  Nevertheless we are making progress.
Dana's smiley face approach to a difficult topic will I am sure be a winner with students especially as they create their face and require to know the constituents of the cellular structure in order to do this.  Therefore there learning will be accelerated through both their research, finding a solution to the problem confronting them and drawing their smiley face.  This approach to learning is fun and should make it easier for them to remember the structure of a cell.

I am getting along with my assignment and hope to post this tomorrow.  I have done a literature search and have some interesting reading that Dana and I will share in discussion and select what is relevant for both of us. 
HOpe everyone else is getting along OK. 

Dana is a great coffee maker!!!
Louise

Working on my assignment

I am working with Dana in the LSC on our assignment. 
WE are considering the various dimensions of the work together and are applying it to our different projects.  We recognise that our discussion surrounds the process and we are working through this.  I have written quite a lengthy background and introduction to establish the context but will need to edit this.  It is helpful in the beginning to clarify my own thinking so by writing out my own thoughts it accomplishes this.  We are moving on this evening together through the process.