Assessment Tools and
Measurements for Course Success

Due to the distance between IVC classrooms, assessment, testing, and evaluations can be more challenging to implement for instructors.

It is very important that IVC faculty members continue to evaluate and revise their IVC courses. As with any instructional design process, evaluation of the course provides an insight to the teaching and learning process.

Self evaluation can be helpful for those faculty members who are new to teaching in an Interactive Video Classroom. Faculty can be observed by IRT staff or fellow department members during selected presentations and receive feedback regarding the effectiveness of course delivery.

Suggestions for improvement can also be solicited from students and other instructors who are experienced IVC instructors. Faculty members may also want to videotape their presentation before actually teaching the first class or during the semester to get immediate feedback on performance. Technical support staff can also provide good feedback.

Meetings between IVC instructors to discuss successes and problems encountered in the classroom can also help in evaluating the IVC courses to see what works, share experiences, and get new ideas.

Below are some general guidelines to follow when delivering or implementing evaluations or course assessment instruments.

1) Review goals and objectives - One purpose of evaluation is to determine if the instructional methods and materials are accomplishing established goals and objectives. Implementation of instruction represents the first real test of what has been developed. Try to pretest instruction on a small scale prior to implementation. If this is not possible, the first actual use will also serve as the "field test" for determining effectiveness.

2) Develop evaluation strategies - Plan how and when to evaluate the effectiveness of the instruction.

Formative evaluation can be used to revise instruction as the course is being developed and implemented. For example, the IVC instructor can give students pre-addressed, stamped postcards to complete and mail after each session. A form or questionnaire could be posted on the instructor's web site to gather information.

Ask open-ended questions and follow by probing questions. Be open to suggestions and encourage feedback from the students. These "mini-evaluations" might focus on course strengths and weaknesses, technical or delivery concerns, and content areas in need of further coverage.

As an instructor, begin formative evaluations with the first lesson. When the lesson is over, review it to help make future adjustments. Consider video taping the session and viewing it later or jot down some notes when the session is completed.

Questions to consider:

• What were the intended outcomes of the lesson? Were they achieved?
• Were expectations clear prior to the lesson?
• Was the lesson technically effective?
• How was the timing?
• Was the lesson plan adequately prepared?
• What would have improved the lesson?
• What should be done differently next time?
• How did the experience compare to a more typical classroom experience?
• What did you like/dislike about technology?
• Technically, what worked? What didn't?
  

Summative evaluation is conducted right before of after instruction is completed. It can act as a spring board in developing a revision plan and provide a base line for revising an existing course or designing a new course.

Multimedia Supporthas developed an IVC Evaluation Form that helps faculty obtain feedback on the course and on the technology found in the room. IVC faculty or facilitators distribute it to students near the end of the semester.

Quantitative and Qualitative Methods

Within the context of formative and summative evaluation, data are collected through quantitative and qualitative methods.

Quantitative evaluation requires that questions be asked in a way which can statistically be tabulated and analyzed. The individual who responds is limited to preset categories of answers.

Qualitative evaluation allows for a range and depth of responses and tends to be more subjective. Qualitative analysis is more flexible and dynamic and not limited to preset questions or categories. The instructor can use a variety of techniques to gather data: participant-observation, non-participant observation, open-ended questions and answers, content analysis and interviewing either one-on-one or one on small groups Qualitative approaches may be of special value because the diversity of distant learners may defy relevant statistical stratification and analysis.

The best approach often combines quantitative measurement of student performance with open-ended interviewing and non-participant observation to collect and assess information about attitudes toward the course's effectiveness and the delivery technology.

Other Evaluation Elements

While Multimedia Supporthas an Interactive Video Classroom Evaluation Form, here are some additional areas you may want to evaluate:

• Use of technology -- how comfortable was the student with the technology?--what were the concerns and/or problems?--what were the positive aspects and attitudes about the technology?
• Class formats -- how effective was the lecture, discussion, questions and answers?
• Class atmosphere -- was it conducive to student learning?
• Student interaction --what was the quality and quantity of the interaction with other students?
• Course content --was it relevant--was there an adequate body of knowledge, and was it organized in a way that enhanced student learning? -- was everything covered?
• Assignments -- were they useful, how difficult were they, how was timeliness of feedback, what was the readability of printed material?
• Tests -- frequency, relevance, was there sufficient review, level of difficulty and timely feedback
• Support Services -- facilitator, technology, library resources, instructor availability
• Student achievement -- adequacy, appropriateness, timeliness, and student involvement
• Student attitudes -- attention, class participation, assignments completed sufficiently
• Student satisfaction -- would you recommend this class to a friend?
• Instructor -- contribution of instructor and discussion leaders, effectiveness, organization, preparation, enthusiasm, openness to student views

  Also, gather background information about the students such as age, grade, number of Multimedia Supportcourses previously taken and prior technological experience

3) Collect and analyze evaluation data - Following implementation of your course/materials, collect the evaluation data. Careful analysis of these results will identify gaps or weaknesses in the instructional process. It is equally important to identify strengths and successes. Results of the evaluation analysis will provide a "springboard" from which to develop the revision plan.

The Revision Stage

There is room for improvement in even the most carefully developed course, and the need for revision should be anticipated. In fact, there will likely be more confidence in a course that has been significantly revised than in one considered flawless the first time through.

Revision plans typically are a direct result of the evaluation process in tandem with feedback from students, colleagues and content specialists. The best source of revision ideas may be the instructor’s own reflection on course strengths and weaknesses. For this reason, revision should be planned as soon as possible after course completion.

Often, course revisions will be minor, such as breaking a large and unwieldy instructional unit into more manageable components, increasing assignment feedback, or improving student-to-student interaction. On other occasions, major revisions will be needed. Significant course changes should be field-tested prior to future course use.

Test revision ideas on small groups of learners, content specialists, and colleagues. Results of this process should be tempered by the knowledge that the characteristics of each distant class will vary and that revisions required for one learner group may be inappropriate for a different student population.