Section 3: Evaluating, Implementing and Managing Instructional Programs and Projects

1. Chapter 10 discusses evaluation in instructional design and provides you with two evaluation models, the CIPP and Kirkpatrick models for evaluation. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction.

The two models I have chosen are the Flashlight Triad Model and the Formative Evaluation Model.

Dr. Steve Ehrmann directed the award-winning Flashlight Program on assessment and evaluation. Flashlight's tools, training, consulting and external evaluations help educators guide their own uses of technology, on- and off-campus. Dr. Ehrmann's work on Flashlight has recently focused on developing research strategies employing matrix surveys. The Flashlight Triad Model covers 5 distinctive steps.

1. Overview and Confronting the Blob. This step involves brainstorming and "bringing to the table" all of the various elements that feed into and flow from a lesson. For example, faculty development and preparation, student prerequisites, budget, assistance with course materials development, student attitudes and satisfaction.
2. From Blob to Issue. To move from the confusing, large-scale blob, to a manageable evaluation, consider the purpose for the evaluation. What matters most to you and others? Who is the intended audience for your completed evaluation? Answer the following questions, then select one or two top issues to study.

• What are the three most crucial things you fear might happen as a result of using the technology?
• What are the three most crucial things you hope will occur?

3. From Issue to Triad. For each issue selected, create several "triads." A triad consists of the type of technology you might employ (T), a specific activity that the technology enables (A), and outcomes expected from that type of activity (O).  After generating several triads, select the triads you wish to focus on.

Sample Triad

4. From Triad to Data. For the triads you have selected, you can now generate questions to gather data about the triads. These questions can be delivered as surveys, interviews, focus groups, etc. At least five types of questions may be generated:

• Question Type 1: Technology. These questions are related to the technical infrastructure, its capacity, its functioning, or the degree to which students/faculty were appropriately trained to use/manage it.
• Question Type 2: Interaction of Technology and Activity. These questions are related to the functions and features inherent in various technical systems and their ability to support or enable a specific activity (i.e., "Did X technology enable you to do Y?").
• Question Type 3: Activity. These questions are related to the nature of student tasks, or what students are asked to do.
• Question Type 4: Interaction of Activity and Outcomes. These questions are related to the capability for specific technology-based activities to generate predicted outcomes (i.e., "Did activity Z help you to accomplish S?").
• Question Type 5: Outcomes. These questions are related to what students learned, how satisfied they were with a lesson or course, and what changes they would recommend.

5. From Data to Next Steps. Based on the evaluation data, you will make decisions about your unit or course.
Source: http://www.tltgroup.org

I would use Flashlight Triad Model to evaluate web based collaboration projects. In my Web Design class, students have website creation project. They use Weebly web platform to create websites. Groups contain five students work collaboratively on the project. I just discovered an open-sourced collaboration program in real time, called "Rizzoma" on the web. It allows communication within a certain context permitting a chat to instantly become a document where topics of a discussion organized into branches of mind-map diagram and minor details are collapsed to avoid distraction. By using Rizzoma, students can share their ideas and opinions on the topic. For example, for pattern background images for website, they can share their pattern background image, and discuss about it.
So, my Flashlight triad would be as follows:

Then, I would develop questions for my survey. Based on the collected data, I would be able to know if using online collaborative editing tool(Rizzoma) was beneficial to my students.

Secondly, I would use Formative Evaluation model. 

A formative evaluation (sometimes referred to as internal) is a method for judging the worth of a program while the program activities are forming (in progress).This part of the evaluation focuses on the process.
Thus, formative evaluations are basically done on the fly. They permit the designers, learners, and instructors to monitor how well the instructional goals and objectives are being met. Its main purpose is to catch deficiencies so that the proper learning interventions can take place that allows the learners to master the required skills and knowledge.

Formative evaluation helps you find out if you are achieving your goals and objects in the formative stages of your project, instead of waiting until you are have finished the project.

Formative evaluation is also useful in analyzing learning materials, student learning and achievements, and teacher effectiveness.... Formative evaluation is primarily a building process which accumulates a series of components of new materials, skills, and problems into an ultimate meaningful whole. - Wally Guyot (1978)

I am usually using this model at the beginning of an instructional period to find out what would be the best materials to be used to teach certain objectives. And, during the instructional period, I evaluate my instruction if it is successful or not. If it is not successful, then I revise my instruction.

2. Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. What else would be useful to know?

Image courtesy of 

marygrove.edu

After reading the textbook about how effective instructional design should be, I would like to include following questions to the instructional design evaluation..

• What were your expectations for this class and how effectively have they been met?
• Was instructor's use of examples or personal experience help learners get points across?
• Were the planned objectives and outcomes in the instruction achieved?
• Were the resources and inputs converted to outputs in a timely and cost-effective manner?
• How relevant was the instruction to learners?

3. Chapter's 12 & 13 focus on project management and how to manage projects when resources are scarce. You have been assigned to develop a series of professional development sessions focusing on technology use in the classroom for teachers during a time of economic decline. How will you use Situational Leadership to facilitate this project and manage scarce resources?

Image courtesy of americasdiversityleader.com

Besides being computer teacher in my school, I have also IT Manager title here. I am in charge of every single technological devices(Servers, Database, Website, Maintenance, Security, lab computers, teachers computers, students/staff domain accounts, etc.). Since beginning of my career here, I have been assigned a task to develop series of professional developments sessions focusing on technology use in the school.   Honestly, I didn't know anything about Situational Leadership model before. I just learned it by reading and researching here. If I were to assign to develop any technology sessions in the future again, I  would facilitate my session using the Situational Leadership model. Based on my personality behavior, I like to work in team more than work by myself. What I understand from Situational Leadership model is, the  successful leaders are those who can adapt their behavior to meet the demands of their own unique situation.
In developing the professional development sessions, I would like to start building my team first. I will ask others to join to team. I would like to incorporate their experiences and knowledge during developing the session. I would then identify roles in the team. I, as a IT Manager, will be the Project Manager to make sure that staff will be trained properly. I would follow the Situational Leadership Model to develop professional development session for staff on the technological projects.

Phase I: First, I would develop an agenda about the project, which contains what to do, how to do.

Phase II- Next, I would ask members in my team if they would like to add/remove anything from agenda. Then, we continue to plan, organize, and develop the professional development session. In this phase, my leadership style will be coaching them.

Phase III- In this phase, my leadership style is going to be supporting. I will be encouraging the team and rewarding them for effort and production. 

Phase IV-The team must feel comfortable to working together. In this phase, my leadership style will be the role of monitor. The team will be able to do decision making and problem solving aspects to finalize the professional development session.