Building KU's Teaching and Learning Community

Using an On-line Tool to Better Prepare Students for Class—Feirong Yuan (2007)

Overview

Developing reading consideration questions with the on-line tool EDU improves student preparation before class, fosters an active learning atmosphere, and helps students reach a higher level of learning than in previous course offerings.

Background

Management 310 (MGMT 310) is a required business course on organizational behavior. When I first taught the course in Fall 2005, one of my observations was that many students failed to understand required readings before arriving at class. I had assumed that students would come prepared but found that I was covering basic material and not having enough time for applications.

I was interested in finding a tool to make sure students would come to class well-prepared, having mastered readings which would enable me to cover more complex materials and give students more opportunities to apply their knowledge to real life settings. I learned that EDU, a web based computer software program, allows instructors to create before-class mastery-oriented on-line reading consideration questions. The program essentially requires students to read assigned material and then answer the on-line reading consideration questions to check their understanding of it before they come to class.

Implementation

For the EDU system, I post multiple choice questions for each reading, which I refer to as reading consideration questions. These questions are posted approximately one week before we'll apply a reading in class, and they cover basic, important material that I expect students to understand. Students need to correctly answer a specific number of questions for each section to demonstrate mastery of the material. There is no time limit, and they do not have to finish a section in one sitting.

The program is set up so that instructors can get feedback on how many students have completed an assignment and how many times it took for each student to complete it. I briefly review EDU reports before class. If I see that lots of students had numerous tries on one concept, I have more class discussion on that concept. But this does not happen very often. Most questions in my assignments can be answered relatively easily if students understand the readings. Also, because I provide comments to help students with the questions they miss, they rarely spend an enormous amount of time on any one question.

Student Work

So far I have been witnessing great results. About 98% of my students read all the assigned chapters and master the on-line reading questions before they come to each class. I now ask more application and analysis questions on course exams as a result of covering more advanced material in class, but find that students' grades are similar. I am able to ask more difficult questions than in the past because EDU enables me to use class time more effectively and students are able to reach a higher level of learning. Most students grasp the readings before class, so class time is now used for activities designed for deeper understanding, such as case analyses, group discussions, and exercises. I've also noticed that discussions and exercises are higher quality and more effective since students are better prepared for class.

Reflections

I have had a lot of success using the EDU system. Students are actively involved with the course material outside of class, and their participation is enhanced by coming to class prepared. Furthermore, class time has profoundly improved because virtually everyone comes better prepared. Using EDU, students learn to master the material rather than being quizzed. I am also able to provide immediate, detailed feedback outside of class when students are missing core concepts.

The process of designing on-line reading consideration questions has led me to reflect on my own teaching philosophy. I now assume that students are intrinsically motivated and interested in learning, whereas in the past I assumed students were extrinsically motivated. In the past, I also needed more control and overemphasized accuracy, whereas now I see the value of mastery learning.

NOTE: The EDU system is now a part of Blackboard Collaborate


^Back to top^

Background

Management 310 (MGMT 310) is a required business course on organizational behavior. A variety of students enroll in the course; most are business majors in their junior year. When I first offered the course in Fall 2005, one of my observations was that many students failed to understand required readings before arriving at class, which resulted in unsatisfactory outcomes from team discussions and exercises. I had assumed that students would come prepared but found that I was covering basic material and not having enough time for applications.

I was interested in finding a tool to make sure students would come to class well-prepared, having mastered readings which would enable me to cover more complex materials and give students more opportunities to apply their knowledge to real life settings. I met with Professor Dan Bernstein at the Center for Teaching Excellence for consultation, and he introduced me to the EDU system. EDU is a web-based computer software program that allows instructors to create before-class mastery-oriented online questions. The program essentially requires students to read assigned material and then answer the on-line reading consideration questions to check their understanding of it before they come to class. Note: The EDU system is now a part of Blackboard Collaborate.

Course goals:

  • To understand and describe a variety of behavioral phenomena in organizations, including issues related to individual behaviors, interpersonal processes, and how organization context influences people’s behaviors.
  • To apply knowledge of organizational behavior to actual situations. To analyze real-life situations by applying the behavioral knowledge from the course and to create action plans to improve the situations.

Scope of EDU
EDU allows you to do more than mastery questions for students to respond to before class. It is computer software that can ask questions, receive answers, and score answers submitted. The instructor supplies and creates the questions. It can also be used for homework assignments, exams, and study materials. Instructors can also insert pictures and graphs and use special symbols and characters when designing their questions. Questions can be several formats such as multiple choice, short answer, fill-in-the-blank, and matching.

For more information, take a look at the products offered by the EDU system (now Blackboard).


^Back to top^

Implementation

At the start of each semester, I explain to students that their learning experience in the course starts with understanding assigned readings before class, and that this is an important first step that sets a basis for class discussions and exercises. The material covered in the reading is the foundation for what will be covered in lecture, discussion, and group work. I emphasize to students that active participation in class is vital for them to develop a higher understanding of the concepts beyond the textbook and to develop the ability to analyze situations, apply organizational behavior concepts, and create action plans. During class we have discussions, class exercises, and group presentations with question and answer sessions that are activities built from the textbook material.

For the EDU system, I post multiple choice questions for each reading, which I refer to as reading consideration questions. These questions are posted approximately one before we’ll discuss a topic, and they cover basic, important material that I expect students to understand. Students need to correctly answer a specific number of questions for each section to demonstrate mastery of the material. There is no time limit, and they do not have to finish a section in one sitting.

In Spring 2006, I had students answer eight to ten questions (pdf) for each reading and was given feedback from students to have more questions. I added more questions for each reading in Fall 2006, so that there were 18 to 20 questions for each. Students felt that was too many. As of Spring 2007, I now use about 12 multiple-choice questions for each reading.

When implementing EDU, I decided to have students earn full credit if they mastered a set of question before class. If they hadn’t mastered material before class, they could still get half credit if they mastered it before it was covered in an exam. The on-line reading consideration questions from EDU are worth one-sixth of students’ course grades. I designed my EDU exercises so that whenever students answer a question wrong, they will see my comments (pdf) that explain the major concepts and analyses involved in the question and I direct them to specific parts of the readings to recapture the concepts (see examples in student performance section). Missing a question alerts students of their insufficient understanding of a topic and allows them to reconsider the reading and come back to check their understanding again with another question on the same topic. In this way, the on-line reading consideration exercise provides teaching opportunities outside class.

The learning approach of the on-line reading questions is also welcoming for students. The purpose is to encourage them to try as many times as necessary to be well prepared for class, rather than deducting quiz points for lack of preparation. With EDU, students’ grades are not adversely affected if they miss a couple questions, since there are other opportunities to master a concept with other questions about it. As long as they successfully complete a specific number of questions for each content topic from the readings, they have mastered the reading before class and will then be given full credit.

Finally, using on-line reading consideration questions fosters an active learning atmosphere among students. Instead of being anxious about and therefore guessing what will be asked on a quiz and being uncertain of what to take from course readings, on-line consideration questions provide students with guidance and with a sense of control. Students are not being passively evaluated; they now actively utilize these on-line questions to improve their learning. They also come to class well prepared, which creates active class discussions and student participation. Furthermore, because most students grasp readings, class time can be used more efficiently for activities designed for a higher level of learning such as case analyses, group discussions, and exercises.

The program is set up so that instructors can see student history to get feedback on how many students have completed an assignment and how many attempts it took for each student to complete it. Instructors can view by student or by topic. Instructors then have an archive of students’ performance, and they can see every student’s score and how each student responded to a question.

I briefly review EDU reports before class. If I see that lots of students had numerous tries on one concept, I have more class discussion on that concept. But this did not happen very often. Most questions in my assignments can be answered relatively easily if students understand the readings. Also, because I provide comments to help students with the questions they miss, they rarely spend an enormous amount of time on any one question.

My use of EDU
I found the EDU system to be flexible and easy to use. I used it for mastery based multiple-choice questions, but one can use it for other extensions as well, such as homework or exams. When I first set up questions, it took a while to enter in all of the questions and write feedback to be given for wrong answers. I also was saving time by not grading the exercises. That being said, it was fairly simple to set up and it does not require much time to maintain.

On the EDU home page (pdf), there is an instructor login section for my use when I want to check student history, enter questions, or make any changes. There is also an option to register for the class or select an assignment for students. After making those choices, students will be prompted to log on.

After I registered and logged in, I built a questions bank. There is a question bank editor that allows you to enter in questions for your course. I typed mine in manually, but instructors can format questions to be uploaded. As instructors enter questions, questions are assigned locations within a hierarchical structure so that each question has a label and a question description within a topic, and each topic is within a question bank. Each question instructors enter will also have a question type.

After instructors enter their questions, they can decide how they would like them to be presented. In setting up an assignment or exam, instructors can set a time period when it can be available to students (starting and ending times and dates). I decided not to use the time limit option. Instructors also set up how many times students can take to complete an assignment and how long they have to do so. I allow students to answer the questions as many times as they would like. I also set up the option for students to get feedback about wrong answers right after they have submitted a wrong answer. This allows me to scaffold students to better understand the material to answer a similar question correctly next time. Instructors can also give explanations for correct answers, as well. There is also an option for students to get feedback after the assignment is completed if the instructor wants to set up assignments to be more like a quiz or exam. Instructors then decide if they want a set order or random. I set my questions as random for each assignment. This is also an option to set up if instructors want the same questions asked each time the quiz is presented, or whether questions are chosen randomly from a pool of questions. I set my questions as random drawing from my question bank.

Since I am using the mastery-learning mode, I also set up how many questions students need to answer correctly for each topic. Instructors can also determine sections within an assignment for students to complete which can be used in a hierarchical manner. I decided to set up hierarchy for some assignments and not for others, depending on if there were any particular knowledge hierarchy in the relationships of the questions. For example, I set up hierarchies for questions in assignment Topic 2: Personalities and Abilities. This assignment includes questions on both the nature of personality and the big five model of personality. Because I would like students to understand the nature of personality before answering questions on big five, I set up a hierarchy so that they needed to complete a number of questions on “the nature of personality” before they answered any question about the “big five model of personality.” For some other assignments, I didn’t set up hierarchy if I didn’t see any particular reason to do so. Finally, instructors set up the number of points to be assigned to each question if the question is answered correctly.

Assignment and question types for EDU
EDU has five assignment types that all have additional features you can customize:

  1. Anonymous
  2. Homework
  3. Proctored
  4. Mastery (what I use)
  5. Study

There are seven question types. The EDU system allows you to combine question types into a single question as well.

  1. Multiple choice (what I use)
  2. Multiple section (student has to identify more than one correct answer and avoid incorrect answers)
  3. Short answer/essay (instructor needs to grade and correct manually—not immediate feedback)
  4. Fill-in-the-blank
  5. Clickable image (the image has places on it for the student to click on and correctly identify)
  6. Matching elements in two lists
  7. True-False

Screen shots from EDU course


EDU login

Assignments list

Reading question

Correct answer

Incorrect answer
 

^Back to top^

Student Work

So far I have been witnessing great results. About 98% of my students read all the assigned chapters and master on-line reading questions before they come to each class. EDU enables me to use class time more effectively and students are able to reach a higher level of learning. I’ve noticed that class discussions and exercises are of higher quality and more effective, due to the fact that students are better prepared than before I started using the on-line reading consideration questions through EDU.

Because my EDU assignments are open-book, I find that definitional questions which can be directly picked from textbooks are easier for students to answer than questions that ask for applications. Examples of questions on the EDU system and reports students receive are provided. It is interesting to note that, because these questions are available on-line for multiple trials, many students use the on-line questions to prepare for exams. However, questions from the EDU system are not on the exams; questions are from class discussions that are more advanced than the reading consideration questions. Because of using EDU, I’ve been able to ask questions that require a higher level of understanding and require analyzing a situation and applying a concept to a case (see example exam questions below). Students are able to answer these types of questions because we are now able to spend class time discussing and analyzing short cases rather than simply covering basic concepts.

Even though I asked more advanced questions and covered more advanced materials after EDU, exam grades stayed about the same. On average, on a 100-point exam, student grades actually improved slightly (about three or four points). Of course, there are many other factors (e.g., the redesign of other aspects of my class sections) other than the use of on-line questions that may have also contributed to student exam grades.

Finally, I find that students enjoy using the EDU system. In my anonymous mid-semester and end-of semester feedback forms, many students have commented, in particular, on the usefulness of the on-line reading questions in helping their learning outside of class.

Example student responses to reading consideration questions

Sample exam questions
Before I began using EDU, one of my exam questions was this:

Job satisfaction is

  1. the collection of feelings and beliefs people have about work in general.
  2. very difficult to measure.
  3. the collection of feelings and beliefs people have about their current jobs.
  4. solely dependent on the work people do.

After EDU:

Mary Miller manages a retail store that sells clothes and cosmetics to the local community. In the past three months, she has been asking employees about whether they like their jobs. As a result, Mary found out that most of the employees working in her store have very low levels of job satisfaction. Based on your knowledge of improving job satisfaction, what should she do next?

  1. increase benefits for her employees
  2. adjust her supervisor style
  3. investigate the specific causes for low job satisfaction
  4. invite OB consultants to redesign jobs for her employees

^Back to top^

Reflections

Professor Feirong Yuan

Feirong Yuan

I have had a lot of success using the EDU system. Students are actively involved with course material outside of class, and their participation is enhanced by coming to class prepared. Furthermore, class time has profoundly improved because virtually everyone comes to class prepared. Using EDU, students learn to master the material rather than being quizzed. I am also able to provide immediate, detailed feedback outside of class when students are missing core concepts.

The initial investment of time to set up the EDU system was well worth it. The program is very user friendly. It was really easy to enter in questions. Now, I can quickly change questions using the test editor. This allows me to add or delete questions, as well as change the mode of assessment, how many questions students need to get right to move on, and the method of grade reporting. In addition, now that I have an on-line test bank set up, I can use the class editor to make any changes when setting up a new course offering.

Impact on my teaching philosophy
The process of designing on-line reading consideration questions has led me to reflect on my own teaching philosophy. One question I had when I first learned about this open-book outside-of-class technology was, is there a possibly for cheating? I found that EDU is designed so that questions come out from question bank in random order. Therefore, it would be difficult for students to get a set of accurate answers from other students, or at least it would probably be easier for them to just check their own readings. However, I was still a little concerned that without either a time limit or in-class monitoring, some students may consult each other for answers without completing the assignment independently.

While discussing this with Professor Dan Bernstein, I discovered that my concerns to some extent reflected my philosophy of teaching. I learned that there could be several goals of teaching. Although evaluating and differentiating are important, it is also essential to develop students’ interest and trust them without close monitoring for a small portion of class credit. I realized I was overemphasizing accuracy and control, which is echoed in the traditional management philosophy (Theory X) of assuming employees (in this case, student) will cheat and are lazy by nature so I have to design clever, “waterproof” methods to control them. Following a different approach, I now assume that students are intrinsically motivated and interested in learning (Theory Y of management). Based on this assumption, what’s important is that students are motivated to use EDU to understand the major concepts in readings. Therefore, if two students get on the phone at 2 a.m. in the morning to debate the correct answer to a question, this may not be such a bad thing.

Contact CTE with comments on this portfolio: cte@ku.edu.


^Back to top^

Flex Teaching

CTE has created a website for helping faculty create flexible courses that can shift between in-person and online. Visit the Flex Teaching site.

GTA Flex and Online Teaching Program

GTAs: The link to the application form for the GTA Flex and Online Teaching Program is now available. It can be accessed here. 

Portfolio & Poster Search



List of all portfolios & posters