OIE Pilot Test Documentation

2009-06-03T03:41:36Z

==Pilot Test Survey Cover Email== Annie’s Project Participants, As part of your Annie’s Project class, you were introduced to the online course, Investing For Your Future for Farm Families. We hope you have had a chance to enroll and take some or all of the course. The purpose of this email is to get your feedback about the course so that we can make any adjustments to make the course as useful as possible for farm families. To get your feedback, we would like you to complete an on-line survey. All Annie’s Project participants who have been introduced to this course in their classes are being asked to participate in this survey, and your participation is very important to us. Please know that your response to this survey will remain anonymous. Survey information will be analyzed in the form of summary data and no individual responses will be identified. The Institutional Review Board (IRB) here at Virginia Tech may view this study’s collected data for auditing purposes. The IRB department is charged with the responsibility for oversight in any study involving protection of human subjects in research. This survey is being administered on a voluntary basis. There are no rewards or gratuities in place for your participation. You are entitled to back out at any time. However, your input in this survey will greatly affect the development of this program. To complete the online survey, please click on the following URL: Survey url Please complete the survey by DATE We appreciate your time and effort to complete this survey. Any responses you provide are kept anonymous. If you should have any questions or concerns about this survey, please feel free to contact us. Respectfully, Mike Lambur, Ph.D. Evaluation Chair
eXtension Evaluation and Research Leader & Research Professor
Virginia Tech
288 Litton Reaves Hall
Phone: 540-231-8177
mike.lambur@extension.org Jeffrey A. Richmond
Graduate Research Assistant
Virginia Tech
288 Litton Reaves Hall
Phone: 540-231-8177
jrichmo@vt.edu ==Pilot Test Survey== [http://tinyurl.com/OIE-pilot-test-survey Pilot Test Survey] Please click the "Next" button at the bottom of each page to move through the survey. ==Pilot Test Facilitators Interview Cover Email== Annie’s Project Facilitators, Thank you for agreeing to participate in the pilot test interview of the Online Investment Education (OIE) course, Investing for Your Future for Farm Families (IFYFfFF). The overall purpose of this interview is to help improve the functionality of the online course (IFYFfFF). We are interested in this information so that we can incorporate any changes in the course for improvement. This interview information will strictly be used to help in this improvement effort for the online course. Your interview is schedule for DATE at TIME and should take about 20 minutes to complete. Jeff Richmond will be conducting the interview. Jeff will be recording your interview in order to most accurately reflect your comments. He will store this interview data on his computer and he and I will be the only people who will have access to it. Once the study is completed, he will destroy your recorded interview data. Please be assured that your responses to the questions will be strictly confidential. Interview information will be released only as summaries in which no individual’s responses can be identified. It is possible that the Institutional Review Board (IRB) may view this study’s collected data for auditing purposes. The IRB is responsible for the oversight of the protection of human subjects involved in research. This interview is completely voluntary. No promise or guarantee of benefits has been made to encourage you to participate. You are free to decide not to participate in the interview and can withdraw at any time without disrupting your relationship with the researcher, Virginia Tech or eXtension. I appreciate your valuable time in helping with this important study. Respectfully, Mike Lambur, Ph.D. Evaluation Chair
eXtension Evaluation and Research Leader & Research Professor
Virginia Tech
288 Litton Reaves Hall
Phone: 540-231-8177
mike.lambur@extension.org Jeffrey A. Richmond
Graduate Research Assistant
Virginia Tech
288 Litton Reaves Hall
Phone: 540-231-8177
jrichmo@vt.edu ==Pilot Test Facilitators Interview Guide== Interview Guide
Facilitator Interviews
Annie’s Project Facilitators '''Problems''' 1. Were there any problems introducing the IFYF course to your class?
2. What were some problems with the course that you noticed in your class?
3. Were there any problems with the survey?
4. Were there any problems associated with the online navigation of the course?
5. Were there any problems accessing the course?
6. Were there any problems with the exercises in the course?
'''Questions''' 1. What were some of the questions participants asked about the course?
2. Were there questions about the survey format?
3. Were there questions about content of the survey?
4. Were there any questions regarding anonymity?
5. Were there any questions about accessibility of the course?
6. Were there any questions about the flow of the course? '''Concerns''' 1. What were some concerns participants shared?
2. Any specific concerns about participating?
3. Any concerns about taking online course?
4. The units were too short/long to complete?
5. The course does not capture this audience?
6. Were there any other concerns about the course? '''Improvements''' 1. What are some improvements participants discussed in your class?
2. What were some suggestions participants made to you during the course?
3. What specific criticisms did participants mention to better the course?
4. Did participants feel the course was easy/difficult to understand the language?
5. What would enhance participant’s experience with this course?
6. Were there any other criticisms of the course in your class? ==Email to Paid Participants== Thank you for agreeing to pilot test the Investing for Your Future for Farm Families (IFYFfFF) online course. This online course is designed to help small farm family households’ financial and investment decisions all across the country. The purpose of the pilot test is to have individuals work through the online course and then provide feedback to us through an online survey so that we can make needed adjustments in the course before we release it nationwide. As a pilot test participant, we would like you to 1) enroll in the online course (see enrollment instructions below), 2) work through the units in the course, and 3) provide your feedback on the online course by responding to an online survey. Upon completion of these tasks and in appreciation for your time in helping us, we will provide $100 to you. This will be a pre-paid Visa card. Please note that we may need additional information from you to provide this incentive payment. Please note that our timeframe for this pilot test is urgent. We need your review of the online course and completion of the pilot test survey by May 15, 2009. To enroll for the IFYFfFF on-line course, please do the following: 1. Go to http://campus.extension.org/ 2. Choose the Create an account link in the login box on the left hand side of the page. After entering the registration data an email will immediately be sent to you. 3. Check your email account for a message from remote learner admin with a verification link. Be sure to check your junk mail folder. 4. When you click on the verification link you will be taken to a page that says your registration has been confirmed with a courses button. Click on that button and you will see a listing of course categories. Choose Personal Finance and click. 5. Choose Investing for Your Future for Farm Families. On the middle of that page you will be asked for an enrollment key. The key for the course is “oie2008ap”. Upon entering the key and hitting enter you will immediately be put into the course. 6. Suggestion: Work through the Lessons menu to see how to navigate the course. 7. When you move through the course, there will be some unique individuals to give you a farm and ranch business perspective. Caution: if you move the mouse off the picture, the dialogue restarts at the beginning. 8. To return to the course you can either visit http://campus.extension.org/ and find it from the list or go to the link in step 1. 9. After you enroll, a welcome e-mail will ask if you want to update your profile. This is optional. Thanks in advance. Please let me know if you are willing and able to participate in our pilot test by May 15. And do get in touch with me if you have questions or problems. Sincerely, Michael Lambur ==Pilot Test Participant Survey Results== [http://tinyurl.com/OIE-Pilot-Test-Survey-Results Pilot Test Participant Survey Results] This url will take you to the overall results from InstantSurvey. ==Pilot Test Facilitators Interview Results== Summary Facilitator Interviews Annie’s Project Facilitators (N=4) '''Problems''' *Registration (number of steps cumbersome)-process was intense *Passwords not found readily (junk mail spam filtered) *Difficult to get back into course and find where left off *Need more detail about what is in packet to cover course- the 11 units that were part of the course *Course needs specific website address instead of finding the link on the website *Need more titles for the chapters to find easier *Not everyone checks emails '''Questions''' *Why no demographics collected in the course *Security of personal information *How in depth would the course be- would it be at a deep enough level to cover needs *Why so many procedures to get to the course *Some concerned whether the units and the material would be over their heads *Terminology is challenging when using “investment” and scares people *Combined financial statements, free credit check tools, and questions on the use of the forms to evaluate a broker *All came up multiple times- several wanted to encourage others to participate *Anonymity- did not want emails and phone numbers to be released *Difficulty finding where they left off in the course '''Concerns''' *“Is it worth my time?” *Wondered if there was a way to get feedback and a way to discuss things *Discouraging that there was no discussion board in the course *Curriculum appeared to be at the middle of the audience level *ID and password obtaining was troublesome *Participant Anonymity- how much private stuff would be made public in doing this course *Not much experience with online course before this *Passwords retrieved from emails was difficult *Middle aged group was more engaged than older group (35-55-middle aged) *Time of year for this course was bad *Investing scares people *Email anonymity *Real estate section seemed a bit scary to some '''Improvements''' *More indexing for each unit to the side of the webpage would help *Need better indexing upon entering each unit *Take some pictures off, maybe too much since took while to load webpages *Course will be more useful and beneficial down the road when younger crowd is majority *Include online group chat discussion *Enrollment issues- stumbling block for several people in the course *Maybe offer as a series of courses *Need an online discussion board or chat room *Participants were not sure this course was worth their time- make the portal to course more attractive *Need discussion area inside the course *Maybe add a facebook page for this course ==Professional Instructional Designer Review of the Course== '''Macro Evaluation Report of Investing for Your Future for Farm Families''' ''' '''INTRODUCTION''' This evaluation was done at a macro level. Many of the existing instructional problems seem to be within the user interaction as a whole rather than within the specific units. I feel that the content and activities contained within the units are very strong and useful. Therefore, I focused methods on the usability taking into consideration the learner characteristics, objectives for individual units, self‐paced learning strategies and the instructional context. The results and recommendations were developed as a problem ~ solution in an effort to provide a concrete method for clearly defining and fixing the areas which could use adjustment. They are broken out by sections of the module. Key words were included to offer some self‐research methods if desired. '''METHODS''' The following section outlines the specific areas that were taken into consideration during the evaluation of this module. These will discuss the points of reference for the reason behind the evaluation, learner characteristics, contextual analysis and usability. All of these will serve as an integral part of the success, impact and flexibility of the module. '''Origin of materials and cause for evaluation''' These materials were originally developed as a paper based instruction. It was implemented into an online learning module a few years later and implemented into the Moodle LMS last year as part of eXtension. A pilot study was run last year and the most significant issue found was that individuals had a difficult time getting past the introduction. The Subject Matter Experts (SME) of this online learning module desire the overall effectiveness of the stand alone learning module reviewed to measure effectiveness and increase impact. There is also interest in developing additional interactive materials, if there are areas which could potentially benefit from doing so. '''Learner Analysis:''' ''General Characteristics'' The general characteristics of our student users were defined as individuals of a variety of ages, incomes and ethnicities. All users are expected to gain information about future financially planning and diversifying their investments. The individuals would be guided to the site by their county extension officer and likely complete these lessons at home in their personal time. Student users will presumably have a family, a busy schedule and be looking for quick and simple ways to gain information and improve their financial standing. These students will predominantly be classified as adult learners. An adult learner is an individual who has already completed a formal form of education (e.g., middle school, high school, college) and is returning to gain expertise in a specific area. In this case, they are farmers, returning to gain specific training in investing for their future. There has been extensive research done on adult learners. Here are some specific, quoted points listed in Morrison et al. (2001) outlining information on adult learner characteristics. *Adults enter an education or training program with a high level of motivation to learn. They appreciate a program that is structured systematically with requirements (objectives) clearly specified. *Adults want to know how the courses content will benefit them. They expect the material to be relevant, and they quickly grasp the practical use of the content. *To adults, time is an important consideration. *Adults bring to a class extensive experience from their personal and working lives. These experiences should be used as major resources for helping student relate to the subject being studied. These characteristics will be examined throughout the document as they are integrated with solutions to the existing problems. '''Contextual Analysis:''' ''Instructional Context'' The instructional context will be predominantly online. Student users will derive all content and materials from an online Learning Management System – Moodle. Most of the content is currently static but there are several exercises available for print and several quizzes and short animations throughout the units. When the student users interact with this module they may have distractions such as noise or poor equipment. These are mostly out of the developers’ control. However, it will be beneficial to the student users for developers to consider these and try to offer useful opportunities for them to interact and gain meaningful learning via different methods. Throughout the document, these opportunities will be pointed out as suggestions for improvement to existing problem areas. '''Usability and how it relates to an online learning module''' Usability is an essential part of the overall effectiveness and impact of an online learning module. Nielsen (2003) defines usability as a “quality attribute that assesses how easy user interfaces are to use. The word “usability” also refers to methods for improving ease‐of‐use during the design process.” Nielsen defined the five basic quality components to break this down: learnability, efficiency, memorability, errors, and satisfaction. Learnability is defined as how easy the user can complete a basic task/action the first time they navigate a site. Efficiency is a measure of how quickly the layout and design of the site allow users to complete a task. Memorability refers to how easily it will be for a returning user to navigate and use the site. Errors is a measure of “how many errors (i.e., mistakes) do users make, how severe are these errors, and how easily can they recover from the errors”. Satisfaction is how positive the experience of using the site is. The navigation and design of this site should be set up to satisfy these five components in the following ways (Nielsen, 2003). '''Nielson's Usability Components - Description''' *Learnability - Signaling and placement of important components *Efficiency - Organization of and naming conventions of navigation, relevant graphics *Memorability - Meaningful navigation and content structure *Errors - Ability to move freely *Satisfaction - Enjoyable, efficient, useful '''RESULTS AND RECOMMENDATIONS''' This section will outline the potential problems that were found with the site when evaluating it against the methods and give suggestions for solutions. Once some of the suggestions have been implemented, a re‐evaluation of the materials will give stronger insight into other areas of the module which may need enhancement. '''Discussion of Objectives''' Objectives serve as the building block for the student user to build knowledge and as the foundation for an effective evaluation. To be able to maximize the time spent in evaluating your learning module, the objectives for the lesson must be available to the student user and clearly stated. Here are some specific, quoted guidelines in Morrison et al. (2001). *Use a statement that clearly indicates the behavior the student needs to master. *If there are several objectives for the unit, create more general statement o keep the list less than seven items long. Too many objectives will place too many requirements on short‐term memory, resulting in confusion rather than mastery of the material. *Write the objectives in a style the learner can understand (e.g., “At the end of this unit you will… “ as opposed to, “At the termination of the instructional presentation, the learner will be able to… “). *Objectives are less effective with units of instruction that are longer than 2,500 words. Researchers theorize that it is too difficult for learners to remember the objectives and the content for lengthy passages. As a result the effectiveness of the objectives as a pre-instructional strategy is decreased. '''Discussion of Self‐Paced Learning''' With the realm of self‐paced learning, activities which relate back to content and tie into well developed objectives are essential. It increases active and meaningful learning; motivating the student user. Study guides, worksheets, multimedia, quizzes and activities are all examples of self‐paced learning. Because these components are key, they need to be readily available to the student user and assist the student user to integrate with the learning materials. '''Discussion of “How to take this course”''' ''Problem:'' Not very many users actually take the time to go through a how‐to, despite the usefulness information available to them. ''Solution:'' Develop a 20‐30 second screen recording using a tool such as Snag It with voice over for the technical use of the site and include it within the introduction. This will be a very quick and easy way for student users to become acclimated and clarify site components while they work their way through the module. '''Discussion of Introduction''' The introduction was one of the bigger problem areas for the module when the pilot study was done. I have outlined the most impacting problems that I discovered and given suggested solutions. ''Problem:'' There is too much information and very little integration within the units. The introduction serves as a launch pad and first impression for the module. If the introduction is vague and long, the student user will likely feel overwhelmed and lose interest. This initial loss of interest may discourage users from continuing with the module. ''Solution:'' Reduce the size of the introduction significantly, set meaningful and motivational objectives. A good idea might be to set up the introduction as a method for the student to assess their current knowledge level and set a clear foundation for new knowledge to build on. The assessment could be printable once the user has filled it out to serve as an external organizer for the rest of the module. Each of your student users will be coming to this site with a different amount of prior knowledge. By developing an exercise that is laid out as a sitemap of the entire module with corresponding questions (5‐10 per lesson), you can assist the users to score themselves on each of the individual units. This will not only give you insight on where your student users previous knowledge and experiences, it will also serve as a motivational and usable tool for your student to begin. ''Problem:'' There are not any suggestions for the student user to get their families involved in the lesson. ''Solution:'' Develop some aspects of the curriculum with tools to empower the whole family on and off of the computer. The exercises could be developed to be a group activity and potentially give parents ideas on getting the children involved. Since this will be an online course and adult learners like the social aspects of learning, giving them opportunities to engage with their spouse, family or business partner would likely lead to a more meaningful learning experience. The option to print out an entire lesson or an abbreviated outline would be desirable to some and would give the user more control on the environment in which they are working in. Another aspect of our student user characteristics is their desire to bring to a class extensive experience from their personal and working lives. These experiences could serve as a very strong foundation to attach new knowledge onto. ''Potential idea for future development:'' Along the same lines of getting families involved, another possibility for future development would include integration with a learning community. Student users can interact with others to get ideas and discuss materials. ''Problem:'' There is not a strong source of motivation. Motivation is really the sum of all of the parts. If your introduction is well organized and the objectives are clear the student is empowered. ''Solution:'' Give the student user an introduction that is organized, visible, with a clear purpose, and consisting of relevant content within itself and with the rest of the module. '''Discussion of Units''' ''Problem:'' The menu does not give the student user an indication of the size and organization of the module. ''Solution:'' Develop a sitemap for the curriculum. Each page should have a title, if a title cannot be derived from the content, it should then become part of another page which does have a title. This will also assist in navigability throughout the module. Users should all have the capability to skip around the entire module freely and not feel restricted in doing so. This is especially true for returning users who are likely just looking to quickly reference a section of content or activity. The titles for each of the units and subunits should be named appropriately and there should not be any hidden content. Sitemaps are a very meaningful way to hierarchically organize information online and give student users a very tactile and approachable sense of the learning module as a whole. Progress bars can certainly be useful but in this case, Moodle does not seem to have a very intuitive progress bar, so it is difficult to use it as a tool. A good example of the importance of a sitemap which depicts the size and scope of the lesson is the tactile sensation you get when you pick up a book. Based on the thickness and weight of the book, you can quickly assess the length and time it will require for you to complete. These can be important factors with a busy lifestyle that most of us have. In addition to the size reference, you can also quickly page through the book to see some of the components and get a feel for whether or not this book is what you have been looking for. Similarly, the structure of online learning modules needs to be readily accessible so that users can easily grasp the attention to detail required when integrating information at various scales. As a final concern for the sitemap’s importance, the research done on learner characteristics of adults tells us that they want to know how the courses content will benefit them. They appreciate a program that is structured systematically. Therefore, spending adequate time developing the site navigation will be a key component to the success. ''Problem:'' There are 3 separate names for each section of the module, “unit”, “lesson” and “topic”, this reduces memorability and efficiency. ''Solution:'' Consistency in naming convention will reduce mental overload, any one of these will be a good choice so long as it does not change. ''Problem:'' The graphics are not always relevant and enhancing to the text and objectives. If imagery does not relate to the objectives, it can actually negatively impact the effectiveness of the content by confusing the user. ''Solution:'' Remove anything that does not enhance content and objectives to ensure that all content is relevant. The user should be able to quickly relate the graphics to the content. Graphics can be very useful, because they can integrate large quantities of information and confusing or abstract subject matter. Pictures and graphics can play several roles within content. They may be decorative, representative, organizational, or interpretational. In the case of adult learners, they tend to dislike pictures and graphics which do not serve a specific purpose. If a picture or graphic is representative, organizational or interpretational in ways which enhance content and objectives it will complement the instruction and satisfy the student user. If the graphic or picture does not serve a purpose and is merely for decoration, it may simply be a distraction. ''Problem:'' Not all of the units have objectives and if they do, they are not consistently placed within the unit structure. ''Solution:'' Make the units consistent with each other, have the objectives be the first thing that the student user sees. The objectives should also be part of the menu structure so that the student can easily navigate to be able to see what the lesson is about. Additionally, the objectives need to be clear and concise, thus allowing the user to quickly assess the units content in relation to their own experience. ''Problem:'' There are activities but not a general location to download the activities. ''Solution:'' Set up an activities pool at the beginning of each unit, make sure their location is consistent between units. Maybe each unit has the objectives listed and then exercises (with explanations) available for download. Ensure that these elements are also included in the menu structure because they will likely be popular. Remember that in self‐paced instruction, these activities are an integral part of the students’ success and satisfaction. ''Potential idea for future development:'' The exercises could actually serve as the basis for navigation throughout each unit. Setting up the activities to be interactive with the rest of the unit content would promote active learning for the students, which is very important when there is not an instructor present. ''Problem:'' The quizzes are not measured in a visible location. ''Solution:'' This would have more to do with the actual capabilities of Moodle. I’m not sure if you would have the option to place a score measure on the page per unit or at a cumulative level but it would likely be motivational for the student to be able to see how they were doing and continue to improve their score. '''Keywords for further research''' Usability Information Architecture Sitemap Naming conventions Adult Learners Self‐paced Learning '''CONCLUSION''' I believe that the content of this learning module is very useful content. The largest areas of concern are the sitemap and the introduction. If the issues with these two areas can be resolved, the usefulness and impact of the module will be increased significantly. The second area of concern is the placement and consistency of the objectives and unit activities. The reality is that not all of the students are going to want or need (based on previous experiences) the information on each of the 11 units. Users will want to be able to clearly see what they have to gain from the individual units at a quick glance and make the decision for themselves. In addition to this, some of the students may just be looking for a quick way to use the tools provided in the module to manage their existing knowledge. Many of the activities that are included within the content will be excellent instructional tools to accomplish this, but there is a high probability that users may not be able to locate the necessary information if they are not planning to complete the units. '''REFERENCES''' Morrison, G R., S.E. Ross, and J.E. Kemp 2001. Designing Effective Instruction, 3rd Edition. John Wiley & Sons, Inc. Nielsen, Jakob (2003). Usability 101: Introduction to Usability, http://www.useit.com/alertbox/20030825.html [[Category:Educational Content]] [[Category:Portal]] [[Category:Online Investor Education]]

Watching: OIE Pilot Test Documentation

OIE Pilot Test Documentation