Talk:PDCoP Evaluation

From Extension Collaborative Wiki

Jump to: navigation, search

Members of Evaluation CoP Leadership Team 2009-2010:

Benjamin Silliman, Department of 4-H Youth Development and Family and Consumer Sciences, P.O. Box 7606, North Carolina State University, Raleigh, NC 27695 ben_silliman@ncsu.edu; PH: (919) 515-8485; FAX: (919) 515-7812; website: nc4-heval.wikispaces.com

Heather Boyd, 111 Hutcheson, Virginia Cooperative Extension, Blacksburg, VA 24061 Email: hboyd@vt.edu Phone: (540) 231-9423

Suzanne LeMenestral, USDA-CSREES 1400 Independence Avenue SW, Stop 2201, Washington, DC 20250-2201 Email: slemenestrel@csrees.usda.gov; Phone: (202) 720 – 2297; Fax: (202) 720 - 9366

Mary Arnold, 105 Ballard Hall, Oregon State University, Corvallis, OR 97331 E-mail: mary.arnold@oregonstate.edu ; PH: 541-737-1315

Michael Newman, College of Agriculture and Life Sciences, Box 9745, MSU, Mississippi State, MS 39762; Email: mnewman@ext.msstate.edu; Phone: 662-325-3462; Fax: 662-325-5858

Joseph Donaldson, UT Extension Evaluation and Staff Development, 2621 Morgan Circle, 212-D Morgan Hall, Knoxville, Tennessee 37996-4525, E-mail: jldonaldson@utk.edu Phone: 865-974-7245; Fax: 865-974-0882

Allison Nichols, West Virginia University Extension, 603 Knapp Hall, Morgantown, WV 26506-5031 E-mail: ahnichols@mail,wvu.edu PH: (304) 293-2796

Sorrell Brown, 101 Curtiss, College of Agriculture and Life Sciences, Iowa State University, Ames, IA 50011Email: sorrel@iastate.edu; URL: http://www.ageds.iastate.edu/undergrad.htm; Phone: 515-294-8802

Melissa Cater, LSU Extension, Courthouse Building, Room 302, Harrisonburg, LA 71340-0160 E-mail: mcater@agcenter.lsu.edu PH: (318) 744-5442; Fax: (318) 744-0430

Rebecca White, Family & Child Development, LSU Extension, 221 Human Ecology, LSU, Baton Rouge, LA 70803 E-mail: bwhite@agcenter.lsu.edu Phone Number: (225) 578-3921 Fax Number: (225) 578-2697

Michael Duttweiler, Cornell Cooperative Extension, 383 Roberts Hall, Ithaca, NY 14853 Email: mwd1@cornell.edu; Phone: 607-255-2105; Fax 607-255-0788

Kimberly J.M. Keller, Evaluation Coordinator, College of Human Environmental Sciences, University of Missouri Extension, 1205 University Ave., Suite 1100, Columbia, MO 65211 E-mail: KellerKJ@missouri.edu; PH: 573-882-1195; Fax: 573-884-5412

Paula I. Threadgill, MSU School of Human Sciences, Box 9745, Mississippi State, MS 39762 E-mail: paulat@ext.msstate.edu; PH: 662-325-3462; Fax: 662-325-8188

Lisa Townson, Extension Specialist - Program Development and Evaluation, University of New Hampshire Cooperative Extension, 201A Taylor Hall, 59 College Road, Durham, NH 03824 Email: lisa.townson@unh.edu; PH: (603) 862-1031; Fax: (603) 862-1585

Elijah Wilson, Extension Agent, 4-H YD , University of Kentucky Cooperative Extension Service, , P.O. Box 39; 130 Keen Street, Burkesville, KY 42717 E-mail: elijah.wilson@uky.edu PH: (270) 864-2681; Fax: (270) 864-4191; Cell: (270) 459-2009

Contents

Scheduled Conference Calls for 2009

8/11 2:30-4:30 EST facilitator: ?

9/15 2:30-4:30 EST

facilitator: ?

10/14 2:30-4:30 EST

facilitator: ?

11/3 2:30-4:30 EST

facilitator: ?



Meeting Minutes 2009 Core Leadership Team

Meeting Minutes May 2009

May 6, 2009 Core Leadership Meeting eXtension Evaluation Community of Practice Meeting Notes

Present: Ben Silliman, Heather Boyd, Suzanne LeMenestrel, Michael Duttweiller, Sorrell Brown, Allison Nichols, Melissa Cater, Kimberly Keller We missed: Mary Arnold, Michael Newman, Becky White Below is a first draft of the notes, using the proposed agenda as guidance. I. Core principles and operating structure as a CoP (intro discussion estimated 30 minutes)

A. What do we want as our ground rules for CoP leadership group discussion? B. How shall we structure ourselves as a group? C. In what ways do we want to work together? D. What shall we consider our working mission, vision and values? E. How much time might we each be able to commit between now and November 2010?

We discussed these as our ground rules and values:

Ten Tips for Successful Learning Communities Nancy Franz, April 15, 2009 Webinar • Founders must have vision • Clear purpose/niche/focus • Strong convener(s) • Interactive Simple vs. complex • Democratic • Easily accessible • Monitored • Learning over task • Learning over information

We emphasized these concepts:

Interactive Democratic Learning over tasks Learning over information Low stress yet still productive

We will calendar meetings between now and the end of 2010. We assume that we are working virtually/online considering the budget issues Extension systems are facing. Heather will get help from her administrative assistant to get dates set.

We reminded ourselves of our purpose and mission.

A reminder from one of our members is: Ours is a Professional Development CoP which by definition is internally focused and separate from public eXtension content. (http://collaborate.extension.org/wiki/PDCoP_Application_Procedure) Success is defined internally: "The ultimate success of all PDCoPs is the positive demonstrable impact on these faculty/staff."

II. Review the accepted plan and scope of work http://about.extension.org/wiki/eXtension_Evaluation_CoP#CoP_Application

A. Scope of Work for the proposed PDCoP through December 31 2010. (estimated 40 minutes) Number of Moodle courses created: 3 For the next conference call, we discussed having an informal review of Moodle modules that Joseph and Mike created. Joseph and Mike both generously said that we could review them for possible use by the CoP. We did not decide on what would constitute an informal peer review during the conference call. For the next call, we will discuss: 1) suggestions for the authors; 2) things we learned about program evaluation in Extension from the modules and 3) whether we would place these Moodle courses on our site. What other suggestions does anyone have for this peer review process? Joseph’s Course 1: "Program Planning and Evaluation" is intended to be the first program planning and evaluation course an agent or specialist takes when employed. I developed the course content, and Richard Karel assisted me with developing the files. Richard used Captivate to crate the "movie-like" features.

http://extol.tennessee.edu/extolsdbx1/

All members of the CoP should click the Course Category labeled "For Guest Access" to see the two courses I created. Access does not require login.

Joseph’s Course 2: "Outcome Measurement" is intended as a six-week, structured course for more experienced specialists and agents. I created this course in Moodle. So, it has a completely different look and feel than the "Program Planning and Evaluation" course. I intended to offer it about twice a year over the six-week period. This course was developed as part of my Ph.D. program to demonstrate instructional technology competence. (You know evaluators, we have all these competencies we have to demonstrate with authentic assessment!) Anyway, this is a very bare bones course, but I do think functionally, it is a good course to demonstrate Moodle features to those who are new to the platform. Mike’s Course Options: Mike is willing to facilitate a guided discussion. Access to his courses is more limited. See his note below: If you want to take a look at the courses, indicate your name and preferred times in this Doodle poll. If the dates don't work, Mike will come up with another one or two. http://www.doodle.com/p64aa6gq477xwwzy

Number of published FAQs: 30 2-3 topics per core leader Content outline: 90 4-H Evaluation for Impact Content ACA Program Evaluation Standards Number of basic information pages: 8 (overview plus one page for each Evaluation for Impact area)

B. Address how we can go about planning for the pieces we committed to, whether we want to make changes, etc.

C. Review commitments made to complete different portions - who committed to what o Allison Nichols —Analyzing and Interpreting Data o Ben Silliman—Methods and Tools o Becky White—Program Planning o Melissa Cater—Collecting and Handling Data o Lisa Guion – Multicultural issues o Whom did we miss from the last volunteer round?

We had additional (or reminder) volunteers for the remaining modules. Michael Duttweiller – Communicating Results Sorrell Brown – Help with a marketing plan for marketing the site and Community to the Extension system Kimberly Keller – Focusing Evaluation By the next conference call, each issue area person is going to put some more detail around how they see their roles and what they consider as part of their scope. Each issue area person will also oversee the FAQ list related to their topic so as to have broad understanding and to track the content.


Additional volunteers needed: 1) Tech person 2) Liasion with Mike Lambur for tracking use of the CoP, evaluating user experiences, etc. 3) Other?

III. Plan for another number of teleconferences and possible in person meeting at AEA (estimated 20 minutes)

We acknowledged that budgets would probably not allow an in-person meeting at AEA, so we will plan to meet online for the remainder of 2009.


Meeting Minutes July 2009

July 22, 2009 Core Leadership Meeting eXtension Evaluation Community of Practice Meeting Notes CoP Leadership Committee Conference Call agenda July 22, 2009 3:30-4:30 PM EST

Dial in information (this is a teleconference, not an Elluminate session) 866 842 5779 Code 540 231 9423

I. Greetings II. Updates on reviews of web based courses from Mike and Joseph and those who participated in the reviews of their courses (Mike and Joseph – If you can’t make this call, we can put this discussion on a subsequent agenda…) III. Beginning discussion of our (budding) peer review process – How do we want to structure this? The process? The rubrics? (Attached is a tool Joseph has used that we could consider as part of the conversation.) Next steps for peer review process and decisions IV. Review of “job descriptions” that section leaders were going to craft for their portions of the From May 6, 2009 meeting minutes By the next conference call, each issue area person is going to put some more detail around how they see their roles and what they consider as part of their scope. Each issue area person will also oversee the FAQ list related to their topic so as to have broad understanding and to track the content. Program Planning for Effective Program Evaluation Becky W Focusing an Evaluation Kim K. Evaluation Questions and Designs ? Evaluation Methods Ben S. Collecting and Handling Data Melissa C. Analyzing and Interpreting Data Allison N. Communicating Evaluation Results Mike D.

If possible, please send your job description to the committee prior to the call so we can reflect ahead of time. V. Wrap Up

Next call August 11, 2009 2:30-4:30 EST Connection details to come Sorrel will facilitate Topics include marketing the CoP

Meeting Minutes August 2009

Minutes taken by Ben Silliman

Present: Boyd, Silliman, Cater, Brown, Nichols, White

Marketing Discussion August 11, 2009 Considerations for discussion in branding the Evaluation CoP:

• Determine our objective What is the objective of the Evaluation CoP? -Evaluation content and help to Extension (Alison)

Why isn't that objective posted on our CoP Welcome page? - Is our objective quantifiable? Does it have to be? -We need to tally interest, use (Becky) -e-Ext does track wiki pages on Google Analytics (Mike) --Pop-up question: Not before content is posted (Mike); in-site surveys can be assessed for consistency, confirmed with more than a pop-up survey (Mike)

  • Identify our target audience (internal to Extension but how deep?)

Who can help us achieve our objective? -We need to decide on ‘first audience’ (specialists, agents, administrators) will shape the look of the site -“All Extension professionals,” reflecting different states’ varied approaches to program evaluation (Becky) -Are there any ways of knowing who in Extension is using eExt (Heather) -eExt not able to differentiate site users, only general profile (Mike) -We may need to target role (e.g., end-user vs trainer) rather than institutional role -Who should we target (Sorrel) -Focus on specialists (Alison) What do they believe in? What can they do for our CoP (and what can we do for them)?

  • Determine our brand positioning

Identify our frame of reference (is it eXtension?) -eExt is simply a delivery platform-our focus is evaluation (Mike) -Create greater program evaluation capacity (Becky) -Develop some general resources, then look for another ECoP that can be a partner with us and create an evaluation model and capacity-building record (Ben) -Educational focus (Melissa) -Explicit training on evaluation is not all that effective in some states—Get involved with a project (Sorrel) Identify our reason to believe in what was are about (education for Extension staff?) Identify our point of difference? How is our CoP different from others within our frame of reference? (internal, not public)—Affirmed by all What benefit do we want people to associate with our CoP? -Greatest benefit is how Extension professionals are able to improve programs (Becky) -Key = easy to use and understand (Heather) -Let specialists and agents tell us what they need most (Ben) Identify the reason to believe in what we are about. What is the most convincing support behind our point of difference? Develop our positioning statement (from our frame of reference, point of difference, and reason to believe).

  • Evaluate our CoP's name

Does our name reflect our objective and clearly communicate what we do? -Professional development for Extension professionals (Becky) -Illustrations on how evaluation is done will help people get beyond the tool only (Ben) When used consistently, a tagline can help clarify a name. Do we need one? -We need a tagline for a more ‘retail orientation’(Sorrel, Becky)

  • Determine our marketing objective

What does our target audience need to do in order to achieve their objectives? -Find and use resources, then share successes with peers (Becky) -Marketing survey is the first step (Sorrel)

  • Determine the marketing challenge

What behavior or belief needs to be overcome (in us or in our target audiences) in order to get to our marketing objective? -Create credible, relevant help and ways to bring their attention to resources on the web (not a place they are used to looking) (Sorrel) What venues are most appropriate for us to promote our CoP? -Where do people find out about eExtension resources now—esp. Professional Development site internal to eExt (Becky) -eXtension is working on some internal professional development resources (Mike) -Our situation is similar to that described by Patton in relation to developmental evaluation (Mike)



FAQ Knowing how to navigate Important to identify types of questions, then group starts addressing them We need to be more systematic about how to approach Review Guidelines and Process, samples

Better KidCare America sample -Begin by generating lots of questions and put them online

Most members concur with initiative to ask questions of evaluation specialists and professional development specialists

Post our content first, then ask for input

FAQ x categories Planning--Becky Focusing—Kim Questions and Design-- Methods—Ben Silliman Data Collection—Melissa Cater Analysis and Interpretation—Alison Reporting—Mike

Intellectual Property/Contributor Agreements (see about.extension.org wiki site) Dan Cotton—“All contributors retain possession of their original material…” --You only give permission to use material (not sign over ownership)


NOTES on NEEDS ASSESSMENT FOR FAQs

During our CoP leadership meeting this week, we discussed generating FAQs for the Collaborative Wiki. We discussed that the chairs of our sections each recruit a few people for their subgroup in order to generate approximately 10 questions for FAQs. Would section chairs be willing to do this? At the same time, Becky volunteered to create a survey where we solicit state Extension evaluation specialists to provide us with their “five top FAQs” they encounter from Extension professionals in their state. We might include a drop down box where each FAQ is categorized into one of our subgroup sections below. This survey would go to our 71 CoP members, the EEE-TIG listserv and the PSD Southern Region listserv, among other places, perhaps. We also discussed that our peer review group might tackle the FAQs before they are posted. What do those who could not be on the call think of this plan? Any suggestions or changes? As a reminder, here are our sub group leaders: Program Planning for Effective Program Evaluation Becky W Focusing an Evaluation Kim K. Evaluation Questions and Designs  ? Evaluation Methods Ben S. Collecting and Handling Data Melissa C. Analyzing and Interpreting Data Allison N. Communicating Evaluation Results Mike D.


Allison, Becky and Melissa are taking the lead on FAQs. We need a section leader for Evaluation Questions and Designs. Who is willing?

Personal tools