Home » Archives for 2012

Introducing Digital Badges Within and Around Universities

Posted by Michael M.K. Tuesday, October 30, 2012 0 comments
Dan Hickey
Sheryl Grant from HASTAC recently posted a detailed summary of resources about uses of digital badges in highereducation.[1]It was a very timely post for me as I had been asked to draft just such a brief by an administrator at Indiana University where I work.  Sheryl is the director of social networking for the MacArthur/Gates Badges for Lifelong Learning initiative.  Her job leaves her uniquely knowledgeable about the explosive growth of digital badges in many settings, including colleges and universities.  In this post, I want to explore one of the issues that Sheryl raised about the ways badges are being introduced in higher education, particularly as it relates to Indiana’s Universities.
IU President McRobbie

Digital Badges at Indiana University?
As someone who sees huge potential in digital badges as recognition of learning and accomplishments, I was pleased that IU President Michael McRobbiementioned them in his recent 2012 State of the University speech.  IU promises to be a fertile environment for exploring the potential of digital badges.  Consider, for example:

●          IU was instrumental in the development of the Sakai open-source course management system, under the leadership ofBrad Wheeler, IU Vice President for Information Technology and CIO. The open-source ethos behind Sakai is consistent with the vision of the Mozilla Foundation who is responsible for the Open Badges Infrastructure (OBI).
●         President McRobbie (who was formerly a Professor of Informatics) recently announced an $8M initiative called IU Online.  This initiative is being led by Barbara Bichelmeyer, a Professor in IU’s top rated Instructional Systems Technologyprogram. As elaborated below, the university is carefully considering the role that badges might play in this effort.
●         Indiana IST Professor Curt Bonk is an internationally known proponent of massive open online courses.  MOOCs are one of the most promising contexts for introducing digital badges in higher education.
●         The International Society for the Scholarship of Teaching and Learning was launched from IU, and IU remains a major center of SOTL work.  The SOTL community could provide the sort of multi-disciplinary consideration needed to identify and refine the appropriate uses of badges in academic programs.

In previous blog post, I outlined some of the questions that universities might askbefore introducing digital badges.  One obvious question is: where to start?.  One of the things Sheryl pointed to in a comment on her blog is that faculty members and other university educators are working in and around universities to implement badges.  This is particularly interesting to me because the way that badges are introduced will impact how institutions learn to use badges.  This has consequences for how those badging practices impact student learning.

Working Around Universities to Implement Digital Badges
Sheryl’s post links to a bunch of higher educators who are introducing badges on their own.  I have been experimenting with issuing badges that say Indiana University in my small doctoral seminar.  IU Learning Sciences Ph.D. student Sophia Bender and I were recently interviewed about this in an article in the Chronicle of Higher Education. While the article’s title implied I was replacing grades with badges, I am actually just supplementing the existing grading structure. It is something that educators who are curious about digital badges should try.  It is an easy way for any educator to experience how it feels to make detailed claims about learning that your students might share with others over Twitter or Facebook.

A more formal badging effort is underway at IU’s InCNTRE networking lab.  They offer a ten-week summer internship program called Summer of Networking and a day-long workshop on OpenFlow networking technology.  InCNTRE Training Coordinator Steven Wallacerecognizes that badges are a natural extension of the existing certificates the inCNTRE lab offers.  Badges can easily provide detailed information about the programs that participants can share with others.  This should increase the credibility of that evidence of accomplishment and help InCNTRE attract new participants.  These workshops are non-accredited and take place outside of the formal academic programs of the IU School of Informatics.  So implementing these badges is a relatively straightforward affair.

Working Within Universities to Implement Digital Badges
Sheryl’s post also highlights more formal efforts to implement digital badges in higher education within accredited academic programs.  Readers who have been involved in efforts to reform accountability in academic programs know that this gets very complicated very quickly.  As I argued in a previous post, changing accountability usually changes assessment, and changing assessment often calls for changes in instruction.

One of the most ambitious formal efforts so far is the Passport System recently unveiled at Purdue University. I don't know much about Passport beyond the relatively straightforward technology components that they have created to allow instructors to create and issue OBI-compatible badges.  Indiana IST graduate Bill Watson helped design the system.  I assume that Dr. Watson and others are working through some of the complex (but consequential) accountability issues as they field applications of up to 200 instructors to participate in beta tests of the system.  This promises to be quite a test bed.

This post is an initial salvo in my effort to convince Purdue to systematically document the badge design policies that emerge in this effort.  This might be potentially more far-reaching than the badge design principles that my graduate students and I are documenting in the DML Design Principles Documentationproject. My project is documenting the reasoning behind the 30 MacArthur/Gates initial plans and practices for using badges. We are drawing on research in software design that captures this useful knowledge before it “evaporates” as features evolve and teams dissolve.  In a similar fashion, reasoning behind initial badging policies will be difficult to recover as policies evolve and committees dissolve. This knowledge will be invaluable to other institutions that are introducing digital badges.

President McRobbie's Approach to Instructional Innovation
It will be interesting to see how digital badges are incorporated in the IU Online initiative.  President McRobbie seems to have tied them closely to MOOCs in his 2012 State of the University address:

This new initiative will accelerate the development and delivery of targeted quality graduate professional programs on the core campuses, joint undergraduate programs on the regional campuses, key gateway courses university-wide, experimental massive open online courses—so-called MOOCs—and educational badges, in order to address Indiana’s economic and professional development needs, and to extend the university’s national and international reach. It will also help with the systematic evaluation and development of new technologies that will underpin the new directions in online education, and coordinate how IU can benefit from economies of scale in deploying these technologies across the university.

Touching on the distinction raised above above, McRobbie indicated that the initiative:

recognizes that the distinction between “traditional” and “non-traditional” students is increasingly blurred and that it no longer makes sense to use different strategies to reach them. It recognizes that all the online courses and degrees must be owned by the schools and campuses as online education is becoming an increasingly fundamental and integral part of what they do.

But he acknowledged the need for caution:

However, as we vigorously move forward with IU Online, we must nevertheless also maintain a skeptical and questioning approach—as is appropriate for a university—to some of the wilder claims being made about online education

As someone who is dismayed by explosive growth of truly awful online instruction, I appreciate these concerns. I was inspired by McRobbie's closing quote on the topic in comments that C. L. Max Nikias, president of USC made to his faculty in an August 2012 memorandum:

The Internet's first wave in the 1990s resulted in a dot-com bubble that was inflated by a fixation on the total number of users that a company's website could collect, rather than the true value that was created through a viable business model. Online education similarly lends itself to a focus on large numbers—yet there is scant evidence that free online classes or viral lectures produce worthy educational or career outcomes. [Our] academic community recognizes, at key inflection points within the development of higher education, that there is a difference between data and wisdom; between mere information and deep insight; and between knowledge disseminated and knowledge absorbed and appreciated. Our goal will always be to produce true academic value, for the fullest benefit of our students (emphasis added).

I really like the sentiment and tone of this observation. While it references the need for evidence, it does not endorse the dreary test-prep practices that currently dominate online education. For me, the highlighted points lobby against a simplistic "what works" search for "best practices" for increasing scores on static tests of content knowledge. 

McRobbie's words leave me with optimism for the IU Online initiative and the broader policies and practices that will follow. While some of us innovators may chafe under some of the restrictions that will likely emerge, these words imply a sensible, balanced approach that makes sense for an institution that has been around for nearly 200 years.



[1] For the latest examples check out the Scoop.it that Sheryl  maintains or another one for professional credentialing and for higher education.

Initial Questions About Digital Badges and Learning

Posted by Michael M.K. Sunday, October 21, 2012 0 comments
by Daniel Hickey
This post suggests some initial questions about learning that you might want to ask if you are considering using digital badges.  A version of this post is being prepared for the November 2012 edition of EvoLLLution magazine.  That article will consider how digital badges can be used to both enhance learning and recognize learning in ways that might help colleges and universities attract larger numbers of adult learners back to school.  This post poses these same questions in a more general context.

Since its announcement in September 2011, the MacArthur Foundation’s Badges for Lifelong Learning initiative has generated immense interest in digital badges as a transformative alternative for recognizing learning and achievement.  One of the products of this initiative is the EDUCAUSE brief Seven Things You Should Know About Badgesby Erin Knight and Carla Casilli from the Mozilla Foundation.  This succinct introduction defines badges as “digital tokens that appear as icons or logos on a web page or other online venue which are awarded by institutions, organizations, groups, or individuals, to signify accomplishments such as completion of a project, mastery of a skill, or marks of experience.”

Thanks to the MacArthur initiative, additional support from the Gates Foundation, and the efforts of Mozilla and HASTAC, roughly thirty diverse educational programs are now systematically incorporating digital badges. But growing appreciation of the potential of digital badges has spurred numerous other programs and schools to consider issuing badges. Because badges are so new, potential issuers have many questions.  Thanks to a burst of work at Mozilla, many questions (especially ones about creating, issuing, earning, and sharing badges) already have answers.  But many important questions can’t be answered yet. And when answers to those questions do emerge, it seems likely that many of the answers will start with “It depends...”  The questions concerning learning seem to be some of the most difficult.  This brief is an initial effort to identify the initial questions about learning that educators, programs, and schools might want to ask if they are thinking about issuing digital badges. Links are added for more details and elaboration.

Functioning
What learning-related functions will your badges serve?  All badges function to recognize learning; as such, most badging practices also function to assess learning. Existing learning systems tend to be organized around teaching rather than learning.  This means that deciding what learning to recognize and how to assess that learning can be surprisingly challenging. Recognizing and assessing learning serves to motivate learning.  But some of the motivational functions are likely to be unplanned and unintended.  Additionally, badging practices offer (mostly unexplored) potential for evaluating and studying learning.  Finally, these functions interact with each other in complex and unpredictable ways.  These functions and their interactions are explored here.

 Assessing
How will you assess the learning that you have decided to recognize with badges?  In many cases, badges will function as summative assessments of prior learning.  Badges can also function as formative assessments in support of learning, by providing guidance, feedback, and motivation.  Additionally, badges can function as transformative assessments that transform existing learning systems or allow new ones to be created.  These different assessment functions will interact with each other in maddeningly complex ways.  In particular, the more salient summative assessment functions can easily overwhelm and undermine intended formative and transformative functions.  This is elaborated here.

Validating
What sorts of claims will your badges make about the earners and what evidence will your badges contain to support those claims? Value is not inherent in the badge itself but in the assertions made by information the badge contains. Traditional notions of validity associated with psychological measurement and educational assessment seem insufficient to address the evidential questions raised by the summative functions of digital badges. Notions like credibility, face validity, and social validity that many measurement theorists dismiss as “unsanctioned” aspects of validity will be more widely embraced. Meanwhile, potential formative functions of digital badges give new importance to consequential aspects of validity, while potential transformative functions call attention to the emerging notion of systemic validity.

Theorizing
What assumptions about learning will frame your consideration and implementation of badges?  By pushing the conversation from teaching to learning, badges and related assessment practices will force many learning systems to grapple with assumptions about learning that have been taken for granted.  It appears that many of the summative functions and evidential aspects of validity can be adequately theorized using traditional “associationist” theories of learning embraced by many instructional designers and measurement specialists.  Arguably, some the formative functions of assessment and consequential aspects of validity call for modern “constructivist” theories of learning embraced by many educational psychologists and cognitive scientists, while some of the transformative functions and systemic aspects of validity will call for emerging “situative” theories of learning being advocated by many learning scientists. This is elaborated here.

Introducing
How will your badges be introduced?  Will it be a centralized effort or pockets of innovation?  Academic institutions may ask whether they will be introduced in a non-academic unit.  Attaching badges to existing assessment practices for an educational program is straightforward.  Adding badges and assessments for an existing educational program is complicated.  Creating an entire assessment and educational system around badges is even more complicated. The “mission creep” that often accompanies good assessment practices may introduce additional complications. That may happen with adding badging to existing assessments reveal shortcomings of those assessments.  More mission creep can occur when improved assessments reveal shortcomings of the instruction.  While the changes are transformative for learning, they can be disruptive as well.

Refining
How are you going to refine your badging practices?  As stated above and nicely elaborated by Carla Casilli, the initial design of badging practices is likely to be a bit chaotic, and you will probably end up doing some things differently than what you initially intended. A driving assumption behind Indiana University’s Badge Design Principles Documentationproject is that many learning systems will not appreciate some of the important aspects of badge system design until badges start actually being issued.  The project further assumes that many of these factors are going to be quite specific to the particular educational context in which particular badges are being awarded.  This means that the search for “best practices” for badges may be quixotic; a more productive question is likely to concern whether particular practices are appropriate in particular contexts. 

Protecting
How will you protect the rights of earners to control what happens with the evidence contained in their badges? Attorneys at Mozilla have been studying the privacy issues associated with digital badges, particularly as they relate to COPPA and FERPA. Privacy concerns also raise issues about badges and learning, particularly when the evidence contained in badges is used to learn whether or not programs are effective and to improve them. While this potential is largely unexplored, the metadata of each badge embodies the values of a particular learning system.  This metadata, and the information linked to it, can be used to evaluate programs. Badges offer entirely different possibilities for learning researchers.   Consider, for example, that some researchers are beginning to associate online attitude surveys with digital badges.  The evidence that badges will or might contain may be incredibly useful for in terms of accountability, evaluation, and research. However, collecting and using this evidence raises new issues concerning consent, ownership, and privacy. 

This document was produced with the support of the MacArthur Foundation’s Digital Media and Learning initiative, via its support for Indiana University’s DML Design Principles Documentation project.  Project members Elyse Buffenbarger, Rebecca Itow, and Andrea Rehak contributed to this brief. This document reflects the opinions of the author and does not necessarily reflect the opinions of the MacArthur Foundation or Indiana University.

Incorporating Open Badges into a Hybrid Course Context

Posted by Michael M.K. Wednesday, October 3, 2012 0 comments
By Dan Hickey
I recently incorporated digital badges into the online aspects of my doctoral course on educational assessment (“Capturing Learning in Context”).  There are two aspects of this effort that readers might find useful.  The first aspect concerns the way students award simple “stamps” to highlight significant contributions or insights from classmates. I use those stamps to award three “one-star” badges each week; I will use the one-star badges to determine how to award three two-star badges at the end of the semester.  I will elaborate on this in a later post.  I also removed the section on using the Mozilla Open Badge backpack to another post as well. This post is already going to be pretty long! 

In this post I want to describe how I used ForAllBadges (from ForAllSystems, a small Chicago firm) to issue digital badges within a typical online course management system (CMS).  Anyone who wants to issue badges that comply with Mozilla’s Open Badge Infrastructure (OBI) can easily sign up for a free account at http://www.forallbadges.com/.  The account can be used as a stand-alone site, or it can be accessed from within any CMS that lets you access outside websites.  I am using OnCourse, the Sakai-based open-source CMS that Indiana University helped develop.


Incorporating ForAllBadges into OnCourse

ForAllBadges Interface Inside of OnCourse
(Click on the images to enlarge them)

After my account was set up, I just had to enter the browser address into the OnCourse Web Content tool and labeled the link ForAllBadges.  Then my students just had to click the link to access that site within my class site.  I added each student as a participant by entering their email address and a default password. 


   


Creating Badges
Six Badges Created for my Class
One of my doc students, Rebecca Itow, helped me to create these badges (Rebecca's experience in P2PU's Badge Maker course was a great help in this process). In order to create badges, we first needed images.  We used a shareware program called InkScape (a tool introduced to us in the P2PU course) to create the images for each badge.  Given the nature of the badges and the class, we did not worry too much about aesthetics.


Interface for Creating Badges

Once the images are created, we just had to click on Manage Badges and then Add Badges to create the badge.  We only used the Name and Description fields. We left the Criteria field blank so it could be filled in when the badge was actually awarded.  You can see that we could have included a lot of information by using the other fields.  What makes ForAllBadges “OBI-compliant” is that these fields are consistent with the metadata structure that the Mozilla Foundation’s Open Badges projectdefined in 2012 as part of the MacArthur Foundations Badges for Lifelong Learning initiative.





 Issuing Badges


One-Star Collaborative Engagement Badge from my Class
Once the participants were added and the badges were created it just took a couple of clicks to issue a badge.  Because of the way I set up the badge, I entered the actual criterion when I issued each badge.  This was simply an explanation of why I awarded that badge to the specific student.  The students can easily see which badges each of them has been issued, and can click on any badge to see the criterion upon which it was awarded. 

I have been playing around some with the wording of the criteria each time I award a badge. This is a nice example why digital badges have such transformative potential.  Because I knew this badge might be “pushed” out beyond my class, I was forced to think about the information that the badge would contain.  And while I had decided to not make badges part of the course grade, I certainly hoped that students would value them. Particularly because every student could inspect every badge that I awarded, it was important that the criterion be credible at least in the context of the class. I still have some work to do to make the badges credible beyond the context of my class.

When I first issued badges, I included links back to the comments from classmates that led me to award the badge.  Those links would work for the students who were enrolled in my class and only then while they were inside my OnCourse site.  While those links supported credibility within the class, they were rather unwieldy.  More importantly, a dead link would undermine the credibility of a given badge when viewed outside of the class context.  So I removed the links. I will write more about credibility in the future, but for now check out Carla Caslilli’s excellent post about credibility and validity for digital badges.

In addition to credibility, I had to think about privacy. When I removed the links, I initially left in the first names of the awardees classmates whose comments led me to award the badge.  But I realized that if the badge was viewed outside of the class context, that some students might object to being identified. Even though it was just the first name, I suspect that putting such identifying information might violate FERPA rules that strictly protect the privacy of anyone enrolled in a public school classroom. The example above shows the badge I eventually awarded.  I left only the awardee's first name in the badge (it is obscured in the picture).  But I provided enough detail so that the students in my class could recognize whose endorsements I was referring to.

I am not sure how meaningful my one-star badges will be to anyone outside of my class.  But I have nine weeks to figure out how to make the three two-star badges more generally meaningful. I now realize that the challenge, if I want to be a badge-savvy educator, is to make the two-star badges so meaningful that the awardee might push it out to Facebook, Google+ and Twitter, so that others might see it and comment on it.  (I hope that this is the point where confused or skeptical readers start to realize why people like me are so excited about digital badges.)

If this is not clear or not helpful, or if you have comments, please let me know!  Now go forth and play!







   

Pushing Badges from ForAllBadges to a Backpack and Beyond

Posted by Michael M.K. 0 comments
By Dan Hickey and Andi Rehak

In a separate post, Dan explained he used ForAllBadges to issue OBI-complaint badges within the Oncourse course management system.  This post explains how these badges earners can "push" their badges out of the class and into their open badges backpack and beyond to Facebook, Google+ and Twitter.

This post is intended to be a very concise explanation for using backpacks when using the ForAllBadges platform.  In particular it highlights the fact that badge earners must have an open badges backpack before they can push their badges to it.

For more general guidelines, check out Mozilla's wiki on badges, information for badge issuers and the open badges FAQs. P2PU's Open Badges 101 sprint and  Mozilla Open Badges google group are also very helpful. If the terms like "issuer" and "earner" are confusing, check out Carla Casilli's blog on the open badges lexicon


Create a Mozilla Open Badges Backpack
The Open Badge Infrastructure (OBI) made it possible for Mozilla to create virtual “backpacks” where earners can store and display their badges, or push them out to other locations such as Facebook.  Initially Dan's students were a little confused about creating backpacks, but this is because the very idea of a virtual backpack is entirely new.  Once he created step-by-step instructions like the ones here, it was pretty simple to create the backpack before pushing a badge to it. Hopefully badge earners will be able to use these instructions as well.


Open Badge Backpack Login
First you go to to Open Badges.org.  There is stuff here that you can play around with but click in the middle where is says Visit Your Mozilla Badges Backpack. Or just go directly to the Open Badges Backpack  Click on the green sign in.

Mozilla's Persona Identity Verification System
This launches Mozilla’s Persona identification system.  Users must log on to open badges with the same email address at the one associated with that person at the ForAllBadges site (or whatever badging platform being used) that issued the badge.  Persona then asks for a password. Users can use whatever password they like but are likely to use the same one as they use for that email address.  

Persona then sends verification to the email address.  Clicking on that address verifies the users identity and leaves them in the Persona account manager.  This step is a little confusing, because from there the users need to go back to http://beta.openbadges.org/

Pushing Badges to the Backpack
Accepting a Badge from ForAllBadges into Backback
From the badge management page, users can click on their magnifying glass and then click again to send the badge to their backpack by clicking the blue Yup.









Prompt to Create a Backpack
If there is no backpack with the email address associated with the badge (shown in red),  the issuer is prompted to create one.  If so, clicking the blue Cool Lets Go takes uses to the steps above for creating a backpack.  Users need to create a backpack with that email address, and that first badge has to be pushed a second time.  At this point, I realized that many students might have preferred me to use their more Gmail address than their university email address. This gets at the crucial issue of permanence that the open badges community is dealing with right now.


Badge Added to Backpack
After a badge is pushed to a valid backpack, a message indicates that the badge was added.  Clicking on the blue hotlink takes the user from there to their backpack. Users can examine their badges in the backpack and push them out from there if they like.

Pushing Badges to Social Networks
Badges and Groups in Backpack
When users go back to their backpack at http://beta.openbadges.org/ they can see all of the badges in their backpack (but they may need to log back in).  The backpack will contain whatever badges and a single unnamed group.  Dragging a badge into the group allows the user to name the group and create additional groups.  Clicking on the Public box on the lower left means that others can see the badges in that group.


Clicking on the on the familiar sharing symbol on the lower right of group box pushes the badge beyond the backpack; clicking on the badge itself shows what anybody else will see if they click on the badge in your backpack or on the wild. Notice that the criteria and evidence have all been turned to hotlinks that go back the badge holder (which in this case is badges.forallschools.com).


The Information in the a Badge
Clicking on the Share This in the upper right of this window prompts the user to select where to share it (Currently limited to Facebook, Google+ and Twitter.  Users are prompted to add a comment or a tweet to go along with the post; if they are not already logged in to those external sites they are asked to do so. 


Badge Pushed to Facebook



Once the badge is pushed, anybody who gets the tweet or is friended in Facebook or Google+  can simply click on the link and see the same info like the one in in the picture above.  They can in turn click on the links to see more information.  Users can add additional comments and others can comment as well.  While the Facebook post can be deleted, a tweet obviously cannot be taken back.  However, the user can go back to the backpack and “disown” the badge and the link will be killed.




Tweeted Badge

Hopefully we have brought some clarity to the world of badge accepting; now go to the open badges homepage and actually earn a badge yourself by taking the Badges 101 Quiz!

Intended Purposes Versus Actual Functions of Digital Badges

Posted by Michael M.K. Sunday, September 9, 2012 29 comments

By Daniel Hickey
On September 4th and 5th, there was a meeting at the National Science Teachers Association in Arlington, VA.  Al Byers of NSTA and Kyle Peck of Penn State organized the meeting to discuss the online NSTA Learning Center for science educator professional development.  I was only able to make it to the second day of the meeting where Kyle discussed the pilot work with the site and his use of digital badges from the Teacher Learning Journeys project.  In the afternoon, Sunny Lee and Erin Knight (Mozilla Foundation) and Brian Mulligan (Sligo Institute of Technology, Ireland) and I did a panel on digital badges that Kyle moderated.. 

One of the questions about badges that came up seems like a crucial issue as we grapple with different ways of characterizing and describing badges.  This post aims to add the category of badge functions to other badge taxonomies like the one by Carla Casilli. Because these issues are complex, this post ended up being rather long.  You may wish to jump directly to the summary at the bottom.  You may also wish to read a condensed version at the HASTAC website.



Badge Purposes Versus Badge Functions
In that panel, I elaborated on a point that I introduced in an earlier post and further in some conversations with Kyle and Eastern Michigan doctoral student Angela Elkordy on Mozilla’s Open Badges discussion list.  This point concerns the difference between the purposes we intend to use digital badges for and the actual functions that they ultimately serve.  I raised this point earlier in my discussions with Kyle when he suggested that Angela concentrate on using badges to recognize (i.e., accredit) learning, but suggested caution in using badges to motivate learning (because of the concerns over extrinsic incentives and other issues).  My response was that if Angela was going to use badges to recognize learning, then those badges will almost certainly serve some motivational functions.  And those functions would depend on how she used badges to recognize learning.

Zipporah Miller 
After the panel, NSTA Associate Executive  Director Zipporah Miller asked me to elaborate on this distinction.  I used the same example in the context of the NSTA Learning Center about how any use of badges to recognize or reward teachers for completing the online activities at the center was going to have some impact on motivation, and that different ways of recognizing learning would lead to different motivation functions.  For example, giving a badge automatically when somebody completed some activity would likely motivate differently than only giving that badge when an expert or a peer reviewed what the student turned in.  To complicate things more, having an expert versus a peer recognize that accomplishment could ultimately have different motivational functions.  As the break was ending and my explanation was getting complicated, I suggested to Zipporah that it might make more sense to continue the conversation as a blog post. 

Purposes Versus Functions in Assessment
Dylan Wiliam
I got interested in this distinction several years ago when thinking about assessment and transfer.  Jim Pellegrino (assessment guru and my doctoral advisor) asked me to co-author a chapter with him on assessment and transfer.  Many leading assessment scholars have followed Paul Black and Dylan Wilam’s lead in categorizing assessments in terms of assessment purposes.  In their groundbreaking work on formative assessment, they argued that assessment should fundamentally be put to formative purposes, which they defined as assessment for learning.  This was an important distinction and helped distinguish formative assessment from summative assessment, which they defined as assessment of learning.  This distinction is now widely embraced, along with the further distinction between evaluative purposes, which is assessment collected primarily for evaluating curricula or programs.   This focus on purposes was particularly helpful at the time because it got people to stop categorizing the assessments themselves as formative, summative, or evaluative.  This helped the field realize that ostensibly formative assessments were often used for summative purposes, and vice versa.  Doing so is always problematic and sometimes disastrous.

In the chapter I eventually wrote with Jim, (available at academia.edu)  I argued that focusing on purposes was also problematic, because it overlooked that fact that the same assessment could be summative for some kinds of learning and formative for other kinds of learning.  Because Jim has a narrower view of learning than I do, he was a bit skeptical at first.  The example I used in the chapter drew from the formative assessment work I had been doing with the GenScope software for teaching introductory genetics.  That research explored the use of multiple levels of assessment:  

We had very informal assessments that served a summative function of the way groups of students completed the inquiry-oriented activities in GenScope, but were formative for each student's understanding of the concepts introduced in those activities. 

We also had semi-formal performance assessments that were summative of each student’s understanding of the concepts, but formative for the teacher’s learning how to use the informal assessments.

Finally we had formal achievement tests that were summative of the teacher’s use of the informal and semiformal assessments, but were formative for our research effort in learning whether we were improving outcomes from one year to the next. 

Once I adopted a broader sociocultural view of cognition that recognizes all three types of learning, I was pretty much forced to adopt the notion of assessment functions to capture this distinction between different functions for the same assessment.  This was partly a response to the concerns that Jim and others had raised in in the 2001 National Research Council Report Knowing What Students Know.  In that, they argued that using the same assessment for multiple purposes was problematic. 

The difference in perspectives stems from our underlying view of learning.  If you view learning as primarily individual conceptual development (as is  in Knowing What Students Know), then it makes sense to focus on assessment purposes.  The intended and unintended consequences can then be treated more as policy or technical issues than learning outcomes.  But if you take a broader view of learning that also includes how a whole class learns to talk about something like genetics and policy makers learning whether their policies are impacting achievement, then it makes more sense to talk about functions.  Thus, the NCLB-induced explosion of test-prep, the narrowing of the curriculum, and the bizarre focus on “bubble kids” just above or below proficiency benchmarks are all ways that teachers and schools learned to accommodate test-driven accountability in the last decade.  I elaborated on this issue in much more detail in a paper about the GenScope research that I wrote with Steven Zuiker that will appear in the next issue of The Journal of the Learning Sciences.

The Varied Functions of Badges
My interest in the  function of badges was spurred along when the MacArthur Foundation asked me to help document the design principles for using digital badges that emerge across the 30 projects underway by the awardees in their Badges for Lifelong Learning project.  We need to come up with a manageable number of categories; I almost always consider “manageable” to be at least three and no more than five.  After reading a bunch of stuff and talking to Barry Joseph at Global Kids and Carla Casilli at Mozilla, Connie Yowell at MacArthur, and MacArthur Scholar Mimi Ito, my team and I settled on the following four categories of functions for digital badges to shape our study:

Recognizing Learning.  This is the most obvious and arguably the primary function of badges.  I sometimes extend it to Recognizing/Accrediting Learning to acknowledge both informal and formal learning recognition processes.  David Wiley has argued cogently that this should be the primary purpose of badges.  If we focus only on purposes, then he may well be right.  His point is that badges are credentials and not assessments.  This is also consistent with the terrifically concise definition in Seven Things You Should Know About Badges by Erin Knight and Carla Casilli.

Assessing Learning.  Nearly every application of digital badges includes some form of assessment.  These assessments have either formative or summative functions and likely have both.  In some cases, these are simply an assessment of whether somebody clicked on a few things or made a few comments.  In other cases, there might be a project or essay that was reviewed and scored, or a test that was graded.  In still other cases, peers might assess an individual, group, or project as badgeworthy. 

Much of the discussion of using badges in the NSTA Learning Center concerned the sorts of assessments associated with digital badges.  Some argue that all badges must be associated with some sort of formal assessment if the badges are to be meaningful. But doing so requires significant infrastructure, scorers, formal assessments, online quizzes, security measures, etc. This echoed a similar discussion in August around how the Department of Education might award badges for teachers participating in Connected Educators Month.  So yes, badges are not assessments in and of themselves.  But nearly every use of badges has some assessment function associated with it.

Motivating Learning.  This is where the controversy comes in.  Much of the debate over badges concerns the well-documented negative consequences of extrinsic incentive on intrinsic motivation and free choice engagement. As first demonstrated by Lepper and Malone in 1973 and reiterated by Mitch Resnick and Henry Jenkins, when you give somebody an arbitrary extrinsic reward for something they already enjoy doing, they will do it less when the reward is no longer offered.  This rich topic deserves another post all to itself.  But for now, it appears that badges are going to happen.  And if we use badges to recognize and assess learning, they are certainly going to impact motivation.  So we might as well harness this crucial function of badges and study these functions carefully while searching for both their positive and negative consequences for motivation.  Perhaps more importantly, we should recognize that badges that offer new opportunities and empowerment and are less likely to leave learners disempowered. (You can read more about these issues at this online handbook entry about motivation) 

Some of the DML awardees articulated specific motivational functions for their badges in their proposal.  We are now interviewing awardees about how those plans are being enacted. Many of them are uncovering entirely new motivational functions, and some are changing their plans because of motivational concerns.  What is particularly interesting to us is how project are realizing that their plans for using badges to recognize and assess learning are pushing them to rethink their expectations regarding motivation.  We are really excited about the many new ideas bubbling to the surface.  I hope that some of the proposals for the new round of research of the DML projects will explore these issues in some detail.  This would be a nice complement to our study of the design principles across projects.  Also check out some of the new scholarly considerations about the motivational functions of badges, including this one by Judd Antin.

Evaluating Learning.  The final category of badge functions we are working with concerns evaluating learning (which we sometime extend to Evaluating/Researching Learning).  Only a few of the DML proposals articulated formal evaluation and research plans.  But the awardees who care about evaluating programs and researching learning are quickly uncovering new ways of using digital badges to help accomplish those goals.   For example, Jim Diamond at the Educational Development Center is working with the Who Built America? project.  They are exploring ways of collecting data for their project and are experimenting with giving participants badges for completing surveys about the project.  These will be different from the badges that they award teachers for completing well-designed lesson plans approved by experts in American history or expert peers.  And they will ultimately need to consider whether the nature of the survey (i.e., the assessment function) interacts with the evaluation function.

Perhaps even more far reaching are the many ways that programs might use the data contained in the badges to evaluate and study learning.  Relative to grades or transcripts, badges provide a
TON of information about learning.  Just the eight metadata tags in the OBI specification include stuff like date, issuer, recipient, and accomplishment.  For many of the programs that are using badges, having a system that tracks just this data will be a huge step forward for evaluating and studying learning.  And much of the information in the actual badge will be hyperlinked back to things like artifacts, testimonials, rubrics, course descriptions, etc.  In the case of the NSTA, much of the data they need to evaluate the impact of their online learning center should be obtained just by compiling the information about all of the stamps and badges that they award.  The potential value of this information was nicely articulated by Stacy Kruse of Pragmatic Solutions.  Stacy is working with two awardees, the Corporation for Public Broadcasting (who are incorporating badges into three different CPB initiatives) and the Digital Onramps Project with Arun Prabhakaran of the Philadephia Urban Affairs Coalition to use badges with comprehensive programs for 21st Century education and workforce training.  To paraphrase Stacy, she pointed out that Pragmatics Solutions is fundamentally a learning analytics firm, so these projects are a dream come true!

Clearly there are additional functions of badges that don’t fall into these categories.  In contrast to Carla Casilli's taxonomy which aimed to be expansive, our goal of documenting a manageable set of design principles from the DML badges projects has pushed us to be concise. We might just need to add a category called “other.”  We have two years to sort it out, and it will be interesting to see where we end up.  In a subsequent post, we will elaborate some on the DML Design Principles Documentation project and say more about how we are using these categories in our project.   We welcome your suggestions and concerns.

Summary of Badge Functions
Okay, this has gotten pretty long.  But these are not simple issues.  These long posts remind me of the first chapter of Daniel Koretz’s 2008 book Measuring Up: What Educational Testing Really Tells Us. Koretz described the frustration that parents, business leading, and policy makers express when assessment and measurement experts insist on giving complicated answers to seemingly simple questions. There is not a simple answer to the question "what are the functions of digital badges?"

The following table summarizes the basic point of this post.  While the categories of badging functions are likely to evolve, I think the basic point will hold that there are a definable set of ways to use badges to support learning, and that functions are a good way to sort these things out and their interactions.  Recognition appears to be an essential to digital badges. Most conceivable uses of badges will serve some assessment function, while many will also serve some motivational function.  Finally while badges offer potential for evaluating and researching learning, these are potential functions which need to be specifically pursued.  
Functions of Badges
Recognizing/Credentialing Learning
Assessing Learning
Motivating Learning
Evaluating/Researching Learning
Nature of Functions
Essential
Functions
Probable Functions
Likely
Functions
Potential
Functions




Copyright © 2012 Online Course. All rights reserved.

Super SEO created by Blogger Tuts | Published by GalleryBloggerTemplates.com