Intended Purposes Versus Actual Functions of Digital Badges
By Daniel Hickey
On September 4th and 5th, there was a meeting at the National Science Teachers Association in Arlington, VA. Al Byers of NSTA and Kyle Peck of Penn State organized the meeting to discuss the online NSTA Learning Center for science educator professional development. I was only able to make it to the second day of the meeting where Kyle discussed the pilot work with the site and his use of digital badges from the Teacher Learning Journeys project. In the afternoon, Sunny Lee and Erin Knight (Mozilla Foundation) and Brian Mulligan (Sligo Institute of Technology, Ireland) and I did a panel on digital badges that Kyle moderated..
One of the questions about badges that came up seems like a crucial issue as we grapple with different ways of characterizing and describing badges. This post aims to add the category of badge functions to other badge taxonomies like the one by Carla Casilli. Because these issues are complex, this post ended up being rather long. You may wish to jump directly to the summary at the bottom. You may also wish to read a condensed version at the HASTAC website.
On September 4th and 5th, there was a meeting at the National Science Teachers Association in Arlington, VA. Al Byers of NSTA and Kyle Peck of Penn State organized the meeting to discuss the online NSTA Learning Center for science educator professional development. I was only able to make it to the second day of the meeting where Kyle discussed the pilot work with the site and his use of digital badges from the Teacher Learning Journeys project. In the afternoon, Sunny Lee and Erin Knight (Mozilla Foundation) and Brian Mulligan (Sligo Institute of Technology, Ireland) and I did a panel on digital badges that Kyle moderated..
One of the questions about badges that came up seems like a crucial issue as we grapple with different ways of characterizing and describing badges. This post aims to add the category of badge functions to other badge taxonomies like the one by Carla Casilli. Because these issues are complex, this post ended up being rather long. You may wish to jump directly to the summary at the bottom. You may also wish to read a condensed version at the HASTAC website.
Badge Purposes Versus Badge Functions
In that panel, I elaborated on a point that I introduced in an earlier post and further in some conversations with Kyle and Eastern Michigan doctoral student Angela Elkordy on Mozilla’s Open Badges discussion list. This point concerns the difference between the purposes we intend to use digital badges for and the actual functions that they ultimately serve. I raised this point earlier in my discussions with Kyle when he suggested that Angela concentrate on using badges to recognize (i.e., accredit) learning, but suggested caution in using badges to motivate learning (because of the concerns over extrinsic incentives and other issues). My response was that if Angela was going to use badges to recognize learning, then those badges will almost certainly serve some motivational functions. And those functions would depend on how she used badges to recognize learning.
 |
| Zipporah Miller |
After the panel, NSTA Associate Executive Director Zipporah Miller asked me to elaborate on this distinction. I used the same example in the context of the NSTA Learning Center about how any use of badges to recognize or reward teachers for completing the online activities at the center was going to have some impact on motivation, and that different ways of recognizing learning would lead to different motivation functions. For example, giving a badge automatically when somebody completed some activity would likely motivate differently than only giving that badge when an expert or a peer reviewed what the student turned in. To complicate things more, having an expert versus a peer recognize that accomplishment could ultimately have different motivational functions. As the break was ending and my explanation was getting complicated, I suggested to Zipporah that it might make more sense to continue the conversation as a blog post.
Purposes Versus Functions in Assessment
 |
| Dylan Wiliam |
I got interested in this distinction several years ago when thinking about assessment and transfer. Jim Pellegrino (assessment guru and my doctoral advisor) asked me to co-author a chapter with him on assessment and transfer. Many leading assessment scholars have followed Paul Black and Dylan Wilam’s lead in categorizing assessments in terms of assessment purposes. In their groundbreaking work on formative assessment, they argued that assessment should fundamentally be put to formative purposes, which they defined as assessment for learning. This was an important distinction and helped distinguish formative assessment from summative assessment, which they defined as assessment of learning. This distinction is now widely embraced, along with the further distinction between evaluative purposes, which is assessment collected primarily for evaluating curricula or programs. This focus on purposes was particularly helpful at the time because it got people to stop categorizing the assessments themselves as formative, summative, or evaluative. This helped the field realize that ostensibly formative assessments were often used for summative purposes, and vice versa. Doing so is always problematic and sometimes disastrous.
In the chapter I eventually wrote with Jim, (available at academia.edu) I argued that focusing on purposes was also problematic, because it overlooked that fact that the same assessment could be summative for some kinds of learning and formative for other kinds of learning. Because Jim has a narrower view of learning than I do, he was a bit skeptical at first. The example I used in the chapter drew from the formative assessment work I had been doing with the GenScope software for teaching introductory genetics. That research explored the use of multiple levels of assessment:
We had very informal assessments that served a summative function of the way groups of students completed the inquiry-oriented activities in GenScope, but were formative for each student's understanding of the concepts introduced in those activities.
We also had semi-formal performance assessments that were summative of each student’s understanding of the concepts, but formative for the teacher’s learning how to use the informal assessments.
Finally we had formal achievement tests that were summative of the teacher’s use of the informal and semiformal assessments, but were formative for our research effort in learning whether we were improving outcomes from one year to the next.
Once I adopted a broader sociocultural view of cognition that recognizes all three types of learning, I was pretty much forced to adopt the notion of assessment functions to capture this distinction between different functions for the same assessment. This was partly a response to the concerns that Jim and others had raised in in the 2001 National Research Council Report Knowing What Students Know. In that, they argued that using the same assessment for multiple purposes was problematic.
The difference in perspectives stems from our underlying view of learning. If you view learning as primarily individual conceptual development (as is in Knowing What Students Know), then it makes sense to focus on assessment purposes. The intended and unintended consequences can then be treated more as policy or technical issues than learning outcomes. But if you take a broader view of learning that also includes how a whole class learns to talk about something like genetics and policy makers learning whether their policies are impacting achievement, then it makes more sense to talk about functions. Thus, the NCLB-induced explosion of test-prep, the narrowing of the curriculum, and the bizarre focus on “bubble kids” just above or below proficiency benchmarks are all ways that teachers and schools learned to accommodate test-driven accountability in the last decade. I elaborated on this issue in much more detail in a paper about the GenScope research that I wrote with Steven Zuiker that will appear in the next issue of The Journal of the Learning Sciences.
The Varied Functions of Badges
My interest in the function of badges was spurred along when the MacArthur Foundation asked me to help document the design principles for using digital badges that emerge across the 30 projects underway by the awardees in their Badges for Lifelong Learning project. We need to come up with a manageable number of categories; I almost always consider “manageable” to be at least three and no more than five. After reading a bunch of stuff and talking to Barry Joseph at Global Kids and Carla Casilli at Mozilla, Connie Yowell at MacArthur, and MacArthur Scholar Mimi Ito, my team and I settled on the following four categories of functions for digital badges to shape our study:
Recognizing Learning. This is the most obvious and arguably the primary function of badges. I sometimes extend it to Recognizing/Accrediting Learning to acknowledge both informal and formal learning recognition processes. David Wiley has argued cogently that this should be the primary purpose of badges. If we focus only on purposes, then he may well be right. His point is that badges are credentials and not assessments. This is also consistent with the terrifically concise definition in Seven Things You Should Know About Badges by Erin Knight and Carla Casilli.
Assessing Learning. Nearly every application of digital badges includes some form of assessment. These assessments have either formative or summative functions and likely have both. In some cases, these are simply an assessment of whether somebody clicked on a few things or made a few comments. In other cases, there might be a project or essay that was reviewed and scored, or a test that was graded. In still other cases, peers might assess an individual, group, or project as badgeworthy.
Much of the discussion of using badges in the NSTA Learning Center concerned the sorts of assessments associated with digital badges. Some argue that all badges must be associated with some sort of formal assessment if the badges are to be meaningful. But doing so requires significant infrastructure, scorers, formal assessments, online quizzes, security measures, etc. This echoed a similar discussion in August around how the Department of Education might award badges for teachers participating in Connected Educators Month. So yes, badges are not assessments in and of themselves. But nearly every use of badges has some assessment function associated with it.
Motivating Learning. This is where the controversy comes in. Much of the debate over badges concerns the well-documented negative consequences of extrinsic incentive on intrinsic motivation and free choice engagement. As first demonstrated by Lepper and Malone in 1973 and reiterated by Mitch Resnick and Henry Jenkins, when you give somebody an arbitrary extrinsic reward for something they already enjoy doing, they will do it less when the reward is no longer offered. This rich topic deserves another post all to itself. But for now, it appears that badges are going to happen. And if we use badges to recognize and assess learning, they are certainly going to impact motivation. So we might as well harness this crucial function of badges and study these functions carefully while searching for both their positive and negative consequences for motivation. Perhaps more importantly, we should recognize that badges that offer new opportunities and empowerment and are less likely to leave learners disempowered. (You can read more about these issues at this online handbook entry about motivation)
Some of the DML awardees articulated specific motivational functions for their badges in their proposal. We are now interviewing awardees about how those plans are being enacted. Many of them are uncovering entirely new motivational functions, and some are changing their plans because of motivational concerns. What is particularly interesting to us is how project are realizing that their plans for using badges to recognize and assess learning are pushing them to rethink their expectations regarding motivation. We are really excited about the many new ideas bubbling to the surface. I hope that some of the proposals for the new round of research of the DML projects will explore these issues in some detail. This would be a nice complement to our study of the design principles across projects. Also check out some of the new scholarly considerations about the motivational functions of badges, including this one by Judd Antin.
Evaluating Learning. The final category of badge functions we are working with concerns evaluating learning (which we sometime extend to Evaluating/Researching Learning). Only a few of the DML proposals articulated formal evaluation and research plans. But the awardees who care about evaluating programs and researching learning are quickly uncovering new ways of using digital badges to help accomplish those goals. For example, Jim Diamond at the Educational Development Center is working with the Who Built America? project. They are exploring ways of collecting data for their project and are experimenting with giving participants badges for completing surveys about the project. These will be different from the badges that they award teachers for completing well-designed lesson plans approved by experts in American history or expert peers. And they will ultimately need to consider whether the nature of the survey (i.e., the assessment function) interacts with the evaluation function.
Perhaps even more far reaching are the many ways that programs might use the data contained in the badges to evaluate and study learning. Relative to grades or transcripts, badges provide a
TON of information about learning. Just the eight metadata tags in the OBI specification include stuff like date, issuer, recipient, and accomplishment. For many of the programs that are using badges, having a system that tracks just this data will be a huge step forward for evaluating and studying learning. And much of the information in the actual badge will be hyperlinked back to things like artifacts, testimonials, rubrics, course descriptions, etc. In the case of the NSTA, much of the data they need to evaluate the impact of their online learning center should be obtained just by compiling the information about all of the stamps and badges that they award. The potential value of this information was nicely articulated by Stacy Kruse of Pragmatic Solutions. Stacy is working with two awardees, the Corporation for Public Broadcasting (who are incorporating badges into three different CPB initiatives) and the Digital Onramps Project with Arun Prabhakaran of the Philadephia Urban Affairs Coalition to use badges with comprehensive programs for 21st Century education and workforce training. To paraphrase Stacy, she pointed out that Pragmatics Solutions is fundamentally a learning analytics firm, so these projects are a dream come true!
Clearly there are additional functions of badges that don’t fall into these categories. In contrast to Carla Casilli's taxonomy which aimed to be expansive, our goal of documenting a manageable set of design principles from the DML badges projects has pushed us to be concise. We might just need to add a category called “other.” We have two years to sort it out, and it will be interesting to see where we end up. In a subsequent post, we will elaborate some on the DML Design Principles Documentation project and say more about how we are using these categories in our project. We welcome your suggestions and concerns.
Summary of Badge Functions
Okay, this has gotten pretty long. But these are not simple issues. These long posts remind me of the first chapter of Daniel Koretz’s 2008 book Measuring Up: What Educational Testing Really Tells Us. Koretz described the frustration that parents, business leading, and policy makers express when assessment and measurement experts insist on giving complicated answers to seemingly simple questions. There is not a simple answer to the question "what are the functions of digital badges?"
The following table summarizes the basic point of this post. While the categories of badging functions are likely to evolve, I think the basic point will hold that there are a definable set of ways to use badges to support learning, and that functions are a good way to sort these things out and their interactions. Recognition appears to be an essential to digital badges. Most conceivable uses of badges will serve some assessment function, while many will also serve some motivational function. Finally while badges offer potential for evaluating and researching learning, these are potential functions which need to be specifically pursued.
Functions of Badges | ||||
Recognizing/Credentialing Learning | Assessing Learning | Motivating Learning | Evaluating/Researching Learning | |
Nature of Functions | Essential Functions | Probable Functions | Likely Functions | Potential Functions |
