Digital Badge Design Principles for Recognizing Learning

Cross-posted at HASTAC

by Andi Rehak and Daniel Hickey


This post introduces the design principles for recognizing learning that are emerging  from the Design Principles Documentation Project (DPD).  A previous post summarized how the DPD project derived these principles. This is the first of four posts, to be followed by posts outlining the principles for using badges to assess, motivate, and study learning.


First and foremost, digital badges serve to recognize some learning or accomplishment.  As succinctly put by Mozilla's Carla Casilli and Erin Knight in their EDUCAUSE Brief, "badges signify accomplishments such as completion of a project, mastery of a skill, or marks of experience."  An important distinction here was made by David Wileywhen he pointed out that "badges are not assessments…badges are things we award to people who pass assessments.”

In more formal educational contexts, this practice is usually called "credentialing."  We are using a more general term of “recognizing” to emphasize that many of the practices for recognizing learning with badges are more informal.  For example, while a Girl Scout badge is technically a credential, few would call it that.  The different DML projects recognize various types of learning.  The way that learning is recognized and who recognizes it also varies. These differences then have consequences for assessing, motivating, and studying learning.

Our Process for Identifying these Principles
We first identified the intended practices for recognizing learning in each project's proposal.  As project teams began implementing those intentions, we interviewed them to uncover their enacted practices.  The 5-10 enacted practices for each project were then sorted into a manageable set of more general principles.  We debated categories and subcategories ourselves. We then discussed and refined them at a half-day workshop before the DML conference with 45 badge project team members.

This resulted in the preliminary set of design principles listed below, starting with the ones that were most widely represented.  These principles are not set in stone.  The principles and (eventually) sub-principles will continue to evolve as we work with them, identify resources that are relevant to each, share them with projects, and begin sharing them more generally.  We seek your questions and suggestions.  Please post comments directly on the blog.

Nine Principles for Recognizing Learning with Digital Badges
The following principles are ordered by prevalence in the current badging practices of the DML Competition 4: Badges for Lifelong Learning awardees. While these principles were developed based on the practices of the DML awardees, these principles should be useful in helping organize efforts to design and refine any badging system. The usefulness of the principles will (hopefully) increase as they are connected to existing literature on credentialing of learning experiences as the DPD team delves into that phase of the project.

1. Use badges to map learning trajectories.  Most of the projects used badges to organize curriculum and learning experiences by either determining levels of badges or offering meta-badges. Alternatively, some projects allow for a more learner-directed process that encourages students to create their own trajectory.

2. Align badges to standards.  Many of the projects used national or international standards to increase the external value of the badge. Alignment to standards is presumed to improve transparency of the credential and help to facilitate better communication of earner knowledge and skills. Some of these standards were more formal such as, the Common Core State Standards, while others were the less formal such as "21st Century Skills." Sometimes the alignment was very formal but other times it was very informal.  Highlighting the relationship between recognition and assessment, the formality of this alignment was usually defined by the formality of the assessment practices involved. 

3.  Have experts issue badges.  Having experts issue badges increases the credibility of the badge and likely influences the usefulness of the credential outside of the issuing community.  At some level, some expert is associated with issuing badges.  But the nature and role of this expert varied quite a bit, as did that way that the expert was him or herself credentialed.  Sometimes the expert held an external credential, while other times the expert was credentialed by the community; some projects include both. 

4. Seek external backing. External backing is presumed to increase the usefulness of the badge as name recognition is a driving force in getting schools or employers to recognize the badge. In the projects that sought external backing, this seemed different than just using badges as a means of external communication. Whether or not the badge is actually externally endorsed, existing formal relationships can increase its external value. (For those wishing to have formal credit granted for a badge, this is the first step.) Partnerships increase communication of the learning recognized in the badge and thus increase the importance/usefulness of the badge for earners. In some cases, a badge is formally endorsed and carries the insignia of the endorsing institution.

5.  Recognize diverse learning.  Credentialing a broad spectrum of experiences helps to legitimize these areas and recognizes knowledge and skills which would otherwise only be implicitly noticed or not at recognized. While this principle could be uncovered in nearly all of the projects at some level, we highlighted several projects that embraced it explicitly. These projects recognized skills and learning outside of what is traditionally recognized in formal learning environments, giving badges for both "hard" and "soft" skills.

6.  Use badges as a means of external communication of knowledge and/or skills.   As with the previous principle, most projects did this at some level.  But some projects really made a concerted effort to increase communication of the learning or accomplishment that the badges represent.

7. Make badges permanent.  While many projects did not explicitly discuss whether or not their badges would expire or require upgrading, a few made strong cases for learners being able to have permanent credentials that will always exist to recognize that specific skill, knowledge, or experience.

8.  Recognize educator learning as well.  Some of the projects awarded badges specifically to educators.  This is a different principle that relates to the several projects where additional badges were included alongside student badges.  These were sometimes that same as the badges for students and other times they were specific to the educators.  Generally speaking, these badges were used to recognize the educators’ participation in the broader learning ecosystem.

9.  Award formal academic credit for badges.  In a few projects, badges were used as a supplement to a formal grade for in-school experiences. Currently, only a couple of projects have successfully created partnerships that allow a badge to directly result in formal academic credit. This of course greatly increases the value of the badge for badge earners.

A Design Implication
We conclude by introducing a dilemma we faced in this process.  This dilemma was embodied in the distinction between (a) integrating badges into an existing curriculum and (b) creating a badge system and a curriculum at the same time.  For a while we treated this as a design principle for recognizing learning.  But we came to see it more as a “design implication.”  This difference seems to have great impact for the design of the badging practices (and likely for the learning ecosystem that results).  But this distinction was largely a result of the context in which badges were being introduced, rather than a deliberate choice.  As such it did not seem like this qualified as an actual design principle. When badges are being added to a pre-existing curriculum, the curriculum will constrain the way learning is recognized. For example, if an existing curriculum is not aligned to standards, it is very difficult to align a badge to standards. Alternatively, when the curriculum is being developed alongside badges, the options for both may seem limitless and overwhelming. A pre-existing curriculum can importantly help to structure design decisions. There are specific advantages for either approach.
We would love to hear what people think about this distinction or about our design principles for recognition of learning with digital badges.

Badges Design Principles Database Project: Update on New Principles


by Daniel Hickey

This post is cross-posted at HASTAC. 
This post is a brief update about the design principles that have emerged in our analyses and interviews of the 30 DML badges awardees. We will begin posting the initial set of design principles for using digital badges to support learning. Specifically we will put up consecutive posts about the principles we have found for using digital badges to recognize, assess, motivate, and research learning. The first blog on recognizing learning with digital badges us up at HASTAC and Remediating Assessment.

previous post introducing our project elaborated on how we were going to identify intended practices for using digital badges by the DML badge awardees. We did that and then interviewed projects to uncover their enacted practices as they put their ideas into place within their particular program or school. We identified practices in four aspects of learning for each project: recognizing, assessing, motivating, and researching. 

We then sorted each set of practices into more general design principles. This is intended to help projects learn what they have in common and link outside resources with particular principles. We hope this will be helpful for building a community of practice around digital badges and help make connections across existing and new badge design projects. Our ultimate goal is a self-sustaining open network organized around an evolving set of principles and resources.

In the coming days, we will post four articles listing the initial set of design principles in each of the four areas. We will describe the principles and the rough proportion of projects that explicitly described corresponding practices. For some principles, we will include an example of a corresponding practice from one or more of the projects that has endorsed our characterizations.i We have been debating the structure and names, and we are getting ready to formalize them, so we really hope people will provide input and suggestions. 

The next phase of our project is identifying outside resources (i.e., research papers, articles, blog posts, etc.) that seem relevant to each of the design principles. We will start by linking principles to the 170 resources in the Annotated Research Bibliography by Sheryl Grant and Kirsten Shawgo. Before doing so we hope to finalize our current set of design principles.ii

Here is our tentative order of posts and authors that we plan to post over the next several weeks.

Design Principles for Recognizing Learning with Digital Badges, by Andi Rehak & Dan Hickey

Design Principles for Assessing Learning with Digital Badges, by Rebecca Itow & Dan Hickey

Design Principles for Motivating Learning with Digital Badges, by Katerina Schenke, Cathy Tran, & Dan Hickey

Design Principles for Researching Learning with Digital Badges, by Dan Hickey

We look forward to hearing from you!


iThis is one of the most challenging aspects of our project. We want to give projects and teams credit for their insights and we want to help people connect with the projects and teams who are enacting practices they are interested in. But we also want to characterize projects and practices accurately. We have been asking (and perhaps begging) team members to review, edit, and endorse our characterizations of individual projects and practices. Once projects are satisfied with the current characterizations, we are asking projects to allow us to share those characterizations widely and publicly. As you can imagine, this is difficult given the number and diversity of projects and their continuing evolution. If you are involved in a project and have yet to review your characterizations, please visit the project website at http://iudpd.indiana.edu/ . And even if you have reviewed and approved our characterizations, you might wish to again visit the new Badge Design Principles section to see which design principles are associated with your project.

iiWe emphasize that these principles will continue to unfold as we learn more about them. Numerous decisions were made in deciding which practices go together and what each principle should be called. We are trying to use more general labels and everyday language for the principles. One of our more general project findings is that the process of identifying practices and principles becomes more subjective and more academic across the four categories of principles. Thus:

It was fairly straightforward to identify the principles for recognizing learning and characterize them with everyday language (e.g., recognizing rather than credentialing). That is to say that every project articulated the learning they were going to recognize and most articulated how they were making those decisions. These practices and the more general principles that emerged can be explained in everyday terms.

Projects were less clear about how they were going to assess learning. More importantly the practices and principles for assessing learning can’t be characterized without using some of the language and distinctions from the assessment research literature (i.e., formative, summative, transformative, etc.).

As for motivating learning, many projects did not offer explicit motivational strategies. This meant that the implicit practices and principles for using badges to motivate learning could only be inferred from the perspective of current research literature on motivation.

Few projects articulated plans for studying learning. This means that the potential practices and principles across the projects were almost entirely inferred from the existing literature and assumptions about educational research.

This reality has some complicated consequences that we are just beginning to understand. For example, some projects have already told us that they are enacting principles for assessing and motivating learning that they did not specifically articulate; it seems likely that projects that explicitly indicated that they were enacting those practices are best positioned to share their insights with others. This also means that that the design principles for studying learning are almost entirely speculative; by defining principles for doing so openly and with lots of community input, we hope that our project will yield a framework for organizing the research now getting underway, and organizing future research as this community evolves.

Online Open Courses Raises Eleven Issues for Higher Education

by Daniel Hickey
            I introduce eleven issues that I am going to have to address with my university in order to teach a free open online course on educational assessment.  I then explore the first issue, Intellectual Property, and how that issue intersects with instructional innovation in open courses.
            Thanks to a grant from Google, I am going to offer a free open online course called Assessment Practices, Principles, and Policies starting in September 2013. This course will help educators improve the way student learning is assessed in classrooms. It will also help them understand the complex issues associated with classroom and school accountability.  I am fortunate to have some assistance from my colleague Cassandra Guarino in Indiana University’s Education Policy Program.  Professor Guarino is going to help out with the aspects of the course that address the controversial “value-added” teacher evaluation policies that most states are introducing this year. 

A Fertile Ground for Exploring New Models of Teaching
            I am really excited about exploring new models of online learning in massively open online courses (or “MOOCs”).  Google invited proposals to scale up existing courses using Coursebuilder, Google’s new course management platform.What was particularly intriguing was that Google would “welcome creative approaches that go beyond the standard MOOC implementation, experimenting with new formats and innovative uses of social experiences.” 
The grant will let me scale up the interactive “wikifolios” that I have been exploring in my graduate-level online Assessment in Schools course. Rather than a massive course enrolling thousands, I am calling this course a “big open online course” (BOOC), and enrollment will be capped at 500.  Such a size will allow me to gradually scale up from the current 25-person for-credit course that I teach most summers. Before I can find out of this will work, there are a few other interesting details that need to be worked out.

Legal Issues for the University
The move to open courses raises all kinds of issues for universities.  As soon as the press release for the BOOC was posted, I was informed that there were a “number of important issues” that would have to be worked out between IU, Google, and the instructors. These issues include

intellectual property, conflicts of interest and commitment, state authorizations, FERPA, terms of use, verification of student identity, ADA compliance, language for any certifications or guarantees, use of the IU name and seal, trademarking and branding, and any revenue generation resulting from the BOOC.

This looked like it might be a lotof work.  Other IU faculty like Instructional Systems Technology Professor Curt Bonk who have delivered MOOCs have sidestepped these issues by doing everything outside of the university.  So I looked elsewhere for legal advice. Fortunately, the IU attorneys had already worked through key issues with Katy Borner, another IU faculty member who had previously opened up her Information Visualization course and who had also been awarded a Google CourseBuilder grant.
Now that I have completed my first meeting with a university lawyer (who happens to specialize in copyright law), I find myself quite fascinated by these issues. I am particularly interested in how these issues intersect with efforts to support networked learning and massively interactive online learning. As I go forward, I am going to explore each of these eleven open course issues, focusing on how they relate to my efforts to support more interactive online learning.

Issue #1: Intellectual Property
The first of these eleven issues concerns ownership of the intellectual property represented by courses (some background here).  If the universities own the course materials, they can hire adjuncts or graduate student to teach a course that faculty members develop, and faculty members can’t take their courses with them when they leave. After simmering for decades, this issue heated up with the advent of online instruction.  Intellectual property was one of the key barriers to online teaching that Zane Berge identified in an influential survey first carried out in 1998.
Course ownership has become a much discussed issue with the recent emergence of for-profit MOOCs like Coursera.  In order to protect the investment of tens of millions of dollars from venture capitalists, these companies naturally have licensing terms that favor the company over the faculty and universities who provide the content.  An EDUCAUSE blog on this topic concluded:

Higher education should pause and reflect on these restrictive licensing terms and the implications for the academic enterprise that has been traditionally built on creating and sharing knowledge as a core value of the teaching and learning mission.  In today’s remix learning culture, what does it mean when users have to give up their IP rights to participate in a MOOC?  When sharing is restricted?   The licenses show that these companies are quite proprietary about the rights for use of their content, but are broadly sweeping in claiming rights for user-generated content

Course ownership, in part or in whole, is a particular issue in content-intensive courses. Naturally, ownership is even more of an issue for courses that use elaborate tutorials based on decades of research. Consider that Carnegie Learning, a leader in developing artificially intelligent tutors, licensed their math tutor to the state of Georgia for $5M in 2008.
            Based on my first meeting with our legal agents and my attempts to personally decipher the relevant Indiana University policies, a critical factor about ownership is whether I use any “exceptional university support” that goes beyond the normal course development process.  Specifically, the intellectual property of my course will belong to the university if I use

…designated technical assistance, such as audio-visual departmental personnel or a qualified graduate assistant, to assist development of an online course, or provision of specialized software purchased for a particular online project, which exceeds normal University support for traditional courses…

Since I have every intention of continuing to take advantage of the specialized knowledge of the staff at IUs Center for Innovation in Teaching and Learning, it is highly unlikely that I will have any claim to the course content.
            Similarly the IP appears to belong to the University if I am

…commissioned by the University by the provision of release time or other compensation to a faculty member as an adjustment to normal assigned duties for the purpose of creating an online course, which exceeds normal University support for traditional courses.

Given that I developed the original online Assessment in Schools course as part of a grant to develop our online certificate program in the Learning Sciences, this would also mean that the university owns the BOOC. Do you agree?
But what if I had not used either of these forms of support?  Since I did not use either of them for my online Learning & Cognition in Schools course, do I own the intellectual property in that course?  And which version of the course are we talking about?  I have been obsessively refining my courses both with and without these forms of support, and will continue doing so.  This gets very complicated very quickly.
            So far, I have little reason to be concerned about IP.  This is because I have developed very little actual “content” in my online classes.  Unlike many online courses, most of the content in my course is contained in a current textbook.  I assume this will be the case with my BOOC. Some students and MOOC proponents will surely take issue with using a textbook an open class and it will likely cut enrollment.  On the one hand, I agree with proponents of the “connectivist” model promoted by George Siemens and Steven Downes that focus on learning to find information.  Plus the current edition of the text lists for $102.  Even the digital version at CouseSmart.com costs $41 and is only good for 180 days.
            On the other hand, I like good textbooks.  I have been using Jim Popham’s assessment textfor a decade.  He updates it every three years or so and does a great job covering the new developments in assessment and accountability.  Because new and changing policies are a focus of my course, I like learners to have a handy source of current disciplinary knowledge.  I want participants to learn to engage with their professional colleagues using current disciplinary knowledge. A good textbook is helpful for doing so. I would love to hear comments and opinions about this from readers.
One possibility is encouraging participants to buy the 2014 edition but allowing them to also use the 2010 edition that Amazon currently sells used for $18 (or even the 2007 edition that goes for $5).  Because of the highly interactive nature of the course, students with the current edition will be able to contribute those insights to the class.  And I can certainly reference the new text. But a quick scan of Indiana Univerity’s  fair use policy, left me unclear as to what my rights are. But the university attorney has made it clear that will be expected to follow them.  What is interesting is that the university is going to be much stricter about fair use in open courses than they are about regular courses. I have already met with the publisher’s representative and asked him to clarify whether they will grant me permission to do what professors now do routinely (like let students download the first chapter of the text in case they run into delays buying the book).  I will update this post as I learn more.

Digital Badges Meeting at the NSF Headquarters Hosted by NYSCI


by Katerina Schenke
This post describes a meeting at the National Science Foundation where sixty leaders in education and research from around the country gathered to discuss digital badges and education.  Three of use presented the initial set of design principles from the Design Principles Documentation Project.
Monday April 1stwe travelled to the NSF headquarters in Arlington, VA. There, Michelle Riconscente and Margaret Honey from the New York Hall of Science hosted a meeting with an impressive list of attendees. STEM educators, members from after school programs, researchers, professors from all different disciplines (computer science, educational psychology, learning sciences) among others met to discuss the current and future research surrounding badges.

Rebecca Itow, Cathy Tran, and I were invited to attend as members of the Badge Design Principles Documentation project and had been asked to serve as official note takers of the meeting. We ended up doing Dan Hickey’s presentation on the project and about digital badges research because Dan instead had to attend to a death in his family.  Our presentation went over well and the audience was very interested in the initial set of design principles emerging across the 30 projects funded by the Gates/MacArthur Badges for Lifelong Learning initiative

Along with discussions about the logistical concerns about the use badges such as how to manage these various systems (Erin Knight from Mozilla), on-the-ground depictions of badge systems (Alejandro Molina from the Providence After School Alliance, Marc Lesser from MOUSE, Inc, and Akili Lee from the DigitalYouth Network, just to name a few), and the potential for badges to optimize student learning (Barry Fishman).  We candidly spoke about some concerns about badging such as “what is the life expectancy of a badge” (Avi Kaplan), and “what are some of the challenges and what are some of the insights as a result of this work?” (Michelle Riconscente).

As someone who is interested in what badges can do for student motivation, a question from Learning Sciences legend Allen Collins really stood out:

“Badges is a low stakes enterprise but at the other end we think about using badges to make decisions like colleges, employers which are kind of high stakes decisions and we kind of know that when you put high stakes on things it distorts the way the system works. It leads to cheating and all of that. So the question is how can the badge community resolve this tension?”

Given that some of the badges efforts are already planning for high-stakes uses, this is a huge question.  Unlike course grades and diplomas, digital badges contain detailed evidence and hyperlinks to more evidence about the issuer and what the individual did to earn the badge.  While this won’t prevent these concerns, it will certainly cause them to unfold differently. Should we resist turning badge systems into high stakes pursuits? Is there a way to design these systems in such a way where badges contain outside value to the learner yet still emphasize the learning process within the badge system?

Comments? Thoughts?



Google Funds an Assessment Practices, Policies and Principles BOOC

by Dan Hickey
This post describe the Big Open Online Course on assessment that was recently funded by Google.
I am happy to announce that Google has kindly supported our proposed Big Open Online Course on Educational Assessment for the Fall of 2013. Here is the press release from IU. If you are interested I just started a Google Group that we will use for announcements: http://tinyurl.com/BOOC-IU
Make sure you click "join group" so you are able to receive updates and post messages.

The BOOC will be capped at 500 students and will use wikifolios and participatory assessment to support much more interaction and professional discourse than is typical in open online courses.  The university is allowing me to embed a for-credit course for student who complete all of the assignments and assessment and assemble their wikifolios into a term paper.  I will also teach this as a smaller for-credit fully online course in the Summer of 2013.

Both the course and the method embrace a lot of topics that we have covered here at Remediating Assessment.  Probably the most relevant is the discussion of digital badges from October 2012.  There is a post about using digital badges in college classes here, and Rebecca Itow and I also wrote a post about the role of reflections in participatory assessment.  



Initial Explorations in Digital Badges and Motivation

By Cathy Tran
This post introduces two of the newest members of the badges Design Principles Documentation Project and describes our efforts to examine the motivational practices and principles that we are uncovering across the 30 project funded to develop digital badges by the MacArthur Foundation's Digital Media and Learning initiative
I just wrapped up a Google+ Hangout session with Katerina Schenke, as we pulled motivation design principles from dozens of projects after viewing their Digital and Media Learning Badges for Lifelong Learning competition applications as well as having conversations with the winners and listening to archived interviews.

Through the Digital Media and Learning Conference poster to unveil our emerging principles are still in the works, I'll give a sneak peek, for those who are way too curious to wait (or maybe I just *really* want to share immediately): One emerging principle is that badge systems are providing different ways for learners to set goals. This includes user-created badges in which the learner decides which badge to pursue or even what a badge is awarded for and what the criteria is for that badge (subject to agreement from the community). In addition, we see different displays of learning trajectories. Some show long-term goals and how the steps along the way connect to one another. Others organize their badges in batches that are not as explicitly interconnected. Another point of interest is the level of autonomy varies. There are paths that are largely determined by the badge system and are intended to scaffold learning in specific ways and others that allow for more user choice of twists and turns.

What does this all mean for learner motivation? Our next step for this motivation principle (and roughly a dozen others) is to connect these practices to the motivation research literature. For instance, what can the more general research on goals and choice--that's not necessarily tied to digital badges or even technology--tell us about the potential benefits and limitations of these different ways in which digital badges allow for goal setting?

Those are big questions, and that's why it is exciting. Back in January of this year, Dan invited me to join the Design Principles Project when all the digital badges folks met at UC Irvine. Right after that meeting, I sent my advisor a quick note that included this:
This whole applying theory (especially motivation) to digital practice thing is so incredibly up my alley and what I've been trying to do--so even if it's tricky, here's a group of people who all want to puzzle through it too and how crazy of an opportunity is that?
Coming to UC Irvine to pursue a PhD, my goal was to connect academic researchers with those who are out there making things. I worked in research, evaluation, and production for children television, museums, and educational software, spending much of that time pondering about how the research I later do can impact development. My overarching interest has always been to connect the scientific community with the public, as my first few years out of college were spent as a science journalist. This project fits right into that big goal of mine.

So I hope you will follow us in this journey as we learn how to make academic research relevant to developers of digital badges--and how to make their work relevant to the academic community.

For those out there with complementary interests, I would love to connect. I can be reached at ctran27@gmail.com or on Twitter @cathytran.