Home » Posts filed under formative assessment
Showing posts with label formative assessment. Show all posts
Showing posts with label formative assessment. Show all posts

Responding to Michael Cole’s Question about Badges

Posted by Michael M.K. Wednesday, July 4, 2012 0 comments

By Dan Hickey

Michael Cole
I was involved in an exchange on XMCA, the listserv established by Michael Cole’s Laboratory of Comparative Human Cognition and the journal Mind, Culture, and Activity.  I mentioned digital badges in the post, and Mike wrote back to ask:

You know there appear to be several people who appear from time to time on xmca involved in the Mac Arthur initiatives where badges are all the rage. For anyone interested in multi-modal representational practices, it is certainly interesting as a subject of CHAT analysis.

Question:  if you are right in your assumption that the BADGE movement will start a trend what do you think that the trend promises or portends more broadly?

I have taken some time to respond, in part because I wanted to get caught up on the latest work by Cole and his students regarding their successes and challenges around the Fifth Dimension after-school computer clubhouses.  The Fifth Dimension is precisely the kind of educational innovation that should be easier to create, sustain, and study when digital badges are widely used.



What Does this Trend Promise?
Regarding the MacArthur/Gates Badges for Lifelong Learning Initiative, a star-studded kickoff led by Secretary Duncan and articles in the New York Times, Chronicle of Higher Ed, and the Wall Street Journal are quite significant for a $3M initiative. At a recent MacArthur meeting, the policy people who are well connected in DC confirmed that interest in digital badges is exploding in many different agencies and contexts.  According to the hardworking folks at Mozilla Learning (who are defining the Open Badges Infrastructure and shepherding the many open badges efforts beyond MacArthur), interest is exploding around the world as well.  If even a small fraction of the many efforts now underway succeed, badges are going to transform numerous existing learning environments and foster the creation of many new ones.  A longer speculation on this near-term potential is here.

One of the most exciting things that badges promise is new versions of old debates.  There was a pretty good article last week Education Week that is current and well written.  The author cites the concerns that Henry Jenkins and others have raised.  I love Henry's work on participatory culture, and a central goal of my assessment research is fostering participatory cultures within the inevitable constraints of schools (i.e., standards and accountability) and most informal learning environments (e.g., resources and persistence).  So I take his concerns quite seriously.

In particular, I agree with Henry’s argument that education is already "gamified".  So the answer to Mike’s question about what badges promise is really another question: Compared to what?  Given the trivial amount of learning supported by many current formal and informal educational contexts, ANY attention to learning outcomes might be an improvement.   Introducing digital badges is sure to change most learning ecosystems.   On the upside, the incentive value of digital badges is likely to draw attention to dubious credentialing practices and lousy assessments. While stakeholders who have a vested interest in the existing ecosystem are likely to blame the badges, most will agree that such attention is needed and generally helpful.

Certainly some of the changes that follow from digital badges will be bad.  In particular, I worry about the fetishistic obsession with test-driven educational reform expanding to badges.  I believe the policy researchers who argue that overconfidence in test-driven reform undermined achievement in many schools that were already high-achieving before No Child Left Behind.  I worry that the same thing may happen as well-meaning administrators and governing boards insist that high-functioning schools and programs incorporate digital badges.  Consider, for example, that the kinds of participatory learning communities like fan fiction websites that Henry studies could quickly come unraveled by the hasty or forceful imposition of badges; the same thing might occur when badges are hastily or needlessly implemented in high-functioning classrooms or programs.  Henry, along with Mike Cole, John Seely Brown, and Jim Gee have helped us realized that it is the culture that emerges around the technology that matters.  It is going to be the same for badges, though I suspect it will be faster and more dramatic.

I believe it was Nora Sabelliwho said in the 1990s that any new technology will increase learning only to the extent that it increases interaction between teachers and students.  (Dr. Sabelli worked in the White House Office of Science and Technology Policy in the 90s; she is now at SRI International and remains a national leader in educational technology).  I think her logic can be extended to badges: If badges reduce knowledgeable interactions between people, learning of that knowledge will go down.  If badges increase knowledgeable interactions between people, learning of that knowledge will go up.   Where badges are valued and are awarded, we are likely to see more of whatever they are associated with.  This applies to memorizing procedures and definitions just as it applies to mentoring and shared meaning-making. Unfortunately, where valued badges can be obtained by cheating, we are likely to see more cheating.  This concern is likely to discourage some from from awarding valued badges for things that have previously been difficult to assess. 

What do Digital Badges Portend More Broadly?
More broadly, I believe that digital badges are going to help innovators transcend traditional paradigms for supporting learning.  To the extent that these paradigms interfere with efforts to use networked digital technology to support learning, badges may well have a profound impact.  Naively (and hubristically), I used to believe that my research could foster broader appreciation of sociocultural approaches to assessment, motivation, and evaluation.  But I gave up on that long ago.  I now think that the exponential rate of change in networked learning ecosystems will transform education and learning in such profound ways, and that these transformation will allow some and force others to transcend traditional paradigms for assessing, motivating, credentialing, and evaluating learning.  

I believe that we are now at that point: digital badges are the perfect tool for making this happen.  More specifically, I agree that digital badges will be what Cathy Davidson called “the tipping point for Digital Media and Learning.”   I believe this initial transformation and eventual transcendence will be supported by other important factors.  These include the vision of the DML 2012 badges awardees, the passion of entrepreneurs and innovators working outside of the competition, the open-source vibe of the fine folks at Mozilla Learning, the broad scope of Mimi Ito's Connected Learning Network, the great pre-badge examples of connected learning practice, such as Katie Salen’s Quest to Learn and Nicole Pinkard’s YouMedia, and the existing badge-driven ecosystems like Global Kids.

Digital Badges Will Encourage New Paradigms of Assessment
Consider, for example, assessment, and the corresponding concern with the validity of assessment evidence.  Digital badges are going to challenge many stakeholders who have ignored validity concerns or simply taken validity for granted.  Many proponents draw inspiration from existing badge-driven learning environments like Stackoverflow.com.  The badges in Stackoverflow are earned by answering coding questions that visitors find useful.  The badges are highly valued because they mark individuals as crack programmers (and perhaps ones who are too busy advancing the knowledge of their coding community to look for a new job).  The backside of Stackoverflow.com is a tech employment agency.  I understand that Stackoverflow does not have problems with cheating or hustling to boost users’ status because doing so will cause you to get called out in a way that diminishes the value of whatever badges you do have. 

If and how the assessment practices like those at Stackoverflow map over to new badge-driven educational ecosystems remains to be seen.  Fortunately, the fine folks at Peer2Peer University are central to the MacArthur effort and seem to be making real progress in this regard.  At P2PU, users can set up their classes to have badges automatically awarded or can have badges set up so that peers award them.  But a system that allows peers to decide how the system should award badges presents huge opportunities.  Such a system also raises questions about reliability and validity that current assessment theorists have yet to consider.  Carla Casilli’s notions of credibility seem to be going in the right direction.  Initially, I predict that assessment scholars like Jim Popham and Jim Pellegrino will lump credibility in with face validity and other “unsanctioned” forms of validity evidence.  But I think digital badges will eventually lead them to expand our definitions of what counts as validity evidence.

Badges Will Encourage New Paradigms of Motivation
Digital badges will also lead us to reconsider our existing paradigms of motivation.  Where I disagree with Henry and others who worry that badges will likely undermine intrinsic motivation and leave leaners feeling disempowered.   In 1989, Carl Berieter and Marlene Scardamalia argued that the distinction between intrinsic and extrinsic motivation was “too crude” to help define new “intentional” learning environments that take full advantage of communities of learners.   Likewise, in 1989, Collins, Brown, and Newman suggested that the documented negative consequences of competition were more the result of lousy classrooms and the lack of feedback and opportunities to improve.  Both points are more relevant than ever given the open and networked learning ecosystems where digital badges are mostly going to be used. 

Henry’s position on badges nicely illustrates one conclusion I have reached after a decade exploring the practical value of sociocultural theories: many people who embrace newer sociocultural paradigms for learning and instruction slip back into conventional individually-oriented paradigms when it comes to assessing, motivating, and evaluating.  This, in turn, re-ignites the simmering tensions between behavioral/associationist and cognitive/rationalist models of each.  These debates seem to obscure the crucial point about incentive practices in educational videogames and newer networked learning environments: If one offers incentives that empower learners and offer new abilities, then the incentive is unlikely to leave the learner feeling disempowered.

What Would Fifth Dimension Do (With Badges)?
This post has gotten way too long.  My original plan was to explore how digital badges could have been useful for organizing and studying Cole’s Fifth Dimension program.  Specifically, I wanted to ponder the role of badges in the thoughtful discussion of design research methods in the 2011 paper by Downing-Wilson, Lecusay, and Cole.  My initial take is that badges could do a lot—too much to say in this post.  So, for now, I will pitch the questions back to Mike and his colleagues and hope that they will respond here:

The Fifth Dimension was a huge inspiration to many of us.  Your frank discussion of the challenges of sustaining them was refreshing.  In what ways might open digital badges have helped you succeed?

Do you think that the salience that digital badges might grant to valued social practices can enhance the mutual appropriation of interactive practices that you describe in your paper, and help you study it?

How might Cultural-Historical Activity Theory (CHAT) help maximize the positive consequences of digital badges while any minimizing the negative consequences?

I would love to hear what others think about this issue or the general issues raised above.  I am going to post it over on XMCA and try to get them to respond here.  I also want to that Elyse Buffenbarger for helping me edit this post.

Digital Badges as “Transformative Assessment”

Posted by Michael M.K. Sunday, June 10, 2012 0 comments
                                                            By Dan Hickey
               The MacArthur Foundation's Badges for Lifelong Learning competition generated immense
interest in using digital badges to motivate and acknowledge informal and formal learning. The
366 proposals submitted in the first round presented a diverse array of functions for digital
badges. As elaborated in a prior post, the various proposals used badges to accomplish one or
more of the following assessment functions:

               Traditional summative functions. This is using badges to indicate that the earner
               previously did something or knows something. This is what the educational assessment
               community calls assessment of learning.

               Newer formative functions. This is where badges are used to enhance motivation,
               feedback, and discourse for individual badge earners and broader communities of earners.
               This is what is often labeled assessment for learning.

               Groundbreaking transformative functions. This is where badges transform existing
               learning ecosystems or allow new ones to be created. These assessment functions impact
               both badge earners and badge issuers, and may be intentional or incidental. I believe we
               should label this assessment as learning.

This diversity of assessment functions was maintained in the 22 badge content awardees who were
ultimately funded to develop content and issue badges, as well as the various entities associated with HIVE collectives in New York and Chicago, who were funded outside of the competition to help their members develop and issue badges.  These awardees will work with one of the three badging platform awardees who are responsible for creating open (i.e., freely-available) systems for issuing digital badges.
            Along the way, the Badges competition attracted a lot of attention.  It certainly raised some eyebrows that the modestly funded program (initially just $2M) was announced by a cabinet-level official at a kickoff meeting attended by heads of numerous other federal agencies.  The competition and the idea of digital badges were mentioned in articles in the Wall Street Journal, New York Times, and The Chronicle of Higher Education.  This attention in turn led to additional interest and helped rekindle the simmering debate over extrinsic incentives.  This attention also led many observers to ask the obvious question: “Will it work?” 
This post reviews the reasons why I think the various awardees are going to succeed in their stated goals for using digital badges to assess learning.  In doing so I want to unpack what “success” means and suggest that the initiative will provide a useful new definition of “success” for learning initiatives.  I will conclude by suggesting that the initiative has already succeeded because it has fostered broader appreciation of the transformative functions of assessment.



Success Factors for the DML Badges Initiative
The deep pool of submissions, a rigorous review process, and the infusion of additional resources from the Gates Foundation make it likely that most of the funded initiatives will succeed at their stated goals.  Other success factors include the following:

Networked learning communities at HASTAC and Mozilla that have been set up to support badging communities;

The open source ethos of sharing and collaborating that permeates the initiative, and particularly placing the Mozilla foundation at the center of the efforts to define the Open Badges Interface (OBI) protocols;

The inclusion of entrepreneurs, with awards to start-ups both for issuing badges and platforms for issuing them;

New educational frameworks to inform and justify awardees' efforts, including Henry Jenkins’ ideas about participatory learning, and Mimi Ito’s ideas about connected learning.

New collaborative topologies for badges including roles for badges led by Philipp Schmidt and Erin Knight at P2PU, badging systems led by Barry Joseph from Global Kids, and the lexicon for badges led by Carla Casilli at Mozilla[1].

In sum, these factors should allow most of the awardees to succeed in using digital badges to assess learning.  More broadly speaking, their shared success should leave behind a community that is bound together by an emerging set of principles and examples for issuing and awarding badges.

What Will it Really Mean to “Succeed?”
The Digital Media and Learning 2012 request for proposals for badge content and programs included no requirement for documenting learning outcomes. Since I have spent much of my career helping educational innovators meet the stringent expectations for program evaluation at the NSF, DOE, and most other agencies and foundations, this omission was noteworthy.  Proposers were only asked to describe outcomes, including  “learning content, programs, or activities that will be supported by badges,” “skills, competencies, and achievements badges will validate,” “specific identities or roles” that learners might assume, and “opportunities or privileges” that the badges would confer. 
While the RFP asked proposers to describe “existing assessments, if any” that could be used for “tracking or measuring performance,” there was no requirement that formal outcome assessments be included.  Examination of the funded proposals [2] reveals little in the way of formally articulated plans for documenting (a) the learning associated with earning a badge, (b) the learning enhanced by offering badges, or (c) the transformations caused by the introduction of badges.  And none of the proposals even began to consider the evidential, consequential, and systemic validity issues associated with any of the evidence that might be used to document summative, formative, and transformative impact. 
While some observers must have been surprised by the lack of formal evaluation requirements, it made a lot of sense to me.  The actual grants were quite small, and the initiative was quite groundbreaking and practically-oriented.  More importantly, stipulations for formal outcome evaluations would have likely focused on the more salient and measurable summative outcomes, and the corresponding evidential validity issues.  From my perspective, this would have come at the expense of the relatively elusive formative outcomes. This is partly because documenting formative outcomes raises complex issues of consequential validity that assessment scholars like Pamela Moss, Lorrie Shepard, and Samuel Messick have struggled with for decades. Because it is so hard to document formative impact, awardees who tried to evaluate those outcomes would likely have fallen short. What little anecdotal evidence they might have obtained would end up being dismissed by skeptics, allowing critics to argue that their program, and by extension, the larger initiative, had failed.
My point here is that requiring formal evaluation of learning outcomes might have squelched the badges movement before it actually got underway. As everyone agrees, there are currently no “best practices” for issuing and awarding digital badges for learning.  While we all point to initial examples like Global Kids, Stackoverflow.com, and Wikipedia Barnstars, there is currently little actual evidence of learning outcomes associated with these examples.  While some of the DML 2012 Research awardees are likely to generate new evidence of the impact of badges, it is simply too early for most of the DML 2012 Badge Content awardees to formally document summative and formative outcomes, or to grapple with the corresponding evidential and consequential validity issues. 
Some content awardees will soon enough be required to formally document learning outcomes when they seek subsequent additional funding from more conventional agencies and programs.  I hope the assessment community can help prepare them for that time.  In the meantime, I am concerned that some content awardees will be pushed to evaluate formal outcomes earlier than they expected or should.  For example, it seems possible that the introduction of digital badges by more established awardees (e.g., Girl Scouts or 4-H) may cause those organizations to wrestle with some enduring tensions in their own learning ecosystems.  This might lead some stakeholders in those organizations to ask for early evidence that digital badges are worth the trouble.  The obvious concern is that premature efforts to formally document summative and formative learning outcomes will draw resources away from or interfere with efforts to obtain those same outcomes.
This raises ad even broader concern that the rigorous evaluations of learning outcomes will be carried out in the badging contexts that are by virtue of those evaluations least likely to obtain those outcomes.  This effect is rooted in the tensions between formative and summative assessment functions.  Think of it as an inversion of the well-known "Hawthorne Effect" where outcomes are generated by virtue of efforts to document them.  Essentially what happens is that the the effort to formally document summative assessment functions transforms the learning ecosystems in ways that are hard to anticipate or even recognize.  This interferes with the formative functions that are needed to increase the learning outcomes associated with badges, while undermining the validity of the very evidence that might be obtained to document those outcomes.

Evaluating Transformative Outcomes of Badges
            To keep this discussion from getting too complicated, I have so far only focused on evaluating summative and formative learning outcomes.  Things gets a lot more complicated when considering the how assessing learning with badges might be used to transform existing ecosystems or create new ones.  This is because the “learning” associated with transformative assessment defies conventional characterizations of learning.  The learning associated with transformative outcomes is really systemic change.  Such learning is highly contextualized within the social and technological practices that collectively define a specific learning ecosystem.  It is certainly possible to use interpretive methods like ethnography and discourse analysis to obtain rigorous evidence of these transformations.  It is also difficult. In my experience, the naturalistic orientation of most interpretive educational researchers can obscure the more interventionist goals of transformative assessment functions.  Even when systemic transformative outcomes are documented, one is still left with the challenge of linking transformative outcomes to individual formative and summative outcomes.[3] 
This challenge of documenting systemic impact of assessment practices and  linking them to individual outcomes is what Allan Collins and John Frederiksen began exploring in a groundbreaking 1989 paper that introduced the notion of systemic validity.  In the ensuing two decades, the primary response to this challenge has consisted of what proponents labeled design-based research (DBR); most recently William Penuel, Barry Fishman, et al. have introduced a helpful new variant called design-based implementation research (DBIR).  In subsequent posts, I will argue why DBR and particularly DBIR are ideal for both documenting and enhancing the entire range of learning outcomes associated with digital badges.  In short, I believe that design-based educational research methods can provide individual awardees and the broader community with a framework for documenting general and specific principles for designing and awarding badges, while gradually increasing broad learning outcomes and evidence of those outcomes.  But doing so will have to wait until the infrastructure for issuing and awarding badges is established.  In the meantime, I hope that the badge developers and issuers will consider positive ways in which their emerging practices might be transforming their learning  learning ecosystem, and search of potentially unexpected (and possibly undesirable) transformations as well.

New Appreciation of Transformative Assessment
As badge developers eventually seek additional support from foundations or investors, they will likely need to craft well-articulated plans for formally documenting the impact of that investment on learning outcomes.  Thanks to continuing efforts of others in the DML community (most notably Penuel and Fishman), it is possible that these funders will be more sensitive to the challenges and risks associated with formal evaluation of learning outcomes.  More fundamentally, I predict that the collective efforts of the awardees will result in a coherent set of principles and practices for issuing and awarding digital badges.  Accompanying these principles and practices will be self-evident examples of learning ecosystems that have been fundamentally transformed or entirely created by digital badges. This should provide a broadly meaningful way of helping others appreciate the transformative functions of learning assessment.  Specific examples are likely to help others appreciate that not all transformative functions can be anticipated, and that some transformations are undesirable. Hopefully other examples will emerge in which DBR methods have been systematically applied from the very start.
In conclusion, I believe that the Badges initiative has already succeeded, because it has provided a broadly meaningful context for appreciating the transformative functions of assessment.  While I don’t expect that all badge developers will find this information immediately useful, I believe that this post illustrates how the DML initiative has moved this issue beyond the small subset of assessment and measurement scholars who have been grappling with it.  I hope that this information will help awardees who are not specialists in assessment or validity (or even learning) to anticipate and appreciate the complex issues that they will encounter as they go forward, and begin addressing them.  I also hope that readers will send along suggestions for doing so.

Next Up:  Before writing about DBR and DBIR, I plan another post about the likely impact of the Badges initiative beyond the awardees, and introduce the notion of transcendent assessment.



[1] Case in point, Carla Casilli’s deliberations over the badging lexicon led to the helpful distinction between learners and earners, which both raises and addresses huge issue in assessment nomenclature.  The more agnostic term earner is more precise; whether an earner has actually learned something is bound up in the social contract between the earner and the issuer.
[2] The winners are listed in this announcement; their original proposals are linked to the summaries of the larger set of Stage 2 winners.
[3] With all due respect, I acknowledge the 2008 book Transformative Assessment by the esteemed assessment scholar Jim Popham.  I have found it useful for thinking about the broader consequences of summative and formative assessment, and I think many awardees will as well.  But the vision of transformative assessment I have in mind goes quite a bit farther than the one outlined in this book.

Some Things about Assessment that Badge Developers Might Find Helpful

Posted by Michael M.K. Sunday, March 18, 2012 0 comments
Erin Knight, Director of Learning at the Mozilla Foundation, was kind enough to introduce me to Greg Wilson, the founder of the non-profit Software Carpentry. Mozilla is supporting their efforts to teach basic computer skills to scientists to help them manage their data and be more productive. Greg and I discussed the challenges and opportunities in assessing the impact of their hybrid mix of face-to-face workshops and online courses. More about that later.
Greg is as passionate about education as he is about programming. We discussed Audrey Watters’ recent tweet regarding “things every techie should know about education.” But the subject of “education” seemed too vast for me right now. Watching the debate unfold around the DML badges competition suggested something more modest and tentative. I have been trying to figure out how existing research literature on assessment, accountability, and validity is (and is not) relevant to the funded and unfunded badge development proposals. In particular I want to explore whether distinctions that are widely held in the assessment community can help show some of the concerns that people have raised about badges (nicely captured at David Theo Goldberg’s “Threading the Needle…” DML post). Greg’s inspiration resulted in six pages, which I managed to trim (!) back to the following with a focus on badges. (An abbreviated version is posted at the HASTAC blog)




 
A. There seem to be three types of primary goals for badge practices.
A review of the funded and unfunded proposals shows quite a range of goals for badges. Based on the information posted at the DML competition website, the primary goal of most of the badging proposals falls into one of three categories:
  1. Use badges to show what somebody has done or might be able to do. This seems like the goal of badges in the Badgework for Vets and 4-H Robotics proposals.
  2. Use badges to motivate more individuals to do or learn more. Badges in 3D Game Lab and BuzzMath proposals seem to accomplish this.
  3. Use badges to transform or even create learning systems. This is what badges have accomplished in Stackoverflow and seems like the goal of badges in the MOUSE Wins! and Pathways for Lifelong Learning proposals.
These are not mutually exclusive categories. In many cases the second goal encompasses the first goal and the third goal encompasses the first and second goals.
B. These three types of goals appear to correspond with the three primary assessment functions.
Most of these goals require some form of assessment. Whether we like it or not, assessment is complex. Arguably these three goals correspond with three assessment functions (or what others have labeled as purposes):
  1. Summative functions, which are often called assessment OF learning.
  2. Formative functions for individuals, which are often called assessment FOR learning.
  3. Transformative functions for systems, which a few are calling assessment AS learning.
C. Different assessment functions generally follow from different theories of knowing and learning, but these assumptions are often taken for granted. And the relationship between assumptions about learning and assessment practices are often in tension.
  1. Summative functions generally follow from conventional associationist views of learning as building organized hierarchies of very specific associations. These concerns are generally consistent with the “folk psychology” views of learning as “more stuff.”
  2. Formative functions follow from modern constructivist theories of learning as constructing conceptual schema by making sense of the world. These include socio-constructivist theories that emphasize the role of sociotechnical contexts in the way that individuals construct meaning.
  3. Transformative functions follow from newer sociocultural theories of learning as participating in social and technological practices.
The key point here is that these three assessment functions often conflict with each other in complex ways. In particular, summative functions often undermine formative and transformative functions. This is because ratcheting up the stakes associated with summative functions (i.e., the value of the badge) often requires assessments that are “indirect” and “objective” like an achievement test. As John Frederiksen and Allan Collins pointed out back in 1989, such assessments have limited formative and transformative potential, compared to more direct and subjective performance and portfolio assessments. Appreciation of this point requires a foray into the complex realm of validity:
D. Each assessment function raises distinct validity concerns.
Getting serious about assessment means thinking about evidence, and that quickly raises the issue of the trustworthiness of the evidence. Validity has to do with the trustworthiness of the evidence for whatever claims one wishes to make about assessment; validity is not the same as reliability, which is a property of the assessment itself. Each of the three assessment functions raises different concerns about the validity of the corresponding evidence:
  1. Summative functions raise concerns about evidential validity: How convincing is the evidence that this person has done or will be able to do what this badge represents? Many assessment theorists like Jim Popham break this down further into content-related, criterion-related, and construct-related evidence. Measurement theorists like Sam Messick break it down even more, but these distinctions are probably too nuanced for now.
  2. Formative functions raise concerns about consequential validity: How convincing is the evidence that these badges led individuals to do or learn more? Consequential validity is often broken down into intended consequences (always desirable) and unintended consequences (usually undesirable).
  3. Transformative functions raise concerns about systemic validity: How trustworthy is the evidence that this educational system might not exist if we had not used badges? Frederiksen and Collins pointed out that the systemic validity of an assessment practice is linked to its directness and subjectivity.
This is where having multiple goals for a single set of badges, or having different goals for different badges, in a single system can get exceedingly complicated. My point here is that badge developers should consider the various goals for their badges, and the assumptions behind those goals. Failing to do so can create “wicked” tensions that are impossible to resolve. This can be toxic to educational systems because stakeholders ascribe those tensions to other things (politics, laziness, culture, faddism, etc.).
In response to my first draft of this post, Greg summarized my point more succinctly and more generally:
People have different philosophical models of education (whether they realize it or not) and that is why they talk across each other so often.
Greg also inspired me to suggest the following contribution to Audry Watters' top ten list of questions you can ask to find out if somebody really knows education, and if you want to know if they know about educational assessment:
Do you understand the difference between summative, formative, and transformative functions of assessment and how they interact?

The Void Between Colleges of Education and the University Teaching and Learning

Posted by Michael M.K. Tuesday, October 27, 2009 1 comment
In this post, I consider the tremendous advances in educational research I am seeing outside of colleges of education and ponder the relevance of mainstream educational research in light of the transformation of learning made possible by new digital social networks.

This weekend, the annual conference of the International Society for the Scholarship of Teaching and Learning took place at Indiana University. ISSOTL is the home of folks who are committed to studying and advancing teaching and learning in university settings. I saw several presentations that are directly relevant to what we care about here at Re-Mediating Assessment. These included a workshop on social pedagogies organized by Randy Bass, the Assistant Provost for Teaching and Learning at Georgetown, and several sessions on open education, including one by Randy and Toru Iiyoshi, who heads the Knowledge Media Lab at the Carnegie Foundation. Toru co-edited the groundbreaking volume Opening up Education, of which we here at RMA are huge fans. (I liked it so much I bought the book, but you can download all of the articles for free—ignore the line at the MIT press about sample chapters).

I presented at a session about e-Portfolios with John Gosney (Faculty Liaison for Learning Technologies at IUPUI) and Stacy Morrone (Associate Dean for Learning Technologies at IU). John talked about the e-Portfolio efforts within the Sakai open source collaboration and courseware platform; Stacy talked about e-Portfolio as it has been implemented in OnCourse, IU’s instantiation of the Sakai open source course collaboration platform. I presented about our efforts to advance participatory assessment in my classroom assessment course using newly available wikis and e-Portfolio tools in Oncourse (earlier deliberation on those efforts are here; more posted here soon). I was flattered that Maggie Ricci of IU’s Office of Instructional Consulting interviewed me about my post on positioning assessment for participation and promised to post the video this week (I will update here when I find out).

I am going to post about these presentations and how they intersect with participatory assessment as time permits over the next week or so. In the meantime, I want to stir up some overdue discussion over the void between the SOTL community and my colleagues in colleges of education at IU and elsewhere. In an unabashed effort to direct traffic to RMA and build interest in past and forthcoming posts, I am going to first write about this issue. I think it raises issues about the relevance of colleges of education and suggests a need for more interdisciplinary approaches to education research.

I should point out that I am new to the SOTL community. I have focused on technology-supported K-12 education for most of my career (most recently within the Quest Atlantis videogaming environment). I have only recently begun studying my own teaching in the context of developing new core courses for the doctoral program in Learning Sciences and in trying to develop online courses that take full advantage of new digital social networking practices (initial deliberations over my classroom assessment course are here). I feel sheepish about my late arrival because I am embarrassed about the tremendous innovations I found in the SOTL community that have mostly been ignored by educational researchers. My departmental colleagues Tom Duffy, who has long been active in SOTL here at IU, and Melissa Gresalfi have recently gotten seriously involved as well. The conference was awash with IU faculty, but I only saw a few colleagues from the School of Education. One notable exception was Melissa’s involvement on a panel on IU’s Interdisciplinary Teagle Colloquium on Inquiry in Action. I could not go because it conflicted with my own session, but this panel described just the sort of cross-campus collaboration I am aiming to promote here. I also ran into Luise McCarty from the Educational Policy program who heads the school’s Carnegie Initiative on the Doctorate for the school.

My search of the program for other folks from colleges of education revealed another session that was scheduled against mine and that focused on the issue I am raising in this post. Karen Swanson of Mercer University and Mary Kayler of George Mason reported on the findings of their meta-analysis of the literature on the tensions between colleges of education and SOTL. The fact that there is enough literature on this topic to meta-analyze points out that this issue has been around for a while (and suggests that I should probably read up before doing anything more than blogging about this issue.) From the abstract, it looks like they focused on the issue of tenure, which I presume refers to a core issue in the broader SOTL community: that SOTL researchers outside of schools of education risk being treated as interlopers by educational researchers, while treated as dilettantes by their own disciplinary communities. This same issue was mentioned in other sessions I attended as well. But significantly from my perspective, it looks like Swanson and Kayler looked at this issue from the perspective of Education faculty, which is what I want to focus on here. I have tenure, but I certainly wonder how my increased foray into the SOTL community will be viewed when I try to get promoted to full professor.

I will start by exploring my own observations about educational researchers who study their own university teaching practices. I am not in teacher education, but I know of a lot of respected education faculty who seem to be conducting high quality, published research about their teacher education practices. However, there is clearly a good deal of pretty mediocre self-study taking place as well. I review for a number of educational research journals and conferences. When I am asked to review manuscripts or proposals for educational research carried out in classrooms in the college of education, I am quite suspect. Because I have expertise in motivation and in formative assessment, I get stacks of submissions of studies of college of education teaching that seem utterly pointless to me. For example, folks love to study whether self______ is correlated with some other education relevant variables. The answer is always yes, (unless their measures are unreliable), and then there is some post hoc explanation of the relationships with some tenuous suggestions for practice. Likewise, I review lots of submissions that examine whether students who get feedback on learning to solve some class of problems learn to solve those problems better than students whose feedback is withheld. Here the answer should be yes, since this is essentially a test of educational malpractice. But the studies often ignore the assessment maxim that feedback must be useful and used, and instead focus on complex random assignment so that their study can be more “scientific.” I understand the appeal, because they are so easy to conduct and there are enough examples of them actually getting published to provide some inspiration (while dragging down the over effect size of feedback in meta-analytic studies). While it is sometimes hard to tell, these “convenience” studies usually appear to be conducted in the author’s own course or academic program. So, yes, I admit that when that looks to be the case, I do not expect to be impressed. I wonder if other folks feel the same way or if perhaps I am being overly harsh.

Much of my interest in SOTL follows from my efforts to help my college take better advantage of new online instructional tools and to help take advantage of social networking tools in my K-12 research. While my colleagues in IU Bloomington and IUPUI are making progress, I am afraid that we are well behind the curve. While I managed to attend a few SOTL sessions, I saw tremendous evidence of success that I will write about in subsequent posts. Randy Bass and Heidi Elmendorf (also of Georgetown) showed evidence of deep engagement on live discussion forums that simply can’t be faked; here at IU, Phillip Quirk showed some very convincing self-report data about student engagement in our new interdisciplinary Human Biology Program, which looks like a great model of practice for team-teaching courses. These initial observations reminded me of the opinion of James Paul Gee, who leads the MacArthur Foundation’s 21st Century Assessment Project (which partly sponsors my work as well). He has stated on several occasions that “the best educational research is no longer being conducted in colleges of education.” That is a pretty bold statement, and my education colleagues and I initially took offense to it. Obviously, it depends on your perspective; but in terms of taking advantage of new digital social networking tools and the movement towards open education and open-source curriculum, it seems like it may already be true.

One concern I had with SOTL was the sense that the excesses of “evidence-based practice” that has infected educational research was occurring in SOTL. But I did not see many of the randomized experimental studies that set out to “prove” that new instructional technology “works.” I have some very strong opinions about this that I will elaborate on in future posts; for now I will just say that I worry that SOTL researchers might get are too caught up in doing controlled comparison studies of conventional and online courses that they completely miss the point that online courses offer an entirely new realm of possibilities for teaching and learning. The “objective” measures of learning normally used in such studies are often biased in favor of traditional lecture/text/practice models that train students to memorize numerous specific associations; as long as enough of those associations appear on a targeted multiple-choice exam, scores will go up. The problem is that such designs can’t capture the important aspects of individual learning and any aspects of the social learning that is possible in these new educational contexts. Educational researchers seem unwilling to seriously begin looking at the potential of these new environments that they have “proven” to work. So, networked computers and online courses end up being used for very expensive test preparation…and that is a shame.

Here at RMA, we are exploring how participatory assessment models can foster and document all of the tremendous new opportunities for teaching and learning made possible by new digital social networks, while also producing convincing evidence on these “scientific” measures. I will close this post with a comment that Heidi Elmendorf made in the social pedagogies workshop. I asked her why she and the other presenters were embracing the distinction between “process” and “product.” In my opinion, this distinction is based on outdated individual models of learning; it dismisses the relevance of substantive communal engagement in powerful forms of learning, while privileging individual tests as the only “scientific” evidence of learning. I don’t recall Heidi’s exact response, but she immediately pointed out that her disciplinary colleagues in Biology leave her no choice. I was struck by the vigorous nods of agreement from her colleagues and the audience. Her response really brought be me back down to earth and reminded me how much work we have to do in this regard. In my subsequent posts, I will try to illustrate how participatory assessment can address precisely the issue that Heidi raised.

Positioning Portfolios for Participation

Posted by Michael M.K. Thursday, October 1, 2009 0 comments
Much of our work in our 21st Century Assessment project this year has focused on communicating participatory assessment to broader audiences whose practices we are trying to inform. This includes:

  • classroom teachers whose practices we are helping reshape to include more participation (like those we are working with in Monroe County right now);

  • other assessment researchers who seem to dismiss participatory accounts of learning as “anecdotal” (like my doctoral mentor Jim Pellegrino who chaired the NRC panel on student assessment);

  • instructional innovators who are trying to support participation while also providing broadly convincing accounts of learning (like my colleagues Sasha Barab and Melissa Gresalfi whose Quest Atlantis immersive environment has been a testbed for many of our idea about assessment);

  • faculty in teacher education who are struggling to help pre-service teachers build professional portfolios while knowing that their score on the Praxis will count for much more (and whose jobs are being threatened by efforts in Indiana to phase out teacher education programs and replace them with more discipline-based instruction);

  • teachers in my graduate-level classroom assessment course who are learning how to do a better job assessing students in their classrooms, as part of their MA degree in educational leadership.


It turns out that participatory approaches to assessment are quite complicated, because they must bridge the void between the socially-defined views of knowing and learning that define participation, and the individually-defined models of knowing and learning that have traditionally been taken for granted by the assessment and measurement communities. As our project sponsor Jim Gee has quite succinctly put: Your challenge is clarity.

As I have come to see most recently, clarity is about entry. Where do we start introducing this comprehensive new approach? Our approach itself is not that complicated really. We have it boiled down to a more participatory version of Wiggins' well known Understanding by Design. In fact we have taken to calling our approach Participation by Design (or if he sues us, Designing for Participation). But the theory behind our approach is maddeningly complex , because it has to span the entire range of activity timescales (from moment-to-moment classroom activity to long-term policy change) and characterizations of learning (from communal discourse to individual understanding to aggregated achievement).

Portfolios and Positioning
Now it is clear to me that the best entry point is the familiar notion of the portfolio. Portfolios consist of any artifacts that learners create. Thanks to Melissa Gresalfi, I have come to realize that the portfolio, and the artifacts that they contain, are ideal for explaining participatory assessment. This is because portfolios position (where position is used as a verb). Before I get to the clarity part, let me first elaborate on what this means.

It turns out that portfolios can be used to position learners and domain content in ways that bridges this void between communal activity and aggregated attainment. In a paper with Caro Williams about the math project that Melissa and I worked on together, Melissa wrote that

“positioning, as a mechanism, helps bridge the space between the opportunities that are available for participation in particular ways and what individual participants do”

Building on the ideas of her doctoral advisor Jim Greeno (e.g., Greeno and Hull, 2002) Melissa explained that positioning refers to how students are positioned relative to content (called disciplinary positioning) and how they are positioned relative to others (called interpersonal positioning). As I will add below, positioning also refer to how instructors are positioned relative to the students and the content (perhaps called professorial positioning). This post will explore how portfolios can support all three types of positioning in more effective and in less effective ways.

Melissa further explained that positioning occurs at two levels. At the more immediate level positioning concerns the moment-to-moment process in which students take up opportunities that they are presented with. Over the longer term, students become associated with particular ways of participating in classroom settings (these ideas are elaborated by scholars like Dorothy Holland and Stanton Wortham). This post will focus on identifying two complementary functions for portfolios helps them support both types of positioning.

Portfolios and Artifacts
Portfolios are collections of artifacts that students created. Artifacts support participation because they are where students apply what they are learning in class to something personally meaningful. In this way they make new meanings. In our various participatory assessment projects, artifacts have included

  • the “Quests” that students complete and revise in Quest Atlantis’ Taiga world where they explain, for example, their hypothesis for why the fish in the Taiga river are in decline;
  • the remixes of Moby Dick and Huck Finn that students in Becky Rupert’s class at Aurora Alternative High School create in their work with the participatory reading curricula that Jenna McWilliams is creating and refining.
  • the various writing assignments that the English teachers in Monroe and Greene County have their students complete in both their introductory and advanced writing classes;
  • the wikifolio entries that my students in my graduate classroom assessment course complete where they draft examples of different assessment items for a lesson in their own classrooms, and state which of the several item writing guidelines in the textbook they found most useful.

    • In each case, various activities scaffold the student learning as they create their artifacts and make new meanings in the process. As a caveat, this means that participatory assessment is not really much use in classrooms where students are not asked to create anything. More specifically, if your students are merely being asked to memorize associations and understand concepts in order to pass a test, stop reading now. Participatory assessment won’t help you. [I learned this the hard way trying to do participatory assessment with the Everyday Mathematics curriculum. Just do drill and practice. It works.]


Problematically Positioned Portfolios
Probably the most important aspect of participatory assessment has to do with the way portfolios are positioned in the classroom. We position them so they serve as a bridge between the communal activities of participatory classroom and the individual accountability associated with compulsory schooling. If portfolios are to serve as a bridge, they must be firmly anchored. On one side they must be anchored to the enactment of classroom activities that support students’ creation of worthwhile portfolios. On the other side they must be anchored to the broader accountability associated with any formal schooling.



To keep portfolio practices from falling apart (as they often do) it is crucial that they rest on these two anchors. If accountability is placed on the portfolio, the portfolio practice will collapse. In other words, don’t use the quality of the actual portfolio artifacts for accountability. Attaching consequences to the actual artifacts means that learners will expect precise specifications regarding those artifacts, and then demand exhausting feedback on whether the artifacts meet particular criteria. And if an instructor’s success is based on the quality of the artifacts, that instructor will comply. Such classrooms are defined by an incessant clamor from learners asking “Is this what you want???”

When portfolios are positioned this way (and they often are), they may or may not represent what students actually learned and are capable of. When positioned this way, the portfolio is more representative of of (a) the specificity of the guidelines, (b) their ability to follow those guidelines, and (3) the amount of feedback they get from the instructor. Accountability-oriented portfolios position disciplinary knowledge as something to be competitively displayed rather than something to be learned and shared, and portfolios position students as competitors rather the supporters. Perhaps most tragically, attaching consequences to artifacts positions instructors (awkwardly) as both piano tuners and gatekeepers. As many instructors (and ex-instructors) know, doing so generates massive amounts of work. This is why it seems that many portfolio-based teacher education programs rely so heavily on doctoral students and adjuncts who may or may not be qualified to teach courses. The more knowledgeable faculty members simply don’t have the time to help students with revision after revision of their artifacts as students struggle to create the perfect portfolio. This is the result of positioning portfolios for production.

Productive Positioning Within Portfolios
Portfolio are more useful when they are positioned to support reflection. Instead of grading the actual artifacts that students create, any accountability should be associated with student reflection on those artifacts. Rather than giving students guidelines for producing their artifact, students need guidelines for reflecting on how that artifact illustrates their use of the “big ideas” of the course. We call these relevant big ideas, or RBIs. The rubrics we provide students for their artifacts essentially ask them to explain how their artifact illustrates (a) the concept behind the RBI, (b) the consequences of the RBI for practice, and (c) what critiques others might have of this characterization of the RBI. For example:

  • Students in my classroom assessment course never actually “submit” their wikifolios of example assessments. Rather, three times a semester they submit a reflection that asks them to explain how they applied the RBIs of the corresponding chapter.
  • Students in Taiga world in Quest Atlantis submit their quests for review by the Park Ranger (actually their teacher but they don’t know that). But the quest instructions (the artifact guidelines) also include a separate reflection section that asks students to reflect on their artifact. The reflection prompts are designed to indirectly cue them what their quest was supposed to address.
  • Students in Becky Rupert’s English class are provided a rubric for their remixes that ask them to explain how that artifact illustrates how an understanding of genre allows a remix to be more meaningful to particular audiences.
Assessing the resulting reflections positions portfolios, students, and teachers in ways that strongly support participation. For example, if the particular student’s artifact actually does not lend itself to applying the RBIs, my classroom assessment students can simply indicate that in their assignment. This is important for at least three reasons:

  1. it allows full individualization for students and avoids a single ersatz assignment that is only half-meaningful to some students and mostly meaningless to the rest;
  2. understanding if and how ideas from a course do not apply is a crucially important part of that expertise.
  3. The reflection itself provides more valid evidence of learning, precisely because it can include very specific guidelines. We give students very specific guidelines asking them to reflect on the RBIs conceptually, consequentially, and critically.

For example, the mathematics teachers in the classroom assessment course are going to discover that it is very difficult to create portfolio assessments for their existing mathematical practices. Rather than forcing them to do so anyways (and giving them a good grade for an absurd example), they can instead reflect on what it is about mathematics that makes it so difficult, and gain some insights into how they might more readily incorporate project-based instruction into their classes. The actual guidelines for creating good portfolios are in the book when they need them; reflecting on those guidelines more generally will set them up to use them more effectively and meaningfully in the future.

Another huge advantage of this way of positioning portfolios is that it greatly eliminate a lot of the grading busywork and allows more broadly useful feedback. In the Quest Atlantis example, our research teacher Jake Summers of Binford Elementary discovered that whenever the reflections were well written and complete, the actual quest submission would also be well done. In the inevitable press for time, he just started looking at the artifacts. Similarly in my classroom assessment course, I will only look need to go back and look at the actual wikifolio entries when a reflection is incomplete or confusing. Given that the 30 students each have 8 entries, it is impossible to carefully review all 240 entries and provide meaningful feedback. Rather throughout the semester, each of the students have been getting feedback from their group members and from me (as they specifically request and as time permits). Because the artifacts are not graded, students understand the feedback they get as more formative than summative, and not as instructions for revision. While some of the groups in class are still getting the hang of it, many of the entries are getting eight or nine comments along with comments on comments. Because the entries are wikis it is simple for the originator go in and revise as appropriate. These students are starting to send me messages that, for me, suggest that the portfolio has indeed been positioned for participation: “Is this what you meant?” (emphasis added). This focus on meaning gets at the essence of participatory culture.

In a subsequent post, I will elaborate on how carefully positioning portfolios relative to (a) the enactment of classroom activities and (b) external accountability can further foster participation.

Copyright © 2012 Online Course. All rights reserved.

Super SEO created by Blogger Tuts | Published by GalleryBloggerTemplates.com