Reflections on setting up and running the TLC Badges debate

When we launched the latest series of TLCs one of our aims was to experiment with the format of webinar sessions.  Our intention was to explore ways in which video conferencing tools can be used to enable TLCs that are actively engaging rather than passive experiences for participants.

The purpose of this post is not to reflect on the topic, content and outcome of the debate but rather to share some thoughts and experiences of setting up and managing a debate activity (in a web conferencing environment) that was an attempt to provide an engaging and participatory experience.

To try and give some structure to the reflections and keep things simple we’ll set out the key things we tried to achieve with the debate format, how we tried to make this work using the tools available in Adobe Connect, and our reflections on how things work in practice. You may also find it useful to take a look at the Debate: Session Plan.

  1. We wanted participants to be informed of the views of Chrissi and Ale in advance so they would have a chance to reflect upon their own experiences and have questions in mind from the start. In other words we wanted to get participants thinking in advance of the debate.
    To achieve this we published short positioning statements from Chrissi and Ale on the TLC website in advance of the session.  We included this in our emails promoting the debate, shared it via our social media networks (primarily Twitter) and prompted participants joining the webinar to read them while waiting for the session to start.
    As a result over 150 people viewed the positioning statements from Chrissi and Ale on the website.  With hindsight we should have asked how many of the attendees had read the statements at the start of the debate.
  2. We wanted attendees to have the opportunity to reflect on the presentations made at the start of the debate, discuss the presentations and their own experiences with other participants and to formulate challenging questions to put to both Chrissi and Ale.
    To achieve this we decided that breaking out into smaller groups (6-7 people) would be a good way of enabling people to contribute.  Adobe Connect allows hosts to set up break out rooms in advance so we set up 10 rooms.  Breakout rooms had a share pod which had a slide explaining the breakout room activity pre-loaded and a chat pod for participants who may be unable to use the audio tools. At the start of the TLC we asked for volunteers to facilitate the breakout groups.  While the initial presentations were underway Rod allocated volunteer facilitators to breakout rooms and then split the rest of the participants equally between the breakout rooms. Rod started the breakouts at the prescribed time and after 10 minutes he broadcast a message informing breakout rooms that the activity was about to end. Subsequently, he ended the breakout session and brought all participants back to the main room.
    In practice this was trickier to manage than anticipated.  We did not know how many participants to expect so it was difficult to know how many breakout rooms to set up. In the end we only needed 5. We used a tool provided by Connect to automatically distribute participants evenly between breakout rooms so we needed to delete the rooms that were not needed.  A technical issue prevented this from being done so the allocation of participants into breakouts rooms had to be done manually.  This worked but was a little bit stressful to do on the fly. Fortunately, Rod had been specifically allocated this role. It would have been very difficult for a solo presenter to do this on the fly.
    I think it is fair to say that asking for volunteers to facilitate the breakout sessions may have put people on the spot and we did not get as many as we would have liked.  Consequently, Calum and Rod each needed to facilitate one break out room.  Again although this was possible it meant that they became a little bit stretched in relation to managing the overall environment. It would have been better to have arranged the breakout room facilitators in advance of the debate so that Rod and Calum could have been free to move between the breakout rooms and pick up and support any problems.
    Informal feedback on the breakout rooms suggest that they worked really well.  Some participants mentioned that they were participating from shared offices and therefore valued the ability to contribute via the chat pod. One or two participants had technical problems that led to them being moved from the breakout room to the main meeting room. As Calum and Rod were busy facilitating breakout rooms they were not available to help resolve the problem. Fortunately, Ale and Chrissi had remained in the main meeting room and although they were not able to fully resolve the issue they were able to engage those participants experiencing technical problems in discussion that meant they did not feel they were missing out completely.
    Overall, we feel that the breakout rooms were an effective way of involving participants in high quality discussion. The questions that were put to Chrissi and Ale in the following Q & A session were certainly well considered and challenging.  Breakout rooms are definitely something we will make use of in future but we recognise that good organisation and advance planning are very important to make things run smoothly. Key to this, from our experience in the TLC, would be knowledge of the number of expected participants.
  3. We wanted participants as well as Chrissi and Ale to have the opportunity to respond to questions and we wanted to capture/record as many questions as possible during the debate so that if we did not have time to address them in the 60 minute TLC we would be able to follow up later using the TLC website.
    To achieve this we included a round robin Q & A session following the breakout rooms and also set up a Q & A pod in the main meeting room.  This pod allows participants to post questions that are captured and can subsequently be allocated to specific individuals.  Calum explained the purpose of this at the start of the webinar and then again at the start of the round robin session.  Furthermore, we encouraged participants to post their thoughts to the chat pod and/or to raise their hands during the round robin Q & A.
    In practice, the round robin Q & A worked perfectly enabling the nominated the spokesperson from breakout groups to put their questions to Ale and Chrissi.  A retrospective look at the text from the chat pod shows that participants were able to post their own comments in responses to the questions and answers provided by Chrissi and Ale. Had time been available the round robin session could have gone on for much longer and we didn’t manage to to questions from all of the breakout groups. We did however have several questions posted to the Q & A pod which we are in the process of addressing on the TLC website (Chrissie and Ale are going to respond to questions left in the Q & A pod via a blog post which we will share via our social media networks). Overall, our impression is that the round robin session and the use of the Q & A pod have been effective in enabling participants to ask questions and also to share their own thoughts, experiences and opinions.

Summary

I think it is fair to say that more planning and preparation went into the debate than into our previous TLC (See reflections on Don’t plan to present, plan to engage). In part this was because it involved more people in the delivery:  Chrissi and Ale as our debaters; Calum as the debate chair; and Rod working behind the scenes managing the meeting room. Furthermore, we needed to plan for facilitators in the breakout sessions.  However, we feel that the level of engagement and the overall participant experience was worth the additional effort and we are confident that we have learned important organisational lessons that will make this type of webinar easier to set up and deliver in future.

Rod and Calum

Advertisements

Badges debate: Follow-up questions

Many thanks for all of the contributions and engagement during our recent webinar debate on the role of badges in Higher Education.  We always knew that it was going to be difficult in a 60 minute session to cover all of the questions and issues that were likely to arrive which is why we asked for questions to be posted to the Q & A pod.  Some of the questions that were posted were addressed during the debate but for those that weren’t addressed Chrissi and Ale to make a brief response below.  Please feel free to post your own thoughts and responses to the questions to this post using the page comments tool or tweet additional questions and thoughts about badges on the #tlc2015.

Vicki Dale (University of Glasgow):

What level of granularity is the most effective for motivation?

Ale: None. Granularity is not the issue. The serious matter here is the use of short-term extrinsic motivators, such as badges, which have negative effects – as explicitly shown, with the relevant evidence from the literature, during our discussion. 

Granting badges for each activity might be onerous and less meaningful than a single badge for a module or week of a course?

Ale: Granting a badge per week is equally meaningless and just as damaging. Either way, in addition to the above explanation, you’d have the additional disadvantage of badges losing their value even further (assuming such a thing were possible) as a result of issuing large numbers of them – per activity or per week.

Is anyone else ‘branding’ their badges so as to officially affiliate them with their institution, and if so, when is this appropriate e.g. credit-bearing versus non-credit bearing courses?

Ale: Not us (University of Northampton), and not many in HE.

Chrissi: As we engage in much more informal learning opportunities without being assessed for a course etc. I think it is valuable to get confirmation that we are learning something, or that it is recognised what we bring or that we are part of something that is happening. Belonging is an important element of learning that is often overlooked. Badges can play a role in this. How many or how often, this is something that will depend on the purpose of these. As we mentioned briefly they can also be used as self-motivators (check out Albert Bandura’s work linked to this). A badges collection can say loads of different things about the badges owner. Would anybody like to share their collection and reflect on this?

Badges can also be used within formal courses, for formative assessment for example but also for self- and peer evaluation. Why not? They could also become currency that can be exchanges for credits. Why not? At university we use Accreditation of Prior Learning via course or experiential routes. Why would badges and the metadata they hold not be appropriate?

Malcolm Murphy (Durham, UK)

If badges are not valued at the moment then why are so many being issued – given you have to actively click a link to get them? Wouldn’t the system just  become tumblewwed very shortly if no-one wanted them?

Chrissi: Hi Malcolm, I definitely agree with you that the technology is still to complicated. Individuals need to create an additional account to collect these and many don’t want to set-up an additional account as they already have too many. I fully understand this.What needs to happen to simplify the process? What could be happen? Do you have any related ideas you would like to share with us? What we have done with the #LTHEchat for example is placing the blue tweeter badge openly on our site and anybody who participates can collect this and display where they wish. There are no criteria attached to this badge, beyond participation. It requires honesty to take the badge and display it elsewhere. If this is not there and we just display it regardless if we participate or not, then it says more about the person than the badge, I think. My second example is again from the #LTHEchat where we give the Golden Tweeter Award. This is a special award and colleagues who receive this are displayed openly on the site. If anybody takes this award without having received it from the LTHEchat team (and this can happen as anybody could take the code from the site), their details won’t appear on the LTHEchat site as nobody beyond the team has access to this. So the person hasn’t been honest to themselves.We have also introduced badges via the p2pu platform linked to the open course #creativeHE and have found it challenging to understand how the badeging system works there. These examples show an alternative way of receiving badges without creating additional accounts. If anybody has further examples, please share with us here.

We are at an explorative stage and individuals, groups, institutions and organisations experiment with them. This is a good thing! Badges for me, are very much part of the web 2.0 or the social web if you like. With this comes opportunities but also challenges, we need to acknowledge and manage this.

Ale: This discussion was on badges in HE. In that sector, not many are being issued. Some are experimenting with them. That said, I agree: often clicking a link is all you need to do to get them, which speaks volumes about their value (“do this and you’ll get that” – a rewards-based approach). Anyone gets a badge for anything, to quote the badge enthusiasts’ own words. The long-term value of badges, or rather, the complete absence of it, is beyond discussion in HE (and also in other sectors, but that’s the subject of a separate discussion). 

John Maguire (University of Glasgow)

Chrissi, are you using Credly as your main issuing platform. How much evidence are you attaching to each badge?

Chrissi: Hi John, We are using Credly as we have found this an easy to use platform. We can create the badges in there but also award them through this platform and get reports about how many have been awarded and to whom. There will be better systems out there and I am very interested to hear. Regarding evidence, it depends what they are for. It is always clearly stated if they are for participation, recognition or specific achievements and who issues them. Sometimes we do include personal messages. I think we should do this more often. Students say that they value personalised feedback, I think something similar might apply to at least some badge-categories.

Ale: I would add: who believes in that “evidence”?

Brenda UANL Mexico

What do you think about having too many badges? is it still worth it when you have 50+ badges?

Chrissi: Hello Brenda, Does it say more about the person perhaps? Are we talking about an autonomous learner who values opportunities that are presented to them and feels that these little rewards are valuable to them and can be shared easily when needed. They also help individuals boost their confidence and helps them capture their learning journey in a visual way. Often we forget what we achieve along the way, especially if the journey is long and very bumpy. Badges can signalise little milestones that are important to the individual, perhaps more than to anybody else. I find this fascinating that they can work as self-motivators for some.

Ale: Not worth having one, not worth having 50. Dangling badges in front of people as rewards for their efforts (efforts? was it that hard to get the badge? really?) gets us nowhere. This rewards-based Skinnerian approach is damaging to motivation and provides no credible or transferable recognition of any learning whatsoever. It may occasionally work for simple or mindless tasks, only in quantitative terms and only for the short term.

Debbie B

Peer awarded badges for reflective work – is this still extrinsic if learner is doing the reflection anyhow and badge awarded ? I see them as a bonus

Chrissi: Hello Debbie, A very good point. The badge in this case means that somebody has read and reviewed your reflections (linked to a specific activity or course) and thinks that you have met the specific criteria for the badge. As the meta-data is build-in, there is the potential to speed up the feedback process. I do see that it can also lead to depersonalisation. However, there are options to include a personal message linked to the work in the metadata which I think, probably gives the badges additional value for the individual and with whom the badge might be shared.

Ale: I’d suggest that you review the literature on intrinsic motivation. What you describe is a textbook example of a short-term extrinsic motivator, with no value or credibility to those receiving the badge, those “awarding” it, or anyone else. Never a bonus.

Adrian (Freire Institute)

Should we be asking about the value of HE instead and discuss epistemology rather than badges?

Chrissi: Hello Adrian. I think there is room and the need for theory and practice.and link the two. How can we balance the too and explore opportunities to actually start from practice and build theory? What are the advantages? Identifying ways to engage students and educators in learning and development and acknowledge their contributions is really important. It might happen on own but others need a scaffold. There are many ways to do this. One of them could be badges.

Ale: We are free to ask ourselves and the academic community anything we like! There is no case for badges in HE.

Footnote from Chrissi

Hello everybody, it was exciting to debate open badges with you in the context of higher education. Despite the stress that it caused me and endless hours of preparation (and my PhD supervisor won’t be happy as I should be working on my thesis… which is not around badges), I have found it extremely valuable for my own thinking and development and I hope the debate made you think.

I am adding here the presentation I used together with the full notes, which in the end I didn’t really use… but you can see from these how I approached the topic and how I constructed my argument fully linked to my own professional practice and how we have used badges for academic CPD and a variety of related purposes. For me badges are not so much about skills. If you were there or accessed the recording, I didn’t talk about skills at all. Open badges have a wide range of uses. Opening our minds to these will be valuable for all of us. As academic developers it is important to have an open mind and support experimentation. Cutting the wings of our colleagues but also our own before they have fully developed and before they and we collectively can come to our own informed conclusions, is perhaps something to avoid. After all, universities are about curiosity and inquiry and making discoveries. Let’s be the enablers!!!

Thank you Ale, Rod and Calum for making this happen. I am pleased it is over so that I can focus on my PhD again…

Footnote from Ale

Case against badges

Debate: Is there a role for badges in Higher Education?

Our next Teaching and Learning Conversation with take place on Monday 26th October (12:30-13:30 BST) and will debate the question “Is there at role for badges in Higher Education?”

The debate format of this TLC will provide participants with an opportunity to experience something different to the common, single room, large group webinar format by using breakout rooms to allow small groups to discuss the topic and formulate questions to put to the debate leaders. We hope that this will provide a very engaging and participatory webinar experience and give lots of opportunities to interact with other participants and the presenters.

For the uninitiated “A badge is a symbol or indicator of an accomplishment, skill, quality or interest.” (Badge Alliance, 2014).

Our debate will focus on the concept of Digital Badges as an online representation of skills earned and Open Badges that take this concept one step further by allowing verification of skills, interests and achievements through credible organizations and attached metadata (see Badge Alliance, 2014 for further details).

Making the case in favor of badges will be National Teaching Fellow and TLC founder Chrissi Neratnzi. The case against will be put by Professor
Ale Armellini.
 Chriss Nerantzi  Ale Armellini
Read Chrissi’s short positioning statement here. Read Ale’s short positioning statement here.

Please take the time to look at Chrissi’s and Ale’s positioning statements before the webinar debate.

Joining the webinar debate: 

Simply follow this link http://mmu.adobeconnect.com/tlc/ and enter as a guest by typing your name, institution and country into the name field and clicking on the “Join Meeting” button.

Whether or not you have previously participated in a webinar or online activity using Adobe Connect we advise that you make sure that you do some checking and preparation in advance. Check your set-up and connection here.

During the breakout groups all participants will be given the opportunity to use audio and video features should they wish.  However, chat box tools will also be made available to enable those who do not wish to do so to contribute.  We highly recommend that participants take the time to check that they have the appropriate equipment and that the computer is set up to provide the best possible audio video experience during the webinar by following the instructions provided above.

You may also find our Adobe Connect Webinar Participant Guide useful to print out in advance of the session.We really hope that you will be able to join for what should be a lively and highly interactive TLC.

The TLC team

If you require any further information about this webinar debate please contact either Rod Cullen (r.cullen@mmu.ac.uk) or Calum Thomson (c.j.m.thomson@salford.ac.uk)