• Making the brilliant, brilliant-er. PebblePad sets out on a mission to build an even better assessment engine.

    by Shane Sutherland

 

Making the brilliant, brilliant-er. PebblePad sets out on a mission to build an even better assessment engine.

~ Assessment and Feedback ~

APRIL 30th 2018· by Shane Sutherland

insert new block here
 
 
 

Our hope for ATLAS was that work would be shared for assessment in its earliest stages of development, and that feedback from teachers or peers would lead to further learning and to ongoing development. I think it’s reasonable to claim that many of our customers use the platform in just this way.

Shane Sutherland, PebblePad CEO & Chief Mischief Maker

 
insert new block here

 


PebblePad and assessment - a brief history

In this blog I’m going to share our ideas and intentions around the re-development of ATLAS, a huge (and hugely important) part of the PebblePad platform. For people who don’t know, PebblePad is a platform of two parts. Pebble+ is the personal side where users create, upload, make sense of, store, and sometimes share their assets. ATLAS is the institutional side of the platform, the part to which assets are often ‘shared’ in order to be reviewed, assessed or graded.

When we first designed ATLAS, a re-development of the earlier Gateways, we were very keen that this should not be thought of as just an assessment tool - and so we named it the Active Teaching Learning and Assessment Space. Our hope was that work would be shared for assessment in its earliest stages of development, and that feedback from teachers or peers would lead to further learning and to ongoing development of the asset. I think it’s reasonable to claim that many of our customers use the platform in just this way - investing their time and effort in providing rich formative feedback for learners as well as summative feedback alongside grading.

 
PebblePad's Assessment Space
ATLAS is already doing a pretty nifty job on the assessment front but it's time for a refresh.
 
insert new block here
 

Six years into the life of ATLAS and it is time for a refresh - of the platform, not the principles. We have been busy gathering feedback, ideas and suggestions for this redevelopment and I realise that I’ve been rather lazy in the language I’ve used to describe what we’re doing, variously talking about ‘new’ ATLAS, ATLAS v2 or other such attributions which suggest a wholesale ‘out with the old, in with the new’. In reality, the redevelopment of ATLAS will be subtle and transitional. But before I get into the detail, let me start with our mission. 

insert new block here
 
 
 

We've already done the hard miles. We already support processes like blind and double blind marking, moderation, external examination, assessment sets, peer-review, along with some very unique features like pausing, look-back-in-time and feedback templates.

 
 
insert new block here
 

Our Mission

We are going to build a world class Submission and Assessment Management System (S&AMS), a tool that can be used to support the assessment of learning beyond just PebblePad assets. In some respects this is tricky territory because file submission and assessment is typically handled by the LMS or by tools like Turnitin. So why take ATLAS in this direction? Partly it’s a case of improving the platform to catch up with those customers who are already using it in this way, and partly it’s because we have already done the hard miles. What I mean by this is that in order to provide our customers with the richest possible palette of tools to assess portfolio-like items, we already support processes like blind and double blind marking, moderation, external examination, assessment sets, peer-review and have some very unique features like pausing, look-back-in-time and feedback templates.  

There are lots of enhancements we want to deliver to what we might think of as our ‘core audience’ before moving on to offering ATLAS as a standalone S&AMS. We know that ATLAS is already highly regarded as an 'assessment’ platform - but think how much more useful it will be when it supports annotations, offline marking, and even better reporting and analytics! These are changes which both enhance our current offer and make the platform much more capable as a standalone S&AMS.

As I noted above, some of our customers already use ATLAS as their S&AMS - and some do so at rather surprising scale. Some do so because they can’t achieve their desired workflows in any other platform (particularly synchronous, double-blind marking), some because of reliability issues experienced elsewhere, and some because of the pedagogic advantages. We were privileged to have David Boud as a keynote at our 2014 PebbleBash conference. Followers of David will know that he advocates much greater engagement and responsibility for learners in the assessment of learning. There are some simple things ATLAS does to support this, such as allowing feedback to be independently released, with grades following only once learners have responded to the feedback. To finesse that slightly, some courses have what I’m going to do with my feedback templates, whilst others, emulating David Boud’s appeals, have students actually complete templates upon submission signalling how well they think they have attended to the relevant learning outcomes and signposting to the tutor the kind of feedback they would particularly appreciate.

Whilst offline marking is always going to be a very difficult trick to pull off for PebblePad assets, it’s quite an easy feat for word docs, PDFs and the like. On the other hand, we’re pretty sure (I’d like to be more certain, but we’re still at the design stage) that we can introduce a method of annotating ‘stuff’ that’s the same for native PebblePad assets and for files - and further down the line maybe even images and video. We have provided shareable feedback comment banks for 10 years, and even the ability to save new comments to the bank as you type. Our plan is to link these to the annotation tools and to provide analytics to show you where and how often the comments were used - across multiple assessments. Add to this improved rubrics to support everything our current scorecards do (weighting, alt scores), feedback templates with auto-summing tables, and embedded hints for assessors, and it all starts to look like it will meet our twin aims of improving feedback to learners whilst making life easier for assessors.

A wee bit further down the timeline (which I’m conscious I haven’t discussed yet) will be the re-emergence of Flourish as a learner-centric space to keep track of all feedback and grades - as well as a place to plan to learn from this by inviting in tutors, academic advisors and even peers.

insert new block here
 
Timeline Banner Image

insert new block here
 


The timeline

So, the timeline! The first piece of work is to rewrite the code that actually displays submitted assets in ATLAS. Although assets look much the same whether viewed in Pebble+ or in ATLAS, it is actually completely different code that allows them to be displayed. One consequence of this is that whenever we add a new thing to Pebble+, for example the embed block on a template, we have to write one set of code to display it in Pebble+ and one set of code to display it in ATLAS. By making Pebble+ responsible for displaying things submitted to ATLAS we can utilise all the advantages of the responsiveness, speed up development, give a completely consistent view of assets, and make assets similarly accessible in all parts of the system. Having a single view is also an essential pre-requisite for our annotation plans. Alongside the viewer will be a raft of enhancements to the feedback sidebar, including making it responsive. Perhaps the most noticeable change will be its ability to support feedback templates alongside the work being reviewed, rather than in a pop-up box. This work will be released over the UK summer in time for the start of the new academic year.

ATLAS RedevelopmentA mockup of how the 'New ATLAS' interface might look following redevelopment


The next piece of work we are scheduling is the submissions page. The mock-ups (like the one above) seem to be eliciting the right kind of noises from the folks we’ve shared them with. Essentially we’re looking to incorporate an activity dashboard into every workspace to make it really quick and easy to find out which work has been marked, not marked, approved, modified, locked or graded. Alongside this will be our first foray into annotations and the provision of an assessment dashboard for users inside Pebble+.

insert new block here
insert new block here
 

And looking further into the future

After this it gets a little hazy - or rather, it’s a bit too far out to make any commitments in written form! We’ve got some very exciting plans for enhancing peer-review and group assessments and a much wider suite of integrations - particularly supporting alternative plagiarism options. This eases us towards the administration side of ATLAS which, whilst it would benefit from a style make-over, remains very user friendly. At least it is if you’ve had some training. What we really want to do is to build in some assessment wizards that allow you to set up a blind, peer-reviewed assessment, in randomised sets of 4, with a mid-semester pause-point and a modification deadline exactly 47 days after the submission deadline - without needing to know where the various settings live. The wizard will handle that for you.

Obviously we’ll be keeping our community up to speed with our progress, both through our community site and our community events - but also through two webinars happening in July, specifically the 18th and the 19th July - see our webinars page (and download the 2018 webinar schedule).

For more than 10 years we’ve been offering advanced assessment options in ATLAS that still aren’t available in many of the other tools widely used across HE. I’m really excited to now be able to throw lots of our design and development expertise into making ATLAS a world-leading submission and assessment management system for the next 10 years.

insert new block here
 

Author

CEO and founder of PebblePad, Shane has been a big presence in the world of portfolios and personal learning ever since the company's inception in 2004. It's not just his boundless enthusiasm and towering capacity for innovation that keeps PebblePad at the top of its game ... when you're 6' 10" having a list of great ideas 'as long as your arm' is somewhat of a competitive advantage!

 

Comments

Priscilla

Thanks Shane for this insight into the 'new' ATLAS (v 2.0!) and the enhancements that are coming. I too agree that the platform already has so many features that are unique in the market and by bringing these further enhancements it sounds like we'll be in for a real treat. Now... if we could just lock down that timeline! Great work team - I am waiting patiently for the first enhancements to land.




 

Share