salvetore

The blog of Michael de Raadt


19 Comments

Analytics: getting helpful information out of an LMS

At the recent Australian Moot, a hot, emerging topic was “analytics”. Analytics have the potential to help students, teachers and institutions make better choices that lead to improved learning outcomes. Analytics were not possible in traditional education – they are, however, something a learning management system (LMS) can easily provide, if we can just figure out what they are…

What are analytics?

Analytics is a relatively new term; so new, in fact, that my spell-checker doesn’t recognise the word. It’s a real word, though; it must be as there is a Wikipedia entry. The term has been popularised in the online world by Google who offer analytics about your website so you can see details about your site visitors.

But what does the term analytics mean for education in the context of an LMS? At a panel session during the aforementioned moot, the topic was raised and this was the first question asked – what are analytics? One of the panel members, Alan Arnold from the University of Canberra bravely offered a definition: “Analytics are any piece of information that can help an LMS user do their job.”

“Analytics are any piece of information that can help an LMS user do their job.”

Thinking more deeply about the subject, I propose that LMS analytics are useful calculable quantitative gathered collections of information, based on the activities of users within or around the LMS, presented in forms that allow decisions leading to improved learning outcomes. (Quite a mouthful.) There’s lots of information about a user that can be collected in the LMS. The trick is to tease out and combine the useful bits and present them simply.

So the question is not so much “what are analytics?” but is instead “what analyics do you need?” and perhaps “how can you get them?”.

What analytics do you need?

Not all analytics are relevant to all users. If you are a teacher, you’re probably thinking about getting information that can allow you to teach better. If you’re a policy maker at an institution, you’re probably wanting to know how successful your teachers are with their students. Bet let’s not forget the students as well; there is information in the LMS that can help them also.

On the plane back from the Moot I decided it would be worth starting a catalogue of all the different analytics that could be captured in Moodle. At lunch I threw these at Martin and we cranked out a few more.

Analytics useful to Students

Progress
My progress bar blockWith an LMS, it is possible to achieve regular assessment within a course based on a rich set of finely chunked multi-modal activities, and while this can lead to deep learning, it can also be overwhelming for students. It is, therefore, useful for a student to know where they are up to in a course and what they have to do next. Students who use short-term planning tend to be more successful; they just need a quick snapshot of their progress.
Relative success
Deep learners are more successful and deep learners are characterised by meta-cognition about their learning. Providing analytics about their relative success can allow students to know whether they are on track of if they need further exposure to a topic. Relative success can also be used to introduce a competitive element into a cohort, which some educationalists recommend.
Opportunities to interact
If students are studying in isolation, it may not always be apparent when there are chances for them to interact with peers or teachers. Determining the level at which a student is interacting could be seen as an analytic that can be used to direct them to opportunities for communication and collaboration.

Analytics useful to Teachers

Student participation
In an online world, it is more difficult for a teacher to know which students are participating and those needing a push. Students can fail to participate for numerous reasons, usually valid ones. Sometimes a student may need to be encouraged to withdraw from a course and re-enrol later. Where analytics can help is in the determination of the timing of when such decisions need to be made. That’s not to say that such information needs to be complex; it could be as simple as “traffic light” coloured icons next to a list of names of students, ordered by risk.
Student success
Engagement analytics block
Assuming a student is involved, a teacher also wants to know how successful they are. This could be the product of assessment and views of resources. If students are progressing through the course with unsuccessful results, then they may need to be encouraged to re-expose themselves to a topic within the course before progressing further.
Student exposures
Moving away from a course modality where “one size fits all”, it is useful to know how many times a student was exposed to a topic before they were successful. This is a differentiating factor among students in a cohort. If students are progressing with few exposures, perhaps they are finding the course too easy, perhaps even boring, and may need to be challenged further. If students are requiring numerous exposures before they are successful, then perhaps alternate presentations of a topic need to be created to suit the learning preference of particular learners. Such an analytical tool can assist a teacher to deliver learning at an individual level.
Student difficulty in understanding
Through an analysis of exposures and assessment results, it may be possible to determine which topics, or areas within a topic, students are finding difficult. This may indicate areas that need to be revisited in the current delivery or enhanced in a future delivery of the course.
Student difficulty in technical tasks
When students are undertaking learning, the last thing they want is to be stifled by an inability to express their understanding because of by the way a course is set up within the LMS. Students patterns of use within the LMS may indicate they are having such difficulties, and a teacher can be alerted to take action.
Feedback attention
Teachers take time and spend effort creating feedback for students as a reflection of their understanding. It is useful to know which students have paid attention to such feedback, and which students may need to be encouraged to do so. Going beyond this it may be possible to deliver information to a teacher about the effectiveness of their feedback on students’ understandings as reflected in subsequent assessment.
Course quality
In several institutions that I know of, part of the measurement of a teacher’s effectiveness is judged by the quality of the courses they are producing within the LMS, based on a set of metrics. Such measurements can be used for promotions and to drive the development of PD activities. If such metrics can be automated, then analytics can be produced for teachers that encourage them to improve their course by increasing the richness of their resources, improving the quality of their activities, including more activities of different kinds, providing more opportunities for students to interact or collaborate.

Analytics useful to Institutions

Student retention
Analytics can provide more information about students than simple pass/fail rates. Analytics can help determine when students may be at risk of failing and in which courses this is more likely to happen. Such analytics can help an institution to send resources to where they are needed most and to plan resources for the future.
Teacher involvement
There may be ethical implications in monitoring teacher involvement in a course as it is akin to workplace survelance. However there is information in an LMS that can be presented in a useful way in relation to training and promotions. It might also be useful to anonymously tie in a teacher involvement analytic with other analytics to find correlations.
Teacher success
As well as looking at success in terms of pass and fail, it may also be possible to determine where teacher interventions have encouraged students to achieve beyond their expected outcomes.
Relative course quality
Clearly not all courses are equal, but how do you determine which is better. There have been a number of attempts to manually measure aspects of a course such as accessibility, organisation, goals and objectives, content, opportunities for practice and transfer, and evaluation mechanisms (Criteria for Evaluating the Quality of Online Courses, Clayton R. Wright). If such metrics can be automated, then analytics can be created with can reflect the quality of courses. Such metrics could also be fed back to teachers as an incentive to improve their courses.
What analytics would you add to this list?

How can you get them?

So, you want these analytics, but how can you get them? Some of them may already be accessible via various mechanisms, however I think we still need to work out how best to draw this information together in a simple way for specific users.

Moodle currently logs most actions that take place within the LMS. It is possible to view log reports, but they are limited to interaction in terms of activities within a course.

There are a number of existing plugins and extensions to Moodle that attempt to provide analytics to users. Among these there are a batch of report generators, many of which are quite configurable.

  • Configurable reportsThe Configurable reports block plugin is a configurable system that allows reports to be created and used by various roles. It may be a good model to use to start a set of standard analytics reports within an institution.
  • The Custom SQL queries report plugin allows an administrator to run any query against the database used by Moodle. It’s clearly flexible, but not something you can put into the hands of all users.
  • The Totara LMS is a rebranded, extended version of Moodle. One of the aspects built onto the standard Moodle install is a reporting facility that provides customisable reports to users of different roles.

There are also a number of blocks available and in-the-works that attempt display analytical information to users.

  • Graph statsMy own Progress Bar block shows a simple view of course progress to students and an overview of student progress to a teacher.
  • The Engagement analytics block is currently in development, but I have seen a demo of this and it looks good. now available. The block allows a teacher to specify expected targets for students then presents to teachers simple traffic-light icons next the names of students at risk.
  • The Course Dedication block estimates the time each student has spent online in a course.
  • The Graph Stats block shows overall student activity in a course over time.

Simple queries

A lot of these analytics can already be queried or calculated from the data already stored in the Moodle databse. The Report plugin type is available for presenting pages if information to users and is applicable for analytics. The Block plugin type is available for simple, compact presentation of information. Both of these APIs can present different displays to users with differing roles.

New logging

Currently, most of the logging that takes place in Moodle ends up in a single table. For simple lookups, this is not a problem, but for complex conjunctive queries, working with a large log table can hog the resources of a server. The current system of logging is likely to be a target of future work at Moodle HQ so that both the recording and retrieval of information can be achieved efficiently.

Measurement of a number of the interactions required for the analytics above is not possible using the current log and module tables. Viewing the recording of user interactions from an analytical perspective may lead to new information being captured for later analysis and presentation.

AI or user optimised queries

When you have a wealth of user interaction information available, why stop at the limits of human developers.

  • Genetic algorithms, neural networks and other heuristic approaches may reveal newly refined analytics or even new analytics altogether.
  • Crowd sourced optimisation of analytics reports may allow a collective intelligence to refine analytics so that they are even more valuable and reliable.
Are there other tools or technologies that you can think of that would help gather analytics?

Analysing analytics

Providing analytics allows us to study the impact that analytics can have on the users who view them. This allows general research questions such as “what analytics promoted better learning, teaching or retention?” Also, specific questions can be asked about individual analytics, such as “does feedback attention have an impact on learning outcomes?” Queue the savvy educationalists…


4 Comments

Moodle is Mainstream

The Underdog

BlackborgA few years ago, while attending a Moodle Moot, I realised that the Moodle community had a serious underdog complex. The mere mention of Blackboard raised emotions reminiscent of an Orwellian hate rally. But even then, I had the impression that Moodle was taking hold and becoming more commonplace, and I was forced to consider: what happens when Moodle is no longer the underdog?

When I first started working at Moodle, I had to explain to my friends and family what Moodle was. Some of my former university-related colleagues knew about Moodle, but weren’t sure how I could be working for Moodle.

Of late, I’ve been frequently encountering random people who know what Moodle is (and have opinions about it), which gives me cause to believe that Moodle is no longer just a small competitor in the LMS game.

  • While at the recent Australian Moot, while in a hotel elevator I was wearing my Moodle shirt; I was asked by another guest if I was here for the “Moodle Conference”. I said I was and asked if she was also. Apparently she was not, but had heard that the conference was happening in town and went on to tell me that her school was using Moodle and she thought it was great. All this in a few floors.
  • At a barbecue with my son’s swimming club, I was asked where I worked. When I mentioned Moodle, several people spontaneously started sharing their Moodle experiences – submitting work online, discussing ideas in forums and monitoring children’s homework. The scientist in me made me sit back and gather evidence, which turned out to be positive.
  • I’ve had similar fleeting experiences on trains, at church gatherings and with old school friends.

It’s even more surprising to hear random people mention Moodle when schools and universities usually re-brand their instance of Moodle, calling it “Study Desk” or “iLearn” or “Wattle” or something else. People are starting to recognise that the underlying system is Moodle, which is the same system their colleagues are using elsewhere.

In truth, Moodle has been mainstream for some time. In some parts of the world, Moodle has reached saturation at the university and vocational training levels. Even Moodle’s competitors are recognising this and attempting to buy into Moodle.

Why has Moodle become mainstream?

There are a number of reasons I can think of that have caused this shift in status.

  • PluginsTeachers want the LMS to do what they want it to do. Moodle, with it’s treasure-trove of plugins, allows them to do more of what they want, and there’s always the potential to create another plugin. Moodle gains much of its strength from the Developer community.
  • People are becoming more accepting of open-source software. With the successes of Firefox, OpenOffice and others, people are starting to see the value in such software and the quality a community effort can produce. Talking to people involved in other open-source projects, I know that Moodle is one of the most organised projects around. The structure and funding model set up for Moodle is very sustainable and ensures Moodle will not disappear for many years yet.
  • Being open source and free, Moodle is making inroads into smaller educational institutions, and this is something commercial LMSs have not been able to do. Not being pigeon-holed in higher education means Moodle is becoming more of a household name.
  • Moodle is open about its flaws. In my position I can claim to know all the blemishes on the face of Moodle, but you can see them too at tracker.moodle.org. Users now realise that no software is bug free and when a commercial product is presented to them as flawless, it is greeted with scepticism.

Perhaps the biggest reason Moodle is shifting into the mainstream mindset is because of the community sentiment that surrounds it…

Satisfaction

This is a bit tongue-in-cheek, but it is clear that public sentiment about Moodle is high.

What does the internet think

This is not infallible evidence, but I dare you to try this test with other LMSs.


2 Comments

Moodle – Now with more SERIOUS

I’ve been thinking of writing this post for a while now, and the list of things to include seems to have been increasing – so here goes.

New SERIOUS staff

Since I have arrived at Moodle HQ there has been a push to increase the number of developers. As Moodle popularity grows, so does the need for maintenance and new feature development. When I arrived in April there were nine employed developers, six more have been added and two more are arriving in coming months. My hope for the near future is to have two teams working on stable development and a third on new features development.

But Moodle is not just about development and in recognition of this, two new roles have been added to the HQ complement.

As a sign of the determination to get things right, we now have a Test Manager. Tim will oversee testing during development and release processes. He brings a great deal of experience as a tester and developer, and will improve testing processes so that the result is serious testing and a more solid Moodle codebase.

Image of Tim Barker

Tim the Test Manager

Barbara brings experience working as a graphic designer in universities, including work with Moodle. She also has teaching experience (in design), so she is highly qualified. But beyond that, she brings a level of style that Moodle has been lacking for some time. A quick look at moodle.org will demonstrate that developers are not very artistic. Undoubtedly the benefit of Barbara joining us will be Moodle – with more serious style.

Image of Barb

Barbara the Designer

New SERIOUS release schedule

It took a long time for Moodle 2.0 to come to fruition. This was a significant step forward (one that many Moodle users are still taking). Afterwards it was decided that it is not necessary to wait for the a complete release roadmap to be fulfilled for each version. Instead, a timed release schedule is now used to create a greater sense of certainty. Part of my role is to “bulldog” developers, to ensure that major releases (such as 2.1 and 2.2) happen on time. We now have major releases every six months in June and December, and since 2.0 releases have happened on time.

Moodle also has minor releases (AKA point releases), such as 2.2.1, that occur between major releases. The main reason for minor releases is to send fixes to security issues out as soon as possible. This might sound like a bit of a ‘number’s game’, but for Moodle Partners and the Moodle community, keeping their Moodle sites up-to-date is a ‘big deal’, and we want to do what we can to help them.

Up to now, minor releases have been completed irregularly when a number of security fixes have come about. But now, we’re getting even more serious. In order to assist in planning upgrades now and into the future, minor upgrades are scheduled to take place every two months, offset by one month to major releases (January, March, May, July, September and November).Release calendar

For more details, see the General release calendar.

New SERIOUS documentation

DocBlock comments in the Moodle codebase

Moodle documentation has, in the past, been fragmented and inconsistent. For new developers, the task of discovering how to work in Moodle as a coding framework has been a daunting one. So to remedy this, HQ developers have put down their development tools to focus on documentation. Starting with core APIs, descriptions are being added and updated to create consistent codebase documentation. This is being linked to Dev Docs wiki pages, again in a consistent manner, to provide developers outside Moodle HQ with the information they need. A planned future addition is consistent ‘getting started’ documentation for each plugin type (currently there are 32 to choose from).

Staying SERIOUS

One of the main focus areas leading up to the release of Moodle 2.3 will be accessibility. With the help of staff from North Carolina State University in the United States, including Greg Kraus and Glenn Ansley, a solid list of accessibility issues has been compiled. Together we will work through these issues to create a release of Moodle that will be accessible to more people. If you would like to be involved in this effort, please help where you can.

As well as this, you can anticipate improvements in usability, specifically file handling, coming soon. (I bet you’re holding your breath.)


2 Comments

The best Moodle extensions you might not have heard about

Last year I wrote a book titled Moodle 1.9 Top Extensions Cookbook. The timing wasn’t great as Moodle 2.0 was just about to be released. Although much of the content is relevant to 1.9 and 2.0, the publishers (Packt) insisted on being version specific. I hope to have the chance to update the book in the future.

I gave a presentation at the Australian Moodle Moot (MootAu11) and I’ve had a few requests to share that presentation since, so I thought I would flesh out my presentation into a more digestible form.

Moodle 1.9 Top Extensions Cookbook Book information:

Moodle receives a 5% royalty on all Moodle books published by Packt.

To receive 20% off when you buy this book or eBook direct from the publisher, enter the discount code MoodleTopEx20.

I’ll start at my number ten and work my way to number one.


10: Course Contents block

Course Contents Block

Click to enlarge

The Course Content block creates an automatic table of contents for a Moodle course. I’ve often seen instructors doing this manually.This is a handy tool when course pages grow long.There is only a version for 1.9 available. In Moodle 2.x there is the Sections block, but it does not show section titles like the Course Contents block.
Author David Mudrák David Mudrak
Type Block
Compatibility 1.9
Links

9: Translation block

Translation Block

Click to enlarge

The Translation block allows you to translate strings into multiple languages, right in Moodle.Application ideas:

  • Language classes
  • Common words in multiple languages (Sudoku, café)
  • Crazy re-translations
  • Translating Moodle strings

There are numerous similar alternatives, but this block works nicely.

Author Paul Holden
Type Block
Compatibility 1.7 to 1.9
Links

8: Hidden Text filter

The Hidden Text filter allows you to reveal hidden snippets of content, as students read it.

Application ideas:

  • Micro-formative assessment
  • Annotations
  • Avoiding over-complicating content
  • Hiding “Easter eggs”

Hidden text before

Hidden text after

Author Dmitry Pupinin Dmitry Pupinin
Type Filter
Compatibility 1.8, 1.9, ?2.0
Links

7: Map module

Map module

Click to enlarge

The Map module allows you to embed dynamic maps into your course. It allows you and your students to add markers on the map.Application ideas:

  • Allowing students to show holiday destinations
  • Geographically, historically or politically significant locations
  • War zones

The potential of this module is really huge.

Author Ted Bowman Ted Bowman
Type Activity Module
Compatibility 1.8, 1.9
Links

6: Twitter Search block

Twitter search

Click to enlarge

The Twitter search block allows you to micro-blog with your students.Searches are based on a configurable hashtag and could be used to search for:

  • Institutional notices
  • Teacher tweets
  • Student tweets
  • Topical news
  • Info popular for students

This is an easy way to bring social media into the classroom.

Author Kevin Hughes Kevin Hughes
Type Block
Compatibility 1.9, 2.x
Links

5: Progress Bar block

Progress Bar block

Click to enlarge

Progress overview

Click to enlarge

The Progress Bar block shows students their progress in a course. It’s an easy way to help students monitor their progress and to motivate them to complete set tasks in the course.In the Moodle 2.x version there is an overview page that shows teachers the progress bar of each student. This allows teachers to see which students are keeping up and which are falling behind.
Author Michael de Raadt (yours’ truly) Michael de Raadt
Type Block
Compatibility 1.9, 2.x
Links

4: Group Selection module

Group Selection module

Click to enlarge

The Group Selection module allows allow students to sort themselves into groups. Once student choose their groups, the groups work as normal Moodle groups.Application ideas:

  • Project groups
  • Interest clubs
  • Research collaborations

This is something I have heard lots of instructors calling out for.

Author David Mudrák, Petr Škoda, Helen Foster, Anna Vanova David MudrakPetr SkodaHelen FosterAnna Vanova
Type Activity module
Compatibility 1.9, 2.x
Links

3: Feedback module

Feedback module

Click to enlarge

Feedback analysis

Click to enlarge

The Feedback module allows you to create custom surveys and analyse the feedback.Application ideas:

  • Attitudinal feedback
  • Student input into future content of a course
  • Express opinions on topical issue
  • Collecting data for statistics lessons

The feedback modules is included as a standard module in Moodle 2.x, but is hidden by default

Author Andreas Grab Andreas Grabs
Type Activity module
Compatibility 1.5 onwards
Links

2: Collapsed formats

Collapsed formats

Click to enlarge

The Collapsed formats allow you to avoid the dreaded “Scroll of death”.Sections can be toggled open and close. The format sets the current week or topic open automatically and each user’s state is remembered when they return to the course page.There is a topics format and a weekly format.
Author Gareth Barnard Gareth Bernard
Type Course formats
Compatibility 1.9, 2.x
Links

1: UploadPDF

UploadPDF

Click to enlarge

The UploadPDF assignment type allows you to mark and annotate assignments without leaving your browser.The only downside is that students have to upload a PDF document, but that’s not such a big deal any more.
Author Davo Smith Davo Smith
Type Assignment type
Compatibility 1.9, 2.x
Links

I struggled to narrow my favourite plugins to a top 10. The following plugins were also great.

  • Lightbox
  • Book
  • Wikipedia filter
  • Peer Review AT
  • Online Users map
  • Mindmap
  • Shoutbox

If you’d like to see my original presentation. Here it is…

http://www.slideshare.net/deraadt/best-extensions-updated-after-moot


Leave a comment

Casual Dev Relationships

So you’re a developer who likes Moodle, but you’re not ready to settle down and make a commitment to an ongoing collaborative project just yet. Well there are bugs out there that are just waiting for someone like you.

Here’s how you can make a small contribution that is likely to help out in a big way.

Step 1. Sign up for an account on Tracker

think of it like a dating site for bugs and devs

The tool used by Moodle to keep track of work is a decked out version of Jira. We just refer to it as Tracker. Head to tracker.moodle.org and create an account.

Step 2. Pick a bug

it’s like speed-dating in binary

You can see a list of bugs by clicking the button labelled Issues and find a bug that’s looking for love.

If you’re just playing the field, pick a minor bug with simple replication steps. If you’re keen to go all the way, check out these searches…

Step 3. Solve the bug

don’t forget to backup first

I’m not going to tell you how to do that. Figuring it out is half the fun.

If you’re looking for some advice, check the Developers forum and and Dev docs.

Step 4. Post a solution

bragging rights

Go back to the bug report and share your solution. You can share a Git branch, a diff patch, or just summarise your solution in a comment. Any form of solution is a good solution.

So your solution can be noticed and integrated, add a label “patch”.

Enjoy!

Our beloved tracker

 


Leave a comment

Hello world!

My Moodle Mugshot

My Moodle Mugshot

Well, I guess this is my obligatory first blog post here on WordPress. I’ll take this as an opportunity to introduce myself and state my intentions for this blog.

So, apart from begin an ordinary guy, I’m Development Manger at Moodle HQ. I’ve come to Moodle after a career in academia. I have been teaching software development for quite a while, in fact, my PhD dissertation focussed on that. But now I’ve escaped and I am making my way through the real world.

In my role as Development Manager I now, apparently, have information that the world is interested in. If you desperately want to know what Moodle developers are working on, I can tell you. If you want to hear what goes on behind the scenes at HQ, I’m there. As well as managing developers at HQ, I also have the privilege of triaging issues as they are reported, so I also know what’s not with Moodle (and if it’s likely to get fixed).

Usually I’m a man of few words. Enough said.