Moodle PDF Conversion Plugins: Good, Bad & Ugly
When talking shop with other Moodlers, one common pain point lies with Document Converter plugins. This is my story.
Introduced with Moodle 3.1, Document Converters allow Moodle to convert common file types like Microsoft Word or OpenOffice that are uploaded as Assignment submissions to PDF. This is so the annotation feedback plugin can be used on a wider variety of student submissions. Naturally, many professors enjoy the flexibility and convenience of annotating without requiring students to manage submitting their assignments in the correct file format (talk about herding cats!)
From an administration standpoint, it’s important to note that Moodle does not perform document conversion itself. Rather, the converter plugins allow integration with a separate utility, which can be run on the same server or elsewhere. When a student adds a file to a submission, an entry is created in the Moodle database in the mdl_assignfeedback_editpdf_queue table (I’m sure there’s more to it, but this is a handy way to check how many conversions your Moodle site is running). A scheduled task runs – every 15 minutes by default – to convert any newly-submitted files to PDF so they can be marked up. If a professor happens to visit the grading screen of a submission before it can be processed, they will have to wait while the file is converted ad hoc. This might take anywhere from a few seconds to several minutes depending on the document and the speed of the conversion process.
Worst of all is when an instructor is working with queued submissions of a file type that isn’t compatible with the conversion process, as this often causes lag and long page loads. One of my users reported wait times of up to 2 minutes trying to grade an assignment that student submitted .zip files to before they could save grades and feedback. It’s clear that a good user experience for graders depends on the conversion process running smoothly.
Unoconv
The first Document converter to become available was Unoconv (Universal Office Converter). Unoconv is about as Open Source and Linux-y as it gets: it’s a command line program that runs a special version of LibreOffice to convert files.
When I took over as the Moodle admin at my school, I mostly knew Unoconv by it’s reputation, which is… bad (really bad.) My predecessor had so much trouble keeping unoconv running reliably that one of his last acts before leaving the university was to just disable document conversion altogether. In the wider Moodle-verse, things didn’t seem much better. Much of the sentiment seemed to be that unoconv was difficult and prone to causing headaches. I’ll let Lafayette College’s Charles Fulton (a developer and Moodler of some renown) summarize the vibe:
My users were missing the annotation capability on Word files, but I was not optimistic about using unoconv. Suffice it to say, it was not long before I started looking for alternatives.
Google Drive Converter – Making Progress
A second Document converter for Google Drive was released with Moodle 3.3 in 2017. The Google Drive document converter was added by the Moodle core team as an easier plug and play solution. Check out this YouTube video from Moodle Moot for a great breakdown of the how document conversion works in Moodle and what problems the team was trying to solve.
My main hesitation in using the Google solution was privacy. The way this converter works is to use a Google system account, the same as used to enable oAuth sign-in and Google Drive integration. For document conversion, it sends the file to the system account’s Google Drive, converts to PDF, downloads the result, and then deletes the original upload.
My university is a lapsed Google school (now on O365), so I wasn’t sure I would be able to get a proper, institution owned Google account set-up, and I didn’t want to use a consumer Google account as it would not be subject to the same FERPA-compliant terms – who knows what type of data mining would be done on student work? But, after working with our IT staff, I was able to get a service account under our Google Apps for Edu domain.
The set-up was easy – Moodle provides good documentation on how to set-up things on the Google side, and then in Moodle it’s plug and play with your oAuth account. I was quite pleased with the Google solution. It was an easy win for a new Moodle admin.
The #pivot
Things continued smoothly for well over a year – aside from some blurriness on the converted documents I had no complaints with the Google converter. Then….COVID-19 and online instruction happened. I’m not sure what the root cause was: the increase in web traffic to our Moodle servers, a sizable uptick in assignments being submitted, Google throttling on their end, or some combination of the three, but all of a sudden our conversions could not keep pace. The queue of pending conversions kept growing. Moreover, the scheduled task for conversions started to fail repeatedly, causing Moodle to add a “Fail delay” which has to be manually cleared, compounding the problem and making lots of extra work for yours truly. Meanwhile, professors already stressed by the current state of affairs began to experience significant wait times or performance degradation while trying to use Moodle for grading.
I tried what I could to remedy the situation: I shortened the interval between scheduled task runs from 15 minutes to 5, deleted the backlog in the queue hoping the process could stay caught up after the initial rush, and even set Annotation Feedback to be off by default on new assignments site wide (guess what: even if an assignment has PDF Annotate turned OFF, Moodle still adds its submissions to the conversion queue). Honestly this was the last thing I needed during all online instruction in a pandemic so within a week I decided to take another look at alternatives.
Enter LibreLambda
When researching alternatives, I found two new file converter plugins of interest in the Moodle plugins database. Onedrive and LibreLamba. I quickly ruled out OneDrive as it seems to only be compatible with consumer grade Microsoft accounts, which again raises privacy issues. LibreLamba, however, seemed to offer the best of both worlds. It uses a slick cloud-based solution but allows the end-user to maintain ownership and privacy of student data. The plugin works by allowing us to set-up and use our own conversion process in AWS with serverless Lamba functions and object storage via S3 buckets.
This sounds a bit scarier than it is. All that it requires to set-up are some basic command line skills and a working knowledge of AWS administration – but is far less complicated to manage than unoconv. The plugin is from CatalystIT, who have done lots of great work in the Moodle community and they really did well making this plugin Moodle admin friendly.
With high hopes, I followed the guide on the LibreLambda GitHub page. Creating a user in AWS, installing the Moodle plugins, and running the script to get things set-up was straightforward and painless – only a little more involved than setting up the Google version. After letting the process run for a few days, I’m impressed. It’s fast, efficient, and critically seems to be keeping up with the surge in online assignments at my school. So far looks to be superior to the Google Drive converter. Thank you, Catalyst! I sincerely hope they are able to keep this plugin updated for new versions of Moodle!
- Using Moodle to Respond to Changing Needs – 19th July 2020
- Some Office 365 favorites – 19th May 2020
- Moodle PDF Conversion Plugins: Good, Bad & Ugly – 19th April 2020
Pingback: Feedback Methods Compared - Candidate 1: PDF Annotation - ElearningWorld.org
Does anyone know if there are plans to support LibreLambda beyond the 3.5 and 3.6 cited in the documentation? TIA 🙂
Are you referring to this plugin:
https://moodle.org/plugins/pluginversions.php?plugin=fileconverter_librelambda
As you say, it’s only available for Moodle versions 3.5.and 3.6.
The plugin hasn’t been updated since Sunday, March 31, 2019.
Given that Moodle 3.9 is now the current release, and work towards Moodle 4.0 is under way, my gut instinct is that this plugin may no longer be updated.
If you check the Git repo for the plugin you will find that it is maintained with updates as recently as October, and it works fine on 3.10…I’ve noticed Catalyst tends to forgo updating the official Moodle plugins page for some of their more IT/enterprise focused projects https://github.com/catalyst/moodle-fileconverter_librelambda
You keep mentioning the lack of privacy/security with various types of solutions; but you seem totally comfortable with “cloud-based” solutions such as Google Docs. Aren’t cloud-based solutions, themselves, about as secure and private as Microsoft Windows?
It’s an interesting point.
Disclaimer: I don’t like Google – I don’t like their licensing, and their marketing machine that seems to convince everyone they are the saviour of the internet, which they definitely are not.
However, there is a continuum, between security and ease, and sadly many schools for example will happily encourage their students to “buy into” the Googleverse and Microsoftverse, with little of zero ethical consideration as to what this means.
However, a Google Docs service provides a similar service to a MacD9nalds when you are hungry – it’s fast, cheap, and easy, and we (mostly) ignore the fact it’s unhealthy.
Hi Ben, I’m very comfortable with mysql. I finally made a couple of queries to identify waste records in database. Also find the moosh command to purge them. BUT, I’m not 100% sure about the Moodle database datamodel, so I’ll keep all this “in the fridge” to test it once course is over. I don’t want to mess it up with COVID-19 all around and more than 400 students depending on Moodle 😉
Thanks Ben. I’ve disabled the Scheduled Task “Prepare Submissions for Annotation.”. I’m now trying to find the way to delete all unnecessary converted documents to get rid of 5 GB of space.
Hi Oscar, I’m not sure how you would be able to do this….there might be a way if you are clever enough with MySQL queries. If you can find a list of the file IDs you want to delete, you can use Moosh to purge them: https://moosh-online.com/commands/
Interesing post.
As you said “even if an assignment has PDF Annotate turned OFF, Moodle still adds its submissions to the conversion queue”. I got into trouble with this. Annotate PDF is consuming huge amount of this space in my small Moodle Server.
Did you find a way to delete all unuseful pdf anotations?
Regards
Hi Oscar, I didn’t find a way to do this. If you want to keep unneeded files from being converted into PDFs you can disable the Scheduled Task “Prepare Submissions for Annotation.” However, that would mean that any graders who are using PDF annotation would have to wait for the files to convert when they visit the assignment grading screen. and this can be undesirable. You may just want to look at disabling the Annotate PDF Feedback plugin altogether if your server can’t handle the space.
Fascinating post Ben !
We find with smaller clients that Unoconv works well, but I think with organisations the size of a University, and the huge number of submissions, it doesn’t sound like it scales to that size well.
Like you, I can’t ethically recommend or use Google for this – total privacy should always be our default starting point.
I haven’t used LibreLambda yet.
But now I know it’s from our friends at Catalyst (the “other” Moodle Partner in New Zealand) I will have a look – they are a great company, and we have lots of friends there 🙂