Reproducibility of research results is one of the core if frequently overlooked axioms of scientific endeavour. Lately it seems to be making something of a comeback through the growth in open data, open source software and open science more generally. If data, code and results are open then anyone can try to reproduce the results with the original or (even better) their own data. Inspired by comments on r-bloggers, one or two new books and our experiences with this approach, we (Stephanie, Luke and I) have tried to apply these research practices to our teaching.
Tools such as R and especially RStudio‘s support for RMarkdown (via knitr) & GitHub have helped in this task by ‘opening up’ the data analysis pipeline and supporting literate programming for data analysis. No longer is there any excuse for forgetting how some result was obtained – if the code is on GitHub and the data is accessible, just review the archived output and if necessary re-run the code. And what if your dataset is updated? No problem, just re-run the code and re-write the interpretation that surrounds it.
But what of learning, teaching, pedagogy? Can the same principles work? We think so. With one year of the University of Southampton’s FEEG 6025 (Data Analysis and Experimental Methods for Civil and Environmental Engineering) module under our belts, we thought we would introduce our graduate students not just to R, RStudio and Git repo (as last year) but to RMarkdown for practical sessions and assignments.
How does it work?
In essence, it works backwards.
At the start of the course we introduced RStudio, RMarkdown and our GitHub repo – although we skimmed over the latter hoping it would enter by osmosis as the course progressed.
Each week we introduce a new concept or method in a lecture session and if the timetabling fairies are nice to us, a day or so later we get to practice the methods in a 2 hour lab.
Backwards? Well we have found that the trick is to write the lab session first as a full RMarkdown document, generating results and capturing outputs. The document is structured into increasingly more complex analyses with questions at the end of each section that we want the students to answer using the results of the analysis they have just run. Initially we asked them to answer these questions in the markdown and then knit the outputs, but more on that later.
Writing the lab session first has the side effect of producing most of the example material we need for the lecture sessions and usually turns the first part of the lab into a race to see who can re-create the results we discussed in the lecture a day or so before. We often use R’s built-in datasets for the simple tasks (mtcars is a fave) but usually migrate to more complex and larger data that is more relevant to the students’ interests. These have included the public use version of the DBEIS NEED household energy consumption data, water porosity of clay slopes and the Irish CER Smart Meter trial data amongst others.
Occasionally we introduce little surprises, sometimes by accident, to keep everyone on their toes. Our favourite is to ‘accidentally’ transform or mess up some of the input data variables to remind the class that real life data is rarely clean and frequently astoundingly messy. Even (perhaps especially) when you have designed the instruments yourself…
Once we are happy with the complete lab RMarkdown ‘results’ document we knit it and keep the results out of harm’s way. And out of sight – so off GitHub!
We then turn off evaluation (knitr::opts_chunk$set(eval = FALSE)), save it as a different file (crucial!) and re-knit the document.
This produces a ‘Task’ html file with the preamble, instructions, questions and chunked R code solutions which we upload to our e-learning system (Blackboard). If we want to stretch the class a little we may leave out some of the code for the more complex tasks. This forces the students to apply what they learn in the first part of the lab to answer the questions in the second. We have found this also helps more able students to progress to more difficult problems while those who need more support concentrate on the initial stages of the tasks.
Finally we take the task RMarkdown document, set knitr::opts_chunk$set(eval = TRUE) and remove all the R code from the main chunks. We leave behind any code which downloads and processes data and put #DO NOT EDIT in big friendly letters all over these chunks. This doesn’t always work – the chunks get edited and everything breaks…
We also leave behind any prose, all the questions and usually any data inspection code (summary(), head(), relevant histograms etc). This is then saved as our lab Template.Rmd file. The Template.Rmd and the Task.html files are made available to the class at the start of the lab (sometimes before) usually via our github repo. The students are asked to clone/copy/download it and work through the tasks, repeatedly knitting the document (or using RStudio’s notebook feature) to produce an incrementally completed session results report.
The answers
After the lab finishes we then release our own original ‘results’ html (or pdf) file to our github repo.
As examples, here are the 3 files used in Week 6 of the 2016-2017 module to introduce linear regression:
How do we grade the outputs?
We don’t.
More precisely: Our grading is done via two assignments and an exam so we do not use the lab sessions for grading purposes. The labs are more like tutorials where we want to support the students to learn how to use appropriate methods and interpret their results.
But of course we want to be able to give feedback on the results produced.
Giving Feedback
Our first attempt at this was to ask each student to upload their knitted outputs (as html or pdf) to the module’s e-learning system. Not only did this provide a permanent archive of work done but it allowed students to share code and results and so learn from their peers. We also reviewed the outputs to give general feedback to the class but found that reviewing each output in a 40+ class was taking a lot of time.
Our next attempt asked the students to work in pairs and upload just one output (with multiple authors). Reviewing these outputs was less time-consuming but still left us with the difficulty of how to give specific feedback both during and after the class.
Our final iteration was to tie the practical session questions to an e-learning system ‘test’ which used the results from the analysis to present automatically marked test questions. Students were able to save and submit their answers on an ongoing basis throughout the lab with incorrect answers given pre-set feedback via the e-learning system and hints of where to look for the correct answers. At the end of the lab the test closed and our own final results output was released via Blackboard. Students were then able to review their ‘test’ answers alongside the correct answers (displayed by the test) and the data analysis results.
What have we learnt?
- Managing data downloads on virtual/managed Windows desktops is tricky. The directory/folder paths that Windows Explorer shows are not consistent with what R/RStudio sees & expects. Don’t try it. Either use the built-in R datasets or hide some real data on a friendly website and use the readr package to load it over the net. This always works – unless you are stuck on a train with no wifi of course…
- Managing RStudio on the same platform is also tricky. Sometimes we have found RStudio mysteriously disappears and/or that we need to re-install a load of packages we installed before. On the other hand our students have got quite used to installing packages at the start of the session;
- Students often forget to load libraries even when they’ve installed them. This can lead to all kinds of confusion;
- Students love to copy & paste. But they don’t like answering questions. We found that some students simply copied the R code from the Task document into the template and ran it. They were pleased with the output and uploaded it to the lab folder but omitted (for whatever reason) to answer any of the questions;
- Using the online ‘test’ seemed to resolve this problem – students seemed to feel that they needed to complete the ‘test’ (even though it did not contribute to their grades).
- Aligning questions (with numbers) in RMarkdown with a test in another medium is tricky. Adding questions out of order (or new ones) can cause confusion. Auto-numbering in RMarkdown has proven difficult (suggestions please!);
- For all its virtues RMarkdown cannot mark itself (yet). So you need to have some way to provide feedback on the results of the session without having to wade through each output file from a 40-strong class. We used Blackboard tests (see above) but other methods might be better.
- That next year we should perhaps try using education.github.com
What’s next?
We’re encouraging our students to prepare their data analysis assignment reports (which are graded) using RMarkdown. We suspect that although they may do this for the analysis/notebook phases, they will probably default to the usual suspects when it comes to writing up. But we will see…
@dataknut 