More or Less (Part Six)

- By Woody Brown

A Seminar on the Massive Open Online Course

(in Seven Easy Lessons)

(Link to Lesson One)
(Link to Lesson Seven)

Lesson Six: I Must Be Mistaken

Amazingly, the MOOC I took graded students based solely on peer reviews. In fact, this is the case in every course Coursera offers that requires students to complete assignments that “do not lend themselves easily to automated grading by a computer,” that is, “courses in the Humanities.” This would seem to offend basic logic (i.e. “How can one student, who is by definition someone who is just as unfamiliar with the material in the course syllabus as any other, judge and grade effectively another student’s work?”),[1] but I am of course open to the possibility that peer assessment can be an effective tool. Sadly, on its website Coursera cites only an article called “Calibrated Peer Review™: An Application to Increase Student Reading and Writing Skills,” published in the journal The Biology Teacher, and “Cheap and Fast—But is it Good?  Evaluating Non-Expert Annotations for Natural Language Tasks,” a technical paper about the effectiveness of crowdsourcing “affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation.” The first article was published in 2001, right before the first round of MOOCs crumbled, by an assistant professor of microbiology and molecular genetics. It is unavailable to the members of the public who do not have a paid subscription to BioOne. That is apparently the extent of Coursera’s defense there.

I was required to complete at least six 800-word assignments with a passing grade (five points out of nine possible points) in order to pass my MOOC. Each grade was derived from the added averages of my scores in three separate areas of assessment: argument, evidence, and exposition. Each section could receive a minimum score of zero (“Nothing submitted”) and a maximum score of three (e.g. “The writing is clear and well-constructed”). In each of these sections, we were given the option to leave written feedback. I never received comments from more than two “peers” on any section of an assignment, which leads me to believe that, far from the scale the term “crowdsourcing” implies, my grade was decided by only two other people. For every assignment, I was required to evaluate the work of three other students. Failure to do so resulted in a 20% penalty applied to my score for that assignment. In order to test whether or not the MOOC would reject evaluations that displayed obvious arbitrariness or caprice, on three of the nine assignments I a) gave the students whose work I evaluated obviously thoughtless scores (e.g. scores of three in every section for three students in a row), and b) left no written feedback. I received no notification that the scores I gave were invalidated as a result.

From the outset, my method for testing the MOOC’s evaluation of my own assignments was twofold. First, I would write at least one error into every sentence of every assignment I wrote, and second, I would do exactly none of the reading or preparation assigned besides watching the video lectures. I wanted to make my written work as plausibly erroneous as possible. I wanted to perform the subtle mistakes (errors of spelling, grammar, punctuation, and usage, as well as overwriting and parapraxis more broadly) that abound in high school and undergraduate writing. As an educator and tutor of writing at a private school, and as someone who has spent his entire academic career studying writing, I am intimately familiar with the ways these problems materialize. In response to the first essay question, “How did Kant define Enlightenment? Use Kant’s definition to discuss whether either Rousseau or Marx is an Enlightenment figure,” I began,

On the second sentence of his famous essay, "An Answer to the Question: What is Enlightenment?", Kant writes, "Enlightenment is Man's emergence from his self-incurred immaturity."  I will show in the course of my essay that Marxis, according to Kant's definition, a thinker of enlightenment, but not himself an enlightened man. Much of the reminder of "What is Enlightenment?" consists of the elucidation of numerous qualifiers and objections to the moral directive that Kant precipitates from this first postulation, which directive we understand as the Latin phrase Sapere aude!, more precisely translated as, "dare for to know."

The first letter is an error of substitution (“On” for “In”), the third word is a factual error (the quote is the first sentence, not the second), and the comma after the quotation mark is an error of punctuation. The second sentence contains an error of punctuation (“Marxis” should be “Marx is”), clunky phrasing (it should either read “a thinker of the Enlightenment” or “an enlightened thinker”), and a dubious ad hominem claim. The third sentence contains a spelling error (“reminder” for “remainder”), general overwriting (“the elucidation of numerous qualifiers and objections to the moral directive that Kant precipitates”), and a completely unfounded mistranslation (“dare for to know”).

The following is the final paragraph of my submission for this assignment:

Though Marx stated his "support every revolutionary movement against the existing social and political order of things" (The Communist Manifesto), he was finally unable to liberate his vision of society from the omnipresent overbearance of a ruling class.  Kant, too, admitted the necessity of a strong and powerful leader who, much like a Nietzschean öbermensch, is an enlightened man who, by virtue of his enlightenment, alone has the sagacity and wherewithal to pro- and prescribe.  Because of these thinker's paradoxical encouragement of revolution and steadfast subterranean attachment to the status quo (despite their stated objection), both Kant and Marx were enlightenment thinkers (according to Kant's definition) who were not enlightened themselves (again, according to Kant's definition.

The first sentence is missing a word (either “stated” should be “spoke of” or an “[of]” or “[for]” should be inserted into the quotation between “support” and “every”) and the citation should be italicized. In the second sentence, “öbermensch” is a misspelling of übermensch, which should in any case be italicized. In the final sentence, “these thinker’s” should be “these thinkers’,” “enlightenment” should be capitalized, and the end parenthesis is missing.

The grade I received on this paper? 7 out of 9: an argument score of 2.5, an evidence score of 2, and an exposition score of 2.5. Peer 1, in response to the question, “Did the student develop a point of view in the essay by moving from premises to conclusion using evidence and logic?” wrote, “yes[.]” Peer 2: “Yes. A clear, precise argument.”  “Did the student use quotations or examples from the reading assigned for the week to support their [sic] argument?” Peer 1: “Some[.]” Peer 2: “Yes.” Finally, “Evaluate the quality of the writing. The sentences should be grammatical, and the paragraphs should be well-organized. The conclusion should be clear.” Peer 1: “Good grammar, paragraph's organization needs more work (i.e. second paragraph starts with ‘but’ and seems like it should merge with first one). conclusion not very clear[.]”[2] Peer 2: “The quality of the writing is good.”

After roughly 4,000 words of this, I received an e-mail with the subject line, “Congratulations! You’ve earned your Verified Certificate from Coursera.”  The e-mail read,

“Dear Signature Track Student,

Congratulations on your stellar performance in The Modern and the Postmodern!

We're delighted to inform you that Wesleyan University and Coursera have issued you an official Verified Certificate for this course. You can access your Certificate from your Course Records page.”

Yay.  If I enrolled at Colorado State, I could get college credit for this.

(Link to Lesson One)
(Link to Lesson Seven)