Design Exercise 1 Grade Explanation

We took extra time in grading Design Exercise 1 in order to be careful to provide consistent scores. We built tools that allowed us to assess the consistency of grading that revealed problems. I worked with Claude to arbitrate among the inconsistencies and provide better feedback.

Short version: I apologize for the DE1 grades taking so long to get back to you. We had some problems with the initial grading, so we built tooling to allow us to see what was happening. I used Claude on anonymized assignments to arbitrate grades (select between Cat’s and mine). In the process, it generated detailed feedback on the assignments, and I am using that in the feedback you receive. The codes and comments generally come from Claude (since it provided better feedback than us). The grades match what we would have given you.

What Happened?

Cat and my grades were very inconsistent, and neither of us provided sufficient feedback. Interestingly, we had very similar aggregate statistics, but lots of individual differences.

So, I built some tools that allowed us to look at the grading consistency, and was able to see that things really were inconsistent.

I tried an experiment where I asked Claude to audit the grades (with the intent of using it to arbitrate). I anonymized the submissions and gave them to Claude (along with access to the assignment specifications), and had a dialog around my expectations. It built a grading mechanism (with a detailed rubric), and iterated a few times (which included showing me examples and asking me to explain how I thought it should be graded). It graded the assignments as well, and compared with the existing grades.

It built a detailed rubric (see Design Exercise 1: The AI Rubric) by looking at the assignment, the published information, and then trying to reverse engineer the grades Cat and I gave. It then iterated with me - showing examples for me to explain, and adjusting its opinions until it matched either mine or Cat’s. It used the codes from DE2 (and past DE2-like assignments) that are overkill for a sketch assignment (but it didn’t penalize for them).

It generated much more detailed and complete explanations for its grading. It’s opinions always matched either Cat’s or mine. But, it usually had much better explanations for why it chose what it did.

So, what I’ve decided: I am going to give you what the arbiter chose as your grade and Claude’s feedback (since it is much better than what either Cat or I had to say). I have de-anonymized the grades.

A bit of a warning: While I agree with the grades (they generally match mine, and if they don’t, they match Cat’s), I did not fully check all of the feedback. In particular, it often gives codes that are questionable (especially on a sketch assignment). Fortunately, it didn’t seem to penalize students for ugly codes (and usually, they can be instructive… if this were a real visualization don’t do that).

The Re-Grade policy is the same. If you dislike the fact that Claude influenced your grade, that is a valid reason for requesting a regrade.

I have said that I won’t use AI for determining grades. I believe this process still follows that: Claude did not give you grades. It providing feedback on our grading and provided me with data to use in grading. And, I have checked the results manually.

And if you are wondering, this audit process probably took longer than the initial grading - which was a price to pay for more consistent grading and much better feedback to students.

I use what I’ve learned in grading future assignments. Generating the detailed rubric really requires a set of assignments, but maybe I can do it from partial data.