Design Exercise 1: The AI Rubric
This is an AI generated rubric used by the AI grader for the arbitration/validation check of Design Exercise 1: Sketching Solutions to Real Data Problems
This rubric was generated by Claude given the assignment, access to the course web (and prior course webs), and the human graders’ grades. It was asked to reverse engineer a rubric from the human grades. In a long session, it asked questions - including asking questions about specific (anonymized) assignments so it could refine its judgment.
CS765 Design Exercise 1 — The AI Rubric
GenAI Disclosure:
What follows is completely generated by Claude. Course staff has read it and approved it.The goal of DE1 is to show that you understand visualization: choosing designs and encodings that make the asked-for comparison easy to see, and explaining why. Artistic skill is not graded. A sketch only needs to be clear enough to tell what is encoded and what a viewer would see.
Items
Your submission was graded as five items. An item is all the sketches plus the rationale for one group of questions.
| Item | Questions | What we looked for |
|---|---|---|
| i1 Urbanness | Q1–3 | The same design used for both data scenarios, so a viewer can compare them; differences visible in Q1, similarity in Q2; sensible grouping and aggregation of the county data. |
| i2 Migration | Q4–6 | Addresses what kinds of places people move to (jobs, education, urbanness); rates rather than raw counts where size matters; two designs that emphasize different things. |
| i3 Life-expectancy gains | Q7–9 | Shows both the change and the final outcome, so the viewer can see that the biggest gains are not the highest end points; two different stories; would work for ~200 countries. |
| i4 Your questions | Q10–12 | Five one-sentence questions about trends that the data can answer; a sketch that answers one of them, and a rationale that says which. |
| i5 Sex gap | Q13–15 | The gap is encoded directly or is easy to read; variation by country is visible; two designs that give different insights. |
Item scores (0–5)
- 3 — Good. Valid designs, appropriate encodings, does what the prompt asks, and the rationale names the encodings. This is the expected score for solid work. Moving up or down from 3 needs a specific reason, recorded as a code.
- 4 — Beyond expectations. Something that clearly lifts the whole item: annotation that tells the story, a transform that makes the comparison direct, an adaptation that fits the question, an excellent rationale.
- 5 — Exceptional. Rare; usually a novel design that works very well.
- 2 — Below expectations. A real problem with the item: the prompt isn’t followed, the key comparison isn’t visible, the rationale is missing or contradicts the sketch.
- 0–1. A required part is missing (for example, only one design where a comparison of two was asked).
What earns a 4 depends on the kind of design.
- Standard designs (line, bar, scatter, pie, …) earn credit through attention to detail: labels, ordering, reference lines, highlighting, annotation. A plain standard chart is a 3.
- Adapted designs (a standard chart changed with annotations, extra encodings, alignment or transforms) earn credit when the adaptation makes the target comparison easier.
- Non-standard or compound designs earn credit when the choice makes sense for the story and is readable and explained. Novelty that makes the comparison harder is not a plus.
Two designs in Q1–3 (NSDG). Q1–3 asks for one design shown with two data scenarios. Using two different designs costs a point unless the item has something substantial that makes up for it.
Rationales matter. A strong sketch with a thin rationale, or a rationale that claims something is easy to see when the design doesn’t support it, is usually a 3, not a 4.
Codes
Codes record the reasons behind a score. They are written VALENCE:CODE, where the letter says how the code applies to your work:
- B Bad — a real mistake or violation of a principle
- U Ugly — a missed opportunity or a principle not applied
- N Neutral — worth noting, not counted for or against
- G Good — a principle applied to good effect
- S Superlative — exceptional
The same code can appear with different letters (for example N:NOTINDATA is noted without penalty). Codes are not exhaustive: a missing code does not mean it doesn’t apply.
Design type
| Code | Meaning |
|---|---|
| AD | Adapted design: a standard design used in a non-standard way. |
| CD | Compound design: a good combination of simple chart types. |
| NONSTD | Non-standard design. Creativity is rewarded, but it is best to justify it. |
| NOVEL | A unique, effective design that goes well beyond a standard or adapted chart. |
| NSDG | Q1–3: two different designs where one design with two scenarios was asked. |
| WEIRD | Unusual; not immediately recognizable. |
What the design does well
| Code | Meaning |
|---|---|
| ANNOTE | Annotation for storytelling: callouts, reference or trend lines, highlighted subsets, labels that point to the story. |
| DET | Attention to detail: labeling, ordering, values, reference marks. |
| XFORM | Useful transform: a derived value (change since a baseline, difference from a reference, rate per decade) that makes the comparison directly visible. |
| STORY | The design itself makes the intended finding visible. |
| RICH | Shows several variables at once in a usable way. |
What gets in the way
| Code | Meaning |
|---|---|
| IE | Inappropriate encoding. |
| CHTS | The key comparison is hard to see with this design. |
| NCS | No clear story. |
| PARTIAL | Shows only part of what was asked (e.g. change but not final level; regions instead of countries). |
| NOISY | Clutter, overdraw or decoration obscures the picture. |
| LABELS | Needed labels (axes, series, which end is which) are missing. |
| SCALES | Different or dual scales make the comparison misleading. |
| RAMP | The color ramp doesn’t fit the data (e.g. rainbow for a quantitative value). |
| IA | Inappropriate aggregation (e.g. averaging county rates instead of summing or weighting by population). |
| CALISBIG | Counts rather than rates: absolute values mostly reflect size. |
| PARTWH | A part-whole design used for something that isn’t part-whole, or that throws away the variation asked about. |
| WHYSEL | Data selection is arbitrary or unexplained (e.g. one state, a handful of counties). |
| NMV | Too few items or variables to show the pattern. |
| ND | The two designs are essentially the same and don’t emphasize different aspects. |
| POORX | Too unclear to read the intended design. |
| NOTINDATA | Uses variables that aren’t in the provided data (e.g. GDP, reasons for moving). |
| MINIMAL | Meets the requirements, not much more. |
| CBE | Not wrong, but a better encoding would help. |
Rationale and format
| Code | Meaning |
|---|---|
| RATIONAL | The rationale goes beyond naming encodings: ties choices to the task, discusses tradeoffs or limitations. |
| SELFCRIT | Recognizes what the design gives up. |
| WEAKRAT | Incomplete rationale (encodings not named, no explanation of why the answer is easy to see). |
| NORAT | The rationale is missing or doesn’t describe the designs. |
| CLAIM | The rationale claims something is easy to see that the design doesn’t support. |
| NEEDRAT | The design needs an explanation the rationale doesn’t give. |
| MISMATCH | The rationale describes something the sketch doesn’t show, or the sketches disagree. |
| Q10FMT | Q10 isn’t five one-sentence questions about trends. |
| NOQ | The Q12 rationale doesn’t say which question the sketch answers. |
| MISSINGFILE | A sketch is missing, duplicated, or in a format that can’t be opened. |
From item scores to a grade
The total is 73 plus the five item scores (all 3s = 88). Grade bands: A 90 and above, AB 85–89, B 80–84.
GenAI Statement: Grading information was checked by a Claude audit of anonymized submissions. The Claude generated feedback is mixed in with other sources.