DC1 Grading

in Posts

This posting will explain how to interpret your DC1 grades. As posted in Canvas, they can be a little hard to read.

Each assignment is ultimately given an 0-100 score (on the standard 90=A, 85=AB, 80=B scale. The University does not give A+s, but we do in class). This grade is only for the final handin - we did not consider other phases.

The final score was determined subjectively - there is no numerical method for combining the ordinal scores in the rubric.

The assignments were examined by two graders. In some cases, both completed rubric sheets. In other cases, only one grader completed the rubric sheet and the other grader agreed. The two rubric sheets were combined (if there were two) and the final grade was averaged.

Canvas does not allow for formatting of comments, so we put vertical bars “|” at the beginning of each line, which may help.

Each rubric element (listed below) is numbered (section period question number). 1,2,3,4 refer to the designs. 0 is for the general checks and the final grade.

Each rubric element is an ordinal score. The definitions are rough to help us achieve consistency. We use letter codes for common comments we have given. Note: we did not give the comments consistently (you might have deserved one of the comments, but we forgot to mark it).

Basic Overall Checks

0.4 Are there 5 designs and rationales

0 - missing many
2 - missing some rationales
6 - everything there

0.5 Is there diversity in the design types?

0 - all the same
2 - all effectively the same
4 - some variance (not always used)
6 - reasonable variance
8 - shows a diversity of choices

0.6 Describes Methodology

0 - no
6 - sufficient
7 - explains non-trivial, non-standard

Summary / Final grade

These are over the whole assignment

0.7 Finds stories 0 - no 2 - stories do not come out easily from visualizations 3 - lack of multiple stories 6 - presents 4 different stories

Numbers are used to show where in a grade bucket things go. So an “89” is “an AB, barely not an A”, etc.

Scores are not determined numerically - they are determined qualitatively, looking at the “whole portfolio”. For example, if you have one exceptional design, we might overlook a mediocre one.

75 BC

  • turns in all files
  • plausible visualizations
  • some flawed visualizations (lack story, bad encodings)

80 B - everything for BC and

  • complete documentation
  • has simple stories
  • avoids “incorrect” encodings
  • uses appropriate designs
  • little adaptation to make story stand out

85 AB - everything for B and

  • multivariate stories
  • good designs that work for stories (not data dumps)
  • some adaptation / details chosen for stories
  • reasonably effective (or good rationale)

90 A - everything for an AB and

  • interesting and clear stories
  • designs well chosen to make stories clear
  • details chosen to make stories stand out
  • effective designs
  • good rationales

95 A+ - everything for an A

  • the A list, but done really well
  • will have at least 1 really exceptional design

Questions per Design (slightly different for Alternate)

These will be repeated for each of 4 designs (so question 1 would be 1.1, 1.2, …)

Note - error codes might not always be given (just because you don’t have a code, doesn’t mean you don’t have the problem)

Note - numbers are meant to be ordinal, not interval, and sometimes they aren’t really ordinal. Follow with letter codes for common reasonings.

  1. Did we swap 1 and 1A (we want to grade the one we think is better) yes/no - no grade impact, just for information

  2. Story support by the design

     0 - Not obvious
 1 - No real story (e.g., data dump)
 2 - With imagination and/or rationale
 4 - With reading the caption
 5 - Notable story, but only after reading rationale
 6 - Yes
 7 - Notable
 8 - Exceptional story/design connection

 Notable means the story is interesting.

 - H - not clear that story emerges from the visualization (but can get from caption)
 - I - not clear what the story is (it neither jumps out from the visualization or caption) - requires reading rationale
 - J - not clear what the story is even after reading rationale
 - K - story is just a data dump
 - IJK = we're not sure which it is, but its one of these

  3. Completeness

     0 - No - insufficient title, caption, or labels means story does not come out without rationale
 2 - Limited - something there, but not enough
 4 - Overkill - too much text (makes the visualization somewhat redundant)
 6 - Yes - effective title, caption and labels guides reader to the story without forcing it

  4. Multi-Variate

     This refers to both the visualization and the story.

 0 - no (univariate - a distribution)
 1 - no (basic bi-variate)
 2 - slightly (a third variable is not central)
 3 - sortof (a hidden third variable)
 4 - data dump (many variables, but not connected)
 6 - yes (at least 3 variables core to story)
 8 - very much (a story with many variables emerges), good use of multiple variables

  5. Do design choices help focus on the story

     0 - data dump (doesn't help guide)
 2 - data dump is the story
 4 - no active choices seemed needed, basic choices OK, no tuning
 5 - a choice made, but not clear how effective
 6 - yes - design choices emphasize important information
 8 - notable - design choices really help

 - P - basic chart not adapted to make story emerge

  6. Reasonable encodings

     0 - clearly incorrect
 2 - poor choice
 3 - non-standard choice isn't incorrect but has issues
 4 - obvious choice isn't incorrect (but might not be the most effective)
 6 - good choice
 8 - well reasoned, specialized design

 error codes:

 - A - clear misuse of part/whole relationship (e.g., treemap or stacked bar for things that cannot be added)
 - B - unclear use of part/whole encoding (e.g., not obvious if bars are stacking or just on the same axis)
 - C - use of too many colors
 - D - continuous/discrete issues (e.g. line chart for categorical data) - (questionable/incorrect use of) continuous axis design for non-continuous variable
 - E - questionable use of a diverging scale (or lackof)
 - F - split scales hard for comparison
 - G - too much overplotting

  7. Design Effectiveness

     0 - poor overall design
 2 - ineffective design choice
 4 - obvious choice is acceptable
 6 - good choice
 8 - clever design specific that is very effective

 error codes:
 Z - obvious comparison not supported (e.g., not putting things to be compared along a common axis)
 Y - data dump (design does not seem to support a task other than (maybe) specific retrieval, questionably providing too many
 X - not clear that story emerges from the visualization (but can get from reading, or maybe no story)
 W - design might obscure the point, rather than enhance it
 V - lack of ordering makes comparison hard

 P - basic chart not adapted to make story emerge
 Q - main correlation that viewer would look for (either implied or stated in rationale/caption) is not exposed in a clear way

  8. Rationale

     0 - non-existing
 2 - not compelling
 4 - straightforward (did the obvious thing)
 6 - good (correct arguments)
 8 - notable (shows insight into the design problem)

 N - reasoning or argumentation error

Kudos Codes (may be applied anywhere, or just put in notes):

a. Good use of combinations of charts to make a visualization.
b. Good use of design to focus on elements in a sea of data.
c. Particularly interesting story (brings together data or focus to create something unexpected).
d. Particularly compelling design to make the story stand out.
e. Good use of providing context
f. Uses design to combat scale
g. Visualizations give provenence and other background for reliability and trust

For the alternate

A.1 Plausible alternate

0 - flawed visualization (bad encodings, ineffective design)
2 - obviously would have been graded poorly
6 - good enough to be graded

A.2 Compare and Contrast

0 - none
2 - not much discussed
6 - good discussion
8 - exceptional discussion

A.3 Presents the same story in a different way

0 - no
2 - somewhat
4 - yes, but so ineffectively that its hard to say its the same story
6 - yes