DE12: Evaluation

in Design Exercises

Understanding the nested model is really important. So, we’ll have an exercise that will force you to use it and think about how it applies to Design Exercise 10: Project Proposal (which you should be working on).

Background

Make sure you’ve done the readings on evaluation (Readings 13: Evaluation). You might want to wait until after the lecture where I will talk about the nested model. It might even help to look at the paper (the book is probably good enough). Figure 2 from the paper is so helpful for this assignment that I will reproduce it here (it was also in the slides). (it is Figure 4.5 in the book)

nested-threats.png
  • Munzner, T. (2009). A Nested Model for Visualization Design and Validation. IEEE Transactions on Visualization and Computer Graphics, 15(6), 921–928. (pdf) (doi)

A warning: the threats and validations at each layer in this picture are just examples. And the picture doesn’t provide an upstream validation for the 2nd layer. But it is still quite instructive. And it shouldn’t be a surprise (since it was in the readings/lecture).

The Idea

The nested model for evaluation and design gives four levels to think about visualization at:

  1. domain situation
  2. data/task abstraction
  3. encoding/interaction techniques
  4. algorithm

Note: I prefer to think of “algorithm” as “detail”, and call “encoding/interaction idiom” “design”. You can choose my levels (Task, Data/Resources, Design, Details/Implementation) or Munzner’s (they are very similar). I think hers are a little too specific, but they are better adjusted for validation.

In fact, you hopefully recognize the whole thing from my “recipe for visualization” (1 - What Is Visualization and How do We Do It?) where I even said this is where it comes from.

In a sense, the nested model shouldn’t be new to you: it is so ingrained in the way that I think that it should have come across throughout the class.

What might be new, however, is to see how the nested model serves for validation (which is the point of the original paper - the fact that it is so useful for design/development came out afterwards).

One point the paper makes is the concept of “upstream/downstream threats.” That is, there are things that can go wrong at each level. Some of them can be thought about before you descend to the next level (i.e. upstream), some of them only come out after you’ve completed the next level (downstream).

In the paper/book, the focus is on the evaluation technique for each level and upstream/downstream - but you can also think of them as different problems.

Overall, there are 8 places where something can go wrong (4 levels, upstream and downstream), and 8 places you can perform validation.

The Assignment

The idea is to apply the nested model to your project. This will help you think about your project, but also help you learn about the nested model.

You may need to adapt the model based on your project. For example, if your “implementation” is a design sketch and analysis, the inner layer (in Munzner) may not make sense. You should think about how the model can be still be applied.

For each of the 8 “places” describe: a specific example of a “threat” that you might encounter, and what you might do to perform validation that you don’t have that problem.

Note: part of the lesson here is that you can do a lot of validation before you ever implement anything.

For the DE12: Evaluation Exercise (due Tue, Dec 6), there will be a Canvas “quiz” where you answer 8 questions: for each “place” (level x up/down) describe a specific threat for DC3, and how you might test it.

A warning: Canvas will allow you to “re-take” the quiz, but you might lose your prior answers. I strongly recommend you write the answers off-line and copy-paste them into Canvas.

One thing I hope you get out of this exercise is to realize all the potential for getting a design right before you do any implementation. For your project this may give you ideas on how to convince us your project is good, even if you aren’t able to make a great implementation.

In order to help better connect the nested model to the project, there are 2 extra questions beyond a question per place (for a total of 10)

  • Briefly describe your project including (a) the key ideas and (b) a description of the intended artifacts (e.g., will there be an implementation, or just a collection of pictures).
  • For each “place” - describe an example of a validity threat, and what you might do to check for it. For the downstream threats, you may need to imagine that you have actually created things.
  • What is your plan for how you intend to evaluate your project. How will you know that you succeeded (or not)? Note: this should focus on what you can actually do, but you may include what you would want to do (given more time and resources). For example, it is impractical for you to run a user study in the time frame of class - you can say “Given more time, we would run a study (with a description)” - but it is also must say what you actually can do. In practice these quicker evaluations should be a pre-condition for more extensive evaluation (don’t run a user study on something you don’t believe in yourself).

Grading

Everyone must do this exercise independently. If you are working on a team, you must do this independently.

This will be considered as part of assessment.