Module 4: Evaluation and Formalization (Oct 13-24)

in Modules
Page content

We will be more explicit in how we evaluate visualizations and visualization research. We will provide more formal approaches to considering aspects of design and analysis. Students will create and critique visualizations.

Introduction

This module is a bit of a grab bag of topics - but all have a similar theme: how do we consider aspects of visualization in a more formal way. We’ll talk about evaluation and the connection to statistical reasoning.

A big thing is the “Design Exercise” - it is a little different than usual. There is only one for the module: Design Exercise 4-2: 2 Data Sets, 6 Visualizations - it will count double. The other design exercise (Design Exercise 4-1: OPT-IN exercise) is an optional “collaborative learning opportunity” (CLO) - where we will let students collaborate (asynchronously) if they agree to do the work of the main assignment on a specific schedule. It’s an experiment.

Other than that, it’s the usual: readings, lectures, a content survey, a class survey and a seek and find.

  • The Design Exercise Design Exercise 4-2: 2 Data Sets, 6 Visualizations is big. Start early.
  • The Lectures will provide an overview of the reading (and might be slightly redundant). The readings should work before or after the corresponding lectures.
  • The surveys are meant to be done at the end (but remember the hard deadlines). The content survey should definitely be after the reading.
  • The seek and find is actually based on last week’s reading. You can do it at any time.

Module Learning Outcomes (Goals)

  1. Understand the challenges of evaluating visualizations
  2. Understand the range of evaluation methods and when they are appropriate
  3. Apply the nested model to analyze visualizations
  4. Be aware of some of the more formal approaches to visualization and apply them to design and analysis
  5. Connect visualization to statistical/data science abstractions
  6. Practice creating and critiquing visualizations

Readings

This module is a set of subtopics, each with its own reading.

Evaluation

How do we know if a visualization (or visualization research) is good?

  • (required) Tamara Munzner. Analysis. Chapter 4 from Munzner's Visualization Analysis & Design. (Canvas File) (UW Library)

    The nested model is key to how I think about visualization (and this course). Here is where you will actually read about it. This chapter comes from an older paper. The Chapter is better. You can skim over the examples in the end.

  • (required) Chris North. Visualization Viewpoints: Toward Measuring Visualization Insight. IEEE Computer Graphics & Applications, 26(3): 6-9, May/June 2006. (doi) (web pdf)

    This is a good introduction to the challenges of visualization evaluation. And it’s short.

  • (required) Alberto Cairo. The five qualities of great visualizations. Chapter 2 of The Truthful Art. (Canvas File) (UW Library)

    Cairo gives you his (less academic) take on what makes a visualization good.

  • (ok to read summary) Gordon Kindlmann and Carlos Scheidegger. An Algebraic Process for Visualization Design. IEEE Transactions on Visualization and Computer Graphics, 20(12) 2014.. (doi) (web pdf) (Summary)

    This is an interesting formalism for thinking about visualization. I want you to see it as an example of how we can think about visualization in a formal way. Reading my summary is probably enough.

I was tempted to put a list of optional papers here. There are so many interesting takes on the topic. But realistically, there is already a lot in this module.

Basics of Cognitive and Perceptual Foundations

Note: I am moving perception readings to the next module.

This chapter is one of the most foundational things in the field. It gets at the basics of the cognitive aspects of visualization and some of the first formalisms for thinking about it. Decades later, this is still an essential reading for our field.

  • (required) Card, Mackinlay, and Schneiderman. Information Visualization. The first 17 pages of the Introduction to “Information Visualization: Using Visualization to Think. (Canvas File)

    This is a 1999 book that consists of this intro, and a bunch of seminal papers. The examples are old, but the main points are timeless. It is the best thing I know of that gets at Vis from the cognitive science perspective. The rest of the chapter (past page 17) is good too, but more redundant with other things we’ll read – so it’s optional. Although, every time I go back to it, I am amazed how good this is - despite being old. The authors are the founders of the field.

    The section “How Visualization Amplifies Cognition” (starting on page 15), with Table 1.3 is particularly important. It really gets at why visualizations help us do things.

Connection to Statistical Analysis

  • (required) J. T. Leek and R. D. Peng. What is the question?. Science 347, 6228 (March 2015), 1314–1315. (doi) (web pdf)

    This is only two pages, but it gives a great introduction to the ways we should think about using data, and the terminology statisticians recommend. There’s another paper I like (below), but it is too much statistics to require.

  • (optional) Galit Shmueli. To Explain or to Predict?. Statistical Science 25, 3 (January 2010), 289–310. (doi) (url) (video)

    This paper fundamentally changed the way I think about data. However, it is a bit too statistically involved to require for class. Actually, watching the video on her web page (it was here, but seems to be gone)~ is the best thing to do as it really gets the point across. (there is another video on YouTube - it’s not as concise as the original one, but still good - better than reading the paper).

  • (optional) Emanuel Zgraggen, Zheguang Zhao, Robert Zeleznik, and Tim Kraska. Investigating the Effect of the Multiple Comparisons Problem in Visual Analysis. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. (doi) (url)

    This paper brings up a somewhat scary point about interactive exploration: we need to be careful about its statistical validity. You don’t need to worry about the second half of the paper (the experiment) - although, it is pretty interesting.

Practical Help

If you need some practical advice for your design exercises, I gave some pointers last week. Module 3: Visualizations and Effectiveness (Sep 29-Oct 10) (Readings Part 3: Practical Help)

Lecture Plan

  • Monday 1 (Oct 13) - Evaluation (Nested Model, Insight Quantification, etc.) - a traditional lecture that should set up the readings.
  • Wednesday 1 (Oct 15) - Student Examples - we’ll look at Design Exercises from years past to help you get a sense of what is good (or not).
  • Monday 2 (Oct 20) - Why Vis? A connection to cognition, statistical reasoning, and the history of the (academic) field.
  • Wednesday 2 (Oct 22) - Introduction to perception. (this connects to the next module)

Assignments

Everything else is due at the end of the module, subject to the usual late policies: