Reading and Discussion 7: Week 7 – Evaluation

by gleicherapi on August 1, 2017

Initial Posting Due: Tue, Oct 17 at (Canvas Link)

Readings

Evaluation is such a big and hard question, and the readings only scratch the surface. The reading list keeps getting longer since there seems to be more and more I want you to know. At some point, I might split empirical studies into its own topic.

  1. Analysis (Chapter 4 from Munzner’s Visualization Analysis & Design) (Munzner-04-Validation.pdf 452 kb)

  2. The five qualities of great visualizations (Chapter 2 of The Truthful Art) (theTruthfulArtCh2.pdf 10.0 mb)

  3. Graphical Integrity (Chapter 2 of Tufte’s The Visual Display of Quantitative Information) (1-VDQI-2-GraphicalIntegrity.pdf 62.2 mb)

  4. Chris North, “Visualization Viewpoints: Toward Measuring Visualization Insight”, IEEE Computer Graphics & Applications, 26(3): 6-9, May/June 2006. pdf (doi; 4 pages)

    This is a good introduction to the challenges of visualization evaluation. And it’s short.

  5. Dragicevic, P., & Jansen, Y. (2018). “Blinded with Science or Informed by Charts? A Replication Study.” IEEE Transactions on Visualization and Computer Graphics, 24(1 (Proceedings InfoVis 2017)), 1–1. DOI PDF

    I want you to read an empirical paper. I pick this one because it takes quite a simple question and tries to be painstakingly thorough with it. Moreover, it is mainly trying to replicate an experiment that got a lot of press. While the authors didn’t set out to contradict the prior paper, it seems they got a different answer to the same question.

  6. You should read at least one of the papers by Michelle Borkin and colleagues on the memorability of visualization. These papers are very provocative, and provoked some people to be downright mean in attacking it. You don’t need to worry about the details – just try to get the essence. The project website has lots of good information.

    Michelle Borkin et. al. What Makes a Visualization Memorable? pdf InfoVis 2013 (10 pages).
    This is another radical thought of “maybe Tufte-ism isn’t all there is – and we can measure it.” Again, we can quibble with the details, but they really re getting at something real here.

    Michelle Borkin et. al. Beyond Memorability: Visualization Recognition and Recall. InfoVis 2015. (pdf); 10 pages

Optional

The “Chartjunk” paper would be required reading – except that we’ve already learned about it from Cairo The Functional Art Chapter 3. It’s worth looking at if you’re really interested in the topic. And the Few blog posting may be more valuable than the article itself

  • Bateman, S., Mandryk, R.L., Gutwin, C., Genest, A.M., McDine, D., Brooks, C. 2010. Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts. In ACM Conference on Human Factors in Computing Systems (CHI 2010), Atlanta, GA, USA. 2573-2582. Best paper award. project page w/pdf (doi). (10 pages)

    This is a pretty provacative paper. You can pick apart the details (and many have), but I think the main ideas are important. There is a ton written about this paper (those of the Tufte religon view this as blasphemy). Stephen Few has a very coherent discussion of it here. In some sense, I’d say it’s as useful than the original paper – but I would really suggest you look at the original first. While more level-headed than most, Few still has an Tufte-ist agenda. Reading the Few article is highly recommended – in some ways, its more interesting than the original.

Chapter 4 of Munzner is based on an earlier paper that was quite influential (at least to my thinking). It is somewhat redundant with what is in the chapter, but for completeness, you might want to see the original:

  • Munzner, T. (2009). A Nested Model for Visualization Design and Validation. IEEE Transactions on Visualization and Computer Graphics, 15(6), 921–928. (pdf) (doi)

In case you cannot get enough of Tufte, you can get his ideas on what is good (Ch5) and bad (Ch6).

If you’re wondering whether the deceptions Tufte mentions actually fool people, here’s an empirical study of it:

  • Pandey, A. V., Rall, K., Satterthwaite, M. L., Nov, O., & Bertini, E. (2015). How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems – CHI ’15 (pp. 1469–1478). New York, New York, USA: ACM Press. (doi)

Some other stuff on evaluation:

  • Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2011). Empirical Studies in Information Visualization: Seven Scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1520–1536. http://doi.org/10.1109/TVCG.2011.279

  • Correll, M., Alexander, E., Albers Szafir, D., Sarikaya, A., Gleicher, M. (2014). Navigating Reductionism and Holism in Evaluation. In Proceedings of the Fifth Workshop on Beyond Time and Errors Novel Evaluation Methods for Visualization – BELIV ’14 (pp. 23–26). New York, New York, USA: ACM Press. (http://graphics.cs.wisc.edu/Papers/2014/CAASG14)

    What happens when I let my students rant.

  • Gleicher, M. (2012). Why ask why? In Proceedings of the 2012 BELIV Workshop on Beyond Time and Errors – Novel Evaluation Methods for Visualization – BELIV ’12 (pp. 1–3). New York, New York, USA: ACM Press. (link)

    Me ranting about how evaluation shouldn’t be an end unto itself. The workshop talk was much better than what I wrote.

Online Discussion

Initial Posting Due: Tue, Oct 17 at (Canvas Link)

This week, the readings will have a lot going on – both about Evaluation in general, and more specific things like experiments in particular.

For your two required postings, I’d like you to address (each in a posting):

  1. Munzner’s framework (the 4 levels of the nested model) is a way to think about all kinds of evaluation (beyond Vis even, but that’s not for us). It’s mainly targeted at “academic” visualization work, but it applies to the thoughts that Cairo and Tufte give us as well. Describe how Munzner’s framework can help us think about each of the other kinds of evaluation perspectives (North’s insight measurement, Tufte, Cairo, and experiments).

  2. You’ve seen some examples of good experimental papers. I’m sortof leaving you to guess about bad empirical evaluation. What do you think makes an empirical study “good”? What can go wrong in terms of making it “bad”? Why do we have to be so careful with studies? This will turn out to be easier than it first appears as there are many things that can go wrong with empirical studies. Think about things that might make you trust a study less, or be less convinced by its findings. (alternatively, think about it in the positive: what do good studies need to convince you, and why do you think this is so important)

A reminder: these questions aren’t just “look it up in the readings.” They are meant to get you thinking about the issues in what you’ve read. The two questions (particularly the 2nd) should give you something to discuss.

There should be enough things here to lead to some conversation.

Print Friendly, PDF & Email

Previous post:

Next post: