All Readings

Evaluation is such a big and hard question. This will get at the key concepts.

1. Analysis (Chapter 4 from Munzner’s Visualization Analysis & Design) (Munzner-04-Validation.pdf 0.5mb)

   This is a variant of the nested model paper below. It gets the main points. The nested model is really influencial in my thinking.

2. The five qualities of great visualizations (Chapter 2 of The Truthful Art) (theTruthfulArtCh2.pdf 10.0mb)

3. Graphical Integrity (Chapter 2 of Tufte’s The Visual Display of Quantitative Information) (1-VDQI-2-GraphicalIntegrity.pdf 62.2mb)

4. Chris North, “Visualization Viewpoints: Toward Measuring Visualization Insight”, IEEE Computer Graphics & Applications, 26(3): 6-9, May/June 2006. pdf (doi; 4 pages)

   This is a good introduction to the challenges of visualization evaluation. And it’s short.

I would like you to look at one of these two papers. They are examples of very thorough empirical methods. I have chosen them less because of what they are about and more because of their methodology. However, the first one gets at some core issues about how we use statistics in experiments. You can watch the talk for #1 since it was at Vis. I would probably recommend that one, even though the actual technical point is more subtle.

1. Jouni Helske, Satu Helske, Matthew Cooper, Anders Ynnerman, and Lonni Besançon. 2021. Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect. IEEE Transactions on Visualization and Computer Graphics 27, 8 (August 2021), 3397–3409. DOI: https://doi.org/10.1109/TVCG.2021.3073466

   You can also see this on the Vis conference page (since you still have access as you were an attendee): https://virtual.ieeevis.org/year/2021/paper_v-tvcg-9405484.html.

   This paper gets at a common problem with the statistical interpretation of

2. Dragicevic, P., & Jansen, Y. (2018). “Blinded with Science or Informed by Charts? A Replication Study.” IEEE Transactions on Visualization and Computer Graphics, 24(1 (Proceedings InfoVis 2017)), 1–1. DOI PDF

   I want you to read an empirical paper. I pick this one because it takes quite a simple question and tries to be painstakingly thorough with it. Moreover, it is mainly trying to replicate an experiment that got a lot of press. While the authors didn’t set out to contradict the prior paper, it seems they got a different answer to the same question.

Optional

The “Chartjunk” paper would be required reading - except that we’ve already learned about it from Cairo, The Functional Art Chapter 3 (theFunctionalArtCh3.pdf 11.4mb). It’s worth looking at if you’re really interested in the topic. And the Few blog posting may be more valuable than the article itself

• Bateman, S., Mandryk, R.L., Gutwin, C., Genest, A.M., McDine, D., Brooks, C. 2010. Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts. In ACM Conference on Human Factors in Computing Systems (CHI 2010), Atlanta, GA, USA. 2573-2582. Best paper award. project page w/pdf (doi). (10 pages)

  This is a pretty provacative paper. You can pick apart the details (and many have), but I think the main ideas are important. There is a ton written about this paper (those of the Tufte religon view this as blasphemy). Stephen Few has a very coherent discussion of it here. In some sense, I’d say it’s as useful than the original paper – but I would really suggest you look at the original first. While more level-headed than most, Few still has an Tufte-ist agenda. Reading the Few article is highly recommended – in some ways, its more interesting than the original.

• Munzner, T. (2009). A Nested Model for Visualization Design and Validation. IEEE Transactions on Visualization and Computer Graphics, 15(6), 921–928. (pdf) (doi)

  Chapter 4 of Munzner’s book is based on this earlier paper that was quite influential (at least to my thinking). It is somewhat redundant with what is in the chapter, but for completeness, you might want to see the original.

In case you cannot get enough of Tufte, you can get his ideas on what is good (Ch5) and bad (Ch6).

• Fundamental Principles of Analytical Design (Chapter 5 of Tufte’s Beautiful Evidence) (4-BeautEvid-5-FundamentalPrinciples.pdf 14.4mb)
• Corruption in Evidence Presentations (Chapter 6 of Tufte’s Beautiful Evidence) {{ book-link “4-BeautEvid-6-Corruption.pdf” }}

If you’re wondering whether the deceptions Tufte mentions actually fool people, here’s an empirical study of it:

• Pandey, A. V., Rall, K., Satterthwaite, M. L., Nov, O., & Bertini, E. (2015). How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15 (pp. 1469–1478). New York, New York, USA: ACM Press. (doi)

Some other stuff on evaluation:

• Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2011). Empirical Studies in Information Visualization: Seven Scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1520–1536. http://doi.org/10.1109/TVCG.2011.279

• Correll, M., Alexander, E., Albers Szafir, D., Sarikaya, A., Gleicher, M. (2014). Navigating Reductionism and Holism in Evaluation. In Proceedings of the Fifth Workshop on Beyond Time and Errors Novel Evaluation Methods for Visualization – BELIV ’14 (pp. 23–26). New York, New York, USA: ACM Press. ( http://graphics.cs.wisc.edu/Papers/2014/CAASG14)

  What happens when I let my students rant.

• Gleicher, M. (2012). Why ask why? In Proceedings of the 2012 BELIV Workshop on Beyond Time and Errors – Novel Evaluation Methods for Visualization – BELIV ’12 (pp. 1–3). New York, New York, USA: ACM Press. (link)

  Me ranting about how evaluation shouldn’t be an end unto itself. The workshop talk was much better than what I wrote.

• You should read at least one of the papers by Michelle Borkin and colleagues on the memorability of visualization. These papers are very provocative, and provoked some people to be downright mean in attacking it. You don’t need to worry about the details – just try to get the essence. The project website has lots of good information.

  Michelle Borkin et. al. What Makes a Visualization Memorable? pdf InfoVis 2013 (10 pages). This is another radical thought of “maybe Tufte-ism isn’t all there is – and we can measure it.” Again, we can quibble with the details, but they really re getting at something real here.

  Michelle Borkin et. al. Beyond Memorability: Visualization Recognition and Recall. InfoVis 2015. (pdf); 10 pages

Read formatted page...

For uncertainty, I’m not sure what to give you, there are too many choices.

For scientific visualization, it is impossible for me to think of how to give you a brief summary of an entire field in a few readings or a lecture. So we won’t read much.

1. Jessica Hullman How to get better at Embracing Unknowns, Scientific American, Volume 29, Special Issue. (pdf) (official paywalled).

   A nice brief survey.

   If you want more, you could watch this talk at OpenVis. Although, these slides would have been even better.

2. Arrange Spatial Data (Chapter 8 from Munzner’s Visualization Analysis & Design) (Munzner-08-ArrangeSpatial.pdf 1.0mb)

   As close as we’ll get to reading about scientific visualization.

3. Rules of Thumb (Chapter 6 from Munzner’s Visualization Analysis & Design) (Munzner-06-RulesOfThumb.pdf 0.5mb)

   We skipped this chapter, so I’ll add it as a reading for completeness.

I have plenty of “optional” readings… on uncertainty, on scientific visualization, on 3D, etc. If you want more things to read, let me know.

Read formatted page...

Helping you think about presentations is something I like to do in this class (and all grad classes).

I’m not sure how well my annual rant about presentations will work in class this year. Normally, it’s at the end of the semester of you seeing me present. And it tends to be more interactive than the typical monologue. But this year, with everything being online. Plus, the future of presentations might mean online.

The “readings” are:

1. My “notes” on presentations (see the caveat below)

2. Watching a recording of a Hans Rosling talk

My Notes on Presentations

Before reading my notes, here are some caveats (note: this is taken from the 2012 class):

• The goals and standard for presentation really vary across venue/discipline. What we value in computer science (in particular the areas I work in) are quite different than in other disciplines. It’s hard for me to discuss this without value judgement (since I am bred to believe in the “CS way”), but I also plead ignorance to the practices in other area. I’d like to use this as a chance to learn about others.
• I don’t consider myself to be a great presenter. Do as I say, not as I do. The upside of this, is that it means I think about how to be better at it.
• A lecture is not the same as a talk, so what you see in class is quite different than what you would see in one of my talks.
• Even within a particular style/venue/type of talk, there is a wide range of opinions on what is good talk, what the goals should be, …
• The “right answer” depends not only on the situation, but on the person. But that will be one of the biggest lessons I hope you get. I may not speak to your specific case, but hopefully, you can see how the general lessons apply.
• As you might guess, I have strong opinions. But you don’t have to guess at what they are, since I’ve written them down.

Given that…

My real goal is to get you to think about what might make for a good presentation, and to form your own strong opinions – even if they are different than mine.

Given that, read my posting about presentations. Yes, it’s from a 2011 class – but I think if I were updating it, it wouldn’t be much different.

Video Presentations

Hans Rosling was a famous presenter – talking about social issues around the world in venues like TED, etc. He was famous for presenting data in a compelling way to make his points for a broad audience. Sadly, he died this year. But his influence is significant (both on presentating data and on the world in general).

If you haven’t seen a Rosling talk, you need to experience one. If you have seen one, you probably won’t mind watching another.

There are lots of videos of rosling presentations – here’s one I have handy, or here’s another one.

The actual point of Rosling is not his visualizations (he does use standard visualization effectively – often with animation), but rather as a way to talk about presentations.

You might also be interested in his son Ola Rosling’s keynote from this year’s EuroVis. Factfulness on YouTube. It might be the “at home on Zoom” version of a Hans Rosling talk.

Read formatted page...