Some thoughts on the first phase of the Design Challenge

I did not have a chance to read all of the Design Challenge 3 Critiques. But I did read many – and they were almost all really good.

Here is some commentary on them, that might help for phase 2:

1. People took a wide range of level of abstraction to consider. Some people critiqued the specific pictures I made, while others considered the most abstract task (looking at a ranked list), and other people picked things in between. All of this is OK: I didn’t really specify which level of abstraction to consider. Each level has its challenges. (I was going to say that the specific examples are easiest to pick apart, but they also suffer most from lack of context).
2. It is good that people called me out on some very basic graph making things. Guilty as charged! My defense would  say “if we were actually to use one of these diagrams, it would have a good figure caption that explained it, and a story in the text of the paper.”
3. There is a whole host of issues that relate to what the Explainers are, and how they are computed. Some of these come from trying to interpret them out of context (if you’re not a literary scholar, some of these might not make any sense – and even then, you need to be familiar with the Docuscope varaibles). In general, I’d say these are not really Visualization issues – but maybe they are? Maybe there are ways to make the viewer familiar with the meaning of the variables, or other pieces of context.
4. Many people pointed out an entire class of tasks these diagrams are bad at: dealing with specific items. (try to find your favorite play in this, or the city you want to visit in this). I couldn’t label all the points (in order to save space). But what could I have done? (even labeling them all doesn’t completely help)
5. People had mixed opinions on my color choices. Hopefully, you can find more interesting things to comment on too – but the colors are fair game. The colors for the genre experiments do come from a Brewer qualitative ramp, but these are not perfect.
6. Scale is certainly the biggest challenge. The big examples (figures 8-10) show lots of items. (getting to over 1000 – but you can imagine cases where you have 10,000 – or more!) But you could also imagine more than 4 classes (for Figure 9, there are actually 12 genres). You could imagine wanting to compare more than a handful of rankings. …
7. As I think I mentioned, there is VERY limited interaction in these. That’s a potential source of improvement. How would you use interaction to address some of the problems with Explainers.
8. I was thinking that people would focus on the “explainers diagram” (the thing for showing scored/ranked lists). But some people had interesting comments on other diagrams (like the scagnostics-like thing), and some people even had interesting comments on the Explainers concepts themselves. All is fair game, but I don’t want people to feel like they need to understand all the math behind explainers in order to do the assignment. Of course, if you want to improve the math behind them, it needs some help too!
9. It might not be obvious, but the Y dimension is used consistently in the diagrams. For the list, its used to show the rank. For everything on the right of the splines, it is the value on the scale (everything uses the same scale).
10. Many people commented that it is hard to learn to read these diagrams. The fact that I needed to make a special extra-simple example with a long explanation for “training” is proof of these points. Some people pointed out that the task of some of the examples was to explain the visualizations for later tasks, …
11. There is no discussion of how you specify what Explainer you want to see, how to put them together in the pictures, how you specify what is in the picture, … If you really want to know… I have to write Python scripts. My collaborators send me email with questions, and I send them back pictures.
12. It is actually possible to allow for more detailed queries “Find me an explainer of American-ness where Minneapolis is the most American city” – or “Find me an explainer of tragicness where Hamlet is the most tragic of all plays.” There is no interface for doing this (something you could try to add). (the math behind the scenes is also a little ugly, but that’s another story)
13. Understanding the variables is a problem. And one that I think actually could be addressed with some visualization.
14. Comparing orderings is hard. It isn’t shown at all in the bigger diagrams (look at figure 8 – there is no sense that the ordering of the documents is completely different, even if the scores are the same). In the smaller diagrams, you are kindof on your own to make the comparisons. This is a hard general problem you might want to solve. It generalizes well beyond Explainers.

Here’s another (hard) example: 550 plays. (figure10) – I am mainly including it to show that I had already fixed some of the little things some people pointed out.