My Survey Reply: An Apology, Clarifications, and Improvements

in Posts

I read the survey responses carefully and would like to respond in three ways: First, I would like to apologize for a framing (both in survey questions and class discussion) that some read in negative ways. Second, I would like to clarify some things that were misinterpreted. Third, I am making some changes to address some of the issues raised.

Claude helped me read through the anonymized responses in a level headed manner. I saw the problems immediately in my first read, and realized I needed some help. It offered to draft a note, but I chose to do it myself. I did discuss the survey results and my responses with it. It helped read drafts and suggest improvements. I’ve iterated/obsessed/ruminated over this note more than I care to admit.

I want to thank everyone who participated in the anonymous surveys. Your answers really are helping improve the class.

First, A Major Apology

I apologize that things that I said and did (in class and in the survey wording) have made some students feel accused or threatened or confused.

Several things (including the AI examples in class and the survey question) were easily interpreted in ways other than I intended.

I can also see how students choosing paths that I want to encourage and support might have felt the most negativity. I am sorry for this.

My intent is to create an environment that is supportive of all different learning pathways. For example, I did not intend to confront students about AI usage, or to make students who avoid AI feel they had to use it to keep up.

My goal for the survey was to understand how students are thinking about the class, and to better understand some behaviors that I was seeing.

Second, some clarifications

I wanted to write a simple clarification where I think I was being misinterpreted. But this turned out to be more complicated than I thought.

Changes (especially how quickly AI has become able to do visualization and visualization programming, but also in its more general effects including to student attitudes) have meant I needed to adapt not only the class, but also my ways of thinking about visualization and education. I am finding it hard to navigate some of the conflicts between goals, in particular to manage flexibility and fairness.

As I struggle to adapt quickly, I have not communicated my intents clearly. The survey gives some specific points I think I should clarify.

  1. My goal is to give students flexibility, while maintaining a level playing field and encouraging responsible behavior. These are in tension.
  2. I am not trying to penalize AI usage, or push students to use AI. We will not use “style” as an AI detection mechanism. Students shouldn’t worry about avoiding certain styles.
  3. I was not clear enough about the intent of the AI generated examples in class. There were many goals. For example: Students should not fear “competing” with AIs because they make different kinds of mistakes. Students should be able to explain the choices in their designs, even if AI made them.

What these all lead to is that I need to create improved mechanisms for designing and assessing assignments that better achieve the course goals and provide fairness no matter how students chose to use AI.

The longer explanation

2.1 My Goal: A Flexible Learning Environment and Level Playing Field

My goal is to give students the flexibility to work the ways that they want. However, I also need to prevent the choices that some students make from hurting others.

The problem is that the flexibility can hurt other students. Students who choose certain tools (such as AI), have certain advantages. This can hurt students who make other choices. I want to have a level playing field where students are rewarded for doing quality work and meeting the learning objectives - no matter how they produce it.

Here is a pre-AI example: Captions are really valuable. Tableau makes captions by default. Making captions by “hand” (for example with some Python libraries) can be a lot of work. Giving points for captions is free points for Tableau users (and maybe an incentive to use it). Also, if we equate “long captions” with “good captions”, Tableau (at least old versions) wrote very long, overly detailed captions that contained a lot of irrelevant details, so if we weren’t careful, Tableau users would be rewarded for having bad captions automatically generated.

Another example is polish. AI gets polish for free (as did Tableau in the past). A student working without AI has to work for it. Polish is a good thing, and I don’t want to stop valuing it. But, it’s hard to reward something that some people get for free. It’s also subtle: a polished submission tends to bias reviewers. Again, I believe the best answer for things like this is through assignment and assessment design that puts the focus on the material.

My preferred answer to this is to design assignments and grading processes that can handle the range of ways that students are learning. Ideally assignments assess how students are mastering the material - whether they choose to work with AI or not. In the caption example, we reward students for effective captions. We could have deducted points for a caption that had extraneous details. Whether or not students used Tableau, they were responsible for making sure they had good captions.

Coming up with these good processes is turning out to be very challenging.

One other point about flexibility, and how I have been unclear in my language:

A problem with flexibility is that it can give students the ability to make choices that are bad for them (e.g., interfere with their learning): There are things that may be legal, but unwise. This is a place that I have slipped up: I conflate what is legal and what is wise. For example, I might say over-using AI tools is “abusing AI” even if it is legal by the rules. Unwise, but legal.

2.2 We are not using style to identify (or penalize) AI usage

It is not my goal to identify AI usage so that I can penalize it. I want to understand AI usage, so that I can support students whether they chose to use AI or not.

I will not use writing (or design) style as a mechanism for AI detection.

Style (writing or visual) alone is not a good indicator of how things were made. I do not want students to fear “sounding like an AI.”

What we care about is effectiveness and evidence of good choices: a design that works, and an explanation that shows why you made it that way. Many styles can do both. A short, plain explanation that names a choice and the reason for it can be effective. A long explanation that hedges and covers every possibility may hide the reasoning, whether a person or an AI wrote it. We encourage you to write (and design) in a style that you think is effective. Plain, direct, even informal writing is fine, providing it communicates to the reader.

The same goes for designs. Agents have their own style (e.g., Claude likes certain fonts on beige backgrounds). If you like this, use it (even if you make it by hand). Common designs are common because they work, and AI tools learned them from designs people made. We look at whether the design works for its purpose and whether you can explain your choices, not at whether it resembles what an AI would produce.

Some AI usage is blatant. For example, an answer may still have instructions from the agent “(insert your name)” or “(replace this with an example from class)”. Part of using AI is checking its answers. In extreme cases, these cues might contradict disclosures, which suggests dishonesty. Honest disclosure helps me understand how students are learning in order to improve the class.

2.3 The AI Examples in Class

I had several lessons (learning goals) in mind with the AI generated Design Exercise 2 examples. However, in hindsight, I can see how the examples can easily be misinterpreted.

I wanted to show how fancy and complex assignments weren’t necessarily better. Uncommon designs may be uncommon for a reason: they may have limited scope, or tradeoffs to consider. This doesn’t mean you shouldn’t try fancy designs: it means that you should consider their tradeoffs and be prepared to defend your choices. This is true if you pick a standard design as well.

I showed examples where an AI came up with complicated designs. But unusual designs can come from anywhere. A complex chart may come from a student with a clever reason for it, or who found it while exploring by hand. Or they may have had an AI make it and then worked to understand why it works, or had an AI make it and turned it in without checking. The chart looks the same in every case, but the student’s contribution and learning are different. Assessment needs to check that the student understands the chart and the decisions behind it, which cannot be seen in the chart itself.

Whether you come up with something novel, find an obscure design, have an AI make something complicated for you, or decide that a standard chart is the most effective choice, you need to “own” the choices.

Similarly, I wanted to show that while AI assignments often get certain basic “polish” elements correct, they sometimes get other things wrong. Again, if you use AI, you need to be careful.

Also, you do not need to compete with agents for polish. Things need to be polished enough to be effective. We will improve the assignment and assessment designs to level the playing field.

This is also a cautionary tale for grading design: we cannot just reward things for being polished, we need to develop better mechanisms to assess whether students understand the choices they have handed in.

Sabotaging agent assignments

I tried to get an agent to make assignments that were more like student assignments by asking it to “get a B.” It decided to do this by making an over-the-top assignment, and then deliberately sabotaging it.

I wanted to use this example to show that agents succeed and fail differently than students. I didn’t want to suggest that you should try to obscure an agent’s work by sabotaging it.

Third, Some Specific Fixes

I very much appreciate the detailed feedback that you have provided. I am trying to use it, as much as possible, to improve the class.

  1. I apologize that I missed the promised Saturday morning releases of assignments: in some cases, they were delayed by as much as a day. I realize that some students want to get an early start on the assignments. I will work harder to get the assignments out earlier, even if I need to release preliminary versions of the assignments.

  2. I understand that the “around class time” deadlines/cutoffs are awkward for students. These were part of the class design, but I have decided the complexity isn’t worth it.

    • I understand the Design Exercise deadline (12:15pm) is awkward for students. We will look into changing the feedback exercise in a way that will allow changing the deadline.
    • The Friday 12:15 deadlines are important for the Design Exercises (DE) so that we can set up the Feedback exercises. However, other assignments just kept this deadline for consistency. All deadlines other than DE deadlines will be end of day (11:59pm).
    • The Monday 10am cutoffs were to force students to complete the Reading and Content surveys before class. In practice, the value of this does not outweigh the complexity it causes. All cutoffs will be changed to end of day (11:59pm).

  3. I am being more careful about assignment design. I am trying to develop assignments that better connect to the objectives and allow for assessment that is more tied to learning, and less to result qualities that can be achieved by the tools.

  4. I am designing more detailed rubrics and grading mechanisms for future assignments. These should provide clarity for students. I seek to design processes that better recognizes good work independently of how the work was produced. If nothing else, more objective criteria should address some of the biases towards certain kinds of responses. Unfortunately, this takes time.

  5. I will provide more assignment-specific AI guidance including examples of recommended usage.

  6. I will clarify the AI policy to match the intents stated here.

  7. The In-Class Activities are generally popular. There are more of them planned in the coming weeks.

GenAI Disclosure:

I used Claude to help me analyze the anonymized responses. I discuss my intents and ideas for improvements with it. I had it review my drafts and provide feedback. I allowed it to fix small errors.

I asked Gemini to read the course policies and describe how it interpreted them and how students might interpret them. I’ve worked with Claude so much that it may know my interpretation of the rules.