DC1 Data Sets

This page lists the data sets that you may use for Design Challenge 1 (DC1): One dataset / Four Stories.

You must pick one of these data sets for your assignment.

For Design Challenge 1, we are providing 4 different data sets. Variants of these data sets have been used successfully by students in the past. All of these data sets are sufficiently large and rich to contain many interesting stories. We may provide more choices, or better versions of the datasets listed. We will not remove things from this list (so if you pick something, it will continue to be OK).

Box Folder with All Data Sets

There are four choices (some with multiple variants):

1. Census Data
2. Airline On-Time Performance
3. Time Usage (how people spend their time)
4. IMDB movie data

Some of the data sets get quite big. However, Tableau is remarkably efficient at working with 100MB+ data sets - it uses fancy database technologies to perform aggregations quickly.

In each case, you may access the original data from its source. However, we are providing versions where we have done a little of the data wrangling work for you. Thank Young, the TA.

In your assignment, be sure to explain which data set you are using (including which variant). If you got your own version of the data, please say so.

Note: we are working on being able to provide these as part of our Tableau site so you can work with them using Tableau online without having to download/upload them yourself. Stay tuned for an announcement!

Census Data by County

This data set aggregates many different quantities of interest over the counties of the US. In past assignments, we’ve provided even more detailed data sets which required even more aggregation.

The USDA provides this data as 4 separate sheets (on This page). Any one of them could tell an interesting story but together, they provide a very rich and complex data set full of stories.

We have joined the 4 spreadsheets together (joining by the “FIPS Code” column) creating a single file. The four files are also combined (joined by the “FIPS Code” column) into one file, and put in Box. The rows for the states (not counties) are also removed. The data is downloaded and processed on September 13, 2020 from USDA/ERS.

• CSV File on Box (~4MB, 3196 columns and 339 rows)

The variable descriptions can be found on here, and some of them are replicated in the following table:

Airline On-Time Performance

The Bureau of Transportation Statistics lets you download a lot of data, one month at a time from this page. You can download many different attributes of every flight.

We’ve chosen a set of attributes for each flight, which leads to huge files (250MB/month), and downloaded several months (one per season for 3 years).

We’ve downloaded a few months for you and assembled them into easier to use files. We’ve put multiple months together and down-sampled the data (using random sampling) to reduce the data set size. While Tableau seems very able to process even the largest of these files, they do get a little unweildy.

We chose 2019 (not 2020) since it has more “usual” patterns (although, comaparing pre- and post-COVID would be interesting).

The “small” datasets were downsampled to produce file sizes of about 5MB. The January file was downsampled to 2% (that is, each flight has a 2% chance of being included). The 4 month data set and 4 month*3 year data set was downsampled to have a similar total number of flights (e.g., the 4 month set has 2%/4).

• Box 2019 Jan (~5MB, 11740 columns and 110 rows)
• Box 2019 Jan-Apr-Jul-Oct (~5MB, 12465 columns and 110 rows)
• Box 2017-2018-2019 Jan-Apr-Jul-Oct (~4MB, 11376 columns and 110 rows)

The “medium” datasets were downsampled to produce file sizes of about 25MB each. The 4 month data set and 4 month*3 year data set was downsampled to have a similar total number of flights (e.g., the 4 month set has 10%/4).

• Box 2019 Jan (~25MB, 58710 columns and 110 rows)
• Box 2019 Jan-Apr-Jul-Oct (~26MB, 62106 columns and 110 rows)
• Box 2017-2018-2019 Jan-Apr-Jul-Oct (~24MB, 56725 columns and 110 rows)

The “large” data sets include 10% of all the months included. You can get a sense of how big they are from this picture (12 months, 40-60K flights a month)

• Box 2019 Jan-Apr-Jul-Oct (~107MB, 249056 columns and 110 rows)
• Box 2017-2018-2019 Jan-Apr-Jul-Oct (~294MB, 682459 columns and 110 rows)

For this data set, you may choose to use the months we downloaded, or download your own (please specify what data you use). You can choose to use just 1 month (version 1), you can pick multiple months to compare (version 2), or multiple years to compare if you want a real challenge (version 3). Note that you may filter by geography (select a particular state or all states), year, and monthly periods. Files are available as .csv.

You may want information on the airports (for example, to get location coordinates for each airport code). This data is available from Federal Aviation Administration. The Airline codes list is useful - especially for Madison where you see airlines like “F9” and “OO” (these are regional commuter companies that fly under another airline’s name).

You might want to refer to Bureau of Transportation Statistics for explanations of all the fields, and look up tables (files that say what the codes mean). Fields include date, carrier, origin/destination information, departure/arrival performance, gate/airport information, flight summaries, cause of delay, and some of them are replicated in the following table:

Time Usage Survey

2003-2019 all samples, Respondents

The American Time Usage Survey (ATUS) tracks how people spend their time. There are corresponding international versions. There are actually lots of different surveys with interesting data available from the IPUMS website.

We have done a data pull for you. We are allowed to share a data pull (see this). We chose to collect data from all of the years (2003-2019), and selected a wide range of different attributes. The data was downloaded on September 13, 2020 from ATUS.

• Data Set on Box (~24MB, 210587 columns and 34 rows)

You may create your own data pull if you’d like to try this with different columns (but be sure to document it in your writeup). Getting a data set requires picking from all the options - there are so many options that picking a good set is pretty time consuming. It is actually an interesting exercise to see how they document their data - they are very careful in documenting everything.

You can find out what the “time use codes” mean on this page.

Interpreting the other codes requires some digging, unfortunately. Some are self-explanatory, but others… I tracked down the “FAMINCOME” columns: explanation here. The state codes are here. Here are some of them are replicated in the following table:

IMDB Movie Data

This is data on approximately 5000 movies with columns including director, genre, plot keywords, and box office statistics. The data comes from Kaggle, and may be noisy.

This data set is small for the design challenge – but we’re still allowing it. The genre field contains texts separated by vertical bars, and we converted them into separate indicator columns and appended to the end of the table.

• CSV file on Box (~1MB, 5044 columns and 54 rows)

The data was downloaded and processed on September 13, 2020 from Kaggle.

The list of genres are summarized in the following table: