{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "**YOUR NAME HERE**\n", "\n", "Spring 2021\n", "\n", "CS 251: Data Analysis and Visualization\n", "\n", "Project 1: Data Analysis and Visualization\n", "\n", "**Week 2: Start this after your 1st lab**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 2) Update `Data` class for data selection\n", "\n", "You will update `Data` to allow the user to select one or more data variables based on their string name. This will give you some practice working with numpy basics.\n", "\n", "Implement the following methods in `Data`:\n", "- `get_all_data()`\n", "- `head()`: Return the 1st five data samples (all variables)\n", "- `tail()`: Return the last five data samples (all variables)\n", "- `limit_samples(self, start_row, end_row)`: Update the data so that the object only stores the samples in the contiguous range `[start_row, end_row)`.\n", "- `select_data(headers, rows=[])`: Return data samples corresponding to the variable names in `headers`. If `rows` is empty, return all samples, otherwise return samples at the indices specified by the `rows` list." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from data import Data\n", "import numpy as np\n", "\n", "# Automatically reload external modules\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2a) Test `get_all_data`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iris_filename = 'data/iris.csv'\n", "iris_data = Data(iris_filename)\n", "dat = iris_data.get_all_data()\n", "dat[0,:] = -9999\n", "new_dat = iris_data.get_all_data()\n", "if new_dat[0, 0] == -9999.:\n", " print('!!You did not return a copy of your data!!\\n')\n", "else:\n", " print('get_all_data seems ok!')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2b) Test `head` and `tail`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (i) Iris" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iris_filename = 'data/iris.csv'\n", "iris_data = Data(iris_filename)\n", "print(f'Your Iris head (shape={iris_data.head().shape}) is:\\n{iris_data.head()}')\n", "print(f'Your Iris tail (shape={iris_data.head().shape}) is:\\n{iris_data.tail()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should see:\n", "\n", " Your Iris head (shape=(5, 4)) is:\n", " [[5.1 3.5 1.4 0.2]\n", " [4.9 3. 1.4 0.2]\n", " [4.7 3.2 1.3 0.2]\n", " [4.6 3.1 1.5 0.2]\n", " [5. 3.6 1.4 0.2]]\n", " Your Iris tail (shape=(5, 4)) is:\n", " [[6.7 3. 5.2 2.3]\n", " [6.3 2.5 5. 1.9]\n", " [6.5 3. 5.2 2. ]\n", " [6.2 3.4 5.4 2.3]\n", " [5.9 3. 5.1 1.8]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (ii) Data with small number of samples" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_filename = 'data/test_data_spaces.csv'\n", "test_data = Data(test_filename)\n", "print(f'Your test head (shape={test_data.head().shape}) is:\\n{test_data.head()}')\n", "print(f'Your test tail (shape={test_data.head().shape}) is:\\n{test_data.tail()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should see:\n", "\n", " Your test head (shape=(3, 4)) is:\n", " [[ 1. 2. 3. 4.]\n", " [ 5. 6. 7. 8.]\n", " [ 9. 10. 11. 12.]]\n", " Your test tail (shape=(3, 4)) is:\n", " [[ 1. 2. 3. 4.]\n", " [ 5. 6. 7. 8.]\n", " [ 9. 10. 11. 12.]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2c) Test `limit_samples`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iris_filename = 'data/iris.csv'\n", "iris_data = Data(iris_filename)\n", "iris_data.limit_samples(50, 100)\n", "print(f'After limiting samples, your test head is:\\n{iris_data.head()}')\n", "print(f'After limiting samples, your test tail is:\\n{iris_data.tail()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should see:\n", "\n", " After limiting samples, your test head is:\n", " [[7. 3.2 4.7 1.4]\n", " [6.4 3.2 4.5 1.5]\n", " [6.9 3.1 4.9 1.5]\n", " [5.5 2.3 4. 1.3]\n", " [6.5 2.8 4.6 1.5]]\n", " After limiting samples, your test tail is:\n", " [[5.7 3. 4.2 1.2]\n", " [5.7 2.9 4.2 1.3]\n", " [6.2 2.9 4.3 1.3]\n", " [5.1 2.5 3. 1.1]\n", " [5.7 2.8 4.1 1.3]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2d) Test `select_data`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (i) Test data with spaces" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_filename = 'data/test_data_spaces.csv'\n", "test_data = Data(test_filename)\n", "\n", "one = test_data.select_data(['spaces'])\n", "print(f'All data in the \"spaces\" variable (shape={one.shape}): \\n{one}')\n", "\n", "two = test_data.select_data(['spaces', 'places'])\n", "print(f'All data in the \"spaces\" and \"places\" variables (shape={two.shape}): \\n{two}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should see:\n", "\n", " All data in the \"spaces\" variable (shape=(3, 1)): \n", " [[ 2.]\n", " [ 6.]\n", " [10.]]\n", " All data in the \"spaces\" and \"places\" variables (shape=(3, 2)): \n", " [[ 2. 4.]\n", " [ 6. 8.]\n", " [10. 12.]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (ii) Test data with spaces, some samples" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_filename = 'data/test_data_spaces.csv'\n", "test_data = Data(test_filename)\n", "\n", "rows = [1, 2]\n", "\n", "one = test_data.select_data(['spaces'], rows=rows)\n", "print(f'All data in the \"spaces\" variable (shape={one.shape}): \\n{one}')\n", "\n", "two = test_data.select_data(['spaces', 'places'], rows=rows)\n", "print(f'All data in the \"spaces\" and \"places\" variables (shape={two.shape}): \\n{two}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should see:\n", "\n", " All data in the \"spaces\" variable (shape=(2, 1)): \n", " [[ 6.]\n", " [10.]]\n", " All data in the \"spaces\" and \"places\" variables (shape=(2, 2)): \n", " [[ 6. 8.]\n", " [10. 12.]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 3) `Analysis` class\n", "\n", "In this task, you will implement the `Analysis` class that analyzes and visualizes complex datasets comprised of many variables. After creating `Analysis`, you’ll use it alongside your `Data` class to analyze an open-access scientific dataset. *Can you use the tools you’ve developed to extract real-world truths from data?*\n", "\n", "You will build on this object-oriented approach to do advanced data analysis in future projects, making it easier for you to apply the tools you develop this week to a wide range of datasets.\n", "\n", "It is crucial for your success in CS251 to become fluent in the **vectorization** of matrix operations (**NO LOOPS**) with Numpy and the visualization of data with Matplotlib. We will use these Python packages each and every week. Please bring your questions to class and office hours so that we can help you get comfortable with these essential tools before the math gets more complex and interesting.\n", "\n", "**We will take off -0.5 points for every loop that we see — vectorized computations only in analysis functions!**\n", "\n", "**TODO:**\n", "- Download the `analysis.py` code template." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Analysis and summary statistic methods\n", "\n", "To actually perform computations on data, you will implement the following methods in `analysis.py`, modeled off of common functions in the `pandas` python package (https://pandas.pydata.org/pandas-docs/stable/user_guide/computation.html) that we will adopt later in the semester:\n", "\n", "- `set_data(data)`\n", "- `min(headers, rows=[])`: Computes the minimum of each variable in `headers` in the data object. Possibly only in a subset of data samples (`rows`) if `rows` is not empty.\n", "- `max(headers, rows=[])`: Computes the maximum of each variable in `headers` in the data object.\n", "- `range(headers, rows=[])`: Computes the range [min, max] for each variable in `headers` in the data object.\n", "- `mean(headers, rows=[])`: Computes the mean for each variable in `headers` in the data object. Equation for mean: $$\\bar{x} = \\frac{1}{N} \\sum_{i=1}^N x_i$$ where $x_i$ is the $i^{th}$ data sample and $N$ is the total number of data samples in the dataset.\n", "- `var(headers, rows=[])`: Computes the variance for each variable in `headers` in the data object. Equation for sample variance: $$s^2 = \\frac{1}{N-1} \\sum_{i=1}^N (x_i - \\bar{x})^2$$ where $x_i$ is the $i^{th}$ data sample, $N$ is the total number of data samples in the dataset, $\\bar{x}$ is the mean.\n", "- `std(headers, rows=[])`: Computes the standard deviation for each variable in `headers` in the data object. Equation for sample standard deviation: $$s^2 = \\sqrt{\\frac{1}{N-1} \\sum_{i=1}^N (x_i - \\bar{x})^2}$$ where $x_i$ is the $i^{th}$ data sample, $N$ is the total number of data samples in the dataset, $\\bar{x}$ is the mean." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from analysis import Analysis\n", "\n", "iris_filename = 'data/iris.csv'\n", "iris_data = Data(iris_filename)\n", "an = Analysis(iris_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (i) Test `min`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"Your minimum values for length vars (all samples) are\\n{an.min(['sepal_length', 'petal_length'])}\\nand should be\\n[4.3 1. ]\")\n", "print(f\"Your minimum values for length vars (1st 10 samples) are\\n{an.min(['sepal_length', 'petal_length'], rows=np.arange(10))}\\nand should be\\n[4.4 1.3]\")\n", "print(f\"Your shape is {an.min(['sepal_length', 'petal_length']).shape}\\nand should be (2,)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (ii) Test `range`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "all_mins, all_maxs = an.range(['sepal_length', 'sepal_width'])\n", "some_mins, some_maxs = an.range(['sepal_length', 'sepal_width'], rows=np.arange(10))\n", "print(f\"Your range for sepal vars (all samples) is\\nmins:{all_mins}\\nmaxs:{all_maxs}\\nand should be\\nmins:[4.3 2. ]\\nmaxs:[7.9 4.4]\\n\")\n", "print(f\"Your range for sepal vars (1st 10 samples) is\\nmins:{some_mins}\\nmaxs:{some_maxs}\\nand should be\\nmins:[4.4 2.9]\\nmaxs:[5.4 3.9]\\n\")\n", "print(f\"Your min shape is {all_mins.shape}\\nand should be (2,)\")\n", "print(f\"Your max shape is {all_maxs.shape}\\nand should be (2,)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (iii) Test `mean`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"Your mean values for length vars (all samples) are\\n{an.mean(['sepal_length', 'petal_length'])}\\nand should be\\n[5.84333333 3.758 ]\")\n", "print(f\"Your mean values for length vars (1st 10 samples) are\\n{an.mean(['sepal_length', 'petal_length'], rows=np.arange(10))}\\nand should be\\n[4.86 1.45]\")\n", "print(f\"Your shape is {an.mean(['sepal_length', 'petal_length']).shape}\\nand should be (2,)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (iv) Test `var`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"Your variance values for all vars (all samples) are\\n{an.var(an.data.get_headers())}\\nand should be\\n[0.68569351 0.18997942 3.11627785 0.58100626]\")\n", "print(f\"Your variance values for all vars (1st 10 samples) are\\n{an.var(an.data.get_headers(), rows=np.arange(10))}\\nand should be\\n[0.08488889 0.09433333 0.01166667 0.00622222]\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (v) Test `std`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"Your stdev values for all vars (all samples) are\\n{an.std(an.data.get_headers())}\\nand should be\\n[0.82806613 0.43586628 1.76529823 0.76223767]\")\n", "print(f\"Your stdev values for all vars (1st 10 samples) are\\n{an.std(an.data.get_headers(), rows=np.arange(10))}\\nand should be\\n[0.29135698 0.30713732 0.10801234 0.07888106]\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 4) Data visualization with matplotlib\n", "\n", "In this task, you will create some plots to practice using matplotlib. You will write several functions to make some kinds of plots that will come up frequently in the coming weeks." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4a) Join data with a smooth curve\n", "\n", "In the cell below, create a single plot of the functions $x$, $x^2$, and $x^3$ in different colors in the range [-5, 5]. Points should be connected with a smooth curve in each case.\n", "\n", "Your plots should be \"high quality\", which means that they should contain:\n", "- a helpful title\n", "- clearly differentiated curve colors. **Do not use the default colors, pick your own colors.**\n", "- a legend showing which color scales map onto which curves\n", "- only the x tick marks -5, 0, +5\n", "- only the y tick marks -100, 0, +100\n", "- there should be no garbage text output below, just the plot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4b) Anscombe's Quartet: Create a 2x2 grid of plots \n", "\n", "In the cell below, create a SINGLE figure with four *scatter plots* in it (2 rows, 2 columns). This should leverage matplotlib's `subplots` function and create ONE cell output (not 4).\n", "\n", "1. You are plotting the data in `anscombe.csv`: The first set of points should be plotted on the top-left (row indices 0-10), the second set should be on the top-right (row indices 11-21), the third set should be on the bottom-left (row indices 22-32), and the last set on the bottom-right (row indices 33-43).\n", "2. Make the markers in each subplot a different shape for fun.\n", "3. You should not have connected curves in these plots.\n", "4. The x/y plot limits should the same in all plots (and include all data samples)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4c) `scatter` method in `Analysis`\n", "\n", "Scatter plots will come up a lot in this course. Fill in the `scatter` method in the `Analysis` class to automate the process of selecting (x, y) data from a dataset, creating a scatter plot, and labeling it with a title." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iris_filename = 'data/iris.csv'\n", "iris_data = Data(iris_filename)\n", "iris_an = Analysis(iris_data)\n", "\n", "x_samps, y_samps = iris_an.scatter('sepal_length', 'sepal_width', 'Iris Sepal width vs. length')\n", "iris_an.show()\n", "\n", "print(f'Your x sample shape is {x_samps.shape} and should be (150,)')\n", "print(f'Your y sample shape is {y_samps.shape} and should be (150,)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4d) `pair_plot` method in `Analysis`\n", "\n", "When doing exploratory data analysis, it is often helpful to create scatter plots of all pairs of variables to visually discover relationships and develop hypotheses to explore quantitatively. For example, if we have variables A, B, C, we would create scatter plots (x=A, y=B), (x=A, y=C), (x=B, y=C). A grid of subplots showing all these scatter plots is called a **pair plot**. If there are $M$ variables, then the grid of plots is $M\\times M$.\n", "\n", "- In `Analysis`, implement `pair_plot` and test your implementation below on the Iris data.\n", " - The y axis of the first column should be labeled with the appropriate variable being plotted there.\n", " - The x axis of the last row should be labeled with the appropriate variable being plotted there.\n", " - There should be no other axis or tick labels (looks too cluttered!)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iris_filename = 'data/iris.csv'\n", "iris_data = Data(iris_filename)\n", "iris_an = Analysis(iris_data)\n", "\n", "fig, axes = iris_an.pair_plot(iris_data.get_headers())\n", "iris_an.show()\n", "\n", "print(f'Do you have a 4x4 grid of plots? {axes.shape[0] == 4 and axes.shape[1] == 4}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 5) Real-world application\n", "\n", "Analyze one real-world dataset of your choice: [e.g. wine quality dataset, Old Faithful geyser dataset, Auto MPG dataset, tips dataset]. Use your **existing** visualization and analysis tools to try to learn something about the real world from this dataset (*you can implement other analyses and explore other visualizations as extensions; see below*). \n", "\n", "- Use scatter plot, pair plot, or other visualizations determine which helps you learn the most about your dataset.\n", " - Create at least one \"good\" graph that presents the data in a helpful way and at least one “bad” graph that is not helpful for gleaning insight into your dataset.\n", "- Briefly (but completely) explain your findings in the Markdown cell below alongside well-labeled graphs that support your conclusions. Explain each variable (its units, its range, mean, stdev, what it means about the world)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code to import your dataset and create visualizations here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Explain your findings here:**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extensions\n", "\n", "To receive credit for any extension, you must:\n", "- Not modify / prevent any code from the core project from working (e.g. make a copy before changing). In other words, **the notebook test code should still work!**\n", "- **You must describe what you did and what you found in detail**. This includes a summary of parameter values used in your simulations.\n", "- Include (*labeled!*) plots and/or numbers to present your results.\n", "- Write up your extensions below or in a separate notebook.\n", "\n", "**Rule of thumb: one deep, thorough extension is worth more than several quick, shallow extensions!**\n", "\n", "**Reminder:** Give credit to all sources, including anyone that you consulted." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1) More visualizations\n", "\n", "Research and create more visualizations of data. If we haven't talked about it in class, all the better! This could be in the matplotlib API or something you build yourself (probably a better extension)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2) More summary statistics\n", "\n", "Research, implement, and apply additional ways to quantitatively summarize a dataset. Compare these other techniques with those that you implemented in this project." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3) More datasets\n", "\n", "Analyze and visualize additional datasets. Document and report on insights/hypothesized relationships that you may have discovered." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4) Anscombe's Quartet\n", "\n", "Despite the visual dissimilarity of the data samples in each group of points, the groups of data have many identical (or nearly identical) summary statistics! Use you analysis code to verify this. Explain what it means / why it happens. Research additional statistics that are identical in the quartet, then implement and verify their similarity. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5) Additional data types\n", "\n", "You wrote `Data` to only store numeric data types. Extend your `Data` class to corectly parse and represent dates, strings, enums, and other types of data.\n", "\n", "NOTE: You will definitely want to make a copy of your `Data` class before attempting this!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 2 }