{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d94102cc",
   "metadata": {},
   "source": [
    "# Week 3: Data Manipulation with Pandas\n",
    "\n",
    "*Kmex Consult — Data Science Fundamentals Course*\n",
    "\n",
    "Work through each code cell in order. Edit and re-run to experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07bb24fc",
   "metadata": {},
   "source": [
    "*Data Science Fundamentals Course*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a61ac3e",
   "metadata": {},
   "source": [
    "## Week 3 Overview\n",
    "Week 3 marks your transition from basic Python to data science libraries. NumPy is the foundation for numerical computing in Python, and pandas builds on NumPy to provide powerful data manipulation capabilities. These two libraries are essential for any data science work.\n",
    "\n",
    "This week you will learn:\n",
    "- NumPy arrays: the core data structure for scientific computing\n",
    "- How to perform efficient numerical operations\n",
    "- Broadcasting: a powerful feature for working with arrays of different shapes\n",
    "- Introduction to pandas: for handling and analyzing tabular data\n",
    "- Creating and manipulating DataFrames\n",
    "- Loading real data from files\n",
    "\n",
    "By the end of Week 3, you will be able to:\n",
    "- Create and manipulate NumPy arrays efficiently\n",
    "- Use array indexing and slicing for data selection\n",
    "- Perform mathematical and statistical operations on arrays\n",
    "- Understand NumPy broadcasting\n",
    "- Create and manipulate pandas Series and DataFrames\n",
    "- Load data from CSV, Excel, and other formats\n",
    "- Perform basic data exploration\n",
    "\n",
    "Week 3 is divided into three 2-hour sessions:\n",
    "- Session 1: NumPy Basics and Array Operations\n",
    "- Session 2: Advanced NumPy: Broadcasting and Functions\n",
    "- Session 3: Introduction to pandas and DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "444fb01e",
   "metadata": {},
   "source": [
    "## SESSION 1: NumPy Basics and Array Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5f516dc",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a54c3173",
   "metadata": {},
   "source": [
    "### 1.1 What is NumPy?\n",
    "NumPy (Numerical Python) is the fundamental package for numerical computing in Python. It provides:\n",
    "- Efficient array data structures\n",
    "- Mathematical and statistical functions\n",
    "- Linear algebra operations\n",
    "- Random number generation\n",
    "- Tools for integrating C/C++ and Fortran code\n",
    "\n",
    "Why NumPy instead of lists?\n",
    "- Speed: NumPy operations are much faster than Python loops\n",
    "- Convenience: Intuitive mathematical operations on arrays\n",
    "- Memory efficiency: NumPy arrays use less memory than lists\n",
    "- Functionality: Extensive mathematical and statistical functions\n",
    "- Broadcasting: Elegant way to handle arrays of different shapes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c0bdc0c",
   "metadata": {},
   "source": [
    "#### Installing NumPy\n",
    "If you installed Anaconda, NumPy is already included. Otherwise, install it:\n",
    "pip install numpy\n",
    "Verify installation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "983bda34",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "print(np.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d95d837",
   "metadata": {},
   "source": [
    "### 1.2 Creating NumPy Arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50c39c31",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Create array from Python list\n",
    "arr1 = np.array([1, 2, 3, 4, 5])\n",
    "print(arr1) # Output: [1 2 3 4 5]\n",
    "print(type(arr1)) # Output: <class 'numpy.ndarray'>\n",
    "\n",
    "# Create 2D array (matrix)\n",
    "arr2d = np.array([[1, 2, 3],\n",
    "[4, 5, 6],\n",
    "[7, 8, 9]])\n",
    "print(arr2d)\n",
    "# Output:\n",
    "# [[1 2 3]\n",
    "# [4 5 6]\n",
    "# [7 8 9]]\n",
    "\n",
    "# Create arrays with specific values\n",
    "zeros = np.zeros(5) # Array of 5 zeros\n",
    "print(zeros) # Output: [0. 0. 0. 0. 0.]\n",
    "\n",
    "ones = np.ones((3, 4)) # 3x4 array of ones\n",
    "print(ones)\n",
    "# Output:\n",
    "# [[1. 1. 1. 1.]\n",
    "# [1. 1. 1. 1.]\n",
    "# [1. 1. 1. 1.]]\n",
    "\n",
    "# Array filled with specific value\n",
    "full = np.full((2, 3), 7) # 2x3 array filled with 7\n",
    "print(full)\n",
    "# Output:\n",
    "# [[7 7 7]\n",
    "# [7 7 7]]\n",
    "\n",
    "# Identity matrix (1s on diagonal, 0s elsewhere)\n",
    "identity = np.eye(3)\n",
    "print(identity)\n",
    "# Output:\n",
    "# [[1. 0. 0.]\n",
    "# [0. 1. 0.]\n",
    "# [0. 0. 1.]]\n",
    "\n",
    "# Create range of values\n",
    "range_arr = np.arange(0, 10, 2) # start, stop, step\n",
    "print(range_arr) # Output: [0 2 4 6 8]\n",
    "\n",
    "# Evenly spaced values\n",
    "linspace = np.linspace(0, 10, 5) # 5 values from 0 to 10\n",
    "print(linspace) # Output: [ 0. 2.5 5. 7.5 10. ]\n",
    "\n",
    "# Random values\n",
    "random_arr = np.random.rand(3, 3) # 3x3 random array [0, 1)\n",
    "print(random_arr)\n",
    "\n",
    "# Random integers\n",
    "random_int = np.random.randint(1, 10, size=(2, 3)) # Random ints 1-9\n",
    "print(random_int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cf59b8e",
   "metadata": {},
   "source": [
    "#### Array Properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a07a557",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.array([[1, 2, 3],\n",
    "[4, 5, 6]])\n",
    "\n",
    "# Shape: dimensions of array\n",
    "print(arr.shape) # Output: (2, 3) - 2 rows, 3 columns\n",
    "\n",
    "# Size: total number of elements\n",
    "print(arr.size) # Output: 6\n",
    "\n",
    "# Dtype: data type of elements\n",
    "print(arr.dtype) # Output: int64\n",
    "\n",
    "# Ndim: number of dimensions\n",
    "print(arr.ndim) # Output: 2\n",
    "\n",
    "# Real-world: Check data dimensions\n",
    "image = np.random.rand(1920, 1080, 3) # RGB image\n",
    "print(f\"Image shape: {image.shape}\") # Output: Image shape: (1920, 1080, 3)\n",
    "print(f\"Total pixels: {image.size}\") # Output: Total pixels: 6220800"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e966acf9",
   "metadata": {},
   "source": [
    "### 1.3 Indexing and Slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eca145e",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.array([10, 20, 30, 40, 50])\n",
    "\n",
    "# Access single element\n",
    "print(arr[0]) # Output: 10\n",
    "print(arr[2]) # Output: 30\n",
    "print(arr[-1]) # Output: 50\n",
    "\n",
    "# Slicing\n",
    "print(arr[1:4]) # Output: [20 30 40]\n",
    "print(arr[:3]) # Output: [10 20 30]\n",
    "print(arr[2:]) # Output: [30 40 50]\n",
    "print(arr[::2]) # Every 2nd element: [10 30 50]\n",
    "\n",
    "# 2D array indexing\n",
    "arr2d = np.array([[1, 2, 3],\n",
    "[4, 5, 6],\n",
    "[7, 8, 9]])\n",
    "\n",
    "# Access single element\n",
    "print(arr2d[0, 0]) # Output: 1\n",
    "print(arr2d[1, 2]) # Output: 6\n",
    "print(arr2d[-1, -1]) # Output: 9\n",
    "\n",
    "# Access entire row\n",
    "print(arr2d[0]) # Output: [1 2 3]\n",
    "print(arr2d[1]) # Output: [4 5 6]\n",
    "\n",
    "# Access entire column\n",
    "print(arr2d[:, 0]) # Output: [1 4 7]\n",
    "print(arr2d[:, 2]) # Output: [3 6 9]\n",
    "\n",
    "# 2D slicing\n",
    "print(arr2d[0:2, 1:3]) # First 2 rows, columns 1-2\n",
    "# Output:\n",
    "# [[2 3]\n",
    "# [5 6]]\n",
    "\n",
    "# Modify elements\n",
    "arr2d[0, 0] = 100\n",
    "print(arr2d)\n",
    "# Output:\n",
    "# [[100 2 3]\n",
    "# [ 4 5 6]\n",
    "# [ 7 8 9]]\n",
    "\n",
    "# Modifying slices affects original array!\n",
    "arr = np.array([1, 2, 3, 4, 5])\n",
    "arr[1:3] = [20, 30]\n",
    "print(arr) # Output: [ 1 20 30 4 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ff0d03e",
   "metadata": {},
   "source": [
    "#### Boolean Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a11affac",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n",
    "\n",
    "# Create boolean mask\n",
    "mask = arr > 5\n",
    "print(mask) # Output: [False False False False False True True True True True]\n",
    "\n",
    "# Use mask to filter\n",
    "filtered = arr[mask]\n",
    "print(filtered) # Output: [ 6 7 8 9 10]\n",
    "\n",
    "# Combine conditions\n",
    "mask2 = (arr > 3) & (arr < 8)\n",
    "print(arr[mask2]) # Output: [4 5 6 7]\n",
    "\n",
    "# Using conditions directly\n",
    "print(arr[arr > 5]) # Output: [ 6 7 8 9 10]\n",
    "print(arr[arr % 2 == 0]) # Even numbers: [ 2 4 6 8 10]\n",
    "\n",
    "# Real-world: Filter data\n",
    "scores = np.array([45, 52, 78, 95, 88, 62, 91, 55])\n",
    "passing = scores[scores >= 60]\n",
    "print(f\"Passing scores: {passing}\")\n",
    "# Output: Passing scores: [78 95 88 62 91]\n",
    "\n",
    "# Count elements meeting condition\n",
    "count = np.sum(scores >= 70)\n",
    "print(f\"Students with score >= 70: {count}\") # Output: Students with score >= 70: 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "151d92e6",
   "metadata": {},
   "source": [
    "## SESSION 2: Advanced NumPy - Broadcasting and Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0d51bd2",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bdf9bf8",
   "metadata": {},
   "source": [
    "### 2.1 Arithmetic Operations and Broadcasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20bb03bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Element-wise operations\n",
    "a = np.array([1, 2, 3, 4, 5])\n",
    "b = np.array([10, 20, 30, 40, 50])\n",
    "\n",
    "print(a + b) # Output: [11 22 33 44 55]\n",
    "print(a - b) # Output: [-9 -18 -27 -36 -45]\n",
    "print(a * b) # Output: [10 40 90 160 250]\n",
    "print(b / a) # Output: [10. 10. 10. 10. 10.]\n",
    "print(a ** 2) # Output: [ 1 4 9 16 25]\n",
    "\n",
    "# Operations with scalars\n",
    "arr = np.array([1, 2, 3, 4, 5])\n",
    "print(arr + 10) # Output: [11 12 13 14 15]\n",
    "print(arr * 2) # Output: [ 2 4 6 8 10]\n",
    "print(arr / 2) # Output: [0.5 1. 1.5 2. 2.5]\n",
    "\n",
    "# 2D operations\n",
    "arr2d = np.array([[1, 2, 3],\n",
    "[4, 5, 6]])\n",
    "\n",
    "print(arr2d * 2)\n",
    "# Output:\n",
    "# [[ 2 4 6]\n",
    "# [ 8 10 12]]\n",
    "\n",
    "# Comparison operations\n",
    "scores = np.array([45, 78, 92, 55, 88])\n",
    "print(scores > 70) # Output: [False True True False True]\n",
    "print(scores == 88) # Output: [False False False False True]\n",
    "print(scores != 70) # Output: [ True True True True True]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f0d5308",
   "metadata": {},
   "source": [
    "#### Broadcasting\n",
    "Broadcasting is NumPy's mechanism for working with arrays of different shapes. It allows you to perform operations on arrays without explicitly replicating data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80c4281f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Broadcasting with scalar\n",
    "arr = np.array([1, 2, 3, 4])\n",
    "scalar = 5\n",
    "print(arr + scalar) # Output: [6 7 8 9]\n",
    "\n",
    "# Broadcasting 1D with 2D\n",
    "arr2d = np.array([[1, 2, 3],\n",
    "[4, 5, 6]])\n",
    "arr1d = np.array([10, 20, 30])\n",
    "\n",
    "result = arr2d + arr1d\n",
    "print(result)\n",
    "# Output:\n",
    "# [[11 22 33]\n",
    "# [14 25 36]]\n",
    "\n",
    "# Broadcasting visualization:\n",
    "# arr2d: arr1d: result:\n",
    "# [[1 2 3] [10 20 30] [[11 22 33]\n",
    "# [4 5 6]] + (broadcast) [14 25 36]]\n",
    "\n",
    "# Column broadcasting\n",
    "col = np.array([[10], [20]]) # Shape (2, 1)\n",
    "arr2d = np.array([[1, 2, 3],\n",
    "[4, 5, 6]]) # Shape (2, 3)\n",
    "\n",
    "result = arr2d + col\n",
    "print(result)\n",
    "# Output:\n",
    "# [[11 12 13]\n",
    "# [24 25 26]]\n",
    "\n",
    "# Real-world: Normalize data (subtract mean from each column)\n",
    "data = np.array([[1, 2, 3],\n",
    "[4, 5, 6],\n",
    "[7, 8, 9]], dtype=float)\n",
    "\n",
    "column_means = np.array([4., 5., 6.]) # Mean of each column\n",
    "normalized = data - column_means\n",
    "print(normalized)\n",
    "# Output:\n",
    "# [[-3. -3. -3.]\n",
    "# [ 0. 0. 0.]\n",
    "# [ 3. 3. 3.]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36838927",
   "metadata": {},
   "source": [
    "### 2.2 Mathematical and Statistical Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f72334d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n",
    "\n",
    "# Basic statistics\n",
    "print(np.sum(arr)) # Output: 55\n",
    "print(np.mean(arr)) # Output: 5.5\n",
    "print(np.median(arr)) # Output: 5.5\n",
    "print(np.std(arr)) # Standard deviation\n",
    "print(np.var(arr)) # Variance\n",
    "\n",
    "# Min and Max\n",
    "print(np.min(arr)) # Output: 1\n",
    "print(np.max(arr)) # Output: 10\n",
    "print(np.argmin(arr)) # Index of minimum: 0\n",
    "print(np.argmax(arr)) # Index of maximum: 9\n",
    "\n",
    "# Mathematical functions\n",
    "print(np.sqrt(arr)) # Square root\n",
    "print(np.exp(arr)) # e^x\n",
    "print(np.log(arr)) # Natural logarithm\n",
    "print(np.abs(np.array([-1, -2, 3, -4]))) # Absolute value\n",
    "\n",
    "# Rounding\n",
    "decimals = np.array([1.234, 2.567, 3.891])\n",
    "print(np.round(decimals, 1)) # Output: [1.2 2.6 3.9]\n",
    "print(np.floor(decimals)) # Output: [1. 2. 3.]\n",
    "print(np.ceil(decimals)) # Output: [2. 3. 4.]\n",
    "\n",
    "# Operations along axis\n",
    "arr2d = np.array([[1, 2, 3],\n",
    "[4, 5, 6],\n",
    "[7, 8, 9]])\n",
    "\n",
    "print(np.sum(arr2d)) # Total sum: 45\n",
    "print(np.sum(arr2d, axis=0)) # Sum of each column: [12 15 18]\n",
    "print(np.sum(arr2d, axis=1)) # Sum of each row: [ 6 15 24]\n",
    "\n",
    "print(np.mean(arr2d, axis=0)) # Mean of each column\n",
    "# Output: [4. 5. 6.]\n",
    "\n",
    "# Real-world: Calculate statistics on test scores\n",
    "scores = np.array([45, 78, 92, 55, 88, 62, 91, 75])\n",
    "print(f\"Mean score: {np.mean(scores):.2f}\")\n",
    "print(f\"Median score: {np.median(scores):.2f}\")\n",
    "print(f\"Std dev: {np.std(scores):.2f}\")\n",
    "print(f\"Best score: {np.max(scores)}\")\n",
    "print(f\"Worst score: {np.min(scores)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83fab677",
   "metadata": {},
   "source": [
    "#### Sorting and Unique Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bff26a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.array([3, 1, 4, 1, 5, 9, 2, 6, 5])\n",
    "\n",
    "# Sorting\n",
    "sorted_arr = np.sort(arr)\n",
    "print(sorted_arr) # Output: [1 1 2 3 4 5 5 6 9]\n",
    "\n",
    "# Get indices that would sort the array\n",
    "indices = np.argsort(arr)\n",
    "print(indices) # Output: [1 3 6 0 2 4 8 7 5]\n",
    "\n",
    "# Reverse sort\n",
    "reverse_sorted = np.sort(arr)[::-1]\n",
    "print(reverse_sorted) # Output: [9 6 5 5 4 3 2 1 1]\n",
    "\n",
    "# Unique values\n",
    "unique = np.unique(arr)\n",
    "print(unique) # Output: [1 2 3 4 5 6 9]\n",
    "\n",
    "# Count occurrences\n",
    "unique_vals, counts = np.unique(arr, return_counts=True)\n",
    "for val, count in zip(unique_vals, counts):\n",
    "print(f\"Value {val} appears {count} times\")\n",
    "\n",
    "# Real-world: Find most common score\n",
    "scores = np.array([78, 92, 78, 55, 88, 78, 91, 75])\n",
    "unique_scores, counts = np.unique(scores, return_counts=True)\n",
    "most_common_idx = np.argmax(counts)\n",
    "most_common_score = unique_scores[most_common_idx]\n",
    "print(f\"Most common score: {most_common_score} (appears {counts[most_common_idx]} times)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ea2003a",
   "metadata": {},
   "source": [
    "#### Reshaping and Combining Arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a76a114",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.array([1, 2, 3, 4, 5, 6])\n",
    "\n",
    "# Reshape to 2D\n",
    "arr_2d = arr.reshape(2, 3)\n",
    "print(arr_2d)\n",
    "# Output:\n",
    "# [[1 2 3]\n",
    "# [4 5 6]]\n",
    "\n",
    "# Reshape to 3D\n",
    "arr_3d = arr.reshape(2, 3, 1)\n",
    "print(arr_3d)\n",
    "\n",
    "# Flatten: convert to 1D\n",
    "flattened = arr_2d.flatten()\n",
    "print(flattened) # Output: [1 2 3 4 5 6]\n",
    "\n",
    "# Transpose (swap rows and columns)\n",
    "transposed = arr_2d.T\n",
    "print(transposed)\n",
    "# Output:\n",
    "# [[1 4]\n",
    "# [2 5]\n",
    "# [3 6]]\n",
    "\n",
    "# Concatenate arrays\n",
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "combined = np.concatenate([a, b])\n",
    "print(combined) # Output: [1 2 3 4 5 6]\n",
    "\n",
    "# Stack arrays vertically\n",
    "arr1 = np.array([1, 2, 3])\n",
    "arr2 = np.array([4, 5, 6])\n",
    "stacked_v = np.vstack([arr1, arr2])\n",
    "print(stacked_v)\n",
    "# Output:\n",
    "# [[1 2 3]\n",
    "# [4 5 6]]\n",
    "\n",
    "# Stack arrays horizontally\n",
    "stacked_h = np.hstack([arr1, arr2])\n",
    "print(stacked_h) # Output: [1 2 3 4 5 6]\n",
    "\n",
    "# Split array\n",
    "arr = np.array([1, 2, 3, 4, 5, 6])\n",
    "split_result = np.split(arr, 3) # Split into 3 equal parts\n",
    "print(split_result) # Output: [array([1, 2]), array([3, 4]), array([5, 6])]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c10e1b30",
   "metadata": {},
   "source": [
    "## SESSION 3: Introduction to pandas and DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "965b3449",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb90b2f",
   "metadata": {},
   "source": [
    "### 3.1 What is pandas?\n",
    "pandas is built on top of NumPy and provides high-level data structures and manipulation tools. The two main data structures are:\n",
    "- Series: 1-dimensional array with labels (like a dictionary)\n",
    "- DataFrame: 2-dimensional table with labeled rows and columns (like a spreadsheet)\n",
    "\n",
    "Why pandas?\n",
    "- Handles missing data easily\n",
    "- Labeled axes (rows and columns have names, not just numbers)\n",
    "- Flexible data alignment and reshaping\n",
    "- Powerful grouping and aggregation\n",
    "- Easy data import from various formats\n",
    "- Integration with NumPy and other libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17447802",
   "metadata": {},
   "source": [
    "#### Installing pandas\n",
    "pandas comes with Anaconda. Or install:\n",
    "pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a981a2d3",
   "metadata": {},
   "source": [
    "### 3.2 pandas Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b34ee870",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create Series from list\n",
    "s1 = pd.Series([10, 20, 30, 40, 50])\n",
    "print(s1)\n",
    "# Output:\n",
    "# 0 10\n",
    "# 1 20\n",
    "# 2 30\n",
    "# 3 40\n",
    "# 4 50\n",
    "# dtype: int64\n",
    "\n",
    "# Series with custom index\n",
    "s2 = pd.Series([10, 20, 30, 40, 50],\n",
    "index=['a', 'b', 'c', 'd', 'e'])\n",
    "print(s2)\n",
    "# Output:\n",
    "# a 10\n",
    "# b 20\n",
    "# c 30\n",
    "# d 40\n",
    "# e 50\n",
    "# dtype: int64\n",
    "\n",
    "# Access elements by index label\n",
    "print(s2['a']) # Output: 10\n",
    "print(s2['c']) # Output: 30\n",
    "\n",
    "# Create Series from dictionary\n",
    "student_grades = {'Alice': 85, 'Bob': 92, 'Charlie': 78, 'Diana': 95}\n",
    "s3 = pd.Series(student_grades)\n",
    "print(s3)\n",
    "# Output:\n",
    "# Alice 85\n",
    "# Bob 92\n",
    "# Charlie 78\n",
    "# Diana 95\n",
    "# dtype: int64\n",
    "\n",
    "# Series properties\n",
    "print(s3.index) # Index(['Alice', 'Bob', 'Charlie', 'Diana'], dtype='object')\n",
    "print(s3.values) # [85 92 78 95]\n",
    "print(len(s3)) # Output: 4\n",
    "\n",
    "# Series operations\n",
    "print(s3 + 5) # Add 5 to all values\n",
    "print(s3[s3 > 80]) # Values greater than 80\n",
    "# Output:\n",
    "# Alice 85\n",
    "# Bob 92\n",
    "# Diana 95\n",
    "# dtype: int64\n",
    "\n",
    "# Series methods\n",
    "print(s3.mean()) # Output: 87.5\n",
    "print(s3.min()) # Output: 78\n",
    "print(s3.max()) # Output: 95\n",
    "print(s3.sum()) # Output: 350\n",
    "\n",
    "# Real-world: Track daily temperature\n",
    "dates = pd.Series([20, 22, 18, 25, 23, 19, 21],\n",
    "index=['Monday', 'Tuesday', 'Wednesday', 'Thursday',\n",
    "'Friday', 'Saturday', 'Sunday'])\n",
    "print(dates)\n",
    "print(f\"Average temperature: {dates.mean():.1f}\")\n",
    "print(f\"Hottest day: {dates.idxmax()} ({dates.max()}°C)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06139aa2",
   "metadata": {},
   "source": [
    "### 3.3 DataFrames: Tabular Data\n",
    "A DataFrame is a 2D table with labeled rows and columns. Think of it as a spreadsheet or SQL table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "119e7e47",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Create DataFrame from dictionary of lists\n",
    "data = {\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],\n",
    "'Age': [25, 30, 28, 32],\n",
    "'City': ['Lagos', 'Accra', 'Nairobi', 'Johannesburg'],\n",
    "'Salary': [60000, 75000, 65000, 80000]\n",
    "}\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "print(df)\n",
    "# Output:\n",
    "# Name Age City Salary\n",
    "# 0 Alice 25 Lagos 60000\n",
    "# 1 Bob 30 Accra 75000\n",
    "# 2 Charlie 28 Nairobi 65000\n",
    "# 3 Diana 32 Johannesburg 80000\n",
    "\n",
    "# DataFrame properties\n",
    "print(df.shape) # Output: (4, 4) - 4 rows, 4 columns\n",
    "print(df.columns) # Index(['Name', 'Age', 'City', 'Salary'], dtype='object')\n",
    "print(df.index) # RangeIndex(start=0, stop=4, step=1)\n",
    "print(df.dtypes) # Data types of each column\n",
    "print(df.info()) # Summary information\n",
    "\n",
    "# View first/last rows\n",
    "print(df.head()) # First 5 rows (default)\n",
    "print(df.head(2)) # First 2 rows\n",
    "print(df.tail()) # Last 5 rows\n",
    "\n",
    "# Access columns\n",
    "print(df['Name']) # Get 'Name' column as Series\n",
    "print(df['Age']) # Get 'Age' column\n",
    "\n",
    "# Access multiple columns\n",
    "print(df[['Name', 'City']]) # Get multiple columns as DataFrame\n",
    "\n",
    "# Access rows by index\n",
    "print(df.loc[0]) # First row by label\n",
    "print(df.loc[2]) # Row with index 2\n",
    "\n",
    "# Access by position\n",
    "print(df.iloc[0]) # First row by position\n",
    "print(df.iloc[1, 2]) # Row 1, column 2 (Row 1, 'City')\n",
    "\n",
    "# Basic statistics\n",
    "print(df.describe()) # Summary statistics for numeric columns\n",
    "print(df['Salary'].mean()) # Average salary\n",
    "print(df['Age'].max()) # Maximum age"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e43899b",
   "metadata": {},
   "source": [
    "#### Modifying DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03eb3598",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame({\n",
    "'Name': ['Alice', 'Bob', 'Charlie'],\n",
    "'Age': [25, 30, 28],\n",
    "'Score': [85, 92, 78]\n",
    "})\n",
    "\n",
    "# Add new column\n",
    "df['Grade'] = ['B', 'A', 'C']\n",
    "print(df)\n",
    "\n",
    "# Calculate new column from existing\n",
    "df['Age_in_months'] = df['Age'] * 12\n",
    "print(df)\n",
    "\n",
    "# Modify existing column\n",
    "df['Score'] = df['Score'] + 5 # Add 5 to all scores\n",
    "print(df)\n",
    "\n",
    "# Drop column\n",
    "df = df.drop('Age_in_months', axis=1) # axis=1 for column\n",
    "print(df)\n",
    "\n",
    "# Drop row\n",
    "df = df.drop(1, axis=0) # axis=0 for row\n",
    "print(df)\n",
    "\n",
    "# Rename columns\n",
    "df = df.rename(columns={'Score': 'Test_Score', 'Grade': 'Letter_Grade'})\n",
    "print(df)\n",
    "\n",
    "# Sort by column\n",
    "df = df.sort_values('Test_Score', ascending=False)\n",
    "print(df)\n",
    "\n",
    "# Filter rows\n",
    "high_scores = df[df['Test_Score'] > 85]\n",
    "print(high_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ccddd0",
   "metadata": {},
   "source": [
    "### 3.4 Loading Data from Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8741513f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read CSV file\n",
    "df = pd.read_csv('data.csv')\n",
    "\n",
    "# Read with options\n",
    "df = pd.read_csv('data.csv',\n",
    "sep=',', # Delimiter\n",
    "encoding='utf-8', # File encoding\n",
    "nrows=100) # Read only first 100 rows\n",
    "\n",
    "# Read Excel file\n",
    "df = pd.read_excel('data.xlsx', sheet_name=0) # First sheet\n",
    "\n",
    "# Read from multiple sheets\n",
    "xls = pd.ExcelFile('data.xlsx')\n",
    "print(xls.sheet_names) # List all sheets\n",
    "df_sheet1 = pd.read_excel('data.xlsx', sheet_name='Sheet1')\n",
    "df_sheet2 = pd.read_excel('data.xlsx', sheet_name=1)\n",
    "\n",
    "# Read from other formats\n",
    "df = pd.read_json('data.json')\n",
    "df = pd.read_html('webpage.html') # Read tables from HTML\n",
    "\n",
    "# Example: Create and save CSV\n",
    "df = pd.DataFrame({\n",
    "'Product': ['Apple', 'Banana', 'Orange'],\n",
    "'Price': [0.50, 0.30, 0.60],\n",
    "'Quantity': [100, 150, 80]\n",
    "})\n",
    "\n",
    "# Save to CSV\n",
    "df.to_csv('products.csv', index=False)\n",
    "\n",
    "# Save to Excel\n",
    "df.to_excel('products.xlsx', index=False)\n",
    "\n",
    "# Save to JSON\n",
    "df.to_json('products.json')\n",
    "\n",
    "# Real-world: Load and explore data\n",
    "df = pd.read_csv('student_data.csv')\n",
    "print(f\"Dataset shape: {df.shape}\")\n",
    "print(f\"Columns: {list(df.columns)}\")\n",
    "print(df.head(10))\n",
    "print(f\"\\nData types:\\n{df.dtypes}\")\n",
    "print(f\"\\nMissing values:\\n{df.isnull().sum()}\")\n",
    "print(f\"\\nBasic statistics:\\n{df.describe()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0695d0",
   "metadata": {},
   "source": [
    "#### Practical Example: Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff5cdb78",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create sample dataset\n",
    "np.random.seed(42)\n",
    "df = pd.DataFrame({\n",
    "'Student': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],\n",
    "'Math': [85, 92, 78, 88, 95],\n",
    "'English': [90, 85, 88, 92, 89],\n",
    "'Science': [88, 88, 92, 85, 93]\n",
    "})\n",
    "\n",
    "print(\"Dataset:\")\n",
    "print(df)\n",
    "print()\n",
    "\n",
    "# Calculate average score for each student\n",
    "df['Average'] = df[['Math', 'English', 'Science']].mean(axis=1)\n",
    "print(df)\n",
    "print()\n",
    "\n",
    "# Find best subject\n",
    "print(\"Best scores in each subject:\")\n",
    "print(df[['Math', 'English', 'Science']].max())\n",
    "print()\n",
    "\n",
    "# Find students with average > 85\n",
    "excellent = df[df['Average'] > 85]\n",
    "print(\"Students with average > 85:\")\n",
    "print(excellent)\n",
    "print()\n",
    "\n",
    "# Summary statistics\n",
    "print(\"Summary statistics:\")\n",
    "print(df[['Math', 'English', 'Science']].describe())\n",
    "print()\n",
    "\n",
    "# Top performer\n",
    "top_student_idx = df['Average'].idxmax()\n",
    "top_student = df.loc[top_student_idx, 'Student']\n",
    "print(f\"Top performer: {top_student}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8eb0e8c",
   "metadata": {},
   "source": [
    "## Week 3 Summary\n",
    "By completing Week 3, you have learned:\n",
    "- NumPy arrays: the foundation of numerical computing\n",
    "- Creating arrays: from lists, ranges, zeros, ones, random values\n",
    "- Array properties: shape, size, dtype, ndim\n",
    "- Indexing and slicing 1D and 2D arrays\n",
    "- Boolean indexing for filtering data\n",
    "- Arithmetic operations on arrays (element-wise)\n",
    "- Broadcasting: working with arrays of different shapes\n",
    "- Mathematical functions: sum, mean, median, std, sqrt, exp, log\n",
    "- Statistical operations: min, max, argmin, argmax\n",
    "- Sorting, unique values, and value counting\n",
    "- Reshaping and combining arrays\n",
    "- pandas Series: 1D labeled data\n",
    "- pandas DataFrames: 2D tabular data\n",
    "- Loading data from CSV, Excel, and other formats\n",
    "- Basic DataFrame exploration and manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c88864af",
   "metadata": {},
   "source": [
    "## Week 3 Assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b70bd69f",
   "metadata": {},
   "source": [
    "### Assignment 1: NumPy Operations\n",
    "Create a Python script that:\n",
    "- Generates a 10x10 random matrix (values 0-100)\n",
    "- Calculates row-wise mean, column-wise mean, and overall mean\n",
    "- Finds the maximum and minimum values and their positions\n",
    "- Normalizes the matrix (subtract mean, divide by std dev)\n",
    "- Uses boolean indexing to find all values > 75\n",
    "- Reshapes a 1D array into a 2D array and performs calculations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f5eca0e",
   "metadata": {},
   "source": [
    "### Assignment 2: Data Analysis with pandas\n",
    "Create a dataset and analyze it:\n",
    "- Create a DataFrame with at least 50 rows and 5 columns (you can use random data or a public dataset)\n",
    "- Use head(), tail(), info(), describe() to explore the data\n",
    "- Calculate summary statistics by groups\n",
    "- Filter data based on multiple conditions\n",
    "- Create new columns based on existing data\n",
    "- Identify and report patterns or trends\n",
    "- Save your analysis results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6567e35d",
   "metadata": {},
   "source": [
    "### Assignment 3: Real Dataset Exploration\n",
    "Download a real dataset (from Kaggle, UCI, or government sources) and:\n",
    "- Load the data into a DataFrame\n",
    "- Perform complete exploratory analysis\n",
    "- Create a summary report with insights\n",
    "- Save clean data for later analysis\n",
    "- Document your findings in a text file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54e07bea",
   "metadata": {},
   "source": [
    "## Recommended Practice Problems\n",
    "- Create functions for common NumPy operations (normalize, standardize, outlier detection)\n",
    "- Practice array slicing with different combinations of indices\n",
    "- Implement mathematical operations without using built-in functions\n",
    "- Create a data cleaning pipeline (handle missing values, duplicates)\n",
    "- Practice merging DataFrames with different join types\n",
    "- Create visualizations of your data (using pandas .plot() method)\n",
    "- Load multiple datasets and combine them\n",
    "- Practice filtering and sorting operations on real datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d53a28",
   "metadata": {},
   "source": [
    "## Additional Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d68b20",
   "metadata": {},
   "source": [
    "### Books\n",
    "- Chapter 4: NumPy Basics - &quot;Python for Data Analysis&quot; by Wes McKinney\n",
    "- Chapter 5: Getting Started with pandas - &quot;Python for Data Analysis&quot; by Wes McKinney\n",
    "- Data Science from Scratch by Joel Grus - NumPy chapter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "466b8693",
   "metadata": {},
   "source": [
    "### Online Documentation\n",
    "- NumPy Documentation: https://numpy.org/doc/\n",
    "- pandas Documentation: https://pandas.pydata.org/docs/\n",
    "- NumPy Tutorial: https://numpy.org/doc/stable/user/basics.broadcasting.html\n",
    "- pandas Getting Started: https://pandas.pydata.org/docs/getting_started/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213a0ea0",
   "metadata": {},
   "source": [
    "### Practice Datasets\n",
    "- Kaggle: https://www.kaggle.com/ - Thousands of free datasets\n",
    "- UCI Machine Learning Repository: https://archive.ics.uci.edu/ml/\n",
    "- Google Dataset Search: https://datasetsearch.research.google.com/\n",
    "- Seaborn Datasets: Pre-built datasets in seaborn library"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}