{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1e3d1d88",
   "metadata": {},
   "source": [
    "# Week 10: Unsupervised Learning — Clustering and PCA\n",
    "\n",
    "*Kmex Consult — Data Science Fundamentals Course*\n",
    "\n",
    "Work through each code cell in order. Edit and re-run to experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c11da21",
   "metadata": {},
   "source": [
    "*Data Science Fundamentals Course*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f9448a",
   "metadata": {},
   "source": [
    "## Week 10 Overview\n",
    "Week 10 explores unsupervised learning - discovering hidden patterns in data WITHOUT labeled targets. This is crucial because:\n",
    "- Most real-world data is unlabeled\n",
    "- Labeling is expensive and time-consuming\n",
    "- Many problems don't have predefined targets\n",
    "- Need to understand data structure and patterns\n",
    "\n",
    "Unsupervised learning applications:\n",
    "- Customer segmentation for targeted marketing\n",
    "- Gene clustering in bioinformatics\n",
    "- Document clustering and topic modeling\n",
    "- Anomaly detection in cybersecurity\n",
    "- Image compression and denoising\n",
    "- Data visualization and exploration\n",
    "- Recommendation systems\n",
    "\n",
    "This week focuses on two main unsupervised techniques:\n",
    "\n",
    "CLUSTERING: Group similar items together\n",
    "- K-means: Fast, scalable, assumes spherical clusters\n",
    "- Hierarchical: Creates dendrograms, no need to specify k\n",
    "- DBSCAN: Density-based, finds arbitrary shapes\n",
    "\n",
    "DIMENSIONALITY REDUCTION: Reduce number of features\n",
    "- PCA: Linear, finds principal components\n",
    "- t-SNE: Nonlinear, for visualization\n",
    "- Autoencoders: Neural network approach\n",
    "\n",
    "By the end of Week 10, you will be able to:\n",
    "- Perform K-means clustering and interpret results\n",
    "- Understand hierarchical clustering and dendrograms\n",
    "- Apply DBSCAN for density-based clustering\n",
    "- Evaluate clustering quality\n",
    "- Apply PCA for dimensionality reduction\n",
    "- Interpret principal components\n",
    "- Handle high-dimensional data\n",
    "- Create 2D visualizations of complex data\n",
    "- Choose appropriate unsupervised learning methods\n",
    "- Avoid common unsupervised learning mistakes\n",
    "\n",
    "Week 10 is divided into three 2-hour sessions:\n",
    "- Session 1: Clustering Fundamentals and K-means\n",
    "- Session 2: Advanced Clustering (Hierarchical and DBSCAN)\n",
    "- Session 3: Dimensionality Reduction and PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c5fbf7",
   "metadata": {},
   "source": [
    "## SESSION 1: Clustering Fundamentals and K-means"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8acaf650",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6bb4cc7",
   "metadata": {},
   "source": [
    "### 1.1 Unsupervised Learning Fundamentals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f686794",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import make_blobs, load_iris\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"UNSUPERVISED LEARNING FUNDAMENTALS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "SUPERVISED vs UNSUPERVISED:\n",
    "\n",
    "SUPERVISED (Week 9):\n",
    "- We have labeled training data (X, y)\n",
    "- Learn mapping from X to y\n",
    "- Examples: emails→spam/not-spam, images→cat/dog\n",
    "- Easy to evaluate: compare predictions to labels\n",
    "\n",
    "UNSUPERVISED (Week 10):\n",
    "- Only have data X, no labels y\n",
    "- Discover structure and patterns\n",
    "- Examples: group customers, compress images\n",
    "- Harder to evaluate: no ground truth\n",
    "\n",
    "TYPES OF UNSUPERVISED LEARNING:\n",
    "\n",
    "1. CLUSTERING\n",
    "- Group similar items\n",
    "- Questions: How many groups? What's similar?\n",
    "- Algorithms: K-means, Hierarchical, DBSCAN, GMM\n",
    "\n",
    "2. DIMENSIONALITY REDUCTION\n",
    "- Reduce number of features\n",
    "- Questions: Which features matter? How to visualize?\n",
    "- Algorithms: PCA, t-SNE, Autoencoders\n",
    "\n",
    "3. ANOMALY DETECTION\n",
    "- Find unusual items\n",
    "- Questions: What's \"normal\"? What's outlier?\n",
    "- Algorithms: Isolation Forest, One-class SVM\n",
    "\n",
    "4. ASSOCIATION RULES\n",
    "- Find relationships\n",
    "- Questions: What items bought together?\n",
    "- Algorithms: Apriori, Eclat\n",
    "\n",
    "THIS WEEK: CLUSTERING AND DIMENSIONALITY REDUCTION\n",
    "\"\"\")\n",
    "\n",
    "# Create sample data\n",
    "print(\"\\nEXAMPLE: CUSTOMER SEGMENTATION\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "# Generate synthetic customer data\n",
    "np.random.seed(42)\n",
    "n_samples = 300\n",
    "\n",
    "# Create clusters\n",
    "cluster1 = np.random.normal([2, 2], 0.5, (100, 2))\n",
    "cluster2 = np.random.normal([8, 1], 0.5, (100, 2))\n",
    "cluster3 = np.random.normal([5, 8], 0.5, (100, 2))\n",
    "\n",
    "X = np.vstack([cluster1, cluster2, cluster3])\n",
    "y_true = np.hstack([np.zeros(100), np.ones(100), np.ones(100)*2])\n",
    "\n",
    "print(f\"Number of customers: {len(X)}\")\n",
    "print(f\"Features: 2 (e.g., spending, frequency)\")\n",
    "print(f\"True underlying clusters: 3\")\n",
    "print(f\"First few samples:\")\n",
    "print(X[:5])\n",
    "\n",
    "# Visualize\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# True labels (for reference - we won't use in unsupervised learning)\n",
    "scatter1 = axes[0].scatter(X[:, 0], X[:, 1], c=y_true, cmap='viridis', s=50, alpha=0.6)\n",
    "axes[0].set_xlabel('Feature 1')\n",
    "axes[0].set_ylabel('Feature 2')\n",
    "axes[0].set_title('True Clusters (for reference only)')\n",
    "axes[0].grid(True, alpha=0.3)\n",
    "plt.colorbar(scatter1, ax=axes[0])\n",
    "\n",
    "# Unlabeled data (what we actually have)\n",
    "axes[1].scatter(X[:, 0], X[:, 1], c='gray', s=50, alpha=0.6)\n",
    "axes[1].set_xlabel('Feature 1')\n",
    "axes[1].set_ylabel('Feature 2')\n",
    "axes[1].set_title('Unlabeled Data (What We Have)')\n",
    "axes[1].grid(True, alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Key considerations\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"KEY CHALLENGES IN UNSUPERVISED LEARNING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "1. NO GROUND TRUTH\n",
    "- Can't directly evaluate correctness\n",
    "- Need validation metrics (silhouette, inertia)\n",
    "- Require domain knowledge\n",
    "\n",
    "2. CHOOSING NUMBER OF CLUSTERS\n",
    "- How many groups are there?\n",
    "- Too few: Oversimplify, miss patterns\n",
    "- Too many: Overcomplicate, find noise\n",
    "- Use elbow method, silhouette score\n",
    "\n",
    "3. INTERPRETING RESULTS\n",
    "- What do clusters mean?\n",
    "- Are they meaningful or just mathematical artifacts?\n",
    "- Require domain expertise\n",
    "\n",
    "4. SCALABILITY\n",
    "- High dimensions, many samples\n",
    "- Distance metrics become problematic\n",
    "- Computational complexity important\n",
    "\n",
    "5. PARAMETER SENSITIVITY\n",
    "- Results depend on parameters\n",
    "- Different parameters → different clusters\n",
    "- Need to explore multiple settings\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0b5daa6",
   "metadata": {},
   "source": [
    "### 1.2 K-means Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ed55434",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"K-MEANS CLUSTERING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW K-MEANS WORKS:\n",
    "\n",
    "1. INITIALIZE\n",
    "- Randomly choose k points as initial centroids\n",
    "- k = number of clusters (we choose this)\n",
    "\n",
    "2. ASSIGN\n",
    "- Assign each point to nearest centroid\n",
    "- \"Nearest\" = smallest Euclidean distance\n",
    "\n",
    "3. UPDATE\n",
    "- Update centroids to mean of assigned points\n",
    "- Centroid moves toward cluster center\n",
    "\n",
    "4. REPEAT\n",
    "- Repeat assign-update until convergence\n",
    "- Stops when centroids don't move much\n",
    "\n",
    "PROPERTIES:\n",
    "- Local optimization (may not find global optimum)\n",
    "- Sensitive to initialization\n",
    "- Assumes spherical clusters\n",
    "- Requires specifying k in advance\n",
    "- Fast: O(n*k*i) where i = iterations\n",
    "\"\"\")\n",
    "\n",
    "# Generate data\n",
    "np.random.seed(42)\n",
    "X, y_true = make_blobs(n_samples=300, centers=3, n_features=2,\n",
    "cluster_std=0.6, random_state=42)\n",
    "\n",
    "# Standardize features\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "print(\"\\nEXAMPLE: CLUSTERING SYNTHETIC DATA\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(f\"Number of samples: {X_scaled.shape[0]}\")\n",
    "print(f\"Number of features: {X_scaled.shape[1]}\")\n",
    "\n",
    "# Fit K-means\n",
    "kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)\n",
    "clusters = kmeans.fit_predict(X_scaled)\n",
    "\n",
    "print(f\"\\nK-means Results (k=3):\")\n",
    "print(f\"Centroids shape: {kmeans.cluster_centers_.shape}\")\n",
    "print(f\"Inertia (sum of squared distances): {kmeans.inertia_:.2f}\")\n",
    "print(f\"\\nCluster sizes:\")\n",
    "unique, counts = np.unique(clusters, return_counts=True)\n",
    "for cluster_id, count in zip(unique, counts):\n",
    "print(f\" Cluster {cluster_id}: {count} samples\")\n",
    "\n",
    "# Predictions for new points\n",
    "new_point = np.array([[0.5, 0.5]])\n",
    "new_point_scaled = scaler.transform(new_point)\n",
    "prediction = kmeans.predict(new_point_scaled)\n",
    "distance_to_centroid = np.min(np.linalg.norm(new_point_scaled - kmeans.cluster_centers_, axis=1))\n",
    "\n",
    "print(f\"\\nNew point prediction:\")\n",
    "print(f\" Point (scaled): {new_point_scaled}\")\n",
    "print(f\" Assigned to cluster: {prediction[0]}\")\n",
    "print(f\" Distance to centroid: {distance_to_centroid:.3f}\")\n",
    "\n",
    "# Choosing k: Elbow Method\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"CHOOSING K: ELBOW METHOD\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "ELBOW METHOD:\n",
    "1. Fit K-means for different k values\n",
    "2. Plot inertia (within-cluster sum of squares) vs k\n",
    "3. Look for \"elbow\" - point where decrease slows\n",
    "4. Choose k at elbow\n",
    "\"\"\")\n",
    "\n",
    "inertias = []\n",
    "silhouette_scores = []\n",
    "K_range = range(1, 10)\n",
    "\n",
    "from sklearn.metrics import silhouette_score\n",
    "\n",
    "for k in K_range:\n",
    "kmeans_k = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
    "clusters_k = kmeans_k.fit_predict(X_scaled)\n",
    "inertias.append(kmeans_k.inertia_)\n",
    "\n",
    "if k > 1: # Silhouette needs at least 2 clusters\n",
    "silhouette_scores.append(silhouette_score(X_scaled, clusters_k))\n",
    "else:\n",
    "silhouette_scores.append(None)\n",
    "\n",
    "print(\"\\nInertia for different k values:\")\n",
    "for k, inertia in zip(K_range, inertias):\n",
    "print(f\" k={k}: {inertia:.2f}\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(2, 2, figsize=(14, 12))\n",
    "\n",
    "# Plot 1: Clustered data\n",
    "scatter = axes[0, 0].scatter(X_scaled[:, 0], X_scaled[:, 1], c=clusters,\n",
    "cmap='viridis', s=50, alpha=0.6)\n",
    "axes[0, 0].scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1],\n",
    "c='red', marker='X', s=200, edgecolors='black', linewidths=2,\n",
    "label='Centroids')\n",
    "axes[0, 0].set_xlabel('Feature 1')\n",
    "axes[0, 0].set_ylabel('Feature 2')\n",
    "axes[0, 0].set_title('K-means Clustering (k=3)')\n",
    "axes[0, 0].legend()\n",
    "axes[0, 0].grid(True, alpha=0.3)\n",
    "plt.colorbar(scatter, ax=axes[0, 0])\n",
    "\n",
    "# Plot 2: Elbow curve\n",
    "axes[0, 1].plot(K_range, inertias, 'b-o', linewidth=2, markersize=8)\n",
    "axes[0, 1].axvline(x=3, color='red', linestyle='--', linewidth=2, label='Elbow at k=3')\n",
    "axes[0, 1].set_xlabel('Number of Clusters (k)')\n",
    "axes[0, 1].set_ylabel('Inertia')\n",
    "axes[0, 1].set_title('Elbow Method')\n",
    "axes[0, 1].legend()\n",
    "axes[0, 1].grid(True, alpha=0.3)\n",
    "\n",
    "# Plot 3: Silhouette scores\n",
    "valid_k = [k for k, s in zip(K_range, silhouette_scores) if s is not None]\n",
    "valid_scores = [s for s in silhouette_scores if s is not None]\n",
    "axes[1, 0].plot(valid_k, valid_scores, 'g-o', linewidth=2, markersize=8)\n",
    "axes[1, 0].axvline(x=3, color='red', linestyle='--', linewidth=2)\n",
    "axes[1, 0].set_xlabel('Number of Clusters (k)')\n",
    "axes[1, 0].set_ylabel('Silhouette Score')\n",
    "axes[1, 0].set_title('Silhouette Score vs k')\n",
    "axes[1, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# Plot 4: Comparison of k values\n",
    "for k in [2, 3, 4]:\n",
    "kmeans_k = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
    "clusters_k = kmeans_k.fit_predict(X_scaled)\n",
    "\n",
    "axes[1, 1].scatter(X_scaled[:, 0], X_scaled[:, 1], c=clusters_k,\n",
    "alpha=0.3, s=30)\n",
    "\n",
    "axes[1, 1].set_xlabel('Feature 1')\n",
    "axes[1, 1].set_ylabel('Feature 2')\n",
    "axes[1, 1].set_title('Comparison: Different k values')\n",
    "axes[1, 1].grid(True, alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Advantages and disadvantages\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"K-MEANS ADVANTAGES AND DISADVANTAGES\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "ADVANTAGES:\n",
    "✓ Fast: O(nki) where n=samples, k=clusters, i=iterations\n",
    "✓ Scalable: Works well with large datasets\n",
    "✓ Simple to understand and implement\n",
    "✓ Easy to interpret: cluster centers are meaningful\n",
    "✓ Works well with roughly spherical clusters\n",
    "\n",
    "DISADVANTAGES:\n",
    "✗ Must specify k in advance\n",
    "✗ Sensitive to initialization (may find local minima)\n",
    "✗ Assumes spherical, similarly-sized clusters\n",
    "✗ Affected by outliers\n",
    "✗ Doesn't scale well with high dimensions\n",
    "✗ All points must be assigned (no outlier detection)\n",
    "\n",
    "WHEN TO USE:\n",
    "- Large datasets\n",
    "- Spherical cluster shapes\n",
    "- Need speed\n",
    "- Know or can estimate k\n",
    "\n",
    "WHEN NOT TO USE:\n",
    "- Complex cluster shapes\n",
    "- Unknown number of clusters\n",
    "- Many outliers\n",
    "- Very high dimensions\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f4355cc",
   "metadata": {},
   "source": [
    "## SESSION 2: Advanced Clustering (Hierarchical and DBSCAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d681eb32",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd2fce5b",
   "metadata": {},
   "source": [
    "### 2.1 Hierarchical Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9da9fd2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.cluster import AgglomerativeClustering\n",
    "from scipy.cluster.hierarchy import dendrogram, linkage\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"HIERARCHICAL CLUSTERING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HIERARCHICAL CLUSTERING:\n",
    "- Build tree of clusters (dendrogram)\n",
    "- Two approaches: Agglomerative and Divisive\n",
    "\n",
    "AGGLOMERATIVE (Bottom-Up):\n",
    "1. Start with each point as own cluster\n",
    "2. Merge closest clusters iteratively\n",
    "3. Continue until one big cluster\n",
    "4. Result: Tree showing merge sequence\n",
    "\n",
    "DIVISIVE (Top-Down):\n",
    "1. Start with all points in one cluster\n",
    "2. Recursively split clusters\n",
    "3. Continue until each point separate\n",
    "4. Less common, more expensive\n",
    "\n",
    "LINKAGE METHODS (how to measure cluster distance):\n",
    "- Single: Distance between closest points\n",
    "- Complete: Distance between farthest points\n",
    "- Average: Average distance between all pairs\n",
    "- Ward: Minimizes within-cluster variance\n",
    "\"\"\")\n",
    "\n",
    "# Generate data\n",
    "np.random.seed(42)\n",
    "X, y_true = make_blobs(n_samples=100, centers=4, n_features=2,\n",
    "cluster_std=0.7, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "print(\"\\nEXAMPLE: HIERARCHICAL CLUSTERING\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(f\"Number of samples: {len(X_scaled)}\")\n",
    "print(f\"Number of features: {X_scaled.shape[1]}\")\n",
    "\n",
    "# Hierarchical clustering with different linkages\n",
    "linkages = ['ward', 'complete', 'average', 'single']\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(14, 10))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for idx, linkage_method in enumerate(linkages):\n",
    "Z = linkage(X_scaled, method=linkage_method)\n",
    "\n",
    "dendrogram(Z, ax=axes[idx], no_labels=True)\n",
    "axes[idx].set_title(f'Dendrogram ({linkage_method.capitalize()} Linkage)')\n",
    "axes[idx].set_xlabel('Sample Index')\n",
    "axes[idx].set_ylabel('Distance')\n",
    "axes[idx].axhline(y=5, color='red', linestyle='--', linewidth=2, label='Cut line')\n",
    "axes[idx].legend()\n",
    "\n",
    "plt.suptitle('Hierarchical Clustering with Different Linkages',\n",
    "fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Use Ward linkage (most common)\n",
    "print(\"\\nUsing Ward linkage:\")\n",
    "Z_ward = linkage(X_scaled, method='ward')\n",
    "\n",
    "# Agglomerative clustering\n",
    "n_clusters = 4\n",
    "hierarchical = AgglomerativeClustering(n_clusters=n_clusters, linkage='ward')\n",
    "clusters = hierarchical.fit_predict(X_scaled)\n",
    "\n",
    "print(f\"\\nResults (k={n_clusters}):\")\n",
    "unique, counts = np.unique(clusters, return_counts=True)\n",
    "for cluster_id, count in zip(unique, counts):\n",
    "print(f\" Cluster {cluster_id}: {count} samples\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# Dendrogram\n",
    "dendrogram(Z_ward, ax=axes[0], no_labels=True)\n",
    "axes[0].axhline(y=10, color='red', linestyle='--', linewidth=2, label='Cut for 4 clusters')\n",
    "axes[0].set_title('Dendrogram (Ward Linkage)')\n",
    "axes[0].set_xlabel('Sample Index')\n",
    "axes[0].set_ylabel('Distance')\n",
    "axes[0].legend()\n",
    "axes[0].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# Clustered data\n",
    "scatter = axes[1].scatter(X_scaled[:, 0], X_scaled[:, 1], c=clusters,\n",
    "cmap='viridis', s=50, alpha=0.6)\n",
    "axes[1].set_xlabel('Feature 1')\n",
    "axes[1].set_ylabel('Feature 2')\n",
    "axes[1].set_title(f'Hierarchical Clustering (k={n_clusters})')\n",
    "axes[1].grid(True, alpha=0.3)\n",
    "plt.colorbar(scatter, ax=axes[1])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Advantages and disadvantages\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"HIERARCHICAL CLUSTERING ADVANTAGES AND DISADVANTAGES\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "ADVANTAGES:\n",
    "✓ No need to specify k in advance (read dendrogram)\n",
    "✓ Dendrogram provides interpretable tree structure\n",
    "✓ Works with different cluster shapes\n",
    "✓ Deterministic (no random initialization)\n",
    "✓ Can use any distance metric\n",
    "✓ Good for hierarchical data\n",
    "\n",
    "DISADVANTAGES:\n",
    "✗ Computationally expensive: O(n²) to O(n³)\n",
    "✗ Can't undo merges (greedy algorithm)\n",
    "✗ Sensitive to noise and outliers\n",
    "✗ Different linkages give different results\n",
    "✗ Not scalable to very large datasets\n",
    "✗ All points must be assigned (no outlier handling)\n",
    "\n",
    "WHEN TO USE:\n",
    "- Small to medium datasets\n",
    "- Hierarchical relationships matter\n",
    "- Want to explore multiple k values\n",
    "- Need interpretable structure\n",
    "\n",
    "WHEN NOT TO USE:\n",
    "- Very large datasets\n",
    "- Need speed/scalability\n",
    "- Have no hierarchical structure\n",
    "- Working with high dimensions\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "701b5a72",
   "metadata": {},
   "source": [
    "### 2.2 DBSCAN: Density-Based Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aebd1b34",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn.datasets import make_moons\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neighbors import NearestNeighbors\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"DBSCAN (Density-Based Spatial Clustering)\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW DBSCAN WORKS:\n",
    "\n",
    "1. CORE POINTS\n",
    "- Points with >= min_samples neighbors within eps distance\n",
    "- \"Dense\" regions\n",
    "\n",
    "2. BORDER POINTS\n",
    "- Non-core points within eps of core point\n",
    "- On cluster boundary\n",
    "\n",
    "3. NOISE POINTS\n",
    "- Points not core or border\n",
    "- Outliers/anomalies\n",
    "\n",
    "4. CLUSTERING\n",
    "- Group core points that are close\n",
    "- Include border points\n",
    "- Mark noise points as outliers (-1)\n",
    "\n",
    "PARAMETERS:\n",
    "- eps: Maximum distance between points (radius)\n",
    "- min_samples: Minimum points in neighborhood for core point\n",
    "Usually: 2*n_features or larger\n",
    "\n",
    "PROPERTIES:\n",
    "- Arbitrary cluster shapes\n",
    "- Automatic outlier detection\n",
    "- No need to specify number of clusters\n",
    "- Good with varying density (if eps set right)\n",
    "\"\"\")\n",
    "\n",
    "# Generate data with non-convex shapes\n",
    "print(\"\\nEXAMPLE: CLUSTERING NON-CONVEX DATA\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "# Create moons dataset (two crescents)\n",
    "X, y_true = make_moons(n_samples=300, noise=0.05, random_state=42)\n",
    "X_scaled = StandardScaler().fit_transform(X)\n",
    "\n",
    "print(f\"Generated moons dataset with {len(X)} samples\")\n",
    "print(f\"True structure: Two crescents (K-means would fail here!)\")\n",
    "\n",
    "# Find optimal eps using k-distance graph\n",
    "neighbors = NearestNeighbors(n_neighbors=5)\n",
    "neighbors_fit = neighbors.fit(X_scaled)\n",
    "distances, indices = neighbors_fit.kneighbors(X_scaled)\n",
    "distances = np.sort(distances[:, -1], axis=0)\n",
    "\n",
    "print(\"\\nFinding optimal eps:\")\n",
    "# Optimal eps usually at \"knee\" of distance curve\n",
    "print(f\"Distance at 90th percentile: {np.percentile(distances, 90):.3f}\")\n",
    "\n",
    "# Fit DBSCAN\n",
    "eps = 0.2\n",
    "min_samples = 5\n",
    "\n",
    "dbscan = DBSCAN(eps=eps, min_samples=min_samples)\n",
    "clusters = dbscan.fit_predict(X_scaled)\n",
    "\n",
    "n_clusters = len(set(clusters)) - (1 if -1 in clusters else 0)\n",
    "n_outliers = list(clusters).count(-1)\n",
    "\n",
    "print(f\"\\nDBSCAN Results (eps={eps}, min_samples={min_samples}):\")\n",
    "print(f\"Number of clusters: {n_clusters}\")\n",
    "print(f\"Number of outliers: {n_outliers}\")\n",
    "print(f\"\\nCluster sizes:\")\n",
    "for cluster_id in sorted(set(clusters)):\n",
    "if cluster_id == -1:\n",
    "print(f\" Outliers: {list(clusters).count(cluster_id)}\")\n",
    "else:\n",
    "print(f\" Cluster {cluster_id}: {list(clusters).count(cluster_id)}\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(2, 2, figsize=(14, 12))\n",
    "\n",
    "# Plot 1: K-distance graph\n",
    "axes[0, 0].plot(distances)\n",
    "axes[0, 0].axhline(y=eps, color='red', linestyle='--', linewidth=2, label=f'eps={eps}')\n",
    "axes[0, 0].set_xlabel('Points sorted by distance')\n",
    "axes[0, 0].set_ylabel('5-nearest neighbor distance')\n",
    "axes[0, 0].set_title('K-distance Graph (for choosing eps)')\n",
    "axes[0, 0].legend()\n",
    "axes[0, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# Plot 2: DBSCAN clustering\n",
    "scatter = axes[0, 1].scatter(X_scaled[:, 0], X_scaled[:, 1], c=clusters,\n",
    "cmap='viridis', s=50, alpha=0.6)\n",
    "# Highlight outliers\n",
    "outlier_mask = clusters == -1\n",
    "axes[0, 1].scatter(X_scaled[outlier_mask, 0], X_scaled[outlier_mask, 1],\n",
    "c='red', marker='X', s=200, edgecolors='black',\n",
    "linewidths=2, label='Outliers')\n",
    "axes[0, 1].set_xlabel('Feature 1')\n",
    "axes[0, 1].set_ylabel('Feature 2')\n",
    "axes[0, 1].set_title(f'DBSCAN Clustering')\n",
    "axes[0, 1].legend()\n",
    "axes[0, 1].grid(True, alpha=0.3)\n",
    "plt.colorbar(scatter, ax=axes[0, 1])\n",
    "\n",
    "# Plot 3: Different eps values\n",
    "eps_values = [0.1, 0.2, 0.3]\n",
    "for idx, eps_val in enumerate(eps_values):\n",
    "ax = axes[1, idx // 2] if idx < 2 else axes[1, 1]\n",
    "\n",
    "dbscan_eps = DBSCAN(eps=eps_val, min_samples=5)\n",
    "clusters_eps = dbscan_eps.fit_predict(X_scaled)\n",
    "\n",
    "ax.scatter(X_scaled[:, 0], X_scaled[:, 1], c=clusters_eps,\n",
    "cmap='viridis', s=50, alpha=0.6)\n",
    "n_clust = len(set(clusters_eps)) - (1 if -1 in clusters_eps else 0)\n",
    "n_out = list(clusters_eps).count(-1)\n",
    "\n",
    "ax.set_title(f'eps={eps_val} (k={n_clust}, outliers={n_out})')\n",
    "ax.grid(True, alpha=0.3)\n",
    "\n",
    "# Remove extra subplot\n",
    "if len(eps_values) < 3:\n",
    "fig.delaxes(axes[1, 1])\n",
    "\n",
    "plt.suptitle('DBSCAN: Effect of eps Parameter', fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Advantages and disadvantages\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"DBSCAN ADVANTAGES AND DISADVANTAGES\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "ADVANTAGES:\n",
    "✓ Finds arbitrary cluster shapes\n",
    "✓ Automatic outlier detection\n",
    "✓ No need to specify number of clusters\n",
    "✓ Works well with varying density (if eps chosen well)\n",
    "✓ Deterministic (no random initialization)\n",
    "\n",
    "DISADVANTAGES:\n",
    "✗ Difficult to choose eps and min_samples\n",
    "✗ Bad with clusters of varying density\n",
    "✗ Sensitivity to parameters\n",
    "✗ O(n²) to O(n log n) complexity\n",
    "✗ High-dimensional data problematic\n",
    "\n",
    "WHEN TO USE:\n",
    "- Non-convex cluster shapes\n",
    "- Need outlier detection\n",
    "- Don't know number of clusters\n",
    "- Varying cluster density acceptable\n",
    "\n",
    "WHEN NOT TO USE:\n",
    "- Uniform cluster density required\n",
    "- Need to specify exact clusters\n",
    "- Very high dimensions\n",
    "- Large datasets (slow)\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55b680fc",
   "metadata": {},
   "source": [
    "## SESSION 3: Dimensionality Reduction and PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6870f41e",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b07a8635",
   "metadata": {},
   "source": [
    "### 3.1 Dimensionality Reduction Fundamentals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7897c9a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"DIMENSIONALITY REDUCTION\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "WHY REDUCE DIMENSIONS?\n",
    "\n",
    "1. VISUALIZATION\n",
    "- Reduce to 2D or 3D for plotting\n",
    "- Understand data structure\n",
    "\n",
    "2. CURSE OF DIMENSIONALITY\n",
    "- Too many features, not enough samples\n",
    "- Model overfitting, poor generalization\n",
    "- Distance metrics become meaningless\n",
    "- Computational cost explodes\n",
    "\n",
    "3. NOISE REDUCTION\n",
    "- Remove irrelevant features\n",
    "- Keep only important information\n",
    "- Improve model performance\n",
    "\n",
    "4. FEATURE EXTRACTION\n",
    "- Create new features from old ones\n",
    "- May be more interpretable\n",
    "\n",
    "METHODS:\n",
    "\n",
    "LINEAR:\n",
    "- PCA: Unsupervised, finds variance directions\n",
    "- Eigenvectors capture principal components\n",
    "- Fast, interpretable\n",
    "\n",
    "NONLINEAR:\n",
    "- t-SNE: Excellent for visualization\n",
    "- UMAP: Fast t-SNE alternative\n",
    "- Autoencoders: Deep learning approach\n",
    "\"\"\")\n",
    "\n",
    "# Load data\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "target_names = iris.target_names\n",
    "feature_names = iris.feature_names\n",
    "\n",
    "print(f\"\\nEXAMPLE: IRIS DATASET\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(f\"Original data shape: {X.shape}\")\n",
    "print(f\"Features: {feature_names}\")\n",
    "print(f\"Classes: {target_names}\")\n",
    "\n",
    "# High-dimensional visualization problem\n",
    "print(\"\\nChallenge:\")\n",
    "print(\"- 4 features (4D data)\")\n",
    "print(\"- Can't visualize 4D directly\")\n",
    "print(\"- Solution: Reduce to 2D for visualization\")\n",
    "\n",
    "# Standardize\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# PCA with different components\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"PCA: PRINCIPAL COMPONENT ANALYSIS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW PCA WORKS:\n",
    "\n",
    "1. STANDARDIZE DATA\n",
    "- Mean = 0, variance = 1\n",
    "\n",
    "2. COMPUTE COVARIANCE MATRIX\n",
    "- Show relationships between features\n",
    "\n",
    "3. FIND EIGENVECTORS AND EIGENVALUES\n",
    "- Eigenvectors: principal component directions\n",
    "- Eigenvalues: variance explained per component\n",
    "\n",
    "4. ORDER BY VARIANCE\n",
    "- Largest eigenvalue = first principal component\n",
    "- Explains most variance\n",
    "\n",
    "5. SELECT COMPONENTS\n",
    "- Keep top k components\n",
    "- Usually retain 90-95% variance\n",
    "\"\"\")\n",
    "\n",
    "# Fit full PCA\n",
    "pca_full = PCA()\n",
    "X_pca_full = pca_full.fit_transform(X_scaled)\n",
    "\n",
    "# Explained variance\n",
    "explained_var = pca_full.explained_variance_ratio_\n",
    "cumsum_var = np.cumsum(explained_var)\n",
    "\n",
    "print(f\"\\nExplained Variance by Component:\")\n",
    "for i, (var, cumsum) in enumerate(zip(explained_var, cumsum_var)):\n",
    "print(f\" PC{i+1}: {var:.4f} ({var*100:.2f}%) [Cumulative: {cumsum*100:.2f}%]\")\n",
    "\n",
    "print(f\"\\nTo retain 90% variance: need {np.argmax(cumsum_var >= 0.9) + 1} components\")\n",
    "print(f\"To retain 95% variance: need {np.argmax(cumsum_var >= 0.95) + 1} components\")\n",
    "\n",
    "# Fit 2-component PCA\n",
    "pca_2 = PCA(n_components=2)\n",
    "X_pca_2 = pca_2.fit_transform(X_scaled)\n",
    "\n",
    "print(f\"\\n2-component PCA:\")\n",
    "print(f\" Explained variance: {pca_2.explained_variance_ratio_.sum()*100:.2f}%\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(2, 2, figsize=(14, 12))\n",
    "\n",
    "# Plot 1: Scree plot (variance explained)\n",
    "axes[0, 0].plot(range(1, len(explained_var)+1), explained_var, 'b-o', linewidth=2)\n",
    "axes[0, 0].set_xlabel('Principal Component')\n",
    "axes[0, 0].set_ylabel('Explained Variance Ratio')\n",
    "axes[0, 0].set_title('Scree Plot')\n",
    "axes[0, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# Plot 2: Cumulative explained variance\n",
    "axes[0, 1].plot(range(1, len(cumsum_var)+1), cumsum_var, 'g-o', linewidth=2)\n",
    "axes[0, 1].axhline(y=0.9, color='red', linestyle='--', label='90% variance')\n",
    "axes[0, 1].axhline(y=0.95, color='orange', linestyle='--', label='95% variance')\n",
    "axes[0, 1].set_xlabel('Number of Components')\n",
    "axes[0, 1].set_ylabel('Cumulative Explained Variance')\n",
    "axes[0, 1].set_title('Cumulative Explained Variance')\n",
    "axes[0, 1].legend()\n",
    "axes[0, 1].grid(True, alpha=0.3)\n",
    "\n",
    "# Plot 3: 2D PCA projection\n",
    "for i, target in enumerate(np.unique(y)):\n",
    "axes[1, 0].scatter(X_pca_2[y == target, 0], X_pca_2[y == target, 1],\n",
    "label=target_names[target], s=50, alpha=0.6)\n",
    "axes[1, 0].set_xlabel(f'PC1 ({pca_2.explained_variance_ratio_[0]*100:.1f}%)')\n",
    "axes[1, 0].set_ylabel(f'PC2 ({pca_2.explained_variance_ratio_[1]*100:.1f}%)')\n",
    "axes[1, 0].set_title('Iris Data Projected to 2 Principal Components')\n",
    "axes[1, 0].legend()\n",
    "axes[1, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# Plot 4: Component loadings (which original features matter)\n",
    "loadings = pca_2.components_.T * np.sqrt(pca_2.explained_variance_)\n",
    "loading_df = pd.DataFrame(loadings, columns=['PC1', 'PC2'], index=feature_names)\n",
    "\n",
    "loading_df.plot(kind='barh', ax=axes[1, 1])\n",
    "axes[1, 1].set_xlabel('Loading')\n",
    "axes[1, 1].set_title('Feature Contributions to Principal Components')\n",
    "axes[1, 1].grid(True, alpha=0.3, axis='x')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Interpretation\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"INTERPRETING PCA\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\\nComponent Loadings (which features matter):\")\n",
    "print(loading_df)\n",
    "\n",
    "print(\"\\nInterpretation:\")\n",
    "print(\"- High loading: Feature important for that component\")\n",
    "print(\"- Positive loading: Feature increases with component\")\n",
    "print(\"- Negative loading: Feature decreases with component\")\n",
    "print(\"- Loadings help understand what components represent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb8ab760",
   "metadata": {},
   "source": [
    "### 3.2 Advanced Techniques: t-SNE and UMAP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "249ad4f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_iris, load_digits\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.manifold import TSNE\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"ADVANCED DIMENSIONALITY REDUCTION\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "PCA LIMITATIONS:\n",
    "- Linear transformation only\n",
    "- Can miss nonlinear structures\n",
    "- Not ideal for visualization\n",
    "- Example: Swiss roll data (PCA fails)\n",
    "\n",
    "NONLINEAR ALTERNATIVES:\n",
    "\n",
    "1. t-SNE (t-Distributed Stochastic Neighbor Embedding)\n",
    "PROS:\n",
    "- Excellent for visualization\n",
    "- Preserves local structure well\n",
    "- Great for exploratory analysis\n",
    "\n",
    "CONS:\n",
    "- Slow: O(n²) or O(n log n)\n",
    "- Non-deterministic (use random_state)\n",
    "- Hyperparameters matter: perplexity, learning_rate\n",
    "- Not good for prediction (can't transform new data easily)\n",
    "- Can create misleading clusters\n",
    "\n",
    "2. UMAP (Uniform Manifold Approximation and Projection)\n",
    "PROS:\n",
    "- Faster than t-SNE\n",
    "- Better preserves global structure\n",
    "- Can transform new data\n",
    "- Works well for high dimensions\n",
    "\n",
    "CONS:\n",
    "- Newer, less established\n",
    "- Still hyperparameter sensitive\n",
    "- Interpretation can be tricky\n",
    "\n",
    "3. AUTOENCODERS\n",
    "PROS:\n",
    "- Deep learning approach\n",
    "- Can learn complex transformations\n",
    "- Good for feature extraction\n",
    "\n",
    "CONS:\n",
    "- Requires neural network knowledge\n",
    "- More parameters to tune\n",
    "- Can overfit\n",
    "\"\"\")\n",
    "\n",
    "# Load a more interesting dataset\n",
    "digits = load_digits()\n",
    "X = digits.data\n",
    "y = digits.target\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "print(f\"\\nEXAMPLE: HANDWRITTEN DIGITS\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(f\"Original shape: {X.shape} (64 features = 8x8 pixels)\")\n",
    "print(f\"Classes: 10 (digits 0-9)\")\n",
    "\n",
    "# t-SNE\n",
    "print(\"\\nFitting t-SNE (this may take a moment)...\")\n",
    "\n",
    "tsne = TSNE(n_components=2, random_state=42, perplexity=30, n_iter=1000)\n",
    "X_tsne = tsne.fit_transform(X_scaled[:500]) # Use subset for speed\n",
    "y_subset = y[:500]\n",
    "\n",
    "print(\"t-SNE fitting complete!\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
    "\n",
    "# PCA\n",
    "from sklearn.decomposition import PCA\n",
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X_scaled[:500])\n",
    "\n",
    "scatter1 = axes[0].scatter(X_pca[:, 0], X_pca[:, 1], c=y_subset,\n",
    "cmap='tab10', s=50, alpha=0.6)\n",
    "axes[0].set_xlabel(f'PC1 ({pca.explained_variance_ratio_[0]*100:.1f}%)')\n",
    "axes[0].set_ylabel(f'PC2 ({pca.explained_variance_ratio_[1]*100:.1f}%)')\n",
    "axes[0].set_title('PCA Projection')\n",
    "axes[0].grid(True, alpha=0.3)\n",
    "plt.colorbar(scatter1, ax=axes[0])\n",
    "\n",
    "# t-SNE\n",
    "scatter2 = axes[1].scatter(X_tsne[:, 0], X_tsne[:, 1], c=y_subset,\n",
    "cmap='tab10', s=50, alpha=0.6)\n",
    "axes[1].set_xlabel('t-SNE 1')\n",
    "axes[1].set_ylabel('t-SNE 2')\n",
    "axes[1].set_title('t-SNE Projection')\n",
    "axes[1].grid(True, alpha=0.3)\n",
    "plt.colorbar(scatter2, ax=axes[1])\n",
    "\n",
    "plt.suptitle('PCA vs t-SNE: Digits Dataset', fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "print(\"\\nComparison:\")\n",
    "print(\"- PCA: Fast, global structure, linear\")\n",
    "print(\"- t-SNE: Slow, local structure, nonlinear\")\n",
    "print(\"- t-SNE shows clusters better but distorts global distances\")\n",
    "\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"CHOOSING DIMENSIONALITY REDUCTION METHOD\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "USE PCA IF:\n",
    "✓ Need speed and interpretability\n",
    "✓ Want explained variance metric\n",
    "✓ Need to transform new data\n",
    "✓ Linear relationships sufficient\n",
    "\n",
    "USE t-SNE IF:\n",
    "✓ Need visualization only\n",
    "✓ Have time for computation\n",
    "✓ Want to see local clusters\n",
    "✓ Non-linear structure important\n",
    "\n",
    "USE UMAP IF:\n",
    "✓ Need balance of speed and quality\n",
    "✓ Want global + local structure\n",
    "✓ Need to transform new data\n",
    "✓ Have large dataset\n",
    "\n",
    "USE AUTOENCODERS IF:\n",
    "✓ Expert in deep learning\n",
    "✓ Complex nonlinear relationships\n",
    "✓ Feature learning important\n",
    "✓ Have lots of data and compute\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ba6af56",
   "metadata": {},
   "source": [
    "### 3.3 Evaluating Clustering and Best Practices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0decddc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import (silhouette_score, davies_bouldin_score,\n",
    "calinski_harabasz_score)\n",
    "import pandas as pd\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"EVALUATING CLUSTERING QUALITY\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "CHALLENGE:\n",
    "- No ground truth labels in unsupervised learning\n",
    "- Can't directly measure correctness\n",
    "- Need internal validation metrics\n",
    "\n",
    "CLUSTERING EVALUATION METRICS:\n",
    "\n",
    "1. SILHOUETTE SCORE (most common)\n",
    "- Range: -1 to 1\n",
    "- 1: Perfect clustering\n",
    "- 0: Overlapping clusters\n",
    "- -1: Wrong clustering\n",
    "- Measures: How similar point is to own cluster vs others\n",
    "\n",
    "2. DAVIES-BOULDIN INDEX\n",
    "- Range: 0 to ∞\n",
    "- Lower is better\n",
    "- Measures: Ratio of distances\n",
    "\n",
    "3. CALINSKI-HARABASZ INDEX\n",
    "- Range: 0 to ∞\n",
    "- Higher is better\n",
    "- Measures: Ratio of between to within cluster variance\n",
    "\n",
    "4. INERTIA (for K-means)\n",
    "- Sum of squared distances to nearest centroid\n",
    "- Lower is better\n",
    "- Can't compare across different k\n",
    "\"\"\")\n",
    "\n",
    "# Generate data\n",
    "np.random.seed(42)\n",
    "X, y_true = make_blobs(n_samples=300, centers=3, cluster_std=0.6, random_state=42)\n",
    "X_scaled = StandardScaler().fit_transform(X)\n",
    "\n",
    "print(\"\\nEXAMPLE: EVALUATE K-MEANS WITH DIFFERENT K\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "results = []\n",
    "\n",
    "for k in range(2, 10):\n",
    "kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
    "clusters = kmeans.fit_predict(X_scaled)\n",
    "\n",
    "silhouette = silhouette_score(X_scaled, clusters)\n",
    "davies_bouldin = davies_bouldin_score(X_scaled, clusters)\n",
    "calinski = calinski_harabasz_score(X_scaled, clusters)\n",
    "inertia = kmeans.inertia_\n",
    "\n",
    "results.append({\n",
    "'k': k,\n",
    "'Silhouette': silhouette,\n",
    "'Davies-Bouldin': davies_bouldin,\n",
    "'Calinski-Harabasz': calinski,\n",
    "'Inertia': inertia\n",
    "})\n",
    "\n",
    "results_df = pd.DataFrame(results)\n",
    "print(results_df.to_string(index=False))\n",
    "\n",
    "# Find optimal k\n",
    "optimal_k_silhouette = results_df.loc[results_df['Silhouette'].idxmax(), 'k']\n",
    "optimal_k_davies = results_df.loc[results_df['Davies-Bouldin'].idxmin(), 'k']\n",
    "optimal_k_calinski = results_df.loc[results_df['Calinski-Harabasz'].idxmax(), 'k']\n",
    "\n",
    "print(f\"\\nOptimal k by:\")\n",
    "print(f\" Silhouette Score: k={int(optimal_k_silhouette)}\")\n",
    "print(f\" Davies-Bouldin Index: k={int(optimal_k_davies)}\")\n",
    "print(f\" Calinski-Harabasz: k={int(optimal_k_calinski)}\")\n",
    "\n",
    "# Visualization\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(14, 10))\n",
    "\n",
    "# Silhouette (higher is better)\n",
    "axes[0, 0].plot(results_df['k'], results_df['Silhouette'], 'b-o', linewidth=2)\n",
    "axes[0, 0].axvline(x=optimal_k_silhouette, color='red', linestyle='--', alpha=0.7)\n",
    "axes[0, 0].set_xlabel('Number of Clusters (k)')\n",
    "axes[0, 0].set_ylabel('Silhouette Score')\n",
    "axes[0, 0].set_title('Silhouette Score (Higher is Better)')\n",
    "axes[0, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# Davies-Bouldin (lower is better)\n",
    "axes[0, 1].plot(results_df['k'], results_df['Davies-Bouldin'], 'g-o', linewidth=2)\n",
    "axes[0, 1].axvline(x=optimal_k_davies, color='red', linestyle='--', alpha=0.7)\n",
    "axes[0, 1].set_xlabel('Number of Clusters (k)')\n",
    "axes[0, 1].set_ylabel('Davies-Bouldin Index')\n",
    "axes[0, 1].set_title('Davies-Bouldin Index (Lower is Better)')\n",
    "axes[0, 1].grid(True, alpha=0.3)\n",
    "\n",
    "# Calinski-Harabasz (higher is better)\n",
    "axes[1, 0].plot(results_df['k'], results_df['Calinski-Harabasz'], 'm-o', linewidth=2)\n",
    "axes[1, 0].axvline(x=optimal_k_calinski, color='red', linestyle='--', alpha=0.7)\n",
    "axes[1, 0].set_xlabel('Number of Clusters (k)')\n",
    "axes[1, 0].set_ylabel('Calinski-Harabasz Score')\n",
    "axes[1, 0].set_title('Calinski-Harabasz Index (Higher is Better)')\n",
    "axes[1, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# Inertia (for reference)\n",
    "axes[1, 1].plot(results_df['k'], results_df['Inertia'], 'c-o', linewidth=2)\n",
    "axes[1, 1].set_xlabel('Number of Clusters (k)')\n",
    "axes[1, 1].set_ylabel('Inertia')\n",
    "axes[1, 1].set_title('Inertia (Lower is Better, but use Elbow)')\n",
    "axes[1, 1].grid(True, alpha=0.3)\n",
    "\n",
    "plt.suptitle('Clustering Evaluation Metrics', fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"BEST PRACTICES FOR UNSUPERVISED LEARNING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "1. DATA PREPARATION\n",
    "✓ Standardize/normalize features\n",
    "✓ Remove highly correlated features\n",
    "✓ Handle missing values\n",
    "✓ Outlier detection/removal appropriate\n",
    "✗ Don't use target variable\n",
    "\n",
    "2. ALGORITHM SELECTION\n",
    "✓ Start with simple methods (K-means)\n",
    "✓ Try multiple algorithms\n",
    "✓ Consider interpretability\n",
    "✗ Don't assume one algorithm works for all\n",
    "\n",
    "3. PARAMETER TUNING\n",
    "✓ Use multiple evaluation metrics\n",
    "✓ Cross-validate results\n",
    "✓ Try range of parameters\n",
    "✓ Visualization helps\n",
    "✗ Don't rely on single metric\n",
    "\n",
    "4. VALIDATION\n",
    "✓ Use internal validation metrics\n",
    "✓ Domain knowledge assessment\n",
    "✓ Stability across runs\n",
    "✓ Interpretability check\n",
    "✗ Don't over-interpret results\n",
    "\n",
    "5. COMMON PITFALLS\n",
    "✗ Expecting ground truth accuracy\n",
    "✗ Using only one evaluation metric\n",
    "✗ Not scaling features\n",
    "✗ Over-interpreting clusters\n",
    "✗ Ignoring outliers\n",
    "✗ Not validating with domain experts\n",
    "\n",
    "6. DIMENSIONALITY REDUCTION\n",
    "✓ Choose method based on goal\n",
    "✓ Retain sufficient variance (90-95%)\n",
    "✓ Interpret components\n",
    "✓ Validate on downstream task\n",
    "✗ Don't remove features blindly\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2acb92c9",
   "metadata": {},
   "source": [
    "## Week 10 Summary\n",
    "By completing Week 10, you have learned:\n",
    "- Unsupervised learning: discovering patterns without labels\n",
    "- Clustering vs dimensionality reduction vs anomaly detection\n",
    "- K-means clustering: algorithm, advantages, disadvantages\n",
    "- Elbow method and silhouette score for choosing k\n",
    "- Hierarchical clustering: agglomerative approach\n",
    "- Dendrograms and linkage methods\n",
    "- DBSCAN: density-based clustering with outlier detection\n",
    "- Choosing eps parameter and k-distance graphs\n",
    "- Clustering evaluation metrics without ground truth\n",
    "- Silhouette score, Davies-Bouldin, Calinski-Harabasz\n",
    "- Dimensionality reduction fundamentals\n",
    "- Principal Component Analysis (PCA)\n",
    "- Explained variance and scree plots\n",
    "- PCA component loadings and interpretation\n",
    "- Non-linear techniques: t-SNE and UMAP\n",
    "- When to use each dimensionality reduction method\n",
    "- Data scaling and preprocessing for clustering\n",
    "- Visualization of high-dimensional data\n",
    "- Common pitfalls in unsupervised learning\n",
    "- Best practices for clustering and reduction\n",
    "- Choosing appropriate algorithms\n",
    "- Validating unsupervised learning results\n",
    "- Domain expertise in interpretation\n",
    "- Handling outliers in clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "908693b6",
   "metadata": {},
   "source": [
    "## Week 10 Assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b9466d7",
   "metadata": {},
   "source": [
    "### Assignment 1: Comprehensive Clustering Analysis\n",
    "Perform complete clustering analysis:\n",
    "- Prepare dataset (scaling, preprocessing)\n",
    "- Implement K-means clustering\n",
    "- Use elbow method to find optimal k\n",
    "- Calculate silhouette scores\n",
    "- Compare with hierarchical clustering\n",
    "- Compare with DBSCAN\n",
    "- Visualize clusters\n",
    "- Evaluate quality with multiple metrics\n",
    "- Interpret and validate results\n",
    "- Create visualizations and report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7adfa27",
   "metadata": {},
   "source": [
    "### Assignment 2: Dimensionality Reduction Project\n",
    "Reduce dimensions using multiple methods:\n",
    "- Load high-dimensional dataset\n",
    "- Apply PCA with full components\n",
    "- Create scree plot and cumulative variance\n",
    "- Choose number of components\n",
    "- Interpret component loadings\n",
    "- Apply t-SNE for comparison\n",
    "- Create visualizations (2D and 3D if possible)\n",
    "- Compare PCA vs t-SNE\n",
    "- Evaluate quality for visualization\n",
    "- Use reduced features in clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d1f7e2d",
   "metadata": {},
   "source": [
    "### Assignment 3: Real-World Unsupervised Learning\n",
    "Apply unsupervised learning to real dataset:\n",
    "- Download or create real-world dataset\n",
    "- Exploratory analysis\n",
    "- Try multiple clustering algorithms\n",
    "- Find optimal parameters for each\n",
    "- Evaluate with multiple metrics\n",
    "- Perform dimensionality reduction\n",
    "- Create 2D visualization with clusters\n",
    "- Interpret findings with domain knowledge\n",
    "- Validate stability across runs\n",
    "- Write comprehensive report with insights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fa3ec61",
   "metadata": {},
   "source": [
    "## Recommended Practice Problems\n",
    "- Manually calculate distances in K-means step by step\n",
    "- Create dendrogram by hand for small dataset\n",
    "- Vary eps and min_samples in DBSCAN on moons dataset\n",
    "- Implement K-means from scratch\n",
    "- Calculate PCA components manually (numpy)\n",
    "- Compare clustering results before and after scaling\n",
    "- Use different linkage methods and compare dendrograms\n",
    "- Evaluate clustering on multiple synthetic datasets\n",
    "- Explain PCA loadings in terms of original features\n",
    "- Create visualizations for all three clustering methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa7aec36",
   "metadata": {},
   "source": [
    "## Additional Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ca02e4",
   "metadata": {},
   "source": [
    "### Books\n",
    "- Hands-On Machine Learning by Aurélien Géron (Chapter 9)\n",
    "- Introduction to Statistical Learning (Chapter 10)\n",
    "- Clustering Algorithms by John Hartigan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f560f6e8",
   "metadata": {},
   "source": [
    "### Online Resources\n",
    "- scikit-learn clustering: https://scikit-learn.org/stable/modules/clustering.html\n",
    "- PCA explained visually: https://setosa.io/ev/principal-component-analysis/\n",
    "- t-SNE visualization: https://distill.pub/2016/misread-tsne/\n",
    "- UMAP documentation: https://umap-learn.readthedocs.io/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}