{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a3f09f19",
   "metadata": {},
   "source": [
    "# Week 9: Machine Learning — Classification and Ensembles\n",
    "\n",
    "*Kmex Consult — Data Science Fundamentals Course*\n",
    "\n",
    "Work through each code cell in order. Edit and re-run to experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b9d45a",
   "metadata": {},
   "source": [
    "*Data Science Fundamentals Course*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a8ac8de",
   "metadata": {},
   "source": [
    "## Week 9 Overview\n",
    "Week 9 introduces machine learning - the bridge between statistical analysis and practical data science. You'll learn how to build, evaluate, and optimize predictive models for classification problems.\n",
    "\n",
    "Machine learning is used extensively in industry:\n",
    "- Email spam detection\n",
    "- Customer churn prediction\n",
    "- Fraud detection\n",
    "- Disease diagnosis\n",
    "- Image and text classification\n",
    "- Recommendation systems\n",
    "- Autonomous vehicles\n",
    "\n",
    "This week focuses on supervised learning where we have labeled training data. You'll learn:\n",
    "- Key machine learning concepts: training, validation, generalization\n",
    "- Decision trees and their interpretability\n",
    "- Ensemble methods that combine multiple models\n",
    "- How to evaluate and compare models rigorously\n",
    "- Hyperparameter tuning for optimal performance\n",
    "- Avoiding common pitfalls (overfitting, data leakage)\n",
    "\n",
    "By the end of Week 9, you will be able to:\n",
    "- Understand the machine learning workflow\n",
    "- Build decision tree classifiers\n",
    "- Create ensemble models (Random Forest, Gradient Boosting)\n",
    "- Evaluate models using appropriate metrics\n",
    "- Tune hyperparameters effectively\n",
    "- Interpret feature importance\n",
    "- Compare models and select the best one\n",
    "- Avoid common machine learning mistakes\n",
    "\n",
    "Week 9 is divided into three 2-hour sessions:\n",
    "- Session 1: Machine Learning Fundamentals and Decision Trees\n",
    "- Session 2: Ensemble Methods and Advanced Classifiers\n",
    "- Session 3: Model Evaluation, Tuning, and Best Practices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40e12ed",
   "metadata": {},
   "source": [
    "## SESSION 1: Machine Learning Fundamentals and Decision Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3798767",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f868dd6",
   "metadata": {},
   "source": [
    "### 1.1 Machine Learning Fundamentals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a80f5912",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"MACHINE LEARNING FUNDAMENTALS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "WHAT IS MACHINE LEARNING?\n",
    "- Algorithms that learn patterns from data\n",
    "- Improve performance with more data\n",
    "- Make predictions on new, unseen data\n",
    "\n",
    "TYPES OF MACHINE LEARNING:\n",
    "\n",
    "1. SUPERVISED LEARNING (labeled data)\n",
    "- Classification: Predict category (yes/no, A/B/C)\n",
    "- Regression: Predict continuous value (price, temperature)\n",
    "- Examples: spam detection, price prediction\n",
    "\n",
    "2. UNSUPERVISED LEARNING (unlabeled data)\n",
    "- Clustering: Group similar items\n",
    "- Dimensionality reduction: reduce features\n",
    "- Examples: customer segmentation, anomaly detection\n",
    "\n",
    "3. REINFORCEMENT LEARNING\n",
    "- Learn from feedback (rewards/penalties)\n",
    "- Examples: game playing, robotics\n",
    "\n",
    "THIS WEEK: SUPERVISED LEARNING - CLASSIFICATION\n",
    "\"\"\")\n",
    "\n",
    "# Machine Learning Workflow\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"MACHINE LEARNING WORKFLOW\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "1. PROBLEM DEFINITION\n",
    "- What are we predicting?\n",
    "- What data do we have?\n",
    "- What's success look like?\n",
    "\n",
    "2. DATA PREPARATION\n",
    "- Clean and preprocess data\n",
    "- Handle missing values\n",
    "- Feature engineering\n",
    "- Train-test split\n",
    "\n",
    "3. MODEL SELECTION\n",
    "- Choose appropriate algorithm\n",
    "- Consider interpretability vs accuracy\n",
    "\n",
    "4. TRAINING\n",
    "- Fit model to training data\n",
    "- Model learns patterns\n",
    "\n",
    "5. VALIDATION\n",
    "- Evaluate on validation set\n",
    "- Prevent overfitting\n",
    "\n",
    "6. HYPERPARAMETER TUNING\n",
    "- Optimize model parameters\n",
    "- Grid search or random search\n",
    "\n",
    "7. TESTING\n",
    "- Final evaluation on test set\n",
    "- Report performance\n",
    "\n",
    "8. DEPLOYMENT\n",
    "- Use model in production\n",
    "- Monitor performance over time\n",
    "\"\"\")\n",
    "\n",
    "# Key Concepts\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"KEY MACHINE LEARNING CONCEPTS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "BIAS-VARIANCE TRADEOFF:\n",
    "- HIGH BIAS: Model too simple, underfits (both train and test error high)\n",
    "- HIGH VARIANCE: Model too complex, overfits (train error low, test error high)\n",
    "- GOAL: Balance bias and variance for good generalization\n",
    "\n",
    "GENERALIZATION:\n",
    "- How well model performs on NEW, unseen data\n",
    "- More important than training accuracy\n",
    "- Achieved through proper validation and regularization\n",
    "\n",
    "DATA LEAKAGE:\n",
    "- Using information that shouldn't be available at prediction time\n",
    "- Common mistakes:\n",
    "- Including target variable in features\n",
    "- Using test set for preprocessing\n",
    "- Using future information\n",
    "- Prevention: Strict train-test separation\n",
    "\n",
    "CURSE OF DIMENSIONALITY:\n",
    "- More features can make learning harder\n",
    "- Need more data as dimensions increase\n",
    "- Feature selection/reduction important\n",
    "\n",
    "IMBALANCED DATA:\n",
    "- When classes have very different frequencies\n",
    "- Standard accuracy misleading\n",
    "- Use precision, recall, F1 instead\n",
    "\"\"\")\n",
    "\n",
    "# Create example dataset\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"EXAMPLE: IRIS FLOWER CLASSIFICATION\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature_names = iris.feature_names\n",
    "target_names = iris.target_names\n",
    "\n",
    "print(f\"\\nDataset size: {X.shape[0]} samples, {X.shape[1]} features\")\n",
    "print(f\"Classes: {target_names}\")\n",
    "print(f\"Features: {feature_names}\")\n",
    "\n",
    "# Create DataFrame\n",
    "df = pd.DataFrame(X, columns=feature_names)\n",
    "df['species'] = y\n",
    "df['species_name'] = df['species'].map({0: 'Setosa', 1: 'Versicolor', 2: 'Virginica'})\n",
    "\n",
    "print(\"\\nFirst few samples:\")\n",
    "print(df.head())\n",
    "\n",
    "# Train-test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "X, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "print(f\"\\nTrain-test split:\")\n",
    "print(f\"Training set: {X_train.shape[0]} samples ({X_train.shape[0]/len(X)*100:.0f}%)\")\n",
    "print(f\"Test set: {X_test.shape[0]} samples ({X_test.shape[0]/len(X)*100:.0f}%)\")\n",
    "\n",
    "# Check class balance\n",
    "print(f\"\\nClass distribution (training set):\")\n",
    "unique, counts = np.unique(y_train, return_counts=True)\n",
    "for target, count in zip(unique, counts):\n",
    "print(f\" {target_names[target]}: {count} ({count/len(y_train)*100:.0f}%)\")\n",
    "\n",
    "# Simple model\n",
    "model = DecisionTreeClassifier(max_depth=3, random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Predictions\n",
    "y_pred_train = model.predict(X_train)\n",
    "y_pred_test = model.predict(X_test)\n",
    "\n",
    "# Evaluation\n",
    "train_accuracy = accuracy_score(y_train, y_pred_train)\n",
    "test_accuracy = accuracy_score(y_test, y_pred_test)\n",
    "\n",
    "print(f\"\\nModel Performance:\")\n",
    "print(f\"Training accuracy: {train_accuracy:.4f}\")\n",
    "print(f\"Test accuracy: {test_accuracy:.4f}\")\n",
    "print(f\"Difference: {train_accuracy - test_accuracy:.4f} (larger = more overfitting)\")\n",
    "\n",
    "print(f\"\\nClassification Report (Test Set):\")\n",
    "print(classification_report(y_test, y_pred_test, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6246601",
   "metadata": {},
   "source": [
    "### 1.2 Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1dbb468",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"DECISION TREES\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW DECISION TREES WORK:\n",
    "- Recursively split data into smaller groups\n",
    "- Each split maximizes information gain\n",
    "- Creates tree-like structure of decision rules\n",
    "\n",
    "ADVANTAGES:\n",
    "- Highly interpretable (can visualize and explain)\n",
    "- No preprocessing needed\n",
    "- Works with non-linear relationships\n",
    "- Handles both classification and regression\n",
    "- Can handle missing values (in some implementations)\n",
    "\n",
    "DISADVANTAGES:\n",
    "- Prone to overfitting\n",
    "- Unstable (small data change → big tree change)\n",
    "- Can be biased with imbalanced data\n",
    "- Greedy algorithm (not globally optimal)\n",
    "\n",
    "WHEN TO USE:\n",
    "- When interpretability is important\n",
    "- Mixed data types\n",
    "- Non-linear relationships\n",
    "- Feature interactions matter\n",
    "\"\"\")\n",
    "\n",
    "# Load data\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature_names = iris.feature_names\n",
    "target_names = iris.target_names\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "X, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "# Build different depth trees\n",
    "print(\"\\nEXAMPLE: IRIS CLASSIFICATION WITH DIFFERENT DEPTHS\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "depths = [2, 3, 5, 10]\n",
    "results = []\n",
    "\n",
    "for depth in depths:\n",
    "model = DecisionTreeClassifier(max_depth=depth, random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "train_acc = model.score(X_train, y_train)\n",
    "test_acc = model.score(X_test, y_test)\n",
    "n_nodes = model.tree_.node_count\n",
    "\n",
    "results.append({\n",
    "'Depth': depth,\n",
    "'Train Accuracy': train_acc,\n",
    "'Test Accuracy': test_acc,\n",
    "'Nodes': n_nodes,\n",
    "'Gap': train_acc - test_acc\n",
    "})\n",
    "\n",
    "print(f\"\\nDepth {depth}:\")\n",
    "print(f\" Training accuracy: {train_acc:.4f}\")\n",
    "print(f\" Test accuracy: {test_acc:.4f}\")\n",
    "print(f\" Train-Test gap: {train_acc - test_acc:.4f}\")\n",
    "print(f\" Number of nodes: {n_nodes}\")\n",
    "\n",
    "if train_acc - test_acc > 0.05:\n",
    "print(f\" Status: OVERFITTING (high gap)\")\n",
    "else:\n",
    "print(f\" Status: GOOD GENERALIZATION\")\n",
    "\n",
    "# Build final model\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"FINAL TREE (Depth=3)\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "model = DecisionTreeClassifier(max_depth=3, random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"\\nAccuracy: {accuracy:.4f}\")\n",
    "print(f\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_pred, target_names=target_names))\n",
    "\n",
    "# Feature importance\n",
    "print(f\"\\nFeature Importance:\")\n",
    "for name, importance in zip(feature_names, model.feature_importances_):\n",
    "print(f\" {name}: {importance:.4f}\")\n",
    "\n",
    "# Tree visualization\n",
    "fig, ax = plt.subplots(figsize=(20, 10))\n",
    "plot_tree(model,\n",
    "feature_names=feature_names,\n",
    "class_names=target_names,\n",
    "filled=True,\n",
    "ax=ax,\n",
    "fontsize=10)\n",
    "plt.title('Decision Tree (Max Depth = 3)', fontsize=14, fontweight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Visualize depth effect\n",
    "import pandas as pd\n",
    "results_df = pd.DataFrame(results)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# Accuracy vs Depth\n",
    "axes[0].plot(results_df['Depth'], results_df['Train Accuracy'], 'b-o', label='Train', linewidth=2)\n",
    "axes[0].plot(results_df['Depth'], results_df['Test Accuracy'], 'r-o', label='Test', linewidth=2)\n",
    "axes[0].fill_between(results_df['Depth'], results_df['Train Accuracy'],\n",
    "results_df['Test Accuracy'], alpha=0.2, color='gray')\n",
    "axes[0].set_xlabel('Max Depth')\n",
    "axes[0].set_ylabel('Accuracy')\n",
    "axes[0].set_title('Impact of Tree Depth on Accuracy')\n",
    "axes[0].legend()\n",
    "axes[0].grid(True, alpha=0.3)\n",
    "\n",
    "# Overfitting gap\n",
    "axes[1].plot(results_df['Depth'], results_df['Gap'], 'g-o', linewidth=2, markersize=8)\n",
    "axes[1].set_xlabel('Max Depth')\n",
    "axes[1].set_ylabel('Train-Test Gap')\n",
    "axes[1].set_title('Overfitting Gap vs Depth')\n",
    "axes[1].grid(True, alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Hyperparameters affecting decision trees\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"DECISION TREE HYPERPARAMETERS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "KEY HYPERPARAMETERS:\n",
    "\n",
    "max_depth:\n",
    "- Maximum depth of tree\n",
    "- Lower → simpler model, less overfitting\n",
    "- Higher → more complex, better training fit\n",
    "\n",
    "min_samples_split:\n",
    "- Minimum samples to split a node\n",
    "- Higher → larger leaves, less overfitting\n",
    "- Lower → more splits allowed\n",
    "\n",
    "min_samples_leaf:\n",
    "- Minimum samples in leaf node\n",
    "- Prevents small, isolated leaves\n",
    "\n",
    "max_features:\n",
    "- Number of features to consider for split\n",
    "- Adds randomness, reduces overfitting\n",
    "- Often sqrt(n_features) or log(n_features)\n",
    "\n",
    "criterion:\n",
    "- 'gini': Uses Gini impurity\n",
    "- 'entropy': Uses information gain\n",
    "\n",
    "TIPS FOR PREVENTING OVERFITTING:\n",
    "- Set max_depth (typical: 5-15 for tabular data)\n",
    "- Increase min_samples_leaf\n",
    "- Decrease max_features\n",
    "- Use ensemble methods (combine multiple trees)\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88ade541",
   "metadata": {},
   "source": [
    "## SESSION 2: Ensemble Methods and Advanced Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69990ab",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a9b309",
   "metadata": {},
   "source": [
    "### 2.1 Random Forest and Ensemble Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "184a1795",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"ENSEMBLE METHODS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "ENSEMBLE LEARNING:\n",
    "- Combine predictions from multiple models\n",
    "- Often better than single model\n",
    "- \"Wisdom of crowds\" principle\n",
    "\n",
    "WHY ENSEMBLES WORK:\n",
    "- Different models make different mistakes\n",
    "- Combining reduces variance\n",
    "- Improves generalization\n",
    "\n",
    "TYPES OF ENSEMBLES:\n",
    "\n",
    "1. BAGGING (Bootstrap Aggregating)\n",
    "- Train models on random samples WITH replacement\n",
    "- Average predictions\n",
    "- Reduces variance, not bias\n",
    "- Example: Random Forest\n",
    "\n",
    "2. BOOSTING\n",
    "- Train models sequentially\n",
    "- Each model corrects previous errors\n",
    "- Reduces bias and variance\n",
    "- Examples: AdaBoost, Gradient Boosting\n",
    "\n",
    "3. STACKING\n",
    "- Train multiple models\n",
    "- Use another model to combine them\n",
    "\n",
    "4. VOTING\n",
    "- Average predictions from different algorithms\n",
    "\"\"\")\n",
    "\n",
    "# Load data\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature_names = iris.feature_names\n",
    "target_names = iris.target_names\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "X, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "# Random Forest\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"RANDOM FOREST\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW IT WORKS:\n",
    "1. Create multiple bootstrap samples of data\n",
    "2. Build decision tree on each sample\n",
    "3. Each tree sees random subset of features\n",
    "4. Average predictions (regression) or majority vote (classification)\n",
    "\n",
    "ADVANTAGES:\n",
    "- Reduces overfitting compared to single tree\n",
    "- Handles non-linear relationships\n",
    "- Feature importance\n",
    "- Works with large datasets\n",
    "- Fast training and prediction\n",
    "- No preprocessing needed\n",
    "\n",
    "DISADVANTAGES:\n",
    "- Less interpretable than single tree\n",
    "- Can be slow with huge datasets\n",
    "- Memory intensive\n",
    "- Biased with highly imbalanced data\n",
    "\"\"\")\n",
    "\n",
    "# Train Random Forest\n",
    "model_rf = RandomForestClassifier(\n",
    "n_estimators=100,\n",
    "max_depth=5,\n",
    "min_samples_leaf=2,\n",
    "random_state=42,\n",
    "n_jobs=-1 # Use all processors\n",
    ")\n",
    "\n",
    "model_rf.fit(X_train, y_train)\n",
    "\n",
    "# Predictions\n",
    "y_pred_rf = model_rf.predict(X_test)\n",
    "acc_rf = accuracy_score(y_test, y_pred_rf)\n",
    "\n",
    "print(f\"\\nRandom Forest Results:\")\n",
    "print(f\"Number of trees: {model_rf.n_estimators}\")\n",
    "print(f\"Max depth: {model_rf.max_depth}\")\n",
    "print(f\"Test accuracy: {acc_rf:.4f}\")\n",
    "\n",
    "print(f\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_pred_rf, target_names=target_names))\n",
    "\n",
    "# Feature importance\n",
    "print(f\"\\nFeature Importance (Random Forest):\")\n",
    "importance_rf = pd.DataFrame({\n",
    "'Feature': feature_names,\n",
    "'Importance': model_rf.feature_importances_\n",
    "}).sort_values('Importance', ascending=False)\n",
    "\n",
    "print(importance_rf.to_string(index=False))\n",
    "\n",
    "# Gradient Boosting\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"GRADIENT BOOSTING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW IT WORKS:\n",
    "1. Start with simple model\n",
    "2. Train next model on residuals (errors)\n",
    "3. Each model corrects previous errors\n",
    "4. Combine additively\n",
    "\n",
    "ADVANTAGES:\n",
    "- Often highest accuracy\n",
    "- Handles complex patterns\n",
    "- Feature importance\n",
    "- Works with various data types\n",
    "- Good with imbalanced data\n",
    "\n",
    "DISADVANTAGES:\n",
    "- Slow training (sequential)\n",
    "- Prone to overfitting if not tuned\n",
    "- Many hyperparameters to tune\n",
    "- Less interpretable\n",
    "\n",
    "KEY HYPERPARAMETERS:\n",
    "- learning_rate: Step size (lower = slower but better)\n",
    "- n_estimators: Number of boosting rounds\n",
    "- max_depth: Depth of trees\n",
    "\"\"\")\n",
    "\n",
    "# Train Gradient Boosting\n",
    "model_gb = GradientBoostingClassifier(\n",
    "n_estimators=100,\n",
    "learning_rate=0.1,\n",
    "max_depth=3,\n",
    "random_state=42\n",
    ")\n",
    "\n",
    "model_gb.fit(X_train, y_train)\n",
    "\n",
    "# Predictions\n",
    "y_pred_gb = model_gb.predict(X_test)\n",
    "acc_gb = accuracy_score(y_test, y_pred_gb)\n",
    "\n",
    "print(f\"\\nGradient Boosting Results:\")\n",
    "print(f\"Number of estimators: {model_gb.n_estimators}\")\n",
    "print(f\"Learning rate: {model_gb.learning_rate}\")\n",
    "print(f\"Test accuracy: {acc_gb:.4f}\")\n",
    "\n",
    "# Feature importance\n",
    "print(f\"\\nFeature Importance (Gradient Boosting):\")\n",
    "importance_gb = pd.DataFrame({\n",
    "'Feature': feature_names,\n",
    "'Importance': model_gb.feature_importances_\n",
    "}).sort_values('Importance', ascending=False)\n",
    "\n",
    "print(importance_gb.to_string(index=False))\n",
    "\n",
    "# Model comparison\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"MODEL COMPARISON\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "comparison = pd.DataFrame({\n",
    "'Model': ['Random Forest', 'Gradient Boosting'],\n",
    "'Accuracy': [acc_rf, acc_gb],\n",
    "'Interpretability': ['Medium', 'Low'],\n",
    "'Speed': ['Fast', 'Slow'],\n",
    "'Memory': ['High', 'Medium']\n",
    "})\n",
    "\n",
    "print(comparison.to_string(index=False))\n",
    "\n",
    "# Visualization: Feature Importance\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# Random Forest\n",
    "axes[0].barh(importance_rf['Feature'], importance_rf['Importance'], color='steelblue')\n",
    "axes[0].set_xlabel('Importance')\n",
    "axes[0].set_title('Feature Importance - Random Forest', fontweight='bold')\n",
    "axes[0].grid(True, alpha=0.3, axis='x')\n",
    "\n",
    "# Gradient Boosting\n",
    "axes[1].barh(importance_gb['Feature'], importance_gb['Importance'], color='coral')\n",
    "axes[1].set_xlabel('Importance')\n",
    "axes[1].set_title('Feature Importance - Gradient Boosting', fontweight='bold')\n",
    "axes[1].grid(True, alpha=0.3, axis='x')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eae9cda",
   "metadata": {},
   "source": [
    "### 2.2 Advanced Classification Techniques"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70a4784d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "import pandas as pd\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"OTHER CLASSIFICATION ALGORITHMS\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "# Load data\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "target_names = iris.target_names\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "X, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "# 1. K-NEAREST NEIGHBORS (KNN)\n",
    "print(\"\\n1. K-NEAREST NEIGHBORS (KNN)\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW IT WORKS:\n",
    "- For prediction, look at k nearest neighbors\n",
    "- Majority vote for classification\n",
    "- Distance usually Euclidean\n",
    "\n",
    "ADVANTAGES:\n",
    "- Simple and intuitive\n",
    "- No training phase\n",
    "- Works with non-linear data\n",
    "\n",
    "DISADVANTAGES:\n",
    "- Slow prediction (compares to all training points)\n",
    "- Sensitive to distance metric\n",
    "- Needs feature scaling\n",
    "- Sensitive to k value\n",
    "- Poor with high dimensions\n",
    "\"\"\")\n",
    "\n",
    "model_knn = KNeighborsClassifier(n_neighbors=5)\n",
    "model_knn.fit(X_train, y_train)\n",
    "acc_knn = model_knn.score(X_test, y_test)\n",
    "\n",
    "print(f\"\\nKNN Accuracy (k=5): {acc_knn:.4f}\")\n",
    "\n",
    "# 2. SUPPORT VECTOR MACHINE (SVM)\n",
    "print(\"\\n2. SUPPORT VECTOR MACHINE (SVM)\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW IT WORKS:\n",
    "- Finds optimal hyperplane separating classes\n",
    "- Maximizes margin between classes\n",
    "- Can handle non-linear with kernel trick\n",
    "\n",
    "ADVANTAGES:\n",
    "- Effective in high dimensions\n",
    "- Memory efficient (uses subset of points)\n",
    "- Various kernel options\n",
    "- Works well with binary classification\n",
    "\n",
    "DISADVANTAGES:\n",
    "- Slow training with large datasets\n",
    "- Hard to interpret\n",
    "- Need feature scaling\n",
    "- Hyperparameter tuning critical\n",
    "\"\"\")\n",
    "\n",
    "model_svm = SVC(kernel='rbf', C=1.0, gamma='scale')\n",
    "model_svm.fit(X_train, y_train)\n",
    "acc_svm = model_svm.score(X_test, y_test)\n",
    "\n",
    "print(f\"\\nSVM Accuracy: {acc_svm:.4f}\")\n",
    "\n",
    "# 3. NAIVE BAYES\n",
    "print(\"\\n3. NAIVE BAYES\")\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HOW IT WORKS:\n",
    "- Uses Bayes' theorem: P(Y|X) = P(X|Y)P(Y) / P(X)\n",
    "- Assumes features are independent (naive)\n",
    "- Fast to train and predict\n",
    "\n",
    "ADVANTAGES:\n",
    "- Fast\n",
    "- Works with small datasets\n",
    "- Handles high dimensions well\n",
    "- Good baseline\n",
    "\n",
    "DISADVANTAGES:\n",
    "- Independence assumption often violated\n",
    "- Less accurate than complex models\n",
    "- Not best for high correlations between features\n",
    "\"\"\")\n",
    "\n",
    "model_nb = GaussianNB()\n",
    "model_nb.fit(X_train, y_train)\n",
    "acc_nb = model_nb.score(X_test, y_test)\n",
    "\n",
    "print(f\"\\nNaive Bayes Accuracy: {acc_nb:.4f}\")\n",
    "\n",
    "# Voting Classifier (Ensemble of different algorithms)\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"VOTING CLASSIFIER (Ensemble of Different Algorithms)\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "COMBINE DIFFERENT ALGORITHMS:\n",
    "- Use multiple different models\n",
    "- Vote on final prediction\n",
    "- Combines strengths of different algorithms\n",
    "\"\"\")\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "voting_clf = VotingClassifier(\n",
    "estimators=[\n",
    "('dt', DecisionTreeClassifier(max_depth=5, random_state=42)),\n",
    "('rf', RandomForestClassifier(n_estimators=50, max_depth=5, random_state=42)),\n",
    "('knn', KNeighborsClassifier(n_neighbors=5)),\n",
    "('svm', SVC(kernel='rbf', C=1.0, gamma='scale'))\n",
    "],\n",
    "voting='hard' # hard voting (majority vote)\n",
    ")\n",
    "\n",
    "voting_clf.fit(X_train, y_train)\n",
    "acc_voting = voting_clf.score(X_test, y_test)\n",
    "\n",
    "print(f\"\\nVoting Classifier Accuracy: {acc_voting:.4f}\")\n",
    "\n",
    "# Summary comparison\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"ALGORITHM COMPARISON\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "comparison = pd.DataFrame({\n",
    "'Algorithm': ['KNN', 'SVM', 'Naive Bayes', 'Voting Ensemble'],\n",
    "'Accuracy': [acc_knn, acc_svm, acc_nb, acc_voting],\n",
    "'Training Speed': ['Fast', 'Slow', 'Fast', 'Medium'],\n",
    "'Interpretability': ['High', 'Low', 'High', 'Low'],\n",
    "'Scalability': ['Poor', 'Medium', 'Good', 'Medium']\n",
    "})\n",
    "\n",
    "print(comparison.to_string(index=False))\n",
    "\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"CHOOSING AN ALGORITHM\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "DECISION GUIDE:\n",
    "\n",
    "START HERE:\n",
    "→ Is interpretability critical?\n",
    "YES: Use Decision Tree or Logistic Regression\n",
    "NO: Use ensemble methods (RF, GB)\n",
    "\n",
    "→ Large dataset (>100K rows)?\n",
    "YES: Use Gradient Boosting or Linear model\n",
    "NO: Most algorithms work\n",
    "\n",
    "→ High-dimensional data (>100 features)?\n",
    "YES: Use SVM or regularized linear model\n",
    "NO: Most algorithms work\n",
    "\n",
    "→ Categorical features?\n",
    "YES: Tree-based methods (no preprocessing)\n",
    "NO: Any algorithm\n",
    "\n",
    "→ Imbalanced classes?\n",
    "YES: Use weighted algorithms or SMOTE\n",
    "NO: Any algorithm\n",
    "\n",
    "GENERAL RULE:\n",
    "- Start simple (Logistic Regression, Decision Tree)\n",
    "- Ensemble if needed (Random Forest)\n",
    "- Advanced tuning if required (Gradient Boosting)\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4c3afa3",
   "metadata": {},
   "source": [
    "## SESSION 3: Model Evaluation, Tuning, and Best Practices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e27c67",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d3791a",
   "metadata": {},
   "source": [
    "### 3.1 Comprehensive Model Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1b47256",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import (accuracy_score, precision_score, recall_score,\n",
    "f1_score, roc_auc_score, roc_curve, auc,\n",
    "confusion_matrix, classification_report)\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"COMPREHENSIVE MODEL EVALUATION\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "# Load data\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "target_names = iris.target_names\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "X, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "# Train model\n",
    "model = RandomForestClassifier(n_estimators=100, random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "print(\"\"\"\n",
    "METRICS FOR CLASSIFICATION:\n",
    "\n",
    "ACCURACY:\n",
    "- Correct predictions / Total predictions\n",
    "- Good overall measure\n",
    "- Can be misleading with imbalanced data\n",
    "- Formula: (TP + TN) / (TP + TN + FP + FN)\n",
    "\n",
    "PRECISION:\n",
    "- Positive predictions that are correct\n",
    "- \"Of predicted positive, how many correct?\"\n",
    "- Important when false positives costly\n",
    "- Formula: TP / (TP + FP)\n",
    "\n",
    "RECALL (Sensitivity):\n",
    "- Actual positives that are detected\n",
    "- \"Of actual positive, how many found?\"\n",
    "- Important when false negatives costly\n",
    "- Formula: TP / (TP + FN)\n",
    "\n",
    "F1-SCORE:\n",
    "- Harmonic mean of precision and recall\n",
    "- Single number balancing both\n",
    "- Better than accuracy for imbalanced data\n",
    "- Formula: 2 × (Precision × Recall) / (Precision + Recall)\n",
    "\n",
    "SPECIFICITY:\n",
    "- True negatives properly identified\n",
    "- Formula: TN / (TN + FP)\n",
    "\n",
    "ROC-AUC:\n",
    "- Receiver Operating Characteristic\n",
    "- Plots TPR vs FPR at different thresholds\n",
    "- AUC = area under curve\n",
    "- 1.0 = perfect, 0.5 = random, <0.5 = worse than random\n",
    "\"\"\")\n",
    "\n",
    "# Binary classification example\n",
    "# Convert to binary (Virginica vs others)\n",
    "y_binary = (y == 2).astype(int)\n",
    "X_train_b, X_test_b, y_train_b, y_test_b = train_test_split(\n",
    "X, y_binary, test_size=0.2, random_state=42, stratify=y_binary\n",
    ")\n",
    "\n",
    "model_b = RandomForestClassifier(n_estimators=100, random_state=42)\n",
    "model_b.fit(X_train_b, y_train_b)\n",
    "y_pred_b = model_b.predict(X_test_b)\n",
    "y_proba_b = model_b.predict_proba(X_test_b)[:, 1]\n",
    "\n",
    "# Metrics\n",
    "accuracy = accuracy_score(y_test_b, y_pred_b)\n",
    "precision = precision_score(y_test_b, y_pred_b)\n",
    "recall = recall_score(y_test_b, y_pred_b)\n",
    "f1 = f1_score(y_test_b, y_pred_b)\n",
    "auc_score = roc_auc_score(y_test_b, y_proba_b)\n",
    "\n",
    "print(f\"\\nBINARY CLASSIFICATION METRICS (Virginica vs Others):\")\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Precision: {precision:.4f}\")\n",
    "print(f\"Recall: {recall:.4f}\")\n",
    "print(f\"F1-Score: {f1:.4f}\")\n",
    "print(f\"AUC-ROC: {auc_score:.4f}\")\n",
    "\n",
    "print(f\"\\nConfusion Matrix:\")\n",
    "cm = confusion_matrix(y_test_b, y_pred_b)\n",
    "print(f\"True Negatives: {cm[0,0]}\")\n",
    "print(f\"False Positives: {cm[0,1]}\")\n",
    "print(f\"False Negatives: {cm[1,0]}\")\n",
    "print(f\"True Positives: {cm[1,1]}\")\n",
    "\n",
    "# Cross-validation\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"CROSS-VALIDATION\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "MULTIPLE TRAIN-TEST SPLITS:\n",
    "- k-fold: Divide into k groups\n",
    "- Train k models, average scores\n",
    "- Better estimate of generalization\n",
    "- More robust than single split\n",
    "\n",
    "STRATIFIED K-FOLD:\n",
    "- Maintains class distribution in each fold\n",
    "- Important for imbalanced data\n",
    "- Better than regular k-fold\n",
    "\"\"\")\n",
    "\n",
    "# Stratified K-Fold\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",
    "cv_scores = cross_val_score(model, X, y, cv=skf, scoring='accuracy')\n",
    "\n",
    "print(f\"\\nStratified 5-Fold Cross-Validation:\")\n",
    "print(f\"Fold scores: {cv_scores}\")\n",
    "print(f\"Mean: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})\")\n",
    "\n",
    "# Visualization\n",
    "fig, axes = plt.subplots(2, 2, figsize=(14, 10))\n",
    "\n",
    "# 1. Confusion Matrix Heatmap\n",
    "import seaborn as sns\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=axes[0, 0],\n",
    "xticklabels=['Not Virginica', 'Virginica'],\n",
    "yticklabels=['Not Virginica', 'Virginica'])\n",
    "axes[0, 0].set_ylabel('Actual')\n",
    "axes[0, 0].set_xlabel('Predicted')\n",
    "axes[0, 0].set_title('Confusion Matrix')\n",
    "\n",
    "# 2. ROC Curve\n",
    "fpr, tpr, _ = roc_curve(y_test_b, y_proba_b)\n",
    "axes[0, 1].plot(fpr, tpr, linewidth=2, label=f'AUC = {auc_score:.3f}')\n",
    "axes[0, 1].plot([0, 1], [0, 1], 'r--', label='Random')\n",
    "axes[0, 1].set_xlabel('False Positive Rate')\n",
    "axes[0, 1].set_ylabel('True Positive Rate')\n",
    "axes[0, 1].set_title('ROC Curve')\n",
    "axes[0, 1].legend()\n",
    "axes[0, 1].grid(True, alpha=0.3)\n",
    "\n",
    "# 3. Cross-validation scores\n",
    "axes[1, 0].bar(range(1, 6), cv_scores, color='steelblue', alpha=0.7)\n",
    "axes[1, 0].axhline(y=cv_scores.mean(), color='r', linestyle='--', linewidth=2)\n",
    "axes[1, 0].set_xlabel('Fold')\n",
    "axes[1, 0].set_ylabel('Accuracy')\n",
    "axes[1, 0].set_title('5-Fold Cross-Validation Scores')\n",
    "axes[1, 0].set_ylim([0.8, 1.0])\n",
    "axes[1, 0].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# 4. Metrics comparison\n",
    "metrics = pd.DataFrame({\n",
    "'Metric': ['Accuracy', 'Precision', 'Recall', 'F1'],\n",
    "'Score': [accuracy, precision, recall, f1]\n",
    "})\n",
    "\n",
    "axes[1, 1].bar(metrics['Metric'], metrics['Score'], color=['blue', 'green', 'orange', 'red'], alpha=0.7)\n",
    "axes[1, 1].set_ylim([0, 1])\n",
    "axes[1, 1].set_ylabel('Score')\n",
    "axes[1, 1].set_title('Classification Metrics')\n",
    "axes[1, 1].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "# Add value labels on bars\n",
    "for i, (metric, score) in enumerate(zip(metrics['Metric'], metrics['Score'])):\n",
    "axes[1, 1].text(i, score + 0.02, f'{score:.3f}', ha='center', va='bottom')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e7715da",
   "metadata": {},
   "source": [
    "### 3.2 Hyperparameter Tuning and Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4697c882",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import (train_test_split, GridSearchCV,\n",
    "RandomizedSearchCV, cross_val_score)\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"HYPERPARAMETER TUNING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "HYPERPARAMETERS:\n",
    "- Parameters we SET before training\n",
    "- Different from model parameters (learned during training)\n",
    "- Critical for model performance\n",
    "- Need tuning to find optimal values\n",
    "\n",
    "TUNING METHODS:\n",
    "\n",
    "1. GRID SEARCH\n",
    "- Try all combinations in parameter grid\n",
    "- Exhaustive search\n",
    "- Works with small grids\n",
    "- Slow but thorough\n",
    "\n",
    "2. RANDOM SEARCH\n",
    "- Random sample from parameter space\n",
    "- Works with large grids\n",
    "- Faster than grid search\n",
    "- Sometimes finds better solutions\n",
    "\n",
    "3. BAYESIAN OPTIMIZATION\n",
    "- Uses probability model\n",
    "- Smart sampling\n",
    "- Efficient\n",
    "- More complex to implement\n",
    "\"\"\")\n",
    "\n",
    "# Load data\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "X, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "# Grid Search\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"GRID SEARCH EXAMPLE\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "param_grid = {\n",
    "'n_estimators': [50, 100, 200],\n",
    "'max_depth': [5, 10, 15, None],\n",
    "'min_samples_split': [2, 5, 10],\n",
    "'min_samples_leaf': [1, 2, 4]\n",
    "}\n",
    "\n",
    "print(f\"\\nParameter grid:\")\n",
    "for param, values in param_grid.items():\n",
    "print(f\" {param}: {values}\")\n",
    "\n",
    "total_combinations = 1\n",
    "for values in param_grid.values():\n",
    "total_combinations *= len(values)\n",
    "\n",
    "print(f\"\\nTotal combinations to try: {total_combinations}\")\n",
    "\n",
    "# Run grid search (can be slow, so we'll do a simplified version)\n",
    "print(\"\\nRunning Grid Search...\")\n",
    "\n",
    "grid_search = GridSearchCV(\n",
    "RandomForestClassifier(random_state=42),\n",
    "param_grid,\n",
    "cv=5, # 5-fold cross-validation\n",
    "scoring='accuracy',\n",
    "n_jobs=-1, # Use all processors\n",
    "verbose=0\n",
    ")\n",
    "\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print(f\"\\nBest parameters: {grid_search.best_params_}\")\n",
    "print(f\"Best CV score: {grid_search.best_score_:.4f}\")\n",
    "\n",
    "# Test performance\n",
    "y_pred = grid_search.predict(X_test)\n",
    "test_accuracy = accuracy_score(y_test, y_pred)\n",
    "print(f\"Test set accuracy: {test_accuracy:.4f}\")\n",
    "\n",
    "# Results summary\n",
    "results_df = pd.DataFrame(grid_search.cv_results_)\n",
    "results_summary = results_df[['param_n_estimators', 'param_max_depth',\n",
    "'param_min_samples_split', 'mean_test_score']].head(10)\n",
    "\n",
    "print(f\"\\nTop 10 parameter combinations:\")\n",
    "results_sorted = results_df.sort_values('mean_test_score', ascending=False).head(10)\n",
    "print(results_sorted[['param_n_estimators', 'param_max_depth',\n",
    "'param_min_samples_split', 'mean_test_score']].to_string(index=False))\n",
    "\n",
    "# Random Search\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"RANDOM SEARCH\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "RANDOM SEARCH:\n",
    "- Faster alternative to grid search\n",
    "- Sample parameter combinations randomly\n",
    "- Good for exploring large search spaces\n",
    "\"\"\")\n",
    "\n",
    "param_dist = {\n",
    "'n_estimators': [50, 100, 150, 200, 250],\n",
    "'max_depth': list(range(3, 20)),\n",
    "'min_samples_split': [2, 5, 10],\n",
    "'min_samples_leaf': [1, 2, 4]\n",
    "}\n",
    "\n",
    "random_search = RandomizedSearchCV(\n",
    "RandomForestClassifier(random_state=42),\n",
    "param_dist,\n",
    "n_iter=20, # Try 20 random combinations\n",
    "cv=5,\n",
    "scoring='accuracy',\n",
    "n_jobs=-1,\n",
    "random_state=42\n",
    ")\n",
    "\n",
    "random_search.fit(X_train, y_train)\n",
    "\n",
    "print(f\"\\nBest parameters: {random_search.best_params_}\")\n",
    "print(f\"Best CV score: {random_search.best_score_:.4f}\")\n",
    "\n",
    "y_pred_rs = random_search.predict(X_test)\n",
    "test_accuracy_rs = accuracy_score(y_test, y_pred_rs)\n",
    "print(f\"Test set accuracy: {test_accuracy_rs:.4f}\")\n",
    "\n",
    "# Learning curves\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"LEARNING CURVES\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "SHOWS:\n",
    "- Training accuracy vs validation accuracy\n",
    "- Effect of training set size\n",
    "- Detect underfitting/overfitting\n",
    "- Whether more data helps\n",
    "\"\"\")\n",
    "\n",
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "train_sizes, train_scores, val_scores = learning_curve(\n",
    "RandomForestClassifier(n_estimators=100, random_state=42),\n",
    "X_train, y_train,\n",
    "cv=5,\n",
    "train_sizes=np.linspace(0.1, 1.0, 10),\n",
    "scoring='accuracy',\n",
    "n_jobs=-1\n",
    ")\n",
    "\n",
    "train_mean = train_scores.mean(axis=1)\n",
    "train_std = train_scores.std(axis=1)\n",
    "val_mean = val_scores.mean(axis=1)\n",
    "val_std = val_scores.std(axis=1)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# Learning curves\n",
    "axes[0].plot(train_sizes, train_mean, 'b-o', label='Training', linewidth=2)\n",
    "axes[0].fill_between(train_sizes, train_mean - train_std,\n",
    "train_mean + train_std, alpha=0.2, color='blue')\n",
    "axes[0].plot(train_sizes, val_mean, 'r-o', label='Validation', linewidth=2)\n",
    "axes[0].fill_between(train_sizes, val_mean - val_std,\n",
    "val_mean + val_std, alpha=0.2, color='red')\n",
    "axes[0].set_xlabel('Training Set Size')\n",
    "axes[0].set_ylabel('Accuracy')\n",
    "axes[0].set_title('Learning Curves')\n",
    "axes[0].legend()\n",
    "axes[0].grid(True, alpha=0.3)\n",
    "\n",
    "# Parameter importance\n",
    "results_sorted_gs = results_df.sort_values('mean_test_score', ascending=False).head(20)\n",
    "param_counts = {}\n",
    "for param in ['n_estimators', 'max_depth', 'min_samples_split', 'min_samples_leaf']:\n",
    "param_name = f'param_{param}'\n",
    "if param_name in results_sorted_gs.columns:\n",
    "values = results_sorted_gs[param_name].value_counts()\n",
    "param_counts[param] = len(values)\n",
    "\n",
    "axes[1].bar(param_counts.keys(), param_counts.values(), color='steelblue', alpha=0.7)\n",
    "axes[1].set_ylabel('Unique Values in Top 20')\n",
    "axes[1].set_title('Parameter Diversity in Best Models')\n",
    "axes[1].grid(True, alpha=0.3, axis='y')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dca66975",
   "metadata": {},
   "source": [
    "### 3.3 Best Practices and Common Pitfalls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10b25cb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "print(\"=\"*50)\n",
    "print(\"MACHINE LEARNING BEST PRACTICES\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "print(\"\"\"\n",
    "1. DATA QUALITY IS CRITICAL\n",
    "✓ Clean and validate data\n",
    "✓ Handle missing values appropriately\n",
    "✓ Check for outliers\n",
    "✓ Ensure data consistency\n",
    "✗ Never skip EDA (exploratory data analysis)\n",
    "✗ Don't ignore data quality issues\n",
    "\n",
    "2. PROPER TRAIN-TEST SPLIT\n",
    "✓ Always split BEFORE preprocessing\n",
    "✓ Use stratified split for imbalanced data\n",
    "✓ Test set should never influence training\n",
    "✓ Use cross-validation for robust estimates\n",
    "✗ Don't fit preprocessor on entire dataset\n",
    "✗ Don't use test set for feature selection\n",
    "✗ Don't tune hyperparameters on test set\n",
    "\n",
    "3. FEATURE ENGINEERING\n",
    "✓ Create meaningful features\n",
    "✓ Domain knowledge helps\n",
    "✓ Test feature importance\n",
    "✓ Start simple, add complexity gradually\n",
    "✗ Don't create too many features (curse of dimensionality)\n",
    "✗ Don't use domain-specific features that won't generalize\n",
    "✗ Don't include future information (data leakage)\n",
    "\n",
    "4. MODEL SELECTION\n",
    "✓ Start with simple models (baseline)\n",
    "✓ Compare multiple algorithms\n",
    "✓ Use appropriate metrics (not just accuracy)\n",
    "✓ Consider interpretability vs accuracy tradeoff\n",
    "✗ Don't jump to complex models immediately\n",
    "✗ Don't use same metric for all problems\n",
    "✗ Don't ignore class imbalance\n",
    "\n",
    "5. EVALUATION AND VALIDATION\n",
    "✓ Use multiple metrics\n",
    "✓ Report confidence intervals\n",
    "✓ Check model assumptions\n",
    "✓ Validate on held-out test set only\n",
    "✗ Don't report only training accuracy\n",
    "✗ Don't cherry-pick best metric\n",
    "✗ Don't evaluate on training data\n",
    "\n",
    "6. COMMON PITFALLS TO AVOID\n",
    "\n",
    "DATA LEAKAGE:\n",
    "- Using information that won't be available at prediction time\n",
    "- Example: Including future values, target-derived features\n",
    "- Prevention: Careful feature engineering, strict train-test separation\n",
    "\n",
    "CLASS IMBALANCE:\n",
    "- One class much more frequent than others\n",
    "- Accuracy misleading (high accuracy, poor minority class detection)\n",
    "- Solutions: Stratified split, class weights, SMOTE, different metrics\n",
    "\n",
    "OVERFITTING:\n",
    "- Model learns training data too well\n",
    "- Poor generalization to new data\n",
    "- Signs: High train accuracy, low test accuracy\n",
    "- Prevention: Regularization, cross-validation, more data\n",
    "\n",
    "UNDERFITTING:\n",
    "- Model too simple to capture relationships\n",
    "- Both train and test accuracy low\n",
    "- Solution: More complex model, feature engineering\n",
    "\n",
    "MULTIPLE TESTING:\n",
    "- Testing many hypotheses → some false positives\n",
    "- Solution: Pre-register analyses, adjust significance levels\n",
    "\n",
    "7. REPRODUCIBILITY\n",
    "✓ Set random seeds\n",
    "✓ Document preprocessing steps\n",
    "✓ Version control code and data\n",
    "✓ Save trained models for later\n",
    "✗ Don't rely on randomness without setting seeds\n",
    "✗ Don't forget to document assumptions\n",
    "\n",
    "8. DEPLOYMENT CONSIDERATIONS\n",
    "✓ Monitor model performance over time\n",
    "✓ Handle data drift\n",
    "✓ Plan for retraining\n",
    "✓ Document decision rules\n",
    "✓ Consider fairness and bias\n",
    "✗ Don't assume model stays accurate forever\n",
    "✗ Don't deploy without testing in production environment\n",
    "✗ Don't ignore ethical implications\n",
    "\n",
    "9. MODEL INTERPRETATION\n",
    "✓ Explain predictions when possible\n",
    "✓ Use feature importance\n",
    "✓ Check for spurious correlations\n",
    "✓ Domain validation of findings\n",
    "✗ Don't trust black-box models blindly\n",
    "✗ Don't assume correlation implies causation\n",
    "✗ Don't ignore feature interactions\n",
    "\n",
    "10. WORKFLOW CHECKLIST\n",
    "\n",
    "PROBLEM DEFINITION:\n",
    "□ Define success metric clearly\n",
    "□ Understand business context\n",
    "□ Identify constraints and requirements\n",
    "\n",
    "DATA PREPARATION:\n",
    "□ Exploratory data analysis\n",
    "□ Handle missing values\n",
    "□ Check for outliers\n",
    "□ Feature engineering\n",
    "□ Data validation\n",
    "\n",
    "MODEL DEVELOPMENT:\n",
    "□ Baseline model\n",
    "□ Multiple algorithms\n",
    "□ Hyperparameter tuning\n",
    "□ Cross-validation\n",
    "□ Feature importance analysis\n",
    "\n",
    "EVALUATION:\n",
    "□ Multiple metrics\n",
    "□ Confidence intervals\n",
    "□ Error analysis\n",
    "□ Assumptions check\n",
    "□ Business impact assessment\n",
    "\n",
    "DEPLOYMENT:\n",
    "□ Final model selection\n",
    "□ Documentation\n",
    "□ Testing in production environment\n",
    "□ Monitoring plan\n",
    "□ Retraining schedule\n",
    "\"\"\")\n",
    "\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"PERFORMANCE TROUBLESHOOTING GUIDE\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "troubleshooting = {\n",
    "'Problem': [\n",
    "'High training accuracy, low test accuracy',\n",
    "'Both training and test accuracy low',\n",
    "'Class imbalance issues',\n",
    "'Inconsistent cross-validation scores',\n",
    "'Model too slow for production'\n",
    "],\n",
    "'Likely Cause': [\n",
    "'Overfitting',\n",
    "'Underfitting',\n",
    "'Class imbalance, inappropriate metric',\n",
    "'High variance, unstable model',\n",
    "'Too complex model, large data'\n",
    "],\n",
    "'Solution': [\n",
    "'Regularization, simpler model, more data',\n",
    "'More complex model, feature engineering',\n",
    "'Use stratified split, class weights, F1/ROC',\n",
    "'Cross-validation, ensemble methods',\n",
    "'Simpler model, feature selection, sampling'\n",
    "]\n",
    "}\n",
    "\n",
    "df_trouble = pd.DataFrame(troubleshooting)\n",
    "print(df_trouble.to_string(index=False))\n",
    "\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"RESOURCES FOR CONTINUED LEARNING\")\n",
    "print(\"=\"*50)\n",
    "\n",
    "resources = {\n",
    "'Topic': [\n",
    "'Deep Learning',\n",
    "'Unsupervised Learning',\n",
    "'Time Series',\n",
    "'NLP',\n",
    "'Computer Vision',\n",
    "'Reinforcement Learning'\n",
    "],\n",
    "'When to Learn': [\n",
    "'After mastering classical ML',\n",
    "'For clustering and dimensionality reduction',\n",
    "'For forecasting problems',\n",
    "'For text data',\n",
    "'For image data',\n",
    "'For sequential decision problems'\n",
    "],\n",
    "'Libraries': [\n",
    "'TensorFlow, PyTorch',\n",
    "'scikit-learn',\n",
    "'statsmodels, ARIMA',\n",
    "'spaCy, NLTK, transformers',\n",
    "'OpenCV, PIL',\n",
    "'Gymnasium, TensorFlow'\n",
    "]\n",
    "}\n",
    "\n",
    "df_resources = pd.DataFrame(resources)\n",
    "print(df_resources.to_string(index=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c90bf94",
   "metadata": {},
   "source": [
    "## Week 9 Summary\n",
    "By completing Week 9, you have learned:\n",
    "- Machine learning fundamentals: supervised vs unsupervised\n",
    "- ML workflow: problem definition through deployment\n",
    "- Bias-variance tradeoff and generalization\n",
    "- Data leakage and how to prevent it\n",
    "- Train-test split and cross-validation\n",
    "- Decision trees: how they work and interpretability\n",
    "- Hyperparameters affecting decision trees\n",
    "- Overfitting and underfitting in trees\n",
    "- Ensemble methods: bagging and boosting\n",
    "- Random Forest: parallel ensemble of trees\n",
    "- Gradient Boosting: sequential error correction\n",
    "- K-Nearest Neighbors (KNN) algorithm\n",
    "- Support Vector Machines (SVM)\n",
    "- Naive Bayes classifier\n",
    "- Voting classifiers and stacking\n",
    "- Accuracy, precision, recall, and F1-score\n",
    "- Confusion matrix interpretation\n",
    "- ROC curves and AUC-ROC\n",
    "- Cross-validation for robust evaluation\n",
    "- Grid search and random search for tuning\n",
    "- Learning curves and bias-variance diagnosis\n",
    "- Feature importance and model interpretation\n",
    "- Handling class imbalance\n",
    "- Common pitfalls and best practices\n",
    "- Reproducibility and documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29432455",
   "metadata": {},
   "source": [
    "## Week 9 Assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7cb0a36",
   "metadata": {},
   "source": [
    "### Assignment 1: Decision Trees and Ensemble Comparison\n",
    "Build and compare multiple classification models:\n",
    "- Preprocess and explore classification dataset\n",
    "- Build single decision tree classifier\n",
    "- Build Random Forest classifier\n",
    "- Build Gradient Boosting classifier\n",
    "- Compare model performance on test set\n",
    "- Plot feature importance for each model\n",
    "- Create learning curves for best model\n",
    "- Analyze overfitting in each model\n",
    "- Write summary of findings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d02e588b",
   "metadata": {},
   "source": [
    "### Assignment 2: Hyperparameter Optimization\n",
    "Perform systematic hyperparameter tuning:\n",
    "- Implement grid search on one model\n",
    "- Implement random search on same model\n",
    "- Compare both approaches (speed, results)\n",
    "- Find optimal hyperparameters\n",
    "- Generate learning curves\n",
    "- Visualize parameter importance\n",
    "- Test on held-out test set\n",
    "- Report final model performance\n",
    "- Discuss computational cost vs improvement"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46e9158b",
   "metadata": {},
   "source": [
    "### Assignment 3: Complete ML Pipeline and Deployment\n",
    "Build end-to-end machine learning pipeline:\n",
    "- Data loading and exploratory analysis\n",
    "- Data preprocessing and feature engineering\n",
    "- Train-test split with stratification\n",
    "- Build multiple models\n",
    "- Evaluate with multiple metrics\n",
    "- Hyperparameter tuning\n",
    "- Cross-validation\n",
    "- Error analysis and interpretation\n",
    "- Save trained model\n",
    "- Create prediction function for new data\n",
    "- Document entire workflow\n",
    "- Present findings with recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8da8e82",
   "metadata": {},
   "source": [
    "## Recommended Practice Problems\n",
    "- Build decision trees with different depths and analyze overfitting\n",
    "- Compare single trees vs Random Forests on same data\n",
    "- Tune Random Forest on real dataset with grid search\n",
    "- Implement voting classifier with different algorithms\n",
    "- Work with highly imbalanced dataset\n",
    "- Compare evaluation metrics for different business costs\n",
    "- Perform manual cross-validation and compare with sklearn\n",
    "- Extract and interpret feature importance\n",
    "- Debug poor model performance systematically\n",
    "- Compare training time vs accuracy across models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85b67a46",
   "metadata": {},
   "source": [
    "## Additional Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fed98a42",
   "metadata": {},
   "source": [
    "### Books\n",
    "- Hands-On Machine Learning by Aurélien Géron\n",
    "- The Hundred-Page Machine Learning Book by Andriy Burkov (free)\n",
    "- Introduction to Statistical Learning (ISLR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9474a769",
   "metadata": {},
   "source": [
    "### Online Resources\n",
    "- scikit-learn documentation and tutorials: https://scikit-learn.org/\n",
    "- Kaggle competitions: https://www.kaggle.com/\n",
    "- Google Colab for free GPU: https://colab.research.google.com/\n",
    "- XGBoost and LightGBM for advanced boosting"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}