Week 9: Machine Learning — Classification and Ensembles

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, classification_report
import matplotlib.pyplot as plt

print("="*50)
print("MACHINE LEARNING FUNDAMENTALS")
print("="*50)

print("""
WHAT IS MACHINE LEARNING?
- Algorithms that learn patterns from data
- Improve performance with more data
- Make predictions on new, unseen data

TYPES OF MACHINE LEARNING:

1. SUPERVISED LEARNING (labeled data)
- Classification: Predict category (yes/no, A/B/C)
- Regression: Predict continuous value (price, temperature)
- Examples: spam detection, price prediction

2. UNSUPERVISED LEARNING (unlabeled data)
- Clustering: Group similar items
- Dimensionality reduction: reduce features
- Examples: customer segmentation, anomaly detection

3. REINFORCEMENT LEARNING
- Learn from feedback (rewards/penalties)
- Examples: game playing, robotics

THIS WEEK: SUPERVISED LEARNING - CLASSIFICATION
""")

# Machine Learning Workflow
print("\n" + "="*50)
print("MACHINE LEARNING WORKFLOW")
print("="*50)

print("""
1. PROBLEM DEFINITION
- What are we predicting?
- What data do we have?
- What's success look like?

2. DATA PREPARATION
- Clean and preprocess data
- Handle missing values
- Feature engineering
- Train-test split

3. MODEL SELECTION
- Choose appropriate algorithm
- Consider interpretability vs accuracy

4. TRAINING
- Fit model to training data
- Model learns patterns

5. VALIDATION
- Evaluate on validation set
- Prevent overfitting

6. HYPERPARAMETER TUNING
- Optimize model parameters
- Grid search or random search

7. TESTING
- Final evaluation on test set
- Report performance

8. DEPLOYMENT
- Use model in production
- Monitor performance over time
""")

# Key Concepts
print("\n" + "="*50)
print("KEY MACHINE LEARNING CONCEPTS")
print("="*50)

print("""
BIAS-VARIANCE TRADEOFF:
- HIGH BIAS: Model too simple, underfits (both train and test error high)
- HIGH VARIANCE: Model too complex, overfits (train error low, test error high)
- GOAL: Balance bias and variance for good generalization

GENERALIZATION:
- How well model performs on NEW, unseen data
- More important than training accuracy
- Achieved through proper validation and regularization

DATA LEAKAGE:
- Using information that shouldn't be available at prediction time
- Common mistakes:
- Including target variable in features
- Using test set for preprocessing
- Using future information
- Prevention: Strict train-test separation

CURSE OF DIMENSIONALITY:
- More features can make learning harder
- Need more data as dimensions increase
- Feature selection/reduction important

IMBALANCED DATA:
- When classes have very different frequencies
- Standard accuracy misleading
- Use precision, recall, F1 instead
""")

# Create example dataset
print("\n" + "="*50)
print("EXAMPLE: IRIS FLOWER CLASSIFICATION")
print("="*50)

from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data
y = iris.target
feature_names = iris.feature_names
target_names = iris.target_names

print(f"\nDataset size: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Classes: {target_names}")
print(f"Features: {feature_names}")

# Create DataFrame
df = pd.DataFrame(X, columns=feature_names)
df['species'] = y
df['species_name'] = df['species'].map({0: 'Setosa', 1: 'Versicolor', 2: 'Virginica'})

print("\nFirst few samples:")
print(df.head())

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"\nTrain-test split:")
print(f"Training set: {X_train.shape[0]} samples ({X_train.shape[0]/len(X)*100:.0f}%)")
print(f"Test set: {X_test.shape[0]} samples ({X_test.shape[0]/len(X)*100:.0f}%)")

# Check class balance
print(f"\nClass distribution (training set):")
unique, counts = np.unique(y_train, return_counts=True)
for target, count in zip(unique, counts):
print(f" {target_names[target]}: {count} ({count/len(y_train)*100:.0f}%)")

# Simple model
model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)

# Predictions
y_pred_train = model.predict(X_train)
y_pred_test = model.predict(X_test)

# Evaluation
train_accuracy = accuracy_score(y_train, y_pred_train)
test_accuracy = accuracy_score(y_test, y_pred_test)

print(f"\nModel Performance:")
print(f"Training accuracy: {train_accuracy:.4f}")
print(f"Test accuracy: {test_accuracy:.4f}")
print(f"Difference: {train_accuracy - test_accuracy:.4f} (larger = more overfitting)")

print(f"\nClassification Report (Test Set):")
print(classification_report(y_test, y_pred_test, target_names=target_names))

import numpy as np
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import matplotlib.pyplot as plt

print("="*50)
print("DECISION TREES")
print("="*50)

print("""
HOW DECISION TREES WORK:
- Recursively split data into smaller groups
- Each split maximizes information gain
- Creates tree-like structure of decision rules

ADVANTAGES:
- Highly interpretable (can visualize and explain)
- No preprocessing needed
- Works with non-linear relationships
- Handles both classification and regression
- Can handle missing values (in some implementations)

DISADVANTAGES:
- Prone to overfitting
- Unstable (small data change → big tree change)
- Can be biased with imbalanced data
- Greedy algorithm (not globally optimal)

WHEN TO USE:
- When interpretability is important
- Mixed data types
- Non-linear relationships
- Feature interactions matter
""")

# Load data
iris = load_iris()
X = iris.data
y = iris.target
feature_names = iris.feature_names
target_names = iris.target_names

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# Build different depth trees
print("\nEXAMPLE: IRIS CLASSIFICATION WITH DIFFERENT DEPTHS")
print("-"*50)

depths = [2, 3, 5, 10]
results = []

for depth in depths:
model = DecisionTreeClassifier(max_depth=depth, random_state=42)
model.fit(X_train, y_train)

train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
n_nodes = model.tree_.node_count

results.append({
'Depth': depth,
'Train Accuracy': train_acc,
'Test Accuracy': test_acc,
'Nodes': n_nodes,
'Gap': train_acc - test_acc
})

print(f"\nDepth {depth}:")
print(f" Training accuracy: {train_acc:.4f}")
print(f" Test accuracy: {test_acc:.4f}")
print(f" Train-Test gap: {train_acc - test_acc:.4f}")
print(f" Number of nodes: {n_nodes}")

if train_acc - test_acc > 0.05:
print(f" Status: OVERFITTING (high gap)")
else:
print(f" Status: GOOD GENERALIZATION")

# Build final model
print("\n" + "="*50)
print("FINAL TREE (Depth=3)")
print("="*50)

model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)

print(f"\nAccuracy: {accuracy:.4f}")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=target_names))

# Feature importance
print(f"\nFeature Importance:")
for name, importance in zip(feature_names, model.feature_importances_):
print(f" {name}: {importance:.4f}")

# Tree visualization
fig, ax = plt.subplots(figsize=(20, 10))
plot_tree(model,
feature_names=feature_names,
class_names=target_names,
filled=True,
ax=ax,
fontsize=10)
plt.title('Decision Tree (Max Depth = 3)', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

# Visualize depth effect
import pandas as pd
results_df = pd.DataFrame(results)

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Accuracy vs Depth
axes[0].plot(results_df['Depth'], results_df['Train Accuracy'], 'b-o', label='Train', linewidth=2)
axes[0].plot(results_df['Depth'], results_df['Test Accuracy'], 'r-o', label='Test', linewidth=2)
axes[0].fill_between(results_df['Depth'], results_df['Train Accuracy'],
results_df['Test Accuracy'], alpha=0.2, color='gray')
axes[0].set_xlabel('Max Depth')
axes[0].set_ylabel('Accuracy')
axes[0].set_title('Impact of Tree Depth on Accuracy')
axes[0].legend()
axes[0].grid(True, alpha=0.3)

# Overfitting gap
axes[1].plot(results_df['Depth'], results_df['Gap'], 'g-o', linewidth=2, markersize=8)
axes[1].set_xlabel('Max Depth')
axes[1].set_ylabel('Train-Test Gap')
axes[1].set_title('Overfitting Gap vs Depth')
axes[1].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Hyperparameters affecting decision trees
print("\n" + "="*50)
print("DECISION TREE HYPERPARAMETERS")
print("="*50)

print("""
KEY HYPERPARAMETERS:

max_depth:
- Maximum depth of tree
- Lower → simpler model, less overfitting
- Higher → more complex, better training fit

min_samples_split:
- Minimum samples to split a node
- Higher → larger leaves, less overfitting
- Lower → more splits allowed

min_samples_leaf:
- Minimum samples in leaf node
- Prevents small, isolated leaves

max_features:
- Number of features to consider for split
- Adds randomness, reduces overfitting
- Often sqrt(n_features) or log(n_features)

criterion:
- 'gini': Uses Gini impurity
- 'entropy': Uses information gain

TIPS FOR PREVENTING OVERFITTING:
- Set max_depth (typical: 5-15 for tabular data)
- Increase min_samples_leaf
- Decrease max_features
- Use ensemble methods (combine multiple trees)
""")

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import matplotlib.pyplot as plt

print("="*50)
print("ENSEMBLE METHODS")
print("="*50)

print("""
ENSEMBLE LEARNING:
- Combine predictions from multiple models
- Often better than single model
- "Wisdom of crowds" principle

WHY ENSEMBLES WORK:
- Different models make different mistakes
- Combining reduces variance
- Improves generalization

TYPES OF ENSEMBLES:

1. BAGGING (Bootstrap Aggregating)
- Train models on random samples WITH replacement
- Average predictions
- Reduces variance, not bias
- Example: Random Forest

2. BOOSTING
- Train models sequentially
- Each model corrects previous errors
- Reduces bias and variance
- Examples: AdaBoost, Gradient Boosting

3. STACKING
- Train multiple models
- Use another model to combine them

4. VOTING
- Average predictions from different algorithms
""")

# Load data
iris = load_iris()
X = iris.data
y = iris.target
feature_names = iris.feature_names
target_names = iris.target_names

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# Random Forest
print("\n" + "="*50)
print("RANDOM FOREST")
print("="*50)

print("""
HOW IT WORKS:
1. Create multiple bootstrap samples of data
2. Build decision tree on each sample
3. Each tree sees random subset of features
4. Average predictions (regression) or majority vote (classification)

ADVANTAGES:
- Reduces overfitting compared to single tree
- Handles non-linear relationships
- Feature importance
- Works with large datasets
- Fast training and prediction
- No preprocessing needed

DISADVANTAGES:
- Less interpretable than single tree
- Can be slow with huge datasets
- Memory intensive
- Biased with highly imbalanced data
""")

# Train Random Forest
model_rf = RandomForestClassifier(
n_estimators=100,
max_depth=5,
min_samples_leaf=2,
random_state=42,
n_jobs=-1 # Use all processors
)

model_rf.fit(X_train, y_train)

# Predictions
y_pred_rf = model_rf.predict(X_test)
acc_rf = accuracy_score(y_test, y_pred_rf)

print(f"\nRandom Forest Results:")
print(f"Number of trees: {model_rf.n_estimators}")
print(f"Max depth: {model_rf.max_depth}")
print(f"Test accuracy: {acc_rf:.4f}")

print(f"\nClassification Report:")
print(classification_report(y_test, y_pred_rf, target_names=target_names))

# Feature importance
print(f"\nFeature Importance (Random Forest):")
importance_rf = pd.DataFrame({
'Feature': feature_names,
'Importance': model_rf.feature_importances_
}).sort_values('Importance', ascending=False)

print(importance_rf.to_string(index=False))

# Gradient Boosting
print("\n" + "="*50)
print("GRADIENT BOOSTING")
print("="*50)

print("""
HOW IT WORKS:
1. Start with simple model
2. Train next model on residuals (errors)
3. Each model corrects previous errors
4. Combine additively

ADVANTAGES:
- Often highest accuracy
- Handles complex patterns
- Feature importance
- Works with various data types
- Good with imbalanced data

DISADVANTAGES:
- Slow training (sequential)
- Prone to overfitting if not tuned
- Many hyperparameters to tune
- Less interpretable

KEY HYPERPARAMETERS:
- learning_rate: Step size (lower = slower but better)
- n_estimators: Number of boosting rounds
- max_depth: Depth of trees
""")

# Train Gradient Boosting
model_gb = GradientBoostingClassifier(
n_estimators=100,
learning_rate=0.1,
max_depth=3,
random_state=42
)

model_gb.fit(X_train, y_train)

# Predictions
y_pred_gb = model_gb.predict(X_test)
acc_gb = accuracy_score(y_test, y_pred_gb)

print(f"\nGradient Boosting Results:")
print(f"Number of estimators: {model_gb.n_estimators}")
print(f"Learning rate: {model_gb.learning_rate}")
print(f"Test accuracy: {acc_gb:.4f}")

# Feature importance
print(f"\nFeature Importance (Gradient Boosting):")
importance_gb = pd.DataFrame({
'Feature': feature_names,
'Importance': model_gb.feature_importances_
}).sort_values('Importance', ascending=False)

print(importance_gb.to_string(index=False))

# Model comparison
print("\n" + "="*50)
print("MODEL COMPARISON")
print("="*50)

comparison = pd.DataFrame({
'Model': ['Random Forest', 'Gradient Boosting'],
'Accuracy': [acc_rf, acc_gb],
'Interpretability': ['Medium', 'Low'],
'Speed': ['Fast', 'Slow'],
'Memory': ['High', 'Medium']
})

print(comparison.to_string(index=False))

# Visualization: Feature Importance
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Random Forest
axes[0].barh(importance_rf['Feature'], importance_rf['Importance'], color='steelblue')
axes[0].set_xlabel('Importance')
axes[0].set_title('Feature Importance - Random Forest', fontweight='bold')
axes[0].grid(True, alpha=0.3, axis='x')

# Gradient Boosting
axes[1].barh(importance_gb['Feature'], importance_gb['Importance'], color='coral')
axes[1].set_xlabel('Importance')
axes[1].set_title('Feature Importance - Gradient Boosting', fontweight='bold')
axes[1].grid(True, alpha=0.3, axis='x')

plt.tight_layout()
plt.show()

import numpy as np
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import VotingClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import pandas as pd

print("="*50)
print("OTHER CLASSIFICATION ALGORITHMS")
print("="*50)

# Load data
iris = load_iris()
X = iris.data
y = iris.target
target_names = iris.target_names

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# 1. K-NEAREST NEIGHBORS (KNN)
print("\n1. K-NEAREST NEIGHBORS (KNN)")
print("-"*50)

print("""
HOW IT WORKS:
- For prediction, look at k nearest neighbors
- Majority vote for classification
- Distance usually Euclidean

ADVANTAGES:
- Simple and intuitive
- No training phase
- Works with non-linear data

DISADVANTAGES:
- Slow prediction (compares to all training points)
- Sensitive to distance metric
- Needs feature scaling
- Sensitive to k value
- Poor with high dimensions
""")

model_knn = KNeighborsClassifier(n_neighbors=5)
model_knn.fit(X_train, y_train)
acc_knn = model_knn.score(X_test, y_test)

print(f"\nKNN Accuracy (k=5): {acc_knn:.4f}")

# 2. SUPPORT VECTOR MACHINE (SVM)
print("\n2. SUPPORT VECTOR MACHINE (SVM)")
print("-"*50)

print("""
HOW IT WORKS:
- Finds optimal hyperplane separating classes
- Maximizes margin between classes
- Can handle non-linear with kernel trick

ADVANTAGES:
- Effective in high dimensions
- Memory efficient (uses subset of points)
- Various kernel options
- Works well with binary classification

DISADVANTAGES:
- Slow training with large datasets
- Hard to interpret
- Need feature scaling
- Hyperparameter tuning critical
""")

model_svm = SVC(kernel='rbf', C=1.0, gamma='scale')
model_svm.fit(X_train, y_train)
acc_svm = model_svm.score(X_test, y_test)

print(f"\nSVM Accuracy: {acc_svm:.4f}")

# 3. NAIVE BAYES
print("\n3. NAIVE BAYES")
print("-"*50)

print("""
HOW IT WORKS:
- Uses Bayes' theorem: P(Y|X) = P(X|Y)P(Y) / P(X)
- Assumes features are independent (naive)
- Fast to train and predict

ADVANTAGES:
- Fast
- Works with small datasets
- Handles high dimensions well
- Good baseline

DISADVANTAGES:
- Independence assumption often violated
- Less accurate than complex models
- Not best for high correlations between features
""")

model_nb = GaussianNB()
model_nb.fit(X_train, y_train)
acc_nb = model_nb.score(X_test, y_test)

print(f"\nNaive Bayes Accuracy: {acc_nb:.4f}")

# Voting Classifier (Ensemble of different algorithms)
print("\n" + "="*50)
print("VOTING CLASSIFIER (Ensemble of Different Algorithms)")
print("="*50)

print("""
COMBINE DIFFERENT ALGORITHMS:
- Use multiple different models
- Vote on final prediction
- Combines strengths of different algorithms
""")

from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier

voting_clf = VotingClassifier(
estimators=[
('dt', DecisionTreeClassifier(max_depth=5, random_state=42)),
('rf', RandomForestClassifier(n_estimators=50, max_depth=5, random_state=42)),
('knn', KNeighborsClassifier(n_neighbors=5)),
('svm', SVC(kernel='rbf', C=1.0, gamma='scale'))
],
voting='hard' # hard voting (majority vote)
)

voting_clf.fit(X_train, y_train)
acc_voting = voting_clf.score(X_test, y_test)

print(f"\nVoting Classifier Accuracy: {acc_voting:.4f}")

# Summary comparison
print("\n" + "="*50)
print("ALGORITHM COMPARISON")
print("="*50)

comparison = pd.DataFrame({
'Algorithm': ['KNN', 'SVM', 'Naive Bayes', 'Voting Ensemble'],
'Accuracy': [acc_knn, acc_svm, acc_nb, acc_voting],
'Training Speed': ['Fast', 'Slow', 'Fast', 'Medium'],
'Interpretability': ['High', 'Low', 'High', 'Low'],
'Scalability': ['Poor', 'Medium', 'Good', 'Medium']
})

print(comparison.to_string(index=False))

print("\n" + "="*50)
print("CHOOSING AN ALGORITHM")
print("="*50)

print("""
DECISION GUIDE:

START HERE:
→ Is interpretability critical?
YES: Use Decision Tree or Logistic Regression
NO: Use ensemble methods (RF, GB)

→ Large dataset (>100K rows)?
YES: Use Gradient Boosting or Linear model
NO: Most algorithms work

→ High-dimensional data (>100 features)?
YES: Use SVM or regularized linear model
NO: Most algorithms work

→ Categorical features?
YES: Tree-based methods (no preprocessing)
NO: Any algorithm

→ Imbalanced classes?
YES: Use weighted algorithms or SMOTE
NO: Any algorithm

GENERAL RULE:
- Start simple (Logistic Regression, Decision Tree)
- Ensemble if needed (Random Forest)
- Advanced tuning if required (Gradient Boosting)
""")

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, roc_auc_score, roc_curve, auc,
confusion_matrix, classification_report)
import matplotlib.pyplot as plt
import pandas as pd

print("="*50)
print("COMPREHENSIVE MODEL EVALUATION")
print("="*50)

# Load data
iris = load_iris()
X = iris.data
y = iris.target
target_names = iris.target_names

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print("""
METRICS FOR CLASSIFICATION:

ACCURACY:
- Correct predictions / Total predictions
- Good overall measure
- Can be misleading with imbalanced data
- Formula: (TP + TN) / (TP + TN + FP + FN)

PRECISION:
- Positive predictions that are correct
- "Of predicted positive, how many correct?"
- Important when false positives costly
- Formula: TP / (TP + FP)

RECALL (Sensitivity):
- Actual positives that are detected
- "Of actual positive, how many found?"
- Important when false negatives costly
- Formula: TP / (TP + FN)

F1-SCORE:
- Harmonic mean of precision and recall
- Single number balancing both
- Better than accuracy for imbalanced data
- Formula: 2 × (Precision × Recall) / (Precision + Recall)

SPECIFICITY:
- True negatives properly identified
- Formula: TN / (TN + FP)

ROC-AUC:
- Receiver Operating Characteristic
- Plots TPR vs FPR at different thresholds
- AUC = area under curve
- 1.0 = perfect, 0.5 = random, <0.5 = worse than random
""")

# Binary classification example
# Convert to binary (Virginica vs others)
y_binary = (y == 2).astype(int)
X_train_b, X_test_b, y_train_b, y_test_b = train_test_split(
X, y_binary, test_size=0.2, random_state=42, stratify=y_binary
)

model_b = RandomForestClassifier(n_estimators=100, random_state=42)
model_b.fit(X_train_b, y_train_b)
y_pred_b = model_b.predict(X_test_b)
y_proba_b = model_b.predict_proba(X_test_b)[:, 1]

# Metrics
accuracy = accuracy_score(y_test_b, y_pred_b)
precision = precision_score(y_test_b, y_pred_b)
recall = recall_score(y_test_b, y_pred_b)
f1 = f1_score(y_test_b, y_pred_b)
auc_score = roc_auc_score(y_test_b, y_proba_b)

print(f"\nBINARY CLASSIFICATION METRICS (Virginica vs Others):")
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-Score: {f1:.4f}")
print(f"AUC-ROC: {auc_score:.4f}")

print(f"\nConfusion Matrix:")
cm = confusion_matrix(y_test_b, y_pred_b)
print(f"True Negatives: {cm[0,0]}")
print(f"False Positives: {cm[0,1]}")
print(f"False Negatives: {cm[1,0]}")
print(f"True Positives: {cm[1,1]}")

# Cross-validation
print("\n" + "="*50)
print("CROSS-VALIDATION")
print("="*50)

print("""
MULTIPLE TRAIN-TEST SPLITS:
- k-fold: Divide into k groups
- Train k models, average scores
- Better estimate of generalization
- More robust than single split

STRATIFIED K-FOLD:
- Maintains class distribution in each fold
- Important for imbalanced data
- Better than regular k-fold
""")

# Stratified K-Fold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(model, X, y, cv=skf, scoring='accuracy')

print(f"\nStratified 5-Fold Cross-Validation:")
print(f"Fold scores: {cv_scores}")
print(f"Mean: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")

# Visualization
fig, axes = plt.subplots(2, 2, figsize=(14, 10))

# 1. Confusion Matrix Heatmap
import seaborn as sns
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=axes[0, 0],
xticklabels=['Not Virginica', 'Virginica'],
yticklabels=['Not Virginica', 'Virginica'])
axes[0, 0].set_ylabel('Actual')
axes[0, 0].set_xlabel('Predicted')
axes[0, 0].set_title('Confusion Matrix')

# 2. ROC Curve
fpr, tpr, _ = roc_curve(y_test_b, y_proba_b)
axes[0, 1].plot(fpr, tpr, linewidth=2, label=f'AUC = {auc_score:.3f}')
axes[0, 1].plot([0, 1], [0, 1], 'r--', label='Random')
axes[0, 1].set_xlabel('False Positive Rate')
axes[0, 1].set_ylabel('True Positive Rate')
axes[0, 1].set_title('ROC Curve')
axes[0, 1].legend()
axes[0, 1].grid(True, alpha=0.3)

# 3. Cross-validation scores
axes[1, 0].bar(range(1, 6), cv_scores, color='steelblue', alpha=0.7)
axes[1, 0].axhline(y=cv_scores.mean(), color='r', linestyle='--', linewidth=2)
axes[1, 0].set_xlabel('Fold')
axes[1, 0].set_ylabel('Accuracy')
axes[1, 0].set_title('5-Fold Cross-Validation Scores')
axes[1, 0].set_ylim([0.8, 1.0])
axes[1, 0].grid(True, alpha=0.3, axis='y')

# 4. Metrics comparison
metrics = pd.DataFrame({
'Metric': ['Accuracy', 'Precision', 'Recall', 'F1'],
'Score': [accuracy, precision, recall, f1]
})

axes[1, 1].bar(metrics['Metric'], metrics['Score'], color=['blue', 'green', 'orange', 'red'], alpha=0.7)
axes[1, 1].set_ylim([0, 1])
axes[1, 1].set_ylabel('Score')
axes[1, 1].set_title('Classification Metrics')
axes[1, 1].grid(True, alpha=0.3, axis='y')

# Add value labels on bars
for i, (metric, score) in enumerate(zip(metrics['Metric'], metrics['Score'])):
axes[1, 1].text(i, score + 0.02, f'{score:.3f}', ha='center', va='bottom')

plt.tight_layout()
plt.show()

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import (train_test_split, GridSearchCV,
RandomizedSearchCV, cross_val_score)
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import pandas as pd
import matplotlib.pyplot as plt

print("="*50)
print("HYPERPARAMETER TUNING")
print("="*50)

print("""
HYPERPARAMETERS:
- Parameters we SET before training
- Different from model parameters (learned during training)
- Critical for model performance
- Need tuning to find optimal values

TUNING METHODS:

1. GRID SEARCH
- Try all combinations in parameter grid
- Exhaustive search
- Works with small grids
- Slow but thorough

2. RANDOM SEARCH
- Random sample from parameter space
- Works with large grids
- Faster than grid search
- Sometimes finds better solutions

3. BAYESIAN OPTIMIZATION
- Uses probability model
- Smart sampling
- Efficient
- More complex to implement
""")

# Load data
iris = load_iris()
X = iris.data
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# Grid Search
print("\n" + "="*50)
print("GRID SEARCH EXAMPLE")
print("="*50)

param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [5, 10, 15, None],
'min_samples_split': [2, 5, 10],
'min_samples_leaf': [1, 2, 4]
}

print(f"\nParameter grid:")
for param, values in param_grid.items():
print(f" {param}: {values}")

total_combinations = 1
for values in param_grid.values():
total_combinations *= len(values)

print(f"\nTotal combinations to try: {total_combinations}")

# Run grid search (can be slow, so we'll do a simplified version)
print("\nRunning Grid Search...")

grid_search = GridSearchCV(
RandomForestClassifier(random_state=42),
param_grid,
cv=5, # 5-fold cross-validation
scoring='accuracy',
n_jobs=-1, # Use all processors
verbose=0
)

grid_search.fit(X_train, y_train)

print(f"\nBest parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.4f}")

# Test performance
y_pred = grid_search.predict(X_test)
test_accuracy = accuracy_score(y_test, y_pred)
print(f"Test set accuracy: {test_accuracy:.4f}")

# Results summary
results_df = pd.DataFrame(grid_search.cv_results_)
results_summary = results_df[['param_n_estimators', 'param_max_depth',
'param_min_samples_split', 'mean_test_score']].head(10)

print(f"\nTop 10 parameter combinations:")
results_sorted = results_df.sort_values('mean_test_score', ascending=False).head(10)
print(results_sorted[['param_n_estimators', 'param_max_depth',
'param_min_samples_split', 'mean_test_score']].to_string(index=False))

# Random Search
print("\n" + "="*50)
print("RANDOM SEARCH")
print("="*50)

print("""
RANDOM SEARCH:
- Faster alternative to grid search
- Sample parameter combinations randomly
- Good for exploring large search spaces
""")

param_dist = {
'n_estimators': [50, 100, 150, 200, 250],
'max_depth': list(range(3, 20)),
'min_samples_split': [2, 5, 10],
'min_samples_leaf': [1, 2, 4]
}

random_search = RandomizedSearchCV(
RandomForestClassifier(random_state=42),
param_dist,
n_iter=20, # Try 20 random combinations
cv=5,
scoring='accuracy',
n_jobs=-1,
random_state=42
)

random_search.fit(X_train, y_train)

print(f"\nBest parameters: {random_search.best_params_}")
print(f"Best CV score: {random_search.best_score_:.4f}")

y_pred_rs = random_search.predict(X_test)
test_accuracy_rs = accuracy_score(y_test, y_pred_rs)
print(f"Test set accuracy: {test_accuracy_rs:.4f}")

# Learning curves
print("\n" + "="*50)
print("LEARNING CURVES")
print("="*50)

print("""
SHOWS:
- Training accuracy vs validation accuracy
- Effect of training set size
- Detect underfitting/overfitting
- Whether more data helps
""")

from sklearn.model_selection import learning_curve

train_sizes, train_scores, val_scores = learning_curve(
RandomForestClassifier(n_estimators=100, random_state=42),
X_train, y_train,
cv=5,
train_sizes=np.linspace(0.1, 1.0, 10),
scoring='accuracy',
n_jobs=-1
)

train_mean = train_scores.mean(axis=1)
train_std = train_scores.std(axis=1)
val_mean = val_scores.mean(axis=1)
val_std = val_scores.std(axis=1)

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Learning curves
axes[0].plot(train_sizes, train_mean, 'b-o', label='Training', linewidth=2)
axes[0].fill_between(train_sizes, train_mean - train_std,
train_mean + train_std, alpha=0.2, color='blue')
axes[0].plot(train_sizes, val_mean, 'r-o', label='Validation', linewidth=2)
axes[0].fill_between(train_sizes, val_mean - val_std,
val_mean + val_std, alpha=0.2, color='red')
axes[0].set_xlabel('Training Set Size')
axes[0].set_ylabel('Accuracy')
axes[0].set_title('Learning Curves')
axes[0].legend()
axes[0].grid(True, alpha=0.3)

# Parameter importance
results_sorted_gs = results_df.sort_values('mean_test_score', ascending=False).head(20)
param_counts = {}
for param in ['n_estimators', 'max_depth', 'min_samples_split', 'min_samples_leaf']:
param_name = f'param_{param}'
if param_name in results_sorted_gs.columns:
values = results_sorted_gs[param_name].value_counts()
param_counts[param] = len(values)

axes[1].bar(param_counts.keys(), param_counts.values(), color='steelblue', alpha=0.7)
axes[1].set_ylabel('Unique Values in Top 20')
axes[1].set_title('Parameter Diversity in Best Models')
axes[1].grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

import numpy as np
import pandas as pd

print("="*50)
print("MACHINE LEARNING BEST PRACTICES")
print("="*50)

print("""
1. DATA QUALITY IS CRITICAL
✓ Clean and validate data
✓ Handle missing values appropriately
✓ Check for outliers
✓ Ensure data consistency
✗ Never skip EDA (exploratory data analysis)
✗ Don't ignore data quality issues

2. PROPER TRAIN-TEST SPLIT
✓ Always split BEFORE preprocessing
✓ Use stratified split for imbalanced data
✓ Test set should never influence training
✓ Use cross-validation for robust estimates
✗ Don't fit preprocessor on entire dataset
✗ Don't use test set for feature selection
✗ Don't tune hyperparameters on test set

3. FEATURE ENGINEERING
✓ Create meaningful features
✓ Domain knowledge helps
✓ Test feature importance
✓ Start simple, add complexity gradually
✗ Don't create too many features (curse of dimensionality)
✗ Don't use domain-specific features that won't generalize
✗ Don't include future information (data leakage)

4. MODEL SELECTION
✓ Start with simple models (baseline)
✓ Compare multiple algorithms
✓ Use appropriate metrics (not just accuracy)
✓ Consider interpretability vs accuracy tradeoff
✗ Don't jump to complex models immediately
✗ Don't use same metric for all problems
✗ Don't ignore class imbalance

5. EVALUATION AND VALIDATION
✓ Use multiple metrics
✓ Report confidence intervals
✓ Check model assumptions
✓ Validate on held-out test set only
✗ Don't report only training accuracy
✗ Don't cherry-pick best metric
✗ Don't evaluate on training data

6. COMMON PITFALLS TO AVOID

DATA LEAKAGE:
- Using information that won't be available at prediction time
- Example: Including future values, target-derived features
- Prevention: Careful feature engineering, strict train-test separation

CLASS IMBALANCE:
- One class much more frequent than others
- Accuracy misleading (high accuracy, poor minority class detection)
- Solutions: Stratified split, class weights, SMOTE, different metrics

OVERFITTING:
- Model learns training data too well
- Poor generalization to new data
- Signs: High train accuracy, low test accuracy
- Prevention: Regularization, cross-validation, more data

UNDERFITTING:
- Model too simple to capture relationships
- Both train and test accuracy low
- Solution: More complex model, feature engineering

MULTIPLE TESTING:
- Testing many hypotheses → some false positives
- Solution: Pre-register analyses, adjust significance levels

7. REPRODUCIBILITY
✓ Set random seeds
✓ Document preprocessing steps
✓ Version control code and data
✓ Save trained models for later
✗ Don't rely on randomness without setting seeds
✗ Don't forget to document assumptions

8. DEPLOYMENT CONSIDERATIONS
✓ Monitor model performance over time
✓ Handle data drift
✓ Plan for retraining
✓ Document decision rules
✓ Consider fairness and bias
✗ Don't assume model stays accurate forever
✗ Don't deploy without testing in production environment
✗ Don't ignore ethical implications

9. MODEL INTERPRETATION
✓ Explain predictions when possible
✓ Use feature importance
✓ Check for spurious correlations
✓ Domain validation of findings
✗ Don't trust black-box models blindly
✗ Don't assume correlation implies causation
✗ Don't ignore feature interactions

10. WORKFLOW CHECKLIST

PROBLEM DEFINITION:
□ Define success metric clearly
□ Understand business context
□ Identify constraints and requirements

DATA PREPARATION:
□ Exploratory data analysis
□ Handle missing values
□ Check for outliers
□ Feature engineering
□ Data validation

MODEL DEVELOPMENT:
□ Baseline model
□ Multiple algorithms
□ Hyperparameter tuning
□ Cross-validation
□ Feature importance analysis

EVALUATION:
□ Multiple metrics
□ Confidence intervals
□ Error analysis
□ Assumptions check
□ Business impact assessment

DEPLOYMENT:
□ Final model selection
□ Documentation
□ Testing in production environment
□ Monitoring plan
□ Retraining schedule
""")

print("\n" + "="*50)
print("PERFORMANCE TROUBLESHOOTING GUIDE")
print("="*50)

troubleshooting = {
'Problem': [
'High training accuracy, low test accuracy',
'Both training and test accuracy low',
'Class imbalance issues',
'Inconsistent cross-validation scores',
'Model too slow for production'
],
'Likely Cause': [
'Overfitting',
'Underfitting',
'Class imbalance, inappropriate metric',
'High variance, unstable model',
'Too complex model, large data'
],
'Solution': [
'Regularization, simpler model, more data',
'More complex model, feature engineering',
'Use stratified split, class weights, F1/ROC',
'Cross-validation, ensemble methods',
'Simpler model, feature selection, sampling'
]
}

df_trouble = pd.DataFrame(troubleshooting)
print(df_trouble.to_string(index=False))

print("\n" + "="*50)
print("RESOURCES FOR CONTINUED LEARNING")
print("="*50)

resources = {
'Topic': [
'Deep Learning',
'Unsupervised Learning',
'Time Series',
'NLP',
'Computer Vision',
'Reinforcement Learning'
],
'When to Learn': [
'After mastering classical ML',
'For clustering and dimensionality reduction',
'For forecasting problems',
'For text data',
'For image data',
'For sequential decision problems'
],
'Libraries': [
'TensorFlow, PyTorch',
'scikit-learn',
'statsmodels, ARIMA',
'spaCy, NLTK, transformers',
'OpenCV, PIL',
'Gymnasium, TensorFlow'
]
}

df_resources = pd.DataFrame(resources)
print(df_resources.to_string(index=False))