{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f28a24bb",
   "metadata": {},
   "source": [
    "# Week 4: Data Cleaning and Preprocessing\n",
    "\n",
    "*Kmex Consult — Data Science Fundamentals Course*\n",
    "\n",
    "Work through each code cell in order. Edit and re-run to experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0339be57",
   "metadata": {},
   "source": [
    "*Data Science Fundamentals Course*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a673fea8",
   "metadata": {},
   "source": [
    "## Week 4 Overview\n",
    "Data cleaning and preprocessing is arguably the most time-consuming and critical part of any data science project. Often called \"data wrangling,\" this phase can consume 60-80% of a data scientist's time. Clean, well-prepared data leads to better analyses, more accurate models, and more reliable results.\n",
    "\n",
    "This week you will learn essential techniques for:\n",
    "- Identifying and handling missing data\n",
    "- Detecting and removing duplicate records\n",
    "- Dealing with outliers\n",
    "- Converting data types appropriately\n",
    "- Scaling and normalizing numerical features\n",
    "- Encoding categorical variables\n",
    "- Creating derived features\n",
    "- Reshaping and combining datasets\n",
    "\n",
    "By the end of Week 4, you will be able to:\n",
    "- Detect and handle missing values using appropriate strategies\n",
    "- Remove duplicates and identify outliers\n",
    "- Scale and normalize features for modeling\n",
    "- Convert and encode different data types\n",
    "- Merge and concatenate datasets\n",
    "- Create new features from existing ones\n",
    "- Build a complete data cleaning pipeline\n",
    "\n",
    "Week 4 is divided into three 2-hour sessions:\n",
    "- Session 1: Missing Data and Duplicates\n",
    "- Session 2: Feature Scaling, Normalization, and Type Conversion\n",
    "- Session 3: Categorical Encoding and Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0931204",
   "metadata": {},
   "source": [
    "## SESSION 1: Missing Data and Duplicates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5221fe42",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdf205ca",
   "metadata": {},
   "source": [
    "### 1.1 Understanding Missing Data\n",
    "Missing data (also called null values or NaN - Not a Number) is one of the most common problems in real datasets. Understanding why data is missing and how to handle it is crucial.\n",
    "\n",
    "Types of missing data:\n",
    "- Missing Completely at Random (MCAR): No pattern to missing data\n",
    "- Missing at Random (MAR): Missingness depends on other variables\n",
    "- Missing Not at Random (MNAR): Missingness depends on the missing values themselves\n",
    "\n",
    "Common causes:\n",
    "- Data entry errors or incomplete forms\n",
    "- Equipment or sensor failures\n",
    "- Data loss during transmission\n",
    "- Merging datasets with different coverage\n",
    "- Intentional omission for privacy\n",
    "- Participants not answering certain questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33e9a2f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create dataset with missing values\n",
    "df = pd.DataFrame({\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],\n",
    "'Age': [25, np.nan, 28, 32, np.nan],\n",
    "'City': ['Lagos', 'Accra', np.nan, 'Johannesburg', 'Cape Town'],\n",
    "'Salary': [60000, 75000, 65000, np.nan, 55000],\n",
    "'Department': ['Sales', np.nan, 'IT', 'Finance', 'Marketing']\n",
    "})\n",
    "\n",
    "print(df)\n",
    "# Output:\n",
    "# Name Age City Salary Department\n",
    "# 0 Alice 25.0 Lagos 60000.0 Sales\n",
    "# 1 Bob NaN Accra 75000.0 NaN\n",
    "# 2 Charlie 28.0 NaN 65000.0 IT\n",
    "# 3 Diana 32.0 Johannesburg NaN Finance\n",
    "# 4 Eve NaN Cape Town 55000.0 Marketing\n",
    "\n",
    "# Detect missing values\n",
    "print(df.isnull()) # Returns boolean DataFrame\n",
    "\n",
    "# Count missing values per column\n",
    "print(df.isnull().sum())\n",
    "# Output:\n",
    "# Name 0\n",
    "# Age 2\n",
    "# City 1\n",
    "# Salary 1\n",
    "# Department 1\n",
    "# dtype: int64\n",
    "\n",
    "# Count total missing values\n",
    "print(df.isnull().sum().sum()) # Output: 5\n",
    "\n",
    "# Percentage of missing data\n",
    "print((df.isnull().sum() / len(df) * 100).round(2))\n",
    "# Output:\n",
    "# Name 0.0\n",
    "# Age 40.0\n",
    "# City 20.0\n",
    "# Salary 20.0\n",
    "# Department 20.0\n",
    "# dtype: float64\n",
    "\n",
    "# Alternative: use .info() to see missing data\n",
    "print(df.info())\n",
    "\n",
    "# Find rows with any missing values\n",
    "rows_with_missing = df[df.isnull().any(axis=1)]\n",
    "print(rows_with_missing)\n",
    "\n",
    "# Find rows with no missing values\n",
    "complete_rows = df.dropna()\n",
    "print(complete_rows)\n",
    "\n",
    "# Find columns with missing values\n",
    "cols_with_missing = df.columns[df.isnull().any()].tolist()\n",
    "print(f\"Columns with missing values: {cols_with_missing}\")\n",
    "# Output: Columns with missing values: ['Age', 'City', 'Salary', 'Department']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea44ea70",
   "metadata": {},
   "source": [
    "### 1.2 Strategies for Handling Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5479fe",
   "metadata": {},
   "source": [
    "#### Strategy 1: Deletion (Removal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bec170a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame({\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],\n",
    "'Age': [25, np.nan, 28, 32, np.nan],\n",
    "'City': ['Lagos', 'Accra', np.nan, 'Johannesburg', 'Cape Town'],\n",
    "'Salary': [60000, 75000, 65000, np.nan, 55000]\n",
    "})\n",
    "\n",
    "# Remove rows with ANY missing values\n",
    "df_clean = df.dropna()\n",
    "print(df_clean)\n",
    "# Only Charlie and Eve remain - others have missing values\n",
    "# Note: Charlie actually has all values!\n",
    "\n",
    "# Remove rows where ALL values are missing\n",
    "df_clean = df.dropna(how='all')\n",
    "print(df_clean) # Removes entire rows only if all values are NaN\n",
    "\n",
    "# Remove rows with missing values in specific column\n",
    "df_clean = df.dropna(subset=['Age'])\n",
    "print(df_clean) # Removes rows where Age is NaN\n",
    "\n",
    "# Remove rows missing values in specific columns\n",
    "df_clean = df.dropna(subset=['Age', 'Salary'])\n",
    "print(df_clean)\n",
    "\n",
    "# Remove columns with ANY missing values\n",
    "df_clean = df.dropna(axis=1)\n",
    "print(df_clean) # Removes City and Salary columns\n",
    "\n",
    "# Remove columns where percentage of missing > threshold\n",
    "threshold = 0.3 # 30%\n",
    "df_clean = df.dropna(thresh=len(df)*(1-threshold), axis=1)\n",
    "print(df_clean)\n",
    "\n",
    "# When to use deletion:\n",
    "# - Very few missing values (< 5%)\n",
    "# - Missing values are random (MCAR)\n",
    "# - You have plenty of data\n",
    "# When NOT to use:\n",
    "# - Missing data represents important information\n",
    "# - Large percentage of data is missing\n",
    "# - You need to preserve all observations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecb78d26",
   "metadata": {},
   "source": [
    "#### Strategy 2: Imputation (Filling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba86b08b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame({\n",
    "'Product': ['A', 'B', 'C', 'D', 'E'],\n",
    "'Price': [10.0, np.nan, 15.0, np.nan, 20.0],\n",
    "'Quantity': [100, 150, np.nan, 200, 250]\n",
    "})\n",
    "\n",
    "# Fill with constant value\n",
    "df_filled = df.fillna(0)\n",
    "print(df_filled)\n",
    "\n",
    "# Fill with specific value per column\n",
    "df_filled = df.fillna({'Price': 12.5, 'Quantity': 175})\n",
    "print(df_filled)\n",
    "\n",
    "# Forward fill: use previous value\n",
    "df_filled = df.fillna(method='ffill')\n",
    "print(df_filled)\n",
    "\n",
    "# Backward fill: use next value\n",
    "df_filled = df.fillna(method='bfill')\n",
    "print(df_filled)\n",
    "\n",
    "# Fill with mean (for numerical columns)\n",
    "df['Price'].fillna(df['Price'].mean(), inplace=True)\n",
    "print(df)\n",
    "\n",
    "# Fill with median (robust to outliers)\n",
    "df['Quantity'].fillna(df['Quantity'].median(), inplace=True)\n",
    "print(df)\n",
    "\n",
    "# Fill with mode (most common value - for categorical)\n",
    "df_cat = pd.DataFrame({\n",
    "'Department': ['Sales', np.nan, 'IT', 'Sales', np.nan],\n",
    "'Level': ['Junior', 'Senior', np.nan, 'Junior', 'Senior']\n",
    "})\n",
    "\n",
    "df_cat['Department'].fillna(df_cat['Department'].mode()[0], inplace=True)\n",
    "print(df_cat)\n",
    "\n",
    "# Interpolation for time series\n",
    "df_ts = pd.DataFrame({\n",
    "'Date': pd.date_range('2023-01-01', periods=5),\n",
    "'Value': [10, np.nan, 20, np.nan, 30]\n",
    "})\n",
    "\n",
    "df_ts['Value'] = df_ts['Value'].interpolate()\n",
    "print(df_ts)\n",
    "# Output: Fills NaN values with interpolated values (12, 20, 24, etc.)\n",
    "\n",
    "# When to use imputation:\n",
    "# - Moderate amount of missing data\n",
    "# - Missing data is random (MCAR/MAR)\n",
    "# - Deleting would lose important information\n",
    "# - Methods:\n",
    "# - Mean/Median: for numerical, skewed distributions\n",
    "# - Mode: for categorical\n",
    "# - Forward/Backward fill: for time series\n",
    "# - Interpolation: for continuous sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91fac877",
   "metadata": {},
   "source": [
    "### 1.3 Handling Duplicate Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b415afd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Create dataset with duplicates\n",
    "df = pd.DataFrame({\n",
    "'ID': [1, 2, 2, 3, 3, 3, 4],\n",
    "'Name': ['Alice', 'Bob', 'Bob', 'Charlie', 'Charlie', 'Charlie', 'Diana'],\n",
    "'Score': [85, 92, 92, 78, 78, 78, 95]\n",
    "})\n",
    "\n",
    "print(df)\n",
    "\n",
    "# Find duplicate rows\n",
    "duplicates = df.duplicated()\n",
    "print(duplicates)\n",
    "# Output: [False False True False True True False]\n",
    "\n",
    "# Count duplicates\n",
    "print(f\"Number of duplicates: {duplicates.sum()}\")\n",
    "\n",
    "# See duplicate rows\n",
    "print(df[df.duplicated(keep=False)]) # keep=False shows all duplicates\n",
    "\n",
    "# Find duplicates based on specific columns\n",
    "duplicates = df.duplicated(subset=['ID'])\n",
    "print(df[duplicates])\n",
    "\n",
    "# Remove duplicates: keep first occurrence (default)\n",
    "df_clean = df.drop_duplicates()\n",
    "print(df_clean)\n",
    "\n",
    "# Remove duplicates: keep last occurrence\n",
    "df_clean = df.drop_duplicates(keep='last')\n",
    "print(df_clean)\n",
    "\n",
    "# Remove duplicates based on specific columns\n",
    "df_clean = df.drop_duplicates(subset=['ID'])\n",
    "print(df_clean)\n",
    "\n",
    "# Remove duplicates in specific columns but keep all columns\n",
    "df_clean = df.drop_duplicates(subset=['Name', 'Score'], keep='first')\n",
    "print(df_clean)\n",
    "\n",
    "# Real-world: Check for duplicate entries in customer database\n",
    "customers = pd.DataFrame({\n",
    "'Email': ['alice@example.com', 'bob@example.com', 'alice@example.com'],\n",
    "'Name': ['Alice', 'Bob', 'Alice'],\n",
    "'Purchase': [100, 200, 150]\n",
    "})\n",
    "\n",
    "# Find duplicate emails\n",
    "print(customers[customers.duplicated(subset=['Email'], keep=False)])\n",
    "\n",
    "# Remove duplicate emails, keeping record with highest purchase\n",
    "customers = customers.sort_values('Purchase', ascending=False)\n",
    "customers_clean = customers.drop_duplicates(subset=['Email'], keep='first')\n",
    "print(customers_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9377b2ff",
   "metadata": {},
   "source": [
    "### 1.4 Detecting and Handling Outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abff8ec9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create dataset with outliers\n",
    "df = pd.DataFrame({\n",
    "'Age': [25, 28, 30, 32, 35, 150, 29, 31], # 150 is outlier\n",
    "'Salary': [50000, 55000, 60000, 65000, 70000, 75000, 80000, 5000000] # 5000000 is outlier\n",
    "})\n",
    "\n",
    "print(df)\n",
    "\n",
    "# Method 1: IQR (Interquartile Range) - Most common\n",
    "Q1 = df['Age'].quantile(0.25)\n",
    "Q3 = df['Age'].quantile(0.75)\n",
    "IQR = Q3 - Q1\n",
    "\n",
    "lower_bound = Q1 - 1.5 * IQR\n",
    "upper_bound = Q3 + 1.5 * IQR\n",
    "\n",
    "outliers = df[(df['Age'] < lower_bound) | (df['Age'] > upper_bound)]\n",
    "print(f\"Outliers in Age: {len(outliers)} found\")\n",
    "print(outliers)\n",
    "\n",
    "# Remove outliers\n",
    "df_clean = df[(df['Age'] >= lower_bound) & (df['Age'] <= upper_bound)]\n",
    "print(df_clean)\n",
    "\n",
    "# Method 2: Z-score\n",
    "from scipy import stats\n",
    "\n",
    "z_scores = np.abs(stats.zscore(df['Salary']))\n",
    "outliers = df[z_scores > 3] # Threshold = 3\n",
    "print(f\"\\nOutliers (Z-score > 3): {len(outliers)}\")\n",
    "print(outliers)\n",
    "\n",
    "# Method 3: Visualization\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df.boxplot(column='Salary')\n",
    "plt.show() # Visual inspection\n",
    "\n",
    "# Real-world: Detect unusual transactions\n",
    "transactions = pd.DataFrame({\n",
    "'Amount': [50, 75, 100, 80, 120, 50000, 90, 110, 95],\n",
    "'Category': ['Food', 'Transport', 'Food', 'Shopping', 'Food', 'Wire', 'Transport', 'Food', 'Shopping']\n",
    "})\n",
    "\n",
    "# IQR method for Amount\n",
    "Q1 = transactions['Amount'].quantile(0.25)\n",
    "Q3 = transactions['Amount'].quantile(0.75)\n",
    "IQR = Q3 - Q1\n",
    "\n",
    "outlier_transactions = transactions[\n",
    "(transactions['Amount'] < Q1 - 1.5*IQR) |\n",
    "(transactions['Amount'] > Q3 + 1.5*IQR)\n",
    "]\n",
    "print(\"Unusual transactions:\")\n",
    "print(outlier_transactions)\n",
    "\n",
    "# Handle outliers - Options:\n",
    "# 1. Remove them\n",
    "# 2. Cap them (set to max reasonable value)\n",
    "# 3. Transform them (log transformation)\n",
    "# 4. Keep them (if legitimate data)\n",
    "\n",
    "# Cap outliers at 95th percentile\n",
    "cap_value = transactions['Amount'].quantile(0.95)\n",
    "transactions['Amount_capped'] = transactions['Amount'].clip(upper=cap_value)\n",
    "print(transactions[['Amount', 'Amount_capped']])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e8e9727",
   "metadata": {},
   "source": [
    "## SESSION 2: Feature Scaling, Normalization, and Type Conversion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc4029f0",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3efd0f2",
   "metadata": {},
   "source": [
    "### 2.1 Data Type Conversion and Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7a08bf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create dataset with wrong data types\n",
    "df = pd.DataFrame({\n",
    "'ID': ['1', '2', '3', '4'], # Should be int\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],\n",
    "'Age': [25, 30, 28, 32],\n",
    "'Salary': ['50000', '75000', '60000', '80000'], # Should be float\n",
    "'HireDate': ['2020-01-15', '2019-05-20', '2021-03-10', '2018-11-30'], # Should be datetime\n",
    "'IsManager': ['True', 'False', 'True', 'False'] # Should be bool\n",
    "})\n",
    "\n",
    "print(df.dtypes)\n",
    "# Output:\n",
    "# ID object\n",
    "# Name object\n",
    "# Age int64\n",
    "# Salary object (but should be numeric)\n",
    "# HireDate object (but should be datetime)\n",
    "# IsManager object (but should be bool)\n",
    "\n",
    "# Convert to correct types\n",
    "df['ID'] = df['ID'].astype('int')\n",
    "df['Salary'] = df['Salary'].astype('float')\n",
    "df['HireDate'] = pd.to_datetime(df['HireDate'])\n",
    "df['IsManager'] = df['IsManager'].astype('bool')\n",
    "\n",
    "print(df.dtypes)\n",
    "\n",
    "# Alternative: astype with errors handling\n",
    "df['ID'] = pd.to_numeric(df['ID'], errors='coerce') # NaN if conversion fails\n",
    "\n",
    "# Convert to category (useful for memory and speed)\n",
    "df['Name'] = df['Name'].astype('category')\n",
    "\n",
    "# Working with datetime\n",
    "print(df['HireDate'].dt.year) # Extract year\n",
    "print(df['HireDate'].dt.month) # Extract month\n",
    "print(df['HireDate'].dt.day) # Extract day\n",
    "\n",
    "# Calculate days employed\n",
    "df['DaysEmployed'] = (pd.Timestamp.now() - df['HireDate']).dt.days\n",
    "print(df)\n",
    "\n",
    "# Real-world: Process raw data\n",
    "raw_data = pd.DataFrame({\n",
    "'TransactionID': ['A001', 'A002', 'A003'],\n",
    "'Amount': ['1500.50', '2300.75', 'Invalid'], # One invalid\n",
    "'Date': ['2023-01-15', '2023-01-16', '2023-01-17'],\n",
    "'Status': ['Complete', 'Pending', 'Complete']\n",
    "})\n",
    "\n",
    "# Safe conversion\n",
    "raw_data['Amount'] = pd.to_numeric(raw_data['Amount'], errors='coerce')\n",
    "raw_data['Date'] = pd.to_datetime(raw_data['Date'])\n",
    "raw_data['TransactionID'] = raw_data['TransactionID'].astype('str')\n",
    "\n",
    "# Remove rows with conversion errors (NaN values)\n",
    "raw_data_clean = raw_data.dropna(subset=['Amount'])\n",
    "print(raw_data_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9e69b93",
   "metadata": {},
   "source": [
    "### 2.2 Feature Scaling and Normalization\n",
    "Scaling is essential because:\n",
    "- Different features have different ranges\n",
    "- Some algorithms are sensitive to feature scale\n",
    "- Makes comparison between features meaningful\n",
    "- Speeds up convergence in optimization algorithms\n",
    "\n",
    "Common approaches:\n",
    "- Standardization (Z-score normalization)\n",
    "- Min-Max scaling (Normalization)\n",
    "- Robust scaling (uses median and IQR)\n",
    "- Log scaling (for skewed distributions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85351375",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler\n",
    "\n",
    "# Create dataset with different scales\n",
    "df = pd.DataFrame({\n",
    "'Age': [25, 30, 28, 35, 42],\n",
    "'Salary': [50000, 75000, 60000, 90000, 120000],\n",
    "'Experience': [2, 5, 3, 8, 15]\n",
    "})\n",
    "\n",
    "print(\"Original data:\")\n",
    "print(df)\n",
    "print(\"\\nDescriptive stats:\")\n",
    "print(df.describe())\n",
    "\n",
    "# Method 1: Standardization (Z-score)\n",
    "# Formula: (x - mean) / std\n",
    "scaler = StandardScaler()\n",
    "df_standardized = scaler.fit_transform(df)\n",
    "df_standardized = pd.DataFrame(df_standardized, columns=df.columns)\n",
    "print(\"\\nStandardized (Z-score):\")\n",
    "print(df_standardized)\n",
    "print(df_standardized.describe())\n",
    "\n",
    "# Manual standardization\n",
    "df_manual = (df - df.mean()) / df.std()\n",
    "print(\"\\nManual standardization:\")\n",
    "print(df_manual)\n",
    "\n",
    "# Method 2: Min-Max Scaling (Normalization)\n",
    "# Formula: (x - min) / (max - min)\n",
    "# Results in range [0, 1]\n",
    "scaler = MinMaxScaler()\n",
    "df_minmax = scaler.fit_transform(df)\n",
    "df_minmax = pd.DataFrame(df_minmax, columns=df.columns)\n",
    "print(\"\\nMin-Max Scaled:\")\n",
    "print(df_minmax)\n",
    "\n",
    "# Manual min-max scaling\n",
    "df_minmax_manual = (df - df.min()) / (df.max() - df.min())\n",
    "print(df_minmax_manual)\n",
    "\n",
    "# Method 3: Robust Scaling (uses median and IQR)\n",
    "# Better for data with outliers\n",
    "scaler = RobustScaler()\n",
    "df_robust = scaler.fit_transform(df)\n",
    "df_robust = pd.DataFrame(df_robust, columns=df.columns)\n",
    "print(\"\\nRobust Scaled:\")\n",
    "print(df_robust)\n",
    "\n",
    "# Method 4: Log Scaling\n",
    "# For right-skewed data\n",
    "df_log = np.log(df + 1) # Add 1 to avoid log(0)\n",
    "print(\"\\nLog Scaled:\")\n",
    "print(df_log)\n",
    "\n",
    "# When to use each:\n",
    "# - Standardization: Most common, use before ML algorithms\n",
    "# - Min-Max: When you need values in specific range [0,1]\n",
    "# - Robust: When data has outliers\n",
    "# - Log: When data is right-skewed\n",
    "\n",
    "# Real-world: Prepare features for machine learning\n",
    "df = pd.DataFrame({\n",
    "'Height_cm': [170, 165, 180, 175, 168],\n",
    "'Weight_kg': [70, 60, 85, 75, 65],\n",
    "'Age': [25, 30, 28, 35, 42]\n",
    "})\n",
    "\n",
    "# Standardize all features\n",
    "scaler = StandardScaler()\n",
    "features_scaled = scaler.fit_transform(df)\n",
    "df_scaled = pd.DataFrame(features_scaled, columns=df.columns)\n",
    "\n",
    "print(\"Original vs Scaled:\")\n",
    "print(pd.DataFrame({\n",
    "'Height_original': df['Height_cm'],\n",
    "'Height_scaled': df_scaled['Height_cm'],\n",
    "'Weight_original': df['Weight_kg'],\n",
    "'Weight_scaled': df_scaled['Weight_kg']\n",
    "}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b83756f0",
   "metadata": {},
   "source": [
    "### 2.3 Handling Categorical Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a658aaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create dataset with categorical variables\n",
    "df = pd.DataFrame({\n",
    "'Product': ['Laptop', 'Phone', 'Tablet', 'Laptop', 'Phone'],\n",
    "'Brand': ['Dell', 'Apple', 'Samsung', 'HP', 'Apple'],\n",
    "'Size': ['Small', 'Medium', 'Large', 'Small', 'Medium'],\n",
    "'Price': [50000, 70000, 20000, 45000, 65000]\n",
    "})\n",
    "\n",
    "print(\"Original data:\")\n",
    "print(df)\n",
    "print(df.dtypes)\n",
    "\n",
    "# Method 1: Label Encoding\n",
    "# Assigns integer to each category\n",
    "# Use when: ordinal data (order matters)\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "le = LabelEncoder()\n",
    "df['Product_encoded'] = le.fit_transform(df['Product'])\n",
    "print(\"\\nLabel Encoding:\")\n",
    "print(df[['Product', 'Product_encoded']])\n",
    "# Laptop -> 0, Phone -> 1, Tablet -> 2 (alphabetical order)\n",
    "\n",
    "# Get mapping\n",
    "mapping = dict(zip(le.classes_, le.transform(le.classes_)))\n",
    "print(f\"Mapping: {mapping}\")\n",
    "\n",
    "# Method 2: One-Hot Encoding\n",
    "# Creates binary column for each category\n",
    "# Use for: nominal data (no order)\n",
    "df_onehot = pd.get_dummies(df['Product'], prefix='Product')\n",
    "print(\"\\nOne-Hot Encoding:\")\n",
    "print(df_onehot)\n",
    "\n",
    "# Drop original and combine\n",
    "df_encoded = pd.concat([df[['Price']], df_onehot], axis=1)\n",
    "print(df_encoded)\n",
    "\n",
    "# Alternative with drop_first (for avoiding multicollinearity)\n",
    "df_onehot_drop = pd.get_dummies(df['Product'], prefix='Product', drop_first=True)\n",
    "print(\"\\nOne-Hot Encoding (drop first):\")\n",
    "print(df_onehot_drop)\n",
    "\n",
    "# Method 3: Ordinal Encoding\n",
    "# For ordinal categorical data\n",
    "size_mapping = {'Small': 1, 'Medium': 2, 'Large': 3}\n",
    "df['Size_encoded'] = df['Size'].map(size_mapping)\n",
    "print(\"\\nOrdinal Encoding:\")\n",
    "print(df[['Size', 'Size_encoded']])\n",
    "\n",
    "# Real-world: Prepare mixed data for modeling\n",
    "df_raw = pd.DataFrame({\n",
    "'Age': [25, 30, 28, 35, 42],\n",
    "'Gender': ['M', 'F', 'M', 'F', 'M'],\n",
    "'Department': ['Sales', 'IT', 'Sales', 'Finance', 'IT'],\n",
    "'Salary': [50000, 75000, 60000, 90000, 120000]\n",
    "})\n",
    "\n",
    "# Encode categorical variables\n",
    "df_processed = df_raw.copy()\n",
    "df_processed = pd.get_dummies(df_processed, columns=['Gender', 'Department'], drop_first=True)\n",
    "\n",
    "print(\"\\nProcessed data:\")\n",
    "print(df_processed)\n",
    "print(df_processed.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db9e62b",
   "metadata": {},
   "source": [
    "## SESSION 3: Categorical Encoding and Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f0b5b73",
   "metadata": {},
   "source": [
    "### Duration: 2 hours"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1724e42",
   "metadata": {},
   "source": [
    "### 3.1 Merging and Concatenating Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "050435f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Create sample datasets\n",
    "customers = pd.DataFrame({\n",
    "'CustomerID': [1, 2, 3, 4],\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],\n",
    "'City': ['Lagos', 'Accra', 'Nairobi', 'Johannesburg']\n",
    "})\n",
    "\n",
    "orders = pd.DataFrame({\n",
    "'OrderID': [101, 102, 103, 104],\n",
    "'CustomerID': [1, 2, 1, 4],\n",
    "'Amount': [5000, 7500, 3000, 12000]\n",
    "})\n",
    "\n",
    "print(\"Customers:\")\n",
    "print(customers)\n",
    "print(\"\\nOrders:\")\n",
    "print(orders)\n",
    "\n",
    "# Method 1: Inner Merge (only matching rows)\n",
    "merged_inner = pd.merge(customers, orders, on='CustomerID', how='inner')\n",
    "print(\"\\nInner Merge:\")\n",
    "print(merged_inner)\n",
    "\n",
    "# Method 2: Left Merge (all from left, matching from right)\n",
    "merged_left = pd.merge(customers, orders, on='CustomerID', how='left')\n",
    "print(\"\\nLeft Merge:\")\n",
    "print(merged_left)\n",
    "\n",
    "# Method 3: Right Merge (matching from left, all from right)\n",
    "merged_right = pd.merge(customers, orders, on='CustomerID', how='right')\n",
    "print(\"\\nRight Merge:\")\n",
    "print(merged_right)\n",
    "\n",
    "# Method 4: Outer Merge (all from both)\n",
    "merged_outer = pd.merge(customers, orders, on='CustomerID', how='outer')\n",
    "print(\"\\nOuter Merge:\")\n",
    "print(merged_outer)\n",
    "\n",
    "# Merge on different column names\n",
    "df1 = pd.DataFrame({'ID': [1, 2, 3], 'Value': ['A', 'B', 'C']})\n",
    "df2 = pd.DataFrame({'CustomerID': [1, 2, 3], 'Score': [85, 92, 78]})\n",
    "\n",
    "merged = pd.merge(df1, df2, left_on='ID', right_on='CustomerID')\n",
    "print(\"\\nMerge with different column names:\")\n",
    "print(merged)\n",
    "\n",
    "# Concatenate along rows (like UNION in SQL)\n",
    "df_2023 = pd.DataFrame({\n",
    "'Product': ['A', 'B', 'C'],\n",
    "'Sales': [1000, 2000, 1500]\n",
    "})\n",
    "\n",
    "df_2024 = pd.DataFrame({\n",
    "'Product': ['A', 'B', 'D'],\n",
    "'Sales': [1200, 2300, 1800]\n",
    "})\n",
    "\n",
    "concat_result = pd.concat([df_2023, df_2024], ignore_index=True)\n",
    "print(\"\\nConcatenated (rows):\")\n",
    "print(concat_result)\n",
    "\n",
    "# Concatenate along columns\n",
    "concat_cols = pd.concat([df_2023, df_2024], axis=1, keys=['2023', '2024'])\n",
    "print(\"\\nConcatenated (columns):\")\n",
    "print(concat_cols)\n",
    "\n",
    "# Real-world: Combine customer and transaction data\n",
    "customers = pd.DataFrame({\n",
    "'CustID': [1, 2, 3, 4],\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana']\n",
    "})\n",
    "\n",
    "transactions = pd.DataFrame({\n",
    "'TransID': [1001, 1002, 1003, 1004, 1005],\n",
    "'CustID': [1, 1, 2, 3, 1],\n",
    "'Amount': [500, 300, 800, 1200, 400]\n",
    "})\n",
    "\n",
    "# Merge and aggregate\n",
    "result = pd.merge(customers, transactions, on='CustID', how='left')\n",
    "customer_totals = result.groupby('Name')['Amount'].sum().reset_index()\n",
    "customer_totals.columns = ['Name', 'TotalSpent']\n",
    "print(\"\\nCustomer totals:\")\n",
    "print(customer_totals)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdd3a29f",
   "metadata": {},
   "source": [
    "### 3.2 Feature Engineering and Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c058ec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create base dataset\n",
    "df = pd.DataFrame({\n",
    "'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],\n",
    "'BirthYear': [1998, 1993, 1995, 1990],\n",
    "'Salary': [50000, 75000, 60000, 90000],\n",
    "'YearsExp': [2, 10, 8, 15]\n",
    "})\n",
    "\n",
    "print(\"Original data:\")\n",
    "print(df)\n",
    "\n",
    "# Feature 1: Create derived feature (age from birth year)\n",
    "current_year = 2024\n",
    "df['Age'] = current_year - df['BirthYear']\n",
    "print(\"\\nWith Age derived from BirthYear:\")\n",
    "print(df)\n",
    "\n",
    "# Feature 2: Create ratio feature\n",
    "df['SalaryPerYear'] = df['Salary'] / df['YearsExp']\n",
    "print(\"\\nWith SalaryPerYear ratio:\")\n",
    "print(df)\n",
    "\n",
    "# Feature 3: Create categorical feature from numerical\n",
    "df['ExperienceLevel'] = pd.cut(df['YearsExp'],\n",
    "bins=[0, 5, 10, 20],\n",
    "labels=['Junior', 'Mid', 'Senior'])\n",
    "print(\"\\nWith ExperienceLevel created from YearsExp:\")\n",
    "print(df)\n",
    "\n",
    "# Feature 4: Create boolean feature\n",
    "df['IsHighEarner'] = df['Salary'] > 65000\n",
    "print(\"\\nWith IsHighEarner boolean:\")\n",
    "print(df)\n",
    "\n",
    "# Feature 5: Polynomial features\n",
    "df['Salary_squared'] = df['Salary'] ** 2\n",
    "df['Salary_sqrt'] = np.sqrt(df['Salary'])\n",
    "print(\"\\nWith polynomial features:\")\n",
    "print(df[['Salary', 'Salary_squared', 'Salary_sqrt']])\n",
    "\n",
    "# Feature 6: Binning/Discretization\n",
    "df['SalaryBand'] = pd.cut(df['Salary'],\n",
    "bins=[0, 55000, 75000, 100000],\n",
    "labels=['Low', 'Medium', 'High'])\n",
    "print(\"\\nWith SalaryBand:\")\n",
    "print(df)\n",
    "\n",
    "# Feature 7: Interaction features\n",
    "df['Age_x_Experience'] = df['Age'] * df['YearsExp']\n",
    "print(\"\\nWith interaction feature:\")\n",
    "print(df[['Age', 'YearsExp', 'Age_x_Experience']])\n",
    "\n",
    "# Real-world: Create features for customer analysis\n",
    "customers = pd.DataFrame({\n",
    "'CustomerID': [1, 2, 3, 4, 5],\n",
    "'JoinDate': pd.date_range('2020-01-01', periods=5, freq='Y'),\n",
    "'TotalPurchase': [10000, 25000, 5000, 50000, 15000],\n",
    "'LastPurchaseDate': pd.date_range('2023-01-01', periods=5, freq='M')\n",
    "})\n",
    "\n",
    "# Feature: Customer tenure (in days)\n",
    "customers['Tenure_days'] = (pd.Timestamp.now() - customers['JoinDate']).dt.days\n",
    "\n",
    "# Feature: Days since last purchase\n",
    "customers['DaysSinceLastPurchase'] = (pd.Timestamp.now() - customers['LastPurchaseDate']).dt.days\n",
    "\n",
    "# Feature: Customer value segment\n",
    "customers['ValueSegment'] = pd.cut(customers['TotalPurchase'],\n",
    "bins=[0, 10000, 30000, float('inf')],\n",
    "labels=['Low', 'Medium', 'High'])\n",
    "\n",
    "# Feature: Churn risk (high days since purchase = high risk)\n",
    "customers['ChurnRisk'] = customers['DaysSinceLastPurchase'] > 180\n",
    "\n",
    "print(\"\\nCustomer features:\")\n",
    "print(customers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d491fa",
   "metadata": {},
   "source": [
    "### 3.3 Building a Complete Data Cleaning Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bd330df",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Step 1: Load and explore data\n",
    "raw_data = pd.DataFrame({\n",
    "'ID': [1, 2, 3, 4, 5, 6, 7, 8],\n",
    "'Name': ['Alice', 'Bob', None, 'Diana', 'Eve', 'Frank', 'Grace', 'Henry'],\n",
    "'Age': [25, 30, 28, np.nan, 35, 25, 28, 40],\n",
    "'Salary': ['50000', '75000', '60000', '80000', 'Unknown', '55000', '70000', '95000'],\n",
    "'Department': ['Sales', 'IT', 'IT', 'Finance', 'Sales', 'IT', 'Finance', 'Sales'],\n",
    "'StartDate': ['2020-01-15', '2019-05-20', '2021-03-10', '2018-11-30',\n",
    "'2020-07-10', '2020-01-15', '2019-06-01', '2017-02-15']\n",
    "})\n",
    "\n",
    "print(\"Step 1: Original Data\")\n",
    "print(raw_data)\n",
    "print(f\"\\nShape: {raw_data.shape}\")\n",
    "print(f\"Missing values: {raw_data.isnull().sum().sum()}\")\n",
    "\n",
    "# Step 2: Handle missing values\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 2: Handle Missing Values\")\n",
    "raw_data['Name'].fillna('Unknown', inplace=True)\n",
    "raw_data['Age'].fillna(raw_data['Age'].median(), inplace=True)\n",
    "\n",
    "# Convert salary and handle invalid values\n",
    "raw_data['Salary'] = pd.to_numeric(raw_data['Salary'], errors='coerce')\n",
    "raw_data['Salary'].fillna(raw_data['Salary'].median(), inplace=True)\n",
    "\n",
    "print(raw_data)\n",
    "\n",
    "# Step 3: Handle data types\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 3: Convert Data Types\")\n",
    "raw_data['ID'] = raw_data['ID'].astype('int')\n",
    "raw_data['Age'] = raw_data['Age'].astype('int')\n",
    "raw_data['Salary'] = raw_data['Salary'].astype('float')\n",
    "raw_data['StartDate'] = pd.to_datetime(raw_data['StartDate'])\n",
    "raw_data['Department'] = raw_data['Department'].astype('category')\n",
    "\n",
    "print(raw_data.dtypes)\n",
    "\n",
    "# Step 4: Remove duplicates\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 4: Remove Duplicates\")\n",
    "initial_rows = len(raw_data)\n",
    "raw_data = raw_data.drop_duplicates()\n",
    "print(f\"Removed {initial_rows - len(raw_data)} duplicate rows\")\n",
    "\n",
    "# Step 5: Handle outliers\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 5: Handle Outliers\")\n",
    "Q1 = raw_data['Age'].quantile(0.25)\n",
    "Q3 = raw_data['Age'].quantile(0.75)\n",
    "IQR = Q3 - Q1\n",
    "lower_bound = Q1 - 1.5 * IQR\n",
    "upper_bound = Q3 + 1.5 * IQR\n",
    "outliers = raw_data[(raw_data['Age'] < lower_bound) | (raw_data['Age'] > upper_bound)]\n",
    "print(f\"Found {len(outliers)} age outliers: {outliers['Age'].tolist()}\")\n",
    "# Option: Remove outliers\n",
    "# raw_data = raw_data[(raw_data['Age'] >= lower_bound) & (raw_data['Age'] <= upper_bound)]\n",
    "\n",
    "# Step 6: Create features\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 6: Create Features\")\n",
    "raw_data['Tenure_years'] = (pd.Timestamp.now() - raw_data['StartDate']).dt.days // 365\n",
    "raw_data['SalaryPerYear'] = raw_data['Salary'] / (raw_data['Tenure_years'] + 1)\n",
    "print(raw_data[['Name', 'StartDate', 'Tenure_years', 'Salary', 'SalaryPerYear']])\n",
    "\n",
    "# Step 7: Encode categorical variables\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 7: Encode Categorical Variables\")\n",
    "raw_data_encoded = pd.get_dummies(raw_data, columns=['Department'], drop_first=True)\n",
    "print(raw_data_encoded.columns.tolist())\n",
    "\n",
    "# Step 8: Scale numerical features\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 8: Scale Numerical Features\")\n",
    "scaler = StandardScaler()\n",
    "numerical_cols = ['Age', 'Salary', 'Tenure_years', 'SalaryPerYear']\n",
    "raw_data_encoded[numerical_cols] = scaler.fit_transform(raw_data_encoded[numerical_cols])\n",
    "\n",
    "print(\"\\nFinal processed data:\")\n",
    "print(raw_data_encoded.head())\n",
    "\n",
    "# Step 9: Save cleaned data\n",
    "print(\"\\n\" + \"=\"*50)\n",
    "print(\"Step 9: Save Cleaned Data\")\n",
    "raw_data_encoded.to_csv('cleaned_data.csv', index=False)\n",
    "print(\"Cleaned data saved to 'cleaned_data.csv'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a6363e",
   "metadata": {},
   "source": [
    "## Week 4 Summary\n",
    "By completing Week 4, you have learned:\n",
    "- Identifying missing data: types, causes, and detection methods\n",
    "- Missing data strategies: deletion, mean/median/mode imputation, forward/backward fill\n",
    "- Detecting and removing duplicate records\n",
    "- Identifying outliers using IQR, Z-score, and visualization\n",
    "- Handling outliers: removal, capping, transformation\n",
    "- Converting data types appropriately (int, float, datetime, category)\n",
    "- Understanding feature scaling and normalization\n",
    "- Standardization (Z-score): (x - mean) / std\n",
    "- Min-Max scaling: (x - min) / (max - min)\n",
    "- Robust scaling for data with outliers\n",
    "- Label encoding for ordinal categorical data\n",
    "- One-hot encoding for nominal categorical data\n",
    "- Merging and concatenating DataFrames (inner, left, right, outer joins)\n",
    "- Feature engineering: creating derived features\n",
    "- Building complete data cleaning pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f42de488",
   "metadata": {},
   "source": [
    "## Week 4 Assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fafc929",
   "metadata": {},
   "source": [
    "### Assignment 1: Missing Data Handling\n",
    "Work with a dataset containing missing values:\n",
    "- Load a dataset with intentional missing values (10-20%)\n",
    "- Identify all missing values and their patterns\n",
    "- Create multiple cleaned versions using different strategies (deletion, mean, median)\n",
    "- Compare and justify which approach is best for each column\n",
    "- Document your decisions with explanations\n",
    "- Save the cleaned dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ef6287a",
   "metadata": {},
   "source": [
    "### Assignment 2: Feature Scaling and Encoding\n",
    "Prepare features for machine learning:\n",
    "- Load a dataset with mixed data types (numerical and categorical)\n",
    "- Identify all data type issues and convert appropriately\n",
    "- Detect and handle outliers in numerical columns\n",
    "- Scale numerical features using multiple methods\n",
    "- Encode categorical variables appropriately\n",
    "- Create a final processed dataset ready for modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58ed2132",
   "metadata": {},
   "source": [
    "### Assignment 3: Complete Data Cleaning Pipeline\n",
    "Build an end-to-end pipeline:\n",
    "- Download a real \"messy\" dataset (from Kaggle or similar)\n",
    "- Apply all cleaning techniques: missing values, duplicates, outliers, type conversion\n",
    "- Create meaningful features from existing columns\n",
    "- Merge with supplementary data if available\n",
    "- Scale and encode appropriately\n",
    "- Generate a detailed data quality report\n",
    "- Save cleaned data and provide analysis summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d03079a",
   "metadata": {},
   "source": [
    "## Recommended Practice Problems\n",
    "- Handle missing data in time series (forward fill, interpolation)\n",
    "- Create custom encoding schemes for domain-specific categorical data\n",
    "- Build a function that automatically detects and handles outliers\n",
    "- Practice different merge types and understand when each is appropriate\n",
    "- Create polynomial features and interaction features\n",
    "- Handle missing data that is MNAR (not missing at random)\n",
    "- Work with datasets having &gt;50% missing data\n",
    "- Create a robust data validation function\n",
    "- Standardize multiple datasets using the same scaler (train/test split)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0ccbb77",
   "metadata": {},
   "source": [
    "## Additional Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c60292c",
   "metadata": {},
   "source": [
    "### Books\n",
    "- Chapter 7: Data Cleaning and Preparation - &quot;Python for Data Analysis&quot; by Wes McKinney\n",
    "- Data Wrangling with pandas, NumPy, and IPython by Wes McKinney"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27624fc7",
   "metadata": {},
   "source": [
    "### Online Documentation\n",
    "- pandas Missing Data: https://pandas.pydata.org/docs/user_guide/missing_data.html\n",
    "- scikit-learn Preprocessing: https://scikit-learn.org/stable/modules/preprocessing.html\n",
    "- pandas Merge/Join: https://pandas.pydata.org/docs/user_guide/merging.html\n",
    "- Feature Engineering Guide: https://en.wikipedia.org/wiki/Feature_engineering"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}