🤖 AI/ML Learning Series

Tracks: CNNs → Transfer Learning → RNNs/LSTMs → NLP Feature Eng. & Ensembling → ML Theory → Decision Trees & Ensembles → Math/NumPy/PyTorch Foundations

1. CNNs & AlexNet

1.1 What are CNNs?

- But what is a convolution? (3Blue1Brown) — https://www.youtube.com/watch?v=KuXjwB4LzSA

- NN Part 8: Image Classification with CNNs — https://www.youtube.com/watch?v=HGwBXDKFk9I

1.2 Basic CNN Code-Along

- CNN_Explainer.ipynb (NumPy from scratch) — https://colab.research.google.com/drive/1IOPQ-LyMqjE6v-YuXO1klJZUOAKKMddY

- (Optional) LeNet-5 paper — http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf

1.3 Dropout

- CS7015 Lec 8.11 — Dropout — https://youtu.be/qfsacbIe9AI

1.4 SGD with Momentum

- Momentum Gradient Descent (3 min) — https://youtu.be/iudXf5n_3ro

- SGD with Momentum — Explained (PDF) — https://www.khoury.northeastern.edu/home/vip/teach/DMcourse/3_dim_reduction/tSNE/tsne_matlab/SGD_Momentum.pdf

- SGD w/ Momentum — Medium (math) — https://medium.com/data-science/stochastic-gradient-descent-with-momentum-a84097641a5d

- (Optional) Why Momentum Really Works (distill.pub) — https://distill.pub/2017/momentum/

1.5 PCA (Data Augmentation)

- StatQuest: PCA Step-by-Step

- StatQuest: PCA in 5 Minutes

- PCA — GeeksforGeeks — https://www.geeksforgeeks.org/principal-component-analysis-pca/

- (Optional) What is PCA Color Augmentation? — https://aparico.github.io/

1.6 AlexNet Paper

- ImageNet Classification with Deep CNNs (original paper) — https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf

1.7 AlexNet From Scratch

- CNN_week6.ipynb — https://colab.research.google.com/drive/1g5qgICAVmhpjSjTecwPmSB4P2-P4mZGH

1.8 AlexNet in PyTorch

- Building AlexNet in PyTorch (Part 2) — https://youtu.be/W9s-4UwlIcU

- Sample FashionMNIST AlexNet (Kaggle) — https://www.kaggle.com/code/aarohiiiy/fahionmnist-alexne

1.9 DIY Project

- Build digit recognition w/ AlexNet on MNIST — https://www.kaggle.com/datasets/hojjatk/mnist-dataset

2. Transfer Learning

- Part 1 — What is Transfer Learning? (GFG) — https://www.geeksforgeeks.org/machine-learning/ml-introduction-to-transfer-learning/

- Part 2 — Fine-Tuning vs Feature Extraction (Keras) — https://www.geeksforgeeks.org/machine-learning/ml-introduction-to-transfer-learning/

- Part 3 — Feature Extraction & Finetuning (Human Emotions) — https://www.youtube.com/watch?v=1ENyc24yOYs

- Practical — model.py on CIFAR-10 — https://drive.google.com/file/d/1zd1UmyVBDSNxSkGrdJ3wMDJLvsQYCU_d

- Alongside: CMU Lectures — https://youtube.com/playlist?list=PL7k0r4t5c10-g7CWCnHfZOAxLaiNinChk

3. RNNs & LSTMs

3.1 RNN Fundamentals

- RNN Document — https://harashish-arora.github.io/RNNs.pdf

- StatQuest: RNNs, Clearly Explained

- RNN Cheatsheet — CS230 — https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks

- BPTT for RNNs — https://mmuratarat.github.io/2019-02-07/bptt-of-rnn

- (Extra) Deep Learning Book — RNN Chapter — https://www.deeplearningbook.org/contents/rnn.html

3.2 Implementing RNNs

- PyTorch RNN Tutorial — Name Classification — https://docs.pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

- NLP From Scratch: Classifying Names — (same link as above)

- NLP From Scratch: Generating Names — https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html

3.3 LSTMs

- LSTM Paper

- StatQuest: LSTM, Clearly Explained

- What is LSTM? (GFG) — https://www.geeksforgeeks.org/deep-learning-introduction-to-long-short-term-memory/

- Understanding LSTM Networks (colah) — https://colah.github.io/posts/2015-08-Understanding-LSTMs/

- PyTorch Tutorial — RNN/LSTM/GRU

3.4 Word Embeddings

- A Complete Overview of Word Embeddings

- Word Embedding in PyTorch (GFG) — https://www.geeksforgeeks.org/word-embedding-in-pytorch/

- Sequence Models & LSTM Networks (PyTorch) — https://docs.pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html

- What Are Word Embeddings?

3.5 Project Ideas

- Sentiment Analysis on Movie Reviews (Kaggle) — https://www.kaggle.com/competitions/sentiment-analysis-on-movie-reviews/overview

- Music Composition with RNN (report) — https://cs229.stanford.edu/proj2016/report/Wang-MusicCompositionWithRNN-report.pdf

4. NLP Feature Engineering & Model Ensembling

Island 1 — Foundations

- CMU Lectures (5–6) — https://youtube.com/playlist?list=PL7k0r4t5c10-g7CWCnHfZOAxLaiNinChk

Island 2 — Preprocessing

- N-grams (worked example) — https://chatgpt.com/share/6846f31f-c6d0-8004-9c92-3c5179642d8d

- TF-IDF (Capital One) — https://www.capitalone.com/tech/machine-learning/understanding-tf-idf/

- SVD — https://www.askpython.com/python/examples/singular-value-decomposition

- (Optional) Distance Features

Landmarks: Drop HTML tags · Word replacement · Stemming · N-grams · TF-IDF · Distance features (optional)

Island 3 — Cross-Validation

- Cross Validation (GFG) — https://www.geeksforgeeks.org/cross-validation-machine-learning/

- Stratified K-Fold (sklearn) — https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html

Optional metrics: Pairwise ranking · Ordinal regression · Soft Kappa · Variance as confidence

Island 4 — Model Training

- Random Forest Regressor

- XGBoost (gblinear) config — https://xgboosting.com/configure-xgboost-linear-booster-gblinear/

- XGBoost (gblinear) Notebook — https://colab.research.google.com/drive/1TJ9qiF5pQdjv3KETi9LgQpL0gbLzMSvG

- Neural Network (Keras)

Landmarks: Eval metric fn · Stratified K-Fold · Train different models · Weighted training via variance

Island 5 — Ensembling

- Ensemble Learning Guide (GFG) — https://www.geeksforgeeks.org/a-comprehensive-guide-to-ensemble-learning/

- Stacking (GFG) — https://www.geeksforgeeks.org/stacking-in-machine-learning/

- Stacking — video — https://youtu.be/O-aDHBGMqXA

- Sample Stacking Project — https://colab.research.google.com/drive/14q-8AjRUv7QlulLGCkG6Q2_dzLIgwBLv

Landmarks: Stack models · Evaluate on test data · Add quadratic weighted kappa

🏁 Capstone: Crowdflower Search Relevance (Kaggle) — https://www.kaggle.com/c/crowdflower-search-relevance

5. ML Theory Fundamentals

CMU Lectures (1–5) — https://youtube.com/playlist?list=PL7k0r4t5c10-g7CWCnHfZOAxLaiNinChk

Stage 1 — Core Concepts

- Some Concept Excerpts

- Statistical Estimators — what is an estimator?

- Estimator — Wikipedia (Quantified Properties section) — https://en.wikipedia.org/wiki/Estimator#Quantified_properties

- Law of Large Numbers — Explained & Visualized

- LLN (Built In) — https://builtin.com/data-science/law-of-large-numbers

- Consistency & Convergence

- But What Is the CLT?

- (Optional) Two Proofs of CLT — https://www.cs.toronto.edu/~yuvalf/CLT.pdf

Stage 2 — Bias, Variance & Regularisation

- ISL Excerpt — estimator_intro.pdf

- Bias-Variance Tradeoff thread — https://x.com/mdancho84/status/1928475379548749912

- [Proof] MSE = Variance + Bias²

- Empirical Risk Minimization (first 3 pages) — https://people.cs.umass.edu/~domke/courses/sml2011/03optimization.pdf

- ERM — Wikipedia — https://en.wikipedia.org/wiki/Empirical_risk_minimization

- (Optional) 7 Common Loss Functions — https://builtin.com/machine-learning/common-loss-functions

- (Optional) train_test_split (Real Python) — https://realpython.com/train-test-split-python-data/

- Cross Validation (GFG) — https://www.geeksforgeeks.org/cross-validation-machine-learning/

- L1/L2 Regularization Explained — https://spotintelligence.com/2023/05/26/l1-l2-regularization/

- Lasso & Ridge in Python (DataCamp) — https://www.datacamp.com/tutorial/tutorial-lasso-ridge-regression

- (Optional) ISLP pp. 248–261

- Linear Regression in Python — https://realpython.com/linear-regression-in-python/#problem-formulation

- DIY 1 — Polynomial Regression Notebook — https://harvard-iacs.github.io/2021-CS109A/lectures/lecture06/notebook-2/

- DIY 2 — Regularisation Experiment (same notebook, degree ~1.5x)

- Logistic Regression in Python (skip digit-classification section) — https://realpython.com/logistic-regression-python/#regularization

- DIY 3 — MNIST Logistic Regression (80/20 split) — https://www.kaggle.com/datasets/hojjatk/mnist-dataset

Stage 3 — Optimisation & Convexity

- (Optional) Optimisation & Gradient Descent (2x speed) — https://www.youtube.com/watch?v=sDv4f4s2SB8

- Gradient Descent Math (matrix form) — https://www.youtube.com/watch?v=9H-s5cQ1iBk

- Intro to Convex Optimisation — https://www.youtube.com/watch?v=AM6BY4btj-M

- Convex Sets, Functions & Optimizations — https://www.youtube.com/watch?v=d0CF3d5aEGc

- Max-Min Inequality, Strong & Weak Duality — https://www.youtube.com/watch?v=oLowhs83aHk

- DL Book — Numerical Computation (from 4.4) — https://www.deeplearningbook.org/contents/numerical.html

- (Optional) Complete Guide to Convexity — https://www.numberanalytics.com/blog/complete-guide-convexity-machine-learning

- KKT & Interior Point Optimization — https://www.youtube.com/watch?v=uh1Dk68cfWs

- (Optional) CS229 Convex Optimization Notes (p.1–10) — https://cs229.stanford.edu/section/cs229-cvxopt.pdf

Stage 4 — Model Mechanics & Evaluation

- (Optional) Closed-Form Solution for Linear Regression — https://www.youtube.com/watch?v=VEluK6Mp340

- (Optional) Sigmoid Function walkthrough — https://www.youtube.com/watch?v=TPqr8t919YM

- Sigmoid Function (GFG) — https://www.geeksforgeeks.org/derivative-of-the-sigmoid-function/

- Entropy, Cross-Entropy & KL-Divergence — https://www.youtube.com/watch?v=ErfnhcEV1O8

- Gradient of Softmax for Cross-Entropy Loss — https://markusthill.github.io/blog/2024/gradient-softmax-function-with-cross-entropy/

- Evaluation Metrics in ML — https://www.appliedaicourse.com/blog/evaluation-metrics-in-machine-learning/

- How to Evaluate ML Models — https://www.youtube.com/watch?v=LbX4X71-TFI

6. Decision Trees & Ensemble Methods

6.1 Scikit-learn Guide — https://medium.com/@sumit.kaul.87/a-comprehensive-guide-to-scikit-learn-machine-learning-in-python-with-code-examples-8e4670877d03

6.2 Building Intuition for Classification Trees (CART) — https://www.youtube.com/watch?v=pR-Of1ua6Dc

6.3 Classification Trees

- Coding a Decision Tree From Scratch (Gini, entropy, info index) — https://anderfernandez.com/en/blog/code-decision-tree-python-from-scratch/

- Classification Trees Explained (Gini Impurity) — https://youtu.be/_L39rN6gz7Y

- Entropy & Information Gain — https://www.youtube.com/watch?v=ZVR2Way4nwQ

- Overfitting & Underfitting in Classification Trees — https://www.youtube.com/watch?v=wpNl-JwwplA

6.4 Regression Trees

- Understanding Regression Trees (SSR) — https://youtu.be/g9c66TUylZ4

- Variance Reduction — https://www.youtube.com/watch?v=UhY5vPfQIrA

- Overfitting/Underfitting Cheatsheet — https://www.superannotate.com/blog/overfitting-and-underfitting-in-machine-learning

- Pruning Regression Trees — https://youtu.be/D0efHEJsfHo

6.5 ID3 & CART

- Demystifying Decision Trees (CART) — https://pub.aimind.so/demystifying-decision-trees-an-intuitive-guide-to-understanding-and-applying-cart-algorithm-for-c77 8a56924cc

- ID3 From Scratch (GFG) — https://www.geeksforgeeks.org/iterative-dichotomiser-3-id3-algorithm-from-scratch/

- CART vs ID3 (KDnuggets) — https://www.kdnuggets.com/2020/02/decision-tree-intuition.html

6.6 Random Forest

- StatQuest: Random Forests Part 1

- StatQuest: Random Forests Part 2

- Random Forests From Scratch — https://carbonati.github.io/posts/random-forests-from-scratch/

6.7 Gradient Descent & Boosting

- The Chain Rule, Clearly Explained

6.8 XGBoost

- XGBoost Part 1: Regression

- XGBoost From Scratch — https://randomrealizations.com/posts/xgboost-from-scratch/

6.9 CatBoost

- CatBoost Part 1: Ordered Target Encoding

- CatBoost Part 2: Building and Using Trees

- CatBoost: Unbiased Boosting w/ Categorical Features (paper) — https://arxiv.org/pdf/1706.09516

7. ML/DL Foundations: Math, NumPy/Pandas, Neural Nets & PyTorch

7.1 Newton's Method

- Intuitive & Physical Approach — https://medium.com/@ruhayel/an-intuitive-and-physical-approach-to-newtons-method-86a0bd812ec3

- Wikipedia — https://en.wikipedia.org/wiki/Newton%27s_method

7.2 Linear Regression

- Simple Explanation w/ Example — https://pujappathak.medium.com/linear-regression-simple-explanation-with-example-fba51b2c181d

- In-depth video — https://www.youtube.com/watch?v=yIYKR4sgzI8

7.3 NumPy, Pandas & Matplotlib

- NumPy Tutorial for Beginners — https://www.youtube.com/watch?v=QUT1VHiLmmI

- NumPy Quickstart — https://numpy.org/doc/stable/user/quickstart.html

- numpy.apply_along_axis() — https://www.geeksforgeeks.org/numpy-apply_along_axis-python/

- Pandas Full Course — https://www.youtube.com/watch?v=gtjxAH8uaP0

- Matplotlib Crash Course — https://www.youtube.com/watch?v=3Xc3CA655Y4

7.4 Intro to Neural Networks

- NN Playlist — https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

- Intro to NNs (victorzhou) — https://victorzhou.com/blog/intro-to-neural-networks/

- Choosing a Loss Function — https://www.perplexity.ai/page/choosing-a-loss-function-c8Vpl.DIS6WusUpDJSIcRQ

- Data Normalization Explained — https://estuary.dev/blog/data-normalization/

- ReLU vs Sigmoid — https://wandb.ai/ayush-thakur/dl-question-bank/reports/ReLU-vs-Sigmoid-Function-in-Deep-Neural-Networks--VmlldzoyMDk0MzI

- Gentle Intro to Sigmoid — https://machinelearningmastery.com/a-gentle-introduction-to-sigmoid-function/

- Loss Functions Explained — https://www.datacamp.com/tutorial/loss-function-in-machine-learning

- Math Behind SGD — https://towardsdatascience.com/stochastic-gradient-descent-math-and-python-code-35b5e66d6f79/

- Learning Rate in ML — https://spotintelligence.com/2024/02/19/learning-rate-machine-learning

- Initializing Neural Networks — https://www.deeplearning.ai/ai-notes/initialization/index.html

- Activation Functions (ML Cheatsheet) — https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html

- Cross-Entropy Loss — https://www.datacamp.com/tutorial/the-cross-entropy-loss-function-in-machine-learning

7.5 PyTorch

- PyTorch Tutorial for DL Lovers — https://www.kaggle.com/code/kanncaa1/pytorch-tutorial-for-deep-learning-lovers

- PyTorch Walkthrough — https://www.youtube.com/watch?v=OIenNRt2bjg

- Learning PyTorch With Examples — https://docs.pytorch.org/tutorials/beginner/pytorch_with_examples.html

7.6 Capstone Project

- Project brief — https://docs.google.com/document/d/1gzpcvvsAfz6YslbgC9tXD3RSANBRUrf4q7H-DS0UDGY/edit