Machine Learning from Scratch: A Full Lab Walkthrough
Machine Learning Code 101
This is the whole pipeline in one place. Load some data, clean it, train a model, check how good it is. Every ML task you get in a lab follows the same skeleton, so once you have this flow in your head you can swap models in and out without thinking about it.
I kept the structure of my notes and filled in the parts that were just headers (k-fold, ensembles) plus a few standard topics that weren’t there (clustering, PCA, grid search). Each step has a short note on what it does and why, then the code. Everything is copy-paste ready.
1. Imports
Start every notebook with these. Numpy and pandas do the data handling, matplotlib and seaborn do the plots. The warnings filter just stops sklearn from spamming you with deprecation messages during a timed lab.
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns2. Load the dataset
Read the CSV into a DataFrame. Then split it into features (X, everything except the last column) and the target (y, the last column). This assumes the thing you’re predicting is the last column, which it usually is in lab datasets. If it isn’t, name the column directly instead of using position.
df = pd.read_csv("data.csv")
X = df.iloc[:, :-1] # all columns except the last
y = df.iloc[:, -1] # the last column3. Look at the data first
Never train on data you haven’t looked at. These commands tell you the shape, the column types, whether there are nulls, and how the target is distributed. The value_counts and countplot matter most for classification: if one class hugely outnumbers the other, you’ll need to deal with that later (see SMOTE).
df.head(10) # first 10 rows
df.shape # (rows, columns)
df.columns # column names
df.info() # types and non-null counts
df.describe() # mean, std, min, max per column
df['target'].value_counts() # class balance
sns.countplot(x=df['target'])
plt.show()4. Preprocessing
This is where most of the work is. A model is only as good as the data you feed it.
Missing values
First count them. Then decide what to do: drop the rows, or fill them in. Dropping is fine when you have plenty of data. Filling (imputation) is better when you can’t afford to lose rows. The replace(0, np.nan) trick is for datasets where a 0 actually means “missing” (like a blood pressure of 0).
# count missing values per column
df.isnull().sum()
# treat 0s as missing, then drop those rows
df = df.replace(0, np.nan)
df.dropna(inplace=True)
# OR fill one column with its mean
df['column'].fillna(df['column'].mean(), inplace=True)For a cleaner, reusable version use SimpleImputer. It learns the fill values and applies them consistently.
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy='mean', missing_values=np.nan)
imputer = imputer.fit(df)
df.iloc[:, :] = imputer.transform(df)Encode the target and any text columns
Models only understand numbers, so any text labels have to become numbers. For a binary target you can just map the two values by hand.
df['target'].replace(['Yes', 'No'], [1, 0], inplace=True)For a column with many categories, use LabelEncoder, which assigns each unique value an integer.
from sklearn import preprocessing
label_encoder = preprocessing.LabelEncoder()
df['column'] = label_encoder.fit_transform(df['column'])One thing my notes skipped: label encoding puts categories on a number line (0, 1, 2...), which implies an order. For categories with no real order (colors, cities) that can mislead a model. Use one-hot encoding instead, which makes a separate 0/1 column for each category.
df = pd.get_dummies(df, columns=['color', 'city'])Drop columns you don’t need
ID columns, names, anything that leaks the answer, or columns that are mostly empty.
df.drop(['column1', 'column2'], axis=1, inplace=True)Split into train and test
You train on one chunk and test on a chunk the model has never seen. That’s the only honest way to know if it actually learned something or just memorized. random_state makes the split reproducible so your results don’t change every run.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.20, random_state=21)Scale the features
Many models (KNN, SVM, logistic regression) care about distance or magnitude, so a feature measured in thousands will drown out one measured in single digits. Scaling fixes that. Two options:
Normalization (MinMax) squeezes everything into the 0 to 1 range.
from sklearn import preprocessing
scaler = preprocessing.MinMaxScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)Standardization (StandardScaler) centers each feature at mean 0 with standard deviation 1. This is the more common default.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)The important rule: fit_transform on train, transform only on test. You learn the scaling from training data and apply the same scaling to test data. Fitting on the test set leaks information.
Fix class imbalance with SMOTE
If your countplot showed one class swamping the other, the model will just predict the majority class and look deceptively accurate. SMOTE creates synthetic examples of the minority class to even things out. Only ever apply it to the training set, never the test set.
from imblearn.over_sampling import SMOTE
oversample = SMOTE()
X_train, y_train = oversample.fit_resample(X_train, y_train)5. Cross-validation
My notes had this as a header with nothing under it, so here it is. A single train/test split can be lucky or unlucky depending on which rows landed where. K-fold cross validation splits the data into k parts, trains on k-1 of them and tests on the one left out, then rotates so every part gets a turn as the test set. You report the average score, which is far more trustworthy than one split.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5) # 5 folds
print("Fold scores:", scores)
print("Mean accuracy:", scores.mean())Define model first (see the next section), then run this.
6. The models
Every sklearn model follows the same three steps: create it, fit it on the training data, predict on the test data. Pick the model below, then run the fit/predict block once at the end. That’s the whole point: the wrapper never changes, only the model line does.
Linear regression (predicting a continuous number).
from sklearn.linear_model import LinearRegression
model = LinearRegression()After fitting you can inspect it and plot the residuals (the gap between predicted and actual). Residuals scattered randomly around zero means a good fit; a pattern means the model is missing something.
print('Coefficients:', model.coef_)
print('Variance score:', model.score(X_test, y_test))
plt.style.use('fivethirtyeight')
plt.scatter(model.predict(X_train), model.predict(X_train) - y_train,
color="green", s=10, label='Train data')
plt.scatter(model.predict(X_test), model.predict(X_test) - y_test,
color="blue", s=10, label='Test data')
plt.hlines(y=0, xmin=0, xmax=50, linewidth=2)
plt.legend(loc='upper right')
plt.title("Residual errors")
plt.show()For a single-feature regression you can plot the actual line through the points.
plt.scatter(X_test, y_test)
plt.plot(X_test, y_pred, color='red')
plt.show()Logistic regression (binary classification, despite the name).
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(random_state=16)KNN classifies a point by looking at its k closest neighbors and taking a majority vote. Small k is noisy, large k is smooth. Always scale your features before using it.
from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors=3)Decision tree splits the data on questions like “is feature X above 5?” until it isolates the classes. max_depth stops it from growing so deep it just memorizes the training set.
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(criterion="entropy", max_depth=3)You can print the tree as text to see the actual rules it learned.
from sklearn import tree
text_representation = tree.export_text(model)
print(text_representation)SVM finds the boundary that separates the classes with the widest possible margin. The points sitting right on that margin are the support vectors.
from sklearn import svm
model = svm.SVC(kernel='linear')After fitting you can pull out and plot those support vectors.
support_vector_indices = model.support_
support_vectors_per_class = model.n_support_
support_vectors = model.support_vectors_
plt.scatter(X_train.iloc[:, 0], X_train.iloc[:, 1])
plt.scatter(support_vectors[:, 0], support_vectors[:, 1], color='red')
plt.title('Linearly separable data with support vectors')
plt.xlabel('X1')
plt.ylabel('X2')
plt.show()Naive Bayes uses probability and assumes every feature is independent. That assumption is rarely true, but it works surprisingly well and is very fast.
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()Random forest builds many decision trees on random slices of the data and averages their votes. More trees, less overfitting than a single tree.
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100)Fit and predict (the same for all of the above)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)7. Ensemble: combine several models
This was another empty header in my notes. The idea is simple: several different models voting together usually beat any single one, because they make different mistakes that cancel out. A VotingClassifier runs each model and takes the majority vote.
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
ensemble = VotingClassifier(estimators=[
('lr', LogisticRegression()),
('dt', DecisionTreeClassifier()),
('nb', GaussianNB())
], voting='hard') # 'hard' = majority vote, 'soft' = average probabilities
ensemble.fit(X_train, y_train)
y_pred = ensemble.predict(X_test)Random forest (above) is itself an ensemble of trees, so you’ve already used the idea once.
8. Evaluate the model
How you score depends on whether you predicted a number (regression) or a category (classification).
Regression metrics
Lower is better for all the error metrics. R2 is the exception: it runs up toward 1, where 1 is perfect.
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
print("MAE:", mean_absolute_error(y_test, y_pred))
print("MSE:", mean_squared_error(y_test, y_pred))
print("RMSE:", np.sqrt(mean_squared_error(y_test, y_pred)))
print("R2:", r2_score(y_test, y_pred))Classification metrics
Accuracy alone lies when classes are imbalanced, so always look at precision and recall too. Precision asks “of the things I flagged positive, how many really were?” Recall asks “of all the real positives, how many did I catch?” The confusion matrix shows you the raw counts of right and wrong for each class.
from sklearn import metrics
print("Accuracy:", metrics.accuracy_score(y_test, y_pred))
print("Precision:", metrics.precision_score(y_test, y_pred, average='macro'))
print("Recall:", metrics.recall_score(y_test, y_pred, average='macro'))
cm = metrics.confusion_matrix(y_test, y_pred)
print("Confusion Matrix:\n", cm)
print(metrics.classification_report(y_test, y_pred))The confusion matrix reads much better as a heatmap.
ax = sns.heatmap(cm, annot=True, cmap='Pastel2')
ax.set_title('CONFUSION MATRIX\n')
ax.set_xlabel('\nPredicted Values')
ax.set_ylabel('Actual Values')
ax.xaxis.set_ticklabels(['False', 'True'])
ax.yaxis.set_ticklabels(['False', 'True'])
plt.show()9. Hyperparameter tuning
Instead of guessing values like n_neighbors=3, let GridSearchCV try a grid of options and keep whichever scored best under cross-validation.
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
param_grid = {'n_neighbors': [3, 5, 7, 9, 11]}
grid = GridSearchCV(KNeighborsClassifier(), param_grid, cv=5)
grid.fit(X_train, y_train)
print("Best parameters:", grid.best_params_)
print("Best score:", grid.best_score_)10. K-Means clustering (unsupervised)
Everything above used labeled data. K-Means is the opposite: no labels, you just ask it to group the data into k clusters by similarity. The elbow method helps you pick k by plotting how much error drops as you add clusters; the “elbow” in the curve is a good choice.
from sklearn.cluster import KMeans
# elbow method to choose k
inertias = []
for k in range(1, 11):
km = KMeans(n_clusters=k, random_state=42)
km.fit(X)
inertias.append(km.inertia_)
plt.plot(range(1, 11), inertias, marker='o')
plt.xlabel('Number of clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')
plt.show()
# fit with the chosen k
kmeans = KMeans(n_clusters=3, random_state=42)
labels = kmeans.fit_predict(X)11. PCA (dimensionality reduction)
When you have a lot of features, PCA compresses them into a few new ones that keep most of the information. Handy for plotting high-dimensional data in 2D, or for speeding up a model. Scale your features before running it.
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_train)
print("Variance explained:", pca.explained_variance_ratio_)
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y_train)
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('Data projected onto 2 components')
plt.show()Quick reference
The mental model for any supervised task is short:
# 1. load
df = pd.read_csv("data.csv")
X, y = df.iloc[:, :-1], df.iloc[:, -1]
# 2. split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=21)
# 3. scale
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
# 4. model (swap this one line for any algorithm)
model = LogisticRegression()
# 5. fit + predict
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
# 6. evaluate
print(metrics.classification_report(y_test, y_pred))Memorize that, and the rest is just knowing which model line to drop in and which metrics to print.