Fine-Tuning vs Transfer Learning: What I Learned Teaching VGG19 to Read Kidney CT Scans
What changed when I stopped treating VGG19 as a fixed feature extractor and let part of it learn.
I started this project with a simple goal: take a network that already knows how to look at images and get it to tell apart four kinds of kidney CT scans, normal, cyst, stone, and tumor. I wanted to know what happens if I stop treating the pretrained network as a fixed feature extractor and actually let part of it learn from my data. That one decision is the whole difference between transfer learning and fine-tuning, and it changed the results more than I expected.
If you have only ever heard these two terms used like synonyms, this is the post I wish I had read first.
The setup
The data is the CT Kidney dataset (normal, cyst, tumor, stone). After augmenting and splitting it 80/10/10, I had about 25,500 training images, roughly 3,200 for validation, and another 3,200 held out for testing. Four balanced classes, around 800 test images each.
Augmentation mattered here because medical scans are not infinite. I used rotation up to 40 degrees, shear, zoom, horizontal flips, and a brightness range of 0.5 to 1.5, then resized everything to 224 by 224 so it would match what VGG19 expects. Nothing fancy, but it gave the model more variety than the raw scans alone.
The backbone in both experiments is VGG19 pretrained on ImageNet, with include_top=False so I could attach my own classifier head:
x = Flatten()(vgg.output)
x = Dense(4096, activation='relu')(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.25)(x)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
prediction = Dense(4, activation='softmax')(x)That head is identical across both runs. The only thing I changed between the two experiments is how much of VGG19 itself was allowed to learn.
What transfer learning actually means
Here is the mental model that finally made it click for me. A convolutional network trained on ImageNet has already learned a hierarchy. The early layers detect edges, corners, and color blobs. The middle layers assemble those into textures and simple shapes. The deep layers combine those into parts of objects. None of that knowledge is specific to cats or cars, edges are edges, so a lot of it transfers to kidney scans even though VGG19 has never seen a single CT image.
Transfer learning takes that idea literally. You freeze the entire pretrained backbone and only train the new head you bolted on top:
vgg = VGG19(input_shape=(224, 224, 3), include_top=False)
for layer in vgg.layers:
layer.trainable = FalseEvery convolutional filter stays exactly where ImageNet left it. The network becomes a fixed feature extractor, and the only thing learning is my dense classifier deciding how to map those frozen features into four categories. It trains fast because most of the weights never move, and it is hard to overfit for the same reason.
I trained this with Adam on its default learning rate, categorical cross entropy, and early stopping watching validation loss with a patience of 10. It got going almost immediately. Validation accuracy was already past 91 percent after the first epoch and climbed from there.
Final test accuracy came in at 96.65 percent, with a macro F1 of 0.97. The per class breakdown:
precision recall f1-score support
Cyst 0.99 0.95 0.97 801
Normal 1.00 0.92 0.96 801
Stone 0.90 1.00 0.95 794
Tumor 0.98 1.00 0.99 800
accuracy 0.97 3196Honestly, for an afternoon of work and a frozen backbone, 96.65 percent is great. But look at the Stone row. Precision is 0.90, lower than everything else, which means the model was flagging things as stones that were not stones. The frozen ImageNet features were good enough to mostly separate the classes, but they were never tuned for the fine, grainy textures that distinguish a stone from a cyst. That gap is exactly what fine-tuning is for.
What fine-tuning changes
Fine-tuning keeps the same idea but loosens the grip. Instead of freezing all of VGG19, I unfroze the last 10 layers so the deepest part of the backbone could adapt to kidney scans, while the early edge and texture detectors stayed frozen:
for layer in vgg.layers[:-10]: # keep the early layers frozen
layer.trainable = False
for layer in vgg.layers[-10:]: # let the deep layers learn
layer.trainable = TrueThe logic is that early features (edges, basic textures) are universal and worth keeping, but the deep features (the abstract “this is a part of an object” patterns) are too ImageNet specific. Those are the ones I want to repoint toward medical imaging.
The other change is small but absolutely critical: the learning rate. When you unfreeze pretrained weights you have to train them gently, because the weights are already good and a normal learning rate will blow them apart before they can adapt. I dropped it by a couple orders of magnitude:
model.compile(
optimizer=Adam(learning_rate=1e-5),
loss='categorical_crossentropy',
metrics=['accuracy']
)That 1e-5 is doing a lot of quiet work. With the default rate, fine-tuning often destroys the very features you were trying to reuse. With a tiny rate, you nudge them instead of shoving them.
I only ran 3 epochs, partly because each epoch was genuinely slow (the unfrozen layers mean a lot more gradients to compute, and an epoch took hours on my setup), and partly because it converged fast. Validation accuracy hit 95.4 percent after epoch one and 99 percent by epoch three.
Final test accuracy: 99.09 percent, macro F1 0.99.
precision recall f1-score support
Cyst 0.98 1.00 0.99 802
Normal 1.00 0.99 0.99 801
Stone 1.00 0.98 0.99 795
Tumor 0.99 1.00 1.00 801
accuracy 0.99 3199The Stone class went from 0.90 precision to 1.00. That is the headline for me. The weakest class in the frozen model became basically perfect once the deep layers were allowed to learn what a kidney stone actually looks like instead of borrowing a texture filter meant for ImageNet.
So which one is better?
It depends on what you are optimizing for, and that is not a cop out.
Transfer learning got me to 96.65 percent fast, with almost no risk of overfitting and a fraction of the compute. If I were prototyping, sanity checking the dataset, or working with very few images, I would stop there. It is the safe, cheap baseline, and it is genuinely strong.
Fine-tuning got me to 99.09 percent and cleaned up the one class the frozen model struggled with, but it cost a lot more training time and it only works if you are careful. Unfreeze too many layers, or use too high a learning rate, and you will overwrite the pretrained knowledge and end up worse than where you started. The tiny learning rate and the choice to only unfreeze the last 10 layers are not optional details, they are the whole reason it worked.
The way I think about it now:
Use transfer learning when your data is small, your compute is limited, or your task is close to what the backbone already knows. Freeze everything and train the head.
Use fine-tuning when you have enough data to support it, you have the compute budget, and that last bit of accuracy actually matters. Unfreeze the deep layers, drop the learning rate hard, and watch validation loss like a hawk.
For a medical task where a misread scan is a real cost, the extra 2.4 points and the fixed Stone class were worth the extra training time to me. The frozen model would say “probably a stone” a little too often. The fine-tuned one stopped guessing.
The thing I actually took away
Transfer learning and fine-tuning are not two different techniques, they are two points on the same dial. The dial is “how much of the pretrained network do I let learn.” All the way frozen is transfer learning. Partially unfrozen with a gentle learning rate is fine-tuning. The art is in choosing how far to turn it and how softly to push once you do.
Both notebooks, the data pipeline, and the full results are in the repo. The original transfer-learning-only version lives at github.com/rishika1099/Kidney-Disorder-Detection if you want to see where this started.