Deep Learning concepts for Medical Imaging — A shallow overview

Rahul S
7 min readNov 19, 2022

U-Net Architechture (src:

In this article I go through a few DL concepts used in Medical Imaging. This is a very limited piece, and depends highly on (can be considered a limited summary of) this paper: []. It is a non-mathematical, non-rigorous treatment, with focus on uses and concepts involved.

I will go through the use of deep learning for

  1. image classification,
  2. object detection,
  3. segmentation,

and provide concise overviews of conceptual ideal per application area.

For a fuller treatment, one must go through the article mentioned.

Let’s jump onto it.

Image Classification

Image or exam classification was one of the first areas in which deep learning contributed majorly to medical image analysis. In exam classification, one typically has one or multiple images (an exam) as input, with a single diagnostic variable as output (e.g., disease present or not).

In such a setting, every diagnostic exam is a sample and dataset sizes are typically small compared to those in computer vision (e.g., hundreds/thousands vs. millions of samples). The popularity of transfer learning for such applications is therefore not surprising.

Transfer learning is the use of pre-trained networks (typically on natural images) to work around the (perceived) requirement of large data sets for deep network training.

Two transfer learning strategies were identified: (1) using a pre-trained network as a feature extractor and (2) fine-tuning a pre-trained network on medical data. The former strategy has the extra benefit of not requiring one to train a deep network at all, allowing the extracted features to be easily plugged in to existing image analysis pipelines. Both strategies are popular and have been widely applied. However, few authors perform a thorough investigation in which strategy gives the best result.

The medical imaging community initially focused on unsupervised pre-training and network architectures, like SAEs and RBMs, for…