The proposed diagnostic pipeline consists of four main blocks: (1) patient-level data partitioning before augmentation; (2) ROI-guided preprocessing and training-only augmentation; (3) a ResNet50 backbone coupled with the proposed IVAF module; and (4) a two-stage transfer-learning procedure with regularization callbacks. The implementation used TensorFlow 2.12/Keras 2.12 and was run on a single NVIDIA A100 GPU with 80 GB VRAM. The overall training time took approximately 4.2 h with 1.8 h for Phase I and 2.4 h for Phase II, considering that a mini-batch size of 32 and the Adam optimizer, which consists of \(\:{\beta\:}_{1}=0.9\), \(\:{\beta\:}_{2}=0.999\), and \(\:\epsilon = 1 \times \:10^{{ – 7}}\)., were employed. The overall procedure can be seen in Fig. 1.
The ResNet50 network has been chosen as the backbone for extracting features due to two reasons: first, residual connections make the optimization process more stable in deep CNN models; second, this architecture is a standard one when it comes to medical image classification using transfer learning. In this way, we obtain a unified feature extractor for CC and MLO pairs without paying too much attention to the backbone.
Fig. 1
End-to-end workflow of the proposed ResNet50 + IVAF pipeline.
Patient-level data stratification and leakage control
Isolation of patient-level data prior to preprocessing and augmentation is the critical methodology in this study. Patient-level partitioning is significant in mammographic imaging datasets due to the fact that several images, views, lateralities, and annotations can belong to one and the same patient. The inclusion of all images of one patient in one and the same partition ensures that patient-specific anatomical information does not find its way into training and test splits simultaneously.
In this study, 1,109 distinct patients were stratified based on pathological class and assigned to non-overlapping training, validation, and test partitions prior to any transformations of images. The training set consisted of 665 patients, the validation set consisted of 222 patients, while the test set consisted of 222 patients. Augmentation was applied only to images in the training set, and hence the training dataset increased from 18,120 samples to 50,025 samples. Validation and test sets remained unaltered throughout the process. Partition statistics are provided in Table 4.
Table 4 Patient-level CBIS-DDSM partition statistics.
Image preprocessing
The mammography images were resampled to 224 × 224 pixels via bicubic interpolation for the sake of meeting the spatial input constraint of the ResNet50 backbone network architecture, while preserving the texture information at the level of lesions. The focal lesion was isolated from the high resolution mammography image via the use of ROI masks from CBIS-DDSM and then resampling was done. These preprocessing steps guide the input of the model towards annotated lesion areas while maintaining the surroundings context.
The pixel intensities were scaled to the continuous range \(\:\left[0,1\right]\) according to:
$$\:{I}_{\text{n}\text{o}\text{r}\text{m}}\left(x,y\right)=\frac{{I}_{\text{o}\text{r}\text{i}\text{g}\text{i}\text{n}\text{a}\text{l}}\left(x,y\right)}{255}.\:\:$$
(1)
Here, \(\:{I}_{\text{o}\text{r}\text{i}\text{g}\text{i}\text{n}\text{a}\text{l}}\left(x,y\right)\) represents the original pixel intensity at coordinates \(\:\left(x,y\right)\). The process of linear scaling ensures that the input intensity values are standardized for consistent optimization while maintaining the relative contrast relationship within the mammogram image.
Fig. 2
ROI-guided preprocessing workflow.
Data augmentation
In order to introduce variability in the training set and resolve class imbalance, stochastic geometric and photometric transformations were performed only on the training data following patient-level stratification. This step ensured that augmented training images did not contain patient-specific anatomical information common to validation or testing images. These augmentations comprised in-plane rotations between\(\:-{15}^{\circ\:}\)and \(\:+{15}^{\circ\:}\), random horizontal and vertical translations up to \(\:\pm\:10\text{\%}\), isotropic scaling between \(\:0.90\) to \(\:1.10\), anatomically realistic flipping, and contrast/brightness adjustments of \(\:\pm\:20\text{\%}\)31.
All these transformations were restricted to maintain the lesion structure and avoid clinically implausible artifacts. The overall transformation procedure for each training image can be mathematically defined as:
$$\:{I}_{\text{a}\text{u}\text{g}}={T}_{\text{c}\text{o}\text{m}\text{p}\text{o}\text{s}\text{i}\text{t}\text{e}}\left({I}_{\text{o}\text{r}\text{i}\text{g}\text{i}\text{n}\text{a}\text{l}}\right).\:\:$$
(2)
whereas \(\:{T}_{\text{c}\text{o}\text{m}\text{p}\text{o}\text{s}\text{i}\text{t}\text{e}}\) stands for the stochastic composition of the active transformations. The post-split augmentation was used in conjunction with the class-weighted loss according to Sect “Class-imbalance mitigation via balanced loss weighting”.
Model architecture
ResNet50 backbone and classification head
Each mammographic view is processed by a ResNet50 backbone initialized with ImageNet weights. The residual connection in each block is summarized by:
$$\:y=F\left(x,\left\{{W}_{i}\right\}\right)+x.\:\:$$
(3)
where \(\:x\) denotes the input to the block, \(\:F\left(x,\left\{{W}_{i}\right\}\right)\) is is the residual mapping with parameters \(\:\left\{{W}_{i}\right\}\), represented by convolutional filters, and \(\:y\) is the output from the block.
Following feature extraction and IVAF fusion, Global Average Pooling turns the fused feature tensor into a compact channel-wise feature representation. Next, a Dropout layer with 0.5 dropout probability, followed by a dense layer of 1,024 neurons and ReLU activations and finally a Softmax layer are used for classifying benign/malignant. Standard CNN layers are concisely reported as our major contributions lie elsewhere in the methodology.
Fig. 3
Overview of the proposed ResNet50 + IVAF architecture.
Inter-view attention fusion (IVAF) module
IVAF module is designed such that two types of complementary CC and MLO views are merged using adaptive attention mechanism, unlike giving equal weightage to both the views. The reason behind such a choice is that complementary information from both the views may add up to help in achieving better accuracy in classification.
In Fig. 4, it can be seen that IVAF takes feature tensors from paired mammographic views extracted using ResNet50 branches of shared weights. This helps in mapping both the views to the same feature space for feature comparison prior to merging:
$$\:{F}_{CC},{F}_{MLO}\in\:{\mathbb{R}}^{H\times\:W\times\:D}.\:\:$$
(4)
The feature tensors are concatenated along the channel dimension:
$$\:{F}_{\text{c}\text{a}\text{t}}=\left[{F}_{CC};{F}_{MLO}\right].\:\:$$
(5)
The attention map is produced using a lightweight gated convolutional network based on a 1 × 1 convolutional layer with ReLU activation and 64 filters, and a second 1 × 1 convolutional layer with 2,048 filters and sigmoid activation:
$$\:A=\sigma\:\left({\text{C}\text{o}\text{n}\text{v}}_{1\times\:1}^{2048}\left(\text{R}\text{e}\text{L}\text{U}\left({\text{C}\text{o}\text{n}\text{v}}_{1\times\:1}^{64}\left({F}_{\text{c}\text{a}\text{t}}\right)\right)\right)\right),\:A\in\:{\mathbb{R}}^{H\times\:W\times\:D}.\:\:$$
(6)
The final fused feature tensor is computed as:
$$\:{F}_{\text{f}\text{u}\text{s}\text{e}\text{d}}=A\odot\:{F}_{CC}+\left(1-A\right)\odot\:{F}_{MLO}.\:\:$$
(7)
where ⊙ represents the Hadamard product of tensors. The combined tensor \(\:{F}_{\text{f}\text{u}\text{s}\text{e}\text{d}}\) is then passed on to the Global Average Pooling layer and the classification layer. The above formulation allows for learning of adaptive contributions of CC-MLO at each spatial position and channel. Classification can be formulated as:
$$\:\widehat{y}=\text{S}\text{o}\text{f}\text{t}\text{m}\text{a}\text{x}\left({W}_{2}\hspace{0.17em}\text{R}\text{e}\text{L}\text{U}\left({W}_{1}\hspace{0.17em}\text{G}\text{A}\text{P}\left({F}_{\text{f}\text{u}\text{s}\text{e}\text{d}}\right)+{b}_{1}\right)+{b}_{2}\right).\:\:$$
(8)
where \(\:{W}_{1}\), \(\:{W}_{2}\), \(\:{b}_{1}\), and \(\:{b}_{2}\) denote trainable classification-head parameters and \(\:\widehat{y}\) is the predicted class-probability vector.
Fig. 4
Enhanced IVAF module architecture.
Two-stage transfer-learning strategy
A two-stage training regimen was implemented to address the domain shift from natural images to mammographic images, following common transfer-learning practice in medical image classification32. In Stage I, the ResNet50 backbone was frozen and only the randomly initialized classification head was trained. The Stage I update is expressed as:
$$\:{\theta\:}_{\text{h}\text{e}\text{a}\text{d}}^{\left(t+1\right)}={\theta\:}_{\text{h}\text{e}\text{a}\text{d}}^{\left(t\right)}-{\eta\:}_{1}{\nabla\:}_{{\theta\:}_{\text{h}\text{e}\text{a}\text{d}}}L\left({\theta\:}_{\text{h}\text{e}\text{a}\text{d}},{\theta\:}_{\text{b}\text{a}\text{c}\text{k}\text{b}\text{o}\text{n}\text{e}}^{\text{f}\text{r}\text{o}\text{z}\text{e}\text{n}}\right).\:\:$$
(9)
The initial learning rate was set to \(\:{\eta\:}_{1}=1\times\:{10}^{-4}\). Freezing the backbone preserves pretrained low- and mid-level feature hierarchies while allowing the classification head to adapt to the mammography label space.
In Stage II, the final two residual blocks of ResNet50 were unfrozen and fine-tuned with the classification head using a lower learning rate of \(\:{\eta\:}_{2}=1\times\:{10}^{-5}\):
$$\:{\theta\:}^{\left(t+1\right)}={\theta\:}^{\left(t\right)}-{\eta\:}_{2}{\nabla\:}_{\theta\:}L\left({\theta\:}_{\text{h}\text{e}\text{a}\text{d}},{\theta\:}_{\text{b}\text{a}\text{c}\text{k}\text{b}\text{o}\text{n}\text{e}}\right).\:\:$$
(10)
Selective fine-tuning was used because earlier convolutional layers capture generic low-level features, whereas deeper layers encode more task-specific representations13,29. The lower Stage II learning rate reduces the risk of destabilizing useful pretrained features.
Class-imbalance mitigation via balanced loss weighting
CBIS-DDSM is moderately imbalanced, with benign lesions representing approximately 55% of the original dataset. To reduce bias toward the majority class, class-balanced categorical cross-entropy was used during both training stages:
$$\:L_{{{\text{weighted}}}} = – \frac{1}{N}\sum\limits_{{i = 1}}^{N} {\sum\limits_{{C = 1}}^{C} {w_{c} y_{{i,c}} {\text{log}}\left( {\hat{y}_{{i,c}} } \right)} }$$
(11)
The class weight for class \(\:c\) was computed as:
$$\:{w}_{c}=\frac{N}{C\hspace{0.17em}{N}_{c}}.\:\:$$
(12)
where \(\:N\) is the number of training samples, \(\:C\) is the number of classes, and \(\:{N}_{c}\) is the number of samples belonging to class \(\:c\). This weighting strategy encourages each class to contribute more evenly to the total loss and is commonly used to reduce imbalance-related bias in medical image classification33.
Training callbacks and regularization
Two adaptive callbacks were used during training. EarlyStopping monitored the validation loss and restored the best weights when improvement plateaued. The stopping condition can be written as:
$$\:{\text{Stopif}}v_{t} \ge \:{\text{min}}\left( {v_{{t – \delta \:}} , \ldots \:,v_{{t – 1}} } \right) – \epsilon ,\,\,\,\,\,\,\,\,\,\,\,\:\:\delta \: = 5,\,\,\epsilon = 10^{{ – 4}} .\:\:$$
(13)
where \(\:{v}_{t}\) is the validation loss at epoch \(\:t\). ReduceLROnPlateau reduced the learning rate when validation loss stopped improving:
$$\:{\eta\:}_{t+1}=\left\{\begin{array}{ll}0.5{\eta\:}_{t},&\:\text{i}\text{f}\hspace{0.25em}\text{p}\text{l}\text{a}\text{t}\text{e}\text{a}\text{u};\\\:{\eta\:}_{t},&\:\text{o}\text{t}\text{h}\text{e}\text{r}\text{w}\text{i}\text{s}\text{e}.\end{array}\right.\:\:$$
(14)
Together with Dropout, class-balanced loss weighting, EarlyStopping, and learning-rate reduction were used to regularize optimization and reduce overfitting.
