Materials
This study uses the public ThyroidXL dataset16, a large-scale, pathology-validated benchmark for thyroid ultrasound analysis containing expert-annotated images with patient-level diagnostic labels confirmed by histopathological examination. The dataset is particularly suited to patient-level modelling because each examination contains multiple ultrasound frames acquired during routine clinical practice. After preprocessing and metadata validation, the cohort comprised 4093 patients and 11,635 images, partitioned by the official split into development and independent test cohorts with differing class balance, alongside TIRADS scores and patient demographics. Full dataset, class-distribution, and split statistics are summarised in Table 1, and representative patient-level bags illustrating intra-patient variability across frames are shown in Fig. 1. Ground-truth malignancy labels were derived from histopathological examination, and binary segmentation masks delineating the thyroid nodule region were available for each image. These masks were not treated as standalone prediction targets; instead, they were used as structured anatomical guidance to separate lesion-focused information from surrounding tissue context.
Table 1 Summary statistics of the ThyroidXL dataset used in this study.Fig. 1
Representative patient-level ultrasound bags from the official test set, illustrating intra-patient variability across frames.
Methods
This section describes the proposed patient-level thyroid ultrasound classification framework, centred on the Region-Aware Context-Aware Fusion (RCAF) model and evaluated under a strict leakage-free patient-level protocol. An overview of the framework is shown in Fig. 2. Starting from raw ultrasound frames and their corresponding segmentation masks, the pipeline proceeds through a leakage-free patient-level split, preprocessing and patient-bag construction, frame-level region-aware encoding with the proposed RCAF model, patient-level aggregation using attention-based multiple instance learning, development-only out-of-fold calibration and threshold selection, and final single-shot evaluation on the official independent test cohort.
The following subsections detail each methodological component of the proposed framework in order, corresponding to the pipeline stages illustrated in Fig. 2.
Fig. 2
Overview of the proposed patient-level thyroid ultrasound classification pipeline.
Leakage-free data split
A strict patient-level partitioning protocol was adopted to prevent information leakage. The official independent test cohort was kept fully isolated and used only once for final model evaluation. The remaining patients formed the development cohort, within which grouped stratified five-fold cross-validation was performed at the patient level, so that all frames from a given patient remained within a single fold and never appeared across both training and validation subsets. This grouped design was essential because image-level splitting would risk inflated performance by allowing patient-specific visual characteristics to leak across subsets. The same partitioning strategy was used across the proposed model and all fair baseline models to ensure unbiased comparison and reliable estimation of patient-level generalisation.
Preprocessing and bag construction
All ultrasound images were resized to a fixed spatial resolution of \(224 \times 224\) pixels, and pixel intensities were normalised to the range [0, 1] to facilitate stable optimisation. Segmentation masks were resized using nearest-neighbour interpolation to preserve their binary structure. During training, data augmentation comprising random horizontal flipping, small geometric perturbations, and mild intensity variation was applied to improve generalisation and reduce sensitivity to acquisition variability. These procedures were applied consistently across the proposed method and all fair baseline models.
To formalise the patient-level learning setting, each examination was represented as a bag of ultrasound frames associated with a single patient-level diagnostic label. For each patient i, the input was represented as
$$\begin{aligned} \mathcal {B}_i = \{x_{i1}, x_{i2}, \dots , x_{iT_i}\}, \end{aligned}$$
(1)
where \(T_i\) denotes the number of ultrasound frames acquired for patient i, and all frames within the same bag (Eq. (1)) share the same patient-level label \(y_i \in \{0,1\}\), where 0 denotes benign and 1 denotes malignant.
To enable efficient patient-level learning with a fixed tensor structure, the number of frames per patient was standardised to a maximum of \(T_{\max } = 10\). Because the dataset maximum coincides with \(T_{\max }\), no patient examination was truncated in this study; all patients had at most 10 frames and are therefore fully represented in their bag. Patients with fewer than \(T_{\max }\) frames were padded with zero-valued frames, and a binary frame-validity mask was used to exclude padded entries from attention weight computation during both training and inference. Should the framework be applied to datasets where examinations routinely exceed 10 frames, a principled truncation policy such as retaining the \(T_{\max }\) frames with the highest individual frame-level malignancy scores from a lightweight pre-filter would be necessary, and its effect on patient-level performance should be validated explicitly.
Proposed RCAF frame-level model
The architecture of the proposed RCAF model is illustrated in Fig. 3. The design is motivated by the observation that thyroid malignancy assessment depends not only on local lesion morphology, but also on the surrounding anatomical and textural context.
To describe the proposed region-aware encoding formally, we denote the input ultrasound frame for patient \(i\) at frame index \(t\) by \(x_{it}\), and its corresponding binary lesion mask by \(m_{it}\): two complementary inputs were constructed:
$$\begin{aligned} x_{it}^{\textrm{les}} = x_{it} \odot m_{it}, \qquad x_{it}^{\textrm{ctx}} = x_{it}, \end{aligned}$$
(2)
where \(\odot\) denotes element-wise multiplication. The lesion branch (Eq. (2)) emphasizes the masked nodule region, whereas the context branch preserves the full ultrasound frame.
Both branches were processed using a shared backbone encoder \(\phi (\cdot )\), yielding frame-level lesion and context features (Eq. (3)):
$$\begin{aligned} z_{it}^{\textrm{les}} = \phi (x_{it}^{\textrm{les}}), \qquad z_{it}^{\textrm{ctx}} = \phi (x_{it}^{\textrm{ctx}}). \end{aligned}$$
(3)
A shared encoder was used to keep the lesion and context branches in a common representation space while controlling model capacity and maintaining a fairer comparison with the baseline architectures. ResNet50 was selected for three reasons. First, with approximately 9500 development images the dataset is moderately sized, and ResNet50 has been shown to offer a strong capacity-to-sample-size trade-off in medical imaging settings of comparable scale4,6. Second, using an identical backbone across all fair baseline models ensures that performance differences are attributable to the fusion architecture rather than to representational capacity. Third, ResNet50 enables a transparent two-stage fine-tuning schedule (frozen lower layers, fine-tuned layer3/layer4) that would generalise straightforwardly to alternative ResNet variants or EfficientNet families; evaluating such alternatives is left for future work alongside the broader multi-center validation effort.
To integrate the two feature streams adaptively, a gated fusion mechanism was applied (Eqs. (4)–(5)):
$$\begin{aligned} & g_{it} = \sigma \!\left( W_g \left[ z_{it}^{\textrm{les}} \,\Vert \, z_{it}^{\textrm{ctx}}\right] \right) , \end{aligned}$$
(4)
$$\begin{aligned} & z_{it} = g_{it} \odot z_{it}^{\textrm{les}} + \left( 1-g_{it}\right) \odot z_{it}^{\textrm{ctx}}, \end{aligned}$$
(5)
where \(\sigma (\cdot )\) denotes the sigmoid function and \(\Vert\) denotes concatenation. This adaptive formulation (Eqs. (4)–(5)) enables the model to emphasize lesion-focused information when local evidence is strong while preserving contextual cues when surrounding tissue remains diagnostically informative.
The fused embedding \(z_{it}\) produced by Eq. (5) was then used as the frame-level representation passed to the patient-level aggregation stage.
Fig. 3
Architecture of the proposed RCAF model.
Patient-level aggregation
After frame-level fusion, the resulting embeddings were aggregated at the patient level using attention-based multiple instance learning12,13,27,28. Formally, each fused frame embedding was assigned a learnable attention weight reflecting its relative diagnostic importance within the patient bag.
For each frame embedding \(z_{it}\), the attention weight was computed as (Eq. (6))
$$\begin{aligned} \alpha _{it} = \frac{\exp \!\left( w^\top \tanh (V z_{it})\right) }{\sum _{t} \exp \!\left( w^\top \tanh (V z_{it})\right) }, \end{aligned}$$
(6)
where V and w are learnable parameters. Attention weights (Eq. (6)) were computed only over valid frames, with padded entries excluded using the frame-validity mask. The patient-level embedding was then formed as (Eq. (7))
$$\begin{aligned} z_i = \sum _{t} \alpha _{it} z_{it}. \end{aligned}$$
(7)
The final patient-level malignancy probability was obtained as
$$\begin{aligned} \hat{y}_i = \sigma (W_o z_i + b_o), \end{aligned}$$
(8)
where \(W_o\) and \(b_o\) denote the classifier parameters. The patient embedding \(z_i\) entering Eq. (8) is the attention-weighted sum of the fused frame embeddings produced by Eqs. (5) and (7). This formulation preserves the patient-level decision setting while enabling the model to focus on the most informative frames within each examination.
Training strategy
Model optimisation was performed at the patient level using the predicted malignancy probability for each patient bag and the corresponding ground-truth label. For clarity, the loss is written below in probability form, although implementation used the logit-equivalent binary cross-entropy formulation. All models were trained using binary cross-entropy loss (Eq. (9)):
$$\begin{aligned} \mathcal {L}_{\textrm{BCE}} = – \frac{1}{N} \sum _{i=1}^{N} \left[ y_i \log (\hat{y}_i) + (1-y_i)\log (1-\hat{y}_i) \right] . \end{aligned}$$
(9)
Deep feature encoders were initialised with pretrained weights, and a two-stage optimisation strategy was adopted. In stage 1, the ResNet50 backbone was kept fully frozen while only the gated fusion module, AttnMIL aggregator, and classification head were optimised, monitored on validation ROC-AUC. In stage 2, the upper two residual blocks of the backbone (layer3 and layer4) were unfrozen and fine-tuned jointly with all remaining components, again monitored on validation ROC-AUC, with the best checkpoint selected on the highest observed validation ROC-AUC. This staged approach prevents the bag-level components from overfitting to uninitialised backbone features and allows the upper backbone layers to specialise to the thyroid ultrasound domain without catastrophic forgetting of lower-layer ImageNet representations. Grouped stratified patient-level folds and class weights were used to reduce imbalance-related bias, and optimisation used AdamW with early stopping on validation ROC-AUC. The same two-stage protocol, optimizer family, preprocessing pipeline, bag construction strategy, and checkpoint-selection logic were applied across the proposed model and all fair baseline models. Complete implementation details, hyperparameters, parameter counts, FLOPs, and inference timings are summarised in Table 2.
The maximum bag length \(T_{\max }=10\) was selected based on the ThyroidXL frame-count distribution, in which more than 95% of patients have at most 10 frames and no patient exceeds this limit. The attention dimensionality \(d_{\textrm{attn}}=128\) follows the standard convention of Ilse et al.13 (\(d_{\textrm{attn}} = d_{\textrm{embed}}/2\)). As summarised in Table 2, total per-patient computational cost scales linearly with the number of valid frames (\(O(T_{\max })\)), since each frame is processed independently by the shared encoder before aggregation, and the gated fusion and AttnMIL aggregation steps contribute less than 0.1% of total per-patient FLOPs regardless of bag length.
Table 2 Implementation details and training hyperparameters used in the experiments.
OOF calibration and threshold selection
Post-hoc calibration was applied to patient-level logits obtained from the out-of-fold development predictions in order to improve probability reliability without altering the underlying discrimination behaviour of the model. Specifically, post-hoc temperature scaling17,25 was applied to reduce miscalibration while leaving the model’s discriminative ordering unchanged (Eq. (10)):
$$\begin{aligned} \hat{p}_i = \sigma \left( \frac{z_i}{T}\right) , \end{aligned}$$
(10)
where \(z_i\) denotes the patient-level logit before calibration and T is a scalar temperature parameter optimized on the OOF development predictions.
The calibration parameter was estimated strictly within the development cohort and then fixed before evaluating the independent test set. In addition to the default threshold of 0.5, two clinically relevant operating thresholds were considered:
-
the threshold maximizing the Youden index on the OOF development predictions, and
-
a sensitivity-constrained threshold selected to satisfy \(\textrm{Sensitivity}\ge 0.90\) on the OOF development predictions.
This design allows discrimination performance, calibration quality, and operating-point behaviour to be assessed without contamination from the official test data.
Final model and independent test
Following development-stage cross-validation, calibration, and threshold selection, the final model was evaluated once on the untouched official test cohort. The calibration parameter and operating thresholds were fixed from the development stage and applied unchanged to the test cohort, preserving the strict leakage-free protocol. This single-shot evaluation provides an unbiased estimate of patient-level generalisation, since no test-set information was used at any point during model development, calibration fitting, or threshold selection.
Final outputs and metrics
All discrimination, calibration, and operating-point metrics were computed at the patient level. The primary discrimination metrics were ROC-AUC and PR-AUC. To assess probability quality, the Brier score and expected calibration error (ECE) were additionally reported. At selected operating thresholds, sensitivity, specificity, precision, F1-score, and confusion-matrix-derived counts were reported. Bootstrap confidence intervals were computed to quantify sampling uncertainty for all reported metrics.
