Abdominal Ultrasound Simulation from Semantic Labels using Paired Label-to-Physics-Based Image Translation
Figure 1. Overview of the proposed inference pipeline. (a) A semantic label map is translated into a simplified US image via a paired model. (b) An unpaired SG-CycleGAN refines realism while preserving anatomy. (c) Optional label transformations T (e.g., motion, lesions, deformations) enable diverse scenario generation without CT input. Yellow arrows highlight the anatomical regions affected by the simulated deformation.
Abstract
Purpose: Current abdominal ultrasound (US) simulation methods often require CT-based anatomical references for ray-casting, limiting deformation and pathology variability. We propose a learning-based pipeline trained to predict physics-based images derived from CT scans from semantic labels, enabling controlled simulations without patient-specific CT volumes at inference time. Methods: We introduce a two-stage pipeline that maps anatomical segmentations to realistic US images through a simplified US image. Stage I synthesizes this image from semantic labels using models trained on CT-based ray-casting outputs. Stage II refines it into a realistic US scan using anatomically guided unpaired translation. Deformations and pathologies are generated by editing anatomical maps. Results: We evaluated Pix2Pix and the Semantic Diffusion Model (SDM) in Stage I, followed by segmentation-guided CycleGAN (SG-CycleGAN) refinement in Stage II. SDM significantly outperformed Pix2Pix in morphological metrics, including MAE↓ (19.85 vs. 21.65), SSIM↑ (0.28 vs. 0.24), and mIoU↑ (0.43 vs. 0.29), whereas Pix2Pix yielded better perceptual point estimates (LPIPS↓: 0.17 vs. 0.19; FID↓: 0.32 vs. 0.37; KID↓: 0.25 vs. 0.48). Conclusion: Training paired generative models with physics-based supervision enables approximation of CT-derived ray-casting outputs at inference time directly from semantic labels. Although the pipeline does not require patient-specific CT volumes at inference time, CT-derived segmentations and ray-casting simulations remain necessary to train Stage I. Once trained, the framework enables controllable healthy and pathological simulations through semantic-map modification.