CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

Универсальная модель на основе CLIP для сегментации органов и обнаружения опухолей
Jie-Neng Chen, Zongwei Zhou, Bennett A. Landman, Alan Yuille, Yucheng Tang, Yongyi Lu, Jie Liu, Yixiao Zhang, Junfei Xiao, Yixuan Yuan, Junfei Xiao
2023-10-01

Beyond The Cranial VaultCLIP-Driven Universal ModelContrastive Language-Image Pre-trainingMedical Segmentation Decathloncomputed tomographymulti-dataset trainingorgan segmentationtext embeddingtransfer learningtumor detection
An increasing number of public datasets have shown a marked impact on automated organ segmentation and tumor detection. However, due to the small size and partially labeled problem of each dataset, as well as a limited investigation of diverse types of tumors, the resulting models are often limited to segmenting specific organs/tumors and ignore the semantics of anatomical structures, nor can they be extended to novel domains. To address these issues, we propose the CLIP-Driven Universal Model, which incorporates text embedding learned from Contrastive Language-Image Pre-training (CLIP) to segmentation models. This CLIP-based label encoding captures anatomical relationships, enabling the model to learn a structured feature embedding and segment 25 organs and 6 types of tumors. The proposed model is developed from an assembly of 14 datasets, using a total of 3,410 CT scans for training and then evaluated on 6,162 external CT scans from 3 additional datasets. We rank first on the Medical Segmentation Decathlon (MSD) public leaderboard and achieve state-of-the-art results on Beyond The Cranial Vault (BTCV). Additionally, the Universal Model is computationally more efficient (6× faster) compared with dataset-specific models, generalized better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks.
1
Incorporating CLIP text embeddings into segmentation models enables capturing anatomical relationships and structured feature embeddings for segmentation.
2
The CLIP-Driven Universal Model segments 25 organs and 6 tumor types using combined supervision from 14 datasets (3,410 CT scans).
3
The Universal Model is computationally more efficient, running approximately 6× faster than dataset-specific models.
4
The model generalizes better to CT scans from varying sites and exhibits stronger transfer learning performance on novel tasks.
5
Trained model evaluated on 6,162 external CT scans from 3 datasets, achieving top rank on the MSD public leaderboard and state-of-the-art on BTCV.

CLIP-Driven Universal Model for organ segmentation and tumor detection applied to CT scans

Ability of the model to segment 25 organs and detect 6 tumor types by incorporating CLIP text embeddings to capture anatomical relationships, including generalization across datasets/sites, computational efficiency, and transfer learning performance

Publication Details
Publication Date
2023-10-01
Journal
Publisher
ISSN
Cited by
346
Access Type
Author Information
Authors
Jie-Neng Chen
Zongwei Zhou
Bennett A. Landman
Alan Yuille
Yucheng Tang
Yongyi Lu
Jie Liu
Yixiao Zhang
Junfei Xiao
Yixuan Yuan
Junfei Xiao
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%