InvNorm: Domain Generalization for Object Detection in Gastrointestinal Endoscopy

Abstract: Domain Generalization is a challenging topic in computer vision, especially in Gastrointestinal Endoscopy image analysis. Due to several device limitations and ethical reasons, current open-source datasets are typically collected on a limited number of patients using the same brand of sensors. Different brands of devices and individual differences will significantly affect the model's generalizability. Therefore, to address the generalization problem in GI(Gastrointestinal) endoscopy, we propose a multi-domain GI dataset and a light, plug-in block called InvNorm(Invertible Normalization), which could achieve a better generalization performance in any structure. Previous DG(Domain Generalization) methods fail to achieve invertible transformation, which would lead to some misleading augmentation. Moreover, these models would be more likely to lead to medical ethics issues. Our method utilizes normalizing flow to achieve invertible and explainable style normalization to address the problem. The effectiveness of InvNorm is demonstrated on a wide range of tasks, including GI recognition, GI object detection, and natural image recognition.

BayeSeg: Bayesian Modeling for Medical Image Segmentation with Interpretable Generalizability

Abstract: Due to the cross-domain distribution shift aroused from diverse medical imaging systems, many deep learning segmentation methods fail to perform well on unseen data, which limits their real-world applicability. Recent works have shown the benefits of extracting domain-invariant representations on domain generalization. However, the interpretability of domain-invariant features remains a great challenge. To address this problem, we propose an interpretable Bayesian framework (BayeSeg) through Bayesian modeling of image and label statistics to enhance model generalizability for medical image segmentation. Specifically, we first decompose an image into a spatial-correlated variable and a spatial-variant variable, assigning hierarchical Bayesian priors to explicitly force them to model the domain-stable shape and domain-specific appearance information respectively. Then, we model the segmentation as a locally smooth variable only related to the shape. Finally, we develop a variational Bayesian framework to infer the posterior distributions of these explainable variables. The framework is implemented with neural networks, and thus is referred to as deep Bayesian segmentation. Quantitative and qualitative experimental results on prostate segmentation and cardiac segmentation tasks have shown the effectiveness of our proposed method. Moreover, we investigated the interpretability of BayeSeg by explaining the posteriors and analyzed certain factors that affect the generalization ability through further ablation studies. Our code is released via https://zmiclab.github.io/projects.html.

Domain Generalization using Causal Matching

Abstract: Learning invariant representations has been proposed as a key technique for addressing the domain generalization problem. However, the question of identifying the right conditions for invariance remains unanswered. In this work, we propose a causal interpretation of domain generalization that defines domains as interventions under a data-generating process. Based on a general causal model for data from multiple domains, we show that prior methods for learning an invariant representation optimize for an incorrect objective. We highlight an alternative condition: inputs across domains should have the same representation if they are derived from the same base object. In practice, knowledge about generation of data or objects is not available. Hence we propose an iterative algorithm called MatchDG that approximates base object similarity by using a contrastive loss formulation adapted for multiple domains. We then match inputs that are similar under the resultant representation to build an invariant classifier. We evaluate MatchDG on rotated MNIST, Fashion-MNIST, and PACS datasets and find that it outperforms prior work on out-of-domain accuracy and learns matches that have over 25\% overlap with ground-truth object matches in MNIST and Fashion-MNIST. Code repository can be accessed here: \textit{this https URL}

Causality-Inspired Single-Source Domain Generalization for Medical Image Segmentation

Abstract: Deep learning models usually suffer from the domain shift issue, where models trained on one source domain do not generalize well to other unseen domains. In this work, we investigate the single-source domain generalization problem: training a deep network that is robust to unseen domains, under the condition that training data are only available from one source domain, which is common in medical imaging applications. We tackle this problem in the context of cross-domain medical image segmentation. In this scenario, domain shifts are mainly caused by different acquisition processes. We propose a simple causality-inspired data augmentation approach to expose a segmentation model to synthesized domain-shifted training examples. Specifically, 1) to make the deep model robust to discrepancies in image intensities and textures, we employ a family of randomly-weighted shallow networks. They augment training images using diverse appearance transformations. 2) Further we show that spurious correlations among objects in an image are detrimental to domain robustness. These correlations might be taken by the network as domain-specific clues for making predictions, and they may break on unseen domains. We remove these spurious correlations via causal intervention. This is achieved by resampling the appearances of potentially correlated objects independently. The proposed approach is validated on three cross-domain segmentation scenarios: cross-modality (CT-MRI) abdominal image segmentation, cross-sequence (bSSFP-LGE) cardiac MRI segmentation, and cross-site prostate MRI segmentation. The proposed approach yields consistent performance gains compared with competitive methods when tested on unseen domains.

Analysis-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:8d13df2a-a30c-4693-86a6-d01fffe6f32b}}), task, organ, and modality used in experiment.

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) No. 2022-0-00959 ((Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making) and No. 2019-0-00079 (Department of Artificial Intelligence (Korea University)).

C. Lian et al., “Hierarchical fully convolutional network for joint atrophy localization and alzheimer's disease diagnosis using structural MRI,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 4, pp. 880–893, 2020.
P. Mukherjee et al., “A shallow convolutional neural network predicts prognosis of lung cancer patients in multi-institutional computed tomography image datasets,” Nature Machine Intelligence, vol. 2, no. 5, pp. 274–282, 2020.
J. Duan et al., “Evaluating the clinical acceptability of deep learning contours of prostate and organs-at-risk in an automated prostate treatment planning process,” Medical Physics, vol. 49, no. 4, pp. 2570–2581, 2022.
Y. Nan et al., “Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions,” Information Fusion, 2022.
R. J. Chen et al., “Scaling vision transformers to gigapixel images via hierarchical self-supervised learning,” in Proceedings of the CVPR, 2022, pp. 16 144–16 155.
L. Feng et al., “5D whole-heart sparse MRI,” Magnetic Resonance in Medicine, vol. 79, no. 2, pp. 826–838, 2018.
D. Ghosh and D. Ekta Ghosh, “A large-scale multi-centre research on domain generalisation in deep learning-based mass detection in mammography: A review,” in Proceedings of the Acta Biology Forum, 2022, pp. 05–09.
A. Apicella et al., “Machine learning strategies to improve generalization in EEG-based emotion assessment: A systematic review,” arXiv preprint arXiv:2212.08744, 2022.
H. Guan and M. Liu, “Domain adaptation for medical image analysis: A survey,” IEEE Transactions on Biomedical Engineering, vol. 69, no. 3, pp. 1173–1185, 2021.
G. Sarafraz et al., “Domain adaptation and generalization on functional medical images: A systematic survey,” arXiv preprint arXiv:2212.03176, 2022.
S. A. Mali et al., “Making radiomics more reproducible across scanner and imaging protocol variations: A review of harmonization methods,” Journal of Personalized Medicine, vol. 11, no. 9, p. 842, 2021.
J. Wang et al., “Generalizing to unseen domains: A survey on domain generalization,” IEEE Transactions on Knowledge and Data Engineering, 2022.
K. Zhou et al., “Domain generalization: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Z. Shen et al., “Towards out-of-distribution generalization: A survey,” arXiv preprint arXiv:2108.13624, 2021.
G. Csurka et al., “Unsupervised domain adaptation for semantic image segmentation: A comprehensive survey,” arXiv preprint arXiv:2112.03241, 2021.
J. Yang et al., “Generalized out-of-distribution detection: A survey,” arXiv preprint arXiv:2110.11334, 2021.
P. Cui and J. Wang, “Out-of-distribution (OOD) detection based on deep learning: A review,” Electronics, vol. 11, no. 21, p. 3500, 2022.
N. Ghassemi and E. Fazl-Ersi, “A comprehensive review of trends, applications and challenges in out-of-distribution detection,” arXiv preprint arXiv:2209.12935, 2022.
P. Sheth et al., “Domain generalization–A causal perspective,” arXiv preprint arXiv:2209.15177, 2022.
H. Li et al., “Out-of-distribution generalization on graphs: A survey,” arXiv preprint arXiv:2202.07987, 2022.
Y. Li et al., “Federated domain generalization: A survey,” arXiv preprint arXiv:2306.01334, 2023.
E. Kodirov et al., “Unsupervised domain adaptation for zero-shot learning,” in Proceedings of the ICCV, 2015, pp. 2452–2460.
D. Adila and D. Kang, “Understanding out-of-distribution: A perspective of data dynamics,” in Proceedings of NeurIPS Workshop.   PMLR, 2022, pp. 1–8.
M. Shah et al., “Evaluating intensity normalization on MRIs of human brain with multiple sclerosis,” Medical Image Analysis, vol. 15, no. 2, pp. 267–282, 2011.
P. Chlap et al., “A review of medical image data augmentation techniques for deep learning applications,” Journal of Medical Imaging and Radiation Oncology, vol. 65, no. 5, pp. 545–563, 2021.
J. Panic et al., “Normalization strategies in multi-center radiomics abdominal MRI: Systematic review and meta-analyses,” IEEE Open Journal of Engineering in Medicine and Biology, 2023.
W. Yu et al., “SAN-Net: Learning generalization to unseen sites for stroke lesion segmentation with self-adaptive normalization,” Computers in Biology and Medicine, vol. 156, p. 106717, 2023.
O. Ronneberger et al., “U-Net: Convolutional networks for biomedical image segmentation,” in Proceedings of the MICCAI.   Springer, 2015, pp. 234–241.
N. Karani et al., “Test-time adaptable neural networks for robust medical image segmentation,” Medical Image Analysis, vol. 68, p. 101907, 2021.
J. Ma, “Histogram matching augmentation for domain adaptation with application to multi-centre, multi-vendor and multi-disease cardiac image segmentation,” in Proceedings of the MICCAI Workshop.   Springer, 2021, pp. 177–186.
L. Li et al., “AtrialGeneral: Domain generalization for left atrial segmentation of multi-center LGE MRIs,” in Proceedings of the MICCAI.   Springer, 2021, pp. 557–566.
H. Gunasinghe et al., “Domain generalisation for glaucoma detection in retinal images from unseen fundus cameras,” in Proceedings of the ACIIDS.   Springer, 2022, pp. 421–433.
R. T. Shinohara et al., “Statistical normalization techniques for magnetic resonance imaging,” NeuroImage: Clinical, vol. 6, pp. 9–19, 2014.
J. T. Pontalba et al., “Assessing the impact of color normalization in convolutional neural network-based nuclei segmentation frameworks,” Frontiers in Bioengineering and Biotechnology, vol. 7, p. 300, 2019.
S. Kondo, S. Kasai, and K. Hirasawa, “Tackling mitosis domain generalization in histopathology images with color normalization,” in MICCAI Challenge on Mitosis Domain Generalization.   Springer, 2022, pp. 217–220.
A. Vahadane et al., “Structure-preserving color normalization and sparse stain separation for histological images,” IEEE Transactions on Medical Imaging, vol. 35, no. 8, pp. 1962–1971, 2016.
J. Xiong et al., “Improve unseen domain generalization via enhanced local color transformation,” in Proceedings of the MICCAI.   Springer, 2020, pp. 433–443.
A. Pakzad, K. Abhishek, and G. Hamarneh, “CIRCLe: Color invariant representation learning for unbiased classification of skin lesions,” in Proceedings of the ECCV.   Springer, 2022, pp. 203–219.
Y. Choi et al., “StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation,” in Proceedings of the CVPR, 2018, pp. 8789–8797.
Q. Xu et al., “Fourier-based augmentation with applications to domain generalization,” Pattern Recognition, vol. 139, p. 109474, 2023.
Xu et al., “A fourier-based framework for domain generalization,” in Proceedings of the CVPR, 2021, pp. 14 383–14 392.
M. Clausel, T. Oberlin, and V. Perrier, “The monogenic synchrosqueezed wavelet transform: A tool for the decomposition/demodulation of AM–FM images,” Applied and Computational Harmonic Analysis, vol. 39, no. 3, pp. 450–486, 2015.
X. Zhao et al., “Test-time Fourier style calibration for domain generalization,” in Proceedings of the IJCAI, 2022, pp. 1721–1727.
H. Zhang et al., “Mixup: Beyond empirical risk minimization,” in ICLR, 2018.
H. Liu et al., “Domain generalization in restoration of cataract fundus images via high-frequency components,” in Proceedings of the ISBI.   IEEE, 2022, pp. 1–5.
D. Hu et al., “Domain generalization for retinal vessel segmentation with vector field transformer,” in Proceedings of the MIDL.   PMLR, 2022, pp. 552–564.
H. Lee et al., “Machine friendly machine learning: Interpretation of computed tomography without image reconstruction,” Scientific reports, vol. 9, no. 1, p. 15540, 2019.
L. Gallardo-Estrella et al., “Normalizing computed tomography data reconstructed with different filter kernels: Effect on emphysema quantification,” European Radiology, vol. 26, pp. 478–486, 2016.
I. Zakazov et al., “Feather-light fourier domain adaptation in magnetic resonance imaging,” in Proceedings of the MICCAI Workshop.   Springer, 2022, pp. 88–97.
L. Zhang et al., “Motion correction in MRI using deep learning and a novel hybrid loss function,” arXiv preprint arXiv:2210.14156, 2022.
R. Zhao, H. Li, and X. Liu, “A survey of dictionary learning in medical image analysis and its application for glaucoma diagnosis,” Archives of Computational Methods in Engineering, vol. 28, pp. 463–471, 2021.
M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736–3745, 2006.
P. Song et al., “Coupled dictionary learning for multi-contrast MRI reconstruction,” IEEE Transactions on Medical Imaging, vol. 39, no. 3, pp. 621–633, 2019.
Q. Liu et al., “Single-domain generalization in medical image segmentation via test-time adaptation from shape dictionary,” in Proceedings of the AAAI, vol. 36, no. 2, 2022, pp. 1756–1764.
C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, no. 1, pp. 1–48, 2019.
L. Li et al., “Random style transfer based domain generalization networks integrating shape and spatial information,” in Proceedings of the MICCAI Workshop.   Springer, 2021, pp. 208–218.
D. Tomar et al., “TeSLA: Test-time self-learning with automatic adversarial augmentation,” in Proceedings of the CVPR, 2023, pp. 20 341–20 350.
M. Scalbert et al., “Test-time image-to-image translation ensembling improves out-of-distribution generalization in histopathology,” in Proceedings of the MICCAI.   Springer, 2022, pp. 120–129.
Y. Choi et al., “StarGAN V2: Diverse image synthesis for multiple domains,” in Proceedings of the CVPR, 2020, pp. 8188–8197.
R. Yamashita et al., “Learning domain-agnostic visual representation for computational pathology using medically-irrelevant style transfer augmentation,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3945–3954, 2021.
Y. Xu et al., “Generative adversarial domain generalization via cross-task feature attention learning for prostate segmentation,” in Proceedings of the NeurIPS.   Springer, 2021, pp. 273–284.
A. Taleb et al., “Multimodal self-supervised learning for medical image analysis,” in Proceedings of the IPMI.   Springer, 2021, pp. 661–673.
Y. Xu et al., “Adversarial consistency for single domain generalization in medical image segmentation,” in Proceedings of the MICCAI.   Springer, 2022, pp. 671–681.
Z. Su et al., “Rethinking data augmentation for single-source domain generalization in medical image segmentation,” in Proceedings of the AAAI, vol. 37, no. 2, 2023, pp. 2366–2374.
C. Xu et al., “Improved domain generalization for cell detection in histopathology images via test-time stain augmentation,” in Proceedings of the MICCAI.   Springer, 2022, pp. 150–159.
J.-R. Chang et al., “Stain mix-up: Unsupervised domain generalization for histopathology images,” in Proceedings of the MICCAI.   Springer, 2021, pp. 117–126.
L. Huang et al., “Normalization techniques in training DNNs: Methodology, analysis and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Z. Zhou et al., “Generalizable cross-modality medical image segmentation via style augmentation and dual normalization,” in Proceedings of the CVPR, 2022, pp. 20 856–20 865.
Y.-P. Liu et al., “SS-Norm: Spectral-spatial normalization for single-domain generalization with application to retinal vessel segmentation,” IET Image Processing, 2023.
K. Stacke et al., “Measuring domain shift for deep learning in histopathology,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 2, pp. 325–336, 2020.
J. Lyu et al., “AADG: Automatic augmentation for domain generalization on retinal image segmentation,” IEEE Transactions on Medical Imaging, vol. 41, no. 12, pp. 3699–3711, 2022.
M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” in Proceedings of the NeurIPS, vol. 26, 2013.
H. Li et al., “Domain generalization for medical imaging classification with linear-dependency regularization,” in Proceedings of the NeurIPS, vol. 33, 2020, pp. 3118–3129.
Q. Meng et al., “Mutual information-based disentangled neural networks for classifying unseen categories in different domains: Application to fetal ultrasound imaging,” IEEE Transactions on Medical Imaging, vol. 40, no. 2, pp. 722–734, 2020.
M. I. Belghazi et al., “Mutual information neural estimation,” in Proceedings of the ICML.   PMLR, 2018, pp. 531–540.
Y. Bi et al., “MI-SegNet: Mutual information-based US segmentation for unseen domain generalization,” in Proceedings of the MICCAI.   Springer, 2023.
Y. Chen et al., “D-Bin: A generalized disentangling batch instance normalization for domain adaptation,” IEEE Transactions on Cybernetics, 2021.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
Z. Li et al., “Domain generalization for mammography detection via multi-style and multi-view contrastive learning,” in Proceedings of the MICCAI.   Springer, 2021, pp. 98–108.
R. Gu et al., “Contrastive domain disentanglement for generalizable medical image segmentation,” arXiv preprint arXiv:2205.06551, 2022.
J. S. Yoon, M.-C. Roh, and H.-I. Suk, “A plug-in method for representation factorization in connectionist models,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 8, pp. 3792–3803, 2022.
M. Ilse et al., “DIVA: Domain invariant variational autoencoders,” in Proceedings of the MIDL.   PMLR, 2020, pp. 322–348.
P. Bandi et al., “From detection of individual metastases to classification of lymph node status at the patient level: The camelyon17 challenge,” IEEE Transactions on Medical Imaging, 2018.
Y. Wang et al., “Variational disentanglement for domain generalization,” Transactions on Machine Learning Research, 2022.
C. Wang et al., “Domain invariant model with graph convolutional network for mammogram classification,” arXiv preprint arXiv:2204.09954, 2022.
Wang et al., “Learning domain-agnostic representation for disease diagnosis,” in Proceedings of the ICLR, 2023.
Liu et al., “Recursively conditional gaussian for ordinal unsupervised domain adaptation,” in Proceedings of the ICCV, 2021, pp. 764–773.
X. Liu et al., “Ordinal unsupervised domain adaptation with recursively conditional gaussian imposed variational disentanglement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
S. Wang et al., “DoFE: Domain-oriented feature embedding for generalizable fundus image segmentation on unseen datasets,” IEEE Transactions on Medical Imaging, vol. 39, no. 12, pp. 4237–4248, 2020.
N. Srivastava et al., “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
M. Islam and B. Glocker, “Frequency dropout: Feature-level regularization via randomized filtering,” in Proceedings of the ECCV.   Springer, 2022, pp. 281–295.
K. P. Nguyen, A. H. Treacher, and A. A. Montillo, “Adversarially-regularized mixed effects deep learning (ARMED) models improve interpretability, performance, and generalization on clustered (non-iid) data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Y. Wang et al., “Embracing the dark knowledge: Domain generalization using regularized knowledge distillation,” in Proceedings of the ACM MM, 2021, pp. 2595–2604.
C. Chen et al., “MaxStyle: Adversarial style composition for robust medical image segmentation,” in Proceedings of the MICCAI.   Springer, 2022, pp. 151–161.
T. Zhou and E. Konukoglu, “FedFA: Federated feature augmentation,” in Proceedings of the ICLR, 2023.
R. Wang, P. Chaudhari, and C. Davatzikos, “Embracing the disharmony in medical imaging: A simple and effective framework for domain adaptation,” Medical Image Analysis, vol. 76, p. 102309, 2022.
K. P. Ayodele et al., “Supervised domain generalization for integration of disparate scalp EEG datasets for automatic epileptic seizure detection,” Computers in Biology and Medicine, vol. 120, p. 103757, 2020.
R. A. Kamraoui et al., “DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation,” Medical Image Analysis, vol. 76, p. 102312, 2022.
M. Philipp et al., “Dynamic CNNs using uncertainty to overcome domain generalization for surgical instrument localization,” in Proceedings of the WACV, 2022, pp. 3612–3621.
Q. Dou et al., “Domain generalization via model-agnostic learning of semantic features,” in Proceedings of the NeurIPS, vol. 32, 2019.
Q. Liu, Q. Dou, and P.-A. Heng, “Shape-aware meta-learning for generalizing prostate MRI segmentation to unseen domains,” in Proceedings of the MICCAI.   Springer, 2020, pp. 475–485.
X. Liu et al., “Semi-supervised meta-learning with disentanglement for domain-generalised medical image segmentation,” in MICCAI.   Springer, 2021, pp. 307–317.
C. Ouyang et al., “Self-supervision with superpixels: Training few-shot medical image segmentation without annotation,” in Proceedings of the ECCV.   Springer, 2020, pp. 762–780.
S. Azizi et al., “Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging,” Nature Biomedical Engineering, pp. 1–24, 2023.
I. E. I. Bekkouch et al., “Adversarial reconstruction loss for domain generalization,” IEEE Access, vol. 9, pp. 42 424–42 437, 2021.
C. Chen et al., “Realistic adversarial data augmentation for MR image segmentation,” in Proceedings of the MICCAI.   Springer, 2020, pp. 667–677.
Z. Zhang et al., “Domain generalization with adversarial intensity attack for medical image segmentation,” arXiv preprint arXiv:2304.02720, 2023.
K. Goel et al., “Model patching: Closing the subgroup performance gap with data augmentation,” ICLR, 2021.
T. Hastie et al., The Elements of Statistical Learning: Data Mining, Inference, and Prediction.   Springer, 2009, vol. 2.
S. Sagawa et al., “Distributionally robust neural networks,” in Proceedings of the ICLR, 2019.
E. Delage and Y. Ye, “Distributionally robust optimization under moment uncertainty with application to data-driven problems,” Operations Research, vol. 58, no. 3, pp. 595–612, 2010.
A. Bissoto et al., “Artifact-based domain generalization of skin lesion models,” in Proceedings of the ECCV.   Springer, 2022, pp. 133–149.
M. M. Bronstein et al., “Geometric deep learning: Grids, groups, graphs, geodesics, and gauges,” arXiv preprint arXiv:2104.13478, 2021.
D. M. Nguyen et al., “LVM-Med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching,” arXiv preprint arXiv:2306.11925, 2023.
L. Seenivasan et al., “Biomimetic incremental domain generalization with a graph network for surgical scene understanding,” Biomimetics, vol. 7, no. 2, p. 68, 2022.
A. Santhirasekaram et al., “Topology preserving compositionality for robust medical image segmentation,” in Proceedings of the CVPR, 2023, pp. 543–552.
G. A. Kaissis et al., “Secure, privacy-preserving and federated machine learning in medical imaging,” Nature Machine Intelligence, vol. 2, no. 6, pp. 305–311, 2020.
Q. Xu et al., “Federated adversarial domain hallucination for privacy-preserving domain generalization,” IEEE Transactions on Multimedia, 2023.
M. Abadi et al., “Deep learning with differential privacy,” in Proceedings of the ACM CCS, 2016, pp. 308–318.
Y. Chen et al., “FedHealth: A federated transfer learning framework for wearable healthcare,” IEEE Intelligent Systems, vol. 35, no. 4, pp. 83–93, 2020.
Q. Liu et al., “FedDG: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space,” in Proceedings of the CVPR, 2021, pp. 1013–1023.
J. Chen et al., “Federated domain generalization for image recognition via cross-client style transfer,” in Proceedings of the WACV, 2023, pp. 361–370.
W. Li et al., “Privacy-preserving federated brain tumour segmentation,” in Proceedings of the MLMI.   Springer, 2019, pp. 133–141.
N. M. Müller et al., “Shortcut removal for improved OOD-generalization,” in Proceedings of the CVPR Workshop, 2023.
J. Yang, K. Zhou, and Z. Liu, “Full-spectrum out-of-distribution detection,” International Journal of Computer Vision, pp. 1–16, 2023.
D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in Proceedings of ICLR, 2017.
S. Ben-David et al., “A theory of learning from different domains,” Machine Learning, vol. 79, pp. 151–175, 2010.
D. Li et al., “Deeper, broader and artier domain generalization,” in Proceedings of the ICCV, 2017, pp. 5542–5550.
D. Mahajan, S. Tople, and A. Sharma, “The connection between out-of-distribution generalization and privacy of ML models,” ICLR, 2022.
A. Puli et al., “Out-of-distribution generalization in the presence of nuisance-induced spurious correlations,” ICLR, 2022.
Puli et al., “Nuisances via negativa: Adjusting for spurious correlations via data augmentation,” ICLR, 2023.
I. Gao et al., “Out-of-distribution robustness via targeted augmentations,” in Proceedings of the NeurIPS Workshop, 2022.
G. Baryannis, S. Dani, and G. Antoniou, “Predicting supply chain risks using machine learning: The trade-off between performance and interpretability,” Future Generation Computer Systems, vol. 101, pp. 993–1004, 2019.
A. Singh, S. Sengupta, and V. Lakshminarayanan, “Explainable deep learning models in medical image analysis,” Journal of Imaging, vol. 6, no. 6, p. 52, 2020.
B. H. Van der Velden et al., “Explainable artificial intelligence (XAI) in deep learning-based medical image analysis,” Medical Image Analysis, p. 102470, 2022.
D. Wang et al., “Proactive pseudo-intervention: Causally informed contrastive learning for interpretable vision models,” arXiv preprint arXiv:2012.03369, 2020.
M. R. Karim et al., “DeepKneeExplainer: Explainable knee osteoarthritis diagnosis from radiographs and magnetic resonance imaging,” IEEE Access, vol. 9, pp. 39 757–39 780, 2021.
Y. Wang et al., “SEEG-Net: An explainable and deep learning-based cross-subject pathological activity detection method for drug-resistant epilepsy,” Computers in Biology and Medicine, vol. 148, p. 105703, 2022.
A. Chattopadhay et al., “Grad-Cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in Proceedings of the WACV.   IEEE, 2018, pp. 839–847.
S. Gao et al., “BayeSeg: Bayesian modeling for medical image segmentation with interpretable generalizability,” arXiv preprint arXiv:2303.01710, 2023.
J. Yuan et al., “Not just pretty pictures: Text-to-image generators enable interpretable interventions for robust representations,” arXiv preprint arXiv:2212.11237, 2022.
D. Mahajan, S. Tople, and A. Sharma, “Domain generalization using causal matching,” in Proceedings of the ICML.   PMLR, 2021, pp. 7313–7324.
R. Wang, P. Chaudhari, and C. Davatzikos, “Harmonization with flow-based causal inference,” in Proceedings of the MICCAI.   Springer, 2021, pp. 181–190.
C. Ouyang et al., “Causality-inspired single-source domain generalization for medical image segmentation,” IEEE Transactions on Medical Imaging, vol. 42, no. 4, pp. 1095–1106, 2023.
V. M. Campello et al., “Multi-centre, multi-vendor and multi-disease cardiac segmentation: The M&Ms challenge,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3543–3554, 2021.
M. Aubreville et al., “Mitosis domain generalization in histopathology images—the midog challenge,” Medical Image Analysis, vol. 84, p. 102699, 2023.
A. F. Kazerooni et al., “The brain tumor segmentation (BraTS) challenge 2023: Focus on pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs),” arXiv preprint arXiv:2305.17033, 2023.
M. Adewole et al., “The brain tumor segmentation (BraTS) challenge 2023: Glioma segmentation in sub-saharan africa patient population (BraTS-Africa),” arXiv preprint arXiv:2305.19369, 2023.
H. Bogunović et al., “RETOUCH: The retinal OCT fluid detection and segmentation benchmark and challenge,” IEEE Transactions on Medical Imaging, vol. 38, no. 8, pp. 1858–1874, 2019.
Q. Meng, D. Rueckert, and B. Kainz, “Learning cross-domain generalizable features by representation disentanglement,” arXiv preprint arXiv:2003.00321, 2020.
R. Gu et al., “CDDSA: Contrastive domain disentanglement and style augmentation for generalizable medical image segmentation,” Medical Image Analysis, vol. 89, p. 102904, 2023.
J. Lee et al., “Site-invariant meta-modulation learning for multisite autism spectrum disorders diagnosis,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14, 2023.
W. Fan et al., “InvNorm: Domain generalization for object detection in gastrointestinal endoscopy,” arXiv preprint arXiv:2205.02842, 2022.
S. Korevaar, R. Tennakoon, and A. Bab-Hadiashar, “Failure to achieve domain invariance with domain generalization algorithms: An analysis in medical imaging,” IEEE Access, 2023.
H. Zhang et al., “An empirical framework for domain generalization in clinical settings,” in Proceedings of the CHIL, 2021, pp. 279–290.
[{{figure:e578bf48-105e-4664-85b5-5d5f1e4f24e4}}
His research interests include explainable AI, computer vision, and representation learning.
[{{figure:981d1c18-7472-417b-9694-3684586c369b}}
His current research interests include explainable AI, computer vision, and machine/deep learning.
[{{figure:749f8433-b6b3-4bf4-b00f-f0d8cb0dd633}}
His research interests include transfer learning, computer vision, and machine/deep learning.
[{{figure:2ea7eb07-9eec-4b4e-a317-829a7c03f8e3}}
[{{figure:6f775985-6d03-4e7b-aafc-c58c96584e07}}
He was awarded a Kakao Faculty Fellowship from Kakao and a Young Researcher Award from the Korean Society for Human Brain Mapping (KHBM) in 2018 and 2019, respectively. His research interests include causal machine/deep learning, explainable AI, biomedical data analysis, and brain-computer interface.
Dr. Suk serves as an Editorial Board Member for Clinical and Molecular Hepatology (Artificial Intelligence Sector), Electronics, Frontiers in Neuroscience, Frontiers in Radiology (Artificial Intelligence in Radiology), International Journal of Imaging Systems and Technology (IJIST), and a Program Committee or a Reviewer for NeurIPS, ICML, ICLR, AAAI, IJCAI, CVPR, MICCAI, AISTATS, etc.

Supplementary Material
Scope of Survey
Number of publications per year on the Google Scholar database.
[
width=0.5height=4cm,
xticklabel style=rotate=330, /pgf/number format/set thousands separator=,
xtick=2013,...,2022,
xtick distance=1,
axis lines=left,
tick label style=font=
]table
Year Publications
2013 13
2014 19
2015 19
2016 23
2017 26
2018 40
2019 73
2020 212
2021 507
2022 990
;

text width = 4cm, align = left] at (2.5,1.4)Keyword: ("domain generalization" OR "domain generalisation") AND (intitle:medical OR ct OR ultrasound OR mri OR pet OR xray OR histology OR histopathology OR pathology OR fundus OR dermoscopy OR endoscopy OR mammography);{{figure:bd2d2ebd-76cf-4735-8c80-7a64d516eabd}}Literature search and selection were conducted by researchers experienced in machine learning and medical image analysis.
We used the Google Scholar search engine with three different search strategies, resulting in a database of 1,621 papers.
First, we searched for all papers that cited the existing domain generalization survey {{cite:3b55acf}}, {{cite:121c83b}}, {{cite:f71156c}}, {{cite:2b4f1ca}}, {{cite:848a605}}, {{cite:b4119d1}}, {{cite:6913ba9}}.
Second, we searched Google Scholar using the exact phrase “domain generaliz(s)ation” and medical-related keywords (“Medical, CT, ultrasound, MRI, PET, X-ray, histology, histopathology, pathology, fundus, dermoscopy, endoscopy, mammography”) and selected 1,000 papers sorted by relevancy.
Lastly, we searched for the top 1,000 papers with the terms “unseen”, “domain”, and medical-related keywords.
The eligibility criteria for papers to be included in this survey are that they have conducted at least one experiment involving the use of medical images within the domain generalization problem settings (see {{formula:9efdd061-96ca-4657-aa7d-e8ec1dc08382}}REF ), regardless of their use of the term “domain generalization” in their paper (small number of papers instead use “unseen”, “out-of-distribution”, or their own terms).
Peer-reviewed published papers were prioritized, but non-peer-reviewed archive papers (e.g., arXiv, bioRxiv) were also included if they had been deemed particularly suitable for selection (e.g., highly relevant, highly significant, highly cited).

Summary of Domain Generalization Methods
Data-level Domain Generalization Methods
Data-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:14c82f83-0a71-4767-9a73-ccf693e7b0ac}}REF ), task, organ, and modality used in experiment.

Method
Strategy
Ref.
Source
Target
Task
Organ
Modality
82.4cmImage Processing
31.5cmIntensity Normalization
{{cite:9f4531a}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:472bf2f}}
Single
Site, Sequence
Segmentation
Brain, Prostate, Heart
MRI
black(lr)2-8black
41.5cmHistogram Matching
{{cite:1491ffc}}
Multiple
Site
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:ea82163}}
Multiple
Site
Segmentation
Atrial
MRI
lightgray!20(lr)3-8black

{{cite:0b8e909}}
Multiple
Site
Detection
Retinal
Fundus
black(lr)2-8black
41.5cmColor Normalization
{{cite:c0b28e0}}
Multiple
Site
Detection
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:e1d0d60}}
Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:a2893d2}}
Multiple
Site
Classification
Skin
Dermatology
92.4cmSurrogate
71.5cmFrequency-based DG
{{cite:490bd53}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:cda2c67}}
Single, Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:7a6e44d}}
Multiple
Site
Restoration
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:2391f80}}
Multiple
Site
Segmentation
Vessel
2D Image
black(lr)2-8black
2.51.5cmUsing Raw Signals
{{cite:0a258ac}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:ad4ef31}}
Single
Sequence
Segmentation
Brain
MRI
black(lr)2-8black
2.51.5cmDictionary Learning
{{cite:5ad8d1d}}
Multiple
Sequence
Reconstruction
Brain
MRI
lightgray!20(lr)3-8black

{{cite:d8c3a02}}
Single
Site
Segmentation
Prostate, Retinal
MRI, Fundus
52.4cmAugmentation
2.51.5cmRandomization-based
{{cite:1c6fbcb}}
Single
Sequence
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:7a6e44d}}
Multiple
Site
Restoration
Retinal
Fundus
black(lr)2-8black
Adversarial-based
{{cite:af60507}}
Multiple
Site
Segmentation
Spinal, Prostate
MRI
black(lr)2-8black
2.51.5cmGenerative Model
{{cite:3973f40}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:b6239e8}}
Single
Site
Classification
Tissue
Histology
72.4cmProblem-specific
6.51.5cmCross-modal Generative Model
{{cite:6dca1c4}}
Multiple
Modality, Sequence
Segmentation
Brain, Prostate, Liver
MRI, CT
lightgray!20(lr)3-8black

{{cite:134ad1c}}
Single
Modality, Sequence
Segmantation
Prostate, Abdominal, Cardiac
MRI, CT
lightgray!20(lr)3-8black

{{cite:4e36271}}
Single
Modality
Segmentation
Cardiac, Abdominal
MRI
black(lr)2-8black
3.51.5cmStain Normalization
{{cite:09f207e}}
Multiple
Site
Detection
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:ad3e51b}}
Multiple
Site
Classification, Segmentation
Tissue
Histology, hematology
{{table:e6765bcd-936e-44ae-b8e2-c25cb5857b7d}}{{table:d870e699-d844-4aba-b056-4fd4eb2809fc}}
Feature-level Domain Generalization Methods
Feature-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:fbe950f5-9df0-4553-a723-f946cb49047c}}REF ), task, organ, and modality used in experiment.

Method
Specific
Ref.
Source
Target
Task
Organ
Modality
5*Feature Alignment
2.51.5cmFeature Normalization
{{cite:028b0f2}}
Single
Sequence, Modality
Segmentation
Brain, Cardiac, Abdominal
MRI, CT
lightgray!20(lr)3-8black

{{cite:897b234}}
Single
Site
Segmentation
Retinal
Fundus
black(lr)2-8black
5.31.5cmDissimilarity-based
{{cite:93a257f}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:7c02e53}}
Multiple
Modality
Segmentation
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:0ed736d}}
Multiple
Sequence, Site
Classification, Segmentation
Skin, Brain
Dermatology, MRI
20*Implicit Disentanglement
141.5cmMutual Information
{{cite:b64a4a0}}
Multiple
Site
Classification
Abdominal, Brain, Femur, Lips
Fetal Ultrasound
lightgray!20(lr)3-8black

{{cite:e5ac965}}
Multiple
Site
Classification
Abdominal, Brain, Femur, Lips
Fetal Ultrasound
lightgray!20(lr)3-8black

{{cite:80fbeda}}
Single
Site
Segmentation
Carotid
Ultrasound
lightgray!20(lr)3-8black

{{cite:fec8a3a}}
Single, Multiple
Sequence
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:134ad1c}}
Single
Modality, Sequence
Segmantation
Prostate, Abdominal, Cardiac
MRI, CT
black(lr)2-8black
2.51.5cmContrastive
{{cite:a43d7a4}}
Multiple
Site
Detection
Breast
X-ray
lightgray!20(lr)3-8black

{{cite:f43e4b0}}
Multiple
Site
Segmentation
Retinal
Fundus
black(lr)2-8black
61.5cmVariational
{{cite:9af3409}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:b2d3294}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:7b9d14c}}
Multiple
Site
Classification
Breast
X-ray
lightgray!20(lr)3-8black

{{cite:a47ac77}}
Multiple
Site
Classification
Breast
X-ray
6.5*Explicit Disentanglement
4.31.5cmConditional Representation Learning
{{cite:cc4f265}}
Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:cce9b83}}
Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:5d7f30c}}
Multiple
Site
Segmentation
Retinal
Fundus
black(lr)2-8black
5.21.5cmFeature Reguarlization
{{cite:99cb615}}
Multiple
Site
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:db2ff2e}}
Multiple
Site
Compression, classification
Brain, Skin
MRI, Dermatology
lightgray!20(lr)3-8black

{{cite:4fc3027}}
Single
Site
Segmentation
Spinal
MRI
6*Others
2.51.5cmFeature Augmentation
{{cite:554c934}}
Single, Multiple
Site
Segmentation
Cardiac, Prostate
MRI
lightgray!20(lr)3-8black

{{cite:3f89dbb}}
Multiple
Site
Segmentation
Prostate
MRI
black(lr)2-8black
2.61.5cmKernel-based
{{cite:f124298}}
Single
Site
Classification
Brain
MRI
lightgray!20(lr)3-8black

{{cite:a6e1be3}}
Multiple
Site
Detection
Brain
EEG
{{table:76f4f39a-9bdd-4a6f-a51e-e022131a81cb}}{{table:2247a600-f84f-411e-bea0-78ca9ebc2906}}
Model-level Domain Generalization Methods
Model-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:59204db7-b73c-4d01-a33c-a345761589ac}}REF ), task, organ, and modality used in experiment.

Method
Specific
Ref.
Source
Target
Task
Organ
Modality
21*Learning Strategy
61.5cmEnsemble Learning
{{cite:2655543}}
Multiple
Sequence, Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:c27d1f0}}
Single
Sequence
Localization
Surgical Scene
Video Frames
lightgray!20(lr)3-8black

{{cite:3973f40}}
Single, Multiple
Site
Classification
Tissue
Histology
black(lr)2-8black
6.51.5cmMeta-learning
{{cite:303e624}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:9a25ddc}}
Multiple
Site
Segmentation
Prostate
MRI
lightgray!20(lr)3-8black

{{cite:b8ec0c1}}
Multiple
Site, Sequence
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:a725c5c}}
Single, Multiple
Site
Classification
Brain
Functional MRI
black(lr)2-8black
71.5cmSelf-supervised Learning
{{cite:f43e4b0}}
Multiple
Site
Segmentation
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:eb215d6}}
Multiple
Modality
Segmentation
Abdominal, Cardiac
CT, MRI
lightgray!20(lr)3-8black

{{cite:a128b07}}
Multiple
Site
Classification
Skin, Retinal, Chest, Breast
Fundus, Dermatology, X-ray, Mammography
black(lr)2-8black
41.5cmAdversarial Learning
{{cite:5885eb2}}
Single, Multiple
Sequence
Detection
Skin, Hip
MRI
lightgray!20(lr)3-8black

{{cite:f0bfbd8}}
Multiple
Site
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:afbd9f4}}
Multiple
Site
Segmentation
Retinal, Prostate
Fundus, MRI
2.5*Distributionally Robust Optimization

{{cite:d5c5153}}
Multiple
Site
Classification
Skin
Dermatology
lightgray!20(lr)3-8black

{{cite:2ffea0f}}
Multiple
Site
Classification
Skin
Dermatology
17.4*Other
5.51.5cmGeometric Learning
{{cite:22ffac2}}
Multiple
Modal, Sequence, Site
Classification, segmentation, detection
Multiple organs, modalities from 55 datasets
lightgray!20(lr)3-8black

{{cite:58fba09}}
Multiple
Site
Segmentation
Surgical Scene
Video Frames
lightgray!20(lr)3-8black

{{cite:778a6fe}}
Single
Site
Segmentation
Abdomen, Cardiac, Prostate
MRI
black(lr)2-8black
5.51.5cmDistributed Learning
{{cite:0e442fe}}
Multiple
Site
Segmentation
Retinal, Prostate
Fundus, MRI
lightgray!20(lr)3-8black

{{cite:f05d11b}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:132a815}}
Multiple
Site
Segmentation
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:0002072}}
Multiple
Modality
Segmentation
Tumor
MRI
black(lr)2-8black
9.21.5cmOOD
{{cite:803c1f2}}
Multiple
Site
Detection
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:96e0a37}}
Single
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:f575f9c}}
Multiple
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:eb09003}}
Single
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:af0a2a7}}
Single
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:30133ba}}
Multiple
Site
Classification
Tissue
Histology
{{table:e1b324ae-f5cb-4845-b836-14b9b7f2c56d}}{{table:9e359065-8c8a-46ed-9fe4-23eca628b680}}
Analysis-level Domain Generalization Methods
Analysis-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:8d13df2a-a30c-4693-86a6-d01fffe6f32b}}REF ), task, organ, and modality used in experiment.

Method
Ref.
Source
Target
Task
Organ
Modality
72.2cmInterpretable AI
{{cite:3986ccf}}
Single
Site
Classification, Segmentation
Retinal, Cardiac
OCT, CT
lightgray!20(lr)3-7black
{{cite:d526c75}}
Single
Site
Classification, Segmentation
Knee
MRI
lightgray!20(lr)3-7black
{{cite:e1974c1}}
Multiple
Site
Classification
Brain
sEEG
lightgray!20(lr)3-7black
{{cite:715a58c}}
Multiple
Sequence
Detection
Gastrointestinal
Endoscopy
3.51.5cmTransferability
{{cite:9ab1ada}}
Single
Sequence, Site
Segmentation
Cardiac, Prostate
MRI, CT
lightgray!20(lr)3-7black
{{cite:d2e67c0}}
Single, Multiple
Site
Classification
Tissue
Histology
71.5cmCausality
{{cite:cec76a3}}
Single, Multiple
Sequence
Classification
Chest
X-ray
lightgray!20(lr)3-7black
{{cite:db6aae6}}
Multiple
Site
Regression, Classification
Brain
MRI
lightgray!20(lr)3-7black
{{cite:d4706ff}}
Single
Modality, Sequence, Site
Segmentation
Prostate, Abdominal, Cardiac
MRI, CT
{{table:b11e2a4d-f86b-4fa6-860e-07a84b56b971}}{{table:187b8858-c6d5-4149-b88a-50758f3d3d6d}}
Suggestions for Domain Generalization for Medical Image Analysis
Problem-specific suggestion for strategies for integrating domain generalization into MedIA workflow. Diamond box indicate the start terminator, angled boxes indicate the process, and round boxes indicate the decision.
rectangleround = [rectangle, text width=1.77cm, rounded corners, minimum width=1.23cm, minimum height=1.33cm, text centered, font=, color=gray, draw=black, line width=0.1, fill=teal!10]
rectangleangled = [rectangle, text width=1.77cm, minimum width=1.23cm, minimum height=1.33cm, text centered, font=, color=gray, draw=black, line width=0.1, fill=teal!20]
diamondbox = [diamond, text width=0.75cm, minimum width=0.75cm, minimum height=0.1cm, text centered, font=, color=gray, draw=black, line width=0.1, fill=teal!30][node distance=3cm]
frequency) [rectangleround] {{formula:0daac278-6b73-4081-be83-17e87b68dcec}} REFFrequency-based;
input) [diamondbox, right of = frequency] Start;
related) [rectangleround , right of=input] {{formula:f72debc2-bc75-4260-a4fa-99854fb7efc4}} REFRelated ML tasks;sufficient) [rectangleangled , below of=frequency, yshift=1cm] Sufficient data?;
raw) [rectangleangled , right of=sufficient] Raw measurement available?;
access) [rectangleangled , right of=raw] Access to {{formula:935ee198-945e-42ce-a0bc-31f77e1db2b6}} ?;augmentation) [rectangleround , below of=sufficient, yshift=1cm] {{formula:ffa1eeec-7960-4ee0-b56a-1ba55c89d1fe}} REFDataAugmentation;
preprocessing) [rectangleangled , right of=augmentation] Preprocessing required?;
specifictask) [rectangleangled , right of=preprocessing] Specific task or problem at hand?;analysis) [rectangleangled , below of=augmentation, yshift=1cm] Analysis or interpretability required?;
problem) [rectangleround , right of=analysis] {{formula:22413421-8b76-4426-866d-e6b1728eee42}} REFProblem-specific;
imageprocessing) [rectangleround , right of=problem] {{formula:7d55863c-c3ba-4cc2-ace0-d273f1f0df3d}} REFImage Processing;analysislevel) [rectangleround , below of=analysis, yshift=1cm] {{formula:689da5f8-865b-4ea8-8995-2dd6e64c8a96}} REFAnalysis-level DG;
specialalgorithm) [rectangleangled , right of=analysislevel] Able to use special algorithm or architecture?;
modellevel) [rectangleround , right of=specialalgorithm] {{formula:d5e4f1d2-bfd8-4a4a-a5ea-336ee42143e9}} REFModel-level DG;alignment) [rectangleround , below of=analysislevel, yshift=1cm] {{formula:5b58b8bf-443f-465c-a04b-8eff7525b569}} REFFeature Alignment;
invariance) [rectangleangled , right of=alignment] Invariant v.s. Disentangled Features;
disentanglement) [rectangleround , right of=invariance] {{formula:39f783fa-f2df-47e7-8cd8-a3d6c39330d1}} REFFeature Disentanglement;[-Stealth] (input) – (access);
[-Stealth] (access) – node[anchor=west] Y (related);
[-Stealth] (access) – node[anchor=north] N (raw);
[-Stealth] (raw) – node[anchor=north] Y (frequency);
[-Stealth] (raw) – node[anchor=north] N (sufficient);
[-Stealth] (sufficient) – node[anchor=north] Y (preprocessing);
[-Stealth] (sufficient) – node[anchor=west] N (augmentation);
[-Stealth] (preprocessing) – node[anchor=north] Y (specifictask);
[-Stealth] (specifictask) – node[anchor=north] Y (problem);
[-Stealth] (specifictask) – node[anchor=west] N (imageprocessing);[-Stealth] (preprocessing) – node[anchor=north] N (analysis);
[-Stealth] (analysis) – node[anchor=west] Y (analysislevel);
[-Stealth] (analysis) – node[anchor=north] N (specialalgorithm);
[-Stealth] (specialalgorithm) – node[anchor=north] Y (modellevel);
[-Stealth] (specialalgorithm) – node[anchor=west] N (invariance);
[-Stealth] (invariance) – node[anchor=north] Inv. (alignment);
[-Stealth] (invariance) – node[anchor=north] Dis. (disentanglement);{{figure:7c44091e-5c93-4ff3-9482-99d947efa557}}There exist a number of benchmarks for commonly used domain generalization techniques suggesting appropriate methods for specific tasks at hand. For example, Korevaar et al.  {{cite:687462e}} evaluated methods on benchmark datasets, whereas Zhang et al.  {{cite:89d4311}} evaluated them on some custom cross-site datasets consisting of publicly available datasets.
While these benchmarks shed light on the capabilities of certain techniques in domain generalization, they do not offer a comprehensive guide for problem-specific and task-specific strategies throughout the MedIA workflow.
To this end, we suggest strategies to incorporate domain generalization into the model, as shown in Fig. {{figure:7c44091e-5c93-4ff3-9482-99d947efa557}} .


Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasets

Abstract: The availability of different pre-trained semantic models enabled the quick development of machine learning components for downstream applications. Despite the availability of abundant text data for low resource languages, only a few semantic models are publicly available. Publicly available pre-trained models are usually built as a multilingual version of semantic models that can not fit well for each language due to context variations. In this work, we introduce different semantic models for Amharic. After we experiment with the existing pre-trained semantic models, we trained and fine-tuned nine new different models using a monolingual text corpus. The models are build using word2Vec embeddings, distributional thesaurus (DT), contextual embeddings, and DT embeddings obtained via network embedding algorithms. Moreover, we employ these models for different NLP tasks and investigate their impact. We find that newly trained models perform better than pre-trained multilingual models. Furthermore, models based on contextual embeddings from RoBERTA perform better than the word2Vec models.

Machine Learning Approaches for Amharic Parts-of-speech Tagging

Abstract: Part-of-speech (POS) tagging is considered as one of the basic but necessary tools which are required for many Natural Language Processing (NLP) applications such as word sense disambiguation, information retrieval, information processing, parsing, question answering, and machine translation. Performance of the current POS taggers in Amharic is not as good as that of the contemporary POS taggers available for English and other European languages. The aim of this work is to improve POS tagging performance for the Amharic language, which was never above 91%. Usage of morphological knowledge, an extension of the existing annotated data, feature extraction, parameter tuning by applying grid search and the tagging algorithms have been examined and obtained significant performance difference from the previous works. We have used three different datasets for POS experiments.

Summary of related works for selected Ethiopian languages in POS tag tasks, Size shows the number of tokens used during the experiment, Score shows the outperformed model results evaluated using accuracy score, Dataset and Model shows the availability of dataset and models in publicly accessible repositories.

Table {{table:c0d0bec4-12b1-47ce-bbe1-9cb96f61409a}}  summarizes the current state of POS tagging research for selected Ethiopian languages. The table shows the name(s) of the author(s), the size of the dataset, the method used, the accuracy score of the models, and the availability of datasets and models in public repositories.

From Table {{table:c0d0bec4-12b1-47ce-bbe1-9cb96f61409a}} , we can conclude that POS tagging is less researched for Ethiopian languages, the majority of the works were found for Amharic than for the other languages. From the works discussed in Table {{table:c0d0bec4-12b1-47ce-bbe1-9cb96f61409a}}  only the work by {{cite:f0a035e}} made their models and datasets available for public use.


SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Teaching Machines to Read and Comprehend

Abstract: Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

Pointer Sentinel Mixture Models

Abstract: Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

Overall performance of adaptive MoE and compared baselines in different NLP tasks. All the models converge to the same loss value.

Table {{table:35da103c-f36a-4137-b2ce-8e54cf6d52b8}}  summarizes the details.


Modeled the tumor density of patients with glioblastoma using a tumor growth model under the Bayesian framework.

GMM-based glioblastoma feature extraction approach.

Personalized Radiotherapy Design for Glioblastoma: Integrating Mathematical Tumor Models, Multimodal Scans, and Bayesian Inference

Abstract: Glioblastoma (GBM) is a highly invasive brain tumor, whose cells infiltrate surrounding normal brain tissue beyond the lesion outlines visible in the current medical scans. These infiltrative cells are treated mainly by radiotherapy. Existing radiotherapy plans for brain tumors derive from population studies and scarcely account for patient-specific conditions. Here, we provide a Bayesian machine learning framework for the rational design of improved, personalized radiotherapy plans using mathematical modeling and patient multimodal medical scans. Our method, for the first time, integrates complementary information from high-resolution MRI scans and highly specific FET-PET metabolic maps to infer tumor cell density in GBM patients. The Bayesian framework quantifies imaging and modeling uncertainties and predicts patient-specific tumor cell density with credible intervals. The proposed methodology relies only on data acquired at a single time point and, thus, is applicable to standard clinical settings. An initial clinical population study shows that the radiotherapy plans generated from the inferred tumor cell infiltration maps spare more healthy tissue thereby reducing radiation toxicity while yielding comparable accuracy with standard radiotherapy protocols. Moreover, the inferred regions of high tumor cell densities coincide with the tumor radioresistant areas, providing guidance for personalized dose-escalation. The proposed integration of multimodal scans and mathematical modeling provides a robust, non-invasive tool to assist personalized radiotherapy design.

Automated Feature Extraction in Brain Tumor by Magnetic Resonance Imaging Using Gaussian Mixture Models

Abstract: This paper presents a novel method for Glioblastoma (GBM) feature extraction based on Gaussian mixture model (GMM) features using MRI. We addressed the task of the new features to identify GBM using T1 and T2 weighted images (T1-WI, T2-WI) and Fluid-Attenuated Inversion Recovery (FLAIR) MR images. A pathologic area was detected using multithresholding segmentation with morphological operations of MR images. Multiclassifier techniques were considered to evaluate the performance of the feature based scheme in terms of its capability to discriminate GBM and normal tissue. GMM features demonstrated the best performance by the comparative study using principal component analysis (PCA) and wavelet based features. For the T1-WI, the accuracy performance was 97.05% (AUC = 92.73%) with 0.00% missed detection and 2.95% false alarm. In the T2-WI, the same accuracy (97.05%, AUC = 91.70%) value was achieved with 2.95% missed detection and 0.00% false alarm. In FLAIR mode the accuracy decreased to 94.11% (AUC = 95.85%) with 0.00% missed detection and 5.89% false alarm. These experimental results are promising to enhance the characteristics of heterogeneity and hence early treatment of GBM.

Overview of brain tumor detection studies based on statistical methods

In this section, we reviewed the statistical approaches used for modeling and optimization of brain tumor segmentation models, summarised in Table {{table:abd4c4a1-6690-48b8-9064-dbb1428d64d8}} . Justifying the underlying probability distribution that drives the data is an important facet of statistical model-based segmentation methods. These methods have the intrinsic limitations of requiring domain expertise and knowledge of prior distribution. Translating information about prior knowledge into a probabilistic model can be challenging, and inference may involve extensive computations {{cite:2cf0744}}. Furthermore, statistical tumor models are often parameterized by many parameters. Estimation of unknown parameters could be computationally challenging due to the difficulties posed by mathematical issues such as non-convexity and model uncertainties. These shortcomings restrict their deployment in clinical practice. However, model-based approaches are effective in terms of model interpretability and the amount of data required for the analysis.


Rethinking the Inception Architecture for Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.

Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Deep Learning Recommendation Model for Personalization and Recommendation Systems

Abstract: With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.

Overview of the six benchmark models evaluated.

Benchmarks. Table {{table:2a6ce3fb-4001-410f-81f8-5127352b8439}}  summarizes the six representative DNN models that we used as benchmarks, they are widely used in prior works {{cite:0ed1950}}, {{cite:f363c88}}, {{cite:f407cc6}}, {{cite:dbb1f33}}. We evaluate throughput with synthetic dataset, which ignores the data loading latency.
Modeling real-world dataset is orthogonal to Proteus.


Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark

Abstract: Psychological research results have confirmed that people can have different emotional reactions to different visual stimuli. Several papers have been published on the problem of visual emotion analysis. In particular, attempts have been made to analyze and predict people's emotional reaction towards images. To this end, different kinds of hand-tuned features are proposed. The results reported on several carefully selected and labeled small image data sets have confirmed the promise of such features. While the recent successes of many computer vision related tasks are due to the adoption of Convolutional Neural Networks (CNNs), visual emotion analysis has not achieved the same level of success. This may be primarily due to the unavailability of confidently labeled and relatively large image data sets for visual emotion analysis. In this work, we introduce a new data set, which started from 3+ million weakly labeled images of different emotions and ended up 30 times as large as the current largest publicly available visual emotion data set. We hope that this data set encourages further research on visual emotion analysis. We also perform extensive benchmarking analyses on this large data set using the state of the art methods including CNNs.

Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks

Abstract: Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.

Statistics of the involved visual emotion datasets.

We evaluate the proposed SOLVER on eight public visual emotion datasets, including the Flickr and Instagram (FI) {{cite:2b84c05}}, Flickr, Instagram {{cite:9dcef20}}, EmotionROI {{cite:22c29f3}}, Twitter I {{cite:360c2c9}}, Twitter II {{cite:dbf19b8}}, ArtPhoto {{cite:35f17cf}} and IAPSa {{cite:b4bd284}}, {{cite:a6eef91}}.
The involved datasets can be roughly divided into large-scale datasets (i.e., FI, Flickr, Instagram) and small-scale datasets (i.e., EmotionROI, Twitter I, Twitter II, Artphoto, IAPSa), for which more details are shown in TABLE {{table:edd9bf5d-20de-4634-8587-1fe3f7bed054}} .


Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts

Abstract: The use of robo-readers to analyze news texts is an emerging technology trend in computational finance. In recent research, a substantial effort has been invested to develop sophisticated financial polarity-lexicons that can be used to investigate how financial sentiments relate to future company performance. However, based on experience from other fields, where sentiment analysis is commonly applied, it is well-known that the overall semantic orientation of a sentence may differ from the prior polarity of individual words. The objective of this article is to investigate how semantic orientations can be better detected in financial and economic news by accommodating the overall phrase-structure information and domain-specific use of language. Our three main contributions are: (1) establishment of a human-annotated finance phrase-bank, which can be used as benchmark for training and evaluating alternative models; (2) presentation of a technique to enhance financial lexicons with attributes that help to identify expected direction of events that affect overall sentiment; (3) development of a linearized phrase-structure model for detecting contextual semantic orientations in financial and economic news texts. The relevance of the newly added lexicon features and the benefit of using the proposed learning-algorithm are demonstrated in a comparative study against previously used general sentiment models as well as the popular word frequency models used in recent financial studies. The proposed framework is parsimonious and avoids the explosion in feature-space caused by the use of conventional n-gram features.

Investigating Societal Biases in a Poetry Composition System

Abstract: There is a growing collection of work analyzing and mitigating societal biases in language understanding, generation, and retrieval tasks, though examining biases in creative tasks remains underexplored. Creative language applications are meant for direct interaction with users, so it is important to quantify and mitigate societal biases in these applications. We introduce a novel study on a pipeline to mitigate societal biases when retrieving next verse suggestions in a poetry composition system. Our results suggest that data augmentation through sentiment style transfer has potential for mitigating societal biases.

Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs

Abstract: People increasingly search online for answers to their medical questions but the rate at which medical questions are asked online significantly exceeds the capacity of qualified people to answer them. This leaves many questions unanswered or inadequately answered. Many of these questions are not unique, and reliable identification of similar questions would enable more efficient and effective question answering schema. COVID-19 has only exacerbated this problem. Almost every government agency and healthcare organization has tried to meet the informational need of users by building online FAQs, but there is no way for people to ask their question and know if it is answered on one of these pages. While many research efforts have focused on the problem of general question similarity, these approaches do not generalize well to domains that require expert knowledge to determine semantic similarity, such as the medical domain. In this paper, we show how a double fine-tuning approach of pretraining a neural network on medical question-answer pairs followed by fine-tuning on medical question-question pairs is a particularly useful intermediate task for the ultimate goal of determining medical question similarity. While other pretraining tasks yield an accuracy below 78.7% on this task, our model achieves an accuracy of 82.6% with the same number of training examples, an accuracy of 80.0% with a much smaller training set, and an accuracy of 84.5% when the full corpus of medical question-answer data is used. We also describe a currently live system that uses the trained model to match user questions to COVID-related FAQs.

Hate Speech Dataset from a White Supremacy Forum

Abstract: Hate speech is commonly defined as any communication that disparages a target group of people based on some characteristic such as race, colour, ethnicity, gender, sexual orientation, nationality, religion, or other characteristic. Due to the massive rise of user-generated web content on social media, the amount of hate speech is also steadily increasing. Over the past years, interest in online hate speech detection and, particularly, the automation of this task has continuously grown, along with the societal impact of the phenomenon. This paper describes a hate speech dataset composed of thousands of sentences manually labelled as containing hate speech or not. The sentences have been extracted from Stormfront, a white supremacist forum. A custom annotation tool has been developed to carry out the manual labelling task which, among other things, allows the annotators to choose whether to read the context of a sentence before labelling it. The paper also provides a thoughtful qualitative and quantitative study of the resulting dataset and several baseline experiments with different classification models. The dataset is publicly available.

TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Abstract: The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.

Datasets used for the experiment.

We evaluate on 17 text classification datasets covering diverse tasks including sentiment analysis, paraphrase detection, natural language inference, hate speech detection and diverse domains including science, social media, finance, and more. All datasets are from Huggigface datasets {{cite:861e608}}. Full list and details about the datasets are provided in Table {{table:1b7896b3-c4f2-45e6-9d60-d7f0cbe99e2d}} .


hi, bn, es, de, fr, en-hi, en-bn, en-es, en-de, en-fr

Fluency-Guided Cross-Lingual Image Captioning

Abstract: Image captioning has so far been explored mostly in English, as most available datasets are in this language. However, the application of image captioning should not be restricted by language. Only few studies have been conducted for image captioning in a cross-lingual setting. Different from these works that manually build a dataset for a target language, we aim to learn a cross-lingual captioning model fully from machine-translated sentences. To conquer the lack of fluency in the translated sentences, we propose in this paper a fluency-guided learning framework. The framework comprises a module to automatically estimate the fluency of the sentences and another module to utilize the estimated fluency scores to effectively train an image captioning model for the target language. As experiments on two bilingual (English-Chinese) datasets show, our approach improves both fluency and relevance of the generated captions in Chinese, but without using any manually written sentences from the target language.

Multi30K: Multilingual English-German Image Descriptions

Abstract: We introduce the Multi30K dataset to stimulate multilingual multimodal research. Recent advances in image description have been demonstrated on English-language datasets almost exclusively, but image description should not be limited to English. This dataset extends the Flickr30K dataset with i) German translations created by professional translators over a subset of the English descriptions, and ii) descriptions crowdsourced independently of the original English descriptions. We outline how the data can be used for multilingual image description and multimodal machine translation, but we anticipate the data will be useful for a broader range of tasks.

COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval

Abstract: This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods. We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags. For effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content. Having 20 342 images annotated with 27 218 Chinese sentences and 70 993 tags, COCO-CN is currently the largest Chinese–English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning, and retrieval. We develop conceptually simple yet effective methods per task for learning from cross-lingual resources. Extensive experiments on the three tasks justify the viability of the proposed dataset and methods. Data and code are publicly available at https://github.com/li-xirong/coco-cn.

STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset

Abstract: In recent years, automatic generation of image descriptions (captions), that is, image captioning, has attracted a great deal of attention. In this paper, we particularly consider generating Japanese captions for images. Since most available caption datasets have been constructed for English language, there are few datasets for Japanese. To tackle this problem, we construct a large-scale Japanese image caption dataset based on images from MS-COCO, which is called STAIR Captions. STAIR Captions consists of 820,310 Japanese captions for 164,062 images. In the experiment, we show that a neural network trained using STAIR Captions can generate more natural and better Japanese captions, compared to those generated using English-Japanese machine translation after generating English captions.

Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

Abstract: Recently there has been a lot of interest in learning common representations for multiple views of data. Typically, such common representations are learned using a parallel corpus between the two views (say, 1M images and their English captions). In this work, we address a real-world scenario where no direct parallel data is available between two views of interest (say, $V_1$ and $V_2$) but parallel data is available between each of these views and a pivot view ($V_3$). We propose a model for learning a common representation for $V_1$, $V_2$ and $V_3$ using only the parallel data available between $V_1V_3$ and $V_2V_3$. The proposed model is generic and even works when there are $n$ views of interest and only one pivot view which acts as a bridge between them. There are two specific downstream applications that we focus on (i) transfer learning between languages $L_1$,$L_2$,...,$L_n$ using a pivot language $L$ and (ii) cross modal access between images and a language $L_1$ using a pivot language $L_2$. Our model achieves state-of-the-art performance in multilingual document classification on the publicly available multilingual TED corpus and promising results in multilingual multimodal retrieval on a new dataset created and released as a part of this work.

Multimodal Pivots for Image Caption Translation

Abstract: We present an approach to improve statistical machine translation of image descriptions by multimodal pivots defined in visual space. The key idea is to perform image retrieval over a database of images that are captioned in the target language, and use the captions of the most similar images for crosslingual reranking of translation outputs. Our approach does not depend on the availability of large amounts of in-domain parallel data, but only relies on available large datasets of monolingually captioned images, and on state-of-the-art convolutional neural networks to compute image similarities. Our experimental evaluation shows improvements of 1 BLEU point over strong baselines.

Cross-lingual Visual Pre-training for Multimodal Machine Translation

Abstract: Pre-trained language models have been shown to improve performance in many natural language tasks substantially. Although the early focus of such models was single language pre-training, recent advances have resulted in cross-lingual and visual pre-training methods. In this paper, we combine these two approaches to learn visually-grounded cross-lingual representations. Specifically, we extend the translation language modelling (Lample and Conneau, 2019) with masked region classification and perform pre-training with three-way parallel vision & language corpora. We show that when fine-tuned for multimodal machine translation, these models obtain state-of-the-art performance. We also provide qualitative insights into the usefulness of the learned grounded representations.

Globetrotter: Connecting Languages by Connecting Images

Abstract: Machine translation between many languages at once is highly challenging, since training with ground truth requires supervision between all language pairs, which is difficult to obtain. Our key insight is that, while languages may vary drastically, the underlying visual appearance of the world remains consistent. We introduce a method that uses visual observations to bridge the gap between languages, rather than relying on parallel corpora or topological properties of the representations. We train a model that aligns segments of text from different languages if and only if the images associated with them are similar and each image in turn is well-aligned with its textual description. We train our model from scratch on a new dataset of text in over fifty languages with accompanying images. Experiments show that our method outperforms previous work on unsupervised word and sentence translation using retrieval. Code, models and data are available on globetrotter.cs.columbia.edu.

MuST-Cinema: a Speech-to-Subtitles corpus

Abstract: Growing needs in localising audiovisual content in multiple languages through subtitles call for the development of automatic solutions for human subtitling. Neural Machine Translation (NMT) can contribute to the automatisation of subtitling, facilitating the work of human subtitlers and reducing turn-around times and related costs. NMT requires high-quality, large, task-specific training data. The existing subtitling corpora, however, are missing both alignments to the source language audio and important information about subtitle breaks. This poses a significant limitation for developing efficient automatic approaches for subtitling, since the length and form of a subtitle directly depends on the duration of the utterance. In this work, we present MuST-Cinema, a multilingual speech translation corpus built from TED subtitles. The corpus is comprised of (audio, transcription, translation) triplets. Subtitle breaks are preserved by inserting special symbols. We show that the corpus can be used to build models that efficiently segment sentences into subtitles and propose a method for annotating existing subtitling corpora with subtitle breaks, conforming to the constraint of length.

Video Highlight Prediction Using Audience Chat Reactions

Abstract: Sports channel video portals offer an exciting domain for research on multimodal, multilingual analysis. We present methods addressing the problem of automatic video highlight prediction based on joint visual features and textual analysis of the real-world audience discourse with complex slang, in both English and traditional Chinese. We present a novel dataset based on League of Legends championships recorded from North American and Taiwanese Twitch.tv channels (will be released for further research), and demonstrate strong results on these using multimodal, character-level CNN-RNN model architectures.

How2: A Large-scale Dataset for Multimodal Language Understanding

Abstract: In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.

VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

Abstract: We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, \vatex is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on \vatex: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the \vatex dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using \vatex for other video-and-language research.

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Abstract: We introduce a novel multimodal machine translation model that utilizes parallel visual and textual information. Our model jointly optimizes the learning of a shared visual-language embedding and a translator. The model leverages a visual attention grounding mechanism that links the visual semantics with the corresponding textual semantics. Our approach achieves competitive state-of-the-art results on the Multi30K and the Ambiguous COCO datasets. We also collected a new multilingual multimodal product description dataset to simulate a real-world international online shopping scenario. On this dataset, our visual attention grounding model outperforms other methods by a large margin.

Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question

Abstract: In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: this http URL

Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation

Abstract: Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.

xGQA: Cross-Lingual Visual Question Answering

Abstract: Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and—vice versa—multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling.

Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

Abstract: We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provide results for a model that learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for research on embodied language agents in simulated, photo-realistic environments.

Visually Grounded Reasoning across Languages and Cultures

Abstract: The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet. While one can hardly overestimate how much this benchmark contributed to progress in computer vision, it is mostly derived from lexical databases and image queries in English, resulting in source material with a North American or Western European bias. Therefore, we devise a new protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. In particular, we let the selection of both concepts and images be entirely driven by native speakers, rather than scraping them automatically. Specifically, we focus on a typologically diverse set of languages, namely, Indonesian, Mandarin Chinese, Swahili, Tamil, and Turkish. On top of the concepts and images obtained through this new protocol, we create a multilingual dataset for {M}ulticultur{a}l {R}easoning over {V}ision and {L}anguage (MaRVL) by eliciting statements from native speaker annotators about pairs of images. The task consists of discriminating whether each grounded statement is true or false. We establish a series of baselines using state-of-the-art models and find that their cross-lingual transfer performance lags dramatically behind supervised performance in English. These results invite us to reassess the robustness and accuracy of current state-of-the-art models beyond a narrow domain, but also open up new exciting challenges for the development of truly multilingual and multicultural systems.

VisualSem: a high-quality knowledge graph for vision and language

Abstract: An exciting frontier in natural language understanding (NLU) and generation (NLG) calls for (vision-and-) language models that can efficiently access external structured knowledge repositories. However, many existing knowledge bases only cover limited domains, or suffer from noisy data, and most of all are typically hard to integrate into neural language pipelines. To fill this gap, we release VisualSem: a high-quality knowledge graph (KG) which includes nodes with multilingual glosses, multiple illustrative images, and visually relevant relations. We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG. This multi-modal retrieval model can be integrated into any (neural network) model pipeline. We encourage the research community to use VisualSem for data augmentation and/or as a source of grounding, among other possible uses. VisualSem as well as the multi-modal retrieval models are publicly available and can be downloaded in this URL: https://github.com/iacercalixto/visualsem.

Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities

Abstract: Contextualizing language technologies beyond a single language kindled embracing multiple modalities and languages. Individually, each of these directions undoubtedly proliferated into several NLP tasks. Despite this momentum, most of the multimodal research is primarily centered around English and multilingual research is primarily centered around contexts from text modality. Challenging this conventional setup, researchers studied the unification of multilingual and multimodal (MultiX) streams. The main goal of this work is to catalogue and characterize these works by charting out the categories of tasks, datasets and methods to address MultiX scenarios. To this end, we review the languages studied, gold or silver data with parallel annotations, and understand how these modalities and languages interact in modeling. We present an account of the modeling approaches along with their strengths and weaknesses to better understand what scenarios they can be used reliably. Following this, we present the high-level trends in the overall paradigm of the field. Finally, we conclude by presenting a road map of challenges and promising research directions.

Datasets along with languages (ISO codes) spanned, gold/silver nature of annotations and parallism across languages.

This section presents various tasks and the corresponding datasets as shown in Table {{table:b0bb43d6-a666-4dcf-93dc-744066e25fb0}} .