Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions

Abstract: Due to the scarcity of dense pixel-level semantic annotations for images recorded in adverse visual conditions, there has been a keen interest in unsupervised domain adaptation (UDA) for the semantic segmentation of such images. UDA adapts models trained on normal conditions to the target adverse-condition domains. Meanwhile, multiple datasets with driving scenes provide corresponding images of the same scenes across multiple conditions, which can serve as a form of weak supervision for domain adaptation. We propose Refign, a generic extension to self-training-based UDA methods which leverages these cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. We design custom modules to streamline both steps and set the new state of the art for domain-adaptive semantic segmentation on several adverse-condition benchmarks, including ACDC and Dark Zurich. The approach introduces no extra training parameters, minimal computational overhead—during training only—and can be used as a drop-in extension to improve any given self-training-based UDA method. Code is available at https://github.com/brdav/refign.

Per-Pixel Classification is Not All You Need for Semantic Segmentation

Abstract: Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Abstract: Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (i.e., without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.

Semantic Segmentation in Art Paintings

Abstract: Semantic segmentation is a difficult task even when trained in a supervised manner on photographs. In this paper, we tackle the problem of semantic segmentation of artistic paintings, an even more challenging task because of a much larger diversity in colors, textures, and shapes and because there are no ground truth annotations available for segmentation. We propose an unsupervised method for semantic segmentation of paintings using domain adaptation. Our approach creates a training set of pseudo‐paintings in specific artistic styles by using style‐transfer on the PASCAL VOC 2012 dataset, and then applies domain confusion between PASCAL VOC 2012 and real paintings. These two steps build on a new dataset we gathered called DRAM (Diverse Realism in Art Movements) composed of figurative art paintings from four movements, which are highly diverse in pattern, color, and geometry. To segment new paintings, we present a composite multi‐domain adaptation method that trains on each sub‐domain separately and composes their solutions during inference time. Our method provides better segmentation results not only on the specific artistic movements of DRAM, but also on other, unseen ones. We compare our approach to alternative methods and show applications of semantic segmentation in art paintings. The code and models for our approach are publicly available at: https://github.com/Nadavc220/SemanticSegmentationInArtPaintings.

An Empirical Study of Remote Sensing Pretraining

Abstract: Deep learning has largely reshaped remote sensing (RS) research for aerial image understanding and made a great success. Nevertheless, most of the existing deep models are initialized with the ImageNet pretrained weights. Since natural images inevitably present a large domain gap relative to aerial images, probably limiting the finetuning performance on downstream aerial scene tasks. This issue motivates us to conduct an empirical study of remote sensing pretraining (RSP) on aerial images. To this end, we train different networks from scratch with the help of the largest RS scene recognition dataset up to now — MillionAID, to obtain a series of RS pretrained backbones, including both convolutional neural networks (CNN) and vision transformers such as Swin and ViTAE, which have shown promising performance on computer vision tasks. Then, we investigate the impact of RSP on representative downstream tasks including scene recognition, semantic segmentation, object detection, and change detection using these CNN and vision transformer backbones. Empirical study shows that RSP can help deliver distinctive performances in scene recognition tasks and in perceiving RS related semantics such as “Bridge” and “Airplane”. We also find that, although RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, it may still suffer from task discrepancies, where downstream tasks require different representations from scene recognition tasks. These findings call for further research efforts on both large-scale pretraining datasets and effective pretraining methods. The codes and pretrained models will be released at https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.

A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images

Abstract: The fully convolutional network (FCN) with an encoder-decoder architecture has been the standard paradigm for semantic segmentation. The encoder-decoder architecture utilizes an encoder to capture multilevel feature maps, which are incorporated into the final prediction by a decoder. As the context is crucial for precise segmentation, tremendous effort has been made to extract such information in an intelligent fashion, including employing dilated/atrous convolutions or inserting attention modules. However, these endeavors are all based on the FCN architecture with ResNet or other backbones, which cannot fully exploit the context from the theoretical concept. By contrast, we introduce the Swin Transformer as the backbone to extract the context information and design a novel decoder of densely connected feature aggregation module (DCFAM) to restore the resolution and produce the segmentation map. The experimental results on two remotely sensed semantic segmentation datasets demonstrate the effectiveness of the proposed scheme.

Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy

Abstract: Gastrointestinal (GI) pathologies are periodically screened, biopsied, and resected using surgical tools. Usually, the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development and amount and size of the resected area get lost.This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic we have released the "Kvasir-Instrument" dataset which consists of 590 annotated frames containing GI procedure tools such as snares, balloons and biopsy forceps, etc. Beside of the images, the dataset includes ground truth masks and bounding boxes and has been verified by two expert GI endoscopists. Additionally, we provide a baseline for the segmentation of the GI tools to promote research and algorithm development. We obtained a dice coefficient score of 0.9158 and a Jaccard index of 0.8578 using a classical U-Net architecture. A similar dice coefficient score was observed for DoubleUNet. The qualitative results showed that the model did not work for the images with specularity and the frames with multiple instruments, while the best result for both methods was observed on all other types of images. Both, qualitative and quantitative results show that the model performs reasonably good, but there is a large potential for further improvements. Benchmarking using the dataset provides an opportunity for researchers to contribute to the field of automatic endoscopic diagnostic and therapeutic tool segmentation for GI endoscopy.

Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding

Abstract: In clinical radiology reports, doctors capture important information about the patient's health status. They convey their observations from raw medical imaging data about the inner structures of a patient. As such, formulating reports requires medical experts to possess wide-ranging knowledge about anatomical regions with their normal, healthy appearance as well as the ability to recognize abnormalities. This explicit grasp on both the patient's anatomy and their appearance is missing in current medical image-processing systems as annotations are especially difficult to gather. This renders the models to be narrow experts e.g. for identifying specific diseases. In this work, we recover this missing link by adding human anatomy into the mix and enable the association of content in medical reports to their occurrence in associated imagery (medical phrase grounding). To exploit anatomical structures in this scenario, we present a sophisticated automatic pipeline to gather and integrate human bodily structures from computed tomography datasets, which we incorporate in our PAXRay: A Projected dataset for the segmentation of Anatomical structures in X-Ray data. Our evaluation shows that methods that take advantage of anatomical information benefit heavily in visually grounding radiologists' findings, as our anatomical segmentations allow for up to absolute 50% better grounding results on the OpenI dataset as compared to commonly used region proposals. The PAXRay dataset is available at https://constantinseibold.github.io/paxray/.

ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes

Abstract: Less than 35% of recyclable waste is being actually recycled in the US [2], which leads to increased soil and sea pollution and is one of the major concerns of environmental researchers as well as the common public. At the heart of the problem are the inefficiencies of the waste sorting process (separating paper, plastic, metal, glass, etc.) due to the extremely complex and cluttered nature of the waste stream. Recyclable waste detection poses a unique computer vision challenge as it requires detection of highly deformable and often translucent objects in cluttered scenes without the kind of context information usually present in human-centric datasets. This challenging computer vision task currently lacks suitable datasets or methods in the available literature. In this paper, we take a step towards computer-aided waste detection and present the first in-the-wild industrial-grade waste detection and segmentation dataset, ZeroWaste. We believe that ZeroWaste will catalyze research in object detection and semantic segmentation in extreme clutter as well as applications in the recycling domain. Our project page can be found at http://ai.bu.edu/zerowaste/

SpiderMesh: Spatial-aware Demand-guided Recursive Meshing for RGB-T Semantic Segmentation

Abstract: For semantic segmentation in urban scene understanding, RGB cameras alone often fail to capture a clear holistic topology in challenging lighting conditions. Thermal signal is an informative additional channel that can bring to light the contour and fine-grained texture of blurred regions in low-quality RGB image. Aiming at practical RGB-T (thermal) segmentation, we systematically propose a Spatial-aware Demand-guided Recursive Meshing (SpiderMesh) framework that: 1) proactively compensates inadequate contextual semantics in optically-impaired regions via a demand-guided target masking algorithm; 2) refines multimodal semantic features with recursive meshing to improve pixel-level semantic analysis performance. We further introduce an asymmetric data augmentation technique M-CutOut, and enable semi-supervised learning to fully utilize RGB-T labels only sparsely available in practical use. Extensive experiments on MFNet and PST900 datasets demonstrate that SpiderMesh achieves state-of-the-art performance on standard RGB-T segmentation benchmarks.

Location-Aware Self-Supervised Transformers

Abstract: Pixel-level labels are particularly expensive to acquire. Hence, pretraining is a critical step to improve models on a task like semantic segmentation. However, prominent algorithms for pretraining neural networks use image-level objectives, e.g. image classification, image-text alignment a la CLIP, or self-supervised contrastive learning. These objectives do not model spatial information, which might be sub-optimal when finetuning on downstream tasks with spatial reasoning. In this work, we pretrain network with a location-aware (LOCA) self-supervised method which fosters the emergence of strong dense features. Specifically, we use both a patch-level clustering scheme to mine dense pseudo-labels and a relative location prediction task to encourage learning about object parts and their spatial arrangements. Our experiments show that LOCA pretraining leads to representations that transfer competitively to challenging and diverse semantic segmentation datasets.

Supervised mIoU results for the datasets.

We looked up the current fully supervised performance to provide an upper threshold for each dataset and present them in Table {{table:2e91f8e3-cbc1-4e65-aab8-554d629b7da3}} .
We did not find any mIoU results for the MHP v1 dataset as it is originally annotated for instance segmentation. Therefore, we trained MaskFormer {{cite:ee2de69}} to provide a reference. We trained the model for 100K steps using the Swin-B ADE20K-150 settings and evaluated the best model based on the val mIoU.


A Bayesian graph convolutional network for reliable prediction of molecular properties with uncertainty quantification

Abstract: Deep neural networks have been increasingly used in various chemical fields. Here, we show that Bayesian inference enables more reliable prediction with quantitative uncertainty analysis.

Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning

Abstract: We report a statistically principled method to quantify the uncertainty of machine learning models for molecular properties prediction. We show that this uncertainty estimate can be used to judiciously design experiments.

Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

Abstract: Uncertainty quantification (UQ) is an important component of molecular property prediction, particularly for drug discovery applications where model predictions direct experimental design and where unanticipated imprecision wastes valuable time and resources. The need for UQ is especially acute for neural models, which are becoming increasingly standard yet are challenging to interpret. While several approaches to UQ have been proposed in the literature, there is no clear consensus on the comparative performance of these models. In this paper, we study this question in the context of regression tasks. We systematically evaluate several methods on five regression datasets using multiple complementary performance metrics. Our experiments show that none of the methods we tested is unequivocally superior to all others, and none produce a particularly reliable ranking of errors across multiple datasets. While we believe these results show that existing UQ methods are not sufficient for all common use cases and further research is needed, we conclude with a practical recommendation as to which existing techniques seem to perform well relative to others.

Uncertainty Estimation for Molecules: Desiderata and Methods

Abstract: Graph Neural Networks (GNNs) are promising surrogates for quantum mechanical calculations as they establish unprecedented low errors on collections of molecular dynamics (MD) trajectories. Thanks to their fast inference times they promise to accelerate computational chemistry applications. Unfortunately, despite low in-distribution (ID) errors, such GNNs might be horribly wrong for out-of-distribution (OOD) samples. Uncertainty estimation (UE) may aid in such situations by communicating the model's certainty about its prediction. Here, we take a closer look at the problem and identify six key desiderata for UE in molecular force fields, three 'physics-informed' and three 'application-focused' ones. To overview the field, we survey existing methods from the field of UE and analyze how they fit to the set desiderata. By our analysis, we conclude that none of the previous works satisfies all criteria. To fill this gap, we propose Localized Neural Kernel (LNK) a Gaussian Process (GP)-based extension to existing GNNs satisfying the desiderata. In our extensive experimental evaluation, we test four different UE with three different backbones and two datasets. In out-of-equilibrium detection, we find LNK yielding up to 2.5 and 2.1 times lower errors in terms of AUC-ROC score than dropout or evidential regression-based methods while maintaining high predictive performance.

Managing uncertainty in data-derived densities to accelerate density functional theory

Abstract: Faithful representations of atomic environments and general models for regression can be harnessed to learn electron densities that are close to the ground state. One of the applications of data-derived electron densities is orbital-free density functional theory (DFT). However, extrapolations of densities learned from a training set to dissimilar structures could result in inaccurate results, which would limit the applicability of the method. Here, we show that a non-Bayesian approach can produce estimates of uncertainty which can successfully distinguish accurate from inaccurate predictions of electron density. We apply our approach to DFT where we initialise calculations with data-derived densities only when we are confident about their quality. This results in a guaranteed acceleration to self-consistency for configurations that are similar to those seen during training and could be useful for sampling-based methods, where previous ground state densities cannot be used to initialise subsequent calculations.

Learning the electronic density of states in condensed matter

Abstract: The electronic density of states (DOS) quantifies the distribution of the energy levels that can be occupied by electrons in a quasiparticle picture, and is central to modern electronic structure theory. It also underpins the computation and interpretation of experimentally observable material properties such as optical absorption and electrical conductivity. We discuss the challenges inherent in the construction of a machine-learning (ML) framework aimed at predicting the DOS as a combination of local contributions that depend in turn on the geometric configuration of neighbours around each atom, using quasiparticle energy levels from density functional theory as training data. We present a challenging case study that includes configurations of silicon spanning a broad set of thermodynamic conditions, ranging from bulk structures to clusters, and from semiconducting to metallic behavior. We compare different approaches to represent the DOS, and the accuracy of predicting quantities such as the Fermi level, the DOS at the Fermi level, or the band energy, either directly or as a side-product of the evaluation of the DOS. The performance of the model depends crucially on the smoothening of the DOS, and there is a tradeoff to be made between the systematic error associated with the smoothening and the error in the ML model for a specific structure. We demonstrate the usefulness of this approach by computing the density of states of a large amorphous silicon sample, for which it would be prohibitively expensive to compute the DOS by direct electronic structure calculations, and show how the atom-centred decomposition of the DOS that is obtained through our model can be used to extract physical insights into the connections between structural and electronic features.

Existing research and benchmark studies on UQ for PDE surrogate solutions, DFT-related tasks, molecular property prediction, and compound-protein binding prediction.

Uncertainty Quantification in AI for Science

Compared to other machine learning applications in computer vision and natural language processing, the problem of training under data scarcity is even more severe for scientific AI as the experiments typically being expensive and time-consuming to be conducted to collect meaningful data.
Similar as any existing data-driven black-box model, the DNN-based scientific AI models can make erroneous but overconfident prediction for unseen input data {{cite:14e60fa}}, and are particularly vulnerable to adversarial attacks {{cite:7a5ddba}}. Therefore, there has been a growing interest in equipping those scientific AI models with the UQ capability. Many different benchmark studies and research efforts have been made to test the idea of using Gaussian process (GP) {{cite:881e59f}}, {{cite:becf856}}, {{cite:c327fd9}}, {{cite:cb9dada}}, {{cite:2fbdd14}}, {{cite:6edb686}}, {{cite:208b1a3}}, {{cite:222965e}}, {{cite:56bdbb1}}, {{cite:f94dbaa}}, deep ensemble {{cite:16cda2a}}, {{cite:c6fb44f}}, {{cite:c327fd9}}, {{cite:cb9dada}}, {{cite:b9d6bc2}}, {{cite:846fdfc}}, {{cite:881e59f}}, {{cite:f94dbaa}}, {{cite:222965e}}, MC dropout {{cite:42ff232}}, {{cite:54b66ff}}, {{cite:c6fb44f}}, {{cite:c327fd9}}, {{cite:cb9dada}}, {{cite:f94dbaa}}, {{cite:222965e}}, Bayesian neural network {{cite:54b66ff}}, {{cite:cb9dada}}, {{cite:881e59f}}, {{cite:222965e}} and EDL {{cite:021b275}}, {{cite:f94dbaa}} to quantify the uncertainty of molecular property prediction {{cite:42ff232}}, {{cite:54b66ff}}, {{cite:c6fb44f}}, {{cite:c327fd9}}, {{cite:cb9dada}}, {{cite:881e59f}}, {{cite:021b275}}, {{cite:2fbdd14}}, {{cite:b9d6bc2}}, {{cite:f94dbaa}}, {{cite:6edb686}}, {{cite:222965e}}, {{cite:208b1a3}}, compound-protein binding prediction {{cite:c327fd9}}, ground-state density prediction tasks {{cite:16cda2a}}, {{cite:becf856}}, as well as PDE surrogate prediction tasks with physics-informed neural network (PINN) {{cite:56bdbb1}}, along with seismic inversion {{cite:9d76c3f}}, {{cite:9bba06d}}. We summarize existing UQ research and benchmark studies applied to the above disciplines with the adopted UQ approaches in Table {{table:daecb5e2-60e3-4ee1-90e4-5357cec93c6f}} .


The Future of Ad Blocking: An Analytical Framework and New Techniques

Abstract: We present a systematic study of ad blocking - and the associated "arms race" - as a security problem. We model ad blocking as a state space with four states and six state transitions, which correspond to techniques that can be deployed by either publishers or ad blockers. We argue that this is a complete model of the system. We propose several new ad blocking techniques, including ones that borrow ideas from rootkits to prevent detection by anti-ad blocking scripts. Another technique uses the insight that ads must be recognizable by humans to comply with laws and industry self-regulation. We have built prototype implementations of three of these techniques, successfully blocking ads and evading detection. We systematically evaluate our proposed techniques, along with existing ones, in terms of security, practicality, and legality. We characterize the order of growth of the development effort required to create/maintain ad blockers as a function of the growth of the web. Based on our state-space model, our new techniques, and this systematization, we offer insights into the likely "end game" of the arms race. We challenge the widespread assumption that the arms race will escalate indefinitely, and instead identify a combination of evolving technical and legal factors that will determine the outcome.

Percival: Making In-Browser Perceptual Ad Blocking Practical With Deep Learning

Abstract: In this paper we present Percival, a browser-embedded, lightweight, deep learning-powered ad blocker. Percival embeds itself within the browser's image rendering pipeline, which makes it possible to intercept every image obtained during page execution and to perform blocking based on applying machine learning for image classification to flag potential ads. Our implementation inside both Chromium and Brave browsers shows only a minor rendering performance overhead of 4.55%, demonstrating the feasibility of deploying traditionally heavy models (i.e. deep neural networks) inside the critical path of the rendering engine of a browser. We show that our image-based ad blocker can replicate EasyList rules with an accuracy of 96.76%. To show the versatility of the Percival's approach we present case studies that demonstrate that Percival 1) does surprisingly well on ads in languages other than English; 2) Percival also performs well on blocking first-party Facebook ads, which have presented issues for other ad blockers. Percival proves that image-based perceptual ad blocking is an attractive complement to today's dominant approach of block lists

Surveyed Web Browser Extensions

Web privacy protection: browser extensions

Different techniques have been introduced to protect users from rowsing exhaust tracking.. Network monitor techniques include DNS filtering and network proxy, which are effective in specific scenarios but have apparent shortcomings against encrypted transmissions and individual URLs {{cite:f1ed349}}.
A popular type of protection comes from browser extensions that can reliably distinguish third-party content from a publisher's web pages and block unwanted content including encrypted web traffic.
In the following section, we describe several popular and featured browser extensions in detail. This information is summarised in Table {{table:101696a6-1345-4e35-bc30-655aa3f0dfcb}} .


International Conference on Software Engineering - Software Engineering in Practice track (ICSE-SEIP)

IEEE Transactions on Software Engineering

How do start-ups estimate technical debt? What are precedents of technical debt in start-ups? What outcomes linked to technical debt do start-ups report?

What patterns pertaining software engineering can be ascertained in start-up companies?

Exploration of Technical Debt in Start-ups

Abstract: Context: Software start-ups are young companies aiming to build and market software-intensive products fast with little resources. Aiming to accelerate time-to-market, start-ups often opt for ad-hoc engineering practices, make shortcuts in product engineering, and accumulate technical debt. Objective: In this paper we explore to what extent precedents, dimensions and outcomes associated with technical debt are prevalent in start-ups. Method: We apply a case survey method to identify aspects of technical debt and contextual information characterizing the engineering context in start-ups. Results: By analyzing responses from 86 start-up cases we found that start-ups accumulate most technical debt in the testing dimension, despite attempts to automate testing. Furthermore, we found that start-up team size and experience is a leading precedent for accumulating technical debt: larger teams face more challenges in keeping the debt under control. Conclusions: This study highlights the necessity to monitor levels of technical debt and to preemptively introduce practices to keep the debt under control. Adding more people to an already di cult to maintain product could amplify other precedents, such as resource shortages, communication issues and negatively a ect decisions pertaining to the use of good engineering practices.

A Progression Model of Software Engineering Goals, Challenges, and Practices in Start-Ups

Abstract: Context: Software start-ups are emerging as suppliers of innovation and software-intensive products. However, traditional software engineering practices are not evaluated in the context, nor adopted to goals and challenges of start-ups. As a result, there is insufficient support for software engineering in the start-up context. Objective: We aim to collect data related to engineering goals, challenges, and practices in start-up companies to ascertain trends and patterns characterizing engineering work in start-ups. Such data allows researchers to understand better how goals and challenges are related to practices. This understanding can then inform future studies aimed at designing solutions addressing those goals and challenges. Besides, these trends and patterns can be useful for practitioners to make more informed decisions in their engineering practice. Method: We use a case survey method to gather first-hand, in-depth experiences from a large sample of software start-ups. We use open coding and cross-case analysis to describe and identify patterns, and corroborate the findings with statistical analysis. Results: We analyze 84 start-up cases and identify 16 goals, 9 challenges, and 16 engineering practices that are common among start-ups. We have mapped these goals, challenges, and practices to start-up life-cycle stages (inception, stabilization, growth, and maturity). Thus, creating the progression model guiding software engineering efforts in start-ups. Conclusions: We conclude that start-ups to a large extent face the same challenges and use the same practices as established companies. However, the primary software engineering challenge in start-ups is to evolve multiple process areas at once, with a little margin for serious errors.

The studies that used the case survey methodology.

This section presents the 12 identified papers and how they answer our research question. Table {{table:f9b289e8-b4a1-471a-8587-d0ba2120c9e1}}  presents a summary of the identified studies, including the research questions they answered.


CIRCLe: Color Invariant Representation Learning for Unbiased Classification of Skin Lesions

Abstract: While deep learning based approaches have demonstrated expert-level performance in dermatological diagnosis tasks, they have also been shown to exhibit biases toward certain demographic attributes, particularly skin types (e.g., light versus dark), a fairness concern that must be addressed. We propose CIRCLe, a skin color invariant deep representation learning method for improving fairness in skin lesion classification. CIRCLe is trained to classify images by utilizing a regularization loss that encourages images with the same diagnosis but different skin types to have similar latent representations. Through extensive evaluation and ablation studies, we demonstrate CIRCLe's superior performance over the state-of-the-art when evaluated on 16k+ images spanning 6 Fitzpatrick skin types and 114 diseases, using classification accuracy, equal opportunity difference (for light versus dark groups), and normalized accuracy range, a new measure we propose to assess fairness on multiple skin type groups.

Test-time Fourier Style Calibration for Domain Generalization

Abstract: The topic of generalizing machine learning models learned on a collection of source domains to unknown target domains is challenging. While many domain generalization (DG) methods have achieved promising results, they primarily rely on the source domains at train-time without manipulating the target domains at test-time. Thus, it is still possible that those methods can overfit to source domains and perform poorly on target domains. Driven by the observation that domains are strongly related to styles, we argue that reducing the gap between source and target styles can boost models’ generalizability. To solve the dilemma of having no access to the target domain during training, we introduce Test-time Fourier Style Calibration (TF-Cal) for calibrating the target domain style on the fly during testing. To access styles, we utilize Fourier transformation to decompose features into amplitude (style) features and phase (semantic) features. Furthermore, we present an effective technique to Augment Amplitude Features (AAF) to complement TF-Cal. Extensive experiments on several popular DG benchmarks and a segmentation dataset for medical images demonstrate that our method outperforms state-of-the-art methods.

Feather-Light Fourier Domain Adaptation in Magnetic Resonance Imaging

Abstract: Generalizability of deep learning models may be severely affected by the difference in the distributions of the train (source domain) and the test (target domain) sets, e.g., when the sets are produced by different hardware. As a consequence of this domain shift, a certain model might perform well on data from one clinic, and then fail when deployed in another. We propose a very light and transparent approach to perform test-time domain adaptation. The idea is to substitute the target low-frequency Fourier space components that are deemed to reflect the style of an image. To maximize the performance, we implement the"optimal style donor"selection technique, and use a number of source data points for altering a single target scan appearance (Multi-Source Transferring). We study the effect of severity of domain shift on the performance of the method, and show that our training-free approach reaches the state-of-the-art level of complicated deep domain adaptation models. The code for our experiments is released.

Abstract: Purpose To develop and evaluate a deep learning-based method (MC-Net) to suppress motion artifacts in brain magnetic resonance imaging (MRI). Methods MC-Net was derived from a UNet combined with a two-stage multi-loss function. T1-weighted axial brain images contaminated with synthetic motions were used to train the network. Evaluation used simulated T1 and T2-weighted axial, coronal, and sagittal images unseen during training, as well as T1-weighted images with motion artifacts from real scans. Performance indices included the peak signal to noise ratio (PSNR), structural similarity index measure (SSIM), and visual reading scores. Two clinical readers scored the images. Results The MC-Net outperformed other methods implemented in terms of PSNR and SSIM on the T1 axial test set. The MC-Net significantly improved the quality of all T1-weighted images (for all directions and for simulated as well as real motion artifacts), both on quantitative measures and visual scores. However, the MC-Net performed poorly on images of untrained contrast (T2-weighted). Conclusion The proposed two-stage multi-loss MC-Net can effectively suppress motion artifacts in brain MRI without compromising image context. Given the efficiency of the MC-Net (single image processing time ~40ms), it can potentially be used in real clinical settings. To facilitate further research, the code and trained model are available at https://github.com/MRIMoCo/DL_Motion_Correction.

Coupled Dictionary Learning for Multi-Contrast MRI Reconstruction

Abstract: Medical imaging tasks often involve multiple contrasts, such as T1-and T2-weighted magnetic resonance imaging (MRI) data. These contrasts capture information associated with the same underlying anatomy and thus exhibit similarities. In this paper, we propose a Coupled Dictionary Learning based multi-contrast MRI reconstruction (CDLMRI) approach to leverage an available guidance contrast to restore the target contrast. Our approach consists of three stages: coupled dictionary learning, coupled sparse denoising, and k-space consistency enforcing. The first stage learns a group of dictionaries that capture correlations among multiple contrasts. By capitalizing on the learned adaptive dictionaries, the second stage performs joint sparse coding to denoise the corrupted target image with the aid of a guidance contrast. The third stage enforces consistency between the denoised image and the measurements in the k-space domain. Numerical experiments on the retrospective under-sampling of clinical MR images demonstrate that incorporating additional guidance contrast via our design improves MRI reconstruction, compared to state-of-the-art approaches.

Single-domain Generalization in Medical Image Segmentation via Test-time Adaptation from Shape Dictionary

Abstract: Domain generalization typically requires data from multiple source domains for model learning. However, such strong assumption may not always hold in practice, especially in medical field where the data sharing is highly concerned and sometimes prohibitive due to privacy issue. This paper studies the important yet challenging single domain generalization problem, in which a model is learned under the worst-case scenario with only one source domain to directly generalize to different unseen target domains. We present a novel approach to address this problem in medical image segmentation, which extracts and integrates the semantic shape prior information of segmentation that are invariant across domains and can be well-captured even from single domain data to facilitate segmentation under distribution shifts. Besides, a test-time adaptation strategy with dual-consistency regularization is further devised to promote dynamic incorporation of these shape priors under each unseen domain to improve model generalizability. Extensive experiments on two medical image segmentation tasks demonstrate the consistent improvements of our method across various unseen domains, as well as its superiority over state-of-the-art approaches in addressing domain generalization under the worst-case scenario.

TeSLA: Test-Time Self-Learning With Automatic Adversarial Augmentation

Abstract: Most recent test-time adaptation methods focus on only classification tasks, use specialized network architectures, destroy model calibration or rely on lightweight information from the source domain. To tackle these issues, this paper proposes a novel Test-time Self-Learning method with automatic Adversarial augmentation dubbed TeSLA for adapting a pre-trained source model to the unlabeled streaming test data. In contrast to conventional self-learning methods based on cross-entropy, we introduce a new test-time loss function through an implicitly tight connection with the mutual information and online knowledge distillation. Furthermore, we propose a learnable efficient adversarial augmentation module that further enhances online knowledge distillation by simulating high entropy augmented images. Our method achieves state-of-the-art classification and segmentation results on several benchmarks and types of domain shifts, particularly on challenging measurement shifts of medical images. TeSLA also benefits from several desirable properties compared to competing methods in terms of calibration, uncertainty metrics, insensitivity to model architectures, and source training strategies, all supported by extensive ablations. Our code and models are available at https://github.com/devavratTomar/TeSLA.

Abstract: Histopathology whole slide images (WSIs) can reveal significant inter-hospital variability such as illumination, color or optical artifacts. These variations, caused by the use of different scanning protocols across medical centers (staining, scanner), can strongly harm algorithms generalization on unseen protocols. This motivates development of new methods to limit such drop of performances. In this paper, to enhance robustness on unseen target protocols, we propose a new test-time data augmentation based on multi domain image-to-image translation. It allows to project images from unseen protocol into each source domain before classifying them and ensembling the predictions. This test-time augmentation method results in a significant boost of performances for domain generalization. To demonstrate its effectiveness, our method has been evaluated on 2 different histopathology tasks where it outperforms conventional domain generalization, standard H&E specific color augmentation/normalization and standard test-time augmentation techniques. Our code is publicly available at https://gitlab.com/vitadx/articles/test-time-i2i-translation-ensembling.

Learning Domain-Agnostic Visual Representation for Computational Pathology Using Medically-Irrelevant Style Transfer Augmentation

Abstract: Suboptimal generalization of machine learning models on unseen data is a key challenge which hampers the clinical applicability of such models to medical imaging. Although various methods such as domain adaptation and domain generalization have evolved to combat this challenge, learning robust and generalizable representations is core to medical image understanding, and continues to be a problem. Here, we propose STRAP (Style TRansfer Augmentation for histoPathology), a form of data augmentation based on random style transfer from non-medical style sources such as artistic paintings, for learning domain-agnostic visual representations in computational pathology. Style transfer replaces the low-level texture content of an image with the uninformative style of randomly selected style source image, while preserving the original high-level semantic content. This improves robustness to domain shift and can be used as a simple yet powerful tool for learning domain-agnostic representations. We demonstrate that STRAP leads to state-of-the-art performance, particularly in the presence of domain shifts, on two particular classification tasks in computational pathology. Our code is available at https://github.com/rikiyay/style-transfer-for-digital-pathology.

Adversarial Consistency for Single Domain Generalization in Medical Image Segmentation

Abstract: . An organ segmentation method that can generalize to unseen contrasts and scanner settings can significantly reduce the need for retraining of deep learning models. Domain Generalization (DG) aims to achieve this goal. However, most DG methods for segmentation require training data from multiple domains during training. We propose a novel adversarial domain generalization method for organ segmentation trained on data from a single domain. We synthesize the new domains via learning an adversarial domain synthesizer (ADS) and presume that the synthetic domains cover a large enough area of plausible distributions so that unseen domains can be interpolated from synthetic domains. We propose a mutual information regularizer to enforce the semantic consistency between images from the synthetic domains, which can be estimated by patch-level contrastive learning. We evaluate our method for various organ segmentation for unseen modalities, scanning protocols, and scanner sites.

Rethinking Data Augmentation for Single-source Domain Generalization in Medical Image Segmentation

Abstract: Single-source domain generalization (SDG) in medical image segmentation is a challenging yet essential task as domain shifts are quite common among clinical image datasets. Previous attempts most conduct global-only/random augmentation. Their augmented samples are usually insufficient in diversity and informativeness, thus failing to cover the possible target domain distribution. In this paper, we rethink the data augmentation strategy for SDG in medical image segmentation. Motivated by the class-level representation invariance and style mutability of medical images, we hypothesize that unseen target data can be sampled from a linear combination of C (the class number) random variables, where each variable follows a location-scale distribution at the class level. Accordingly, data augmented can be readily made by sampling the random variables through a general form. On the empirical front, we implement such strategy with constrained Bezier transformation on both global and local (i.e. class-level) regions, which can largely increase the augmentation diversity. A Saliency-balancing Fusion mechanism is further proposed to enrich the informativeness by engaging the gradient information, guiding augmentation with proper orientation and magnitude. As an important contribution, we prove theoretically that our proposed augmentation can lead to an upper bound of the generalization risk on the unseen target domain, thus confirming our hypothesis. Combining the two strategies, our Saliency-balancing Location-scale Augmentation (SLAug) exceeds the state-of-the-art works by a large margin in two challenging SDG tasks. Code is available at https://github.com/Kaiseem/SLAug.

Data-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:14c82f83-0a71-4767-9a73-ccf693e7b0ac}}), task, organ, and modality used in experiment.

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) No. 2022-0-00959 ((Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making) and No. 2019-0-00079 (Department of Artificial Intelligence (Korea University)).

C. Lian et al., “Hierarchical fully convolutional network for joint atrophy localization and alzheimer's disease diagnosis using structural MRI,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 4, pp. 880–893, 2020.
P. Mukherjee et al., “A shallow convolutional neural network predicts prognosis of lung cancer patients in multi-institutional computed tomography image datasets,” Nature Machine Intelligence, vol. 2, no. 5, pp. 274–282, 2020.
J. Duan et al., “Evaluating the clinical acceptability of deep learning contours of prostate and organs-at-risk in an automated prostate treatment planning process,” Medical Physics, vol. 49, no. 4, pp. 2570–2581, 2022.
Y. Nan et al., “Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions,” Information Fusion, 2022.
R. J. Chen et al., “Scaling vision transformers to gigapixel images via hierarchical self-supervised learning,” in Proceedings of the CVPR, 2022, pp. 16 144–16 155.
L. Feng et al., “5D whole-heart sparse MRI,” Magnetic Resonance in Medicine, vol. 79, no. 2, pp. 826–838, 2018.
D. Ghosh and D. Ekta Ghosh, “A large-scale multi-centre research on domain generalisation in deep learning-based mass detection in mammography: A review,” in Proceedings of the Acta Biology Forum, 2022, pp. 05–09.
A. Apicella et al., “Machine learning strategies to improve generalization in EEG-based emotion assessment: A systematic review,” arXiv preprint arXiv:2212.08744, 2022.
H. Guan and M. Liu, “Domain adaptation for medical image analysis: A survey,” IEEE Transactions on Biomedical Engineering, vol. 69, no. 3, pp. 1173–1185, 2021.
G. Sarafraz et al., “Domain adaptation and generalization on functional medical images: A systematic survey,” arXiv preprint arXiv:2212.03176, 2022.
S. A. Mali et al., “Making radiomics more reproducible across scanner and imaging protocol variations: A review of harmonization methods,” Journal of Personalized Medicine, vol. 11, no. 9, p. 842, 2021.
J. Wang et al., “Generalizing to unseen domains: A survey on domain generalization,” IEEE Transactions on Knowledge and Data Engineering, 2022.
K. Zhou et al., “Domain generalization: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Z. Shen et al., “Towards out-of-distribution generalization: A survey,” arXiv preprint arXiv:2108.13624, 2021.
G. Csurka et al., “Unsupervised domain adaptation for semantic image segmentation: A comprehensive survey,” arXiv preprint arXiv:2112.03241, 2021.
J. Yang et al., “Generalized out-of-distribution detection: A survey,” arXiv preprint arXiv:2110.11334, 2021.
P. Cui and J. Wang, “Out-of-distribution (OOD) detection based on deep learning: A review,” Electronics, vol. 11, no. 21, p. 3500, 2022.
N. Ghassemi and E. Fazl-Ersi, “A comprehensive review of trends, applications and challenges in out-of-distribution detection,” arXiv preprint arXiv:2209.12935, 2022.
P. Sheth et al., “Domain generalization–A causal perspective,” arXiv preprint arXiv:2209.15177, 2022.
H. Li et al., “Out-of-distribution generalization on graphs: A survey,” arXiv preprint arXiv:2202.07987, 2022.
Y. Li et al., “Federated domain generalization: A survey,” arXiv preprint arXiv:2306.01334, 2023.
E. Kodirov et al., “Unsupervised domain adaptation for zero-shot learning,” in Proceedings of the ICCV, 2015, pp. 2452–2460.
D. Adila and D. Kang, “Understanding out-of-distribution: A perspective of data dynamics,” in Proceedings of NeurIPS Workshop.   PMLR, 2022, pp. 1–8.
M. Shah et al., “Evaluating intensity normalization on MRIs of human brain with multiple sclerosis,” Medical Image Analysis, vol. 15, no. 2, pp. 267–282, 2011.
P. Chlap et al., “A review of medical image data augmentation techniques for deep learning applications,” Journal of Medical Imaging and Radiation Oncology, vol. 65, no. 5, pp. 545–563, 2021.
J. Panic et al., “Normalization strategies in multi-center radiomics abdominal MRI: Systematic review and meta-analyses,” IEEE Open Journal of Engineering in Medicine and Biology, 2023.
W. Yu et al., “SAN-Net: Learning generalization to unseen sites for stroke lesion segmentation with self-adaptive normalization,” Computers in Biology and Medicine, vol. 156, p. 106717, 2023.
O. Ronneberger et al., “U-Net: Convolutional networks for biomedical image segmentation,” in Proceedings of the MICCAI.   Springer, 2015, pp. 234–241.
N. Karani et al., “Test-time adaptable neural networks for robust medical image segmentation,” Medical Image Analysis, vol. 68, p. 101907, 2021.
J. Ma, “Histogram matching augmentation for domain adaptation with application to multi-centre, multi-vendor and multi-disease cardiac image segmentation,” in Proceedings of the MICCAI Workshop.   Springer, 2021, pp. 177–186.
L. Li et al., “AtrialGeneral: Domain generalization for left atrial segmentation of multi-center LGE MRIs,” in Proceedings of the MICCAI.   Springer, 2021, pp. 557–566.
H. Gunasinghe et al., “Domain generalisation for glaucoma detection in retinal images from unseen fundus cameras,” in Proceedings of the ACIIDS.   Springer, 2022, pp. 421–433.
R. T. Shinohara et al., “Statistical normalization techniques for magnetic resonance imaging,” NeuroImage: Clinical, vol. 6, pp. 9–19, 2014.
J. T. Pontalba et al., “Assessing the impact of color normalization in convolutional neural network-based nuclei segmentation frameworks,” Frontiers in Bioengineering and Biotechnology, vol. 7, p. 300, 2019.
S. Kondo, S. Kasai, and K. Hirasawa, “Tackling mitosis domain generalization in histopathology images with color normalization,” in MICCAI Challenge on Mitosis Domain Generalization.   Springer, 2022, pp. 217–220.
A. Vahadane et al., “Structure-preserving color normalization and sparse stain separation for histological images,” IEEE Transactions on Medical Imaging, vol. 35, no. 8, pp. 1962–1971, 2016.
J. Xiong et al., “Improve unseen domain generalization via enhanced local color transformation,” in Proceedings of the MICCAI.   Springer, 2020, pp. 433–443.
A. Pakzad, K. Abhishek, and G. Hamarneh, “CIRCLe: Color invariant representation learning for unbiased classification of skin lesions,” in Proceedings of the ECCV.   Springer, 2022, pp. 203–219.
Y. Choi et al., “StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation,” in Proceedings of the CVPR, 2018, pp. 8789–8797.
Q. Xu et al., “Fourier-based augmentation with applications to domain generalization,” Pattern Recognition, vol. 139, p. 109474, 2023.
Xu et al., “A fourier-based framework for domain generalization,” in Proceedings of the CVPR, 2021, pp. 14 383–14 392.
M. Clausel, T. Oberlin, and V. Perrier, “The monogenic synchrosqueezed wavelet transform: A tool for the decomposition/demodulation of AM–FM images,” Applied and Computational Harmonic Analysis, vol. 39, no. 3, pp. 450–486, 2015.
X. Zhao et al., “Test-time Fourier style calibration for domain generalization,” in Proceedings of the IJCAI, 2022, pp. 1721–1727.
H. Zhang et al., “Mixup: Beyond empirical risk minimization,” in ICLR, 2018.
H. Liu et al., “Domain generalization in restoration of cataract fundus images via high-frequency components,” in Proceedings of the ISBI.   IEEE, 2022, pp. 1–5.
D. Hu et al., “Domain generalization for retinal vessel segmentation with vector field transformer,” in Proceedings of the MIDL.   PMLR, 2022, pp. 552–564.
H. Lee et al., “Machine friendly machine learning: Interpretation of computed tomography without image reconstruction,” Scientific reports, vol. 9, no. 1, p. 15540, 2019.
L. Gallardo-Estrella et al., “Normalizing computed tomography data reconstructed with different filter kernels: Effect on emphysema quantification,” European Radiology, vol. 26, pp. 478–486, 2016.
I. Zakazov et al., “Feather-light fourier domain adaptation in magnetic resonance imaging,” in Proceedings of the MICCAI Workshop.   Springer, 2022, pp. 88–97.
L. Zhang et al., “Motion correction in MRI using deep learning and a novel hybrid loss function,” arXiv preprint arXiv:2210.14156, 2022.
R. Zhao, H. Li, and X. Liu, “A survey of dictionary learning in medical image analysis and its application for glaucoma diagnosis,” Archives of Computational Methods in Engineering, vol. 28, pp. 463–471, 2021.
M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736–3745, 2006.
P. Song et al., “Coupled dictionary learning for multi-contrast MRI reconstruction,” IEEE Transactions on Medical Imaging, vol. 39, no. 3, pp. 621–633, 2019.
Q. Liu et al., “Single-domain generalization in medical image segmentation via test-time adaptation from shape dictionary,” in Proceedings of the AAAI, vol. 36, no. 2, 2022, pp. 1756–1764.
C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, no. 1, pp. 1–48, 2019.
L. Li et al., “Random style transfer based domain generalization networks integrating shape and spatial information,” in Proceedings of the MICCAI Workshop.   Springer, 2021, pp. 208–218.
D. Tomar et al., “TeSLA: Test-time self-learning with automatic adversarial augmentation,” in Proceedings of the CVPR, 2023, pp. 20 341–20 350.
M. Scalbert et al., “Test-time image-to-image translation ensembling improves out-of-distribution generalization in histopathology,” in Proceedings of the MICCAI.   Springer, 2022, pp. 120–129.
Y. Choi et al., “StarGAN V2: Diverse image synthesis for multiple domains,” in Proceedings of the CVPR, 2020, pp. 8188–8197.
R. Yamashita et al., “Learning domain-agnostic visual representation for computational pathology using medically-irrelevant style transfer augmentation,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3945–3954, 2021.
Y. Xu et al., “Generative adversarial domain generalization via cross-task feature attention learning for prostate segmentation,” in Proceedings of the NeurIPS.   Springer, 2021, pp. 273–284.
A. Taleb et al., “Multimodal self-supervised learning for medical image analysis,” in Proceedings of the IPMI.   Springer, 2021, pp. 661–673.
Y. Xu et al., “Adversarial consistency for single domain generalization in medical image segmentation,” in Proceedings of the MICCAI.   Springer, 2022, pp. 671–681.
Z. Su et al., “Rethinking data augmentation for single-source domain generalization in medical image segmentation,” in Proceedings of the AAAI, vol. 37, no. 2, 2023, pp. 2366–2374.
C. Xu et al., “Improved domain generalization for cell detection in histopathology images via test-time stain augmentation,” in Proceedings of the MICCAI.   Springer, 2022, pp. 150–159.
J.-R. Chang et al., “Stain mix-up: Unsupervised domain generalization for histopathology images,” in Proceedings of the MICCAI.   Springer, 2021, pp. 117–126.
L. Huang et al., “Normalization techniques in training DNNs: Methodology, analysis and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Z. Zhou et al., “Generalizable cross-modality medical image segmentation via style augmentation and dual normalization,” in Proceedings of the CVPR, 2022, pp. 20 856–20 865.
Y.-P. Liu et al., “SS-Norm: Spectral-spatial normalization for single-domain generalization with application to retinal vessel segmentation,” IET Image Processing, 2023.
K. Stacke et al., “Measuring domain shift for deep learning in histopathology,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 2, pp. 325–336, 2020.
J. Lyu et al., “AADG: Automatic augmentation for domain generalization on retinal image segmentation,” IEEE Transactions on Medical Imaging, vol. 41, no. 12, pp. 3699–3711, 2022.
M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” in Proceedings of the NeurIPS, vol. 26, 2013.
H. Li et al., “Domain generalization for medical imaging classification with linear-dependency regularization,” in Proceedings of the NeurIPS, vol. 33, 2020, pp. 3118–3129.
Q. Meng et al., “Mutual information-based disentangled neural networks for classifying unseen categories in different domains: Application to fetal ultrasound imaging,” IEEE Transactions on Medical Imaging, vol. 40, no. 2, pp. 722–734, 2020.
M. I. Belghazi et al., “Mutual information neural estimation,” in Proceedings of the ICML.   PMLR, 2018, pp. 531–540.
Y. Bi et al., “MI-SegNet: Mutual information-based US segmentation for unseen domain generalization,” in Proceedings of the MICCAI.   Springer, 2023.
Y. Chen et al., “D-Bin: A generalized disentangling batch instance normalization for domain adaptation,” IEEE Transactions on Cybernetics, 2021.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
Z. Li et al., “Domain generalization for mammography detection via multi-style and multi-view contrastive learning,” in Proceedings of the MICCAI.   Springer, 2021, pp. 98–108.
R. Gu et al., “Contrastive domain disentanglement for generalizable medical image segmentation,” arXiv preprint arXiv:2205.06551, 2022.
J. S. Yoon, M.-C. Roh, and H.-I. Suk, “A plug-in method for representation factorization in connectionist models,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 8, pp. 3792–3803, 2022.
M. Ilse et al., “DIVA: Domain invariant variational autoencoders,” in Proceedings of the MIDL.   PMLR, 2020, pp. 322–348.
P. Bandi et al., “From detection of individual metastases to classification of lymph node status at the patient level: The camelyon17 challenge,” IEEE Transactions on Medical Imaging, 2018.
Y. Wang et al., “Variational disentanglement for domain generalization,” Transactions on Machine Learning Research, 2022.
C. Wang et al., “Domain invariant model with graph convolutional network for mammogram classification,” arXiv preprint arXiv:2204.09954, 2022.
Wang et al., “Learning domain-agnostic representation for disease diagnosis,” in Proceedings of the ICLR, 2023.
Liu et al., “Recursively conditional gaussian for ordinal unsupervised domain adaptation,” in Proceedings of the ICCV, 2021, pp. 764–773.
X. Liu et al., “Ordinal unsupervised domain adaptation with recursively conditional gaussian imposed variational disentanglement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
S. Wang et al., “DoFE: Domain-oriented feature embedding for generalizable fundus image segmentation on unseen datasets,” IEEE Transactions on Medical Imaging, vol. 39, no. 12, pp. 4237–4248, 2020.
N. Srivastava et al., “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
M. Islam and B. Glocker, “Frequency dropout: Feature-level regularization via randomized filtering,” in Proceedings of the ECCV.   Springer, 2022, pp. 281–295.
K. P. Nguyen, A. H. Treacher, and A. A. Montillo, “Adversarially-regularized mixed effects deep learning (ARMED) models improve interpretability, performance, and generalization on clustered (non-iid) data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Y. Wang et al., “Embracing the dark knowledge: Domain generalization using regularized knowledge distillation,” in Proceedings of the ACM MM, 2021, pp. 2595–2604.
C. Chen et al., “MaxStyle: Adversarial style composition for robust medical image segmentation,” in Proceedings of the MICCAI.   Springer, 2022, pp. 151–161.
T. Zhou and E. Konukoglu, “FedFA: Federated feature augmentation,” in Proceedings of the ICLR, 2023.
R. Wang, P. Chaudhari, and C. Davatzikos, “Embracing the disharmony in medical imaging: A simple and effective framework for domain adaptation,” Medical Image Analysis, vol. 76, p. 102309, 2022.
K. P. Ayodele et al., “Supervised domain generalization for integration of disparate scalp EEG datasets for automatic epileptic seizure detection,” Computers in Biology and Medicine, vol. 120, p. 103757, 2020.
R. A. Kamraoui et al., “DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation,” Medical Image Analysis, vol. 76, p. 102312, 2022.
M. Philipp et al., “Dynamic CNNs using uncertainty to overcome domain generalization for surgical instrument localization,” in Proceedings of the WACV, 2022, pp. 3612–3621.
Q. Dou et al., “Domain generalization via model-agnostic learning of semantic features,” in Proceedings of the NeurIPS, vol. 32, 2019.
Q. Liu, Q. Dou, and P.-A. Heng, “Shape-aware meta-learning for generalizing prostate MRI segmentation to unseen domains,” in Proceedings of the MICCAI.   Springer, 2020, pp. 475–485.
X. Liu et al., “Semi-supervised meta-learning with disentanglement for domain-generalised medical image segmentation,” in MICCAI.   Springer, 2021, pp. 307–317.
C. Ouyang et al., “Self-supervision with superpixels: Training few-shot medical image segmentation without annotation,” in Proceedings of the ECCV.   Springer, 2020, pp. 762–780.
S. Azizi et al., “Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging,” Nature Biomedical Engineering, pp. 1–24, 2023.
I. E. I. Bekkouch et al., “Adversarial reconstruction loss for domain generalization,” IEEE Access, vol. 9, pp. 42 424–42 437, 2021.
C. Chen et al., “Realistic adversarial data augmentation for MR image segmentation,” in Proceedings of the MICCAI.   Springer, 2020, pp. 667–677.
Z. Zhang et al., “Domain generalization with adversarial intensity attack for medical image segmentation,” arXiv preprint arXiv:2304.02720, 2023.
K. Goel et al., “Model patching: Closing the subgroup performance gap with data augmentation,” ICLR, 2021.
T. Hastie et al., The Elements of Statistical Learning: Data Mining, Inference, and Prediction.   Springer, 2009, vol. 2.
S. Sagawa et al., “Distributionally robust neural networks,” in Proceedings of the ICLR, 2019.
E. Delage and Y. Ye, “Distributionally robust optimization under moment uncertainty with application to data-driven problems,” Operations Research, vol. 58, no. 3, pp. 595–612, 2010.
A. Bissoto et al., “Artifact-based domain generalization of skin lesion models,” in Proceedings of the ECCV.   Springer, 2022, pp. 133–149.
M. M. Bronstein et al., “Geometric deep learning: Grids, groups, graphs, geodesics, and gauges,” arXiv preprint arXiv:2104.13478, 2021.
D. M. Nguyen et al., “LVM-Med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching,” arXiv preprint arXiv:2306.11925, 2023.
L. Seenivasan et al., “Biomimetic incremental domain generalization with a graph network for surgical scene understanding,” Biomimetics, vol. 7, no. 2, p. 68, 2022.
A. Santhirasekaram et al., “Topology preserving compositionality for robust medical image segmentation,” in Proceedings of the CVPR, 2023, pp. 543–552.
G. A. Kaissis et al., “Secure, privacy-preserving and federated machine learning in medical imaging,” Nature Machine Intelligence, vol. 2, no. 6, pp. 305–311, 2020.
Q. Xu et al., “Federated adversarial domain hallucination for privacy-preserving domain generalization,” IEEE Transactions on Multimedia, 2023.
M. Abadi et al., “Deep learning with differential privacy,” in Proceedings of the ACM CCS, 2016, pp. 308–318.
Y. Chen et al., “FedHealth: A federated transfer learning framework for wearable healthcare,” IEEE Intelligent Systems, vol. 35, no. 4, pp. 83–93, 2020.
Q. Liu et al., “FedDG: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space,” in Proceedings of the CVPR, 2021, pp. 1013–1023.
J. Chen et al., “Federated domain generalization for image recognition via cross-client style transfer,” in Proceedings of the WACV, 2023, pp. 361–370.
W. Li et al., “Privacy-preserving federated brain tumour segmentation,” in Proceedings of the MLMI.   Springer, 2019, pp. 133–141.
N. M. Müller et al., “Shortcut removal for improved OOD-generalization,” in Proceedings of the CVPR Workshop, 2023.
J. Yang, K. Zhou, and Z. Liu, “Full-spectrum out-of-distribution detection,” International Journal of Computer Vision, pp. 1–16, 2023.
D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in Proceedings of ICLR, 2017.
S. Ben-David et al., “A theory of learning from different domains,” Machine Learning, vol. 79, pp. 151–175, 2010.
D. Li et al., “Deeper, broader and artier domain generalization,” in Proceedings of the ICCV, 2017, pp. 5542–5550.
D. Mahajan, S. Tople, and A. Sharma, “The connection between out-of-distribution generalization and privacy of ML models,” ICLR, 2022.
A. Puli et al., “Out-of-distribution generalization in the presence of nuisance-induced spurious correlations,” ICLR, 2022.
Puli et al., “Nuisances via negativa: Adjusting for spurious correlations via data augmentation,” ICLR, 2023.
I. Gao et al., “Out-of-distribution robustness via targeted augmentations,” in Proceedings of the NeurIPS Workshop, 2022.
G. Baryannis, S. Dani, and G. Antoniou, “Predicting supply chain risks using machine learning: The trade-off between performance and interpretability,” Future Generation Computer Systems, vol. 101, pp. 993–1004, 2019.
A. Singh, S. Sengupta, and V. Lakshminarayanan, “Explainable deep learning models in medical image analysis,” Journal of Imaging, vol. 6, no. 6, p. 52, 2020.
B. H. Van der Velden et al., “Explainable artificial intelligence (XAI) in deep learning-based medical image analysis,” Medical Image Analysis, p. 102470, 2022.
D. Wang et al., “Proactive pseudo-intervention: Causally informed contrastive learning for interpretable vision models,” arXiv preprint arXiv:2012.03369, 2020.
M. R. Karim et al., “DeepKneeExplainer: Explainable knee osteoarthritis diagnosis from radiographs and magnetic resonance imaging,” IEEE Access, vol. 9, pp. 39 757–39 780, 2021.
Y. Wang et al., “SEEG-Net: An explainable and deep learning-based cross-subject pathological activity detection method for drug-resistant epilepsy,” Computers in Biology and Medicine, vol. 148, p. 105703, 2022.
A. Chattopadhay et al., “Grad-Cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in Proceedings of the WACV.   IEEE, 2018, pp. 839–847.
S. Gao et al., “BayeSeg: Bayesian modeling for medical image segmentation with interpretable generalizability,” arXiv preprint arXiv:2303.01710, 2023.
J. Yuan et al., “Not just pretty pictures: Text-to-image generators enable interpretable interventions for robust representations,” arXiv preprint arXiv:2212.11237, 2022.
D. Mahajan, S. Tople, and A. Sharma, “Domain generalization using causal matching,” in Proceedings of the ICML.   PMLR, 2021, pp. 7313–7324.
R. Wang, P. Chaudhari, and C. Davatzikos, “Harmonization with flow-based causal inference,” in Proceedings of the MICCAI.   Springer, 2021, pp. 181–190.
C. Ouyang et al., “Causality-inspired single-source domain generalization for medical image segmentation,” IEEE Transactions on Medical Imaging, vol. 42, no. 4, pp. 1095–1106, 2023.
V. M. Campello et al., “Multi-centre, multi-vendor and multi-disease cardiac segmentation: The M&Ms challenge,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3543–3554, 2021.
M. Aubreville et al., “Mitosis domain generalization in histopathology images—the midog challenge,” Medical Image Analysis, vol. 84, p. 102699, 2023.
A. F. Kazerooni et al., “The brain tumor segmentation (BraTS) challenge 2023: Focus on pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs),” arXiv preprint arXiv:2305.17033, 2023.
M. Adewole et al., “The brain tumor segmentation (BraTS) challenge 2023: Glioma segmentation in sub-saharan africa patient population (BraTS-Africa),” arXiv preprint arXiv:2305.19369, 2023.
H. Bogunović et al., “RETOUCH: The retinal OCT fluid detection and segmentation benchmark and challenge,” IEEE Transactions on Medical Imaging, vol. 38, no. 8, pp. 1858–1874, 2019.
Q. Meng, D. Rueckert, and B. Kainz, “Learning cross-domain generalizable features by representation disentanglement,” arXiv preprint arXiv:2003.00321, 2020.
R. Gu et al., “CDDSA: Contrastive domain disentanglement and style augmentation for generalizable medical image segmentation,” Medical Image Analysis, vol. 89, p. 102904, 2023.
J. Lee et al., “Site-invariant meta-modulation learning for multisite autism spectrum disorders diagnosis,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14, 2023.
W. Fan et al., “InvNorm: Domain generalization for object detection in gastrointestinal endoscopy,” arXiv preprint arXiv:2205.02842, 2022.
S. Korevaar, R. Tennakoon, and A. Bab-Hadiashar, “Failure to achieve domain invariance with domain generalization algorithms: An analysis in medical imaging,” IEEE Access, 2023.
H. Zhang et al., “An empirical framework for domain generalization in clinical settings,” in Proceedings of the CHIL, 2021, pp. 279–290.
[{{figure:e578bf48-105e-4664-85b5-5d5f1e4f24e4}}
His research interests include explainable AI, computer vision, and representation learning.
[{{figure:981d1c18-7472-417b-9694-3684586c369b}}
His current research interests include explainable AI, computer vision, and machine/deep learning.
[{{figure:749f8433-b6b3-4bf4-b00f-f0d8cb0dd633}}
His research interests include transfer learning, computer vision, and machine/deep learning.
[{{figure:2ea7eb07-9eec-4b4e-a317-829a7c03f8e3}}
[{{figure:6f775985-6d03-4e7b-aafc-c58c96584e07}}
He was awarded a Kakao Faculty Fellowship from Kakao and a Young Researcher Award from the Korean Society for Human Brain Mapping (KHBM) in 2018 and 2019, respectively. His research interests include causal machine/deep learning, explainable AI, biomedical data analysis, and brain-computer interface.
Dr. Suk serves as an Editorial Board Member for Clinical and Molecular Hepatology (Artificial Intelligence Sector), Electronics, Frontiers in Neuroscience, Frontiers in Radiology (Artificial Intelligence in Radiology), International Journal of Imaging Systems and Technology (IJIST), and a Program Committee or a Reviewer for NeurIPS, ICML, ICLR, AAAI, IJCAI, CVPR, MICCAI, AISTATS, etc.

Supplementary Material
Scope of Survey
Number of publications per year on the Google Scholar database.
[
width=0.5height=4cm,
xticklabel style=rotate=330, /pgf/number format/set thousands separator=,
xtick=2013,...,2022,
xtick distance=1,
axis lines=left,
tick label style=font=
]table
Year Publications
2013 13
2014 19
2015 19
2016 23
2017 26
2018 40
2019 73
2020 212
2021 507
2022 990
;

text width = 4cm, align = left] at (2.5,1.4)Keyword: ("domain generalization" OR "domain generalisation") AND (intitle:medical OR ct OR ultrasound OR mri OR pet OR xray OR histology OR histopathology OR pathology OR fundus OR dermoscopy OR endoscopy OR mammography);{{figure:bd2d2ebd-76cf-4735-8c80-7a64d516eabd}}Literature search and selection were conducted by researchers experienced in machine learning and medical image analysis.
We used the Google Scholar search engine with three different search strategies, resulting in a database of 1,621 papers.
First, we searched for all papers that cited the existing domain generalization survey {{cite:3b55acf}}, {{cite:121c83b}}, {{cite:f71156c}}, {{cite:2b4f1ca}}, {{cite:848a605}}, {{cite:b4119d1}}, {{cite:6913ba9}}.
Second, we searched Google Scholar using the exact phrase “domain generaliz(s)ation” and medical-related keywords (“Medical, CT, ultrasound, MRI, PET, X-ray, histology, histopathology, pathology, fundus, dermoscopy, endoscopy, mammography”) and selected 1,000 papers sorted by relevancy.
Lastly, we searched for the top 1,000 papers with the terms “unseen”, “domain”, and medical-related keywords.
The eligibility criteria for papers to be included in this survey are that they have conducted at least one experiment involving the use of medical images within the domain generalization problem settings (see {{formula:9efdd061-96ca-4657-aa7d-e8ec1dc08382}}REF ), regardless of their use of the term “domain generalization” in their paper (small number of papers instead use “unseen”, “out-of-distribution”, or their own terms).
Peer-reviewed published papers were prioritized, but non-peer-reviewed archive papers (e.g., arXiv, bioRxiv) were also included if they had been deemed particularly suitable for selection (e.g., highly relevant, highly significant, highly cited).

Summary of Domain Generalization Methods
Data-level Domain Generalization Methods
Data-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:14c82f83-0a71-4767-9a73-ccf693e7b0ac}}REF ), task, organ, and modality used in experiment.

Method
Strategy
Ref.
Source
Target
Task
Organ
Modality
82.4cmImage Processing
31.5cmIntensity Normalization
{{cite:9f4531a}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:472bf2f}}
Single
Site, Sequence
Segmentation
Brain, Prostate, Heart
MRI
black(lr)2-8black
41.5cmHistogram Matching
{{cite:1491ffc}}
Multiple
Site
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:ea82163}}
Multiple
Site
Segmentation
Atrial
MRI
lightgray!20(lr)3-8black

{{cite:0b8e909}}
Multiple
Site
Detection
Retinal
Fundus
black(lr)2-8black
41.5cmColor Normalization
{{cite:c0b28e0}}
Multiple
Site
Detection
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:e1d0d60}}
Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:a2893d2}}
Multiple
Site
Classification
Skin
Dermatology
92.4cmSurrogate
71.5cmFrequency-based DG
{{cite:490bd53}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:cda2c67}}
Single, Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:7a6e44d}}
Multiple
Site
Restoration
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:2391f80}}
Multiple
Site
Segmentation
Vessel
2D Image
black(lr)2-8black
2.51.5cmUsing Raw Signals
{{cite:0a258ac}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:ad4ef31}}
Single
Sequence
Segmentation
Brain
MRI
black(lr)2-8black
2.51.5cmDictionary Learning
{{cite:5ad8d1d}}
Multiple
Sequence
Reconstruction
Brain
MRI
lightgray!20(lr)3-8black

{{cite:d8c3a02}}
Single
Site
Segmentation
Prostate, Retinal
MRI, Fundus
52.4cmAugmentation
2.51.5cmRandomization-based
{{cite:1c6fbcb}}
Single
Sequence
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:7a6e44d}}
Multiple
Site
Restoration
Retinal
Fundus
black(lr)2-8black
Adversarial-based
{{cite:af60507}}
Multiple
Site
Segmentation
Spinal, Prostate
MRI
black(lr)2-8black
2.51.5cmGenerative Model
{{cite:3973f40}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:b6239e8}}
Single
Site
Classification
Tissue
Histology
72.4cmProblem-specific
6.51.5cmCross-modal Generative Model
{{cite:6dca1c4}}
Multiple
Modality, Sequence
Segmentation
Brain, Prostate, Liver
MRI, CT
lightgray!20(lr)3-8black

{{cite:134ad1c}}
Single
Modality, Sequence
Segmantation
Prostate, Abdominal, Cardiac
MRI, CT
lightgray!20(lr)3-8black

{{cite:4e36271}}
Single
Modality
Segmentation
Cardiac, Abdominal
MRI
black(lr)2-8black
3.51.5cmStain Normalization
{{cite:09f207e}}
Multiple
Site
Detection
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:ad3e51b}}
Multiple
Site
Classification, Segmentation
Tissue
Histology, hematology
{{table:e6765bcd-936e-44ae-b8e2-c25cb5857b7d}}{{table:d870e699-d844-4aba-b056-4fd4eb2809fc}}
Feature-level Domain Generalization Methods
Feature-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:fbe950f5-9df0-4553-a723-f946cb49047c}}REF ), task, organ, and modality used in experiment.

Method
Specific
Ref.
Source
Target
Task
Organ
Modality
5*Feature Alignment
2.51.5cmFeature Normalization
{{cite:028b0f2}}
Single
Sequence, Modality
Segmentation
Brain, Cardiac, Abdominal
MRI, CT
lightgray!20(lr)3-8black

{{cite:897b234}}
Single
Site
Segmentation
Retinal
Fundus
black(lr)2-8black
5.31.5cmDissimilarity-based
{{cite:93a257f}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:7c02e53}}
Multiple
Modality
Segmentation
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:0ed736d}}
Multiple
Sequence, Site
Classification, Segmentation
Skin, Brain
Dermatology, MRI
20*Implicit Disentanglement
141.5cmMutual Information
{{cite:b64a4a0}}
Multiple
Site
Classification
Abdominal, Brain, Femur, Lips
Fetal Ultrasound
lightgray!20(lr)3-8black

{{cite:e5ac965}}
Multiple
Site
Classification
Abdominal, Brain, Femur, Lips
Fetal Ultrasound
lightgray!20(lr)3-8black

{{cite:80fbeda}}
Single
Site
Segmentation
Carotid
Ultrasound
lightgray!20(lr)3-8black

{{cite:fec8a3a}}
Single, Multiple
Sequence
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:134ad1c}}
Single
Modality, Sequence
Segmantation
Prostate, Abdominal, Cardiac
MRI, CT
black(lr)2-8black
2.51.5cmContrastive
{{cite:a43d7a4}}
Multiple
Site
Detection
Breast
X-ray
lightgray!20(lr)3-8black

{{cite:f43e4b0}}
Multiple
Site
Segmentation
Retinal
Fundus
black(lr)2-8black
61.5cmVariational
{{cite:9af3409}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:b2d3294}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:7b9d14c}}
Multiple
Site
Classification
Breast
X-ray
lightgray!20(lr)3-8black

{{cite:a47ac77}}
Multiple
Site
Classification
Breast
X-ray
6.5*Explicit Disentanglement
4.31.5cmConditional Representation Learning
{{cite:cc4f265}}
Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:cce9b83}}
Multiple
Site
Classification
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:5d7f30c}}
Multiple
Site
Segmentation
Retinal
Fundus
black(lr)2-8black
5.21.5cmFeature Reguarlization
{{cite:99cb615}}
Multiple
Site
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:db2ff2e}}
Multiple
Site
Compression, classification
Brain, Skin
MRI, Dermatology
lightgray!20(lr)3-8black

{{cite:4fc3027}}
Single
Site
Segmentation
Spinal
MRI
6*Others
2.51.5cmFeature Augmentation
{{cite:554c934}}
Single, Multiple
Site
Segmentation
Cardiac, Prostate
MRI
lightgray!20(lr)3-8black

{{cite:3f89dbb}}
Multiple
Site
Segmentation
Prostate
MRI
black(lr)2-8black
2.61.5cmKernel-based
{{cite:f124298}}
Single
Site
Classification
Brain
MRI
lightgray!20(lr)3-8black

{{cite:a6e1be3}}
Multiple
Site
Detection
Brain
EEG
{{table:76f4f39a-9bdd-4a6f-a51e-e022131a81cb}}{{table:2247a600-f84f-411e-bea0-78ca9ebc2906}}
Model-level Domain Generalization Methods
Model-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:59204db7-b73c-4d01-a33c-a345761589ac}}REF ), task, organ, and modality used in experiment.

Method
Specific
Ref.
Source
Target
Task
Organ
Modality
21*Learning Strategy
61.5cmEnsemble Learning
{{cite:2655543}}
Multiple
Sequence, Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:c27d1f0}}
Single
Sequence
Localization
Surgical Scene
Video Frames
lightgray!20(lr)3-8black

{{cite:3973f40}}
Single, Multiple
Site
Classification
Tissue
Histology
black(lr)2-8black
6.51.5cmMeta-learning
{{cite:303e624}}
Multiple
Site
Segmentation
Brain
MRI
lightgray!20(lr)3-8black

{{cite:9a25ddc}}
Multiple
Site
Segmentation
Prostate
MRI
lightgray!20(lr)3-8black

{{cite:b8ec0c1}}
Multiple
Site, Sequence
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:a725c5c}}
Single, Multiple
Site
Classification
Brain
Functional MRI
black(lr)2-8black
71.5cmSelf-supervised Learning
{{cite:f43e4b0}}
Multiple
Site
Segmentation
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:eb215d6}}
Multiple
Modality
Segmentation
Abdominal, Cardiac
CT, MRI
lightgray!20(lr)3-8black

{{cite:a128b07}}
Multiple
Site
Classification
Skin, Retinal, Chest, Breast
Fundus, Dermatology, X-ray, Mammography
black(lr)2-8black
41.5cmAdversarial Learning
{{cite:5885eb2}}
Single, Multiple
Sequence
Detection
Skin, Hip
MRI
lightgray!20(lr)3-8black

{{cite:f0bfbd8}}
Multiple
Site
Segmentation
Cardiac
MRI
lightgray!20(lr)3-8black

{{cite:afbd9f4}}
Multiple
Site
Segmentation
Retinal, Prostate
Fundus, MRI
2.5*Distributionally Robust Optimization

{{cite:d5c5153}}
Multiple
Site
Classification
Skin
Dermatology
lightgray!20(lr)3-8black

{{cite:2ffea0f}}
Multiple
Site
Classification
Skin
Dermatology
17.4*Other
5.51.5cmGeometric Learning
{{cite:22ffac2}}
Multiple
Modal, Sequence, Site
Classification, segmentation, detection
Multiple organs, modalities from 55 datasets
lightgray!20(lr)3-8black

{{cite:58fba09}}
Multiple
Site
Segmentation
Surgical Scene
Video Frames
lightgray!20(lr)3-8black

{{cite:778a6fe}}
Single
Site
Segmentation
Abdomen, Cardiac, Prostate
MRI
black(lr)2-8black
5.51.5cmDistributed Learning
{{cite:0e442fe}}
Multiple
Site
Segmentation
Retinal, Prostate
Fundus, MRI
lightgray!20(lr)3-8black

{{cite:f05d11b}}
Multiple
Site
Classification
Tissue
Histology
lightgray!20(lr)3-8black

{{cite:132a815}}
Multiple
Site
Segmentation
Retinal
Fundus
lightgray!20(lr)3-8black

{{cite:0002072}}
Multiple
Modality
Segmentation
Tumor
MRI
black(lr)2-8black
9.21.5cmOOD
{{cite:803c1f2}}
Multiple
Site
Detection
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:96e0a37}}
Single
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:f575f9c}}
Multiple
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:eb09003}}
Single
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:af0a2a7}}
Single
Site
Classification
Chest
X-ray
lightgray!20(lr)3-8black

{{cite:30133ba}}
Multiple
Site
Classification
Tissue
Histology
{{table:e1b324ae-f5cb-4845-b836-14b9b7f2c56d}}{{table:9e359065-8c8a-46ed-9fe4-23eca628b680}}
Analysis-level Domain Generalization Methods
Analysis-level domain generalization methods. Methods categorized by different settings for source and target domains (see {{formula:8d13df2a-a30c-4693-86a6-d01fffe6f32b}}REF ), task, organ, and modality used in experiment.

Method
Ref.
Source
Target
Task
Organ
Modality
72.2cmInterpretable AI
{{cite:3986ccf}}
Single
Site
Classification, Segmentation
Retinal, Cardiac
OCT, CT
lightgray!20(lr)3-7black
{{cite:d526c75}}
Single
Site
Classification, Segmentation
Knee
MRI
lightgray!20(lr)3-7black
{{cite:e1974c1}}
Multiple
Site
Classification
Brain
sEEG
lightgray!20(lr)3-7black
{{cite:715a58c}}
Multiple
Sequence
Detection
Gastrointestinal
Endoscopy
3.51.5cmTransferability
{{cite:9ab1ada}}
Single
Sequence, Site
Segmentation
Cardiac, Prostate
MRI, CT
lightgray!20(lr)3-7black
{{cite:d2e67c0}}
Single, Multiple
Site
Classification
Tissue
Histology
71.5cmCausality
{{cite:cec76a3}}
Single, Multiple
Sequence
Classification
Chest
X-ray
lightgray!20(lr)3-7black
{{cite:db6aae6}}
Multiple
Site
Regression, Classification
Brain
MRI
lightgray!20(lr)3-7black
{{cite:d4706ff}}
Single
Modality, Sequence, Site
Segmentation
Prostate, Abdominal, Cardiac
MRI, CT
{{table:b11e2a4d-f86b-4fa6-860e-07a84b56b971}}{{table:187b8858-c6d5-4149-b88a-50758f3d3d6d}}
Suggestions for Domain Generalization for Medical Image Analysis
Problem-specific suggestion for strategies for integrating domain generalization into MedIA workflow. Diamond box indicate the start terminator, angled boxes indicate the process, and round boxes indicate the decision.
rectangleround = [rectangle, text width=1.77cm, rounded corners, minimum width=1.23cm, minimum height=1.33cm, text centered, font=, color=gray, draw=black, line width=0.1, fill=teal!10]
rectangleangled = [rectangle, text width=1.77cm, minimum width=1.23cm, minimum height=1.33cm, text centered, font=, color=gray, draw=black, line width=0.1, fill=teal!20]
diamondbox = [diamond, text width=0.75cm, minimum width=0.75cm, minimum height=0.1cm, text centered, font=, color=gray, draw=black, line width=0.1, fill=teal!30][node distance=3cm]
frequency) [rectangleround] {{formula:0daac278-6b73-4081-be83-17e87b68dcec}} REFFrequency-based;
input) [diamondbox, right of = frequency] Start;
related) [rectangleround , right of=input] {{formula:f72debc2-bc75-4260-a4fa-99854fb7efc4}} REFRelated ML tasks;sufficient) [rectangleangled , below of=frequency, yshift=1cm] Sufficient data?;
raw) [rectangleangled , right of=sufficient] Raw measurement available?;
access) [rectangleangled , right of=raw] Access to {{formula:935ee198-945e-42ce-a0bc-31f77e1db2b6}} ?;augmentation) [rectangleround , below of=sufficient, yshift=1cm] {{formula:ffa1eeec-7960-4ee0-b56a-1ba55c89d1fe}} REFDataAugmentation;
preprocessing) [rectangleangled , right of=augmentation] Preprocessing required?;
specifictask) [rectangleangled , right of=preprocessing] Specific task or problem at hand?;analysis) [rectangleangled , below of=augmentation, yshift=1cm] Analysis or interpretability required?;
problem) [rectangleround , right of=analysis] {{formula:22413421-8b76-4426-866d-e6b1728eee42}} REFProblem-specific;
imageprocessing) [rectangleround , right of=problem] {{formula:7d55863c-c3ba-4cc2-ace0-d273f1f0df3d}} REFImage Processing;analysislevel) [rectangleround , below of=analysis, yshift=1cm] {{formula:689da5f8-865b-4ea8-8995-2dd6e64c8a96}} REFAnalysis-level DG;
specialalgorithm) [rectangleangled , right of=analysislevel] Able to use special algorithm or architecture?;
modellevel) [rectangleround , right of=specialalgorithm] {{formula:d5e4f1d2-bfd8-4a4a-a5ea-336ee42143e9}} REFModel-level DG;alignment) [rectangleround , below of=analysislevel, yshift=1cm] {{formula:5b58b8bf-443f-465c-a04b-8eff7525b569}} REFFeature Alignment;
invariance) [rectangleangled , right of=alignment] Invariant v.s. Disentangled Features;
disentanglement) [rectangleround , right of=invariance] {{formula:39f783fa-f2df-47e7-8cd8-a3d6c39330d1}} REFFeature Disentanglement;[-Stealth] (input) – (access);
[-Stealth] (access) – node[anchor=west] Y (related);
[-Stealth] (access) – node[anchor=north] N (raw);
[-Stealth] (raw) – node[anchor=north] Y (frequency);
[-Stealth] (raw) – node[anchor=north] N (sufficient);
[-Stealth] (sufficient) – node[anchor=north] Y (preprocessing);
[-Stealth] (sufficient) – node[anchor=west] N (augmentation);
[-Stealth] (preprocessing) – node[anchor=north] Y (specifictask);
[-Stealth] (specifictask) – node[anchor=north] Y (problem);
[-Stealth] (specifictask) – node[anchor=west] N (imageprocessing);[-Stealth] (preprocessing) – node[anchor=north] N (analysis);
[-Stealth] (analysis) – node[anchor=west] Y (analysislevel);
[-Stealth] (analysis) – node[anchor=north] N (specialalgorithm);
[-Stealth] (specialalgorithm) – node[anchor=north] Y (modellevel);
[-Stealth] (specialalgorithm) – node[anchor=west] N (invariance);
[-Stealth] (invariance) – node[anchor=north] Inv. (alignment);
[-Stealth] (invariance) – node[anchor=north] Dis. (disentanglement);{{figure:7c44091e-5c93-4ff3-9482-99d947efa557}}There exist a number of benchmarks for commonly used domain generalization techniques suggesting appropriate methods for specific tasks at hand. For example, Korevaar et al.  {{cite:687462e}} evaluated methods on benchmark datasets, whereas Zhang et al.  {{cite:89d4311}} evaluated them on some custom cross-site datasets consisting of publicly available datasets.
While these benchmarks shed light on the capabilities of certain techniques in domain generalization, they do not offer a comprehensive guide for problem-specific and task-specific strategies throughout the MedIA workflow.
To this end, we suggest strategies to incorporate domain generalization into the model, as shown in Fig. {{figure:7c44091e-5c93-4ff3-9482-99d947efa557}} .


HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation

Abstract: Minimizing the discrepancy of feature distributions between different domains is one of the most promising directions in unsupervised domain adaptation. From the perspective of distribution matching, most existing discrepancy-based methods are designed to match the second-order or lower statistics, which however, have limited expression of statistical characteristic for non-Gaussian distributions. In this work, we explore the benefits of using higher-order statistics (mainly refer to third-order and fourth-order statistics) for domain matching. We propose a Higher-order Moment Matching (HoMM) method, and further extend the HoMM into reproducing kernel Hilbert spaces (RKHS). In particular, our proposed HoMM can perform arbitrary-order moment tensor matching, we show that the first-order HoMM is equivalent to Maximum Mean Discrepancy (MMD) and the second-order HoMM is equivalent to Correlation Alignment (CORAL). Moreover, the third-order and the fourth-order moment tensor matching are expected to perform comprehensive domain alignment as higher-order statistics can approximate more complex, non-Gaussian distributions. Besides, we also exploit the pseudo-labeled target samples to learn discriminative representations in the target domain, which further improves the transfer performance. Extensive experiments are conducted, showing that our proposed HoMM consistently outperforms the existing moment matching methods by a large margin. Codes are available at \url{https://github.com/chenchao666/HoMM-Master}

Unsupervised Domain Adaptation Using Feature-Whitening and Consensus Loss

Abstract: A classifier trained on a dataset seldom works on other datasets obtained under different conditions due to domain shift. This problem is commonly addressed by domain adaptation methods. In this work we introduce a novel deep learning framework which unifies different paradigms in unsupervised domain adaptation. Specifically, we propose domain alignment layers which implement feature whitening for the purpose of matching source and target feature distributions. Additionally, we leverage the unlabeled target data by proposing the Min-Entropy Consensus loss, which regularizes training while avoiding the adoption of many user-defined hyper-parameters. We report results on publicly available datasets, considering both digit classification and object recognition tasks. We show that, in most of our experiments, our approach improves upon previous methods, setting new state-of-the-art performances.

Abstract: Recently, remarkable progress has been made in learning transferable representation across domains. Previous works in domain adaptation are majorly based on two techniques: domain-adversarial learning and self-training. However, domain-adversarial learning only aligns feature distributions between domains but does not consider whether the target features are discriminative. On the other hand, self-training utilizes the model predictions to enhance the discrimination of target features, but it is unable to explicitly align domain distributions. In order to combine the strengths of these two methods, we propose a novel method called Adversarial-Learned Loss for Domain Adaptation (ALDA). We first analyze the pseudo-label method, a typical self-training method. Nevertheless, there is a gap between pseudo-labels and the ground truth, which can cause incorrect training. Thus we introduce the confusion matrix, which is learned through an adversarial manner in ALDA, to reduce the gap and align the feature distributions. Finally, a new loss function is auto-constructed from the learned confusion matrix, which serves as the loss for unlabeled target samples. Our ALDA outperforms state-of-the-art approaches in four standard domain adaptation datasets. Our code is available at https://github.com/ZJULearning/ALDA.

Performance Comparison (Classification Accuracy in %) of Different Methods on Office-Home Dataset for object classification. `Ar', `Cl', `Pr', `Rw' are different domains in the Office-Home dataset.

All the experiment result comparisons are shown in Table {{table:816f14d5-5a3e-467a-b244-047cad594e72}} , {{table:12183196-612e-44f8-9510-254e06d0e515}} , {{table:a8b91040-694c-48e8-b077-aa3ecc38c0a5}} , {{table:1d2faad5-5348-43c4-9b10-45b7e2c343c2}}  (image classification); Table {{table:ec89894f-14cb-4b96-8040-09be1b8df687}} , {{table:47b50a18-86d6-4305-9c53-56c16a1dc26e}}  (object detection); and Table {{table:6d808aed-8ebc-44e5-9b64-da50aebeeee1}} , {{table:13c1396e-7ed9-46b8-9bdb-f56f00a4feb8}}  (semantic segmentation). For each backbone, the methods are sorted by average classification accuracy, mAP and mIoU.


Swin Transformer V2: Scaling Up Capacity and Resolution

Abstract: We present techniques for scaling Swin Transformer [35] up to 3 billion parameters and making it capable of training with images of up to 1,536x1,536 resolution. By scaling up capacity and resolution, Swin Transformer sets new records on four representative vision benchmarks: 84.0% top-1 accuracy on ImageNet- V2 image classification, 63.1 / 54.4 box / mask mAP on COCO object detection, 59.9 mIoU on ADE20K semantic segmentation, and 86.8% top-1 accuracy on Kinetics-400 video action classification. We tackle issues of training instability, and study how to effectively transfer models pre-trained at low resolutions to higher resolution ones. To this aim, several novel technologies are proposed: 1) a residual post normalization technique and a scaled cosine attention approach to improve the stability of large vision models; 2) a log-spaced continuous position bias technique to effectively transfer models pre-trained at low-resolution images and windows to their higher-resolution counterparts. In addition, we share our crucial implementation details that lead to significant savings of GPU memory consumption and thus make it feasi-ble to train large vision models with regular GPUs. Using these techniques and self-supervised pre-training, we suc-cessfully train a strong 3 billion Swin Transformer model and effectively transfer it to various vision tasks involving high-resolution images or windows, achieving the state-of-the-art accuracy on a variety of benchmarks. Code is avail-able at https://github.com/microsoft/Swin-Transformer.

Multilingual Denoising Pre-training for Neural Machine Translation

Abstract: Abstract This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART—a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective (Lewis et al., 2019). mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, whereas previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine-tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task- specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show that it enables transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.1

Model architecture with increasing number of GPUs. M: million. B: billion.

We perform weak scaling test to evaluate the training performance of all these models. Following the practice of Megatron {{cite:1bb9785}} and Alpa {{cite:73ddaac}}, we expand model size with the increasingly larger number of GPUs.
The batch size is set to 128 for AlphaFold2 following {{cite:4e8d58a}}, with other models set to 512.
The complete model configurations are listed in Table {{table:59bbdc57-4011-41be-9693-ee55db646220}} .
Similar to {{cite:1bb9785}}, {{cite:73ddaac}}, we evaluate the performance in aggregated tera floating point per second (TFLOPS).