Modeling the role of respiratory droplets in Covid-19 type pandemics

Abstract: In this paper, we develop a first principles model that connects respiratory droplet physics with the evolution of a pandemic such as the ongoing Covid-19. The model has two parts. First, we model the growth rate of the infected population based on a reaction mechanism. The advantage of modeling the pandemic using the reaction mechanism is that the rate constants have sound physical interpretation. The infection rate constant is derived using collision rate theory and shown to be a function of the respiratory droplet lifetime. In the second part, we have emulated the respiratory droplets responsible for disease transmission as salt solution droplets and computed their evaporation time, accounting for droplet cooling, heat and mass transfer, and finally, crystallization of the dissolved salt. The model output favourably compares with the experimentally obtained evaporation characteristics of levitated droplets of pure water and salt solution, respectively, ensuring fidelity of the model. The droplet evaporation/desiccation time is, indeed, dependent on ambient temperature and is also a strong function of relative humidity. The multi-scale model thus developed and the firm theoretical underpinning that connects the two scales—macro-scale pandemic dynamics and micro-scale droplet physics—thus could emerge as a powerful tool in elucidating the role of environmental factors on infection spread through respiratory droplets.

A Molecular Communication Perspective on Airborne Pathogen Transmission and Reception via Droplets Generated by Coughing and Sneezing

Abstract: Infectious diseases spread via pathogens such as viruses and bacteria. Airborne pathogen transmission via droplets is an important mode for infectious diseases. In this paper, the spreading mechanism of infectious diseases by airborne pathogen transmission between two humans is modeled with a molecular communication perspective. An end-to-end system model which considers the pathogen-laden cough/sneeze droplets as the input and the infection state of the human as the output is proposed. This model uses the gravity, initial velocity and buoyancy for the propagation of droplets and a receiver model which considers the central part of the human face as the reception interface is proposed. Furthermore, the probability of infection for an uninfected human is derived by modeling the number of propagating droplets as a random process. The numerical results reveal that exposure time affects the probability of infection. In addition, the social distance for a horizontal cough should be at least 1.7 m and the safe coughing angle of a coughing human to infect less people should be less than −25°.

Modeling of Viral Aerosol Transmission and Detection

Abstract: In this paper, we propose studying the disease spread mechanism in the atmosphere as an engineering problem. Aerosol transmission is the most significant mode among the viral transmission mechanisms that do not include physical contact, where airflows carry virus-laden droplets over long distances. Throughout this work, we study the transport of these droplets as a molecular communication problem, where one has no control over the transmission source, but a robust receiver can be designed using bio-sensors. To this end, we present a complete system model and derive an end-to-end mathematical model for the transmission channel under certain constraints and boundary conditions. We derive the system response for both continuous sources such as breathing and jet or impulsive sources such as coughing and sneezing. In addition to transmitter and channel, we assumed a receiver architecture composed of air sampler and Silicon Nanowire field-effect transistor. Then, we formulate a detection problem to maximize the likelihood decision rule and minimize the corresponding missed detection probability. Finally, we present several numerical results to observe the impact of parameters that affect the performance and justify the feasibility of the proposed setup in related applications.

Literature review summary on channel models for virus air propagation

We summarise the literature on existing droplet propagation models with certain properties of the droplet and airborne viral particles in Table {{table:ba1d5f4b-7561-49af-853b-68340c11a3d7}} . Even though they are not entirely classified as pure Molecular Communications channels, they present not only the physical modelling of droplet propagation but also the effects of droplet propagation and thus can be considered by the community for characterising Molecular Communication Systems. We analyse the literature in terms of the completeness of the physics that govern the droplet propagation. First, we look at their modes of propagation, either air-based or molecular-based. These modes dictate the way these models are constructed. Then, we classify each type of medium that can be utilised for modelling these Molecular Communications approaches. The types of species include airflow behaviour with average droplet concentration (transient air, air cloud) to more focused on the characterisation of the number of droplets (single, distribution, concentration and concentration/rate). We also analyse the airflow properties that are mostly secreted from a person as the transmitting point, this includes turbulent flow and puff flow. Turbulent flow accounts for the advection-diffusion of particles that are influenced by a force, in this case, the air turbulence and flow created from the transmitting point. The puff flow can be regarded as the Brownian diffusion in the air and can be influenced by gravity. Lastly, we analyse the properties of the environment that affects the state of droplets once they are excreted and this includes evaporation and crystallisation. Since the majority of the droplets is comprised of water, it is subject to effects from temperature change that can result in evaporation, as well as the quantity of salt in the droplet that lead to its crystallisation. From Table {{table:ba1d5f4b-7561-49af-853b-68340c11a3d7}} , we observe that all species comply with the puff and turbulent flow ({{cite:bc5e135}}, {{cite:0c62a89}}, {{cite:b3be751}}, {{cite:c0a37ce}}). However, we do recognise that environmental effects on the droplets are not fully explored for Molecular Communications models. The environmental effects have a significant impact on the propagation of the droplets, as it can either impact (a) on the flow behaviour in space, or (b) on the rate of virus reception by the receiving organ (e.g., nose or lung). Besides future investigation in environmental effects on the droplet propagation, there are also needs for further investigation on the effects of jet streams that affect the viral propagation behaviour. This includes understanding the aerodynamic airflow within confined and open areas and how this affects the flow of the viral particle propagation.

Leveraging Experimental Data for Molecular Communications

The airborne virus propagation models through droplets based on Model 8 need a set of data to characterise diffusion properties when a human transmitter sneezes, coughs or talks. The availability of this data is a critical issue as we found very limited resources that can be used. However, existing testbeds can be used to generate data that can be used by the Molecular Communications researchers. A particular testbed that is appropriate for this scenario is the Tabletop Molecular Communication {{cite:1ea5956}}, where the authors presented the release of isopropyl alcohol molecules through an electronic-activated spray. These airborne molecules flow towards an alcohol sensor through the airflow produced by a fan. The authors demonstrate the successful encoding, transmission and reception of information encoded using molecules concentration. This testbed can be used to study the effects of droplet propagation, i.e., coupling it with Model 8. The usage of fans can provide modifications to the airflow that drives the propagation of the droplets, and hence air jet streams found in different indoor or outdoor scenarios. This would help estimate parameters for velocity and turbulence measures, such as {{formula:3993db64-3e2d-4a76-ab1a-1f578b753a59}}  and {{formula:0cee7ab2-1458-4cba-b621-d8521f214b6a}}  in (REF ). The production of droplets is not clear, especially when trying to emulate events of sneeze, cough and speech. However, we believe the testbed can be extended to include different versions of the electronic-activated spray that can modulate the release rate of molecules, for example. In that way, parameter {{formula:686ef1d4-0802-43dc-841e-84af47a3ea9f}}  in (REF ) would be associated with a proper virus release rate profile in each emulated event. Lastly, the propagation of the droplets itself can be used to determine other model parameters including the droplet degradation loss derived from environmental effects, which is {{formula:ab6766b1-bee3-4568-8580-af23a15b3c8e}}  in REF . The value of {{formula:a1e41e7f-227b-40ad-90a7-ab51e9ee3297}}  can be broken down into different components as shown in Table {{table:ba1d5f4b-7561-49af-853b-68340c11a3d7}} , where the testbed can be used to estimate {{formula:6ae3130a-49cb-443c-be7a-81f5ccb2609a}} .


Improved Speech Reconstruction from Silent Video

Abstract: Speechreading is the task of inferring phonetic information from visually observed articulatory facial movements, and is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible and natural-sounding acoustic speech signal from silent video frames of a speaking person. We train our model on speakers from the GRID and TCD-TIMIT datasets, and evaluate the quality and intelligibility of reconstructed speech using common objective measurements. We show that speech predictions from the proposed model attain scores which indicate significantly improved quality over existing models. In addition, we show promising results towards reconstructing speech from an unconstrained dictionary.

Lip2AudSpec: Speech reconstruction from silent lip movements video

Abstract: In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.

Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feed

Abstract: Speechreading or lipreading is the technique of understanding and getting phonetic features from a speaker's visual features such as movement of lips, face, teeth and tongue. It has a wide range of multimedia applications such as in surveillance, Internet telephony, and as an aid to a person with hearing impairments. However, most of the work in speechreading has been limited to text generation from silent videos. Recently, research has started venturing into generating (audio) speech from silent video sequences but there have been no developments thus far in dealing with divergent views and poses of a speaker. Thus although, we have multiple camera feeds for the speech of a user, but we have failed in using these multiple video feeds for dealing with the different poses. To this end, this paper presents the world's first ever multi-view speech reading and reconstruction system. This work encompasses the boundaries of multimedia research by putting forth a model which leverages silent video feeds from multiple cameras recording the same subject to generate intelligent speech for a speaker. Initial results confirm the usefulness of exploiting multiple camera views in building an efficient speech reading and reconstruction system. It further shows the optimal placement of cameras which would lead to the maximum intelligibility of speech. Next, it lays out various innovative applications for the proposed system focusing on its potential prodigious impact in not just security arena but in many other multimedia analytics problems.

Lipper: Synthesizing Thy Speech using Multi-View Lipreading

Abstract: Lipreading has a lot of potential applications such as in the domain of surveillance and video conferencing. Despite this, most of the work in building lipreading systems has been limited to classifying silent videos into classes representing text phrases. However, there are multiple problems associated with making lipreading a text-based classification task like its dependence on a particular language and vocabulary mapping. Thus, in this paper we propose a multi-view lipreading to audio system, namely Lipper, which models it as a regression task. The model takes silent videos as input and produces speech as the output. With multi-view silent videos, we observe an improvement over single-view speech reconstruction results. We show this by presenting an exhaustive set of experiments for speaker-dependent, out-of-vocabulary and speaker-independent settings. Further, we compare the delay values of Lipper with other speechreading systems in order to show the real-time nature of audio produced. We also perform a user study for the audios produced in order to understand the level of comprehensibility of audios produced using Lipper.

Video-Driven Speech Reconstruction using Generative Adversarial Networks

Abstract: Speech is a means of communication which relies on both audio and visual information. The absence of one modality can often lead to confusion or misinterpretation of information. In this paper we present an end-to-end temporal model capable of directly synthesising audio from silent video, without needing to transform to-and-from intermediate features. Our proposed approach, based on GANs is capable of producing natural sounding, intelligible speech which is synchronised with the video. The performance of our model is evaluated on the GRID dataset for both speaker dependent and speaker independent scenarios. To the best of our knowledge this is the first method that maps video directly to raw audio and the first to produce intelligible speech when tested on previously unseen speakers. We evaluate the synthesised audio not only based on the sound quality but also on the accuracy of the spoken words.

Vocoder-Based Speech Synthesis from Silent Videos

Abstract: Both acoustic and visual information influence human perception of speech. For this reason, the lack of audio in a video sequence determines an extremely low speech intelligibility for untrained lip readers. In this paper, we present a way to synthesise speech from the silent video of a talker using deep learning. The system learns a mapping function from raw video frames to acoustic features and reconstructs the speech with a vocoder synthesis algorithm. To improve speech reconstruction performance, our model is also trained to predict text information in a multi-task learning fashion and it is able to simultaneously reconstruct and recognise speech in real time. The results in terms of estimated speech quality and intelligibility show the effectiveness of our method, which exhibits an improvement over existing video-to-speech approaches.

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

Abstract: Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speaker-specific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the single-speaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space.

Deep-learning-based approaches for speech reconstruction from silent videos. MV: multi-view. SI: speaker-independent. VSR: visual speech recognition.

Speech Reconstruction from Silent Videos

As an alternative to VSR-TTS systems, techniques that directly perform a video-to-speech mapping have been proposed (cf. Table {{table:7ab6d7b7-40db-438d-917f-5fa9a8adbbe7}} ).
Although some attempts were made to reconstruct intelligible speech from silent articulations captured with several sensorsThe data used in these works include (but are not limited to) recordings obtained with electromagnetic articulography, electropalatography and laryngography sensors. {{cite:0b29a22}}, {{cite:b470856}}, {{cite:6cf8ba1}}, {{cite:d7cb349}}, Le Cornu and Milner {{cite:fa9ce4a}} were the first to employ a neural network using only the silent video of a speaker's frontal face. They decided to base their system on STRAIGHT {{cite:7a75125}}, a vocoder which allows to perform speech synthesis from three time-varying parameters describing fundamental aspects of a given speech signal: fundamental frequency (F0), aperiodicity (AP) and spectral envelope (SP). Supported by the results of some previous works {{cite:d6dc0e3}}, {{cite:b67ace3}}, {{cite:084a70d}}, they assumed that only SP could be inferred from visual features. Therefore, AP and F0 were not estimated from the silent video, but artificially produced without taking the visual information into account, while SP was estimated with a Gaussian mixture model (GMM) and FFNN within a regression-based framework. As input to the models, two different visual features were considered, 2-D DCT and AAM, while the explored SP representations were linear predictive coding (LPC) coefficients and mel-filterbank amplitudes. While the choice of visual features did not have a big impact on the results, the use of mel-filterbank amplitudes allowed to outperform the systems based on LPC coefficients.


Rethinking the Inception Architecture for Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.

Attention is All you Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Towards Better Decoding and Language Model Integration in Sequence to Sequence Models

Abstract: The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion. In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters. We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used. We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.

Survey of literature label smoothing results according to {{cite:f8ee366}}.

Label smoothing has emerged as a useful technique in training neural networks {{cite:de4f22d}}, {{cite:2a5f433}}, {{cite:32865a8}}, {{cite:9fdc029}}, {{cite:d4136e2}}, {{cite:97587f1}} and empirical results have pointed towards its ability to improve performance across a suite of tasks in image classification, machine translation, and language modeling {{cite:f8ee366}} (see Table {{table:d2a39cbc-697f-425e-9155-48c24dbffd37}}  for a summary of positive empirical results). Given a classification problem where the training targets {{formula:efc64234-cdcc-4b20-803f-8f023639cf93}}  take the form of a one-hot vector (1 for the target class index and 0 elsewhere), label smoothing uses a smoothing hyperparameter {{formula:32cca298-f349-49bb-9ba6-a34e0000b425}}  to define a “soft” version of these training targets:
{{formula:de6bd84f-f04c-49d9-90dd-762c6e997b37}}


FedMix performs data augmentation based on the MixUp strategy.

Astraea performs data augmentation based on the global data distribution, generated by collecting the local data distribution.

FedMD enables clients to independently design their models and implements communication between heterogeneous models.

They mitigate overfitting in personalized updates by enhancing the logits similarity between the global model and the local models.

Collecting local data distributions may bring potential information leakage.

Uploading local data distributions may expose potential backdoors to attacks.

Global knowledge cannot describe rich domain knowledge in feature skew scenarios.

Logits Exploitation may lead to insufficient learning of local information.

FedMix: Approximation of Mixup under Mean Augmented Federated Learning

Abstract: Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device, thus preserving privacy and eliminating the need to store data globally. While there are promising results under the assumption of independent and identically distributed (iid) local data, current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. To resolve this issue, we propose a simple framework, Mean Augmented Federated Learning (MAFL), where clients send and receive averaged local data, subject to the privacy requirements of target applications. Under our framework, we propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup, but does not require local raw data to be directly shared among devices. Our method shows greatly improved performance in the standard benchmark datasets of FL, under highly non-iid federated settings, compared to conventional algorithms.

Astraea: Self-Balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications

Abstract: Federated learning (FL) is a distributed deep learning method which enables multiple participants, such as mobile phones and IoT devices, to contribute a neural network model while their private training data remains in local devices. This distributed approach is promising in the edge computing system where have a large corpus of decentralized data and require high privacy. However, unlike the common training dataset, the data distribution of the edge computing system is imbalanced which will introduce biases in the model training and cause a decrease in accuracy of federated learning applications. In this paper, we demonstrate that the imbalanced distributed training data will cause accuracy degradation in FL. To counter this problem, we build a self-balancing federated learning framework call Astraea, which alleviates the imbalances by 1) Global data distribution based data augmentation, and 2) Mediator based multi-client rescheduling. The proposed framework relieves global imbalance by runtime data augmentation, and for averaging the local imbalance, it creates the mediator to reschedule the training of clients based on Kullback-Leibler divergence (KLD) of their data distribution. Compared with FedAvg, the state-of-the-art FL algorithm, Astraea shows +5.59% and +5.89% improvement of top-1 accuracy on the imbalanced EMNIST and imbalanced CINIC-10 datasets, respectively. Meanwhile, the communication traffic of Astraea can be 92% lower than that of FedAvg.

FedMD: Heterogenous Federated Learning via Model Distillation

Abstract: Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.

Salvaging Federated Learning by Local Adaptation

Abstract:
Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the federated model, latest FL approaches use differential privacy or robust aggregation to limit the influence of "outlier" participants.
First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants.
Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each technique is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model and who have no incentive to participate in FL today improve less, but sufficiently to make the adapted federated model better than their local models.

Federated unsupervised representation learning

Abstract: To leverage the enormous amount of unlabeled data on distributed edge devices, we formulate a new problem in federated learning called federated unsupervised representation learning (FURL) to learn a common representation model without supervision while preserving data privacy. FURL poses two new challenges: (1) data distribution shift (non-independent and identically distributed, non-IID) among clients would make local models focus on different categories, leading to the inconsistency of representation spaces; (2) without unified information among the clients in FURL, the representations across clients would be misaligned. To address these challenges, we propose the federated contrastive averaging with dictionary and alignment (FedCA) algorithm. FedCA is composed of two key modules: a dictionary module to aggregate the representations of samples from each client which can be shared with all clients for consistency of representation space and an alignment module to align the representation of each client on a base model trained on public data. We adopt the contrastive approach for local model training. Through extensive experiments with three evaluation protocols in IID and non-IID settings, we demonstrate that FedCA outperforms all baselines with significant margins.

Heterogeneous Federated Learning: State-of-the-art and Research Challenges

Abstract: Federated learning (FL) has drawn increasing attention owing to its potential use in large-scale industrial applications. Existing FL works mainly focus on model homogeneous settings. However, practical FL typically faces the heterogeneity of data distributions, model architectures, network environments, and hardware devices among participant clients. Heterogeneous Federated Learning (HFL) is much more challenging, and corresponding solutions are diverse and complex. Therefore, a systematic survey on this topic about the research challenges and state-of-the-art is essential. In this survey, we firstly summarize the various research challenges in HFL from five aspects: statistical heterogeneity, model heterogeneity, communication heterogeneity, device heterogeneity, and additional challenges. In addition, recent advances in HFL are reviewed and a new taxonomy of existing HFL methods is proposed with an in-depth analysis of their pros and cons. We classify existing methods from three different levels according to the HFL procedure: data-level, model-level, and server-level. Finally, several critical and promising future research directions in HFL are discussed, which may facilitate further developments in this field. A periodically updated collection on HFL is available at https://github.com/marswhu/HFL_Survey.

blackData-level methods.

blackIn this subsection, we introduce the classification of data-level methods and some representative methods in each category, as shown in Tab. {{table:9a5d6a8d-9c2d-468f-be16-d27cf24fe1ba}} . Data-level methods refer to operations performed at the data level, including private data processing and external data utilization. Private data processing means that clients internally process private data to improve data quality, diversity, and security, thereby optimizing the performance of federated learning. These methods include data preparation and data privacy protection. Data preparation includes operations such as data collection, filtering, cleaning, and augmentation, which can directly alleviate statistical heterogeneity. Data privacy protection aims to ensure that the original data information is not disclosed. External data utilization refers to performing model knowledge distillation or imposing constraints on model updates by introducing additional data. Knowledge distillation is usually employed to deal with communication difficulties caused by model heterogeneity and can alleviate data heterogeneity and communication heterogeneity to some extent. Unsupervised representation learning can alleviate the statistical heterogeneity between the local data of clients.


Amoxicillin, Cefotaxime, Ceftazidime, Cefuroxime, Ciprofloxacin, Gentamicin

Comprehensive Identification of Single Nucleotide Polymorphisms Associated with Beta-lactam Resistance within Pneumococcal Mosaic Genes

Abstract: Traditional genetic association studies are very difficult in bacteria, as the generally limited recombination leads to large linked haplotype blocks, confounding the identification of causative variants. Beta-lactam antibiotic resistance in Streptococcus pneumoniae arises readily as the bacteria can quickly incorporate DNA fragments encompassing variants that make the transformed strains resistant. However, the causative mutations themselves are embedded within larger recombined blocks, and previous studies have only analysed a limited number of isolates, leading to the description of “mosaic genes” as being responsible for resistance. By comparing a large number of genomes of beta-lactam susceptible and non-susceptible strains, the high frequency of recombination should break up these haplotype blocks and allow the use of genetic association approaches to identify individual causative variants. Here, we performed a genome-wide association study to identify single nucleotide polymorphisms (SNPs) and indels that could confer beta-lactam non-susceptibility using 3,085 Thai and 616 USA pneumococcal isolates as independent datasets for the variant discovery. The large sample sizes allowed us to narrow the source of beta-lactam non-susceptibility from long recombinant fragments down to much smaller loci comprised of discrete or linked SNPs. While some loci appear to be universal resistance determinants, contributing equally to non-susceptibility for at least two classes of beta-lactam antibiotics, some play a larger role in resistance to particular antibiotics. All of the identified loci have a highly non-uniform distribution in the populations. They are enriched not only in vaccine-targeted, but also non-vaccine-targeted lineages, which may raise clinical concerns. Identification of single nucleotide polymorphisms underlying resistance will be essential for future use of genome sequencing to predict antibiotic sensitivity in clinical microbiology.

Systematic longitudinal survey of invasive Escherichia coli in England demonstrates a stable population structure only transiently disturbed by the emergence of ST131

Abstract: Escherichia coli associated with urinary tract infections and bacteremia has been intensively investigated, including recent work focusing on the virulent, globally disseminated, multidrug-resistant lineage ST131. To contextualize ST131 within the broader E. coli population associated with disease, we used genomics to analyze a systematic 11-yr hospital-based survey of E. coli associated with bacteremia using isolates collected from across England by the British Society for Antimicrobial Chemotherapy and from the Cambridge University Hospitals NHS Foundation Trust. Population dynamics analysis of the most successful lineages identified the emergence of ST131 and ST69 and their establishment as two of the five most common lineages along with ST73, ST95, and ST12. The most frequently identified lineage was ST73. Compared to ST131, ST73 was susceptible to most antibiotics, indicating that multidrug resistance was not the dominant reason for prevalence of E. coli lineages in this population. Temporal phylogenetic analysis of the emergence of ST69 and ST131 identified differences in the dynamics of emergence and showed that expansion of ST131 in this population was not driven by sequential emergence of increasingly resistant subclades. We showed that over time, the E. coli population was only transiently disturbed by the introduction of new lineages before a new equilibrium was rapidly achieved. Together, these findings suggest that the frequency of E. coli lineages in invasive disease is driven by negative frequency-dependent selection occurring outside of the hospital, most probably in the commensal niche, and that drug resistance is not a primary determinant of success in this niche.

Summary of data sets tested

We investigate performance of the different methods using four public bacterial datasets suitable for GWAS and simulated phenotypes based on the genetic architecture present in the such data. We focus on estimating heritability of the antibiotic resistance phentotypes. Table {{table:9152f0c5-2478-436b-95e2-dcf86925c79e}}  summarizes our test datasets. The heritability of the antibiotic resistance phenotype is expected to be high, meaning that the variability stems primarily from the observed genetic differences among these bacteria.


Geom-GCN: Geometric Graph Convolutional Networks

Abstract: Message-passing neural networks (MPNNs) have been successfully applied in a wide variety of applications in the real world. However, two fundamental weaknesses of MPNNs' aggregators limit their ability to represent graph-structured data: losing the structural information of nodes in neighborhoods and lacking the ability to capture long-range dependencies in disassortative graphs. Few studies have noticed the weaknesses from different perspectives. From the observations on classical neural network and network geometry, we propose a novel geometric aggregation scheme for graph neural networks to overcome the two weaknesses. The behind basic idea is the aggregation on a graph can benefit from a continuous space underlying the graph. The proposed aggregation scheme is permutation-invariant and consists of three modules, node embedding, structural neighborhood, and bi-level aggregation. We also present an implementation of the scheme in graph convolutional networks, termed Geom-GCN, to perform transductive learning on graphs. Experimental results show the proposed Geom-GCN achieved state-of-the-art performance on a wide range of open datasets of graphs.

Multi-scale Attributed Node Embedding

Abstract: We present network embedding algorithms that capture information about a node from the local distribution over node attributes around it, as observed over random walks following an approach similar to Skip-gram. Observations from neighbourhoods of different sizes are either pooled (AE) or encoded distinctly in a multi-scale approach (MUSAE). Capturing attribute-neighbourhood relationships over multiple scales is useful for a range of applications, including latent feature identification across disconnected networks with similar features. We prove theoretically that matrices of node-feature pointwise mutual information are implicitly factorized by the embeddings. Experiments show that our algorithms are computationally efficient and outperform comparable models on social networks and web graphs.

Statistics of the six heterophily datasets.

As shown in Table {{table:22dcbc26-331c-460a-b9a5-227100c48463}} , Actor {{cite:d10b5da}} is a graph representing actor co-occurrence in Wikipedia pages, Texas, Wisconsin and Cornell are the hyperlinked web pages of various universities provided by {{cite:d10b5da}}, Chameleon and Squirrel are the subgraphs of web pages in Wikipedia discussing the corresponding topics provided by {{cite:f40a8dc}}.


Public, online and household activities, gathering, mobility data, mobile phone data, social media data, etc.

Social and political data, case data, surveys, questionnaires, sociopolitical events, etc.

Understanding the Impact of the COVID-19 Pandemic on Transportation-related Behaviors with Human Mobility Data

Abstract: The constrained outbreak of COVID-19 in Mainland China has recently been regarded as a successful example of fighting this highly contagious virus. Both the short period (in about three months) of transmission and the sub-exponential increase of confirmed cases in Mainland China have proved that the Chinese authorities took effective epidemic prevention measures, such as case isolation, travel restrictions, closing recreational venues, and banning public gatherings. These measures can, of course, effectively control the spread of the COVID-19 pandemic. Meanwhile, they may dramatically change the human mobility patterns, such as the daily transportation-related behaviors of the public. To better understand the impact of COVID-19 on transportation-related behaviors and to provide more targeted anti-epidemic measures, we use the huge amount of human mobility data collected from Baidu Maps, a widely-used Web mapping service in China, to look into the detail reaction of the people there during the pandemic. To be specific, we conduct data-driven analysis on transportation-related behaviors during the pandemic from the perspectives of 1) means of transportation, 2) type of visited venues, 3) check-in time of venues, 4) preference on "origin-destination" distance, and 5) "origin-transportation-destination" patterns. For each topic, we also give our specific insights and policy-making suggestions. Given that the COVID-19 pandemic is still spreading in more than 200 countries and territories worldwide, infecting millions of people, the insights and suggestions provided here may help fight COVID-19.

Modelling Threat Causation for Religiosity and Nationalism in Europe

Abstract: Europe's contemporary political landscape has been shaped by massive shifts in recent decades caused by geopolitical upheavals such as Brexit and now, COVID-19. The way in which policy makers respond to the current pandemic could have large effects on how the world looks after the pandemic subsides. We aim to investigate complex questions post COVID-19 around the relationships and intersections concerning nationalism, religiosity, and anti-immigrant sentiment from a socio-cognitive perspective by applying a mixed-method approach (survey and modelling); in a context where unprecedented contagion threats have caused huge instability. There are still significant gaps in the scholarly literature on populism and nationalism. In particular, there is a lack of attention to the role of evolved human psychology in responding to persistent threats, which can fall into four broad categories in the literature: predation (threats to one's life via being eaten or killed in some other way), contagion (threats to one's life via physical infection), natural (threats to one's life via natural disasters), and social (threats to one's life by destroying social standing). These threats have been discussed in light of their effects on religion and other forms of behaviour, but they have not been employed to study nationalist and populist behaviours. In what follows, two studies are presented that begin to fill this gap in the literature. The first is a survey used to inform our theoretical framework and explore the different possible relationships in an online sample. The second is a study of a computer simulation. Both studies (completed in 2020) found very clear effects among the relevant variables, enabling us to identify trends that require further explanation and research as we move toward models that can adequately inform policy discussions.

Examples of COVID-19 Influence and Impact Modeling.

Discussion. Table {{table:c218f685-1317-4f23-bd84-4f159b266ac3}}  summarizes examples of modeling the COVID-19 influence and impact. Typical research on COVID-19 influence and impact modeling only involves local and regional COVID-19 data and their affected objects, the resultant conclusions have limited indication of their applicability to general practice and broad pandemic control. More robust results are expected to inform medical and public health policy-making on medication, business and society. No to rare outcomes are available on how NPIs influence the threshold and effects of COVID-19 vaccinations and herd immunity and on how to balance NPIs and the economic and social revivification. It is demanding to have actionable evidence and guidelines on what policies should be taken and what tradeoff is appropriate in balancing the COVID-19 outbreak and resurgence containment and the economic and social business recovery.


Trajectories of 20 buses in Athens, Georgia

Taxi and bus trajectories in San Francisco, USA

Chemical Reaction Optimization (meta-heuristic)

Sensor deployment for air pollution monitoring using public transportation system

Abstract: Air pollution monitoring is a very popular research topic and many monitoring systems have been developed. In this paper, we formulate the Bus Sensor Deployment Problem (BSDP) to select the bus routes on which sensors are deployed, and we use Chemical Reaction Optimization (CRO) to solve BSDP. CRO is a recently proposed metaheuristic designed to solve a wide range of optimization problems. Using the real world data, namely Hong Kong Island bus route data, we perform a series of simulations and the results show that CRO is capable of solving this optimization problem efficiently.

Spatio-Temporal Coverage Enhancement in Drive-By Sensing Through Utility-Aware Mobile Agent Selection

Abstract: In recent years, the drive-by sensing paradigm has become increasingly popular for cost-effective monitoring of urban areas. Drive-by sensing is a form of crowdsensing wherein sensor-equipped vehicles (aka, mobile agents) are the primary data gathering agents. Enhancing the efficacy of drive-by sensing poses many challenges, an important one of which is to select non-dedicated mobile agents on which a limited number of sensors are to be mounted. This problem, which we refer to as the mobile-agent selection problem, has a significant impact on the spatio-temporal coverage of the drive-by sensing platforms and the resultant datasets. The challenge here is to achieve maximum spatiotemporal coverage while taking the relative importance levels of geographical areas into account. In this paper, we address this problem in the context of the SCOUTS project, the goal of which is to map and analyze the urban heat island phenomenon accurately. Our work makes several major technical contributions. First, we delineate a model for representing the mobile agents selection problem. This model takes into account the trajectories of the vehicles (public transportation buses in our case) and the relative importance of the urban regions, and formulates it as an optimization problem. Second, we provide two algorithms that are based upon the utility (coverage) values of mobile agents, namely, a hotspot-based algorithm that limits the search space to important sub-regions and a utility-aware genetic algorithm that enables the latter algorithm to make unbiased selections. Third, we design a highly efficient coverage redundancy minimization algorithm that, at each step, chooses the mobile agent, which provides maximal improvement to the spatio-temporal coverage. This paper reports a series of experiments on a real-world dataset from Athens, GA, USA, to demonstrate the effectiveness of the proposed approaches.

Modulo: Drive-by Sensing at City-scale on the Cheap

Abstract: Drive-by sensing is gaining popularity as an inexpensive way to perform fine-grained, city-scale, spatiotemporal monitoring of physical phenomena. Prior work explores several challenges in the design of low-cost sensors, the reliability of these sensors, and their application for specific use-cases like pothole detection and pollution monitoring. However, the process of deployment of a drive-by sensing network at a city-scale is still unexplored. Despite the rise of ride-sharing services, there is still no way to optimally select vehicles from a fleet that can accomplish the sensing task by providing enough coverage of the city. In this paper, we propose Modulo -- a system to bootstrap drive-by sensing deployment by taking into consideration a variety of aspects such as spatiotemporal coverage, budget constraints. Further, Modulo is well-suited to satisfy unique deployment constraints such as colocations with other sensors (needed for gas and PM sensor calibration), etc. We compare Modulo with two baseline algorithms on real-world taxi and bus datasets. We find that Modulo marginally outperforms the two baselines for datasets with just random-routes vehicles such as taxis. However, it significantly outperforms the baselines when a fleet comprises of both taxis and fixed-route vehicles such as public transport buses. Finally, we present a real-deployment that uses Modulo to select vehicles for an air pollution sensing application.

Summary of sensor deployment problems.

Table {{table:c9713f0b-342f-404a-a4d4-d879a987d4f5}}  summarizes relevant literature on the sensor location problem for various vehicle types. They all focused on allocating sensors to vehicles or bus/tram lines based on actual or predicted vehicle trajectories and employ heuristic, meta-heuristic, or approximate algorithms to solve the problem. It is noted that all these studies fall within the category of opportunistic sensing {{cite:fb42969}}, {{cite:5ee7367}}.


ImageNet(tag label), ILSVRC13 validation subset(box annotation)

Active learning for identifying object cluster

Mask out initialization + SVM detector training

Transferring detectors and representation from auxiliary data

Improving Weakly-Supervised Object Localization By Micro-Annotation

Abstract: Weakly-supervised object localization methods tend to fail for object classes that consistently co-occur with the same background elements, e.g. trains on tracks. We propose a method to overcome these failures by adding a very small amount of model-specific additional annotation. The main idea is to cluster a deep network's mid-level representations and assign object or distractor labels to each cluster. Experiments show substantially improved localization results on the challenging ILSVC2014 dataset for bounding box detection and the PASCAL VOC2012 dataset for semantic segmentation.

Weakly Supervised Localization using Deep Feature Maps

Abstract: Object localization is an important computer vision problem with a variety of applications. The lack of large scale object-level annotations and the relative abundance of image-level labels makes a compelling case for weak supervision in the object localization task. Deep Convolutional Neural Networks are a class of state-of-the-art methods for the related problem of object recognition. In this paper, we describe a novel object localization algorithm which uses classification networks trained on only image labels. This weakly supervised method leverages local spatial and semantic patterns captured in the convolutional layers of classification networks. We propose an efficient beam search based approach to detect and localize multiple objects in images. The proposed method significantly outperforms the state-of-the-art in standard object localization data-sets with a 8 point increase in mAP scores.

Self-taught Object Localization with Deep Networks

Abstract: This paper introduces self-taught object localization, a novel approach that leverages deep convolutional networks trained for whole-image recognition to localize objects in images without additional human supervision, i.e., without using any ground-truth bounding boxes for training. The key idea is to analyze the change in the recognition scores when artificially masking out different regions of the image. The masking out of a region that includes the object typically causes a significant drop in recognition score. This idea is embedded into an agglomerative clustering technique that generates self-taught localization hypotheses. Our object localization scheme outperforms existing proposal methods in both precision and recall for small number of subwindow proposals (e.g., on ILSVRC-2012 it produces a relative gain of 23.4% over the state-of-the-art for top-1 hypothesis). Furthermore, our experiments show that the annotations automatically-generated by our method can be used to train object detectors yielding recognition results remarkably close to those obtained by training on manually-annotated bounding boxes.

Detector discovery in the wild: Joint multiple instance and representation learning

Abstract: We develop methods for detector learning which exploit joint training over both weak (image-level) and strong (bounding box) labels and which transfer learned perceptual representations from strongly-labeled auxiliary tasks. Previous methods for weak-label learning often learn detector models independently using latent variable optimization, but fail to share deep representation knowledge across classes and usually require strong initialization. Other previous methods transfer deep representations from domains with strong labels to those with only weak labels, but do not optimize over individual latent boxes, and thus may miss specific salient structures for a particular category. We propose a model that subsumes these previous approaches, and simultaneously trains a representation and detectors for categories with either weak or strong labels present. We provide a novel formulation of a joint multiple instance learning method that includes examples from classification-style data when available, and also performs domain transfer learning to improve the underlying detector representation. Our model outperforms known methods on ImageNet-200 detection with weak labels.

Summary of the approaches using visual cues, which is a subcategory in the weakly supervised object localization and detection approaches based on the off-the-shelf deep models. * indicates a certain variation of the corresponding model. An approach is considered for general object category when it is tested for detecting more than five object categories in the corresponding literature. The approaches with None detector indicate the weakly supervised object localization approaches.

Instead of using the pre-trained deep models as feature extractor, the methods of this category obtain useful information cues (such as the activations in the intermediate network layers and the semantic scores in the output network layer) from the pre-trained deep neural networks to facilitate the weakly supervised learning process.
The focus of these approaches mainly lies in the initialization stage of the weakly supervised learning process.
A brief summary of these approaches is shown in Table {{table:89af18e4-c03d-43f4-bfbc-6bca0ea65334}} .


An open access repository of images on plant health to enable the development of mobile disease diagnostics through machine learning and crowdsourcing

Abstract: Human society needs to increase food production by an estimated 70% by 2050 to feed an expected population size that is predicted to be over 9 billion people. Currently infectious diseases reduce the potential yield by an average of 40% with many farmers in the developing world experiencing yield losses as high as 100%. Infectious diseases of crops are not new and historic examples such as the Irish Potato Famine of 1845-49 demonstrate this. But what is new is the widespread distribution of smartphones among crop growers around the world with an expected 5 billion smartphones by 2020. This offers the potential of turning the smartphone into a valuable tool for diverse communities growing food. One potential application is the development of mobile disease diagnostics through machine learning and crowdsourcing. Computer vision and machine learning have shown their potential to automatically classify images. To do this for plant diseases requires a training set that facilitates the development of the algorithms. Here we announce the release of >50,000 expertly curated images on healthy and infected leaves of crops plants through the existing platform www.PlantVillage.org. We describe both the data and the platform. These data are the beginning of an on-going, crowdsourcing effort to enable computer vision approaches to help solve the problem of yield losses in crop plants due to infectious diseases.

DiaMOS Plant: A Dataset for Diagnosis and Monitoring Plant Disease

Abstract: The classification and recognition of foliar diseases is an increasingly developing field of research, where the concepts of machine and deep learning are used to support agricultural stakeholders. Datasets are the fuel for the development of these technologies. In this paper, we release and make publicly available the field dataset collected to diagnose and monitor plant symptoms, called DiaMOS Plant, consisting of 3505 images of pear fruit and leaves affected by four diseases. In addition, we perform a comparative analysis of existing literature datasets designed for the classification and recognition of leaf diseases, highlighting the main features that maximize the value and information content of the collected data. This study provides guidelines that will be useful to the research community in the context of the selection and construction of datasets.

Publicly available plant disease datasets

Several open plant disease datasets emerged in recent years (Table {{table:fdbd559a-0711-4038-9ecf-a48371d0655f}} ). The PlantVillage dataset is the largest and most studied plant disease dataset. It contains more than 54,000 images of leaves on a homogenous background. This sparked a plethora of studies on plant disease detection using deep learning{{cite:a923624}}, {{cite:cb2e3c0}}. Most of the papers reported classification accuracies above 98%.


Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Xception: Deep Learning with Depthwise Separable Convolutions

Abstract: We present an interpretation of Inception modules in convolutional neural networks as being an intermediate step in-between regular convolution and the depthwise separable convolution operation (a depthwise convolution followed by a pointwise convolution). In this light, a depthwise separable convolution can be understood as an Inception module with a maximally large number of towers. This observation leads us to propose a novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions. We show that this architecture, dubbed Xception, slightly outperforms Inception V3 on the ImageNet dataset (which Inception V3 was designed for), and significantly outperforms Inception V3 on a larger image classification dataset comprising 350 million images and 17,000 classes. Since the Xception architecture has the same number of parameters as Inception V3, the performance gains are not due to increased capacity but rather to a more efficient use of model parameters.

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less memory and computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet.

In our ablation studies we show the impact of different design decisions on the performance of our network. The final LRPD configuration is marked in italic.

To verify the effectiveness of all components in our approach, we derived ablation studies.
These were performed on the KITTI pedestrian BEV benchmark regarding moderate AP.
We swapped out the encoder, changed the input modality and the proposal augmentation separately to observe the impacts.
The ablation studies were done on a 80:20 split on the KITTI Benchmark different from the 50:50 split used for our comparison with other approaches to avoid a bias.
In our ablation studies (Table {{table:3aca19d6-2955-4ed1-a1d2-5aac34fd60c9}} ) we show that both LiDAR and RGB are integral parts to the success of our approach.
By removing RGB or LiDAR from the second stage, described in Section III.C, the performance degrades significantly, highlighting the importance of using both sensor modalities.
Furthermore, the encoder choice has as strong impact on performance of the network. Where VGG19 {{cite:a562873}} and DenseNet {{cite:93fb99e}} significantly increase the performance, MobileNet {{cite:016d049}} slightly decreases the performance.

When varying the proposal method described in Section III.B, we can show that the choice of proposal augmentation has a significant impact on mAP (up to +11.4).
Baseline describes creating the proposals without any data augmentation and achieves the worst results of our proposal configurations.
Even though random augmentation has a far better overall moderate BEV mAP (Table {{table:3aca19d6-2955-4ed1-a1d2-5aac34fd60c9}} ) grounding yields better results for medium and long range detection (Figure {{figure:c55a4289-5bc0-414a-a0ba-7a917629f5ed}} ).
Combining the strong performance of random augmentation for short range detection with grounding for medium and long range detection, as described in Section III.B, yields best overall results, with only 0.1 mAP reduction on long range.
Despite a low overall BEV mAP of 43.5 grounding achieves the best results on long range (Table {{table:6bbd4fd3-a4dc-447a-8802-8cc15846fa0f}} ).
This proves that best mAP on KITTI does not correlate to detecting pedestrians better at long range.
Therefore, our distance range based analysis of the mAP is mandatory to gain insights into the performance characteristics of approaches,
unveiling that on long range detection LRPD significantly outperforms other state-of-the-art approaches.


A Convex Framework for Fair Regression

Abstract: We introduce a flexible family of fairness regularizers for (linear and logistic) regression problems. These regularizers all enjoy convexity, permitting fast optimization, and they span the rang from notions of group fairness to strong individual fairness. By varying the weight on the fairness regularizer, we can compute the efficient frontier of the accuracy-fairness trade-off on any given dataset, and we measure the severity of this trade-off via a numerical quantity we call the Price of Fairness (PoF). The centerpiece of our results is an extensive comparative study of the PoF across six different datasets in which fairness is a primary consideration.

Scalable Fair Clustering

Abstract: We study the fair variant of the classic $k$-median problem introduced by Chierichetti et al. [2017]. In the standard $k$-median problem, given an input pointset $P$, the goal is to find $k$ centers $C$ and assign each input point to one of the centers in $C$ such that the average distance of points to their cluster center is minimized. In the fair variant of $k$-median, the points are colored, and the goal is to minimize the same average distance objective while ensuring that all clusters have an "approximately equal" number of points of each color. Chierichetti et al. proposed a two-phase algorithm for fair $k$-clustering. In the first step, the pointset is partitioned into subsets called fairlets that satisfy the fairness requirement and approximately preserve the $k$-median objective. In the second step, fairlets are merged into $k$ clusters by one of the existing $k$-median algorithms. The running time of this algorithm is dominated by the first step, which takes super-quadratic time. In this paper, we present a practical approximate fairlet decomposition algorithm that runs in nearly linear time. Our algorithm additionally allows for finer control over the balance of resulting clusters than the original work. We complement our theoretical bounds with empirical evaluation.

InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity

Abstract: We demonstrate an approach to face attribute detection that retains or improves attribute detection accuracy across gender and race subgroups by learning demographic information prior to learning the attribute detection task. The system, which we call InclusiveFaceNet, detects face attributes by transferring race and gender representations learned from a held-out dataset of public race and gender identities. Leveraging learned demographic representations while withholding demographic inference from the downstream face attribute detection task preserves potential users' demographic privacy while resulting in some of the best reported numbers to date on attribute detection in the Faces of the World and CelebA datasets.

Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems

Abstract: Automatic machine learning systems can inadvertently accentuate and perpetuate inappropriate human biases. Past work on examining inappropriate biases has largely focused on just individual systems. Further, there is no benchmark dataset for examining inappropriate biases in systems. Here for the first time, we present the Equity Evaluation Corpus (EEC), which consists of 8,640 English sentences carefully chosen to tease out biases towards certain races and genders. We use the dataset to examine 219 automatic sentiment analysis systems that took part in a recent shared task, SemEval-2018 Task 1 'Affect in Tweets'. We find that several of the systems show statistically significant bias; that is, they consistently provide slightly higher sentiment intensity predictions for one race or one gender. We make the EEC freely available.

Identifying and Reducing Gender Bias in Word-Level Language Models

Abstract: Many text corpora exhibit socially problematic biases, which can be propagated or amplified in the models trained on such data. For example, doctor cooccurs more frequently with male pronouns than female pronouns. In this study we (i) propose a metric to measure gender bias; (ii) measure bias in a text corpus and the text generated from a recurrent neural network language model trained on the text corpus; (iii) propose a regularization loss term for the language model that minimizes the projection of encoder-trained embeddings onto an embedding subspace that encodes gender; (iv) finally, evaluate efficacy of our proposed method on reducing gender bias. We find this regularization method to be effective in reducing gender bias up to an optimal weight assigned to the loss term, beyond which the model becomes unstable as the perplexity increases. We replicate this study on three training corpora---Penn Treebank, WikiText-2, and CNN/Daily Mail---resulting in similar conclusions.

Getting Gender Right in Neural Machine Translation

Abstract: Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996). One such difference is related to the way gender is expressed in a language. Saying “I am happy” in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either “Je suis heureux”, for a male speaker or “Je suis heureuse” for a female one. Apart from morphological agreement, demographic factors (gender, age, etc.) also influence our use of language in terms of word choices or syntactic constructions (Tannen, 1991; Pennebaker et al., 2003). We integrate gender information into NMT systems. Our contribution is two-fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.

Does Gender Matter? Towards Fairness in Dialogue Systems

Abstract: Recently there are increasing concerns about the fairness of Artificial Intelligence (AI) in real-world applications such as computer vision and recommendations. For example, recognition algorithms in computer vision are unfair to black people such as poorly detecting their faces and inappropriately identifying them as "gorillas". As one crucial application of AI, dialogue systems have been extensively applied in our society. They are usually built with real human conversational data; thus they could inherit some fairness issues which are held in the real world. However, the fairness of dialogue systems has not been well investigated. In this paper, we perform a pioneering study about the fairness issues in dialogue systems. In particular, we construct a benchmark dataset and propose quantitative measures to understand fairness in dialogue models. Our studies demonstrate that popular dialogue models show significant prejudice towards different genders and races. Besides, to mitigate the bias in dialogue systems, we propose two simple but effective debiasing methods. Experiments show that our methods can reduce the bias in dialogue systems significantly. The dataset and the implementation are released to foster fairness research in dialogue systems.

Compositional Fairness Constraints for Graph Embeddings

Abstract: Learning high-quality node embeddings is a key building block for machine learning models that operate on graph data, such as social networks and recommender systems. However, existing graph embedding techniques are unable to cope with fairness constraints, e.g., ensuring that the learned representations do not correlate with certain attributes, such as age or gender. Here, we introduce an adversarial framework to enforce fairness constraints on graph embeddings. Our approach is compositional---meaning that it can flexibly accommodate different combinations of fairness constraints during inference. For instance, in the context of social recommendations, our framework would allow one user to request that their recommendations are invariant to both their age and gender, while also allowing another user to request invariance to just their age. Experiments on standard knowledge graph and recommender system benchmarks highlight the utility of our proposed framework.

Say No to the Discrimination: Learning Fair Graph Neural Networks with Limited Sensitive Attribute Information

Abstract: Graph neural networks (GNNs) have shown great power in modeling graph structured data. However, similar to other machine learning models, GNNs may make predictions biased on protected sensitive attributes, e.g., skin color and gender. Because machine learning algorithms including GNNs are trained to reflect the distribution of the training data which often contains historical bias towards sensitive attributes. In addition, the discrimination in GNNs can be magnified by graph structures and the message-passing mechanism. As a result, the applications of GNNs in sensitive domains such as crime rate prediction would be largely limited. Though extensive studies of fair classification have been conducted on i.i.d data, methods to address the problem of discrimination on non-i.i.d data are rather limited. Furthermore, the practical scenario of sparse annotations in sensitive attributes is rarely considered in existing works. Therefore, we study the novel and important problem of learning fair GNNs with limited sensitive attribute information. FairGNN is proposed to eliminate the bias of GNNs whilst maintaining high node classification accuracy by leveraging graph structures and limited sensitive information. Our theoretical analysis shows that FairGNN can ensure the fairness of GNNs under mild conditions given limited nodes with known sensitive attributes. Extensive experiments on real-world datasets also demonstrate the effectiveness of FairGNN in debiasing and keeping high accuracy.

A summary of bias detection works in different data domains.

In this subsection, we summarize the studies regarding bias and fairness issues in real-world AI systems in different tasks. We introduce the works following the order of different data domains, including tabular data, images, texts, audios, and graphs. For each domain, we describe several representative tasks and present how AI systems can be biased on these tasks. A summary of the representative works can be found in Table {{table:e0765dc8-0824-4dda-a30e-5991e66fd5c7}} .


Comprehensive Identification of Single Nucleotide Polymorphisms Associated with Beta-lactam Resistance within Pneumococcal Mosaic Genes

Abstract: Traditional genetic association studies are very difficult in bacteria, as the generally limited recombination leads to large linked haplotype blocks, confounding the identification of causative variants. Beta-lactam antibiotic resistance in Streptococcus pneumoniae arises readily as the bacteria can quickly incorporate DNA fragments encompassing variants that make the transformed strains resistant. However, the causative mutations themselves are embedded within larger recombined blocks, and previous studies have only analysed a limited number of isolates, leading to the description of “mosaic genes” as being responsible for resistance. By comparing a large number of genomes of beta-lactam susceptible and non-susceptible strains, the high frequency of recombination should break up these haplotype blocks and allow the use of genetic association approaches to identify individual causative variants. Here, we performed a genome-wide association study to identify single nucleotide polymorphisms (SNPs) and indels that could confer beta-lactam non-susceptibility using 3,085 Thai and 616 USA pneumococcal isolates as independent datasets for the variant discovery. The large sample sizes allowed us to narrow the source of beta-lactam non-susceptibility from long recombinant fragments down to much smaller loci comprised of discrete or linked SNPs. While some loci appear to be universal resistance determinants, contributing equally to non-susceptibility for at least two classes of beta-lactam antibiotics, some play a larger role in resistance to particular antibiotics. All of the identified loci have a highly non-uniform distribution in the populations. They are enriched not only in vaccine-targeted, but also non-vaccine-targeted lineages, which may raise clinical concerns. Identification of single nucleotide polymorphisms underlying resistance will be essential for future use of genome sequencing to predict antibiotic sensitivity in clinical microbiology.

Systematic longitudinal survey of invasive Escherichia coli in England demonstrates a stable population structure only transiently disturbed by the emergence of ST131

Abstract: Escherichia coli associated with urinary tract infections and bacteremia has been intensively investigated, including recent work focusing on the virulent, globally disseminated, multidrug-resistant lineage ST131. To contextualize ST131 within the broader E. coli population associated with disease, we used genomics to analyze a systematic 11-yr hospital-based survey of E. coli associated with bacteremia using isolates collected from across England by the British Society for Antimicrobial Chemotherapy and from the Cambridge University Hospitals NHS Foundation Trust. Population dynamics analysis of the most successful lineages identified the emergence of ST131 and ST69 and their establishment as two of the five most common lineages along with ST73, ST95, and ST12. The most frequently identified lineage was ST73. Compared to ST131, ST73 was susceptible to most antibiotics, indicating that multidrug resistance was not the dominant reason for prevalence of E. coli lineages in this population. Temporal phylogenetic analysis of the emergence of ST69 and ST131 identified differences in the dynamics of emergence and showed that expansion of ST131 in this population was not driven by sequential emergence of increasingly resistant subclades. We showed that over time, the E. coli population was only transiently disturbed by the introduction of new lineages before a new equilibrium was rapidly achieved. Together, these findings suggest that the frequency of E. coli lineages in invasive disease is driven by negative frequency-dependent selection occurring outside of the hospital, most probably in the commensal niche, and that drug resistance is not a primary determinant of success in this niche.

Summary of test datasets

We investigate performance of the different methods using four public bacterial datasets suitable for GWAS and simulated phenotypes based on the genetic architecture present in the data. We focus on estimating heritability of the antibiotic resistance phenotypes. Table {{table:865fd275-d924-419f-8b56-bb3ff51484ef}}  summarizes our test datasets. The heritability of the antibiotic resistance phenotype is expected to be high, meaning that the variability stems primarily from the observed genetic differences among these bacteria.


Zero-Shot Relation Extraction via Reading Comprehension

Abstract: We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training examples. Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relation types with high accuracy, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels, setting the bar for future work on this task.

Locating and Editing Factual Associations in GPT

Abstract: We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at https://rome.baulab.info/

Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark

Abstract: Recent model editing techniques promise to mitigate the problem of memorizing false or outdated associations during LLM training. However, we show that these techniques can introduce large unwanted side effects which are not detected by existing specificity benchmarks. We extend the existing CounterFact benchmark to include a dynamic component and dub our benchmark CounterFact+. Additionally, we extend the metrics used for measuring specificity by a principled KL divergence-based metric. We use this improved benchmark to evaluate recent model editing techniques and find that they suffer from low specificity. Our findings highlight the need for improved specificity benchmarks that identify and prevent unwanted side effects.

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

Abstract: The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option. This has recently given rise to a range of techniques for injecting new facts through updating model weights. Current evaluation paradigms are extremely limited, mainly validating the recall of edited facts, but changing one fact should cause rippling changes to the model's related beliefs. If we edit the UK Prime Minister to now be Rishi Sunak, then we should get a different answer to Who is married to the British Prime Minister? In this work, we present a benchmark, MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. While we find that current knowledge-editing approaches can recall edited facts accurately, they fail catastrophically on the constructed multi-hop questions. We thus propose a simple memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers that are consistent with the edited facts. While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (up to 175B) and outperforms previous model editors by a large margin.

Evaluating the Ripple Effects of Knowledge Editing in Language Models

Abstract: Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple effect'' in the form of additional facts that the model needs to update (e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we propose a novel set of evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct RippleEdits, a diagnostic benchmark of 5K factual edits, capturing a variety of types of ripple effects. We evaluate prominent editing methods on RippleEdits, showing that current methods fail to introduce consistent changes in the model's knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.

Eva-KELLM: A New Benchmark for Evaluating Knowledge Editing of LLMs

Abstract: Large language models (LLMs) possess a wealth of knowledge encoded in their parameters. However, this knowledge may become outdated or unsuitable over time. As a result, there has been a growing interest in knowledge editing for LLMs and evaluating its effectiveness. Existing studies primarily focus on knowledge editing using factual triplets, which not only incur high costs for collection but also struggle to express complex facts. Furthermore, these studies are often limited in their evaluation perspectives. In this paper, we propose Eva-KELLM, a new benchmark for evaluating knowledge editing of LLMs. This benchmark includes an evaluation framework and a corresponding dataset. Under our framework, we first ask the LLM to perform knowledge editing using raw documents, which provides a more convenient and universal approach compared to using factual triplets. We then evaluate the updated LLM from multiple perspectives. In addition to assessing the effectiveness of knowledge editing and the retention of unrelated knowledge from conventional studies, we further test the LLM's ability in two aspects: 1) Reasoning with the altered knowledge, aiming for the LLM to genuinely learn the altered knowledge instead of simply memorizing it. 2) Cross-lingual knowledge transfer, where the LLM updated with raw documents in one language should be capable of handling queries from another language. To facilitate further research, we construct and release the corresponding dataset. Using this benchmark, we investigate the effectiveness of several commonly-used knowledge editing methods. Experimental results indicate that the current methods for knowledge editing using raw documents are not effective in yielding satisfactory results, particularly when it comes to reasoning with altered knowledge and cross-lingual knowledge transfer.

Comparison of knowledge editing evaluation benchmarks, including language, size and evaluation.

After editing the input and model, assessing the extent of knowledge integration can be accomplished by scrutinizing the output. This subsection will primarily introduce the characteristics of model evaluation and provide an overview of general benchmarks for knowledge editing in Table {{table:6d0d8297-f3bc-49df-80a6-63ec4d885eeb}} .


Attention is All you Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Scaling Neural Machine Translation

Abstract: Sequence to sequence learning models still require several days to reach state of the art performance on large benchmark datasets using a single machine. This paper shows that reduced precision and large batch training can speedup training by nearly 5x on a single 8-GPU machine with careful tuning and implementation. On WMT’14 English-German translation, we match the accuracy of Vaswani et al. (2017) in under 5 hours when training on 8 GPUs and we obtain a new state of the art of 29.3 BLEU after training for 85 minutes on 128 GPUs. We further improve these results to 29.8 BLEU by training on the much larger Paracrawl dataset. On the WMT’14 English-French task, we obtain a state-of-the-art BLEU of 43.2 in 8.5 hours on 128 GPUs.

Pay Less Attention with Lightweight and Dynamic Convolutions

Abstract: Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

Scaling Neural Machine Translation

Scaling Neural Machine Translation

Results on WMT en-de Machine Translation (newstest2014 test set)

Similarly, as shown in Table {{table:894387d8-44df-4f69-b94d-7059f03c2e42}}  and Table {{table:bf7d70ac-a706-449d-be5a-80484d424f02}} , applying LayerDrop to Transformers on text generation tasks such as neural machine translation, summarization, and long form question answering also boosts performance for all tasks.
In these experiments, we take the Transformer architectures that are state-the-art and train them with LayerDrop.
In neural machine translation on newstest2014, our 12 encoder layer Transformer model with LayerDrop further improves the state of the art, reaching {{formula:035b2388-70a3-4b07-86e3-b8bf5c044137}}  BLEU. In comparison, a standard Transformer trained without LayerDrop diverges with 12 encoder layers. This is a known problem, and techniques such as improved initialization could be used to maintain stability {{cite:cb508e7}}, {{cite:9c084e5}}, but are out of the scope of this work. Similar results are seen in summarization.


Gender differences in research productivity: a bibliometric analysis of the Italian academic system

Abstract: The literature dedicated to analysis of the difference in research productivity between the sexes tends to agree in indicating better performance for men. This study enters in the vein of work on the subject. Through bibliometric examination of the entire population of research personnel working in the scientific-technological disciplines of Italian university system, it confirms the presence of significant differences in productivity between men and women. But such differences result as being smaller than reported in a large part of the literature, confirming an ongoing tendency towards decline, and are also seen as more noticeable for quantitative performance indicators than other indicators. The gap between the sexes presents important sectorial specificities. In spite of the generally better performance of men, it can not be ignored that there are a significant number of scientific sectors in which the performance of women does not result as inferior.

Just a Matter of Time? Women’s Career Advancement in Neo-Liberal Academia. An Analysis of Recruitment Trends in Italian Universities

Abstract: Recently, the Italian higher education system has experienced two profound changes: the strong feminization of its academic staff and the implementation of market-based reforms aimed at fostering cost efficiency and economic productivity. Such reforms include the reshaping of the academic career ladder envisaged by the last university reform, the so called Gelmini reform (law 240/2010), and the adoption of a performance-based funding system. Both elements occurred in parallel with a strong cut in turnover. By accessing unique data on recruitment covering the last two decades, which were provided by the Italian Ministry of Education, University, and Research’s statistical office, this study aims at investigating these changes from a gendered perspective. More specifically, it firstly aims at analyzing if the feminization of the academic staff is due to an effective improvement of gender equality in recruitment or, rather, to demographic dynamics; secondly, it investigates to what extent the recent neo-liberal transformations, and more specifically the reshaping of the career structure combined with the limitations on hiring, has had any implications in terms of women’s recruitment and advancement. The results suggest that the road to gender equality is extremely slow and non-linear. The introduction, with the Gelmini reform, of the new fixed-term assistant professor has tightened female access to the tenure track. Moreover, female recruitment remained substantially unchanged over the period among associate and full professors, thus suggesting that the feminization of the academic staff is not due to an effective improvement of gender equality in recruitment, but also to demographic dynamics, such as the retirement of men who are concentrated in the older cohorts.

Summary of the Literature Review.

Table {{table:e061f3a9-1b2f-4586-9e18-bc84b320ef05}}  summarises such features for each paper. Note that papers with the same features have been grouped in the same row.