A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

Abstract: We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.

1.5 billion words Arabic Corpus

Abstract: This study is an attempt to build a contemporary linguistic corpus for Arabic language. The corpus produced, is a text corpus includes more than five million newspaper articles. It contains over a billion and a half words in total, out of which, there is about three million unique words. The data were collected from newspaper articles in ten major news sources from eight Arabic countries, over a period of fourteen years. The corpus was encoded with two types of encoding, namely: UTF-8, and Windows CP-1256. Also it was marked with two mark-up languages, namely: SGML, and XML.

CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Abstract: Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia.

Shamela: A Large-Scale Historical Arabic Corpus

Abstract: Arabic is a widely-spoken language with a rich and long history spanning more than fourteen centuries. Yet existing Arabic corpora largely focus on the modern period or lack sufficient diachronic information. We develop a large-scale, historical corpus of Arabic of about 1 billion words from diverse periods of time. We clean this corpus, process it with a morphological analyzer, and enhance it by detecting parallel passages and automatically dating undated texts. We demonstrate its utility with selected case-studies in which we show its application to the digital humanities.

Overview of the data-sources used to develop the proposed corpus

The proposed Arabic corpus is composed of 22 constituent data-sources which are listed in Table {{table:ae2a22a1-7627-4a3c-a32d-d9d41b3a4492}}  and their statistics are shown in Table {{table:1b8e7df7-03cb-44f3-8b73-43e3a6407440}} . The sources belong to news, academic, social, religious, cultural and other related domains. Those sources represent a good proportion of most of the commonly used Arabic dialects including MEGLN along with CA.


Masked Face Recognition Dataset and Application

Abstract: In order to effectively prevent the spread of COVID-19 virus, almost everyone wears a mask during coronavirus epidemic. This almost makes conventional facial recognition technology ineffective in many cases, such as community access control, face access control, facial attendance, facial security checks at train stations, etc. Therefore, it is very urgent to improve the recognition performance of the existing face recognition technology on the masked faces. Most current advanced face recognition approaches are designed based on deep learning, which depend on a large number of face samples. However, at present, there are no publicly available masked face recognition datasets. To this end, this work proposes three types of masked face datasets, including Masked Face Detection Dataset (MFDD), Real-world Masked Face Recognition Dataset (RMFRD) and Simulated Masked Face Recognition Dataset (SMFRD). Among them, to the best of our knowledge, RMFRD is currently theworld's largest real-world masked face dataset. These datasets are freely available to industry and academia, based on which various applications on masked faces can be developed. The multi-granularity masked face recognition model we developed achieves 95% accuracy, exceeding the results reported by the industry. Our datasets are available at: https://github.com/X-zhangyang/Real-World-Masked-Face-Dataset.

Real-Time Face Mask Detection Method Based on YOLOv3

Abstract: The rapid outbreak of COVID-19 has caused serious harm and infected tens of millions of people worldwide. Since there is no specific treatment, wearing masks has become an effective method to prevent the transmission of COVID-19 and is required in most public areas, which has also led to a growing demand for automatic real-time mask detection services to replace manual reminding. However, few studies on face mask detection are being conducted. It is urgent to improve the performance of mask detectors. In this paper, we proposed the Properly Wearing Masked Face Detection Dataset (PWMFD), which included 9205 images of mask wearing samples with three categories. Moreover, we proposed Squeeze and Excitation (SE)-YOLOv3, a mask detector with relatively balanced effectiveness and efficiency. We integrated the attention mechanism by introducing the SE block into Darknet53 to obtain the relationships among channels so that the network can focus more on the important feature. We adopted GIoUloss, which can better describe the spatial difference between predicted and ground truth boxes to improve the stability of bounding box regression. Focal loss was utilized for solving the extreme foreground-background class imbalance. Besides, we performed corresponding image augmentation techniques to further improve the robustness of the model on the specific task. Experimental results showed that SE-YOLOv3 outperformed YOLOv3 and other state-of-the-art detectors on PWMFD and achieved a higher 8.6% mAP compared to YOLOv3 while having a comparable detection speed.

How to Correctly Detect Face-Masks for COVID-19 from Visual Information?

Abstract: The new Coronavirus disease (COVID-19) has seriously affected the world. By the end of November 2020, the global number of new coronavirus cases had already exceeded 60 million and the number of deaths 1,410,378 according to information from the World Health Organization (WHO). To limit the spread of the disease, mandatory face-mask rules are now becoming common in public settings around the world. Additionally, many public service providers require customers to wear face-masks in accordance with predefined rules (e.g., covering both mouth and nose) when using public services. These developments inspired research into automatic (computer-vision-based) techniques for face-mask detection that can help monitor public behavior and contribute towards constraining the COVID-19 pandemic. Although existing research in this area resulted in efficient techniques for face-mask detection, these usually operate under the assumption that modern face detectors provide perfect detection performance (even for masked faces) and that the main goal of the techniques is to detect the presence of face-masks only. In this study, we revisit these common assumptions and explore the following research questions: (i) How well do existing face detectors perform with masked-face images? (ii) Is it possible to detect a proper (regulation-compliant) placement of facial masks? and iii) How useful are existing face-mask detection techniques for monitoring applications during the COVID-19 pandemic? To answer these and related questions we conduct a comprehensive experimental evaluation of several recent face detectors for their performance with masked-face images. Furthermore, we investigate the usefulness of multiple off-the-shelf deep-learning models for recognizing correct face-mask placement. Finally, we design a complete pipeline for recognizing whether face-masks are worn correctly or not and compare the performance of the pipeline with standard face-mask detection models from the literature. To facilitate the study, we compile a large dataset of facial images from the publicly available MAFA and Wider Face datasets and annotate it with compliant and non-compliant labels. The annotation dataset, called Face-Mask-Label Dataset (FMLD), is made publicly available to the research community.

Available Websites of Open Source Datasets.

Dey et al {{cite:1b0867f}} created a dataset containing 4095 images that can be obtained from the available link in Table {{table:8072f493-afb1-45e5-b5cf-6ead631c2b4b}} . Most of images have only one face. The images are selected from MFDD {{cite:01dc871}} and SMFD {{cite:8b77eb1}}. Dey's dataset consists of two classes: 1930 faces without masks and 2165 faces with masks. Head poses vary from frontal to profile. Most of scenes are simple because face regions account for large ration in the whole image. However, annotations are not provided.

In summary, detailed information for above mentioned datasets is illustrated in Table {{table:d3f69b4a-d0e0-4c89-8f44-c9b4a1ef9b3e}} . The corresponding available links are also provided in Table {{table:8072f493-afb1-45e5-b5cf-6ead631c2b4b}} . All the links had been verified to be effective before May 10, 2021.


Source-Independent Quantum Random Number Generation

Abstract: Quantum random number generators can provide genuine randomness by appealing to the fundamental principles of quantum mechanics. In general, a physical generator contains two parts---a randomness source and its readout. The source is essential to the quality of the resulting random numbers; hence, it needs to be carefully calibrated and modeled to achieve information-theoretical provable randomness. However, in practice, the source is a complicated physical system, such as a light source or an atomic ensemble, and any deviations in the real-life implementation from the theoretical model may affect the randomness of the output. To close this gap, we propose a source-independent scheme for quantum random number generation in which output randomness can be certified, even when the source is uncharacterized and untrusted. In our randomness analysis, we make no assumptions about the dimension of the source. For instance, multiphoton emissions are allowed in optical implementations. Our analysis takes into account the finite-key effect with the composable security definition. In the limit of large data size, the length of the input random seed is exponentially small compared to that of the output random bit. In addition, by modifying a quantum key distribution system, we experimentally demonstrate our scheme and achieve a randomness generation rate of over $5\times 10^3$ bit/s.

Source-Device-Independent Ultrafast Quantum Random Number Generation.

Abstract: Secure random numbers are a fundamental element of many applications in science, statistics, cryptography and more in general in security protocols. We present a method that enables the generation of high-speed unpredictable random numbers from the quadratures of an electromagnetic field without any assumption on the input state. The method allows us to eliminate the numbers that can be predicted due to the presence of classical and quantum side information. In particular, we introduce a procedure to estimate a bound on the conditional min-entropy based on the entropic uncertainty principle for position and momentum observables of infinite dimensional quantum systems. By the above method, we experimentally demonstrated the generation of secure true random bits at a rate greater than 1.7 Gbit/s.

Self-testing quantum random number generator.

Abstract: The generation of random numbers is a task of paramount importance in modern science. A central problem for both classical and quantum randomness generation is to estimate the entropy of the data generated by a given device. Here we present a protocol for self-testing quantum random number generation, in which the user can monitor the entropy in real time. Based on a few general assumptions, our protocol guarantees continuous generation of high quality randomness, without the need for a detailed characterization of the devices. Using a fully optical setup, we implement our protocol and illustrate its self-testing capacity. Our work thus provides a practical approach to quantum randomness generation in a scenario of trusted but error-prone devices.

A summary of self-testing and semi-self-testing QRNG demonstrations. MDI: measurement device independent, SI: source independent, CV: continuous variable.

In the last two decades, there have been tremendous development for all the three types of QRNG, trusted-device, self-testing, and semi-self-testing. In fact, there are commercial QRNG products available in the market. A brief summary of representative practical QRNG demonstrations that highlights the broad variety of optical QRNG is presented in Table {{table:aa50b557-8589-4b68-a101-8034cfe4e899}} . These QRNG schemes will be discussed further in Section REF  and REF . A summary of self-testing and semi-self-testing QRNG demonstrations is presented in Table {{table:b7e8b1a7-bec5-4dc6-8d17-3256b4b08b0b}} , which will be reviewed in details in Section REF  and REF .


An Empirical Study of Training Self-Supervised Vision Transformers

Abstract: This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.

Emerging Properties in Self-Supervised Vision Transformers

Abstract: In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) [16] that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder [26], multi-crop training [9], and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

Semi-ViT results with other self-pretraining techniques.

Beyond MAE, we also experiment on other self-pretraining technique, including MoCo-v3 {{cite:81db014}} and DINO {{cite:b04f3d8}}, in Table {{table:91c13d81-5964-4857-8852-575214f739ab}} . By comparing the fine-tuning results, DINO is close to MoCo-v3 for ViT-Base but much better for ViT-Small, and both of them are better than MAE for ViT-Base, suggesting that DINO could be a better self-pretraining technique for smaller scales of ViT models. On top of the strong fine-tuning baselines, the semi-supervised fine-tuning, using EMA-Teacher, still has nontrivial improvements for both DINO and MoCo-v3, e.g., 5.8 (2.1) points on 1% (10%) labels for DINO-ViT-Base. In addition, the probabilistic pseudo mixup can further improve over the EMA-Teacher, independent of the self-pretraining algorithms. And the final Semi-ViT-Base of DINO is 2.1 (0.5) points better than that of MAE on 1% (10%) labels.


Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose

Abstract: In this work we adapt multi-person pose estimation architecture to use it on edge devices. We follow the bottom-up approach from OpenPose, the winner of COCO 2016 Keypoints Challenge, because of its decent quality and robustness to number of people inside the frame. With proposed network design and optimized post-processing code the full solution runs at 28 frames per second (fps) on Intel$\unicode{xAE}$ NUC 6i7KYB mini PC and 26 fps on Core$^{TM}$ i7-6850K CPU. The network model has 4.1M parameters and 9 billions floating-point operations (GFLOPs) complexity, which is just ~15% of the baseline 2-stage OpenPose with almost the same quality. The code and model are available as a part of Intel$\unicode{xAE}$ OpenVINO$^{TM}$ Toolkit.

Whole-Body Human Pose Estimation in the Wild

Abstract: This paper investigates the task of 2D human whole-body pose estimation, which aims to localize dense landmarks on the entire human body including face, hands, body, and feet. As existing datasets do not have whole-body annotations, previous methods have to assemble different deep models trained independently on different datasets of the human face, hand, and body, struggling with dataset biases and large model complexity. To fill in this blank, we introduce COCO-WholeBody which extends COCO dataset with whole-body annotations. To our best knowledge, it is the first benchmark that has manual annotations on the entire human body, including 133 dense landmarks with 68 on the face, 42 on hands and 23 on the body and feet. A single-network model, named ZoomNet, is devised to take into account the hierarchical structure of the full human body to solve the scale variation of different body parts of the same person. ZoomNet is able to significantly outperform existing methods on the proposed COCO-WholeBody dataset. Extensive experiments show that COCO-WholeBody not only can be used to train deep models from scratch for whole-body pose estimation but also can serve as a powerful pre-training dataset for many different tasks such as facial landmark detection and hand keypoint estimation. The dataset is publicly available at https://github.com/jin-s13/COCO-WholeBody.

Standards for Different Types of HPE

Standards in HPE refer to the number of landmarks used to carry out the estimation, meaning the number of human joints or predefined landmarks needed to be localized. This number varies by the type of pose estimation being done and by the method itself. Table {{table:d649ed98-b043-4fcd-b35f-1f9a1c360fda}}  lists a few methods for body, face, and hand pose estimation each, as well as the corresponding number of landmarks they used. See Figure {{figure:da00f9c2-17b4-4011-83f6-d8eaceca8650}}  for examples of the same type of pose estimation that vary in the number of the landmarks used.

As shown in Table {{table:d649ed98-b043-4fcd-b35f-1f9a1c360fda}} , most of the methods for face pose estimation used 68 landmarks because finding algorithms besides MediaPipe Face Mesh that used a number of face landmarks other than 68 proved to be unsuccessful. However, if methods used one of the face datasets for training that does not label its images using 68 landmarks, then there would have been resulting standards that had a number of landmarks other than 68 besides the MediaPipe Face Mesh method that used 468. Similarly, all the methods for hand pose estimation used 21 landmarks per hand because finding algorithms that used a number of hand landmarks other than 21 per hand proved to be unsuccessful as well. If methods used one of the datasets for training that does not label its images using 21 landmarks per hand, then there would have been resulting standards that had a number of landmarks other than 21 per hand. The different types of datasets in HPE are covered in Section .


Double energy limitations at both the UAV and ground sensors

Uplink cooperative NOMA for A2G interference cancelation

Interference-aware UAV trajectory design

Energy-Constrained UAV Data Collection Systems: NOMA and OMA

Abstract: This paper investigates unmanned aerial vehicle (UAV) data collection systems with different multiple access schemes, where a rotary-wing UAV is dispatched to collect data from multiple ground nodes (GNs). Our goal is to maximize the minimum UAV data collection throughput from GNs for both orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) transmission, subject to the energy budgets at both the UAV and GNs, namely double energy limitations. 1) For OMA, we propose an efficient algorithm by invoking alternating optimization (AO) method, where each subproblem is alternately solved by applying successive convex approximation (SCA) technique. 2) For NOMA, we first handle subproblems with fixed decoding order using SCA technique. Then, we develop a penalty-based algorithm to solve the decoding order design subproblem. Numerical results show that: i) The proposed algorithms are capable of improving the max-min throughput performance compared with other benchmark schemes; and ii) NOMA yields a higher performance gain than OMA when GNs have sufficient energy.

Uplink Cooperative NOMA for Cellular-Connected UAV

Abstract: Aerial–ground interference mitigation is a challenging issue in the emerging cellular-connected unmanned aerial vehicle (UAV) communications. Due to the strong line-of-sight (LoS) air-to-ground (A2G) channels, the UAV may impose/suffer more severe uplink/downlink interference to/from the cellular base stations (BSs) as compared to the ground users. To tackle this challenge, we propose in this paper to apply the non-orthogonal multiple access (NOMA) technique to the uplink communication from a UAV to cellular BSs, under spectrum sharing with the existing ground users. However, for our considered system, traditional NOMA with only local interference cancellation (IC) at individual BSs, termed non-cooperative NOMA, may provide very limited gain compared to the orthogonal multiple access (OMA). This is because there are a large number of co-channel BSs due to the LoS A2G channels, and thus, the rate performance of the UAV is severely limited by the BS with the worst channel condition with the UAV. To mitigate the UAV's uplink interference without significantly compromising its achievable rate, a new cooperative NOMA scheme is proposed in this paper by exploiting the existing backhaul links among BSs. Specifically, some BSs with better channel conditions are selected to decode the UAV's signals first, and then forward the decoded signals to their backhaul-connected BSs for IC. To investigate the optimal design of cooperative NOMA and air-ground performance tradeoff, we maximize the weighted sum-rate of the UAV and ground users by jointly optimizing the UAV's rate and power allocations over multiple resource blocks as well as their associated BSs. However, this problem is difficult to be solved optimally. To obtain useful insights, we first consider two special cases with egoistic and altruistic transmission strategies of the UAV, respectively, and solve their corresponding problems optimally. Next, we consider the general case and propose an efficient suboptimal solution by applying the alternating optimization and successive convex approximation techniques. Numerical results show that the proposed cooperative NOMA scheme yields significant throughput gains than those by the traditional OMA as well as the non-cooperative NOMA benchmark.

Non-Orthogonal Multiple Access for Air-to-Ground Communication

Abstract: This paper investigates ground-aerial uplink non-orthogonal multiple access (NOMA) cellular networks. A rotary-wing unmanned aerial vehicle (UAV) user and multiple ground users (GUEs) are served by ground base stations (GBSs) by utilizing the uplink NOMA protocol. The UAV is dispatched to upload specific information bits to each target GBSs. Specifically, our goal is to minimize the UAV mission completion time by jointly optimizing the UAV trajectory and UAV-GBS association order while taking into account the UAV’s interference to non-associated GBSs. The formulated problem is a mixed integer non-convex problem and involves infinite variables. To tackle this problem, we efficiently check the feasibility of the formulated problem by utilizing graph theory and topology theory. Next, we prove that the optimal UAV trajectory needs to satisfy the fly-hover-fly structure. With this insight, we first design an efficient solution with predefined hovering locations by leveraging graph theory techniques. Furthermore, we propose an iterative UAV trajectory design by applying successive convex approximation (SCA) technique, which is guaranteed to coverage to a locally optimal solution. We demonstrate that the two proposed designs exhibit polynomial time complexity. Finally, numerical results show that: 1) the SCA based design outperforms the fly-hover-fly based design; 2) the UAV mission completion time is significantly minimized with proposed NOMA schemes compared with the orthogonal multiple access (OMA) scheme; 3) the increase of GUEs’ quality of service (QoS) requirements will increase the UAV mission completion time.

Summary of existing works on the application of NOMA in UAV-aided communications.

With the rapid development of manufacturing technology and the continuous reduction of costs, UAV-aided communications have been recognized as an emerging technique for next generation wireless networks due to the resulting high mobility and flexibility {{cite:ae59d64}}, {{cite:262f551}}, {{cite:0d574c6}}. Equipped with communication devices, UAVs can act as BSs, relays, and users in wireless networks and facilitate a wide range of applications (e.g., wireless service recovery for emergencies, traffic offloading for temporary hotspots, environment monitoring, and cargo delivery) by exploiting their unique line-of-sight (LOS) dominated air-to-ground (A2G) channel. However, these applications also present significant challenges, such as the management of the limited available wireless resources and the cancelation of the A2G interference. To cope with these issues, there are two promising approaches of employing NOMA in UAV-aided communications, namely, UAV-BS-enabled NOMA and NOMA-assisted cellular-connected UAVs, which are depicted in Fig. {{figure:3b694bb1-2c3e-4d26-9411-493842aae764}} . A summary of the corresponding research contributions is listed in Table {{table:93629a3c-ef56-4229-9a08-4332ad0ca0d2}} .


Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Abstract: Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

TableQA: a Large-Scale Chinese Text-to-SQL Dataset for Table-Aware SQL Generation

Abstract: Parsing natural language to corresponding SQL (NL2SQL) with data driven approaches like deep neural networks attracts much attention in recent years. Existing NL2SQL datasets assume that condition values should appear exactly in natural language questions and the queries are answerable given the table. However, these assumptions may fail in practical scenarios, because user may use different expressions for the same content in the table, and query information outside the table without the full picture of contents in table. Therefore we present TableQA, a large-scale cross-domain Natural Language to SQL dataset in Chinese language consisting 64,891 questions and 20,311 unique SQL queries on over 6,000 tables. Different from exisiting NL2SQL datasets, TableQA requires to generalize well not only to SQL skeletons of different questions and table schemas, but also to the various expressions for condition values. Experiment results show that the state-of-the-art model with 95.1% condition value accuracy on WikiSQL only gets 46.8% condition value accuracy and 43.0% logic form accuracy on TableQA, indicating the proposed dataset is challenging and necessary to handle. Two table-aware approaches are proposed to alleviate the problem, the end-to-end approaches obtains 51.3% and 47.4% accuracy on the condition value and logic form tasks, with improvement of 4.7% and 3.4% respectively.

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

Abstract: We present Spider, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables covering 138 different domains. We define a new complex and cross-domain semantic parsing and text-to-SQL task so that different complicated SQL queries and databases appear in train and test sets. In this way, the task requires the model to generalize well to both new SQL queries and new database schemas. Therefore, Spider is distinct from most of the previous semantic parsing tasks because they all use a single database and have the exact same program in the train set and the test set. We experiment with various state-of-the-art models and the best model achieves only 9.7% exact matching accuracy on a database split setting. This shows that Spider presents a strong challenge for future research. Our dataset and task with the most recent updates are publicly available at https://yale-lily.github.io/seq2sql/spider.

SParC: Cross-Domain Semantic Parsing in Context

Abstract: We present SParC, a dataset for cross-domainSemanticParsing inContext that consists of 4,298 coherent question sequences (12k+ individual questions annotated with SQL queries). It is obtained from controlled user interactions with 200 complex databases over 138 domains. We provide an in-depth analysis of SParC and show that it introduces new challenges compared to existing datasets. SParC demonstrates complex contextual dependencies, (2) has greater semantic diversity, and (3) requires generalization to unseen domains due to its cross-domain nature and the unseen databases at test time. We experiment with two state-of-the-art text-to-SQL models adapted to the context-dependent, cross-domain setup. The best model obtains an exact match accuracy of 20.2% over all questions and less than10% over all interaction sequences, indicating that the cross-domain setting and the con-textual phenomena of the dataset present significant challenges for future research. The dataset, baselines, and leaderboard are released at https://yale-lily.github.io/sparc.

Comparisons of seven text-to-SQL datasets.MD (multi-domain), MTa (multi-table) and MTu (multi-turn) denote the research focuses of these datsets and # SQL Len. denotes the complexity of predicted SQL.

We compare UniSAr with previous task-specific parsers using seven popular text-to-SQL benchmark datasets.
The dataset statistics are shown in Table {{table:5545e8ef-093c-43e0-a11c-61f12a23a5c2}}  (Appendix ).
To systemically compare our unified parser with previous task-specific models, we divided the dataset into three group to study the effectiveness of UniSAr:
(1) Multi-DomainUniSAr still shows a powerful performance in an internal single-domain dataset. Note that for Spider, CoSQL, SparC and DuSQL, we conduct experiments on dev set since the test set is not publicly available.: WikiSQL {{cite:603d00b}} and TableQA {{cite:b16a5c1}};
(2) Multi-Table: Spider {{cite:90cf269}} and DuSQL {{cite:7470440}};
(3) Multi-Turn:
CoSQL {{cite:5546ac5}}, SParC {{cite:8850035}} and Chase {{cite:8072cce}}.

Table {{table:5545e8ef-093c-43e0-a11c-61f12a23a5c2}}  lists the dataset we used.
We divided the dataset into three group to study the effectiveness of UniSAr:
(1) Multi-Domain: WikiSQL {{cite:603d00b}} and TableQA {{cite:b16a5c1}};
(2) Multi-Table: Spider {{cite:90cf269}} and DuSQL {{cite:7470440}};
(3) Multi-Turn:
CoSQL {{cite:5546ac5}}, SParC {{cite:8850035}} and Chase {{cite:8072cce}}.


Android malware detection on real devices

DL-Droid: Deep learning based android malware detection using real devices

Abstract: The Android operating system has been the most popular for smartphones and tablets since 2012. This popularity has led to a rapid raise of Android malware in recent years. The sophistication of Android malware obfuscation and detection avoidance methods have significantly improved, making many traditional malware detection methods obsolete. In this paper, we propose DL-Droid, a deep learning system to detect malicious Android applications through dynamic analysis using stateful input generation. Experiments performed with over 30,000 applications (benign and malware) on real devices are presented. Furthermore, experiments were also conducted to compare the detection performance and code coverage of the stateful input generation method with the commonly used stateless approach using the deep learning system. Our study reveals that DL-Droid can achieve up to 97.8% detection rate (with dynamic features only) and 99.6% detection rate (with dynamic + static features) respectively which outperforms traditional machine learning techniques. Furthermore, the results highlight the significance of enhanced input generation for dynamic analysis as DL-Droid with the state-based input generation is shown to outperform the existing state-of-the-art approaches.

Android HIV: A Study of Repackaging Malware for Evading Machine-Learning Detection

Abstract: Machine learning based solutions have been successfully employed for automatic detection of malware on Android. However, machine learning models lack robustness to adversarial examples, which are crafted by adding carefully chosen perturbations to the normal inputs. So far, the adversarial examples can only deceive detectors that rely on syntactic features (e.g., requested permissions, API calls, etc), and the perturbations can only be implemented by simply modifying application's manifest. While recent Android malware detectors rely more on semantic features from Dalvik bytecode rather than manifest, existing attacking/defending methods are no longer effective. In this paper, we introduce a new attacking method that generates adversarial examples of Android malware and evades being detected by the current models. To this end, we propose a method of applying optimal perturbations onto Android APK that can successfully deceive the machine learning detectors. We develop an automated tool to generate the adversarial examples without human intervention. In contrast to existing works, the adversarial examples crafted by our method can also deceive recent machine learning based detectors that rely on semantic features such as control-flow-graph. The perturbations can also be implemented directly onto APK's Dalvik bytecode rather than Android manifest to evade from recent detectors. We demonstrate our attack on two state-of-the-art Android malware detection schemes, MaMaDroid and Drebin. Our results show that the malware detection rates decreased from 96% to 0% in MaMaDroid, and from 97% to 0% in Drebin, with just a small number of codes to be inserted into the APK.

Top cited papers with Minimum 25 citations per year

In order to locate current research concerns, we sorted collected studies according to the number of citations per year that are based on the citation counts from Google Scholar before Nov 2020. Table {{table:274b1516-8665-422c-90a8-fa7119405d19}}  mentions the primary publications with minimum 25 citations per year. It is apparent from this table that integrating deep learning techniques into Android malware detection is still the main focus. But specifically, these research studies are not limited to introducing a single deep learning algorithm to detect malicious applications, and specific solutions including hybrid models and on-device detection have also attracted high research interest. Grosse et al. {{cite:f5237df}}, {{cite:17b8e36}} has the largest number of citations, and one reason for the high citation counts is that this work investigates the viability of adversarial attack techniques against neural networks in this field, which sets the foundations for future work. Another work {{cite:e5ef5ad}} published in 2019 that examines adversarial attacks and defense techniques on two state-of-the-art Android malware detection models is also a highly cited paper. These facts state that the robustness of DL-based detection models to adversarial samples is attracting more and more research attention.

Table {{table:274b1516-8665-422c-90a8-fa7119405d19}}  and {{table:4c7f907b-9d52-4c16-baa0-a6b3ca9bc522}}  further summarize the deep learning architectures used in these listed studies. Although Figure {{figure:04c68dec-039b-4157-9e29-8e9dd952448f}}  displays that CNN, RNN and MLP are widely used in Android malware defenses in recent years, MLP appears most frequently among listed important works. This is not a surprising outcome. MLP is the simplest but quintessential deep neural network, and thus researchers would tend to conduct experiments based on MLP networks first when they have a new research idea. At the same time, this finding also illustrates that the application of advanced deep learning techniques to the Android malware defenses domain is actually in a preliminary stage. In recent years, deep learning has brought impressive progress in many domains and many modern DL approaches are up-to-come, such as deep active learning, reinforcement learning, transfer learning, controllable generative models and etc. However, the application of deep learning technologies in the field of mobile security is far behind the development of deep learning itself. Research toward DL-based Android malware defenses is active currently, hence we believe more advancements are expected in the near future.


ShapeNet: An Information-Rich 3D Model Repository

Abstract: We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects. ShapeNet contains 3D models from a multitude of semantic categories and organizes them under the WordNet taxonomy. It is a collection of datasets providing many semantic annotations for each 3D model such as consistent rigid alignments, parts and bilateral symmetry planes, physical sizes, keywords, as well as other planned annotations. Annotations are made available through a public web-based interface to enable data visualization of object attributes, promote data-driven geometric analysis, and provide a large-scale quantitative benchmark for research in computer graphics and vision. At the time of this technical report, ShapeNet has indexed more than 3,000,000 models, 220,000 models out of which are classified into 3,135 categories (WordNet synsets). In this report we describe the ShapeNet effort as a whole, provide details for all currently available datasets, and summarize future plans.

Joint 2D-3D-Semantic Data for Indoor Scene Understanding

Abstract: We present a dataset of large-scale indoor spaces that provides a variety of mutually registered modalities from 2D, 2.5D and 3D domains, with instance-level semantic and geometric annotations. The dataset covers over 6,000m2 and contains over 70,000 RGB images, along with the corresponding depths, surface normals, semantic annotations, global XYZ images (all in forms of both regular and 360{\deg} equirectangular images) as well as camera information. It also includes registered raw and semantically annotated 3D meshes and point clouds. The dataset enables development of joint and cross-modal learning models and potentially unsupervised approaches utilizing the regularities present in large-scale indoor spaces. The dataset is available here: this http URL

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Abstract: Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy (~92\%). Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background. Our dataset and code are publicly available in our project page https://hkust-vgd.github.io/scanobjectnn/.

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Abstract: A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available – current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.

nuScenes: A Multimodal Dataset for Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

PCN: Point Completion Network

Abstract: Shape completion, the problem of estimating the complete geometry of objects from partial observations, lies at the core of many vision and robotics applications. In this work, we propose Point Completion Network (PCN), a novel learning-based approach for shape completion. Unlike existing shape completion methods, PCN directly operates on raw point clouds without any structural assumption (e.g. symmetry) or annotation (e.g. semantic class) about the underlying shape. It features a decoder design that enables the generation of fine-grained completions while maintaining a small number of parameters. Our experiments show that PCN produces dense, complete point clouds with realistic structures in the missing regions on inputs with various levels of incompleteness and noise, including cars from LiDAR scans in the KITTI dataset.

Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision

Abstract: We propose a CNN-based approach for 3D human body pose estimation from single RGB images that addresses the issue of limited generalizability of models trained solely on the starkly limited publicly available 3D pose data. Using only the existing 3D pose data and 2D pose data, we show state-of-the-art performance on established benchmarks through transfer of learned features, while also generalizing to in-the-wild scenes. We further introduce a new training set for human body pose estimation from monocular images of real humans that has the ground truth captured with a multi-camera marker-less motion capture system. It complements existing corpora with greater diversity in pose, human appearance, clothing, occlusion, and viewpoints, and enables an increased scope of augmentation. We also contribute a new benchmark that covers outdoor and indoor scenes, and demonstrate that our 3D pose dataset shows better in-the-wild performance than existing annotated data, which is further improved in conjunction with transfer learning from 2D pose data. All in all, we argue that the use of transfer learning of representations in tandem with algorithmic and data contributions is crucial for general 3D body pose estimation.

Common datasets for 3D vision tasks. These datasets cover a range of 3D applications: classification, segmentation, detection, completion, and pose estimation. They are collected using different sensors for indoor scenes, outdoor scenes, and objects.

Datasets play two important roles in the advancement of computer vision tasks.
First, annotations allow training deep learning models to solve challenging problems.
Second, datasets provide a basis for quantitative comparisons to measure the effectiveness of a proposed method.
Numerous datasets have been developed for 3D vision tasks.
For classification of 3D objects, ModelNet40 {{cite:e96c59d}} is widely used, as well as ScanObjectNN {{cite:85d8d4a}}.
For 3D segmentation, ShapeNet {{cite:8fc2f46}} provides part segmentation annotation, and S3DIS {{cite:5b3bb08}} and ScanNet {{cite:4fc7c14}} provide indoor scenes segmentation.
For indoor 3D object detection, SUN RGB-D {{cite:2198f03}} provides oriented bounding boxes annotations for scenes from a single RGB-D frame, and ScanNet {{cite:4fc7c14}} can also be utilized by transforming instance segmentation labels into axis aligned bounding box annotations. For 3D object detection in outdoor scenes, KITTI {{cite:3485b60}} and nuScenes {{cite:0c3855b}} are the most commonly used.
For 3D point cloud completion, Completion3D {{cite:0569627}} and PCN {{cite:ef84884}} are two of the most widely used datasets.
For 3D pose estimation, Human3.6M {{cite:b8a1ab3}} and MPI-INF-3DHP {{cite:64e3d21}} are among the most popular datasets.
The main attributes of these datasets are shown in Table {{table:3d4ec947-e36a-448a-bd5a-a4bbd7798a64}} .


Aging in CMOS transistors of neuron circuitry

Read disturb issues in ReRAM cells in a neuromorphic crossbar

Time-Dependant Dielectric Breakdown (TDDB), Bias Temperature Instability (BTI)

Incorporate bitline and wordline parasitics

A Case for Lifetime Reliability-Aware Neuromorphic Computing

Abstract: Neuromorphic computing with non-volatile memory (NVM) can significantly improve performance and lower energy consumption of machine learning tasks implemented using spike-based computations and bio-inspired learning algorithms. High voltages required to operate certain NVMs such as phase-change memory (PCM) can accelerate aging in a neuron's CMOS circuit, thereby reducing the lifetime of neuromorphic hardware. In this work, we evaluate the long-term, i.e., lifetime reliability impact of executing state-of-the-art machine learning tasks on a neuromorphic hardware, considering failure models such as negative bias temperature instability (NBTI) and time-dependent dielectric breakdown (TDDB). Based on such formulation, we show the reliability-performance trade-off obtained due to periodic relaxation of neuromorphic circuits, i.e., a stop-and-go style of neuromorphic computing.

Reliability-Performance Trade-offs in Neuromorphic Computing

Abstract: Neuromorphic architectures built with Non-Volatile Memory (NVM) can significantly improve the energy efficiency of machine learning tasks designed with Spiking Neural Networks (SNNs). A major source of voltage drop in a crossbar of these architectures are the parasitic components on the crossbar's bitlines and wordlines, which are deliberately made longer to achieve lower cost-per-bit. We observe that the parasitic voltage drops create a significant asymmetry in programming speed and reliability of NVM cells in a crossbar. Specifically, NVM cells that are on shorter current paths are faster to program but have lower endurance than those on longer current paths, and vice versa. This asymmetry in neuromorphic architectures create reliability-performance trade-offs, which can be exploited efficiently using SNN mapping techniques. In this work, we demonstrate such trade-offs using a previously-proposed SNN mapping technique with 10 workloads from contemporary machine learning tasks for a state-of-the art neuromoorphic hardware.

Improving Inference Lifetime of Neuromorphic Systems via Intelligent Synapse Mapping

Abstract: Non-Volatile Memories (NVMs) such as Resistive RAM (RRAM) are used in neuromorphic systems to implement high-density and low-power analog synaptic weights. Unfortunately, an RRAM cell can switch its state after reading its content a certain number of times. Such behavior challenges the integrity and program-once-read-many-times philosophy of implementing machine learning inference on neuromorphic systems, impacting the Quality-of-Service (QoS). Elevated temperatures and frequent usage can significantly shorten the number of times an RRAM cell can be reliably read before it becomes absolutely necessary to reprogram. We propose an architectural solution to extend the read endurance of RRAM-based neuromorphic systems. We make two key contributions. First, we formulate the read endurance of an RRAM cell as a function of the programmed synaptic weight and its activation within a machine learning workload. Second, we propose an intelligent workload mapping strategy incorporating the endurance formulation to place the synapses of a machine learning model onto the RRAM cells of the hardware. The objective is to extend the inference lifetime, defined as the number of times the model can be used to generate output (inference) before the trained weights need to be reprogrammed on the RRAM cells of the system. We evaluate our architectural solution with machine learning workloads on a cycle-accurate simulator of an RRAM-based neuromorphic system. Our results demonstrate a significant increase in inference lifetime with only a minimal performance impact.

Neuromorphic system software approaches targeting thermal and reliability optimization for platform based design.

System Software for Thermal and Reliability Optimization

Summary: Table {{table:8376c673-db18-405b-aede-18230d35d894}}  summarizes these approaches.


Multi-Differential Fairness Auditor for Black Box Classifiers

Abstract: Machine learning algorithms are increasingly involved in sensitive decision-making processes with adversarial implications on individuals. This paper presents a new tool,  mdfa that identifies the characteristics of the victims of a classifier's discrimination. We measure discrimination as a violation of multi-differential fairness.  Multi-differential fairness is a guarantee that a black box classifier's outcomes do not leak information on the sensitive attributes of a small group of individuals. We reduce the problem of identifying worst-case violations to matching distributions and predicting where sensitive attributes and classifier's outcomes coincide. We apply mdfa to a recidivism risk assessment classifier widely used in the United States and demonstrate that for individuals with little criminal history, identified African-Americans are three-times more likely to be considered at high risk of violent recidivism than similar non-African-Americans.

Dynamic fairness - Breaking vicious cycles in automatic decision making

Abstract:
In recent years, machine learning techniques have been increasingly applied in sensitive decision making processes, raising fairness concerns. Past research has shown that machine learning may reproduce and even exacerbate human bias due to biased training data or flawed model assumptions, and thus may lead to discriminatory actions. To counteract such biased models, researchers have proposed multiple mathematical definitions of fairness according to which classifiers can be optimized. However, it has also been shown that the outcomes generated by some fairness notions may be unsatisfactory.
In this contribution, we add to this research by considering decision making processes in time. We establish a theoretic model in which even perfectly accurate classifiers which adhere to almost all common fairness definitions lead to stable long-term inequalities due to vicious cycles. Only demographic parity, which enforces equal rates of positive decisions across groups, avoids these effects and establishes a virtuous cycle, which leads to perfectly accurate and fair classification in the long term.

Analyzing Bias in Sensitive Personal Information Used to Train Financial Models

Abstract: Bias in data can have unintended consequences that propagate to the design, development, and deployment of machine learning models. In the financial services sector, this can result in discrimination from certain financial instruments and services. At the same time, data privacy is of paramount importance, and recent data breaches have seen reputational damage for large institutions. Presented in this paper is a trusted model-lifecycle management platform that attempts to ensure consumer data protection, anonymization, and fairness. Specifically, we examine how datasets can be reproduced using deep learning techniques to effectively retain important statistical features in datasets whilst simultaneously protecting data privacy and enabling safe and secure sharing of sensitive personal information beyond the current state-of-practice.

A Causal Bayesian Networks Viewpoint on Fairness

Abstract: We offer a graphical interpretation of unfairness in a dataset as the presence of an unfair causal path in the causal Bayesian network representing the data-generation mechanism. We use this viewpoint to revisit the recent debate surrounding the COMPAS pretrial risk assessment tool and, more generally, to point out that fairness evaluation on a model requires careful considerations on the patterns of unfairness underlying the training data. We show that causal Bayesian networks provide us with a powerful tool to measure unfairness in a dataset and to design fair models in complex unfairness scenarios.

Fair Adversarial Gradient Tree Boosting

Abstract: Fair classification has become an important topic in machine learning research. While most bias mitigation strategies focus on neural networks, we noticed a lack of work on fair classifiers based on decision trees even though they have proven very efficient. In an up-to-date comparison of state-of-the-art classification algorithms in tabular data, tree boosting outperforms deep learning. For this reason, we have developed a novel approach of adversarial gradient tree boosting. The objective of the algorithm is to predict the output $Y$ with gradient tree boosting while minimizing the ability of an adversarial neural network to predict the sensitive attribute $S$. The approach incorporates at each iteration the gradient of the neural network directly in the gradient tree boosting. We empirically assess our approach on 4 popular data sets and compare against state-of-the-art algorithms. The results show that our algorithm achieves a higher accuracy while obtaining the same level of fairness, as measured using a set of different common fairness definitions.

Fairness in Credit Scoring: Assessment, Implementation and Profit Implications

Abstract: The rise of algorithmic decision-making has spawned much research on fair machine learning (ML). Financial institutions use ML for building risk scorecards that support a range of credit-related decisions. Yet, the literature on fair ML in credit scoring is scarce. The paper makes three contributions. First, we revisit statistical fairness criteria and examine their adequacy for credit scoring. Second, we catalog algorithmic options for incorporating fairness goals in the ML model development pipeline. Last, we empirically compare different fairness processors in a profit-oriented credit scoring context using real-world data. The empirical results substantiate the evaluation of fairness measures, identify suitable options to implement fair credit scoring, and clarify the profit-fairness trade-off in lending decisions. We find that multiple fairness criteria can be approximately satisfied at once and recommend separation as a proper criterion for measuring the fairness of a scorecard. We also find fair in-processors to deliver a good balance between profit and fairness and show that algorithmic discrimination can be reduced to a reasonable level at a relatively low cost. The codes corresponding to the paper are available on GitHub.

A Survey on Bias and Fairness in Machine Learning

Abstract: With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

Evolution and impact of bias in human and machine learning algorithm interaction

Abstract: Traditionally, machine learning algorithms relied on reliable labels from experts to build predictions. More recently however, algorithms have been receiving data from the general population in the form of labeling, annotations, etc. The result is that algorithms are subject to bias that is born from ingesting unchecked information, such as biased samples and biased labels. Furthermore, people and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data. Algorithms can also make biased predictions, leading to what is now known as algorithmic bias. On the other hand, human’s reaction to the output of machine learning methods with algorithmic bias worsen the situations by making decision based on biased information, which will probably be consumed by algorithms later. Some recent research has focused on the ethical and moral implication of machine learning algorithmic bias on society. However, most research has so far treated algorithmic bias as a static factor, which fails to capture the dynamic and iterative properties of bias. We argue that algorithmic bias interacts with humans in an iterative manner, which has a long-term effect on algorithms’ performance. For this purpose, we present an iterated-learning framework that is inspired from human language evolution to study the interaction between machine learning algorithms and humans. Our goal is to study two sources of bias that interact: the process by which people select information to label (human action); and the process by which an algorithm selects the subset of information to present to people (iterated algorithmic bias mode). We investigate three forms of iterated algorithmic bias (personalization filter, active learning, and random) and how they affect the performance of machine learning algorithms by formulating research questions about the impact of each type of bias. Based on statistical analyses of the results of several controlled experiments, we found that the three different iterated bias modes, as well as initial training data class imbalance and human action, do affect the models learned by machine learning algorithms. We also found that iterated filter bias, which is prominent in personalized user interfaces, can lead to more inequality in estimated relevance and to a limited human ability to discover relevant data. Our findings indicate that the relevance blind spot (items from the testing set whose predicted relevance probability is less than 0.5 and who thus risk being hidden from humans) amounted to 4% of all relevant items when using a content-based filter that predicts relevant items. A similar simulation using a real-life rating data set found that the same filter resulted in a blind spot size of 75% of the relevant testing set.

Fairness for Image Generation with Uncertain Sensitive Attributes

Abstract: This work tackles the issue of fairness in the context of generative procedures, such as image super-resolution, which entail different definitions from the standard classification setting. Moreover, while traditional group fairness definitions are typically defined with respect to specified protected groups -- camouflaging the fact that these groupings are artificial and carry historical and political motivations -- we emphasize that there are no ground truth identities. For instance, should South and East Asians be viewed as a single group or separate groups? Should we consider one race as a whole or further split by gender? Choosing which groups are valid and who belongs in them is an impossible dilemma and being"fair"with respect to Asians may require being"unfair"with respect to South Asians. This motivates the introduction of definitions that allow algorithms to be \emph{oblivious} to the relevant groupings. We define several intuitive notions of group fairness and study their incompatibilities and trade-offs. We show that the natural extension of demographic parity is strongly dependent on the grouping, and \emph{impossible} to achieve obliviously. On the other hand, the conceptually new definition we introduce, Conditional Proportional Representation, can be achieved obliviously through Posterior Sampling. Our experiments validate our theoretical results and achieve fair image reconstruction using state-of-the-art generative models.

Fairness via Representation Neutralization

Abstract: Existing bias mitigation methods for DNN models primarily work on learning debiased encoders. This process not only requires a lot of instance-level annotations for sensitive attributes, it also does not guarantee that all fairness sensitive information has been removed from the encoder. To address these limitations, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the classification head, even with biased representations as inputs? To this end, we propose a new mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. To this end, we leverage samples with the same ground-truth label but different sensitive attributes, and use their neutralized representations to train the classification head of the DNN model. The key idea of RNF is to discourage the classification head from capturing spurious correlation between fairness sensitive information in encoder representations with specific class labels. To address low-resource settings with no access to sensitive attribute annotations, we leverage a bias-amplified model to generate proxy annotations for sensitive attributes. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models with minimal degradation in task-specific performance.

Papers returned by the search string

This section presents and analyses the 40 selected studies, which are included in Table {{table:95f585b5-0379-4fa0-a956-b817c10fee2e}} , according to the research questions Q1 and Q2 set in the Scope Definition step.
The results are organized in five sections: Types of Bias, Identified Datasets, Mitigation techniques and models,
Technique for identification of the sensitive attribute and Fairness metrics. Those sections represent fundamental aspects on the discussion of bias and fairness.


DAiSEE: Towards User Engagement Recognition in the Wild

Abstract: We introduce DAiSEE, the first multi-label video classification dataset comprising of 9068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration in the wild. The dataset has four levels of labels namely - very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. We have also established benchmark results on this dataset using state-of-the-art video classification methods that are available today. We believe that DAiSEE will provide the research community with challenges in feature extraction, context-based inference, and development of suitable machine learning methods for related tasks, thus providing a springboard for further research. The dataset is available for download at https://people.iith.ac.in/vineethnb/resources/daisee/index.html.

Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network

Abstract: Automatic detection of students' engagement in online learning settings is a key element to improve the quality of learning and to deliver personalized learning materials to them. Varying levels of engagement exhibited by students in an online classroom is an affective behavior that takes place over space and time. Therefore, we formulate detecting levels of students' engagement from videos as a spatio-temporal classification problem. In this paper, we present a novel end-to-end Residual Network (ResNet) and Temporal Convolutional Network (TCN) hybrid neural network architecture for students' engagement level detection in videos. The 2D ResNet extracts spatial features from consecutive video frames, and the TCN analyzes the temporal changes in video frames to detect the level of engagement. The spatial and temporal arms of the hybrid network are jointly trained on raw video frames of a large publicly available students' engagement detection dataset, DAiSEE. We compared our method with several competing students' engagement detection methods on this dataset. The ResNet+TCN architecture outperforms all other studied methods, improves the state-of-the-art engagement level detection accuracy, and sets a new baseline for future research.

An Optimized CNN Model for Engagement Recognition in an E-Learning Environment

Abstract: In the wake of the restrictions imposed on social interactions due to the COVID-19 pandemic, traditional classroom education was replaced by distance education in many universities. Under the changed circumstances, students are required to learn more independently. The challenge for teachers has been to duly ascertain students’ learning efficiency and engagement during online lectures. This paper proposes an optimized lightweight convolutional neural network (CNN) model for engagement recognition within a distance-learning setup through facial expressions. The ShuffleNet v2 architecture was selected, as this model can easily adapt to mobile platforms and deliver outstanding performance compared to other lightweight models. The proposed model was trained, tested, evaluated and compared with other CNN models. The results of our experiment showed that an optimized model based on the ShuffleNet v2 architecture with a change of activation function and the introduction of an attention mechanism provides the best performance concerning engagement recognition. Further, our proposed model outperforms many existing works in engagement recognition on the same database. Finally, this model is suitable for student engagement recognition for distance learning on mobile platforms.

End-to-end video-based engagement measurement approaches and their deep-learning models, with an explanation of how they analyze sequential video data for engagement measurement, NA: Not Analyzed.

A summary of existing video-based end-to-end, and feature-based engagement measurement approaches are presented in Tables {{table:f8cabf2b-7a81-4cfc-90fc-eb872718a7c8}} , and {{table:3f8c7802-da04-4edf-9d28-d17c09876be5}} , respectively. In the end-to-end approaches, consecutive raw video frames are fed to spatio-temporal 3D CNNs or a combination of 2D CNNs and sequential networks, including RNNs (such as vanilla RNN, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU)), TCNs, and transformers. In the former methods, 3D CNNs utilize 3D convolutions to analyze the spatial information in the frames and the temporal changes between consecutive frames, thereby determining the engagement level. The 3D CNNs used for the purpose of engagement measurement were C3D {{cite:be6e0b6}}, {{cite:d2ac396}}, Inflated 3D CNN (I3D) {{cite:215968e}}, and 3D DenseNet {{cite:dc98e94}}. In the latter methods, 2D CNNs and sequential models are jointly trained to automatically extract visual features from raw video frames, analyze the sequences of extracted features, and output the engagement level. Various methods have been explored to this end, Gupta et al. {{cite:be6e0b6}} used Long-Term Recurrent Convolutional Network (LRCN), a combination of 2D CNN and LSTM, Abedi and Khan {{cite:d2ac396}} used 2D ResNet along with TCN, Liao et al. {{cite:556feb0}} used Squeeze-and-Excitation ResNet along with LSTM with global attention, and Selim et al. {{cite:54ed86b}} used EfficientNet along with LSTM and TCN.


101 categories of human action video frames

Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Information on the image classification datasets used for transfer learning.

We perform experiments on 8 image classification datasets which span many different domains including common objects {{cite:12949ed}}, {{cite:0a40879}}, fine-grained animals {{cite:a8f2252}}, indoor and outdoor scenes {{cite:ab769da}}, foods {{cite:f123b75}}, traffic signs {{cite:782cc2c}},
natural textures {{cite:6860842}}, and human actions {{cite:9177ed6}}.
Table {{table:bbda2184-1ee0-4f0a-80d2-0f47e4a395cd}}  shows the detailed information of each dataset.


Exploring Plausible Patches Using Source Code Embeddings in JavaScript

Abstract: Despite the immense popularity of the Automated Program Repair (APR) field, the question of patch validation is still open. Most of the present-day approaches follow the so-called Generate-and-Validate approach, where first a candidate solution is being generated and after validated against an oracle. The latter, however, might not give a reliable result, because of the imperfections in such oracles; one of which is usually the test suite. Although (re-) running the test suite is right under one's nose, in real life applications the problem of over- and underfitting often occurs, resulting in inadequate patches. Efforts that have been made to tackle with this problem include patch filtering, test suite expansion, careful patch producing and many more. Most approaches to date use post-filtering relying either on test execution traces or make use of some similarity concept measured on the generated patches. Our goal is to investigate the nature of these similarity-based approaches. To do so, we trained a Doc2Vec model on an open-source JavaScript project and generated 465 patches for 10 bugs in it. These plausible patches alongside with the developer fix are then ranked based on their similarity to the original program. We analyzed these similarity lists and found that plain document embeddings may lead to misclassification - it fails to capture nuanced code semantics. Nevertheless, in some cases it also provided useful information, thus helping to better understand the area of Automated Program Repair.

Predicting Patch Correctness Based on the Similarity of Failing Test Cases

Abstract: Towards predicting patch correctness in APR, we propose a simple, but novel hypothesis on how the link between the patch behaviour and failing test specifications can be drawn: similar failing test cases should require similar patches. We then propose BATS, an unsupervised learning-based system to predict patch correctness by checking patch Behaviour Against failing Test Specification. BATS exploits deep representation learning models for code and patches: for a given failing test case, the yielded embedding is used to compute similarity metrics in the search for historical similar test cases in order to identify the associated applied patches, which are then used as a proxy for assessing generated patch correctness. Experimentally, we first validate our hypothesis by assessing whether ground-truth developer patches cluster together in the same way that their associated failing test cases are clustered. Then, after collecting a large dataset of 1278 plausible patches (written by developers or generated by some 32 APR tools), we use BATS to predict correctness: BATS achieves an AUC between 0.557 to 0.718 and a recall between 0.562 and 0.854 in identifying correct patches. Compared against previous work, we demonstrate that our approach outperforms state-of-the-art performance in patch correctness prediction, without the need for large labeled patch datasets in contrast with prior machine learning-based approaches. While BATS is constrained by the availability of similar test cases, we show that it can still be complementary to existing approaches: used in conjunction with a recent approach implementing supervised learning, BATS improves the overall recall in detecting correct patches. We finally show that BATS can be complementary to the state-of-the-art PATCH-SIM dynamic approach of identifying the correct patches for APR tools.

A summary and comparison of learning-based APCA studies

Table {{table:727b1738-31f5-4f4e-b35f-9d327bdecade}}  presents a summary of existing learning-based techniques to predict patch correctness automatically in the literature.
The first and second columns list the APCA technique and the time of publication.
The third column lists the targeted programming languages.
The fourth column lists the features adopted by the APCA technique.
The remaining columns list the employed datasets and the public repositories.

Now, we discuss and summarize these individual approaches as follows.

[colback=gray!15, colframe=black, boxsep=-0.15cm, middle=-0.15cm,breakable]
✎ Summary
{{formula:5c3edaca-178a-4580-81f0-5ecfa429e304}}

The overfitting issue has become a key focus in the field of APR, which has led to the emergence and rapid development of recent APCA techniques.
DL techniques have been gradually used to predict the correctness of patches by learning features from historical corpora.
Compared to traditional dynamic and static APCA, learning-based APCA has shown impressive performance in prediction accuracy and recall.
We provide a summary of the existing learning-based APCA techniques in Table {{table:727b1738-31f5-4f4e-b35f-9d327bdecade}} .
In the literature, most existing APCA techniques employ a two-component pipeline, i.e., the feature extractor and the classifier.
The former extracts the features from the source code of patches, e.g., hand-crafted features, static representation features and dynamic execution features, while the latter trains a classifier to perform binary prediction, e.g., Random Forest and Decision tree.
Despite increasing research efforts being put into this phase and encouraging progress being made, the problem of patch overfitting still hinders the application and deployment of repair techniques in practice.
Therefore, the APR community needs more advanced APCA techniques to improve the correctness of returned patches, e.g., patch-aware feature extraction and more powerful pre-trained models.

{{formula:14f79c02-341a-4bae-96f7-9515d6346b97}}


Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

TADAM: Task dependent adaptive metric for improved few-shot learning

Abstract: Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100. Our code is publicly available at this https URL.

Using Deep Learning for Image-Based Plant Disease Detection

Abstract: Crop diseases are a major threat to food security, but their rapid identification remains difficult in many parts of the world due to the lack of the necessary infrastructure. The combination of increasing global smartphone penetration and recent advances in computer vision made possible by deep learning has paved the way for smartphone-assisted disease diagnosis. Using a public dataset of 54,306 images of diseased and healthy plant leaves collected under controlled conditions, we train a deep convolutional neural network to identify 14 crop species and 26 diseases (or absence thereof). The trained model achieves an accuracy of 99.35% on a held-out test set, demonstrating the feasibility of this approach. Overall, the approach of training deep learning models on increasingly large and publicly available image datasets presents a clear path toward smartphone-assisted crop disease diagnosis on a massive global scale.

Dataset information.

Table {{table:1ca839fb-e455-4cbf-9039-3dd718e50495}}  summarizes detailed descriptions of (a) pretraining datasets, (c) linear evaluation benchmarks, and (c) few-shot learning benchmarks. For linear evaluation benchmarks, we randomly choose validation samples in the training split for each dataset when the validation split is not officially provided. For few-shot benchmarks, we use the meta-test split for FC100 {{cite:fa5910e}}, and whole datasets for CUB200 {{cite:a50a476}} and Plant Disease {{cite:b7c3eb5}}. The evaluation details are described in Section .