Transformer (machine learning model)

Subsequent_work The original transformer uses ReLU activation function.  Other activation functions were developed, such as SwiGLU.  Transformers may use other positional encoding methods than sinusoidal.  RoPE (rotary positional embedding), is best explained by considering a list of 2-dimensional vectors [ ( x 1 ( 1 ) , x 1 ( 2 ) ) , ( x 2 ( 1 ) , x 2 ( 2 ) ) , ( x 3 ( 1 ) , x 3 ( 2 ) ) , .  .  .  ] {\displaystyle [(x_{1}^{(1)},x_{1}^{(2)}),(x_{2}^{(1)},x_{2}^{(2)}),(x_{3}^{(1)},x_{3}^{(2)}),. . . ]} .  Now pick some angle �� {\displaystyle \theta } .  Then RoPE encoding isThe benefit of RoPE is that the dot-product between two vectors depends on their relative location only: ALiBi (Attention with Linear Biases) is not a replacement for the positional encoder on the original transformer.  Instead, it is an additional positional encoder that is directly plugged into the attention mechanism.  Specifically, the ALiBi attention mechanism isALiBi allows pretraining on short context windows, then finetuning on longer context windows.  Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).  FlashAttention is an algorithm that implements the transformer attention mechanism efficiently on a GPU.  It performs matrix multiplications in blocks, such that each block fits within the cache of a GPU, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow).  An improved version, FlashAttention-2, was developed to cater to the rising demand for language models capable of handling longer context lengths.  It offers enhancements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs (FP16/BF16), a 2x speed increase over the original FlashAttention.  Key advancements in FlashAttention-2 include the reduction of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head dimensions up to 256 and multi-query attention (MQA) and grouped-query attention (GQA).  Benchmarks revealed FlashAttention-2 to be up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch.  Future developments include optimization for new hardware like H100 GPUs and new data types like FP8.  Multi-Query Attention changes the multiheaded attention mechanism.  Whereas normally, This has a neutral effect on model quality and training speed, but increases inference speed.  Training transformer-based architectures can be expensive, especially for long inputs.  Alternative architectures include the Reformer (which reduces the computational load from O ( N 2 ) {\displaystyle O(N^{2})} to O ( N ln ��� N ) {\displaystyle O(N\ln N)} ), or models like ETC/BigBird (which can reduce it to O ( N ) {\displaystyle O(N)} ) where N {\displaystyle N} is the length of the sequence.  This is done using locality-sensitive hashing and reversible layers.


Transformer (machine learning model)

Subsequent_work Ordinary transformers require a memory size that is quadratic in the size of the context window.  Attention-free transformers reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.  Long Range Arena (2020) is a standard benchmark for comparing the behavior of transformer architectures over long inputs.  Random Feature Attention (2021) uses Fourier random features:This approximation can be computed in linear time, as we can compute the matrix �� ( k i ) v i T {\displaystyle \varphi (k_{i})v_{i}^{T}} first, then multiply it with the query.  In essence, we have managed to obtain a more precise version of Performer (2022) uses the same Random Feature Attention, but w 1 , .  .  .  , w D {\displaystyle w_{1},. . . ,w_{D}} are first independently sampled from the normal distribution N ( 0 , �� 2 I ) {\displaystyle N(0,\sigma ^{2}I)} , then they are Gram-Schmidt processed.  Transformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to "tokenize" the modality.  Vision transformers adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like tokens in a standard transformer.  Conformer and later Whisper follow the same pattern for speech recognition, first turning the speech signal into a spectrogram, which is then treated like an image, i. e.  broken down into a series of patches, turned into vectors and treated like tokens in a standard transformer.  Perceivers by Andrew Jaegle et al.  (2021) can learn from large amounts of heterogeneous data.  Regarding image outputs, Peebles et al introduced a diffusion transformer (DiT) which facilitates use of the transformer architecture for diffusion-based image production.  Also, Google released a transformer-centric image generator called "Muse" based on parallel decoding and masked generative transformer technology.  (Transformers played a less-central role with prior image-producing technologies, albeit still a significant one.


Transformer (machine learning model)

See_also Perceiver ��� Machine learning algorithm for non-textual data BERT (language model) ��� Language model developed by Google GPT-3 ��� 2020 large language model GPT-4 ��� 2023 text-generating language model ChatGPT ��� AI chatbot developed by OpenAI Wu Dao ��� Chinese multimodal artificial intelligence program Vision transformer ��� Machine learning algorithm for vision processing BLOOM (language model) ��� Open-access multilingual language model


Dimensionality reduction

Feature_projection Feature projection (also called feature extraction) transforms the data from the high-dimensional space to a space of fewer dimensions.  The data transformation may be linear, as in principal component analysis (PCA), but many nonlinear dimensionality reduction techniques also exist.  For multidimensional data, tensor representation can be used in dimensionality reduction through multilinear subspace learning.  The main linear technique for dimensionality reduction, principal component analysis, performs a linear mapping of the data to a lower-dimensional space in such a way that the variance of the data in the low-dimensional representation is maximized.  In practice, the covariance (and sometimes the correlation) matrix of the data is constructed and the eigenvectors on this matrix are computed.  The eigenvectors that correspond to the largest eigenvalues (the principal components) can now be used to reconstruct a large fraction of the variance of the original data.  Moreover, the first few eigenvectors can often be interpreted in terms of the large-scale physical behavior of the system, because they often contribute the vast majority of the system's energy, especially in low-dimensional systems.  Still, this must be proven on a case-by-case basis as not all systems exhibit this behavior.  The original space (with dimension of the number of points) has been reduced (with data loss, but hopefully retaining the most important variance) to the space spanned by a few eigenvectors. [citation needed] NMF decomposes a non-negative matrix to the product of two non-negative ones, which has been a promising tool in fields where only non-negative signals exist, such as astronomy.  NMF is well known since the multiplicative update rule by Lee & Seung, which has been continuously developed: the inclusion of uncertainties, the consideration of missing data and parallel computation, sequential construction which leads to the stability and linearity of NMF, as well as other updates including handling missing data in digital image processing.  With a stable component basis during construction, and a linear modeling process, sequential NMF is able to preserve the flux in direct imaging of circumstellar structures in astronomy, as one of the methods of detecting exoplanets, especially for the direct imaging of circumstellar discs.  In comparison with PCA, NMF does not remove the mean of the matrices, which leads to unphysical non-negative fluxes; therefore NMF is able to preserve more information than PCA as demonstrated by Ren et al.  Principal component analysis can be employed in a nonlinear way by means of the kernel trick.  The resulting technique is capable of constructing nonlinear mappings that maximize the variance in the data.  The resulting technique is called kernel PCA.  Other prominent nonlinear techniques include manifold learning techniques such as Isomap, locally linear embedding (LLE), Hessian LLE, Laplacian eigenmaps, and methods based on tangent space analysis.  These techniques construct a low-dimensional data representation using a cost function that retains local properties of the data, and can be viewed as defining a graph-based kernel for Kernel PCA.  More recently, techniques have been proposed that, instead of defining a fixed kernel, try to learn the kernel using semidefinite programming.


Dimensionality reduction

Feature_projection The most prominent example of such a technique is maximum variance unfolding (MVU).  The central idea of MVU is to exactly preserve all pairwise distances between nearest neighbors (in the inner product space), while maximizing the distances between points that are not nearest neighbors.  An alternative approach to neighborhood preservation is through the minimization of a cost function that measures differences between distances in the input and output spaces.  Important examples of such techniques include: classical multidimensional scaling, which is identical to PCA; Isomap, which uses geodesic distances in the data space; diffusion maps, which use diffusion distances in the data space; t-distributed stochastic neighbor embedding (t-SNE), which minimizes the divergence between distributions over pairs of points; and curvilinear component analysis.  A different approach to nonlinear dimensionality reduction is through the use of autoencoders, a special kind of feedforward neural networks with a bottle-neck hidden layer.  The training of deep encoders is typically performed using a greedy layer-wise pre-training (e. g. , using a stack of restricted Boltzmann machines) that is followed by a finetuning stage based on backpropagation.  Linear discriminant analysis (LDA) is a generalization of Fisher's linear discriminant, a method used in statistics, pattern recognition and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events.  GDA deals with nonlinear discriminant analysis using kernel function operator.  The underlying theory is close to the support-vector machines (SVM) insofar as the GDA method provides a mapping of the input vectors into high-dimensional feature space.  Similar to LDA, the objective of GDA is to find a projection for the features into a lower dimensional space by maximizing the ratio of between-class scatter to within-class scatter.  Autoencoders can be used to learn nonlinear dimension reduction functions and codings together with an inverse function from the coding to the original representation.  T-distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality reduction technique useful for visualization of high-dimensional datasets.  It is not recommended for use in analysis such as clustering or outlier detection since it does not necessarily preserve densities or distances well.  Uniform manifold approximation and projection (UMAP) is a nonlinear dimensionality reduction technique.  Visually, it is similar to t-SNE, but it assumes that the data is uniformly distributed on a locally connected Riemannian manifold and that the Riemannian metric is locally constant or approximately locally constant.


Mixture of experts

Deep_learning The previous section described MoE as it was used before the era of deep learning.  After deep learning, MoE found applications in running the largest models, as a simple way to perform conditional computation: only parts of the model are used, the parts chosen according to what the input is.  The earliest paper that applies MoE to deep learning is, which proposes to use a different gating network at each layer in a deep neural network.  Specifically, each gating is a linear-ReLU-linear-softmax network, and each expert is a linear-ReLU network.  The key design desiderata for MoE in deep learning is to reduce computing cost.  Consequently, for each query, only a small subset of the experts should be queried.  This makes MoE in deep learning different from classical MoE.  In classical MoE, the output for each query is a weighted sum of all experts' outputs.  In deep learning MoE, the output for each query can only involve a few experts' outputs.  Consequently, the key design choice in MoE becomes routing: given a batch of queries, how to route the queries to the best experts.  The sparsely-gated MoE layer, published by researchers from Google Brain, uses feedforward networks as experts, and linear-softmax gating.  Similar to the previously proposed hard MoE, they achieve sparsity by a weighted sum of only the top-k experts, instead of the weighted sum of all of them.  Specifically, in a MoE layer, there are feedforward networks f 1 , .  .  .  , f n {\displaystyle f_{1},. . . ,f_{n}} , and a gating network w {\displaystyle w} .  The gating network is defined by w ( x ) = s o f t m a x ( t o p k ( W x + noise ) ) {\displaystyle w(x)=\mathrm {softmax} (\mathrm {top} _{k}(Wx+{\text{noise}}))} , where t o p k {\displaystyle \mathrm {top} _{k}} is a function that keeps the top-k entries of a vector the same, but sets all other entries to ��� ��� {\displaystyle -\infty } .  The addition of noise helps with load balancing.  The choice of k {\displaystyle k} is a hyperparameter that is chosen according to application.  Typical values are k = 1 , 2 {\displaystyle k=1,2} .  The k = 1 {\displaystyle k=1} version is also called the Switch Transformer.  As demonstration, they trained a series of models for machine translation with alternating layers of MoE and LSTM, and compared with deep LSTM models.  Table 3 shows that the MoE models used less inference time compute, despite having 30x more parameters.  Vanilla MoE tend to have issues of load balancing: some experts are consulted often, while other experts rarely or not at all.  To encourage the gate to select each expert with equal frequency (proper load balancing) within each batch, each MoE layer has two auxiliary loss functions.  This is improved by into a single auxiliary loss function.


Mixture of experts

Deep_learning Specifically, let n {\displaystyle n} be the number of experts, then for a given batch of queries { x 1 , x 2 , .  .  .  , x T } {\displaystyle \{x_{1},x_{2},. . . ,x_{T}\}} , the auxiliary loss for the batch isIn sparsely-gated MoE, only the top-k experts are queried, and their outputs are weighted-summed.  There are other methods.  In Hash MoE, routing is performed deterministically by a hash function, fixed before learning begins.  For example, if the model is a 4-layered Transformer, and input is a token for word "eat", and the hash of "eat" is ( 1 , 4 , 2 , 3 ) {\displaystyle (1,4,2,3)} , then the token would be routed to the 1st expert in layer 1, 4th expert in layer 2, etc.  Despite its simplicity, it achieves competitive performance as sparsely gated MoE with k = 1 {\displaystyle k=1} .  In soft MoE, suppose in each batch, each expert can process p {\displaystyle p} queries, then there are n �� p {\displaystyle n\times p} queries that can be assigned per batch.  Now for each batch of queries { x 1 , x 2 , .  .  .  , x T } {\displaystyle \{x_{1},x_{2},. . . ,x_{T}\}} , the soft MoE layer computes an array w i , j , k {\displaystyle w_{i,j,k}} , such that ( w i , j , 1 , .  .  .  , w i , j , T ) {\displaystyle (w_{i,j,1},. . . ,w_{i,j,T})} is a probability distribution over queries, and the i {\displaystyle i} -th expert's j {\displaystyle j} -th query is ��� k w i , j , k x k {\displaystyle \sum _{k}w_{i,j,k}x_{k}} .  However, this does not work with autoregressive modelling, since the weights w i , j , k {\displaystyle w_{i,j,k}} over one token depends on all other tokens'.  Other approaches include solving it as a constrained linear programming problem, making each expert choose the top-k queries it wants (instead of each query choosing the top-k experts for it), using reinforcement learning to train the routing algorithm (since picking an expert is a discrete action, like in RL).  Suppose there are n {\displaystyle n} experts in a layer.  For a given batch of queries { x 1 , x 2 , .  .  .  , x T } {\displaystyle \{x_{1},x_{2},. . . ,x_{T}\}} , each query is routed to one ore more experts.  For example, if each query is routed to one expert as in Switch Transformers, and if the experts are load-balanced, then each expert should expect on average T / n {\displaystyle T/n} queries in a batch.  In practice, the experts cannot expect perfect load balancing: in some batches, one expert might be underworked, while in other batches, it would be overworked.  Since the inputs cannot move through the layer until every expert in the layer has finished the queries it is assigned, load balancing is important.


Mixture of experts

Deep_learning As a hard constraint on load balancing, there is the capacity factor: each expert is only allowed to process up to c ��� T / n {\displaystyle c\cdot T/n} queries in a batch.  found c ��� [ 1.25 , 2 ] {\displaystyle c\in [1.25,2]} to work in practice.  MoE layers are used in very large Transformer models, for which learning and inferring over the full model is too costly.  In Transformer models, the MoE layers are often used to select the feedforward layers (typically a linear-ReLU-linear network), appearing in each Transformer block after the multiheaded attention.  This is because the feedforward layers take up an increasing portion of the computing cost as models grow larger.  For example, 90% of parameters in PALM-540B are in feedforward layers.  A series of large language models from Google used MoE.  GShard uses MoE with up to top-2 experts per layer.  Specifically, the top-1 expert is always selected, and the top-2th expert is selected with probability proportional to that experts' weight according to the gating function.  Later, GLaM demonstrated a language model with 1.2 trillion parameters, eath MoE layer using top-2 out of 64 experts.  Switch Transformers use top-1 in all MoE layers.  The NLLB-200 by Meta AI is a machine translation model for 200 languages.  Each MoE layer uses a hierarchical MoE with two levels.  On the first level, the gating function chooses to use either a "shared" feedforward layer, or to use the experts.  If using the experts, then another gating function computes the weights and chooses the top-2 experts (see Figure 19).  MoE large language models can be adapted for downstream tasks by instruction tuning.  Generally, MoE are used when dense models have become too costly.  As of 2023, the largest models tend to be large language models.  Outside of those, Vision MoE is a Transformer model with MoE layers.  They demonstrated it by training a model with 15 billion parameters.


Stochastic gradient descent

Extensions_and_variants Many improvements on the basic stochastic gradient descent algorithm have been proposed and used.  In particular, in machine learning, the need to set a learning rate (step size) has been recognized as problematic.  Setting this parameter too high can cause the algorithm to diverge; setting it too low makes it slow to converge.  A conceptually simple extension of stochastic gradient descent makes the learning rate a decreasing function ��t of the iteration number t, giving a learning rate schedule, so that the first iterations cause large changes in the parameters, while the later ones do only fine-tuning.  Such schedules have been known since the work of MacQueen on k-means clustering.  Practical guidance on choosing the step size in several variants of SGD is given by Spall.  As mentioned earlier, classical stochastic gradient descent is generally sensitive to learning rate ��.  Fast convergence requires large learning rates but this may induce numerical instability.  The problem can be largely solved by considering implicit updates whereby the stochastic gradient is evaluated at the next iterate rather than the current one: This equation is implicit since w n e w {\displaystyle w^{\rm {new}}} appears on both sides of the equation.  It is a stochastic form of the proximal gradient method since the update can also be written as: As an example, consider least squares with features x 1 , ��� , x n ��� R p {\displaystyle x_{1},\ldots ,x_{n}\in \mathbb {R} ^{p}} and observations y 1 , ��� , y n ��� R {\displaystyle y_{1},\ldots ,y_{n}\in \mathbb {R} } .  We wish to solve: where x j ��� w = x j 1 w 1 + x j , 2 w 2 + .  .  .  + x j , p w p {\displaystyle x_{j}'w=x_{j1}w_{1}+x_{j,2}w_{2}+. . . +x_{j,p}w_{p}} indicates the inner product.  Note that x {\displaystyle x} could have "1" as the first element to include an intercept.  Classical stochastic gradient descent proceeds as follows: where i {\displaystyle i} is uniformly sampled between 1 and n {\displaystyle n} .  Although theoretical convergence of this procedure happens under relatively mild assumptions, in practice the procedure can be quite unstable.  In particular, when �� {\displaystyle \eta } is misspecified so that I ��� �� x i x i ��� {\displaystyle I-\eta x_{i}x_{i}'} has large absolute eigenvalues with high probability, the procedure may diverge numerically within a few iterations.  In contrast, implicit stochastic gradient descent (shortened as ISGD) can be solved in closed-form as: This procedure will remain numerically stable virtually for all �� {\displaystyle \eta } as the learning rate is now normalized.  Such comparison between classical and implicit stochastic gradient descent in the least squares problem is very similar to the comparison between least mean squares (LMS) and normalized least mean squares filter (NLMS).  Even though a closed-form solution for ISGD is only possible in least squares, the procedure can be efficiently implemented in a wide range of models.


Stochastic gradient descent

Extensions_and_variants Specifically, suppose that Q i ( w ) {\displaystyle Q_{i}(w)} depends on w {\displaystyle w} only through a linear combination with features x i {\displaystyle x_{i}} , so that we can write ��� w Q i ( w ) = ��� q ( x i ��� w ) x i {\displaystyle \nabla _{w}Q_{i}(w)=-q(x_{i}'w)x_{i}} , where q ( ) ��� R {\displaystyle q()\in \mathbb {R} } may depend on x i , y i {\displaystyle x_{i},y_{i}} as well but not on w {\displaystyle w} except through x i ��� w {\displaystyle x_{i}'w} .  Least squares obeys this rule, and so does logistic regression, and most generalized linear models.  For instance, in least squares, q ( x i ��� w ) = y i ��� x i ��� w {\displaystyle q(x_{i}'w)=y_{i}-x_{i}'w} , and in logistic regression q ( x i ��� w ) = y i ��� S ( x i ��� w ) {\displaystyle q(x_{i}'w)=y_{i}-S(x_{i}'w)} , where S ( u ) = e u / ( 1 + e u ) {\displaystyle S(u)=e^{u}/(1+e^{u})} is the logistic function.  In Poisson regression, q ( x i ��� w ) = y i ��� e x i ��� w {\displaystyle q(x_{i}'w)=y_{i}-e^{x_{i}'w}} , and so on.  In such settings, ISGD is simply implemented as follows.  Let f ( �� ) = �� q ( x i ��� w o l d + �� | | x i | | 2 ) {\displaystyle f(\xi )=\eta q(x_{i}'w^{old}+\xi ||x_{i}||^{2})} , where �� {\displaystyle \xi } is scalar.  Then, ISGD is equivalent to: The scaling factor �� ��� ��� R {\displaystyle \xi ^{\ast }\in \mathbb {R} } can be found through the bisection method since in most regular models, such as the aforementioned generalized linear models, function q ( ) {\displaystyle q()} is decreasing, and thus the search bounds for �� ��� {\displaystyle \xi ^{\ast }} are [ min ( 0 , f ( 0 ) ) , max ( 0 , f ( 0 ) ) ] {\displaystyle [\min(0,f(0)),\max(0,f(0))]} .  Further proposals include the momentum method or the heavy ball method, which in ML context appeared in Rumelhart, Hinton and Williams' paper on backpropagation learning and borrowed the idea from Soviet mathematician Boris Polyak's 1964 article on solving functional equations.  Stochastic gradient descent with momentum remembers the update ��w at each iteration, and determines the next update as a linear combination of the gradient and the previous update: that leads to: where the parameter w {\displaystyle w} which minimizes Q ( w ) {\displaystyle Q(w)} is to be estimated, �� {\displaystyle \eta } is a step size (sometimes called the learning rate in machine learning) and �� {\displaystyle \alpha } is an exponential decay factor between 0 and 1 that determines the relative contribution of the current gradient and earlier gradients to the weight change.


Stochastic gradient descent

Extensions_and_variants The name momentum stems from an analogy to momentum in physics: the weight vector w {\displaystyle w} , thought of as a particle traveling through parameter space, incurs acceleration from the gradient of the loss ("force").  Unlike in classical stochastic gradient descent, it tends to keep traveling in the same direction, preventing oscillations.  Momentum has been used successfully by computer scientists in the training of artificial neural networks for several decades.  The momentum method is closely related to underdamped Langevin dynamics, and may be combined with Simulated Annealing.  In mid-1980s the method was modified by Yurii Nesterov to use the gradient predicted at the next point, and the resulting so-called Nesterov Accelerated Gradient was sometimes used in ML in the 2010s.  Averaged stochastic gradient descent, invented independently by Ruppert and Polyak in the late 1980s, is ordinary stochastic gradient descent that records an average of its parameter vector over time.  That is, the update is the same as for ordinary stochastic gradient descent, but the algorithm also keeps track of When optimization is done, this averaged parameter vector takes the place of w.  AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011.  Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse.  This strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative.  Examples of such applications include natural language processing and image recognition.  It still has a base learning rate ��, but this is multiplied with the elements of a vector {Gj,j} which is the diagonal of the outer product matrix where g �� = ��� Q i ( w ) {\displaystyle g_{\tau }=\nabla Q_{i}(w)} , the gradient, at iteration ��.  The diagonal is given by This vector essentially stores a historical sum of gradient squares by dimension and is updated after every iteration.  The formula for an update is now or, written as per-parameter updates, Each {G(i,i)} gives rise to a scaling factor for the learning rate that applies to a single parameter wi.  Since the denominator in this factor, G i = ��� �� = 1 t g �� 2 {\displaystyle {\sqrt {G_{i}}}={\sqrt {\sum _{\tau =1}^{t}g_{\tau }^{2}}}} is the ���2 norm of previous derivatives, extreme parameter updates get dampened, while parameters that get few or small updates receive higher learning rates.  While designed for convex problems, AdaGrad has been successfully applied to non-convex optimization.  RMSProp (for Root Mean Square Propagation) is a method invented by Geoffrey Hinton in 2012 in which the learning rate is, like in Adagrad, adapted for each of the parameters.  The idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight.  Unusually, it was not published in an article but merely described in a Coursera lecture.


Stochastic gradient descent

Extensions_and_variants [citation needed] So, first the running average is calculated in terms of means square, where, �� {\displaystyle \gamma } is the forgetting factor.  The concept of storing the historical gradient as sum of squares is borrowed from Adagrad, but "forgetting" is introduced to solve Adagrad's diminishing learning rates in non-convex problems by gradually decreasing the influence of old data.  And the parameters are updated as, RMSProp has shown good adaptation of learning rate in different applications.  RMSProp can be seen as a generalization of Rprop and is capable to work with mini-batches as well opposed to only full-batches.  Adam (short for Adaptive Moment Estimation) is a 2014 update to the RMSProp optimizer combining it with the main feature of the Momentum method.  In this optimization algorithm, running averages with exponential forgetting of both the gradients and the second moments of the gradients are used.  Given parameters w ( t ) {\displaystyle w^{(t)}} and a loss function L ( t ) {\displaystyle L^{(t)}} , where t {\displaystyle t} indexes the current training iteration (indexed at 0 {\displaystyle 0} ), Adam's parameter update is given by: where �� {\displaystyle \epsilon } is a small scalar (e. g.  10 ��� 8 {\displaystyle 10^{-8}} ) used to prevent division by 0, and �� 1 {\displaystyle \beta _{1}} (e. g.  0.9) and �� 2 {\displaystyle \beta _{2}} (e. g.  0.999) are the forgetting factors for gradients and second moments of gradients, respectively.  Squaring and square-rooting is done element-wise.  The profound influence of this algorithm inspired multiple newer, less well-known momentum-based optimization schemes using Nesterov-enhanced gradients (eg: NAdam and FASFA) and varying interpretations of second-order information (eg: Powerpropagation and AdaSqrt).  However, the most commonly used variants are AdaMax, which generalizes Adam using the infinity norm, and AMSGrad, which addresses convergence problems from Adam by using maximum of past squared gradients instead of the exponential average.  AdamW is a later update which mitigates an unoptimal choice of the weight decay algorithm in Adam.  Even though sign-based optimization goes back to the aforementioned Rprop, only in 2018 researchers tried to simplify Adam by removing the magnitude of the stochastic gradient from being taked into account and only considering its sign.  Backtracking line search is another variant of gradient descent.  All of the below are sourced from the mentioned link.  It is based on a condition known as the Armijo���Goldstein condition.  Both methods allow learning rates to change at each iteration; however, the manner of the change is different.  Backtracking line search uses function evaluations to check Armijo's condition, and in principle the loop in the algorithm for determining the learning rates can be long and unknown in advance.  Adaptive SGD does not need a loop in determining learning rates.  On the other hand, adaptive SGD does not guarantee the "descent property" ��� which Backtracking line search enjoys ��� which is that f ( x n + 1 ) ��� f ( x n ) {\displaystyle f(x_{n+1})\leq f(x_{n})} for all n.


Stochastic gradient descent

Extensions_and_variants If the gradient of the cost function is globally Lipschitz continuous, with Lipschitz constant L, and learning rate is chosen of the order 1/L, then the standard version of SGD is a special case of backtracking line search.  A stochastic analogue of the standard (deterministic) Newton���Raphson algorithm (a "second-order" method) provides an asymptotically optimal or near-optimal form of iterative optimization in the setting of stochastic approximation[citation needed].  A method that uses direct measurements of the Hessian matrices of the summands in the empirical risk function was developed by Byrd, Hansen, Nocedal, and Singer.  However, directly determining the required Hessian matrices for optimization may not be possible in practice.  Practical and theoretically sound methods for second-order versions of SGD that do not require direct Hessian information are given by Spall and others.  (A less efficient method based on finite differences, instead of simultaneous perturbations, is given by Ruppert. ) Another approach to the approximation Hessian matrix is replacing it with the Fisher information matrix, which transforms usual gradient to natural.  These methods not requiring direct Hessian information are based on either values of the summands in the above empirical risk function or values of the gradients of the summands (i. e. , the SGD inputs).  In particular, second-order optimality is asymptotically achievable without direct calculation of the Hessian matrices of the summands in the empirical risk function.


Reinforcement learning

Algorithms_for_control_learning Even if the issue of exploration is disregarded and even if the state was observable (assumed hereafter), the problem remains to use past experience to find out which actions lead to higher cumulative rewards.  The agent's action selection is modeled as a map called policy: The policy map gives the probability of taking action a {\displaystyle a} when in state s {\displaystyle s} .  There are also deterministic policies.  The value function V �� ( s ) {\displaystyle V_{\pi }(s)} is defined as, expected return starting with state s {\displaystyle s} , i. e.  s 0 = s {\displaystyle s_{0}=s} , and successively following policy �� {\displaystyle \pi } .  Hence, roughly speaking, the value function estimates "how good" it is to be in a given state.  where the random variable R {\displaystyle R} denotes the return, and is defined as the sum of future discounted rewards: where r t {\displaystyle r_{t}} is the reward at step t {\displaystyle t} , �� ��� [ 0 , 1 ) {\displaystyle \gamma \in [0,1)} is the discount-rate.  Gamma is less than 1, so events in the distant future are weighted less than events in the immediate future.  The algorithm must find a policy with maximum expected return.  From the theory of Markov decision processes it is known that, without loss of generality, the search can be restricted to the set of so-called stationary policies.  A policy is stationary if the action-distribution returned by it depends only on the last state visited (from the observation agent's history).  The search can be further restricted to deterministic stationary policies.  A deterministic stationary policy deterministically selects actions based on the current state.  Since any such policy can be identified with a mapping from the set of states to the set of actions, these policies can be identified with such mappings with no loss of generality.  The brute force approach entails two steps: For each possible policy, sample returns while following it Choose the policy with the largest expected returnOne problem with this is that the number of policies can be large, or even infinite.  Another is that the variance of the returns may be large, which requires many samples to accurately estimate the return of each policy.  These problems can be ameliorated if we assume some structure and allow samples generated from one policy to influence the estimates made for others.  The two main approaches for achieving this are value function estimation and direct policy search.  Value function approaches attempt to find a policy that maximizes the return by maintaining a set of estimates of expected returns for some policy (usually either the "current" [on-policy] or the optimal [off-policy] one).  These methods rely on the theory of Markov decision processes, where optimality is defined in a sense that is stronger than the above one: A policy is called optimal if it achieves the best-expected return from any initial state (i. e. , initial distributions play no role in this definition).


Reinforcement learning

Algorithms_for_control_learning Again, an optimal policy can always be found amongst stationary policies.  To define optimality in a formal manner, define the value of a policy �� {\displaystyle \pi } by where R {\displaystyle R} stands for the return associated with following �� {\displaystyle \pi } from the initial state s {\displaystyle s} .  Defining V ��� ( s ) {\displaystyle V^{*}(s)} as the maximum possible value of V �� ( s ) {\displaystyle V^{\pi }(s)} , where �� {\displaystyle \pi } is allowed to change, A policy that achieves these optimal values in each state is called optimal.  Clearly, a policy that is optimal in this strong sense is also optimal in the sense that it maximizes the expected return �� �� {\displaystyle \rho ^{\pi }} , since �� �� = E [ V �� ( S ) ] {\displaystyle \rho ^{\pi }=E[V^{\pi }(S)]} , where S {\displaystyle S} is a state randomly sampled from the distribution �� {\displaystyle \mu } of initial states (so �� ( s ) = Pr ( s 0 = s ) {\displaystyle \mu (s)=\Pr(s_{0}=s)} ).  Although state-values suffice to define optimality, it is useful to define action-values.  Given a state s {\displaystyle s} , an action a {\displaystyle a} and a policy �� {\displaystyle \pi } , the action-value of the pair ( s , a ) {\displaystyle (s,a)} under �� {\displaystyle \pi } is defined by where R {\displaystyle R} now stands for the random return associated with first taking action a {\displaystyle a} in state s {\displaystyle s} and following �� {\displaystyle \pi } , thereafter.  The theory of Markov decision processes states that if �� ��� {\displaystyle \pi ^{*}} is an optimal policy, we act optimally (take the optimal action) by choosing the action from Q �� ��� ( s , ��� ) {\displaystyle Q^{\pi ^{*}}(s,\cdot )} with the highest value at each state, s {\displaystyle s} .  The action-value function of such an optimal policy ( Q �� ��� {\displaystyle Q^{\pi ^{*}}} ) is called the optimal action-value function and is commonly denoted by Q ��� {\displaystyle Q^{*}} .  In summary, the knowledge of the optimal action-value function alone suffices to know how to act optimally.  Assuming full knowledge of the Markov decision process, the two basic approaches to compute the optimal action-value function are value iteration and policy iteration.  Both algorithms compute a sequence of functions Q k {\displaystyle Q_{k}} ( k = 0 , 1 , 2 , ��� {\displaystyle k=0,1,2,\ldots } ) that converge to Q ��� {\displaystyle Q^{*}} .  Computing these functions involves computing expectations over the whole state-space, which is impractical for all but the smallest (finite) Markov decision processes.  In reinforcement learning methods, expectations are approximated by averaging over samples and using function approximation techniques to cope with the need to represent value functions over large state-action spaces.  Monte Carlo methods can be used in an algorithm that mimics policy iteration.  Policy iteration consists of two steps: policy evaluation and policy improvement.


Reinforcement learning

Algorithms_for_control_learning Monte Carlo is used in the policy evaluation step.  In this step, given a stationary, deterministic policy �� {\displaystyle \pi } , the goal is to compute the function values Q �� ( s , a ) {\displaystyle Q^{\pi }(s,a)} (or a good approximation to them) for all state-action pairs ( s , a ) {\displaystyle (s,a)} .  Assume (for simplicity) that the Markov decision process is finite, that sufficient memory is available to accommodate the action-values and that the problem is episodic and after each episode a new one starts from some random initial state.  Then, the estimate of the value of a given state-action pair ( s , a ) {\displaystyle (s,a)} can be computed by averaging the sampled returns that originated from ( s , a ) {\displaystyle (s,a)} over time.  Given sufficient time, this procedure can thus construct a precise estimate Q {\displaystyle Q} of the action-value function Q �� {\displaystyle Q^{\pi }} .  This finishes the description of the policy evaluation step.  In the policy improvement step, the next policy is obtained by computing a greedy policy with respect to Q {\displaystyle Q} : Given a state s {\displaystyle s} , this new policy returns an action that maximizes Q ( s , ��� ) {\displaystyle Q(s,\cdot )} .  In practice lazy evaluation can defer the computation of the maximizing actions to when they are needed.  Problems with this procedure include: 1.  The procedure may spend too much time evaluating a suboptimal policy.  2.  It uses samples inefficiently in that a long trajectory improves the estimate only of the single state-action pair that started the trajectory.  3.  When the returns along the trajectories have high variance, convergence is slow.  4.  It works in episodic problems only.  5.  It works in small, finite Markov decision processes only.  The first problem is corrected by allowing the procedure to change the policy (at some or all states) before the values settle.  This too may be problematic as it might prevent convergence.  Most current algorithms do this, giving rise to the class of generalized policy iteration algorithms.  Many actor-critic methods belong to this category.  The second issue can be corrected by allowing trajectories to contribute to any state-action pair in them.  This may also help to some extent with the third problem, although a better solution when returns have high variance is Sutton's temporal difference (TD) methods that are based on the recursive Bellman equation.  The computation in TD methods can be incremental (when after each transition the memory is changed and the transition is thrown away), or batch (when the transitions are batched and the estimates are computed once based on the batch).  Batch methods, such as the least-squares temporal difference method, may use the information in the samples better, while incremental methods are the only choice when batch methods are infeasible due to their high computational or memory complexity.  Some methods try to combine the two approaches.  Methods based on temporal differences also overcome the fourth issue.


Reinforcement learning

Algorithms_for_control_learning Another problem specific to TD comes from their reliance on the recursive Bellman equation.  Most TD methods have a so-called �� {\displaystyle \lambda } parameter ( 0 ��� �� ��� 1 ) {\displaystyle (0\leq \lambda \leq 1)} that can continuously interpolate between Monte Carlo methods that do not rely on the Bellman equations and the basic TD methods that rely entirely on the Bellman equations.  This can be effective in palliating this issue.  In order to address the fifth issue, function approximation methods are used.  Linear function approximation starts with a mapping �� {\displaystyle \phi } that assigns a finite-dimensional vector to each state-action pair.  Then, the action values of a state-action pair ( s , a ) {\displaystyle (s,a)} are obtained by linearly combining the components of �� ( s , a ) {\displaystyle \phi (s,a)} with some weights �� {\displaystyle \theta } : The algorithms then adjust the weights, instead of adjusting the values associated with the individual state-action pairs.  Methods based on ideas from nonparametric statistics (which can be seen to construct their own features) have been explored.  Value iteration can also be used as a starting point, giving rise to the Q-learning algorithm and its many variants.  Including Deep Q-learning methods when a neural network is used to represent Q, with various applications in stochastic search problems.  The problem with using action-values is that they may need highly precise estimates of the competing action values that can be hard to obtain when the returns are noisy, though this problem is mitigated to some extent by temporal difference methods.  Using the so-called compatible function approximation method compromises generality and efficiency.  An alternative method is to search directly in (some subset of) the policy space, in which case the problem becomes a case of stochastic optimization.  The two approaches available are gradient-based and gradient-free methods.  Gradient-based methods (policy gradient methods) start with a mapping from a finite-dimensional (parameter) space to the space of policies: given the parameter vector �� {\displaystyle \theta } , let �� �� {\displaystyle \pi _{\theta }} denote the policy associated to �� {\displaystyle \theta } .  Defining the performance function by under mild conditions this function will be differentiable as a function of the parameter vector �� {\displaystyle \theta } .  If the gradient of �� {\displaystyle \rho } was known, one could use gradient ascent.  Since an analytic expression for the gradient is not available, only a noisy estimate is available.  Such an estimate can be constructed in many ways, giving rise to algorithms such as Williams' REINFORCE method (which is known as the likelihood ratio method in the simulation-based optimization literature).  Policy search methods have been used in the robotics context.  Many policy search methods may get stuck in local optima (as they are based on local search).  A large class of methods avoids relying on gradient information.  These include simulated annealing, cross-entropy search or methods of evolutionary computation.


Reinforcement learning

Algorithms_for_control_learning Many gradient-free methods can achieve (in theory and in the limit) a global optimum.  Policy search methods may converge slowly given noisy data.  For example, this happens in episodic problems when the trajectories are long and the variance of the returns is large.  Value-function based methods that rely on temporal differences might help in this case.  In recent years, actor���critic methods have been proposed and performed well on various problems.  Finally, all of the above methods can be combined with algorithms that first learn a model.  For instance, the Dyna algorithm learns a model from experience, and uses that to provide more modelled transitions for a value function, in addition to the real transitions.  Such methods can sometimes be extended to use of non-parametric models, such as when the transitions are simply stored and 'replayed' to the learning algorithm.  There are other ways to use models than to update a value function.  For instance, in model predictive control the model is used to update the behavior directly.


Semisupervised learning

Semisupervised learning Weak supervision, also called semi-supervised learning, is a paradigm in machine learning, the relevance and notability of which increased with the advent of large language models due to large amount of data required to train them. It is characterized by using a combination of a small amount of human-labeled data (exclusively used in more expensive and time-consuming supervised learning paradigm), followed by a large amount of unlabeled data (used exclusively in unsupervised learning paradigm). In other words, the desired output values are provided only for a subset of the training data. The remaining data is unlabeled or imprecisely labeled. Intuitively, it can be seen as an exam and labeled data as sample problems that the teacher solves for the class as an aid in solving another set of problems. In the transductive setting, these unsolved problems act as exam questions. In the inductive setting, they become practice problems of the sort that will make up the exam. Technically, it could be viewed as performing clustering and then labeling the clusters with the labeled data, pushing the decision boundary away from high-density regions, or learning an underlying one-dimensional manifold where the data reside.


Neural network

Overview A biological neural network is composed of a group of chemically connected or functionally associated neurons. A single neuron may be connected to many other neurons and the total number of neurons and connections in a network may be extensive. Connections, called synapses, are usually formed from axons to dendrites, though dendrodendritic synapses and other connections are possible. Apart from electrical signalling, there are other forms of signalling that arise from neurotransmitter diffusion. Artificial intelligence, cognitive modelling, and neural networks are information processing paradigms inspired by how biological neural systems process data. Artificial intelligence and cognitive modelling try to simulate some properties of biological neural networks. In the artificial intelligence field, artificial neural networks have been applied successfully to speech recognition, image analysis and adaptive control, in order to construct software agents (in computer and video games) or autonomous robots. Historically, digital computers evolved from the von Neumann model, and operate via the execution of explicit instructions via access to memory by a number of processors. On the other hand, the origins of neural networks are based on efforts to model information processing in biological systems. Unlike the von Neumann model, neural network computing does not separate memory and processing. Neural network theory has served to identify better how the neurons in the brain function and provide the basis for efforts to create artificial intelligence.


Neural scaling law

Introduction In general, a neural model can be characterized by 4 parameters: size of the model, size of the training dataset, cost of training, performance after training.  Each of these four variables can be precisely defined into a real number, and they are empirically found to be related by simple statistical laws, called "scaling laws". [citation needed] These are usually written as N , D , C , L {\displaystyle N,D,C,L} (number of parameters, dataset size, computing cost, loss).  In most cases, the size of the model is simply the number of parameters.  However, one complication arises with the use of sparse models, such as mixture-of-expert models.  In sparse models, during every inference, only a fraction of the parameters are used.  In comparison, most other kinds of neural networks, such as Transformer networks, always use all their parameters during every inference.  The size of the training dataset is usually quantified by the number of data points it contains.  Larger training datasets are typically preferred as they provide a richer and more diverse source of information for the model to learn from.  This in turn can lead to improved generalization performance when the model is applied to unseen data.  However, increasing the size of the training dataset also increases the computational resources and time required for model training.  With the "pretrain, then finetune" method used in most large language models, there are two kinds of training dataset: the pretraining dataset and the finetuning dataset.  Their sizes would have different effects on model performance.  Generally, the finetuning dataset is less than 1% the size of pretraining dataset.  In some cases, a small amount of high quality data suffices for finetuning, and more data does not improve performance.  The cost of training is typically measured in terms of time (how long it takes to train the model) and computational resources (how much processing power and memory are required to train the model).  It's important to note that the cost of training can be significantly reduced with efficient training algorithms, optimized software libraries, and parallel computing on specialized hardware like GPUs or TPUs.  The cost of training a neural model is a function of several factors including the size of the model, the size of the training dataset, the complexity of the training algorithm, and the computational resources available.  In particular, doubling the training dataset does not necessarily double the cost of training, because one may train the model for several times over the same dataset (each being an "epoch").  The performance of a neural model is evaluated based on its ability to accurately predict the output given the input data.  Common metrics for evaluating model performance include: accuracy, precision, recall, and F1 score for classification tasks; mean squared error (MSE) or mean absolute error (MAE) for regression tasks; negative log-likelihood per token (logarithm of perplexity) for language modeling.


Neural scaling law

Introduction Elo rating in a competition against other models, such as gameplay or preference by a human judgePerformance can be improved by using more data, larger models, different training algorithms, regularizing the model to prevent overfitting, and early stopping using a validation set.


Neural scaling law

Examples A 2020 analysis studied statistical relations between C , N , D , L {\displaystyle C,N,D,L} over a wide range of values and found similar scaling laws, over the range of N ��� [ 10 3 , 10 9 ] {\displaystyle N\in [10^{3},10^{9}]} , C ��� [ 10 12 , 10 21 ] {\displaystyle C\in [10^{12},10^{21}]} , and over multiple modalities (text, video, image, text to image, etc. ).  In particular, the scaling laws it found are (Table 1 of ): For each modality, they fixed one of the two C , N {\displaystyle C,N} , and varying the other one ( D {\displaystyle D} is varied along using D = C / 6 N {\displaystyle D=C/6N} ), the achievable test loss satisfies L = L 0 + ( x 0 x ) �� {\displaystyle L=L_{0}+\left({\frac {x_{0}}{x}}\right)^{\alpha }} where x {\displaystyle x} is the varied variable, and L 0 , x 0 , �� {\displaystyle L_{0},x_{0},\alpha } are parameters to be found by statistical fitting.  The parameter �� {\displaystyle \alpha } is the most important one.  When N {\displaystyle N} is the varied variable, �� {\displaystyle \alpha } ranges from 0.037 {\displaystyle 0.037} to 0.24 {\displaystyle 0.24} depending on the model modality.  This corresponds to the �� = 0.34 {\displaystyle \alpha =0.34} from the Chinchilla scaling paper.  When C {\displaystyle C} is the varied variable, �� {\displaystyle \alpha } ranges from 0.048 {\displaystyle 0.048} to 0.19 {\displaystyle 0.19} depending on the model modality.  This corresponds to the �� = 0.28 {\displaystyle \beta =0.28} from the Chinchilla scaling paper.  Given fixed computing budget, optimal model parameter count is consistently around N o p t ( C ) = ( C 5 �� 10 ��� 12 petaFLOP-day ) 0.7 = 9.0 �� 10 ��� 7 C 0.7 {\displaystyle N_{opt}(C)=\left({\frac {C}{5\times 10^{-12}{\text{petaFLOP-day}}}}\right)^{0.7}=9.0\times 10^{-7}C^{0.7}} The parameter 9.0 �� 10 ��� 7 {\displaystyle 9.0\times 10^{-7}} varies by a factor of up to 10 for different modalities.  The exponent parameter 0.7 {\displaystyle 0.7} varies from 0.64 {\displaystyle 0.64} to 0.75 {\displaystyle 0.75} for different modalities.  This exponent corresponds to the ��� 0.5 {\displaystyle \approx 0.5} from the Chinchilla scaling paper.  It's "strongly suggested" (but not statistically checked) that D o p t ( C ) ��� N o p t ( C ) 0.4 ��� C 0.28 {\displaystyle D_{opt}(C)\propto N_{opt}(C)^{0.4}\propto C^{0.28}} .  This exponent corresponds to the ��� 0.5 {\displaystyle \approx 0.5} from the Chinchilla scaling paper. When N {\displaystyle N} is the varied variable, �� {\displaystyle \alpha } ranges from 0.037 {\displaystyle 0.037} to 0.24 {\displaystyle 0.24} depending on the model modality.  This corresponds to the �� = 0.34 {\displaystyle \alpha =0.34} from the Chinchilla scaling paper.  When C {\displaystyle C} is the varied variable, �� {\displaystyle \alpha } ranges from 0.048 {\displaystyle 0.048} to 0.19 {\displaystyle 0.19} depending on the model modality.  This corresponds to the �� = 0.28 {\displaystyle \beta =0.28} from the Chinchilla scaling paper.


Neural scaling law

Examples The scaling law of L = L 0 + ( C 0 / C ) 0.048 {\displaystyle L=L_{0}+(C_{0}/C)^{0.048}} was confirmed during the training of GPT-3 (Figure 3.1 ).  One particular scaling law ("Chinchilla scaling") states that, for a large language model (LLM) autoregressively trained for one epoch, with a cosine learning rate schedule, we have: C {\displaystyle C} is the cost of training the model, in FLOPs.  N {\displaystyle N} is the number of parameters in the model.  D {\displaystyle D} is the number of tokens in the training set.  L {\displaystyle L} is the average negative log-likelihood loss per token (nats/token), achieved by the trained LLM on the test dataset.  L 0 {\displaystyle L_{0}} represents the loss of an ideal generative process on the test data A N �� {\displaystyle {\frac {A}{N^{\alpha }}}} captures the fact that a Transformer language model with N {\displaystyle N} parameters underperforms the ideal generative process B D �� {\displaystyle {\frac {B}{D^{\beta }}}} captures the fact that the model trained on D {\displaystyle D} tokens underperforms the ideal generative process L 0 {\displaystyle L_{0}} represents the loss of an ideal generative process on the test data A N �� {\displaystyle {\frac {A}{N^{\alpha }}}} captures the fact that a Transformer language model with N {\displaystyle N} parameters underperforms the ideal generative process B D �� {\displaystyle {\frac {B}{D^{\beta }}}} captures the fact that the model trained on D {\displaystyle D} tokens underperforms the ideal generative processand the statistical parameters are C 0 = 6 {\displaystyle C_{0}=6} , meaning that it costs 6 FLOPs per parameter to train on one token.  This is estimated by Kaplan et al.  Note that training cost is much higher than inference cost, as training entails both forward and backward passes, whereas inference costs 1 to 2 FLOPs per parameter to infer on one token.  �� = 0.34 , �� = 0.28 , A = 406.4 , B = 410.7 , L 0 = 1.69 {\displaystyle \alpha =0.34,\beta =0.28,A=406.4,B=410.7,L_{0}=1.69} . The statistical laws were fitted over experimental data with N ��� [ 7 �� 10 7 , 1.6 �� 10 10 ] , D ��� [ 5 �� 10 9 , 5 �� 10 11 ] , C ��� [ 10 18 , 10 24 ] {\displaystyle N\in [7\times 10^{7},1.6\times 10^{10}],D\in [5\times 10^{9},5\times 10^{11}],C\in [10^{18},10^{24}]} .  Since there are 4 variables related by 2 equations, imposing 1 additional constraint and 1 additional optimization objective allows us to solve for all four variables.  In particular, for any fixed C {\displaystyle C} , we can uniquely solve for all 4 variables that minimizes L {\displaystyle L} .  This provides us with the optimal D o p t ( C ) , N o p t ( C ) {\displaystyle D_{opt}(C),N_{opt}(C)} for any fixed C {\displaystyle C} :There are other estimates for "Chinchilla efficient" model size and training dataset size.


Neural scaling law

Examples The above is based on a statistical model of L = A N �� + B D �� + L 0 {\displaystyle L={\frac {A}{N^{\alpha }}}+{\frac {B}{D^{\beta }}}+L_{0}} .  One can also directly fit a statistical law for D o p t ( C ) , N o p t ( C ) {\displaystyle D_{opt}(C),N_{opt}(C)} without going through the detour, for which one obtains:In simpler terms, the Chinchilla scaling law for training Transformer language models suggests that when given an increased budget (in FLOPs), to achieve compute-optimal, the number of model parameters (N) and the number of tokens for training the model (D) should scale in approximately equal proportions.  This conclusion differs from the previous scaling law for neural language models, which states that N should be scaled faster than D.  The discrepancy arises from setting different cycle lengths for cosine learning rate schedulers.  In estimating the Chinchilla scaling, the authors set the cycle length to be the same as the training steps, as experimental results indicate that larger cycles overestimate the loss of the models.  A 2022 analysis found that many scaling behaviors of artificial neural networks follow a smoothly broken power law functional form: y = a + ( b x ��� c 0 ) ��� i = 1 n ( 1 + ( x d i ) 1 / f i ) ��� c i ��� f i {\displaystyle y=a+{\bigg (}bx^{-c_{0}}{\bigg )}\prod _{i=1}^{n}\left(1+\left({\frac {x}{d_{i}}}\right)^{1/f_{i}}\right)^{-c_{i}*f_{i}}} in which x {\displaystyle x} refers to the quantity being scaled (i. e.  C {\displaystyle C} , N {\displaystyle N} , D {\displaystyle D} , number of training steps, number of inference steps, or model input size) and y {\displaystyle y} refers to the downstream (or upstream) performance evaluation metric of interest (e. g.  prediction error, cross entropy, calibration error, AUROC, BLEU score percentage, F1 score, reward, Elo rating, solve rate, or FID score) in zero-shot, prompted, or fine-tuned settings.  The parameters a , b , c 0 , c 1 .  .  .  c n , d 1 .  .  .  d n , f 1 .  .  .  f n {\displaystyle a,b,c_{0},c_{1}. . . c_{n},d_{1}. . . d_{n},f_{1}. . . f_{n}} are found by statistical fitting.  On a log���log plot, when f i {\displaystyle f_{i}} is not too large and a {\displaystyle a} is subtracted out from the y-axis, this functional form looks like a series of linear segments connected by arcs; the n {\displaystyle n} transitions between the segments are called "breaks", hence the name Broken Neural Scaling Laws (BNSL).  The scenarios in which the scaling behaviors of artificial neural networks were found to follow this functional form include large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, out-of-distribution detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, arithmetic, emergent abilities, double descent, supervised learning, unsupervised/self-supervised learning, and reinforcement learning (single agent and multi-agent).


Neural scaling law

Examples The architectures for which the scaling behaviors of artificial neural networks were found to follow this functional form include ResNets, Transformers, MLPs, MLP-Mixers, Recurrent Neural Networks, Graph Neural Networks, U-Nets, Ensembles (and Non-Ensembles), MoE (Mixture of Experts) (and Non-MoE) Models, and Sparse Pruned (and Non-Sparse Unpruned) Models.  Vision transformers, similar to language transformers, exhibit scaling laws.  A 2022 research trained vision transformers, with parameter counts N ��� [ 5 �� 10 6 , 2 �� 10 9 ] {\displaystyle N\in [5\times 10^{6},2\times 10^{9}]} , on image sets of sizes D ��� [ 3 �� 10 7 , 3 �� 10 9 ] {\displaystyle D\in [3\times 10^{7},3\times 10^{9}]} , for computing C ��� [ 0.2 , 10 4 ] {\displaystyle C\in [0.2,10^{4}]} (in units of TPUv3-core-days).  After training the model, it is finetuned on ImageNet training set.  Let L {\displaystyle L} be the error probability of the finetuned model classifying ImageNet test set.  They found min N , D L = 0.09 + 0.26 ( C + 0.01 ) 0.35 {\displaystyle \min _{N,D}L=0.09+{\frac {0.26}{(C+0.01)^{0.35}}}} .  Ghorbani, Behrooz et al.  studied scaling laws for neural machine translation (specifically, English as source, and German as target) in encoder-decoder Transformer models, trained until convergence on the same datasets (thus they did not fit scaling laws for computing cost C {\displaystyle C} or dataset size D {\displaystyle D} ).  They varied N ��� [ 10 8 , 3.5 �� 10 9 ] {\displaystyle N\in [10^{8},3.5\times 10^{9}]} They found three results: L {\displaystyle L} is a scaling law function of N E , N D {\displaystyle N_{E},N_{D}} , where N E , N D {\displaystyle N_{E},N_{D}} are encoder and decoder parameter count.  It is not simply a function of total parameter count N = N E + N D {\displaystyle N=N_{E}+N_{D}} .  The function has form L ( N e , N d ) = �� ( N �� e N e ) p e ( N �� d N d ) p d + L ��� {\displaystyle L\left(N_{e},N_{d}\right)=\alpha \left({\frac {{\bar {N}}_{e}}{N_{e}}}\right)^{p_{e}}\left({\frac {{\bar {N}}_{d}}{N_{d}}}\right)^{p_{d}}+L_{\infty }} , where �� , p e , p d , L ��� , N �� e , N �� d {\displaystyle \alpha ,p_{e},p_{d},L_{\infty },{\bar {N}}_{e},{\bar {N}}_{d}} are fitted parameters.  They found that N d / N ��� 0.55 {\displaystyle N_{d}/N\approx 0.55} minimizes loss if N {\displaystyle N} is held fixed.  L {\displaystyle L} "saturates" (that is, it reaches L ��� {\displaystyle L_{\infty }} ) for smaller models when the training and testing datasets are "source-natural" than "target-natural".  A "source-natural" data point means a pair of English-German sentences, and the model is asked to translate the English sentence into German, and the English sentence is written by a natural English writer, while the German sentence is translated from the English sentence by a machine translator.  To construct the two kinds of datasets, the authors collected natural English and German sentences online, then used machine translation to generate their translations.


Neural scaling law

Examples As models grow larger, models trained on source-original datasets can achieve low loss but bad BLEU score.  In contrast, models trained on target-original datasets achieve low loss and good BLEU score in tandem (Figure 10, 11 ). The authors hypothesize that source-natural datasets have uniform and dull target sentences, and so a model that is trained to predict the target sentences would quickly overfit.  trained Transformers for machine translations with sizes N ��� [ 4 �� 10 5 , 5.6 �� 10 7 ] {\displaystyle N\in [4\times 10^{5},5.6\times 10^{7}]} on dataset sizes D ��� [ 6 �� 10 5 , 6 �� 10 9 ] {\displaystyle D\in [6\times 10^{5},6\times 10^{9}]} .  They found the Kaplan et al (2020) scaling law applied to machine translation: L ( N , D ) = [ ( N C N ) �� N �� D + D C D ] �� D {\displaystyle L(N,D)=\left[\left({\frac {N_{C}}{N}}\right)^{\frac {\alpha _{N}}{\alpha _{D}}}+{\frac {D_{C}}{D}}\right]^{\alpha _{D}}} .  They also found the BLEU score scaling as B L E U ��� C e ��� k L {\displaystyle BLEU\approx Ce^{-kL}} .  Hernandez, Danny et al.  studied scaling laws for transfer learning in language models.  They trained a family of Transformers in three ways: pretraining on English, finetuning on Python pretraining on an equal mix of English and Python, finetuning on Python training on PythonThe idea is that pretraining on English should help the model achieve low loss on a test set of Python text.  Suppose the model has parameter count N {\displaystyle N} , and after being finetuned on D F {\displaystyle D_{F}} Python tokens, it achieves some loss L {\displaystyle L} .  We say that its "transferred token count" is D T {\displaystyle D_{T}} , if another model with the same N {\displaystyle N} achieves the same L {\displaystyle L} after training on D F + D T {\displaystyle D_{F}+D_{T}} Python tokens.  They found D T = 1.9 e 4 ( D F ) .18 ( N ) .38 {\displaystyle D_{T}=1.9e4\left(D_{F}\right)^{.18}(N)^{.38}} for pretraining on English text, and D T = 2.1 e 5 ( D F ) .096 ( N ) .38 {\displaystyle D_{T}=2.1e5\left(D_{F}\right)^{.096}(N)^{.38}} for pretraining on English and non-Python code.


Vision transformer

Vision transformer A Vision Transformer (ViT) is a transformer designed for computer vision. Transformers were introduced in 2017, and have found widespread use in Natural Language Processing. In 2020, they were adapted for computer vision, yielding ViT. The basic structure is to break down input images as a series of patches, then tokenized, before applying the tokens to a standard Transformer architecture. The attention mechanism in a ViT repeatedly transforms representation vectors of image patches, incorporating more and more semantic relations between image patches in an image. This is analogous to how in NLP, as representation vectors flow through a Transformers, they incorporate more and more semantic relations between words, from syntax to semantics. ViT has found applications in image recognition, image segmentation, and autonomous driving.


Vision transformer

Architecture The basic architecture, used by the original 2020 paper, is as follows. In summary, it is a BERT-like encoder-only Transformer. The input image is of type R H �� W �� C {\displaystyle \mathbb {R} ^{H\times W\times C}} , where H , W , C {\displaystyle H,W,C} are height, width, channel (RGB). It is then split into square-shaped patches of type R P �� P �� C {\displaystyle \mathbb {R} ^{P\times P\times C}} . For each patch, the patch is pushed through a linear operator, to obtain a vector ("patch embedding"). The position of the patch is also transformed into a vector by "position encoding". The two vectors are added, then pushed through several Transformer encoders. The above architecture turns an image into a sequence of vector representations. To use the vector representation for downstream applications, one needs to add some network modules on top of it. For example, to use it for classification, one can add a shallow MLP on top of it that outputs a probability distribution over classes. The original paper uses a linear-GeLU-linear-softmax network.


Vision transformer

Vision_Transformers Transformers found their initial applications in natural language processing (NLP) tasks, as demonstrated by language models such as BERT and GPT-3. By contrast the typical image processing system uses a convolutional neural network (CNN). Well-known projects include Xception, ResNet, EfficientNet, DenseNet, and Inception. Transformers measure the relationships between pairs of input tokens (words in the case of text strings), termed attention. The cost is quadratic in the number of tokens. For images, the basic unit of analysis is the pixel. However, computing relationships for every pixel pair in a typical image is prohibitive in terms of memory and computation. Instead, ViT computes relationships among pixels in various small sections of the image (e.g., 16x16 pixels), at a drastically reduced cost. The sections (with positional embeddings) are placed in a sequence. The embeddings are learnable vectors. Each section is arranged into a linear sequence and multiplied by the embedding matrix. The result, with the position embedding is fed to the transformer. As in the case of BERT, a fundamental role in classification tasks is played by the class token. A special token that is used as the only input of the final MLP Head as it has been influenced by all the others. The architecture for image classification is the most common and uses only the Transformer Encoder in order to transform the various input tokens. However, there are also other applications in which the decoder part of the traditional Transformer Architecture is also used.


Vision transformer

History In 2021 a pure transformer model demonstrated better performance and greater efficiency than CNNs on image classification. A study in June 2021 added a transformer backend to ResNet, which dramatically reduced costs and increased accuracy. In the same year, some important variants of the Vision Transformers were proposed. These variants are mainly intended to be more efficient, more accurate or better suited to a specific domain. Among the most relevant is the Swin Transformer, which through some modifications to the attention mechanism and a multi-stage approach achieved state-of-the-art results on some object detection datasets such as COCO. Another interesting variant is the TimeSformer, designed for video understanding tasks and able to capture spatial and temporal information through the use of divided space-time attention.


Vision transformer

Comparison_with_Convolutional_Neural_Networks Due to the commonly used (comparatively) large patch size, ViT performance depends more heavily on decisions including that of the optimizer, dataset-specific hyperparameters, and network depth than convolutional networks. Preprocessing with a layer of smaller-size, overlapping (stride < size) convolutional filters helps with performance and stability. The CNN translates from the basic pixel level to a feature map. A tokenizer translates the feature map into a series of tokens that are then fed into the transformer, which applies the attention mechanism to produce a series of output tokens. Finally, a projector reconnects the output tokens to the feature map. The latter allows the analysis to exploit potentially significant pixel-level details. This drastically reduces the number of tokens that need to be analyzed, reducing costs accordingly. The differences between CNNs and Vision Transformers are many and lie mainly in their architectural differences. In fact, CNNs achieve excellent results even with training based on data volumes that are not as large as those required by Vision Transformers. This different behaviour seems to derive from the different inductive biases they possess. The filter-oriented architecture of CNNs can be somehow exploited by these networks to grasp more quickly the particularities of the analysed images even if, on the other hand, they end up limiting them making it more complex to grasp global relations. On the other hand, the Vision Transformers possess a different kind of bias toward exploring topological relationships between patches, which leads them to be able to capture also global and wider range relations but at the cost of a more onerous training in terms of data. Vision Transformers also proved to be much more robust to input image distortions such as adversarial patches or permutations. However, choosing one architecture over another is not always the wisest choice, and excellent results have been obtained in several Computer Vision tasks through hybrid architectures combining convolutional layers with Vision Transformers.


Vision transformer

The_Role_of_Self-Supervised_Learning The considerable need for data during the training phase has made it essential to find alternative methods to train these models, and a central role is now played by self-supervised methods. Using these approaches, it is possible to train a neural network in an almost autonomous way, allowing it to deduce the peculiarities of a specific problem without having to build a large dataset or provide it with accurately assigned labels. Being able to train a Vision Transformer without having to have a huge vision dataset at its disposal could be the key to the widespread dissemination of this promising new architecture.


Vision transformer

Applications Vision Transformers have been used in many Computer Vision tasks with excellent results and in some cases even state-of-the-art. Among the most relevant areas of application are: Image Classification Object Detection Video Deepfake Detection Image segmentation Anomaly detection Image Synthesis Cluster analysis Autonomous Driving


Vision transformer

Implementations There are many implementations of Vision Transformers and its variants available in open source online. The main versions of this architecture have been implemented in PyTorch but implementations have also been made available for TensorFlow.


Vision transformer

See_also Transformer (machine learning model) Attention (machine learning) Perceiver Deep learning PyTorch TensorFlow


Open-source artificial intelligence

Open-source artificial intelligence Open-source artificial intelligence is the application of open source practices to the development of artificial intelligence resources. Many open-source artificial intelligence products are variations of other existing tools and technology which major companies have shared as open-source software. Companies often developed closed products in an attempt to keep a competitive advantage in the marketplace. A journalist for Wired explored the idea that open-source AI tools have a development advantage over closed products, and could overtake them in the marketplace. Popular open-source artificial intelligence project categories include large language models, Machine translation tools, and chatbots. For software developers to produce open-source artificial intelligence resources, they must trust the various other open-source software components they use in its development.


Open-source artificial intelligence

Large_Language_Models LLaMA (Large Language Model Meta AI) is a family of large language models (LLMs), released by Meta AI starting in February 2023.


Ethics of artificial intelligence

Approaches Machine ethics (or machine morality) is the field of research concerned with designing Artificial Moral Agents (AMAs), robots or artificially intelligent computers that behave morally or as though moral.  To account for the nature of these agents, it has been suggested to consider certain philosophical ideas, like the standard characterizations of agency, rational agency, moral agency, and artificial agency, which are related to the concept of AMAs.  Isaac Asimov considered the issue in the 1950s in his I, Robot.  At the insistence of his editor John W.  Campbell Jr. , he proposed the Three Laws of Robotics to govern artificially intelligent systems.  Much of his work was then spent testing the boundaries of his three laws to see where they would break down, or where they would create paradoxical or unanticipated behavior.  His work suggests that no set of fixed laws can sufficiently anticipate all possible circumstances.  More recently, academics and many governments have challenged the idea that AI can itself be held accountable.  A panel convened by the United Kingdom in 2010 revised Asimov's laws to clarify that AI is the responsibility either of its manufacturers, or of its owner/operator.  In 2009, during an experiment at the Laboratory of Intelligent Systems in the Ecole Polytechnique F��d��rale of Lausanne, Switzerland, robots that were programmed to cooperate with each other (in searching out a beneficial resource and avoiding a poisonous one) eventually learned to lie to each other in an attempt to hoard the beneficial resource.  Some experts and academics have questioned the use of robots for military combat, especially when such robots are given some degree of autonomous functions.  The US Navy has funded a report which indicates that as military robots become more complex, there should be greater attention to implications of their ability to make autonomous decisions.  The President of the Association for the Advancement of Artificial Intelligence has commissioned a study to look at this issue.  They point to programs like the Language Acquisition Device which can emulate human interaction.  Vernor Vinge has suggested that a moment may come when some computers are smarter than humans.  He calls this "the Singularity".  He suggests that it may be somewhat or possibly very dangerous for humans.  This is discussed by a philosophy called Singularitarianism.  The Machine Intelligence Research Institute has suggested a need to build "Friendly AI", meaning that the advances which are already occurring with AI should also include an effort to make AI intrinsically friendly and humane.  There are discussions on creating tests to see if an AI is capable of making ethical decisions.  Alan Winfield concludes that the Turing test is flawed and the requirement for an AI to pass the test is too low.  A proposed alternative test is one called the Ethical Turing Test, which would improve on the current test by having multiple judges decide if the AI's decision is ethical or unethical.


Ethics of artificial intelligence

Approaches In 2009, academics and technical experts attended a conference organized by the Association for the Advancement of Artificial Intelligence to discuss the potential impact of robots and computers, and the impact of the hypothetical possibility that they could become self-sufficient and able to make their own decisions.  They discussed the possibility and the extent to which computers and robots might be able to acquire any level of autonomy, and to what degree they could use such abilities to possibly pose any threat or hazard.  They noted that some machines have acquired various forms of semi-autonomy, including being able to find power sources on their own and being able to independently choose targets to attack with weapons.  They also noted that some computer viruses can evade elimination and have achieved "cockroach intelligence".  They noted that self-awareness as depicted in science-fiction is probably unlikely, but that there were other potential hazards and pitfalls.  However, there is one technology in particular that has the potential to make the idea of morally capable robots a reality.  In a paper on the acquisition of moral values by robots, Nayef Al-Rodhan mentions the case of neuromorphic chips, which aim to process information similarly to humans, nonlinearly and with millions of interconnected artificial neurons.  Robots embedded with neuromorphic technology could learn and develop knowledge in a uniquely humanlike way.  Inevitably, this raises the question of the environment in which such robots would learn about the world and whose morality they would inherit ��� or if they end up developing human 'weaknesses' as well: selfishness, a pro-survival attitude, hesitation etc.  In Moral Machines: Teaching Robots Right from Wrong, Wendell Wallach and Colin Allen conclude that attempts to teach robots right from wrong will likely advance understanding of human ethics by motivating humans to address gaps in modern normative theory and by providing a platform for experimental investigation.  As one example, it has introduced normative ethicists to the controversial issue of which specific learning algorithms to use in machines.  Nick Bostrom and Eliezer Yudkowsky have argued for decision trees (such as ID3) over neural networks and genetic algorithms on the grounds that decision trees obey modern social norms of transparency and predictability (e. g.  stare decisis), while Chris Santos-Lang argued in the opposite direction on the grounds that the norms of any age must be allowed to change and that natural failure to fully satisfy these particular norms has been essential in making humans less vulnerable to criminal "hackers".  According to a 2019 report from the Center for the Governance of AI at the University of Oxford, 82% of Americans believe that robots and AI should be carefully managed.  Concerns cited ranged from how AI is used in surveillance and in spreading fake content online (known as deep fakes when they include doctored video images and audio generated with help from AI) to cyberattacks, infringements on data privacy, hiring bias, autonomous vehicles, and drones that do not require a human controller.


Ethics of artificial intelligence

Approaches Similarly, according to a five-country study by KPMG and the University of Queensland Australia in 2021, 66-79% of citizens in each country believe that the impact of AI on society is uncertain and unpredictable; 96% of those surveyed expect AI governance challenges to be managed carefully.  The term "robot ethics" (sometimes "roboethics") refers to the morality of how humans design, construct, use and treat robots.  Robot ethics intersect with the ethics of AI.  Robots are physical machines whereas AI can be only software.  Not all robots function through AI systems and not all AI systems are robots.  Robot ethics considers how machines may be used to harm or benefit humans, their impact on individual autonomy, and their effects on social justice.  In the review of 84 ethics guidelines for AI, 11 clusters of principles were found: transparency, justice and fairness, non-maleficence, responsibility, privacy, beneficence, freedom and autonomy, trust, sustainability, dignity, solidarity.  Luciano Floridi and Josh Cowls created an ethical framework of AI principles set by four principles of bioethics (beneficence, non-maleficence, autonomy and justice) and an additional AI enabling principle ��� explicability.  Bill Hibbard argues that because AI will have such a profound effect on humanity, AI developers are representatives of future humanity and thus have an ethical obligation to be transparent in their efforts.  Ben Goertzel and David Hart created OpenCog as an open source framework for AI development.  OpenAI is a non-profit AI research company created by Elon Musk, Sam Altman and others to develop open-source AI beneficial to humanity.  There are numerous other open-source AI developments.  Unfortunately, making code open source does not make it comprehensible, which by many definitions means that the AI code is not transparent.  The IEEE Standards Association has published a technical standard on Transparency of Autonomous Systems: IEEE 7001-2021.  The IEEE effort identifies multiple scales of transparency for different stakeholders.  Further, there is concern that releasing the full capacity of contemporary AI to some organizations may be a public bad, that is, do more damage than good.  For example, Microsoft has expressed concern about allowing universal access to its face recognition software, even for those who can pay for it.  Microsoft posted an extraordinary blog on this topic, asking for government regulation to help determine the right thing to do.  Not only companies, but many other researchers and citizen advocates recommend government regulation as a means of ensuring transparency, and through it, human accountability.  This strategy has proven controversial, as some worry that it will slow the rate of innovation.  Others argue that regulation leads to systemic stability more able to support innovation in the long term.  The OECD, UN, EU, and many countries are presently working on strategies for regulating AI, and finding appropriate legal frameworks.  On June 26, 2019, the European Commission High-Level Expert Group on Artificial Intelligence (AI HLEG) published its "Policy and investment recommendations for trustworthy Artificial Intelligence".  This is the AI HLEG's second deliverable, after the April 2019 publication of the "Ethics Guidelines for Trustworthy AI".


Ethics of artificial intelligence

Approaches The June AI HLEG recommendations cover four principal subjects: humans and society at large, research and academia, the private sector, and the public sector.  The European Commission claims that "HLEG's recommendations reflect an appreciation of both the opportunities for AI technologies to drive economic growth, prosperity and innovation, as well as the potential risks involved" and states that the EU aims to lead on the framing of policies governing AI internationally.  To prevent harm, in addition to regulation, AI-deploying organizations need to play a central role in creating and deploying trustworthy AI in line with the principles of trustworthy AI, and take accountability to mitigate the risks.  On 21 April 2021, the European Commission proposed the Artificial Intelligence Act.


Ethics of artificial intelligence

Ethical_challenges AI has become increasingly inherent in facial and voice recognition systems.  Some of these systems have real business applications and directly impact people.  These systems are vulnerable to biases and errors introduced by its human creators.  Also, the data used to train these AI systems itself can have biases.  For instance, facial recognition algorithms made by Microsoft, IBM and Face++ all had biases when it came to detecting people's gender; these AI systems were able to detect gender of white men more accurately than gender of darker skin men.  Further, a 2020 study reviewed voice recognition systems from Amazon, Apple, Google, IBM, and Microsoft found that they have higher error rates when transcribing black people's voices than white people's.  Furthermore, Amazon terminated their use of AI hiring and recruitment because the algorithm favored male candidates over female ones.  This was because Amazon's system was trained with data collected over 10-year period that came mostly from male candidates.  Bias can creep into algorithms in many ways.  The most predominant view on how bias is introduced into AI systems is that it is embedded within the historical data used to train the system.  For instance, Amazon's AI-powered recruitment tool was trained with its own recruitment data accumulated over the years, during which time the candidates that successfully got the job were mostly white males.  Consequently, the algorithms learned the (biased) pattern from the historical data and generated predictions for the present/future that these types of candidates are most likely to succeed in getting the job.  Therefore, the recruitment decisions made by the AI system turn out to be biased against female and minority candidates.  Friedman and Nissenbaum identify three categories of bias in computer systems: existing bias, technical bias, and emergent bias.  In natural language processing, problems can arise from the text corpus ��� the source material the algorithm uses to learn about the relationships between different words.  Large companies such as IBM, Google, etc.  have made efforts to research and address these biases.  One solution for addressing bias is to create documentation for the data used to train AI systems.  Process mining can be an important tool for organizations to achieve compliance with proposed AI regulations by identifying errors, monitoring processes, identifying potential root causes for improper execution, and other functions.  The problem of bias in machine learning is likely to become more significant as the technology spreads to critical areas like medicine and law, and as more people without a deep technical understanding are tasked with deploying it.  Some experts warn that algorithmic bias is already pervasive in many industries and that almost no one is making an effort to identify or correct it.  There are some open-sourced tools by civil societies that are looking to bring more awareness to biased AI.  "Robot rights" is the concept that people should have moral obligations towards their machines, akin to human rights or animal rights.


Ethics of artificial intelligence

Ethical_challenges It has been suggested that robot rights (such as a right to exist and perform its own mission) could be linked to robot duty to serve humanity, analogous to linking human rights with human duties before society.  These could include the right to life and liberty, freedom of thought and expression, and equality before the law.  The issue has been considered by the Institute for the Future and by the U. K.  Department of Trade and Industry.  Experts disagree on how soon specific and detailed laws on the subject will be necessary.  Glenn McGee reported that sufficiently humanoid robots might appear by 2020, while Ray Kurzweil sets the date at 2029.  Another group of scientists meeting in 2007 supposed that at least 50 years had to pass before any sufficiently advanced system would exist.  The rules for the 2003 Loebner Prize competition envisioned the possibility of robots having rights of their own: 61.  If in any given year, a publicly available open-source Entry entered by the University of Surrey or the Cambridge Center wins the Silver Medal or the Gold Medal, then the Medal and the Cash Award will be awarded to the body responsible for the development of that Entry.  If no such body can be identified, or if there is disagreement among two or more claimants, the Medal and the Cash Award will be held in trust until such time as the Entry may legally possess, either in the United States of America or in the venue of the contest, the Cash Award and Gold Medal in its own right.  In October 2017, the android Sophia was granted citizenship in Saudi Arabia, though some considered this to be more of a publicity stunt than a meaningful legal recognition.  Some saw this gesture as openly denigrating of human rights and the rule of law.  The philosophy of Sentientism grants degrees of moral consideration to all sentient beings, primarily humans and most non-human animals.  If artificial or alien intelligence show evidence of being sentient, this philosophy holds that they should be shown compassion and granted rights.  Joanna Bryson has argued that creating AI that requires rights is both avoidable, and would in itself be unethical, both as a burden to the AI agents and to human society.  In 2020, professor Shimon Edelman noted that only a small portion of work in the rapidly growing field of AI ethics addressed the possibility of AIs experiencing suffering.  This was despite credible theories having outlined possible ways by which AI systems may became conscious, such as Integrated information theory.  Edelman notes one exception had been Thomas Metzinger, who in 2018 called for a global moratorium on further work that risked creating conscious AIs.  The moratorium was to run to 2050 and could be either extended or repealed early, depending on progress in better understanding the risks and how to mitigate them.


Ethics of artificial intelligence

Ethical_challenges Metzinger repeated this argument in 2021, highlighting the risk of creating an "explosion of artificial suffering", both as an AI might suffer in intense ways that humans could not understand, and as replication processes may see the creation of huge quantities of artificial conscious instances.  Several labs have openly stated they are trying to create conscious AIs.  There have been reports from those with close access to AIs not openly intended to be self aware, that consciousness may already have unintentionally emerged.  These include OpenAI founder Ilya Sutskever in February 2022, when he wrote that today's large neural nets may be "slightly conscious".  In November 2022, David Chalmers argued that it was unlikely current large language models like GPT-3 had experienced consciousness, but also that he considered there to be a serious possibility that large language models may become conscious in the future.  Joseph Weizenbaum argued in 1976 that AI technology should not be used to replace people in positions that require respect and care, such as: A customer service representative (AI technology is already used today for telephone-based interactive voice response systems) A nursemaid for the elderly (as was reported by Pamela McCorduck in her book The Fifth Generation) A soldier A judge A police officer A therapist (as was proposed by Kenneth Colby in the 70s)Weizenbaum explains that we require authentic feelings of empathy from people in these positions.  If machines replace them, we will find ourselves alienated, devalued and frustrated, for the artificially intelligent system would not be able to simulate empathy.  Artificial intelligence, if used in this way, represents a threat to human dignity.  Weizenbaum argues that the fact that we are entertaining the possibility of machines in these positions suggests that we have experienced an "atrophy of the human spirit that comes from thinking of ourselves as computers. " Pamela McCorduck counters that, speaking for women and minorities "I'd rather take my chances with an impartial computer", pointing out that there are conditions where we would prefer to have automated judges and police that have no personal agenda at all.  However, Kaplan and Haenlein stress that AI systems are only as smart as the data used to train them since they are, in their essence, nothing more than fancy curve-fitting machines; Using AI to support a court ruling can be highly problematic if past rulings show bias toward certain groups since those biases get formalized and engrained, which makes them even more difficult to spot and fight against.  Weizenbaum was also bothered that AI researchers (and some philosophers) were willing to view the human mind as nothing more than a computer program (a position now known as computationalism).  To Weizenbaum, these points suggest that AI research devalues human life.  AI founder John McCarthy objects to the moralizing tone of Weizenbaum's critique.  "When moralizing is both vehement and vague, it invites authoritarian abuse," he writes.


Ethics of artificial intelligence

Ethical_challenges Bill Hibbard writes that "Human dignity requires that we strive to remove our ignorance of the nature of existence, and AI is necessary for that striving. " As the widespread use of autonomous cars becomes increasingly imminent, new challenges raised by fully autonomous vehicles must be addressed.  Recently,[when? ] there has been debate as to the legal liability of the responsible party if these cars get into accidents.  In one report where a driverless car hit a pedestrian, the driver was inside the car but the controls were fully in the hand of computers.  This led to a dilemma over who was at fault for the accident.  In another incident on March 18, 2018, Elaine Herzberg was struck and killed by a self-driving Uber in Arizona.  In this case, the automated car was capable of detecting cars and certain obstacles in order to autonomously navigate the roadway, but it could not anticipate a pedestrian in the middle of the road.  This raised the question of whether the driver, pedestrian, the car company, or the government should be held responsible for her death.  Currently, self-driving cars are considered semi-autonomous, requiring the driver to pay attention and be prepared to take control if necessary. [failed verification] Thus, it falls on governments to regulate the driver who over-relies on autonomous features.  as well educate them that these are just technologies that, while convenient, are not a complete substitute.  Before autonomous cars become widely used, these issues need to be tackled through new policies.  Some experts and academics have questioned the use of robots for military combat, especially when such robots are given some degree of autonomy.  On October 31, 2019, the United States Department of Defense's Defense Innovation Board published the draft of a report recommending principles for the ethical use of artificial intelligence by the Department of Defense that would ensure a human operator would always be able to look into the 'black box' and understand the kill-chain process.  However, a major concern is how the report will be implemented.  The US Navy has funded a report which indicates that as military robots become more complex, there should be greater attention to implications of their ability to make autonomous decisions.  Some researchers state that autonomous robots might be more humane, as they could make decisions more effectively.  Within this last decade, there has been intensive research in autonomous power with the ability to learn using assigned moral responsibilities.  "The results may be used when designing future military robots, to control unwanted tendencies to assign responsibility to the robots. " From a consequentialist view, there is a chance that robots will develop the ability to make their own logical decisions on whom to kill and that is why there should be a set moral framework that the AI cannot override.  There has been a recent outcry with regard to the engineering of artificial intelligence weapons that have included ideas of a robot takeover of mankind.