Writing / Machine learning notes / Regression & Gaussian classification

Statistical regression and
Gaussian classification

Learn a distribution from data, turn it into a prediction, and derive the boundary between classes. Follow the path from Gaussian conditioning to least squares and discriminant analysis.

The probability and Gaussian chapter explained what a model implies when its parameters are known. Now we learn those parameters from observations. The same Gaussian machinery will predict a numerical response in regression and a discrete label in classification.

This chapter combines the statistical-regression lecture and its Gaussian discriminant analysis continuation. It expands the estimation calculations, the handwritten question about joint versus conditional likelihood, and the decision-boundary derivation. Gaussian mixtures appear as a limitation of a single bell-shaped model. Logistic regression, unequal covariances, and cost-sensitive decisions are clearly identified connections beyond the core shared-covariance classifier.

Reading route: sections 1–6 build statistical estimation; 7–10 complete regression and its worked example. Sections 11–12 explain mixtures and the logistic connection. Sections 13–18 develop Gaussian classification. Work through one route at a time and return to the revision sheet after attempting the problems.

1. A model, a fitting rule, and a prediction rule

Lecture connection: the statistical/non-statistical model-building map and generative/discriminative branches.

A supervised dataset consists of pairs \(D=\{(x_i,y_i)\}_{i=1}^N\), where the input is a column vector \(x_i\in\mathbb R^d\). For regression, yᵢ is a real number; for the binary classifier later, yᵢ is 0 or 1. N counts observations, while d counts features. A parameter θ describes what the model learns. We use different letters for specialized parameters once their roles are clear.

There are three separate choices. A model family states what distributions or functions are possible. A fitting criterion chooses parameters from the data. A prediction rule turns the fitted model into an action. A straight line can be fitted by squared error without first stating a noise distribution. Alternatively, a Gaussian-noise model can lead to that same squared-error criterion by maximum likelihood. The algebra may coincide while the interpretation and additional predictions differ.

A generative approach models a joint distribution. In classification it commonly uses p(x|y) together with p(y): how features vary within a class and how frequent each class is. These define p(x,y)=p(x|y)p(y), from which Bayes' rule gives p(y|x). A discriminative approach can model p(y|x) directly without specifying the feature distribution p(x). Discriminative methods can also learn a decision function without a full conditional probability model.

A generative model can also generate synthetic pairs: sample a label from its prior, then sample features from the corresponding class density. A conditional predictor can generate a response given an input, but it does not thereby supply a model for generating the inputs. Neither label promises that its assumptions match a particular dataset.

2. Maximum likelihood: vary parameters, hold observations fixed

Lecture connection: estimating parameters through products over independent examples.

For independent and identically distributed observations \(z_1,\ldots,z_N\), a likelihood is the joint mass or density evaluated at the observed data, regarded as a function of θ:

“Independent” justifies the product; “identically distributed” says each observation uses the same distributional parameters. Dependencies across examples require a different factorization. A continuous observation contributes a density, not a positive probability of exactly that point. A likelihood is not a normalized probability distribution over θ. A Bayesian posterior over parameters would additionally need a parameter prior and normalization.

Taking logarithms changes multiplication into addition. Because log is strictly increasing on positive values, it preserves the maximizer:

This is also numerically useful: products of many small densities can underflow. Minimizing the negative log-likelihood gives the same answer. A term can be dropped only if it is constant in the variable being optimized. A density's normalizer may depend on a variance that is being learned.

Write the conditional direction explicitly

These are three different objectives. The first can be factored as either conditional plus the relevant marginal. Fixing observed xᵢ does not make pθ(xᵢ) constant in θ. Later we will find a particular Gaussian parameterization in which joint fitting separates, and a particular GDA parameterization in which fitting class densities separates from fitting class priors. Those are properties to establish, not consequences of merely writing Bayes' rule.

3. A joint Gaussian becomes a regression model

Lecture connection: the joint model for [Y,X], the parameter-count annotation, and Gaussian conditioning.

Assume that a scalar response Y and d-dimensional input X are jointly Gaussian:

The cross-covariance ΣYX is a 1×d row; ΣXY is its d×1 transpose. Assume the full covariance is positive definite. Then ΣXX is invertible and the conditional Gaussian from the previous chapter gives:

Define a d×1 coefficient vector \(\beta=\Sigma_{XX}^{-1}\Sigma_{XY}\), and intercept \(a=\mu_Y-\beta^\top\mu_X\). The conditional mean becomes \(m(x)=a+\beta^\top x\). A linear prediction function has emerged from a distributional assumption. The conditional variance is constant in x in this model, while the mean moves with x.

The intercept matters: “linear regression” commonly includes this affine offset. It can be folded into a dot product by augmenting each input with a leading 1. The joint model has d+1 means and (d+1)(d+2)/2 distinct covariance entries. The latter count is 1 response variance, d cross-covariances, and d(d+1)/2 input-covariance entries. Symmetry means we do not learn both copies of each off-diagonal entry.

The earlier additive-noise model \(Y=a+\beta^\top X+\varepsilon\), with ε independent of X and ε∼N(0,τ²), gives the same conditional Gaussian. It does not require X itself to be Gaussian. Joint Gaussianity is the stronger assumption used when we also model the distribution of X.

4. Why predict the conditional mean?

Lecture connection: predicting by the peak of a conditional density and the sketches of non-Gaussian conditional shapes.

A fitted distribution is not yet a unique prediction rule. Under squared-error loss, choose a number c minimizing the conditional expected loss. Let m=E[Y|X=x]. Expanding around m gives:

The cross term vanishes because E[Y−m|X=x]=0. The variance term does not change with c, so the minimum is attained at c=m. This argument needs a finite second moment but does not need a Gaussian. Absolute-error loss instead leads to a conditional median. Different losses answer different prediction questions.

The lecture also writes a density maximizer: \(\operatorname*{arg\,max}_y p(y\mid x)\). This selects a conditional mode. For an ordinary Gaussian, mode, mean, and median coincide, so both routes give a+βᵀx. For a multimodal distribution, the mean can lie between peaks. Maximizing a continuous density is not the same as maximizing the probability of an exact real-valued response, which is zero in a density model.

If parameters were known, a Gaussian conditional puts about 95% of future responses in m(x)±1.96τ. Substituting fitted parameters gives a plug-in conditional range. It does not account for uncertainty in the fitted mean or variance and is not a confidence interval for a regression coefficient. The figures use it only to visualize residual response variation.

5. Estimate a Gaussian mean from data

Lecture connection: the multivariate Gaussian maximum-likelihood slide, repeated in both lectures.

Temporarily use zᵢ∈ℝᵖ for any p-dimensional Gaussian observation. For regression, zᵢ=[yᵢ,xᵢᵀ]ᵀ and p=d+1. Assume independent observations with common mean μ and positive-definite covariance Σ. Taking logs of their Gaussian densities gives:

The determinant controls how much volume the model assigns to its spread. The quadratic term measures how far the data fall from the proposed mean relative to that spread. Both terms depend on Σ; keeping only the quadratic would let increasing the covariance avoid distance penalties without the corresponding volume penalty.

For a symmetric matrix M, differentiating (z−μ)ᵀM(z−μ) with respect to μ gives −2M(z−μ). This follows by expanding the quadratic into zᵀMz−2μᵀMz+μᵀMμ. Thus:

The Hessian in μ is −NΣ⁻¹, negative definite for positive-definite Σ. Therefore the sample mean is the unique maximizing mean for every fixed allowed covariance. This also means we can substitute μ̂ before optimizing the covariance.

6. Derive the covariance estimate using a trace

Lecture connection: the trace trick, derivative with respect to precision, and the covariance estimate.

The trace converts a quadratic form into a matrix inner product

The trace tr(A) is the sum of A's diagonal entries. For compatible matrices, tr(AB)=tr(BA); a scalar equals its own trace. If r is a p×1 residual, then:

The outer product rrᵀ is p×p. It is not the scalar squared length rᵀr and is not an elementwise square. This is the precise matrix meaning of the squared-residual shorthand in the handwritten covariance formula. Let \(T=\sum_i(z_i-\bar z)(z_i-\bar z)^\top\). Since log|Σ|=−log|Λ|, the variable part of the likelihood is:

Two matrix-derivative rules are enough: the derivative of log|Λ| is Λ⁻ᵀ, and the derivative of tr(ΛT) is Tᵀ. The first follows from the differential d log|Λ|=tr(Λ⁻¹ dΛ); the second follows by expanding the trace entry by entry. For symmetric Λ and T, the rules simplify to ordinary inverses and untransposed T.

To see the log-determinant rule, perturb Λ by tH. Factor the determinant as \(|\Lambda+tH|=|\Lambda|\,|I+t\Lambda^{-1}H|\). Expanding a determinant near I gives \(|I+tM|=1+t\operatorname{tr}(M)+O(t^2)\): its first-order contributions are the diagonal changes. Taking the log and differentiating at t=0 yields \(\frac{d}{dt}\log|\Lambda+tH|\big|_{t=0}=\operatorname{tr}(\Lambda^{-1}H)\). This directional derivative is what the matrix gradient represents.

Why this stationary point is the maximum, when it exists

If S=T/N is positive definite, put C=S¹ᐟ²ΛS¹ᐟ². The symmetric square root S¹ᐟ² keeps S's eigenvectors and takes positive square roots of its eigenvalues. Apart from a constant, the precision objective is (N/2)(log|C|−tr(C)). For each positive eigenvalue λ of C, log λ−λ is maximized uniquely at λ=1: its derivative is 1/λ−1 and its second derivative is negative. Hence C=I, Λ=S⁻¹, and Σ=S is the global maximum.

If S is singular, there is no positive-definite covariance MLE in this unrestricted full-dimensional model. Shrinking covariance in a direction with no observed residual variation can make the likelihood unbounded. Reporting S as a sample covariance is still meaningful, but inserting it into an inverse/determinant density is not. The centered scatter has rank at most N−1, so N≥p+1 is necessary for full rank; the points must also span all p directions after centering.

Why the denominator is N rather than N−1

Maximum likelihood and unbiased estimation are different criteria. With an unknown mean, the likelihood calculation gives T/N. For independent observations with finite population covariance, E[T]=(N−1)Σ; therefore T/(N−1) is an unbiased covariance estimator. The likelihood estimate is biased downward at finite N. Neither denominator should be silently substituted for the other. Regression residual variance will have another degrees-of-freedom correction because it fits more than one coefficient.

7. Fit a complete small Gaussian regression example

The following four observations are invented for transparent arithmetic:

The fitted means are μ̂X=1.5 and μ̂Y=2.5. The centered sums are ∑(xᵢ−x̄)²=5, ∑(yᵢ−ȳ)²=9, and ∑(xᵢ−x̄)(yᵢ−ȳ)=6. Dividing by N=4 gives SXX=1.25, SYY=2.25, and SYX=1.5. In [Y,X] order:

Conditioning the fitted joint Gaussian gives:

The fitted mean is ŷ=0.7+1.2x. At the observed x values, predictions are 0.7,1.9,3.1,4.3; residuals yᵢ−ŷᵢ are 0.3,0.1,−1.1,0.7. Their squared sum is 0.09+0.01+1.21+0.49=1.8, and 1.8/4=0.45 matches the conditional variance. At x=2.5, the predicted response distribution is N(3.7,0.45), with standard deviation about 0.6708.

8. Conditional Gaussian likelihood gives least squares

Lecture connection: revisit the earlier linear-noise model and compare conditional with joint fitting.

Now specify only \(Y_i\mid x_i\sim\mathcal N(a+\beta^\top x_i,\tau^2)\), independently across observations conditional on the inputs. The xᵢ may be treated as a fixed design. Let A be the N×(d+1) matrix whose i-th row is [1,xᵢᵀ], and w=[a,βᵀ]ᵀ. The target vector y has length N. Then:

For fixed positive τ², maximizing the likelihood minimizes the sum of squared errors. Expanding yᵀy−2wᵀAᵀy+wᵀAᵀAw gives:

These are the normal equations. If A has full column rank, AᵀA is invertible and the unique solution is (AᵀA)⁻¹Aᵀy. In numerical work, solve the least-squares system with QR or SVD rather than explicitly forming an inverse. If columns are dependent, the fitted values still have a unique least-squares projection, but the coefficient vector need not be unique.

For a scalar input with variation, the intercept equation sets the mean residual to zero. Substitute a=ȳ−βx̄ into the slope equation:

With several features, write the same centered equations as β̂=SXX⁻¹SXY when SXX is invertible. The fitted joint-Gaussian conditional mean therefore agrees with the ordinary least-squares mean, subject to the existence conditions discussed next.

Estimate the residual variance separately

Writing v=τ², the derivative is −N/(2v)+SSE/(2v²), so v̂=SSE/N when SSE>0. Under the correctly specified independent constant-variance regression model with full-rank A, SSE/(N−d−1) is an unbiased residual-variance estimator when that denominator is positive. In the four-point example, d=1 and the two estimates are 1.8/4=0.45 versus 1.8/2=0.9. They answer different estimation criteria.

If SSE=0, the variance likelihood grows without bound as v approaches zero; there is no finite positive-variance Gaussian MLE. A perfect line fit can thus be a valid least-squares result while the ordinary noise-density MLE fails to exist.

9. When do joint and conditional fitting agree?

Lecture connection: the handwritten question comparing argmax p(X,Y) with argmax p(X|Y).

Bayes' rule supplies an identity, not a universal equality of optimization problems. For the forward regression direction:

The second term can depend on θ. To see why the unrestricted Gaussian case nevertheless works, reparameterize it by (μX,ΣXX) for the input distribution and (a,β,τ²) for the conditional response distribution. Any positive-definite ΣXX and positive τ², together with arbitrary means and coefficients, define a valid joint Gaussian through:

In these coordinates, the input likelihood uses only μX,ΣXX and the conditional likelihood uses only a,β,τ². The allowed parameter choices do not constrain one block through the other. Thus the joint maximization separates. If both parts have finite nonsingular maxima, the resulting forward regression coefficients and residual variance equal their conditional MLEs.

For a concrete counterexample to general equivalence, constrain independent X∼N(θ,1) and Y∼N(θ,1) to share one mean θ. With the single pair x=10,y=0, conditional fitting pθ(y|x) gives θ̂=0, while joint fitting gives θ̂=5. The shared mean couples the input and response objectives.

The complete joint model still learns extra input-distribution parameters. Equality of a prediction rule is not equality of every learned parameter or every objective value. If covariance restrictions couple the blocks, a prior or penalty couples them, or the empirical scatter is singular, the preceding argument no longer gives this unrestricted equivalence. A Gaussian-noise conditional regression can also be meaningful with a non-Gaussian X, while the joint-Gaussian feature model is then misspecified.

The lecture's reverse conditional p(X|Y) must also be named as such. The identity is ℓjoint=ℓX|Y+∑log pθ(yᵢ). In a fully unrestricted joint Gaussian one can similarly use an independent reverse-conditional/marginal parameterization, but fitting p(X|Y) alone does not supply p(Y), and therefore does not by itself determine the forward predictor p(Y|X). For GDA, the class-density and prior separation is made explicit in section 14.

10. Work through the regression fit yourself

Edit the four pairs and query input. The tool computes the centered statistics, fits a line, and compares the residual variance with the Schur-complement expression from the joint model. Counts of degrees of freedom are displayed separately. Each row is one observation, not a separate model.

11. A mixture is a different model from a single Gaussian

Lecture connection: the two-component Gaussian mixture and the sketches of multiple conditional peaks.

Let a latent indicator Z choose component 1 with probability q and component 2 with probability 1−q. Conditional on that choice, draw a real variable U from N(μ₁,σ₁²) or N(μ₂,σ₂²). Summing over the unobserved choice gives:

This is a weighted sum of densities, not a product and not the random sum of two Gaussian variables. Except for special parameter choices, it is not one Gaussian. Its mean is qμ₁+(1−q)μ₂. Expanding its second moment gives variance qσ₁²+(1−q)σ₂²+q(1−q)(μ₁−μ₂)²: within-component variation plus between-component variation.

A single jointly Gaussian regression forces a unimodal normal conditional and constant conditional variance. A mixture or a richer conditional family can express shapes outside that model. The lecture introduces the mixture construction, not a full mixture-fitting algorithm; latent-variable estimation such as EM is a later topic. In GDA, the marginal feature density will itself be a mixture after the class label is summed out.

12. Logistic regression as the direct conditional route

Connection: the logistic-regression heading and discriminative branch; the likelihood expansion below supplies explanatory context.

For a binary target, a Gaussian response model permits values other than 0 and 1. Logistic regression instead models a Bernoulli conditional with \(p_i=P(Y_i=1\mid x_i)=s(b+w^\top x_i)\), where \(s(t)=1/(1+e^{-t})\). It does not require a Gaussian model for the inputs.

Use s′(t)=s(t)(1−s(t)) and the chain rule. The derivative of one log-likelihood term with respect to its score tᵢ is yᵢ−pᵢ; multiplying by the augmented input gives the coefficient gradient. Thus maximizing the Bernoulli likelihood minimizes binary cross-entropy. This gives the probabilistic meaning of a familiar loss, rather than interpreting a sigmoid output as a Gaussian response.

GDA will also yield a sigmoid of an affine score when both classes share a covariance. It reaches that form by fitting feature distributions and priors. The same posterior form therefore does not imply the same fitted parameters. Full logistic optimization is outside this chapter's shared-Gaussian derivation; the connection here explains what is being compared.

13. Gaussian discriminant analysis: model each class

Lecture connection: binary Gaussian class models with two means and a shared covariance.

Return to a binary label Y∈{0,1} and continuous features X∈ℝᵈ. The shared-covariance GDA model is:

Assume 0<φ<1 and positive-definite Σ for the density formulas. Each class gets its own mean but the same covariance. This says the within-class clouds have the same spread and orientation, while their centers and frequencies may differ. It does not require the off-diagonal covariances to vanish. Shared covariance is not the same assumption as independent features.

The class priors π₀=1−φ and π₁=φ are probabilities; the class-conditional feature functions are densities. The joint mass-density is p(x,y=k)=πₖN(x;μₖ,Σ). There are 1+2d+d(d+1)/2 free scalar parameters. The binary Y and real X are not one jointly multivariate-Gaussian vector: Y has a discrete distribution. The Gaussian assumption is inside each class.

Marginalizing the label gives p(x)=(1−φ)N(x;μ₀,Σ)+φN(x;μ₁,Σ), usually a two-component Gaussian mixture rather than one Gaussian. Conditioning on a known class and marginalizing over an unknown class are different operations, just as in the previous chapter.

14. Derive the GDA parameter estimates

Lecture connection: conditional versus joint generative likelihoods, class means, prior factors, and the pooled covariance derivation.

Let Nₖ count observations with yᵢ=k and assume both classes appear. The joint likelihood groups observations by their labels:

Each Gaussian density has prefactor \((2\pi)^{-d/2}|\Sigma|^{-1/2}\). Its exponent uses the feature dimension d, not the class count Nₖ. Multiplying N densities supplies the N factors in the log-likelihood. Taking logs gives:

Prior: count the labels

Differentiating the prior part gives N₁/φ−N₀/(1−φ)=0. Rearrangement gives φ̂=N₁/N. Its second derivative is −N₁/φ²−N₀/(1−φ)²<0, so the interior solution is the maximum. If a class is absent, its prior MLE lies at a boundary and its mean cannot be estimated from that sample; do not apply the two-class interior formulas blindly.

Means: average within each class

For μₖ, the derivative contains only the class-k observations: Σ⁻¹∑ᵢ:yᵢ=k(xᵢ−μₖ)=0. Thus:

A common Σ does not make the two class means equal. Each is fitted to its own labeled observations. Conversely, estimating class means separately does not allow separate covariances when the chosen model requires sharing.

Covariance: pool residuals around the matching class means

Define \(T_W=\sum_i(x_i-\hat\mu_{y_i})(x_i-\hat\mu_{y_i})^\top\). The determinant term still counts all N examples. The same precision/trace derivative as section 6 gives:

These are within-class residuals. Subtracting the overall feature mean would also include the separation between classes, estimating a different covariance. Equivalently, if Sₖ is each class's ML covariance with denominator Nₖ, then Σ̂=(N₀S₀+N₁S₁)/N. Weight by observation counts; simply averaging S₀ and S₁ is correct only when those counts are equal.

For this unrestricted covariance likelihood to have a finite positive-definite maximum, TW must have full rank. Its rank is at most N−2 because two class means were fitted. Therefore N≥d+2 is necessary, along with both classes appearing and the residuals spanning the feature directions. Sharing can produce a full-rank pooled covariance even when an individual class scatter is singular. A common optional unbiased estimator uses TW/(N−2), but the likelihood derivation gives TW/N.

What happens if only p(X|Y) is fitted?

In this model the class-density parameters (μ₀,μ₁,Σ) are independent of the prior parameter φ in the parameter space. The class-conditional log-likelihood ∑log p(xᵢ|yᵢ) is exactly the Gaussian part above. Maximizing it gives the same class means and shared covariance as joint fitting, but it says nothing about φ. Maximizing the full joint likelihood additionally fits φ from the labels. Directly maximizing ∑log p(yᵢ|xᵢ) under these same generative parameters is a different, normalized objective and generally need not produce these closed-form estimates.

A six-point example, from data to parameters

Class 0 contains (−2,−1),(−1,0),(0,−1); class 1 contains (1,1),(2,0),(3,1). Then N₀=N₁=3, φ̂=1/2, μ̂₀=[−1,−2/3]ᵀ and μ̂₁=[2,2/3]ᵀ. The centered x₁ squared residuals sum to 4 over both classes; the centered x₂ squared residuals sum to 4/3; cross-products cancel. Hence:

The smaller second-feature variance means a given displacement along that direction receives a larger Mahalanobis penalty. Section 16 will turn these parameters into an explicit boundary and posterior.

15. From class densities to posterior probabilities

Lecture connection: Bayes' rule, prediction by maximum posterior, and dropping the common denominator for class comparison.

For fixed learned parameters and an observed input x, Bayes' rule gives:

The denominator is p(x), the marginal feature density. It is the same for both candidate labels at this fixed x. Thus, under equal misclassification costs, choosing the most probable class is equivalent to comparing the unnormalized numerators. To report an actual posterior probability, retain the normalization. If the two scores are 0.06 and 0.14, the class-1 posterior is 0.14/0.20=0.7, not 0.14.

For numerical stability, use log scores gₖ=log πₖ−(1/2)log|Σ|−(1/2)(x−μₖ)ᵀΣ⁻¹(x−μₖ), after removing the common −(d/2)log(2π). Normalize by subtracting M=max(g₀,g₁) before exponentiating: pₖ=e^(gₖ−M)/(e^(g₀−M)+e^(g₁−M)). This avoids underflow from very small raw densities while preserving their ratios.

In binary classification, let Δ=g₁−g₀. Then P(Y=1|x)=1/(1+e^(−Δ)). Predict class 1 for posterior greater than 1/2, class 0 for less, and state how a tie is handled. The workbench assigns a tie to class 1, treating tiny score differences within numerical precision as ties. The lecture's argmax over labels concerns this prediction comparison, not parameter estimation.

16. Derive the linear decision boundary

Lecture connection: equal posterior probabilities and the unfinished handwritten Gaussian-boundary expansion.

The equal-cost binary boundary is where π₁N(x;μ₁,Σ)=π₀N(x;μ₀,Σ). Take the logarithm of their ratio. Because Σ is shared, the determinant terms cancel:

Expand each quadratic as xᵀΣ⁻¹x−2μₖᵀΣ⁻¹x+μₖᵀΣ⁻¹μₖ. The xᵀΣ⁻¹x terms cancel too. The remaining expression is affine:

For w≠0 the decision boundary wᵀx+b=0 is a hyperplane: a point in one dimension, a line in two, and a plane in three. Its normal direction is w. If μ₀=μ₁, w=0 and only priors remain; unequal priors select one class everywhere, while equal priors tie everywhere. Do not divide by a zero slope to invent a boundary point.

Interpret the geometry and priors

With equal priors, choose the nearer mean in Mahalanobis distance. This is nearest-mean Euclidean classification only when the shared covariance is a scalar multiple of I, or after an appropriate whitening transformation. Equal covariance does not mean equal influence for every raw feature. Correlation and feature spread change Σ⁻¹ and therefore the score direction.

Holding class densities fixed, increasing π₁ adds a positive amount to b and expands the class-1 region. In one dimension, μ₀=−1, μ₁=1 and σ²=1 give Δ(x)=2x+log(π₁/π₀). Equal priors give boundary 0. With π₁=0.8 the boundary is −log(4)/2≈−0.6931, moving toward the less frequent class-0 center. Priors are not an extra distance to the class mean; they shift the log score.

Finish the six-point example

From section 14, μ₁−μ₀=[3,4/3]ᵀ and Σ⁻¹=diag(3/2,9/2). Hence w=[4.5,6]ᵀ. The two mean quadratic forms are 8 and 3.5, so b=−(8−3.5)/2=−2.25. The fitted boundary is:

The input [0.5,0]ᵀ lies on the boundary. At [1,0]ᵀ the score is 2.25, giving posterior about 0.90465 and class 1. These calculations use the full density ratio; they are not obtained by reading a raw Gaussian height as a probability.

This shared-covariance Gaussian classifier is commonly called linear discriminant analysis (LDA). The name also appears in related projection formulations. Here its precise meaning is the Gaussian model and classifier derived above. It implies an affine logistic posterior; a logistic posterior alone does not imply Gaussian class densities.

17. Explore priors, scores, and decision boundaries

The core mode below shares one variance, matching the lecture's GDA model in one dimension. Change a prior, mean, spread, or query x. The left panel plots the weighted class densities πₖfₖ(x), which sum to the marginal density. The right panel plots the normalized class-1 posterior.

18. Extensions and limits of the shared-Gaussian model

Supplementary comparisons: these clarify which assumptions the lecture's boundary and estimates rely on.

Separate covariances produce quadratic discriminants

If each class has its own positive-definite covariance Σₖ, compare:

The difference g₁−g₀ generally retains a quadratic term \(-\tfrac12x^\top(\Sigma_1^{-1}-\Sigma_0^{-1})x\), as well as linear and constant terms. The determinant terms no longer cancel. This is quadratic discriminant analysis (QDA). The boundary can be curved, disconnected, or have no real points. Special parameter values can still make the equation simpler; “quadratic” means an equation of degree at most two.

For the one-dimensional extension N(0,1) versus N(0,4) with equal priors, the log ratio is Δ(x)=−log2+3x²/8. The equal-posterior points are \(x=\pm\sqrt{8\log2/3}\approx\pm1.3596\). The broader class 1 wins in both tails; class 0 wins near the common mean. A rule with one ordered cutoff would miss this structure.

Thresholds encode a decision objective

Equal misclassification costs lead to threshold 1/2. If a false positive costs C_FP and a false negative costs C_FN, with zero cost for correct predictions, the expected costs are C_FP(1−p) for predicting 1 and C_FNp for predicting 0. Choose 1 when \(p>C_{\rm FP}/(C_{\rm FP}+C_{\rm FN})\). Both costs must be positive for this interior threshold.

For a threshold t, the score boundary is Δ(x)=log(t/(1−t)). Changing t leaves the posterior distribution unchanged. Changing the class prior changes the posterior. For the equal-prior model Δ=2x, a threshold of 0.8 moves the decision cutoff to +log4/2≈0.6931, in the opposite direction from increasing the class-1 prior to 0.8 at threshold 0.5.

Within-class covariance is not total feature covariance

Using conditional second moments, the GDA model implies:

The first term is within-class variation; the second is variation of the class means. This follows by writing E[XXᵀ|Y=k]=Σ+μₖμₖᵀ, averaging over k, and subtracting E[X]E[X]ᵀ. Using overall covariance in place of within-class covariance silently changes the fitted classifier.

Singularity, assumptions, and evaluation

Too few independent residual directions, redundant features, or features with no within-class variation can make Σ̂ singular. Adding λI with λ>0 is one regularization choice, but it changes the estimator and is not the unrestricted MLE derived here. Feature scaling, covariance restrictions, and regularization must be chosen using training/validation data rather than fitted from the held-out test set. Solve linear systems for discriminant coefficients rather than numerically forming a matrix inverse.

A linear GDA boundary follows from shared Gaussian class densities. A linear boundary by itself does not validate those density assumptions. Logistic regression and GDA can yield different fitted boundaries and probabilities even though both use an affine sigmoid in the shared-covariance case. A richer covariance model has more parameters and may require more data to estimate reliably. Evaluate the chosen task and probabilities on held-out data; a Gaussian fitting calculation alone is not evidence of generalization.

19. Practice: estimation, regression, and classification

These are original problems for these notes. Start with the model assumptions and parameter dimensions, then calculate. Use the hints only after trying a step; checking a remembered formula is not the same as deriving it.

20. Explain the model in an interview

21. Revision: derive the formula and state its conditions

Mastery checklist

Saved in this browser where local storage is available.

• I can distinguish joint, forward-conditional, and reverse-conditional likelihoods.
• I can derive a Gaussian conditional regression mean and variance.
• I can explain mean, mode, and loss-dependent predictions.
• I can derive the Gaussian mean and covariance MLEs using the trace.
• I can check the rank and variance conditions needed by the density formulas.
• I can fit the small regression example and derive the normal equations.
• I can state the conditions for joint/conditional regression agreement.
• I can derive GDA priors, class means, and pooled covariance.
• I can normalize log class scores into a posterior.
• I can derive the shared-covariance boundary and explain prior shifts.
• I can distinguish within-class and total covariance.
• I can explain how separate covariances and costs change the decision rule.

22. Sources and continuation

These personal notes expand Rahul Garg's statistical-regression and Gaussian discriminant analysis lectures (Lectures 24–25), including handwritten likelihood questions, mixture construction, parameter estimates, and the decision-boundary prompt. The Gaussian identities recalled at the start of the continuation are developed in the preceding chapter. The following lecture starts Naive Bayes and a factor-model transition.

The derivation steps, numerical datasets, figures, workbenches, practice problems, and interview prompts are original explanatory additions. Mean-versus-mode decisions, the least-squares bridge, existence/rank conditions, logistic likelihood expansion, and the clearly marked unequal-covariance/cost extensions make assumptions and consequences explicit. They do not imply that every extension appears as a complete derivation in the supplied slides.

• Primary coverage: the supplied statistical-regression and GDA lectures, including their annotations.
• Andrew Ng: Generative Learning Algorithms — an additional reference for Gaussian classification, its parameter estimates, and the logistic-regression connection.

Next: Naive Bayes reuses class priors, class-conditional models, Bayes normalization, and log scores, while replacing the full Gaussian feature model with a conditional-independence structure. Continue with Naive Bayes and its text event models.