Linear Algebra for AI
A data record can be written as a row of numbers. What does its direction mean, how can many rows be transformed together, and how can a model find the closest answer when no exact answer exists? Start with two- and three-number vectors, draw their combinations, and turn small equation systems into tables of coefficients. Matrices, projections, least-squares fitting, eigen-directions, and singular-value decomposition are built from those visible jobs. Only then connect them to features, compressed representations, attention calculations, and numerical reliability, always checking dimensions and reconstructing a small result by hand.
Coordinate systems and feature spaces
Objective Translate real observations into vectors while preserving units, meaning, and valid comparisons.
Core explanation
Linear algebra begins by choosing a vector space and a coordinate system. In machine learning, a row of features is a vector only after every coordinate has a defined meaning, unit, and ordering. A different basis can describe the same underlying object with different coordinates, just as a location can be expressed relative to different map axes. Scaling is therefore not cosmetic: a large-unit feature can dominate distance, gradients, and regularization. Centering moves the origin, while standardization changes the geometry by assigning a scale to each axis. Sparse text vectors and dense embeddings occupy very different spaces even when they represent the same document. Before applying an algorithm, document the representation, fit any transformation on training data only, and test whether distances or directions correspond to useful domain relationships.
Define the objects and assumptions in Coordinate systems and feature spaces
A vector space is a set with addition and scalar multiplication satisfying stated axioms. A basis is an independent spanning list, and coordinates are coefficients relative to that basis. The same abstract vector has different coordinate lists under a basis change; real observations become vectors only through an encoding whose semantics and validity must be documented. For a unit-aware feature-space atlas, begin by naming the mathematical objects, their domains and codomains, the population or data source, and which equalities are definitions, derived results, approximations, or empirical claims. Translate every symbol into a sentence and every sentence back into a typed quantity so a beginner can check that the problem being solved is the problem that was asked.
For every feature, record source, unit, scale, valid range, missingness, category encoding, population, time, and whether the transformation is fitted. Separate points from displacement vectors, physical coordinates from learned embeddings, and affine offsets from linear directions. Standardization changes geometry and must travel with the model. Create a notation and shape table before calculating. Include units, support, coordinate or basis choices, conditioning information, fitted state, and invalid inputs. Give one ordinary example, one boundary example, and one counterexample. This prevents syntactically legal arithmetic from silently answering a different question.
Derive and interpret the central result for Coordinate systems and feature spaces
Express a vector as a linear combination of basis vectors, assemble a change-of-basis matrix from new basis columns, and derive the inverse coordinate conversion when the bases are valid. Show algebraically and geometrically how a unit conversion can preserve the underlying object while altering naive coordinate distance. Write the derivation line by line, cite the definition or theorem used at each transition, and check dimensions, signs, normalization constants, and limiting behavior. A derivation is complete only when the learner can explain why each transformation is allowed and what the final expression says about the original problem.
Build two hand-checkable instances of a unit-aware feature-space atlas: one satisfying the premises and one changing exactly one premise. The chapter-specific interpretation is this: For every feature, record source, unit, scale, valid range, missingness, category encoding, population, time, and whether the transformation is fitted. Separate points from displacement vectors, physical coordinates from learned embeddings, and affine offsets from linear directions. Standardization changes geometry and must travel with the model. Substitute small values, calculate intermediate quantities, and explain why the two outcomes differ before using software; the contrast separates mathematical necessity from representation, data, approximation, and product policy.
Compute and diagnose a unit-aware feature-space atlas
Build two-dimensional examples by hand, transform them between standard and rotated bases, and verify reconstruction. Then encode a small mixed-unit dataset with a training-fitted pipeline and compare distances before and after scaling. Test dependent basis vectors, zero variance, unseen categories, missing values, outliers, and deployment unit changes. Preserve the input fixture, code version, dtype, random generator state, fitted transformations, expected invariants, and a reference calculation. Inspect intermediate shapes and values instead of accepting a plausible final scalar. Complexity, memory, convergence, and numerical range belong to the algorithm contract just as much as the formula.
The diagnostic mutation is: The atlas fails when coordinates are treated as the object itself, a dependent list is called a basis, evaluation data fits scaling, or distance in an arbitrary encoding is described as objective semantic similarity. Reproduce it with the smallest example, locate the first violated definition, assumption, shape, independence, numerical, or evaluation invariant, and repair that boundary. Re-run ordinary, boundary, adversarial, shifted, and repeated cases so the repair does not merely hide the final symptom.
Laboratory: build a unit-aware feature-space atlas
First write a one-page specification from these controls: A vector space is a set with addition and scalar multiplication satisfying stated axioms. A basis is an independent spanning list, and coordinates are coefficients relative to that basis. The same abstract vector has different coordinate lists under a basis change; real observations become vectors only through an encoding whose semantics and validity must be documented. For every feature, record source, unit, scale, valid range, missingness, category encoding, population, time, and whether the transformation is fitted. Separate points from displacement vectors, physical coordinates from learned embeddings, and affine offsets from linear directions. Standardization changes geometry and must travel with the model. Then derive the expected result for a tiny fixture, implement the simplest transparent version, compare it with an independent calculation or trusted library, and visualize the geometry, distribution, optimization path, graph, or error curve that carries the chapter’s meaning. Record failures and uncertainty rather than deleting inconvenient trials.
Acceptance requires the learner to explain the derivation without the code, reproduce and repair “The atlas fails when coordinates are treated as the object itself, a dependent list is called a basis, evaluation data fits scaling, or distance in an arbitrary encoding is described as objective semantic similarity.”, and then add a new basis or feature while preserving reconstruction, lineage, and neighbor meaning. Deliver formulas with assumptions, annotated computations, tests and counterexamples, numerical and statistical tolerances, plots with labeled axes, exact environment and artifact identity, limitations, and a short teach-back that predicts an unseen case.