Calculus & Optimization for AI
If changing one model number changes its error, which direction should that number move, and by how much? Start with slopes on ordinary graphs and a tiny loss function you can calculate by hand. Then extend the same idea to many parameters, explain every gradient entry, trace how error moves backward through a computation, and test why a training step helps or fails. Curvature, random batches, penalties, and constraints appear only when the simpler update rule needs them. The final diagnostic project separates correct calculus from unstable arithmetic, poor data, and an unsuitable model.
Functions, limits, and local approximation
Objective Interpret functions as models and use limits to reason about local behavior and continuity.
Core explanation
A function maps inputs to outputs under a declared domain. AI models are functions whose inputs may be vectors, images, or token sequences and whose outputs may be scores, probabilities, or generated values. Limits describe what a function approaches as an input moves near a point, allowing calculus to define derivatives even when direct substitution is insufficient. Continuity means nearby inputs produce nearby outputs locally, but it does not guarantee smoothness, robustness, or useful predictions. Local linear approximation replaces a complicated function near one point with a tangent line. This is the seed of gradient-based optimization. Always distinguish the mathematical function from finite-precision software, where clipping, branches, and overflow can change apparent behavior.
Define the objects and assumptions in Functions, limits, and local approximation
A function maps each input in a declared domain to one output in a codomain. Limits describe values approached near a point independently of the value at the point; continuity joins the limit, function value, and domain. A first-order local approximation replaces a smooth function near a with f(a)+f prime(a)(x-a). For a local-approximation and continuity explorer, begin by naming the mathematical objects, their domains and codomains, the population or data source, and which equalities are definitions, derived results, approximations, or empirical claims. Translate every symbol into a sentence and every sentence back into a typed quantity so a beginner can check that the problem being solved is the problem that was asked.
State domain, codomain, units, parameterization, valid neighborhood, one- or two-sided approach, and whether variables are scalar or vector. Discontinuities, saturation, clipping, branches, undefined logarithms, and discrete inputs break assumptions differently. A good approximation needs an error statement and scale, not merely closeness in one plot. Create a notation and shape table before calculating. Include units, support, coordinate or basis choices, conditioning information, fitted state, and invalid inputs. Give one ordinary example, one boundary example, and one counterexample. This prevents syntactically legal arithmetic from silently answering a different question.
Derive and interpret the central result for Functions, limits, and local approximation
Evaluate limits algebraically and numerically, derive tangent-line approximation from the derivative definition, and connect the remainder to curvature. Compare removable, jump, infinite, and oscillatory behavior. Show how composition preserves continuity only when the inner limit lands in the outer function domain. Write the derivation line by line, cite the definition or theorem used at each transition, and check dimensions, signs, normalization constants, and limiting behavior. A derivation is complete only when the learner can explain why each transformation is allowed and what the final expression says about the original problem.
Build two hand-checkable instances of a local-approximation and continuity explorer: one satisfying the premises and one changing exactly one premise. The chapter-specific interpretation is this: State domain, codomain, units, parameterization, valid neighborhood, one- or two-sided approach, and whether variables are scalar or vector. Discontinuities, saturation, clipping, branches, undefined logarithms, and discrete inputs break assumptions differently. A good approximation needs an error statement and scale, not merely closeness in one plot. Substitute small values, calculate intermediate quantities, and explain why the two outcomes differ before using software; the contrast separates mathematical necessity from representation, data, approximation, and product policy.
Compute and diagnose a local-approximation and continuity explorer
Calculate exact and approximate values over shrinking neighborhoods, plot absolute and relative error, and compare float64 with higher precision. Test cancellation, boundary domains, piecewise branches, large scale, log near zero, sigmoid saturation, and inputs far from the expansion point. Preserve the input fixture, code version, dtype, random generator state, fitted transformations, expected invariants, and a reference calculation. Inspect intermediate shapes and values instead of accepting a plausible final scalar. Complexity, memory, convergence, and numerical range belong to the algorithm contract just as much as the formula.
The diagnostic mutation is: The explorer fails when sampled values are taken as proof of a limit, a two-sided limit is asserted at a boundary, a local line is used globally, or floating-point cancellation is mistaken for a discontinuous mathematical function. Reproduce it with the smallest example, locate the first violated definition, assumption, shape, independence, numerical, or evaluation invariant, and repair that boundary. Re-run ordinary, boundary, adversarial, shifted, and repeated cases so the repair does not merely hide the final symptom.
Laboratory: build a local-approximation and continuity explorer
First write a one-page specification from these controls: A function maps each input in a declared domain to one output in a codomain. Limits describe values approached near a point independently of the value at the point; continuity joins the limit, function value, and domain. A first-order local approximation replaces a smooth function near a with f(a)+f prime(a)(x-a). State domain, codomain, units, parameterization, valid neighborhood, one- or two-sided approach, and whether variables are scalar or vector. Discontinuities, saturation, clipping, branches, undefined logarithms, and discrete inputs break assumptions differently. A good approximation needs an error statement and scale, not merely closeness in one plot. Then derive the expected result for a tiny fixture, implement the simplest transparent version, compare it with an independent calculation or trusted library, and visualize the geometry, distribution, optimization path, graph, or error curve that carries the chapter’s meaning. Record failures and uncertainty rather than deleting inconvenient trials.
Acceptance requires the learner to explain the derivation without the code, reproduce and repair “The explorer fails when sampled values are taken as proof of a limit, a two-sided limit is asserted at a boundary, a local line is used globally, or floating-point cancellation is mistaken for a discontinuous mathematical function.”, and then add a piecewise model and state its continuity, valid local regions, and approximation error. Deliver formulas with assumptions, annotated computations, tests and counterexamples, numerical and statistical tolerances, plots with labeled axes, exact environment and artifact identity, limitations, and a short teach-back that predicts an unseen case.