Information Theory & Numerical Methods for AI
How much information does an uncertain answer contain, and why can a mathematically correct AI formula still fail on a computer? Begin with coin-like outcomes, ordinary probabilities, and the number of yes-or-no questions needed to identify a result. This builds the meaning of information before naming entropy. Then compare predicted and observed probabilities, study what finite computer numbers can and cannot store, and repair calculations that overflow, underflow, or lose important digits. Running summaries, sensitivity, and evaluation are added step by step, with hand calculations beside executable checks.
Logarithms, information units, and surprise
Objective Use logarithms to quantify information and understand why independent evidence adds.
Core explanation
Information theory assigns more self-information to less probable events. The quantity minus log p has the properties we want: an event with probability one carries zero surprise, rarer events carry more, and independent event probabilities multiply while their information adds. The logarithm base determines the unit—base two gives bits, base e gives nats—but does not change qualitative comparisons. A probability must refer to a defined model and population; an unlikely event under a poor model is not automatically meaningful. Exact zero probability implies infinite surprise when the event occurs, exposing an overconfident or incomplete model. In software, protect domain boundaries and use log-domain operations instead of forcing arbitrary probabilities after overflow.
Define the objects and assumptions in Logarithms, information units, and surprise
A logarithm is the inverse of exponentiation and turns products into sums. Self-information I(x)=-log p(x) assigns more surprise to less probable outcomes. The log base declares units: base two gives bits, natural logarithm gives nats. Information is nonnegative for valid probabilities in zero to one and infinite at a claimed zero-probability event. For an information-unit and log-likelihood calculator, begin by naming the mathematical objects, their domains and codomains, the population or data source, and which equalities are definitions, derived results, approximations, or empirical claims. Translate every symbol into a sentence and every sentence back into a typed quantity so a beginner can check that the problem being solved is the problem that was asked.
Declare outcome alphabet, conditioning context, probability source, log base, units, zero policy, smoothing, and whether values are exact, estimated, or model predictions. Independent evidence adds information because joint probability factors, but dependence requires the actual joint or conditional terms. Create a notation and shape table before calculating. Include units, support, coordinate or basis choices, conditioning information, fitted state, and invalid inputs. Give one ordinary example, one boundary example, and one counterexample. This prevents syntactically legal arithmetic from silently answering a different question.
Derive and interpret the central result for Logarithms, information units, and surprise
Derive log product, quotient, and power rules, then derive additivity of surprise for independent events. Convert bits to nats and work fair, biased, rare, and impossible outcomes. Express likelihood products as sums of log likelihood and explain why this helps both interpretation and computation. Write the derivation line by line, cite the definition or theorem used at each transition, and check dimensions, signs, normalization constants, and limiting behavior. A derivation is complete only when the learner can explain why each transformation is allowed and what the final expression says about the original problem.
Build two hand-checkable instances of an information-unit and log-likelihood calculator: one satisfying the premises and one changing exactly one premise. The chapter-specific interpretation is this: Declare outcome alphabet, conditioning context, probability source, log base, units, zero policy, smoothing, and whether values are exact, estimated, or model predictions. Independent evidence adds information because joint probability factors, but dependence requires the actual joint or conditional terms. Substitute small values, calculate intermediate quantities, and explain why the two outcomes differ before using software; the contrast separates mathematical necessity from representation, data, approximation, and product policy.
Compute and diagnose an information-unit and log-likelihood calculator
Calculate small cases by hand, verify base conversions, compare direct products with summed logs over long sequences, and test p near zero or one. Include invalid negative or greater-than-one probabilities, dependent signals, underflow, log1p for near-one changes, smoothing, and unit-mislabeled plots. Preserve the input fixture, code version, dtype, random generator state, fitted transformations, expected invariants, and a reference calculation. Inspect intermediate shapes and values instead of accepting a plausible final scalar. Complexity, memory, convergence, and numerical range belong to the algorithm contract just as much as the formula.
The diagnostic mutation is: The calculator fails when bits and nats are compared without conversion, dependent events are multiplied as independent, a zero forecast is silently clipped without policy, or a finite-precision underflowed product is interpreted as mathematical impossibility. Reproduce it with the smallest example, locate the first violated definition, assumption, shape, independence, numerical, or evaluation invariant, and repair that boundary. Re-run ordinary, boundary, adversarial, shifted, and repeated cases so the repair does not merely hide the final symptom.
Laboratory: build an information-unit and log-likelihood calculator
First write a one-page specification from these controls: A logarithm is the inverse of exponentiation and turns products into sums. Self-information I(x)=-log p(x) assigns more surprise to less probable outcomes. The log base declares units: base two gives bits, natural logarithm gives nats. Information is nonnegative for valid probabilities in zero to one and infinite at a claimed zero-probability event. Declare outcome alphabet, conditioning context, probability source, log base, units, zero policy, smoothing, and whether values are exact, estimated, or model predictions. Independent evidence adds information because joint probability factors, but dependence requires the actual joint or conditional terms. Then derive the expected result for a tiny fixture, implement the simplest transparent version, compare it with an independent calculation or trusted library, and visualize the geometry, distribution, optimization path, graph, or error curve that carries the chapter’s meaning. Record failures and uncertainty rather than deleting inconvenient trials.
Acceptance requires the learner to explain the derivation without the code, reproduce and repair “The calculator fails when bits and nats are compared without conversion, dependent events are multiplied as independent, a zero forecast is silently clipped without policy, or a finite-precision underflowed product is interpreted as mathematical impossibility.”, and then add conditional evidence and preserve units, dependence, and zero-probability meaning. Deliver formulas with assumptions, annotated computations, tests and counterexamples, numerical and statistical tolerances, plots with labeled axes, exact environment and artifact identity, limitations, and a short teach-back that predicts an unseen case.