Loss
A Notion of Error in Machine Learning
DOI:
https://doi.org/10.55283/jhk.18717Keywords:
error, statistics, machine learning, computingAbstract
This essay compares two statistical notions of error to draw out their distinctive epistemological and normative implications. The first sense crystallized in the nineteenth century as practical techniques for producing estimates from discrepant observations were interpreted as metaphysical laws of error. Historians have shown how these interpretations produced new forms of social knowledge and control over normal types. Although the metaphysical understanding of these laws was abandoned, error continued to be a major theme in twentieth-century statistical thought. A second sense of error associated with machine learning emerged in the second part of the century. This began when John von Neumann and Frank Rosenblatt developed statistical theories of computing, notably as machines able to learn from their environments. In the 1980s, researchers developed techniques such as backpropagation that used error measurements to improve a model’s performance on learning tasks. Comparing these conceptions of error, I argue that we can perceive a larger shift from a politics of normal types revealed by the regularity of error to what I term a “politics of tasks” in which errors are used to refine desired behaviors.
Downloads
References
Aldrich, John. “R. A. Fisher and the Making of Maximum Likelihood 1912–1922.” Statistical Science 12, no. 3 (1997): 162–76. https://doi.org/10.1214/ss/1030037906.
Amoore, Louise. “Machine Learning Political Orders.” Review of International Studies 49, no. 1 (2023): 20–36. https://doi.org/10.1017/S0260210522000031.
Anderson, James A., and Edward Rosenfeld, eds. Talking Nets: An Oral History of Neural Networks. MIT Press, 2000.
Annany, Mike. “Seeing Like an Algorithmic Error: What Are Algorithmic Mistakes, Why Do They Matter, How Might They Be Public Problems?” Yale Journal of Law and Technology 24 (2022): 342–64.
Aradau, Claudia, and Tobias Blanke. “Algorithmic Surveillance and the Political Life of Error.” Journal for the History of Knowledge 2 (2021): 1–13. https://doi.org/10.5334/jhk.42.
Aspray, William. John von Neumann and the Origins of Modern Computing. MIT Press, 1990.
Boden, Margaret. Mind As Machine: A History of Cognitive Science. Clarendon Press, 2006.
Breiman, Leo. “Statistical Modeling: The Two Cultures.” Statistical Science 16, no. 3 (2001): 199–231. https://doi.org/10.1214/ss/1009213726.
Campolo, Alexander. “State-of-the-Art: The Temporal Order of Benchmarking Culture.” Digital Society 4, no. 35 (2025): 1–10. https://doi.org/10.1007/s44206-025-00190-x.
Campolo, Alexander, and Katia Schwerzmann. “From Rules to Examples: Machine Learning’s Type of Authority.” Big Data & Society 10, no. 2 (2023): 1–13. https://doi.org/10.1177/20539517231188725.
Canales, Jimena. Bedeviled: A Shadow History of Demons in Science. Princeton University Press, 2020.
Canguilhem, Georges. The Normal and the Pathological. Translated by Carolyn R. Fawcett. Zone Books, 1991.
Daston, Lorraine. “Scientific Error and the Ethos of Belief.” Social Research 72, no. 1 (2005): 1–28. https://doi.org/10.1353/sor.2005.0016.
Dick, Stephanie. “Of Models and Machines: Implementing Bounded Rationality.” Isis 106, no. 3 (2015): 623–34. https://doi.org/10.1086/683527.
Donoho, David. “Data Science at the Singularity.” Harvard Data Science Review 6, no. 1 (2024). https://doi.org/10.1162/99608f92.b91339ef.
Dupuy, Jean-Pierre. The Mechanization of the Mind: On the Origins of Cognitive Science. Translated by M. B. DeBevoise. Princeton University Press, 2000.
Edwards, A. W. F. “The History of Likelihood.” International Statistical Review 42, no. 1 (1974): 9–15. https://doi.org/10.2307/1402681.
Eisenhart, Churchill. “Laws of Error.” In Encyclopedia of Statistical Sciences. Vol. 4, Icing the Tails to Limit Theorems, edited by Samuel Kotz, Norman Lloyd Johnson, Campbell B. Read. Wiley, 1983.
Fisher, Ronald A. “On the Mathematical Foundations of Theoretical Statistics.” Philosophical Transactions of the Royal Society Series A, Containing Papers of a Mathematical or Physical Character 222 (1922): 309–68. https://doi.org/10.1098/rsta.1922.0009.
Fisher, Ronald A. “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society. Series B (Methodological) 17, no. 1 (1955): 69–78. https://doi.org/10.1111/j.2517-6161.1955.tb00180.x.
Fourier, Joseph. “Extrait d’un Mémoire sur la Théorie analytique des assurances.” Annales de Chimie et de Physique 2, no. 10 (1819): 177–89.
Galison, Peter. “The Ontology of the Enemy: Norbert Wiener and the Cybernetic Vision.” Critical Inquiry 21, no. 1 (1994): 228–66. https://doi.org/10.1086/448747.
Galison, Peter. “Author of Error.” Social Research 72, no. 1 (2005): 63–76. https://doi.org/10.1353/sor.2005.0032.
Gigerenzer, Gerd, Zeno Swijtink, Theodore Porter, Lorraine Daston, John Beatty, and Lorenz Krüger. “The Inference Experts.” In The Empire of Chance: How Probability Changed Science and Everyday Life. Cambridge University Press, 1989. https://doi.org/10.1017/CBO9780511720482.005.
Grill, Gabriel. “Constructing Capabilities: The Politics of Testing Infrastructures for Generative AI.” Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (2024): 1838–49. https://doi.org/10.1145/3630106.3659009.
Grosman, Jérémy, and Tyler Reigeluth. “Perspectives on Algorithmic Normativities: Engineers, Objects, Activities.” Big Data & Society 6, no. 2 (2019): 1–12. https://doi.org/10.1177/2053951719858742.
Hacking, Ian. The Taming of Chance. Cambridge University Press, 1990.
Halpern, Orit. “The Future Will Not Be Calculated: Neural Nets, Neoliberalism, and Reactionary Politics.” Critical Inquiry 48, no. 2 (2022): 334–59. https://doi.org/10.1086/717313.
Hendrycks, Dan, Collin Burns, Steven Basart, et al. “Measuring Massive Multitask Language Understanding.” Preprint, ArXiv, January 12, 2021. https://doi.org/10.48550/arXiv.2009.03300.
Jones, Matthew L. Reckoning with Matter: Calculating Machines, Innovation, and Thinking about Thinking from Pascal to Babbage. University of Chicago Press, 2016.
Jones, Matthew L. “How We Became Instrumentalists (Again): Data Positivism since World War II.” Historical Studies in the Natural Sciences 48, no. 5 (2018): 673–84. https://doi.org/10.1525/hsns.2018.48.5.673.
Joque, Justin. Revolutionary Mathematics: Artificial Intelligence, Statistics and the Logic of Capitalism. Verso, 2022.
Klein, Judy L. “Economics for a Client: The Case of Statistical Quality Control and Sequential Analysis.” History of Political Economy 32, (2000): 25–70. https://doi.org/10.1215/00182702-32-Suppl_1-25.
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” Proceedings of the 26th International Conference on Neural Information Processing Systems (2012): 1097–105.
Law, Harry. “Bell Labs and the ‘Neural’ Network, 1986–1996.” BJHS Themes 8 (2023): 143–54. https://doi.org/10.1017/bjt.2023.1.
LeCun, Yann. “Learning Process in an Asymmetric Threshold Network.” In Disordered Systems and Biological Organization, edited by E. Bienenstock, F. Fogelman Soulié, and G. Weisbuch. Springer, 1986. https://doi.org/10.1007/978-3-642-82657-3_24.
Lin, Cindy Kaiying, and Steven J. Jackson. “From Bias to Repair: Error as a Site of Collaboration and Negotiation in Applied Data Science Work.” Proceedings of the ACM on Human-Computer Interaction 7 (2023): 131:1–131:32. https://doi.org/10.1145/3579607.
Luitse, Dieuwertje, Tobias Blanke, and Thomas Poell. “AI Competitions as Infrastructures of Power in Medical Imaging.” Information, Communication & Society (2024): 1–22. https://doi.org/10.1080/1369118X.2024.2334393.
Mackenzie, Adrian. Machine Learners: Archaeology of a Data Practice. MIT Press, 2017.
Mayo, Deborah G. Error and the Growth of Experimental Knowledge. University of Chicago Press, 1996.
McCorduck, Pamela. Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence. AK Peters, 2004.
Minsky, Marvin, and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. MIT Press, 1969.
Neumann, John von. “Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components.” In Automata Studies, edited by Claude E. Shannon and John McCarthy. Princeton University Press, 1956.
Neyman, Jerzy, and Egon S. Pearson. “On the Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference: Part I.” Biometrika 20, no. 1/2 (1928): 175–240. https://doi.org/10.2307/2331945.
Olazaran, Mikel. “A Sociological Study of the Official History of the Perceptrons Controversy.” Social Studies of Science 26, no. 3 (1996): 611–59. https://doi.org/10.1177/030631296026003005.
Porter, Theodore M. The Rise of Statistical Thinking: 1820–1900. Princeton University Press, 1986.
Rosenblatt, Frank. “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.” Psychological Review 65, no. 6 (1958): 386–408. https://doi.org/10.1037/h0042519.
Rosenblatt, Frank. “Two Theorems of Statistical Separability in the Perceptron.” In Mechanisation of Thought Processes: Proceedings of a Symposium Held at the National Physical Laboratory. H.M. Stationary Office, 1959.
Rosenblatt, Frank. “Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms.” Cornell Aeronautical Laboratory, March 16, 1961.
Rosenblueth, Arturo, Norbert Wiener, and Julian Bigelow. “Behavior, Purpose and Teleology.” Philosophy of Science 10, no. 1 (1943): 18–24. https://doi.org/10.1086/286788.
Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. “Learning Internal Representations by Error Propagation.” In Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations. Edited by David E. Rumelhart and Hames L. McClelland. MIT Press, 1986. https://doi.org/10.7551/mitpress/5236.003.0012.
Savage, Leonard J. The Foundations of Statistics. John Wiley & Sons, 1954.
Shannon, Claude E. “A Mathematical Theory of Communication.” Bell System Technical Journal 27, no. 3 (1948): 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x.
Sheynin, Oscar. “Density Curves in the Theory of Errors.” Archive for History of Exact Sciences 49, no. 2 (1995): 163–96. https://doi.org/10.1007/BF00376546.
Simpson, Thomas. “XIX. A Letter to the Right Honourable George Earl of Macclesfield, President of the Royal Society, on the Advantage of Taking the Mean of a Number of Observations, in Practical Astronomy.” Philosophical Transactions of the Royal Society of London 49 (1755): 82–93. https://doi.org/10.1098/rstl.1755.0020.
Skinner, B.F. The Behavior of Organisms: An Experimental Analysis. B.F. Skinner Foundation, 1991.
Stigler, Stephen M. The History of Statistics: The Measurement of Uncertainty before 1900. Belknap Press, 1986.
Talcott, Samuel. Georges Canguilhem and the Problem of Error. Palgrave Macmillan, 2019.
Tversky, Amos, and Daniel Kahneman. “Judgment under Uncertainty: Heuristics and Biases.” Science 185, no. 4157 (1974): 1124–31. https://doi.org/10.1126/science.185.4157.1124.
Vapnik, Vladimir N. “An Overview of Statistical Learning Theory.” IEEE Transactions on Neural Networks 10, no. 5 (1999): 988–99. https://doi.org/10.1109/72.788640.
Vapnik, Vladimir N. Statistical Learning Theory. Wiley, 1998.
Vapnik, Vladimir N. The Nature of Statistical Learning Theory. Springer, 1995.
Vapnik, Vladimir N, and A. Ya. Chervonenkis. “On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities.” Translated by B. Seckler. Theory of Probability & Its Applications 16, no. 2 (1971): 264–80. https://doi.org/10.1137/1116025.
Wald, Abraham. “Contributions to the Theory of Statistical Estimation and Testing Hypotheses.” The Annals of Mathematical Statistics 10, no. 4 (1939): 299–326.
Wallis, W. Allen. “The Statistical Research Group, 1942–1945.” Journal of the American Statistical Association 75, no. 370 (1980): 320–30. https://doi.org/10.1080/01621459.1980.10477469.
Werbos, Paul. “Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences.” PhD diss., Harvard University, 1974.
Williamson, Robert C. “Loss Functions.” In Empirical Inference: Festschrift in Honor of Vladimir N. Vapnik, edited by Bernhard Schölkopf, Zhiyuan Luo, and Vladimir Vovk. Springer, 2013. https://doi.org/10.1007/978-3-642-41136-6_8.
Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. “Understanding Deep Learning (Still) Requires Rethinking Generalization.” Communications of the ACM 64, no. 3 (2021): 107–15. https://doi.org/10.1145/3446776.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Alexander Campolo
This work is licensed under a Creative Commons Attribution 4.0 International License.
Funding data
-
H2020 European Research Council
Grant numbers ERC-2019-ADG-883107-ALGOSOC
