A problem of counting
Long before computers, governments faced a data problem: counting people. The United States census of 1880 was tabulated largely by hand and took years to finish, which raised fears that the next count would not be done before the one after it began. For the 1890 census, an engineer named Herman Hollerith built electric tabulating machines that read holes punched in cards. The count was completed much faster than its predecessor. Hollerith’s company was one of those that later merged to form what became IBM, and punched cards remained a standard way to store and sort records for decades. The same card idea had earlier steered patterns on the Jacquard loom, though the purpose there was weaving rather than counting.
Storage gets bigger
Early electronic computers kept data on cards and magnetic tape, which could only be read in order. In 1956 IBM introduced the RAMAC, an early commercial disk storage system that allowed records to be reached directly, without winding through a reel. Capacity was tiny by modern standards, but the principle of random access made interactive record-keeping practical.
Over the following decades the cost of storing a byte fell dramatically, while the electronic computers doing the work grew faster. That made it realistic to keep records that people previously threw away, such as every sale at a store or every click on a page.
Databases bring order
Storing data is only useful if it can be found. In 1970 Edgar Codd, then at IBM, described the relational model, in which data is organized in tables and queried by describing what is wanted rather than how to fetch it. The query language SQL grew out of this work and became the standard way to question a database. Relational systems spread through banks, airlines and businesses in the 1970s and 1980s.
By the late 1980s and 1990s, organizations had begun building data warehouses, which combined records from many operational systems so that analysts could look for trends. Barcode scanners at checkouts were one important source of such records, since every sale could be logged automatically.
Where the term came from
The phrase “big data” gained ground in the 1990s and spread widely in the 2000s. Its exact origin is hard to pin down, and several people have been credited with early uses. A commonly cited framing, from analyst Doug Laney in 2001, describes data challenges in terms of volume, velocity and variety. Many later writers added further words beginning with the same letter. These are descriptive labels rather than a formal definition, and there is no agreed size at which data becomes big. A rough working idea is data too large, too fast or too varied for a single conventional database and one machine to handle comfortably.
Splitting the work
The web changed the scale. Search companies needed to process enormous numbers of pages, and buying ever larger single machines became expensive. Instead, engineers spread the work over many inexpensive computers and designed software to cope with the fact that some of them would fail.
Google published papers describing its file system in 2003 and a processing model called MapReduce in 2004. The idea of MapReduce is to split a job into many small pieces that run in parallel, then combine the results. An open-source project called Hadoop, started by Doug Cutting and others in the mid-2000s with support from Yahoo, implemented similar ideas and made them available to anyone. Later tools handled streaming data and faster in-memory processing, and various “NoSQL” databases relaxed some relational rules to scale out more easily.
Rented computing power from cloud computing providers lowered the barrier further, since a company could use a large cluster for a few hours without buying one.
What it was used for
Typical uses include recommending products, detecting suspicious card transactions, planning logistics, analyzing scientific instruments such as telescopes and particle detectors, and improving search and translation. The growth of connected sensors adds another stream, as does the training data for machine learning and artificial intelligence.
Limits and criticisms
More data does not automatically mean better answers. Large datasets can contain biases, errors or gaps, and patterns found in them may reflect how the data was collected rather than something real. Collecting personal information also raises privacy questions that societies and legislatures are still working through. A responsible reading of the history is that the tools for handling large data developed quickly, while the habits and rules for using it well are still catching up.