Friday, September 9, 2016

The First Domestic Mass Data Mining Systems

Mass data mining first was developed by the Federal Bureau of Investigation (FBI) during the late 1990s and supervised by its Data Intercept Technology Unit. The first program, whose name remains a secret, debuted in 1996 and was replaced by the programs known as Omnivore in 1997 and Carnivore in 1999. (Carnivore later was renamed DCS-1000.) Carnivore was “packet sniffer” software: It captures the e-mail messages of a specific target from the e-mail traffic that travelled through the network of an Internet service provider (ISP). In 2000, when the program was publicly disclosed, the FBI said it “chews all the data on the network, but it only actually eats the information authorized by the court order.”[i] In another statement of this view, an assistant FBI Director told Congress in 2000: “[Carnivore] does NOT search through the contents of every message and collect those that contain certain key words like ‘bomb’ or ‘drugs.’ It selects messages based on criteria expressly set out in the court order, for example, messages transmitted to or from a particular account or to or from a particular us.”[ii] So before 9/11, the FBI said the program did not conduct mass data mining, although it had that capability.

Carnivore became a mass surveillance tool, bypassing the warrant requirement to search records, sometime after the attacks of September 11, 2001. I have tried to identify when exactly this changeover to mass surveillance for phone and internet records occurred, but it is difficult to pinpoint based on the current state of public knowledge. To be sure, FBI mass data-mining efforts are authorized under the USA Patriot Act (2001) and FBI Guidelines (2002) for investigation issued by the U.S. Justice Department. Carnivore (or DSC- 1000) was used until 2005 and replaced by a program known as NarusInsight. The technology writer Declan McCullagh writes that NarusInsight “can record all Internet traffic, including Web browsing--or, optionally, only certain subsets such as all e-mail messages flowing through the network. Interception typically takes place inside an Internet provider's network at the junction point of a router or network switch.” According to Kevin Bankston, a staff attorney at the Electronic Frontier Foundation, "What they're doing is intercepting everyone and then choosing their targets." This approach is called “full-pipe” surveillance and critics have questioned its legality. [iii]


[i] E. Judson Jennings, “Carnivore: US Government Surveillance of Internet Transmissions,” Virginia Journal of Law and Technology, 6: 10 (Summer 2001). See also Talitha Nabbali and Mark Perry, “Going for the Throat: Carnivore in an Echelon Word- Part I,” Computer Law and Security Report Vol. 19 no. 6 (2003): 460.

[ii] Oversight Hearings on “Fourth Amendment Issues Raised by the FBI’s ‘Carnivore’ Program,” Before the House Committee on the Judiciary, Subcommittee on the Constitution, 106th Cong. (2000) (statement of Donald M. Kerr, Assistant Director, Federal Bureau of Investigation), July 24, 2000 (quoted in Jennings, “Carnivore”).

[iii] Dean McCullagh, “FBI turns to Broad New Wiretap Method,” ZDNeT, January 30, 2007; Richard Koman, “FBI ‘Full-Pipe’ Surveillance May be Illegal,” ZDNet, January 31, 2007.