'Google search on steroids' brings dark Web into the light

[Renew Deal]

'Google search on steroids' brings dark Web into the light

The government agency that brought us the Internet has now developed a powerful new search engine that is shedding light on the contents of the so-called deep Web.

The Defense Advanced Research Projects Agency (DARPA) began work on the Memex Deep Web Search Engine a year ago, and this week unveiled its tools to Scientific American and "60 Minutes."

Memex, which is being developed by 17 different contractor teams, aims to build a better map of Internet content and uncover patterns in online data that could help law enforcement officers and others. While early trials have focused on mapping the movements of human traffickers, the technology could one day be applied to investigative efforts such as counterterrorism, missing persons, disease response, and disaster relief.

Dan Kaufman, director of the information innovation office at DARPA, says Memex is all about making the unseen seen. "The Internet is much, much bigger than people think," DARPA program manager Chris White told "60 Minutes." "By some estimates Google, Microsoft Bing, and Yahoo only give us access to around 5 percent of the content on the Web."
<snip>

http://www.infoworld.com/article/2883793/government/darpa-search-engine-casts-light-on-dark-web.html

New search engine exposes the "dark web"

This week on 60 Minutes, Lesley Stahl and producer Shachar Bar-On got an early look at Memex, a powerful new search engine developed by DARPA, the U.S. military's Defense Advanced Research Projects Agency.

The inventor of Memex, Chris White, sat down with Stahl and Bar-On and explained how Memex works--and how it could revolutionize law enforcement investigations. (See Chris White's demonstration in the video player above.)

"The internet is much, much bigger than people think," White said. "By some estimates Google, Microsoft Bing, and Yahoo only give us access to around 5% of the content on the Web." That leaves a lot of room for bad actors to operate freely in the shadows.

White says that Memex goes far beyond the realm of traditional search engines and gives law enforcement a powerful new tool to search the "dark web," where criminals buy, sell, and advertise in the illegal weapons trade and sex trafficking.
<snip>

60 Minutes Overtime Video: http://www.cbsnews.com/news/new-search-engine-exposes-the-dark-web

DARPA: Nobody's safe on the Internet

<snip>
Dan Kaufman: And we watched, and they did what you'd think. You know, they put an address of a massage parlor or something, and then they'd write it down on a yellow stickie, and then they'd try to build in each to each to each. And we looked at that, and we said, "There has to be a better way."

Especially considering that Google and Bing don't penetrate the dark web, where most illegal goods are advertised and sold. So DARPA invented Memex, with which you can click just one button and all the hidden information scattered deep in the web about an illicit activity is pulled together and revealed.

Lesley Stahl: So the--you're building the network.

Chris White: Building the network. That's right.
<snip>

60 Minutes video about security in general including Memex: http://www.cbsnews.com/news/darpa-dan-kaufman-internet-security-60-minutes

Human Traffickers Caught on Hidden Internet

In November 2012 a 28-year-old woman plunged 15 meters from a bedroom window to the pavement in New York City, a devastating fall that left her body broken but alive. The accident was an act of both desperation and hope—the woman had climbed out of the sixth-floor window to escape a group of men who had been sexually abusing her and holding her captive for two days.

Four months ago the New York County District Attorney’s Office sent Benjamin Gaston, one of the men responsible for the woman’s ordeal, to prison for 50-years-to-life. A key weapon in the prosecutor’s arsenal, according to the NYDA’s Office: an experimental set of Internet search tools the U.S. Department of Defense is developing to help catch and lock up human traffickers.

Although the Defense Department and the prosecutor’s office had not publicly acknowledged using the new tools, they confirmed to Scientific American that the Defense Advanced Research Projects Agency’s (DARPA) Memex program provided advanced Internet search capabilities that helped secure the conviction. DARPA is creating Memex to scour the Internet in search of information about human trafficking, in particular advertisements used to lure victims into servitude and to promote their sexual exploitation.

Much of this information is publically available, but it exists in the 90 percent of the so-called “deep Web” that Google, Yahoo and other popular search engines do not index. That leaves untouched a multitude of information that may not be valuable to the average Web surfer but could provide crucial information to investigators. Google would not confirm that it indexes no more than 10 percent of the Internet, a statistic that has been widely reported, but a spokesperson pointed out that the company’s focus is on whether its search results are relevant and useful in answering users' queries, not whether it has indexed 100 percent of the data on the Internet.
<snip>

http://www.scientificamerican.com/article/human-traffickers-caught-on-hidden-internet

[safeinOhio]

1. So when they catch

That lady that calls me every day to lower my credit card interest. That has to be connected to the web some how.

[Downwinder]

2. robots.txt no longer respected?

[JaneyVee]

3. I don't even wanna know WTF goes on on the other 90% of internet.

The internet is the new wild wild west.