Whitehouse.gov scrubbing and Search Engines

[ThoughtCriminal]

Regarding Brad Friedman's article about scrubbing at Whitehouse.gov
(http://www.democraticunderground.com/articles/04/10/26_scrubbing.html) may explain a recent discovery I made about the site. The web sites Robots.txt file is huge and excludes large portions of the site from inclusion by search engines. You can view the whitehouse.gov search engine here:

http://www.whitehouse.gov/robots.txt

So, why are so many pages disallowed? Many search engines like Google, cache the pages that they search. These cached pages can provide proof of alterations or deletions.

More about web robots:
http://www.robotstxt.org/wc/robots.html

About Google caching:
http://www.google.com/help/features.html#cached

Internet Archive "Wayback Machine"
http://www.archive.org/web/web.php

[whistle]

1. Uhmmm, must neater than book burnings and also a lot less....

...conspicuous, eh!

[AlamoDemoc]

2. our hard-earned tax dollars at work !!!

[shaolinmonkey]

3. I believe that robots page excludes ALL the pages.

Nothing like open government, eh?

[Southpaw Bookworm]

4. Another site to know about

The Memory Hole

I believe they do some archiving of Web pages as well, in addition to their other work.