New whitehouse.gov robots.txt file
by thatbaldguy on 21 Jan 2009 at 20:07:40, under public interest, technomancy
In what we nerds take as a very positive sign, the robots.txt file for the official White House website went from about 2,400 of Disallow lines to just two lines:
User-agent: *
Disallow: /includes/
A sampling of the previous Bush-era file:
User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text
via Boing Boing.
Leave a Reply
-
Welcome to the HFAYHD
This blog is a handcrafted product. The slight variations in spelling and grammar from accepted norms have been employed to enhance its natural beauty and cultural relevance, and should in no way be considered flaws or defects. Thank you.
