Don't generate http requests with two Host: headers
authorMagnus Hagander <magnus@hagander.net>
Tue, 7 Feb 2012 12:05:41 +0000 (13:05 +0100)
committerMagnus Hagander <magnus@hagander.net>
Tue, 7 Feb 2012 12:05:41 +0000 (13:05 +0100)
commitf9486b54f1e01c29d1c842cf1caee0d539ebb8e6
treed33fe0cb2bd0e5eccb2ed20d231f05af64e7803a
parent03794e315794f863c81d8b89c428f4822d81293f
Don't generate http requests with two Host: headers

This would cause http 400 requests at the server, unsurprisingly.
tools/search/crawler/lib/basecrawler.py