Forum

DenyBot in Haiwatha 9 +

luk

20 March 2015, 18:44

How to DenyBot = Baiduspider+:/ in Haiwatha 9 + ?

Hiawatha version: v9.8
Operating System:

20 March 2015, 21:55

i found https://www.hiawatha-webserver.org/forum/topic/1737

luk

20 March 2015, 22:31

Hmmm I have this list of bot usings.ru/bots.php?bot=&page=all

and I want my site to be accessible for some of them but rule may become too big:

UrlToolkit {
 	ToolkitID = block_bots
 	Header User-Agent Ask DenyAccess
	Header User-Agent Baiduspider DenyAccess
	Header User-Agent Baiduspider/2.0 DenyAccess
	Header User-Agent BaiDuSpider DenyAccess
	Header User-Agent Baiduspider+ DenyAccess
        ............
}

How to optimize it ?

Hugo Leisink

21 March 2015, 12:09

The Header pattern matching is done case insensitive. And 'Baiduspider' is also present in 'Baiduspider/2.0' and 'Baiduspider+'. So, the 'Baiduspider' line is sufficient.

Balwinder S Dheeman

3 April 2015, 12:02

How do I kick or block spiders, bots or the such when there is no user agent header, in extended logs I can see that as "-" only.

Hugo Leisink

4 April 2015, 09:54

By IP address?

This topic has been closed.

Back