Forum

Hello,

We get an extremely slow loading for reverse proxy feature, tried this on several installations.

Site: http://rentremotedesktop.com

Config file of hiawatha:

# Hiawatha main configuration file
#


# GENERAL SETTINGS
#
#ServerId = www-data
ConnectionsTotal = 1500
ConnectionsPerIP = 500
SystemLogfile = /var/log/hiawatha/system.log
GarbageLogfile = /var/log/hiawatha/garbage.log
ServerString = none
 LogFormat = extended

# BINDING SETTINGS
# A binding is where a client can connect to.
#
Binding {
        Port = 80
#       Interface = 5.104.105.24
#       MaxKeepAlive = 30
#       TimeForRequest = 3,20
#       MaxRequestSize = 256
}
#
#Binding {
#       Port = 443
#       Interface = ::1
#       MaxKeepAlive = 30
#       TimeForRequest = 3,20
#       SSLcertFile = hiawatha.pem
#}


# BANNING SETTINGS
# Deny service to clients who misbehave.
#
BanOnGarbage = 300
BanOnMaxPerIP = 60
BanOnMaxReqSize = 300
KickOnBan = yes
RebanDuringBan = yes


# COMMON GATEWAY INTERFACE (CGI) SETTINGS
# These settings can be used to run CGI applications. Use the 'php-fcgi'
# tool to start PHP as a FastCGI daemon.
#
#CGIhandler = /usr/bin/perl:pl
#CGIhandler = /usr/bin/php-cgi:php
#CGIhandler = /usr/bin/python:py
#CGIhandler = /usr/bin/ruby:rb
#CGIhandler = /usr/bin/ssi-cgi:shtml
#CGIextension = cgi
#
#FastCGIserver {
#       FastCGIid = PHP5
#       ConnectTo = 127.0.0.1:2005
#       Extension = php
#}


# URL TOOLKIT
# This URL toolkit rule was made for the Banshee PHP framework, which
# can be downloaded from http://www.hiawatha-webserver.org/banshee
#
#UrlToolkit {
#       ToolkitID = banshee
#       RequestURI isfile Return
#       Match ^/(css|files|images|js|slimstat)($|/) Return
#       Match ^/(favicon.ico|robots.txt|sitemap.xml)$ Return
#       Match .*\?(.*) Rewrite /index.php?$1
#       Match .* Rewrite /index.php
#}


# DEFAULT WEBSITE
# It is wise to use your IP address as the hostname of the default website
# and give it a blank webpage. By doing so, automated webscanners won't find
# your possible vulnerable website.
#
Hostname = 127.0.0.1
ReverseProxy ^/.* http://64.120.231.50:80/
WebsiteRoot = /var/www/hiawatha
StartFile = index.php
AccessLogfile = /var/log/hiawatha/access.log
ErrorLogfile = /var/log/hiawatha/error.log
#ErrorHandler = 404:/error.cgi

# VIRTUAL HOSTS
# Use a VirtualHost section to declare the websites you want to host.
#
#VirtualHost {
#       Hostname = mydomain.com
#       ReverseProxy .* http://67.23.163.87:80/
#       WebsiteRoot = /var/www/my-domain/public
#       StartFile = index.php
#       AccessLogfile = /var/www/access.log
#       ErrorLogfile = /var/www/error.log
#       TimeForCGI = 5
#       UseFastCGI = PHP5
#       UseToolkit = banshee
#}


# DIRECTORY SETTINGS
# You can specify some settings per directory.
#
#Directory {
#       Path = /home/baduser
#       ExecuteCGI = no
#       UploadSpeed = 10,2
#}

Can you please try this beta version [www.leisink.net].

Hello Hugo,

Thank you for your reply. This did slightly improve things, but it's still a long way to go for the proxy feature I guess:)

http://tools.pingdom.com/fpt/#!/xvQb94vF7/rentremotedesktop.com

It seems it's all about CSS and JS files.

Regards
Andrew

I'm not sure this is a Hiawath issue. I did some manual HTTP requests on your website. The CSS and JS files load fast, but then the webserver I connected to (nginx) keeps waiting for some unknown reason. It's the delay between recieving the last byte of the requested file and the disconnection that's causing the trouble.

I did another test. I placed a website I created, which runs on Hiawatha, behind a reverse proxy of another Hiawatha webserver. Original website: www.reki.nl [www.reki.nl], proxied version of that website: reki.leisink.net [reki.leisink.net]. Load fast enough, doesn't it?

I've looked into this issue before, but I have no idea what goes wrong.

Thank you for looking into this, Hugo.

Not sure either, we tested with backend apache and nginx on several sites, same issue.

Anyway, your server is very promising, according to our test it does stop a number of nasty attacks.

If reverse proxiying is fixed in the future, it will be great

Thanks
Andrew

Hello Hugo,

On which operating system are you running Hiawatha? On Debian?

Here is what we tested:

ReverseProxy .* http://178.18.90.135:80/

Where this is your ip for "reki.nl"

Then we tried setting HOSTS file to: 5.104.105.24 reki.nl

Where 5.104.105.24 is our reverse-proxy...

It was also loading slow.

Maybe our configuration file is wrong or something else? Or maybe we complied something wrongly...?

Regards
Andrew

All my servers are running Ubuntu.

I'm not exactly sure what you did. You configured a virtual host and in it you set ReverseProxy .* http://178.18.90.135:80/. Correct? That one was loading slow? No other webserver was involved?

Then you set 5.104.105.24 to reki.nl in your hosts file. Then what?

Hello Hugo,

Setting hosts file to "5.104.105.25 reki.nl" allows us to view your website via our reverse proxy using any web browser.

Your proxy works fast somehow (http://reki.leisink.net/) but our proxy does not (5.104.105.25).

So maybe its some misconfiguration on our side...

Regards
Andrew

Are you sure no nginx / cloudfare webserver is involved?

Btw, your cherokee webserver is quite buggy.

Trying 5.199.128.225...
Connected to rentremotedesktop.com.
Escape character is '^]'.
GET / HTTP/1.0
Host: rentremotedesktop.com

HTTP/1.0 200 OK
Connection: Keep-Alive
Keep-Alive: timeout=15, max=500
Date: Fri, 04 Jan 2013 12:42:50 GMT
Server: Cherokee/1.2.101 (UNIX)
Date: Fri, 04 Jan 2013 01:43:10 GMT
Server: Cherokee/1.2.101 (UNIX)
X-Pingback: http://rentremotedesktop.com/xmlrpc.php
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Via: rentremotedesktop.com (Cherokee/1.2.101)
Content-Length: 4151
X-Cache: HIT from rentremotedesktop.com
Age: 39581

?<{wӸ?'??T/g?@'}??&?0~;??`?egg????J?ֱ?%7??گ??l??W?)? .... 

Double Date and Server HTTP header. I never asked for gzip content-encoding. And HTTP/1.0 should set Connection to close by default. Not according to specs!!

Hello Hugo,

We tested cherokee as hiawatha failed, it's not a production environment.

Do you think you can add our site to your proxy as vhost so that we can test the speed via your proxy?

If our site works fast via your proxy, then it must be our problem...

Regards
Andrew.

http://remote.leisink.net/ Works fine... But this doesn't me that this case is closed for me. I still think it has something to do with the combination of Hiawatha and nginx.

Hello Hugo,

Interesting, we now changed server to Nginx:

http://remote.leisink.net/

curl -I remote.leisink.net
HTTP/1.1 200 OK
Connection: close
Server: nginx
Date: Fri, 04 Jan 2013 13:31:51 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 17522
X-Pingback: http://rentremotedesktop.com/xmlrpc.php
Vary: Accept-Encoding,User-Agent
X-Age: 28
X-Edge-IP: 46.37.167.226
X-Edge-Location: Scranton, US
X-Cache: HIT

And it still loads fast...

But not via our own proxy...

Added you on skype, ours - wooservers

If you have 5 minutes, we will appreciate a quick chat with you.

Thanks
Andrew.

I'm not at home at the moment. Will try at the end of the day. Don't know if my Skype account is still active.

I think I solved it. Please, redownload this beta version [www.leisink.net] and try again.

New version works amazingly fast so far, we will keep testing for several days and I will get back to you on experience.

Thank you so much!

Can we ask if the following rules apply to reverse proxy mode as well? Or this is not implemented?

PreventCMDI
PreventCSRF
PreventSQLi
PreventXSS

Thanks
Andrew

The Prevent* rules do apply to the reverse proxy as well. It was the whole idea to use Hiawatha as a security layer for other webserver.

Great to hear that it now works as it should!

Experienced the same with a setup here, which resulted in a temporary use of nginx as reverse proxy to solve the problem quickly.

Will this fix make it into soon-to-released 8.6.x version?

If not, any estimates for when 8.7 might be available?

I want to release 8.7 as soon as possible. I'm waiting for Andrew's response. If I don't hear anything, I will release 8.7 at the end of this week.

Hello Hugo,

Well, everything is working as expected. We have implemented Prevent* rules and have been testing them using various tools. So far, everything is good.

There is only one little thing we would want to see in maybe some of the future releases:

184.22.156.212|Sun 06 Jan 2013 16:39:50 +0000|Client banned because of too many simultaneous connections

It would be great to know which VHOST or which request caused this, like:

184.22.156.212|Sun 06 Jan 2013 16:39:50 +0000|Client banned because of too many simultaneous connections on Somewebsite.com

184.22.156.212|Tue 08 Jan 2013 20:26:08 +0000|Client banned because of SQL injection on Somewebsite.com

Because when you pass many sites via Hiawatha, in logs it would be nice to know which website causes the issues and maybe ban it.

But of course, 8.7 can be released without this, it's just a "nice to have" feature in the future. =)

We can cross-reference access.log and other logs for now using some bash scripting.

Regards
Andrew.

Hi Andrew. Great to hear everything is working well. I will release Hiawatha 8.7 tonight.

The feature you request is hard to implement. Say, the limit is 10 connections. A client has already 10 connections for different websites and makes an 11th connection. Hiawatha immediately drops that new connection, so the request is never read. What virtual host should I include in the message? So, I'm afraid that I can't implement your feature request.

Including the virtual host in case of a SQL injection is of course possible. It will be available in the official release.

Great, thanks a lot for that.

If the same can be done to "exploit.log" and "garbage.log" it would be great, as now it shows:

174.127.133.103|Wed 09 Jan 2013 01:36:58 +0000|127.0.0.1|/services/support/.tweet|invalid URL
5.199.128.119|Tue 08 Jan 2013 20:36:28 +0000|GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1

Regards
Andrew

Exploit.log already contains the hostname. The 'invalid URL' is in error.log and I already added the hostname to that entry. The stuff in garbage.log is unparsed garbage, so no hostname is extracted. But if it is a malformed HTTP request, the hostname can be read in the garbage itself.

I see, thanks for explaining Hugo, we plan on using Hiawatha on a large scale setup, that's why asking so many questions

No problem, that's what the forum is for.

If that's not really hard to implement, I'd love to see the host name in the system.log for time out errors?

i.e.

255.255.255.255|Fri 11 Jan 2013 13:26:18 +0200|Timeout while waiting for request [domain.tld]

My system.log is full of these errors and I wonder in which domain does these occur.

They don't occur in a specific domain. A client connects but sends no request. No request, no domain.

So, it's normal to have these entries in the system.log, nothing to worry about?

Well, it's not normal for a client to connect and then do nothing. But since some do, it's normal to have those in your logfile. I have them too, lots of them. You don't have to worry about it.

Thanks for the clarification Hugo, it's comforting to hear that

Hello Hugo,

Can we ask the following: ReverseProxy <pattern> http[s]://<hostname>[:<port>][/<path>] [<timeout>]

This [timeout] value, what is the Default setting for it?

Regards
Andrew.

5 seconds. Forgot to mention it in the manual page.