Forum

Hello,

I'm opening new topic after discussing in another thread [www.hiawatha-webserver.org] after I received some useful info from my support stuff...

Hiawatha version:
Hiawatha v9.8, cache, IPv6, Monitor, reverse proxy, SSL (1.3.8), URL toolkit, XSLT
Copyright (c) by Hugo Leisink <hugo@leisink.net>

Operating System: Linux (Debian) 3.10.44 #1 SMP Sat Jun 21 11:57:38 CEST 2014 x86_64 GNU/Linux

So, Hiawatha is working behind Nginx which is setup as reverse proxy with the following config:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X_FORWARDED_PROTO $scheme;
proxy_set_header X_FORWARDED_PROTOCOL $scheme;
proxy_set_header SCHEME $scheme; 

When I try test.php script calling phpinfo() via https, the Hiawatha gives the following output (shortened):

_SERVER["FCGI_ROLE"]	RESPONDER
_SERVER["GATEWAY_INTERFACE"]	CGI/1.1
_SERVER["REQUEST_METHOD"]	GET
_SERVER["REQUEST_URI"]	/test.php
_SERVER["SCRIPT_NAME"]	/test.php
_SERVER["SERVER_PORT"] some_local_port_number
_SERVER["SERVER_NAME"]	some.testing.domain.name
_SERVER["SERVER_PROTOCOL"]	HTTP/1.0
_SERVER["SERVER_SOFTWARE"]	Hiawatha v9.8
_SERVER["SERVER_ADDR"]	127.0.0.1
_SERVER["REDIRECT_STATUS"]	200
_SERVER["HTTP_HOST"]	some.testing.domain.name
_SERVER["HTTP_X_REAL_IP"]	some.real.ip.address
_SERVER["HTTP_X_FORWARDED_FOR"]	some.real.ip.address
_SERVER["HTTP_SCHEME"]	http
_SERVER["HTTPS"]	off
_SERVER["HTTP_RETURN_CODE"]	200
_SERVER["PHP_SELF"]	/test.php

I was spending lot of time trying to debug and patch Tiki's code to properly run & install being puzzled why is Hiawatha setting HTTP_SCHEME to 'http' as well as putting HTTPS to 'off'.

After receiving info from support stuff that it works properly with Lighttpd running in the same environment with local port number being the only difference, I tried it for myself and here is the output of Lighttpd:

_SERVER["HTTP_SCHEME"]	https
_SERVER["HTTP_X_FORWARDED_PROTOCOL"]	https
_SERVER["HTTP_X_FORWARDED_PROTO"]	https
_SERVER["HTTP_X_FORWARDED_FOR"]	some.real.ip.address
_SERVER["HTTP_X_REAL_IP"]	some.real.ip.address
_SERVER["HTTP_HOST"]	some.testing.domain.name
_SERVER["SERVER_PROTOCOL"]	HTTP/1.0
_SERVER["REDIRECT_STATUS"]	200
_SERVER["REQUEST_METHOD"]	GET
_SERVER["QUERY_STRING"]	no value
_SERVER["REQUEST_URI"]	/test.php
_SERVER["PATH_INFO"]	no value
_SERVER["SCRIPT_NAME"]	/test.php
_SERVER["REMOTE_ADDR"]	some.real.ip.address
_SERVER["REMOTE_PORT"]	0
_SERVER["SERVER_ADDR"]	127.0.0.1
_SERVER["SERVER_PORT"] some_local_port_number	
_SERVER["GATEWAY_INTERFACE"]	CGI/1.1
_SERVER["SERVER_NAME"]	some.testing.domain.name
_SERVER["SERVER_SOFTWARE"]	lighttpd
_SERVER["FCGI_ROLE"]	RESPONDER

Is there any explanation in the difference of behaviour?

I also wonder what is with

_SERVER["HTTP_X_FORWARDED_PROTOCOL"]	https
_SERVER["HTTP_X_FORWARDED_PROTO"]	https

variables which are checked in Tiki code and not set by Hiawatha?

Let me say, that there is another difference in setup:

• Lighttpd is using php-cgi
  
• Hiawatha is using php-fpm
  

Any hint?

Just to confirm that I tried running lighttpd with php-fpm to make running environment exactly the same as for Hiawatha and there is no change in the performance of Lighttpd.

If HTTPS is set to 'off' and HTTP_SCHEME set to 'http', then HTTP is used for sure. I have no idea why. I have no view at what's happening in your reverse proxy.

> If HTTPS is set to 'off' and HTTP_SCHEME set to 'http', then HTTP is used for sure. I have no idea why.

Well, I wonder why in the same environment, when accessing the script served by Lighttpd via https, there is e.g. lock in Firefox, Firebug shows that Remote IP goes on port 443 and phpinfo() shows it's https, while Hiawatha reports http?

> I have no view at what's happening in your reverse proxy.

I can give you access to my server...

I've tried to use curl...

Here is the snippet when I access script served by Hiawatha:

...
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
...
* 	 start date: 2014-09-21 10:29:07 GMT
* 	 expire date: 2016-04-25 22:25:57 GMT
* 	 issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
* 	 SSL certificate verify ok.
> GET /test.php HTTP/1.1
> User-Agent: curl/7.37.0
...
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server nginx/1.6.2 is not blacklisted
< Server: nginx/1.6.2
< Date: Sun, 19 Oct 2014 14:47:05 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/5.5.17-1~dotdeb.1

Does it qualify as https?

Looks like a HTTPS connection to me.

Moreover, Hiawatha log files contains things like:

X-Forwarded-For: some.real.ip.address|X_FORWARDED_PROTO: https|X_FORWARDED_PROTOCOL: https|SCHEME: https

and still:

HTTP_SCHEME = http
HTTPS = off

Do you have any explanation for it or are you inspired to login to the server?

As I wrote, previously, shows/set variables to https.

Ok, you can send the login to hugo@leisink.net. Also mention the URL which I can use to test.

First, your Hiawatha configuration doesn't even have a HTTPS binding. So, there is no way that the connection to Hiawatha was done via HTTPS. Perhaps your reverse proxy first tries it via HTTPS and falls back to HTTP because it doesn't work. I haven't looked at your lighttpd config, but that probably does contain a HTTPS binding config if you say that it works via lighttpd.

Second, the "X_FORWARDED_PROTO: https|X_FORWARDED_PROTOCOL: https|SCHEME: https" in your logfile is information that was sent via the HTTP request. It has nothing to do with whether the request was done via HTTP or HTTPS. It's just some text in the HTTP request. However, this kind of information shouldn't be there and specially not in this format! Seems you have a buggy reverse proxy!

I see...now I wonder considering that binding for http is via non-standard (!=80) port, does it mean that I'd have to create another non-standard port and use it for HTTPS?


There is, afaict, no HTTPS binding for lighttpd (its config is in ~/lighttpd/lighttpd.conf).



Yeah, that's sent by Nginx which contains:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X_FORWARDED_PROTO $scheme;
proxy_set_header X_FORWARDED_PROTOCOL $scheme;
proxy_set_header SCHEME $scheme; 

Can you provide some explanation for learning purposes?


If you give some explanation above, I'll inform my support staff about it.

Thank you.

Yes, a binding can only do HTTP or HTTPS, not both at the same time. So, yes, you have to specify a new binding with HTTPS (for Hiawatha, this means use the SSLcertFile option for that binding).

The X_FORWARDED_PROTO and X_FORWARDED_PROTOCOL are no HTTP header variables. They look like CGI environment variables. Rename them to X-Forwarded-Proto and X-Forwarded-Protocol. I think the last one can even be removed, because it's not an official one. SCHEME can also be removed.

I was following info from the manual [www.hiawatha-webserver.org], but after restarting server there is:

Warning: the certificate signature algoritm in serverkey.pem should at least be SHA256.

Anything missing in the docs?

No, docs are fine. It's just a warning, the server will run fine with it. The warning is for the SHA signature in your SSL certificate. SHA will soon be deprecated [community.qualys.com]. Also browsers will soon no longer accept SHA signed certificates [konklone.com]. You are advised to replace them with stronger certificates. Many SSL certificate providers offer free updates, so no reason to wait with it.

Considering that we do not have need for expensive certificates - personal site, non-profit site, small company without ecommerce etc. - do you have any suggestion for some stronger and not too expensive certs?

If it's just for testing, you can use self-signed certificates or you can try www.startssl.com [www.startssl.com]. On the other hand, SSL certificates aren't really expensive. The ones I bought for my websites are about €10 per year.

The funny thing is that those cheap certificates are exactly the same (from a crypto point of view) as the ones for which you pay several hundreds euro's per year (the EV certificates).

Yes, I could use free one from startssl. com for my personal site.


If you can give me some pointer...The ones from Comodo are more expensive, afaict.

With $10/yr I'd use them for all sites and have all traffic through https.

Btw, if you can provide some non-expensive VPS provider to put Hiawatha on the front...with some helped to setup/admin email...

I only know providers in the Netherlands: SSL certificate [www.sslcertificaten.nl] and VPS [www.cloudvps.com].

Thank you.

Hello Hugo,


I've received from support stuff which wrote me the following:



This explains why lighttpd is showing 'https' despite I could not find any https-related setup in my config.

Now, not being an expert - leaving that for you to answer - I can't say whether the traffic between front-end (reverse proxy) and back-end servers should be encrypted in https scheme, but since I consider to take advantage of SNI to use SSL certificates for some of my virtual hosts sits, I wonder if that would work if the traffic between front and back is not encrypted?

I'm a bit suspcicious that depending on just headers sent by front end is enought?

Please, shed some light!

Sincerely,
Gour

Of course not. X_FORWARDED_PROTOCOL is not a valid HTTP header. It should be X-Forwarded-Proto. And isn't it weird that Lighttpd shows 'https' while in fact the connection to Lighttpd was HTTP, specially because Lighttpd had no HTTPS bindings. The problem here is that it is undefined / unspecified how a web server should handle those headers. Hiawatha looks for the HTTP_SCHEME variable to its own connection, Lighttpd apparently also looks at HTTP headers sent by a reverse proxy.

The biggest problem here is that many proxies that sent a X-Forwarded-For don't sent a X-Forwarded-Proto. Yes, it's a mess. So, instead of relying on some crappy, incomplete shit sent by (reverse) proxies, I chose to let Hiawatha look at its own connection. And while X-Forwarded-For may contain multiple IP address, X-Forwarded-Proto only contains one value. So, what if a connection via 2 reverse proxies reaches my webserver and I see X-Forwarded-For with 2 IP addresses and no X-Forwarded-Proto or one with only a single value, than what? To keep the way Hiawatha works consistent, I decided to ignore those headers. In my opinion, what Lighttpd does is wrong or at least can cause some confusion due to inconsistency caused by crappy HTTP headers.

If you want to enforce HTTPS, do it at your reverse proxy. Redirections within the web application should only be done by using a relative path, not via a complete URL. If you did it all well, you'll never need the X-Forwarded-* stuff anyway.

Thank you Hugo, for elaborate answer!


Does it mean that PHP's server variable is (should be) set based on X-Forwarde-Proto header?


Indeed!!


So, based on this, it means that one should have traffic between proxy and Hiawatha encrypted as well in order to qualify as https?


That could be fixed by (proper) setup?


It makes sense.


Heh, that's interesting scenario.


Can we say this is example of 'defensive coding/programming' ?


After your explanation, I tend to agree with you.


Well, the 1st problem is that it's not my server - I'm only client using its services and do not like being forced to use Apache which eats my memory in order to serve PHP sites. Secondly, I want to use Hiawatha (instead of Lighttpd) server as back-end.

Now, few (more questions) with hopefully short answers:

• how to enforce HTTPS a reverse proxy?
  
• if reverse proxy would add/send X-Forwarded-Proto header, would it be sufficient for Hiawatha to set HTTP_SCHEME to 'https'?
  
• is it possible with present setup and using Lightttp to use SSL certificates by deploying SNI technology?
  
• would 2) above be enough to use SNI with Hiawatha (which I want to)?
  

Finally let me say, that I'd be more happy to use VPS and setup Hiawatha at the front-end and not fiddling with reverse proxies, but the main obstacle is the pain of maintaining one's own mail server and all the external services which I looked at are pricey which are much above the cost if VPS itself.

A connection from a client to a webserver via a reverse proxy are in fact two separate connections. Both can be HTTP or HTTPS.

Only use HTTPS where you don't trust the underlying network. For the internet, that would be a 'yes'. For the network between reverse proxy and webserver, that's probably a 'no'. If it's your network, than a 'no' for sure. If it's the provider's network then you should ask yourself what you're doing there if you would answer it with a 'yes'.


Not by you or me. All reverse proxies must be 'fixed' to solve this issue.


Yes, it's the result of thinking things though with a 'hacker's mindset'. Don't consider it 'done' when it works. Also try to think about 'what can go wrong with this solution'.


That depends on what software you use for the reverse proxy. If it were Hiawatha, a 'RequireSSL = yes' would be enough.


No, Hiawatha doesn't look at HTTP headers for that setting and never will. Hiawatha needs an incoming HTTPS connection for that.


yes.


Yes. But if both client-proxy and proxy-webserver connections are HTTPS, the SSL certificate needs to be present at both proxy and webserver. My advice, use the reverse proxy as SSL offloader. Let proxy-webserver connection be simple HTTP. If you don't have a good feeling about this solution, consider a provider where incoming connections are going directly to your webserver instead of via a reverse proxy.

About your last statement, well, can't decide for you but setting up e-mail may cost a few hours and than you're done. I'm running Postfix which I got running with online documentation within an hour (including spam filtering). After that, it doesn't require any time. But what's stopping you from directing e-mail to some external mailservice and only HTTP(S) to your own VPS. That's what DNS is for.

OK, that's what I was thinking.



OK, that's what support stuff said: "...it doesn't make sense to encrypt the traffic inside a local machine."



It's provider's network and I have enough trust that the answer is 'no'.



It's Nginx, but it's obvious that the provider does not enforce https for all the traffic.



Hmm...I can enforce that connection to virtual host served by Hiawatha is HTTPS- there is option in provider's control panel to 'Allow HTTPS only (redirect HTTP requests to HTTPS)' and will try to see if it helps.




What would be the role of set X-Forwarded-Proto in this case?



The problem with this is that I do not have influence how reverse-proxy (nginx) is setup. Using direct connections is possible only when one uses system's Apache which I do not want/like.



I only did qmail setup/admin long ago and never touched neither Exim nor Postfix, but yesterday I was told in #linode that maintaining mail server is PIDA and it takes lot of time, especially constant battling with spam. In any case, even just for the sake of experiment, I'm going to buy one month VPS subscription along with some testing domain to see how it goes.

Thank you for your encouragement in regard to mail server.

When I see how much time I already spent fighting this reverse-proxy <---> webserver setup, I could already setup mail server.



I was looking for such solution - using VPS for web stuff and external mailservice for mail stuff, but all what I've seen is very pricey and for few domains the cost goes several time above the price for VPS itself.

For the sake of experiment I did create testing account at another provider which has similar/same environment...

Here are some of the SERVER envvars set by their reverse-proxy:

_SERVER["HTTP_HTTP_X_FORWARDED_PROTO"]	https
_SERVER["HTTP_HTTPS"]	on
_SERVER["HTTP_X_FORWARDED_PROTO"]	https
_SERVER["HTTP_X_FORWARDED_SSL"]	on
_SERVER["HTTP_CONNECTION"]	close
_SERVER["HTTP_SCHEME"]	http
_SERVER["HTTPS"]	off

so I wonder if such environment has possibility to use SNI & run web sites with SSL certs?

Whether a webserver can do SNI has nothing to do with any of those environment variable. It's all about the SSL library and how it's used.

Well, I know Hiawatha supports SNI, the main provider does it as well as the 'new' testing provider, but I assume that in order to have working https connection with appropriate SSL certificate, it would be required that front-end & back-end servers combo - Nginx & Hiawatha - are properly configured?