Weblog

I will stick to Hiawatha until a more secure and faster web server is available.

Did you ever think about implementing an asynchronous, event driven architecture? I know that select and poll are not the best performers, but there are epoll (Linux), kqueue (FreeBSD), etc. that provide better scalability. As for the "no SMP" issue, other people from the web server crowd already have solutions for things like this: lighttpd and Cherokee use a single process with multiple listener threads while nginx and LiteSpeed use worker processes. All of them use epoll under Linux, while the thread/process number is kept to minimal in order to avoid the expensive context switches since anyway, there's no real improvement in speed by using too many processes/threads as epoll does the job. Don't hate me because I mention sendfile as well.

Don't know about sendfile, but I know that Hiawatha uses a multithreaded model. I remember reading something about this into the forums.

The functions select() and poll() are indeed not the best performers if you use them to handle large amount of simultaneous connections. In Hiawatha, select() is only used for its timeout ability, so it's not a problem. The problem with a single process handling multiple connections, is that a system call blocks the whole process, while a multithreaded solution can switch to another thread and continue serving clients. And since every modern computer has a processor with multiple cores, a multithreaded solution is far more effecient than a single process. Also, a thread is capable of handling multiple requests (due to HTTP's keep-alive feature), it's not as heavy as you might think.

In my opinion, the 10k connections problem is not a problem that should be solved, but should be avoided. If you have a webserver that is likely to have 10k simultaneous client connections, you should be thinking about redundancy and load balancing. Because, apparently you are hosting a popular website. So, in my opinion, Lighttpd was build for the wrong reason.

Another reason for me to use a multithreaded solution, is that a single process solution requires more complex code to handle multiple connections. More complex code can more easily lead to bugs and security issues.

I've tried using sendfile() in Hiawatha, but for several reasons decided not to use it. It appeared that using Hiawatha's internal cache was faster than using sendfile(). I don't know the exact reason, but my guess is that because of Hiawatha is first filling the internal cache before sending, it can read the file in one continues read from disk, instead of reading it in delayed smaller blocks because of limited upload speed. Reading it fast from disk and than sending the file from memory might be the reason it was faster than using sendfile().

Another reason for not using sendfile() is because Hiawatha will not be able to do traffic throttling and to handle stagnating connections.

That's why I mentioned epoll/kqueue instead of select/poll.

lighttpd powers our large objects serving. As for the load balancing and redundancy, we already have this for our large website that gets a lot of traffic (including the downloads, powered by lighttpd and a Layer-4 balancer). Hiawatha won't fit the task because of the technology behind the application (Java Servlets), but I am currently investigating alternatives for our smaller applications, PHP based, in order to use less machines to serve more traffic (the classic "cost reduction" as the management loves to say). Most probably the static objects will move towards S3/CloudFront, thus a bunch of servers handling this type of task becomes a "deprecated" solution. Which moves us to the dynamic generated content.

For a web service, I am investigating web servers that can handle a lot of idle keep-alive connections without eating all the available memory as Apache loves to do with its brain dead architecture. So far, nginx with epoll are the closest options to our needs. Hiawatha isn't fully evaluated, yet.

As for the sendfile support, I understand your reasons.

It was never my intention to make the fastest webserver in the world, because other projects already have that goal. So, if you're looking for the fastest webserver available, it's likely you won't pick Hiawatha. My goal is to make the most secure webserver. Security comes at a small price: a little bit of performance. Compare Hiawatha with Lighttpd at secunia.com and you see the results of my effort. Because I advertise with 'being secure', I recieve a lot (and I really mean 'a lot') of hack attemps on my server. None was ever successful.

A few weeks ago, a researcher performed a security audit on the Hiawatha source code. He said that with Hiawatha I'm 'doing a great job as far as secure coding'. He only found small issues which will be fixed in the next release. I think that's another of the last small steps to actually being the most secure webserver in the world. I think it's worth the price of a little bit performance.

Nevertheless, I'm very interested in the results of the Hiawatha evaluation. Can you please share them with me?

I know. As a former web developer while being a security obsessed person, I know how certain security features can hit the performance. However, I do sleep better having those features in production.

As for lighty, some internal features that people can not find in every server are used. Along with FreeBSD, it kinda makes the CPU and memory to be mostly useless (110MiB of ram OS+server, 5-10% CPU, under load) while the TCP stack implementation absolutely shines. The "kewl, Lenux" crowd still has to learn a few things from the old fart, traditional UNIX, FreeBSD.

The beauty of the nginx setup is the fact that it can be easily be converted to a reverse proxy for dynamic content serving. Since you were so receptive about a couple of my requests (mod_rpaf like IP headers handling and UNIX socket support for FastCGI) Hiawatha is still a good candidate for this job, behind the proxy. Some commercial CMSes that I need to deploy have some dull rewrite rules (fairly easy to convert from mod_rewrite to nginx's UrlRewrite module) that I don't know if they can be easily converted to Hiawatha's format, thus the proxy can do the rewrite as well. Some people just won't use the Front Controller Pattern Design. This makes the sysadmin life a giant pain in the ass with hundreds of dumb rewrite rules to manage the application beyond the "Apache" mindset. Still curios about how fast your FastCGI implementation can do, since for this kind of requests, the web server is merely a proxy for the PHP upstream. And yes, I'll share the results. Actually, I am going to publish my overall results, any time soon.

If you're looking for a webserver with good rewrite abilities, you came to the right place here. In Hiawatha, rewrite rules are written in a sort of scripting language, which makes it very flexible and powerfull. In 7.2, I've rewritten the rewrite engine to make it easier to add features. So, I've anything is missing, just let me know and I'll see what I can do.

edit: comment moved to forum (http://www.hiawatha-webserver.org/forum/topic/641)

I recently discovered Hiawatha by accident. I am in the process of migrating to a new server and have been looking at various different software alternatives to those in use today. However I never thought about replacing Apache. Well, I guess I will be. I wanted to try out lighttp, and then found nginx - and then Hiawatha? I got curious and spent some time researching what this what. I decided to give it a try. Under 30 min later I had Hiawatha serve out all my sites. I am impressed! I don't need all the fancy stuff Apache "can do".

I sincerely hope Hiawatha will continue to develop and grow. The word is spreading so don't give up on "changing the world" :-) As long as Hiawatha is maintained I am sure more will see the advantages it offers.

There is one feature I would like to see in a future release. Bandwidth throttling per virtual host. Ability to throttle per user connection would be great. I am not sure if that exists already. I read about throttling scripts I think.

I can't wait to put Hiawatha into production!

Don't be discouraged; Hiawatha is my favorite webserver, and I use it wherever I can. Really appreciate all the work you've done. Thank you!

The downside to threading is that it will cost (at least) one stack per connection, which is vastly more memory than actually is needed. So using epoll() as an alternative could be rather important to achieve a maximum number of connections per given amount of memory.