Forum

This one looks nasty. Assume this file structure:

/home/www/
/home/www/randomfiles
/home/www/domain-1.net/
/home/www/domain-2.net/
/home/www/domain-1.net/pics

When I browse http://127.0.0.1/domain-1.net/pics/ I have no problems. I have tested several conditions and found no problem. However, browsing http://127.0.0.1/randomfiles/ is dangerous. Hiawatha crashes if there isn't a .hiawatha file with the StartFile instruction in that directory. The contents of that directory don't matter, it crashes the server even if empty. As soon as I create the .hiawatha file with the StartFile directive, the problem is gone. An empty .hiawatha file won't do the trick. Hiawatha really wants the StartFile directive.

The chances of that happening by accident are very thin. Someone would have to browse my server by IP instead of domain name, and also browse one directory that is not under any of the domains and lacks a .hiawatha file. But an attacker could use this flaw to bring the server down.


Hiawatha version: v6.15
Operating System: Linux Ubuntu Hardy

I tried to reproduce the problem, but I couldn't make Hiawatha crash here. Can you please follow these instructions for me?

That procedure isn't working for me.

I recompiled Hiawatha with ./configure && make and ran it with Valgrind right from its build directory. I ran the two tests (for the two bugs) and it didn't crash.

I shut it down and ran the tests again with the old build. It crashed on both counts. Just to make sure the old one still crashed.

So I renamed /usr/sbin/hiawatha to something else and copied the new, fatter (556 Kb) hiawatha to /usr/sbin/. Ran the tests and it crashed! It didn't crash when I ran it with Valgrind, but crashed without Valgrind. So I ran it with Valgrind again, except this time from /usr/sbin/ instead of the build directory. It didn't crash.

Summing up, it won't crash when I run it with Valgrind.

Gotta go sleep now, I'll get your new instructions when I get back.

Oke, that's really weird. Please, give me the following information.
- The Ubuntu version you are using (desktop/server, 32/64 bit)
- What were the parameters you gave to the configure script.
- How did you install Hiawatha (make install?)
- Please, post all your configuration files. Including the .hiawatha files.
- Give me any more information that might be helpfull to me in order to reproduce the crash. Like custom kernel or other OS adjustments.

Thanks.

1) Ubuntu 8.04.2 (Hardy).

2) ./configure --prefix=/

3) make deb && cd .. && dpkg -i hiawatha_6.15_i386.deb
But note that the unstripped binary I tested with Valgrind was just copied over from the build directory to /usr/sbin/, and it crashes too.

Maybe I could send you my entire build directory after a ./configure && make? Maybe my binaries could tell you something?

4) Config files. In a moment.

5) No custom kernel. What other "OS adjustments" could there be?

6) Config files. Most of my .hiawatha files have nothing but this:
ShowIndex = no
StartFile = index.cgi

Some of them have index-not.cgi, which produce a blank page. The presence, absence or content of index.* files do not seem to have any effect on the tests.

Some of them override the error code pages. Example:
ShowIndex = no
StartFile = index.cgi
ErrorHandler = 401:/domain-1.net/401.html
ErrorHandler = 403:/domain-1.net/403.html
ErrorHandler = 404:/domain-1.net/404.html
ErrorHandler = 501:/domain-1.net/501.html
ErrorHandler = 503:/domain-1.net/503.html


Here is my httpd.conf file:

# GENERAL SETTINGS
ServerId = nobody
ConnectionsTotal = 500
ConnectionsPerIP = 10
SystemLogfile = /var/log/hiawatha/system.log
GarbageLogfile = /var/log/hiawatha/garbage.log

# BINDING SETTINGS
Binding {
	Port = 80
	TimeForRequest = 2, 25
}

# COMMON GATEWAY INTERFACE (CGI) SETTINGS
CGIextension = cgi
NoExtensionAs = cgi
ExecuteCGI = yes
TimeForCGI = 35
KillTimedoutCGI = yes
TriggerOnCGIstatus = yes

# DEFAULT WEBSITE
Hostname = 127.0.0.1
WebsiteRoot = /home/www
StartFile = index.cgi
AccessLogfile = /var/log/hiawatha/access.log
ErrorLogfile = /var/log/hiawatha/error.log
LogFormat = hiawatha
WrapUserCGI = yes
EnablePathInfo = yes
ErrorHandler = 404:/404.cgi
ShowIndex = yes
IndexStyle = /index.css

# VIRTUAL HOSTS
VirtualHost {
	Hostname = www.domain-1.net,domain-1.net
	WebsiteRoot = /home/www/domain-1.net
	StartFile = index.cgi
	ErrorHandler = 404:/domain-1.net/404.cgi
}
VirtualHost {
	Hostname = www.domain-2.net,domain-2.net
	WebsiteRoot = /home/www/domain-2.net
	StartFile = index.cgi
	ErrorHandler = 404:/domain-2.net/404.cgi
}

# DIRECTORY SETTINGS
Directory {
	Path = /home/www/domain-1.net
	ExecuteCGI = yes
	UploadSpeed = 10,2
}
Directory {
	Path = /home/www/domain-2.net
	ExecuteCGI = yes
	UploadSpeed = 10,2
}
Directory {
	Path = /home/www/domain-3.net
	ExecuteCGI = yes
	UploadSpeed = 10,2
}
# BANNING SETTINGS
BanOnFlooding = 10/1:30
BanOnGarbage = 60
BanOnMaxPerIP = 6
BanOnMaxReqSize = 3
BanOnSQLi = 10800
BanOnTimeout = 10
BanOnWrongPassword = 4:300
KickOnBan = yes
RebanDuringBan = yes
BanOnCMDi = 60
BanlistMask = deny 192.168.0.0/16, deny 127.0.0.1
ReconnectDelay = 6
PreventXSS = yes

The domain names have been changed to protect the innocent.

I never touched any of the other files in /etc/hiawatha/

TIA

In the meantime, can you please retry the valgrind test. I added a parameter to valgrind in the HOWTO page. Please try to reproduce the crash. Send the output to hugo at leisink dot net. Whether Hiawatha crashes or not. It might contain valuable information for me. Thanks.

I just did. Once again, Hiawatha did not crash.

The output is here: http://pastebin.com/m47ef3138

Thank you for your attention once again.

Thanks for the valgrind output. I've made some changes based on that output. A test version of Hiawatha can be found here: http://www.leisink.org/~hugo/hiawatha-6.16.tar.gz. Please note that this is NOT an official release so do not distribute. Thanks.

Read the ChangeLog before installing. Some other changes have been made. Most important one is the renaming of httpd.conf to hiawatha.conf.

Sorry, man. It still crashes. :-(

Here is some new Valgrind output. I hope it's useful in some way.

http://pastebin.com/m2c9162fd

I've used your configuration, ran it on a Ubuntu 8.04 machine, but no crash....

I'm going to ask you to do me a favor. Can you please find out for me which line of code causes the crash? It requires a little knowledge of C. In hiawatha.c, there is a function called serve_client(). In the latest version you have, it starts on line 827. This is the main routine for every request Hiawatha recieves. Can you please add the following statement to that routine

fprintf(stderr, "OKE\n");

and run hiawatha like this:

sudo ./hiawatha -d

Try to reproduce the crash. If Hiawatha prints OKE before it crashes, move the fprintf statement a few lines lower and repeat the steps. This way, you can locate which line of code causes the crash.

I know I'm asking a lot from you, but since we have an exeptional situation here (I can't reproduce it and Hiawatha won't crash with valgrind) I hope you are willing to help me. I really wanna smash this bug.

I don't mind, I am glad to help. I just may need a little more clarification. I added the printf line to line 828, caused the crash and seem to have gotten quite a lot of output already:

http://pastebin.com/m7e3a8587

What do you think? Is that enough or should I move that line further down the code?

Well, since you see 'OKE' in the output, you haven't reached the line that crashes Hiawatha. Please move the fprintf line down and try again. Continue until you see a crash but no 'OKE'. That means that the lines above the printf caused Hiawatha to crash. Otherwise it would have reached the printf statement. So please please, continue the search.

Remember NOT to strip the binary. Lines like ./hiawatha[0x8054bb1] don't help me much. I need to see function names, not addresses.

That part about not stripping the binary isn't clear to me. I am doing './configure --path=/ && make' then running the binary. It is 557.1 Kb large (against 116 Kb of the binary installed with 'make deb && dpkg -i hiawatha.deb'). Isn't that procedure correct? Isn't it unstripped? What am I doing wrong?

The stripped version is the 116kb one. The 557kb version is the not-stripped one. Stripping a binary means removing all debug information. When debugging a binary, you definitely want that information to be present. So, make sure you use the one that is 557kb big. FYI: you can strip a binary by using 'strip <binary>'. Try it on the hiawatha binary. It will become 116kb big. To get the unstripped one, just remove it and do a make again.

I already knew everything you just said. I know something at least. But it's the unstripped, freshly built binary that I am using in the tests. Still, you seem to complain that the output is not correct and useful. That is confusing me.

I will run more tests. Be back shortly.

Oke. I was just asking because I saw a lot of addresses in the debug output instead of function names. Maybe it's just the way that kind of output looks like. I'm no expert in everything. Just in Hiawatha

I seem to have narrowed it down to the switch statement on line 1189:

(There is no preview button in this forum, so let's hope it doesn't get messy.)

        case 503:
            if (session->data_sent == false) {
                switch (handle_error(session, result)) {
                    case -1:
                        session->keep_alive = false;
                        break;
                    case 200:
                        break;
                    default:
                        if (session->data_sent == false) {
                            if (send_code(session, result) == -1) {
                                session->keep_alive = false;
                            }
                        }
                }
            }
            break;
        case 500:
            session->keep_alive = false;
        default:
            if (session->data_sent == false) {
                send_code(session, result);
            }
    }

I replaced it with this code:

		case 501:
						fprintf(stderr, "OKE-501\n");
		case 503:
						fprintf(stderr, "OKE-503\n");
			if (session->data_sent == false) {
						fprintf(stderr, "OKE-503a\n");
				switch (handle_error(session, result)) {
					case -1:
						fprintf(stderr, "OKE-503b\n");
						session->keep_alive = false;
						fprintf(stderr, "OKE-503c\n");
						break;
					case 200:
						fprintf(stderr, "OKE-503-200\n");
						break;
					default:
						fprintf(stderr, "OKE-503-default-a\n");
						if (session->data_sent == false) {
							fprintf(stderr, "OKE-503-default-b\n");
							if (send_code(session, result) == -1) {
								fprintf(stderr, "OKE-503-default-c\n");
								session->keep_alive = false;
							}
						fprintf(stderr, "OKE-503-default-d\n");
						}
					fprintf(stderr, "OKE-503-default-e\n");
				}
				fprintf(stderr, "OKE-503-default-f\n");
			}
			fprintf(stderr, "OKE-503-default-g\n");
			break;
		case 500:
			fprintf(stderr, "OKE-500\n");
			session->keep_alive = false;
		default:
			fprintf(stderr, "OKE-500-default-a\n");
			if (session->data_sent == false) {
				fprintf(stderr, "OKE-500-default-b\n");
				send_code(session, result);
			}
	}

With those changes, I get this output:

Press Ctrl-C to shutdown the Hiawatha webserver.
OKE-501
OKE-503
OKE-503a
*** glibc detected *** ./hiawatha: corrupted double-linked list: 0x0807b2a8 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb7ca6149]
/lib/tls/i686/cmov/libc.so.6[0xb7ca7edd]
/lib/tls/i686/cmov/libc.so.6(__libc_malloc+0x8d)[0xb7ca9cad]
./hiawatha[0x804ad59]
./hiawatha[0x805f77f]
./hiawatha[0x804f6a7]
./hiawatha[0x804fc06]
./hiawatha[0x804fe61]
/lib/tls/i686/cmov/libpthread.so.0[0xb7d904fb]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb7d12e5e]
======= Memory map: ========
Aborted

I guess thenthat something happens if (session->data_sent == false).

HTH

It looks like something goes wrong in handle_error(). Otherwise at least one more printf would have been executed. Can you please take a look at that function for me? Can you print the content of 'result' please? I'd like to know about which error code we're talking here.

Anyway, thanks a lot for the work so far!!!

Sorry, I don't know how to do that. Give me more detailed instructions.

handle_error() is just another function in hiawatha.c. It starts at line 184. Please, continue adding printf statements like you did before in that function until a crash doesn't show the latest printf string. If a function used in handle_error() can't be found in hiawatha.c, it can be found in one of the other .c files. Probably in target.c.

I know I'm asking a lot from you, but I need the information in order to fix the bug. Maybe it's possible I get an account on your machine so I can do the testing myself. If you open a port higher than 1024 in the firewall on that machine, I don't even need root access. Just a normal account. If that's fine with you, please send the login information to hugo at hiawatha dash webserver dot org.

OK. Now I got it narrowed down to line 241: case no: result = send_file(session);

which leads me to file target.c, line 378: add_to_cache

which leads me to file cache.c, line 103: t_cached_object *add_to_cache(t_session *session, char *file)

I modified that part of the code with the following:

t_cached_object *add_to_cache(t_session *session, char *file) {
	t_cached_object *object;
	struct stat status;
	off_t size;
	int fd, i;
	size_t bytes_read, bytes_total = 0;

	fprintf(stderr, "OKE1\n");

	if (stat(file, &status) == -1) {
		fprintf(stderr, "OKE2\n");
		return NULL;

	} else if ((size = status.st_size) == -1) {
		fprintf(stderr, "OKE3\n");
		return NULL;

	} else if ((size < session->config->cache_min_filesize) || (size > session->config->cache_max_filesize)) {
		fprintf(stderr, "OKE4\n");
		return NULL;

	} else if (cachesize + size > session->config->cache_size) {
		fprintf(stderr, "OKE5\n");
		return NULL;

	} else if ((object = (t_cached_object*)malloc(sizeof(t_cached_object))) == NULL) {
		fprintf(stderr, "OKE6\n");
		return NULL;

	} else if ((object->file = strdup(file)) == NULL) {
		fprintf(stderr, "OKE7\n");
		free(object);
		return NULL;

	} else if ((object->data = (char*)malloc(size)) == NULL) {
		fprintf(stderr, "OKE8\n");
		free(object->file);
		free(object);
		return NULL;
	}

fprintf(stderr, "OKE9\n");

	if ((fd = open(file, O_RDONLY)) != -1) {
		while (bytes_total < size) {
			if ((bytes_read = read(fd, object->data + bytes_total, size - bytes_total)) == -1) {
				if (errno != EINTR) {
					free(object->data);
					free(object->file);
					free(object);
					close(fd);

					return NULL;
				}
			} else {
				bytes_total += bytes_read;
			}
		}
		close(fd);
	} else {
		free(object->data);
		free(object->file);
		free(object);

		return NULL;
	}
fprintf(stderr, "OKE10\n");

The output only prints up to "OKE1":

Press Ctrl-C to shutdown the Hiawatha webserver.
OKE1
*** glibc detected *** ./hiawatha: corrupted double-linked list: 0x0807b2a8 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb7c72149]
/lib/tls/i686/cmov/libc.so.6[0xb7c73edd]
/lib/tls/i686/cmov/libc.so.6(__libc_malloc+0x8d)[0xb7c75cad]
./hiawatha[0x804ade9]
./hiawatha[0x805f6bf]
./hiawatha[0x804f857]
./hiawatha[0x804fc15]
./hiawatha[0x804fda1]
/lib/tls/i686/cmov/libpthread.so.0[0xb7d5c4fb]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb7cdee5e]
======= Memory map: ========
Aborted

That is confusing me because I expected at least one of the IF clauses to be triggered. None of them did. OKE10, way further down, never gets triggered, so I guess the code crashes in one the IF test conditions, like else if ((size = status.st_size) == -1), not in the execution statements.

I can't go any further without additional instructions.

What you can do is remove the else statement and move the if to the next line. Like this:

if (stat(file, &status) == -1) {
	fprintf(stderr, "OKE2\n");
	return NULL;
}

if ((size = status.st_size) == -1) {
	fprintf(stderr, "OKE3\n");
	return NULL;
...

You can now place printf statements in between to find out which if statement causes the crash.

How about this:

	if ((object->file = strdup(file)) == NULL) {
		fprintf(stderr, "OKE11\n");
		free(object);
		return NULL;
	}
	fprintf(stderr, "OKE12\n");

	if ((object->data = (char*)malloc(size)) == NULL) {
		fprintf(stderr, "OKE13\n");
		free(object->file);
		free(object);
		return NULL;
	}
	fprintf(stderr, "OKE14\n");

Press Ctrl-C to shutdown the Hiawatha webserver.
OKE0
OKE2
OKE4
OKE6
OKE8
OKE10
OKE12
*** glibc detected *** ./hiawatha: corrupted double-linked list: 0x0807bd70 ***

How come neither 13 nor 14 is triggered?

Catching some sleep now...

Perhaps because the malloc() statement is the one that causes the crash! Can you please tell me the value of size? Place the following line after the OKE12 print.

fprintf(stderr, "size:%ld\n", size);

Can you try if Hiawatha still crashes when you add the following setting to your configuration file.

CacheMinFilesize = 1

OKE12
size:1816
*** glibc detected *** ./hiawatha: corrupted double-linked list: 0x0807be98 ***

CacheMinFilesize = 1 didn't make any difference.

What is the value of file?

fprintf(stderr, "%s\n", file);

Press Ctrl-C to shutdown the Hiawatha webserver.
OKE12
size:1816
/home/devel/www/404.html
*** glibc detected *** ./hiawatha: corrupted double-linked list: 0x0807be98 ***

I rewrote that page and made it a very, very basic HTML file:

Press Ctrl-C to shutdown the Hiawatha webserver.
OKE12
size:280
/home/devel/www/404.html
OKE14
*** glibc detected *** hiawatha-6.16/hiawatha: munmap_chunk(): invalid pointer: 0x0807bea0 ***
*** glibc detected *** hiawatha-6.16/hiawatha: corrupted double-linked list: 0x0807be98 ***

Then I deleted that file and got this:

Press Ctrl-C to shutdown the Hiawatha webserver.
*** glibc detected *** hiawatha-6.16/hiawatha: corrupted double-linked list: 0x0807be98 ***

I suppose that means it's now crashing elsewhere?

Oke, may I ask you to run memcheck on your machine? Boot from your Ubuntu CD/DVD and chose memcheck from the main menu. Just to be sure your machine is oke.

Is it possible I get an account on your machine? I really like to test some things myself.

Yes, one of these days...

Sorry, too busy now. This Hiawatha debugging is beginning to consume too much time. I ran memcheck about four or five months ago, if that means anything. I was testing a memory module for possible upgrade. The machine crashed often, so I checked it and it failed. I removed that one module and the machine passed the test. I returned the module (to a friend) and postponed the upgrade altogether.

Also, nothing else has been crashing here, so I am giving this memcheck task a very, very low priority for the time being.

Sorry.

Thank you for your attention. I appreciate it.

My machine is behind two firewalls over which I have no control. I can't ssh to it myself from a remote account.

I've been doing some extra testing myself based on your findings and found a way to crash Hiawatha. I've made fix for it. You can find a new version here [www.leisink.org]. Can you please test it for me?

Hello. I came here to tell you that I am now running Hiawatha on one of my servers. It's live. The bad news is the two bugs I had reported before are easily reproduced on that machine too. It's a whole different machine, it's remote and runs Slackware, not Ubuntu.

But then I just saw you posted a new version...

Good news! It's fixed! I don't know what you did this time, but the two bugs are gone in my desktop machine. Yay! I'm going to try it on my remote server then. I am sure to come back here and pester you again if any problem comes up.

This is fantastic. Thanks again!

That's good to hear!! I used the valgrind information you supplied and was able to crash Hiawatha myself. And from that point, it was very easy to track down the bug.

Thanks a lot for all the debugging help!!! And if you have another problem with Hiawatha (I hope not), you are of course most welcome to report it.

Sorry, man, problems still abound. I left the server running on a live, test site and noticed that it crashed a few times.

I thought that maybe it was happening because I had just replaced the one 'hiawatha' binary with a freshly-compiled executable, not the whole installation, so I removed it all and reinstalled a new Slackware package made with the new v6.16 you offered on this page. Then I created a cron job to monitor the site every minute. More than a day went by without a glitch. I thought that everything was OK.

But today I ran a download from my own site for the first time and the connection was quickly lost. My cron job didn't trigger, so I checked it and, damn, there was a typo in the cron line. The monitor wasn't running at all.

I fixed the cron job and bang! The cron job itself had the power to take the site down immediatey. All it does is run wget every minute to fetch a certain page, a CGI page that outputs the date and time. Strange that I visited that page with the browser many times and Hiawatha never crashed. But the cron job seemed to be very good at crashing Hiawatha.

I gave it a last shot: I downloaded the latest package you offer on the "download" page. I am not sure it's different from the package you posted here, I didn't remember to compare, but anyway, it's what I use now. I compiled it and installed it with checkinstall. The cronjob doesn't kill the site every minute anymore, but it's been running for 2 hours and the site has crashed 5 times already. Pretty abysmal server performance for such a tiny site with close to zero traffic. I completely doubt that Hiawatha would withstand any real traffic.

That is going to be well nigh impossible to debug, because I have no idea what is causing the crash. I don't think there is any page in particular. I browsed all of the pages (about 12) at random for one or two minutes and saw the server crash as many as 6 times. None of the pages seemed more likely to cause the crash. It is very random.

During all these tests, I was also disappointed to see that Hiawatha does not support resuming a broken download (wget -c). It restarted from zero every time. I finally gave up, shut Hiawatha down, put Apache on duty again and could resume my download from where it had been broken up while using Hiawatha. That was a relief. It felt like being warm and safe again after being out in the pouring rain.

Irk! I just refreshed this page before posting my comment and suddenly I had raw HTML all over the screen. I should have taken a screenshot!

Sigh. I really like how small and light this server is, the choice features and ease of use also seems just right, but I'm afraid I can't deem it stable to go on production just yet.

I'm sorry to hear that Hiawatha does not live up to what I promised on the website.

Can you please send that CGI script and your wget command to hugo at hiawatha dash webserver dot org? I'd like to try to reproduce the problem. The big difference between a request from a browser and one from a tool like wget is that a browser sends a lot of HTTP headers along with the request. I think I'll start searching in the HTTP header handling routines.

I find it really strange you experience these problems. I use Hiawatha on several productions machines. Files are requested via browers, wgets, index bots, scan and hack tools, etc. I use Hiawatha for many years now without a single crash. Which doesn't mean that Hiawatha is bugfree. The big difference in crash experience just amazes me. But, I will not rest before I find the cause of all this.

Btw, did you made the Slackware package yourself or did you download it from somewhere?

I don't think I can/should send you the scripts for clear reasons:

There are two different CGI scripts (two domains). One is tiny, the other is quite big. Too many auxiliary functions, although probably just a dozen or so ever get loaded/run. The tiny script was crashing every minute with wget polling the site, but it didn't crash the site every minute anymore after I installed the v6.16 version from your "Download" section (which I built myself on Slackware and made a package with it). The big CGI script commands an entire site (small, but meant to scale) and that one crashes randomly as I browse the site. The small script is just one boring page, so I thought it made sense to test the site with the big script, because it outputs different pages.

The thing is, there were crashes with two completely different scripts. The current uptime on that box currently is 72 days. It was running on Apache until 2 or 3 days ago and NEVER had crashed. I can't see any relationship between these crashes and my scripts since they are different from each other, both crash and they have run flawlessly on Apache for two years.

Besides, the big script is sensitive/classified. I am definitely not keen on disclosing it.

I think that maybe it would make more sense for you to run my Slackware package if you have a Slackware box handy. I can send you the .tgz package if you want.

Another idea I entertain often is having some kind of debug output dumped constantly into some text file -- overwriting it, or it will grow too large. When Hiawatha crashes, my cron job script will detect the crash and email me the contents of that file, with Hiawatha's "last words" before dying. But that is not something I can do. You would have to provide a version of Hiawatha that will do that, dump debugging info to a file.

Let me know.

I you could e-mail me your slackware package, that would be great. Please include the options you gave to the configure script while building the package. If you can include your configuration files, that would help me a lot. I need those to see what Hiawatha features you are using.

I will make a debugging version for you very soon. Thanks a lot for helping me finding the bug.