Forum

Hello,

We are still having issues with monitor. We get around 1.5 Mln requests via hiawatha per day now and 1 thread of monitor is just not able to catch up with such speed. So we need to launch several threads (up to 10) but they overload the server.

Please let us know if there is any other solution to this.

Regards
Andrew

Try setting MonitorStatsInterval to 1. There is a mistake in the manualpage for this option. The parameter is in minutes, not in seconds.

it seems we are also getting error 404 on our hiawatha nodes in monitor-access.log:



xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:06 +0000|404|889||GET /1369838017.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838028.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838035.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838037.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838039.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:08 +0000|404|889||GET /1369838045.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:08 +0000|404|889||GET /1369838050.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:08 +0000|200|5022||GET /1369838054.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip

That's because you run multiple log fetch threads at the same time. Don't do that. They all request an index. But if one downloads a .gz file, another thread might also try to download it. After downloading a monitor log file (the .txt.gz files), Hiawatha deletes it automaticly.

we now run 1 thread only, it seems it processes around 200,000 requests per 2-3 hours.

we are worried that we wont be able to process more than 5-10 mln. requests this way

It could very well be possible that my monitoring solution isn't capable of handling that amount of requests. My problem is that I don't have a server/website with that amount of traffic to do the required testing.

Do you use 'MonitorRequests = yes' for one or more virtual hosts?

we have MonitorRequests set to yes for Default virtual host, and generally we wanted to use it for all other virtual hosts, but clearly it will be difficult to meet with 1 monitoring thread.

if you have in mind some high performance design to handle 10-100 million requests per day, we are ready to hire developers for implementing it and even making the code public after that...contributing to this community

I will give this some thought. The downloading of the statistical information from the webservers is not the issue, right? The load is on the database server while inserting the statistical information, correct?

yes, that's correct. i understand that inserting hundreds of thousands of entries into db leads to high load, even on SSD disks that we have.

I've made some changes to fetch_webserver_logs, which can now insert multiple records via one query instead of a separate query per record. Perhaps that will lower the load on the database.

I will do some last checks and bug testing before sending it to you. Please send me your e-mailaddress via the contact form.

hello hugo, thank you for the script, we are getting the following error:

[\u@\h \W]#\[\033[0m\]./fetch_webserver_logs
PHP Notice: Undefined offset: 1 in /var/www/monitor/database/fetch_webserver_logs on line 202
PHP Fatal error: Call to private method database_connection::type_to_format() from context 'MySQLi_connection_di' in /var/www/monitor/database/fetch_webserver_logs on line 166
[\u@\h \W]#\[\033[0m\]

I think I told to make the function type_to_format() in libraries/database/database_connection.php 'protected'. It's 'private' now. Did I forget to mention that in the e-mail?

Thank you, Hugo, my bad.

http://i42.tinypic.com/n65and.png

We are now seeing this going at super high speed in ssh window.

Is that normal? For 1 day we see no update in any of the statistics in web pages.

Those are SQL queries to insert log information into the database. I have no idea why they end up on your screen. I don't print them in my script. What did you do to see those SQL queries?

And to answer your question: no, that's not normal.