Forum

Hi,
I'm getting many errors on my virtual host's error log that say: "access denied due to URL toolkit rules"

I can't find out what's wrong with the URL toolkit, since the webserver runs flawlessly since two years and the website code (Magento 1.x) is always the same.

Is it possible to debug what is breaking those rules or do you know if is there any known problem related to Magento?

The hiawatha version is 10.1 and os is Centos 7 64 bit, PHP is 5.6

Thanks in advance

What UrlToolkit are you using?

I'm using this two Toolkits (but it doesn't work even if I disable the "slashes" toolkit):

UrlToolkit {
        ToolkitID = magento
        #Do Call scannerblocker
        #Do Call vulnerabilityblocker
        Match ^/var DenyAccess
        Match ^/api/rest Rewrite /api.php?type=rest
        Match ^/(media|skin|js)(/|$) Return
        #RequestURI exists Return
        Match ^/.*\?(.*) Rewrite /index.php?$1
        Match ^/.* Rewrite /index.php
}

UrlToolkit {
        ToolkitID = slashes
        Match /{2,} Rewrite / 9
}

Can you also show me the errorlog entries containing the error message you mentioned?

64.228.xx.195|Sat 26 Mar 2016 21:25:13 +0100|access denied via URL toolkit rule

Do you have any idea on what URL the error triggers?

No, that's the problem. I can't find out the url. Is it possible to make Hiawatha more verbose?

The URL is very likely already in the logfiles. Look for a 403 in the access log. The only line with a DenyAccess is the /var entry. What if you disable that one? Not permanently of course, just for testing.

Hugo, we finally found out that Magento caching system was redirecting users to incomplete urls due to a bug, so URL toolkits are correct and Hiawatha is processing them as it always does. However, we found that Hiawatha 10.1 is randomly and unexpectedly getting stuck in an unreliable state in some scenarios. Mostly this bug occurs when I try to upload big files. Hiawatha generates an high (100%) cpu load while it gets at 3% max on older releases.
I had to revert all of my deployments to 10.0 release in order to getting them working flawlessly again.

Are you aware of this strange CPU overload?

No, I will take a look at it. You mentioned uploading big files. What size are you talking about?

It seems to be sufficient to upload a little file to see it squashing my CPU.
It is noticeable with Pydio and Magento and it's not so hard to reproduce.

Hi Hugo,
could you test it? Were you able to encounter this bug?

I will take a look at it soon. I'm quite busy at the moment.

Ok, don't worry. I reverted my installations to 10.0 and I got them working stably again.

I've done some testing, but I can't reproduce your issue. Uploading files does not increase the CPU load on my server.

How is the uploading done? POST request? PUT request? And is it really Hiawatha causing the high CPU load or a Magento PHP script?

I've made a diff between 10.0 and 10.1, but I can't find any code change that could lead to a loop. I installed Pydio 6.4.1 and was able to upload files without any problem.

What Hiawatha configuration are you using for your Pydio website?

Hi Hugo,
sorry for my huge delay but I was busy and I didn't know that you replied to me since there's no notification system on this forum

However, I made some tests and I found the feature that causes the hangs on the upload process:

PreventSQLi = yes

If I disable this protection, Hiawatha 10.1 works flawlessly, while Hiawatha <10.0 worked even when this protection was enabled.
I also encountered this problem on a Magento website.

Can I help you to debug it?

Thanks,
Alessandro

P.S. PreventSQLi works on version 10.0 and prior versions.

The PreventSQLi option is meant to protect applications which *are* vulnerable for SQL injection. It does a pattern match at the entire request. The bigger the request (large uploads), the more work it is. My advice is to not use that option, unless you are sure your application is vulnerable for SQL injection or you have a strong indication that it is.

In v10.1, I changed the SQL injection detection significantly, because there were ways to bypass it. It now offers much more certainty to block actual SQL injection attacks, at the price of a higher chance of causing problems for non-vulnerable applications.

If the PreventSQLi option causes trouble, than the request would also have failed when your applications was vulnerable for SQL injection. With that in mind, I rewrote the SQL injection detection routine. In previous versions, my focus was not to disrupt non-vulnerable applications, at the price of not being able to stop all possible attacks.

Several other users are having problems with the new detection routine. I'll see what I can do to make it a bit more configurable. It's now only on or off.

Great,
thanks for your great explaination! I'll keep in mind it.

So, we can consider this topic closed.
Thanks Hugo.