Well, drat. Strangely 3 environments I updated last week stopped having the issue after I pushed some changes and restarted the containers. However, one today did not. In fact, it had a new container that suddenly started having the issue.
This environment I am working with today is a big QA environment, so I pushed my debug lines into the conf files and restarted again to trace. I then watched our logging system for indications of pods which were incorrectly firing the SecRules where the variables were empty.
... and of course you are correct. I have no idea why, but out of tons of Linux systems, as you pointed out, some just don't return the files in order apparently. I can see this clearly with the SecRules I placed at the top of each rule file to log and ordering # that increments. Strangely, most systems are returning the rules in order, or at least I think. They could just be loading the variable file before the rules that need them.
Now onto an Ansible play to concatenate and push them as one rule until you all can resolve. Thanks Scott.
PS: This was the ordering I saw in that big QA environment: 901 init loads way after 920 protocol enforcement, which is a glaring issue.
/var\/pdweb\/shared\/waf\/rules\/REQUEST-911-METHOD-ENFORCEMENT.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-955-WEB-SHELLS.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-913-SCANNER-DETECTION.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-921-PROTOCOL-ATTACK.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-930-APPLICATION-ATTACK-LFI.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-931-APPLICATION-ATTACK-RFI.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-932-APPLICATION-ATTACK-RCE.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-933-APPLICATION-ATTACK-PHP.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-941-APPLICATION-ATTACK-XSS.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-944-APPLICATION-ATTACK-JAVA.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-949-BLOCKING-EVALUATION.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-950-DATA-LEAKAGES.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-951-DATA-LEAKAGES-SQL.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-953-DATA-LEAKAGES-PHP.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-954-DATA-LEAKAGES-IIS.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-959-BLOCKING-EVALUATION.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-901-INITIALIZATION.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-980-CORRELATION.conf
/var\/pdweb\/shared\/waf\/rules\/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-934-APPLICATION-ATTACK-GENERIC.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-905-COMMON-EXCEPTIONS.conf
/var\/pdweb\/shared\/waf\/rules\/REQUEST-922-MULTIPART-ATTACK.conf
Edit: I can confirm that concatenating all the rule files together into a large file works and resolves our issues. Unfortunately the ModSecurity events obviously do not have the correct rule file name in them, which is going to be frowned upon by our security team. However, this gets us going until this can be addressed.
------------------------------
Matt Jenkins
------------------------------
Original Message:
Sent: Tue March 28, 2023 05:03 PM
From: Scott Exton
Subject: ModSecurity Details
Matt,
It is definitely possible that the order is different on different machines. The operating system doesn't guarantee the order of the returned files – and so the order is likely to be different on different machines. Here is a stackoverflow conversation which discusses the ordering of returned files: https://stackoverflow.com/questions/8977441/does-readdir-guarantee-an-order
Thanks.
Scott A. Exton
Senior Software Engineer
Chief Programmer - IBM Security Verify Access
IBM Master Inventor
Original Message:
Sent: 3/28/2023 12:14:00 PM
From: Matt Jenkins
Subject: RE: ModSecurity Details
Seems like the rules are loading in order. I added a SecRule to the top of each rule file with a unique ID and had it fire on every event. In the console logs, I saw the events fire in the correct order with the filenames, and double verified with the rule IDs I had added.
So the ordering I witnessed in this test was correct:
/var/pdweb/shared/waf/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
/var/pdweb/shared/waf/rules/REQUEST-901-INITIALIZATION.conf
/var/pdweb/shared/waf/rules/REQUEST-905-COMMON-EXCEPTIONS.conf
/var/pdweb/shared/waf/rules/REQUEST-911-METHOD-ENFORCEMENT.conf
/var/pdweb/shared/waf/rules/REQUEST-913-SCANNER-DETECTION.conf
/var/pdweb/shared/waf/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
/var/pdweb/shared/waf/rules/REQUEST-921-PROTOCOL-ATTACK.conf
/var/pdweb/shared/waf/rules/REQUEST-922-MULTIPART-ATTACK.conf
/var/pdweb/shared/waf/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf
/var/pdweb/shared/waf/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf
/var/pdweb/shared/waf/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf
/var/pdweb/shared/waf/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
/var/pdweb/shared/waf/rules/REQUEST-934-APPLICATION-ATTACK-GENERIC.conf
/var/pdweb/shared/waf/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf
/var/pdweb/shared/waf/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
/var/pdweb/shared/waf/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
/var/pdweb/shared/waf/rules/REQUEST-944-APPLICATION-ATTACK-JAVA.conf
/var/pdweb/shared/waf/rules/REQUEST-949-BLOCKING-EVALUATION.conf
/var/pdweb/shared/waf/rules/RESPONSE-950-DATA-LEAKAGES.conf
/var/pdweb/shared/waf/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf
/var/pdweb/shared/waf/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
/var/pdweb/shared/waf/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf
/var/pdweb/shared/waf/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf
/var/pdweb/shared/waf/rules/RESPONSE-955-WEB-SHELLS.conf
/var/pdweb/shared/waf/rules/RESPONSE-959-BLOCKING-EVALUATION.conf
/var/pdweb/shared/waf/rules/RESPONSE-980-CORRELATION.conf
/var/pdweb/shared/waf/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
Now the question is why in our other environments some of the rules are running without anything in the variables. Those should have gotten loaded in REQUEST-901-INITIALIZATION.conf. So I might need to add a debug rule in that initialization rule file and let it slide into those environments and see if we log that rule before the other rules are firing incorrectly. Because I never saw this issue in my lab in the first place, so it is possible the other environments perhaps are loading the files in a different order. IDK.
------------------------------
Matt Jenkins
Original Message:
Sent: Sun March 26, 2023 09:22 PM
From: Matt Jenkins
Subject: ModSecurity Details
Scott, thanks for the update. I'll have to write some SecRules at the top of each rule file to append the log file name to a variable and log it each time. I can't log it at the end as I cannot guarantee the log statement would get executed at the end. I would use debug statements but I can't seem to get the ModSec debug output on containers (I do see a waf_debug.log on the virtual appliances). I'll try to remember to update this thread with my findings.
------------------------------
Matt Jenkins
Original Message:
Sent: Sun March 26, 2023 05:31 PM
From: Scott Exton
Subject: ModSecurity Details
Matt,
Unfortunately there is no guarantee over the order in which the rules are loaded (I know that there is an outstanding RFE on this). However, once the rules are loaded they should always be executed in the same order. Unfortunately there is no WebSEAL trace available which you could use to see the order that the rules were loaded.
Your idea of merging the rules into a single file is a good one, at least to verify whether there is an issue with the order of the rules.
I'm sorry that I don't have better news.
Thanks.
Scott A. Exton
Senior Software Engineer
Chief Programmer - IBM Security Verify Access
IBM Master Inventor
Original Message:
Sent: 3/24/2023 10:39:00 PM
From: Matt Jenkins
Subject: RE: ModSecurity Details
@Scott Exton A while back you recall I asked about ordering of the rule files loading. Is there any way we can verify this by enabling some debugging? I attempted to turn up debugging in ModSecurity by setting SecDebugLogLevel to 9 but the containers don't write waf_debug.log and the debug output did not go to the console. I'll have to open a PMR to address the WAF debug logging. However, is there something like pdweb.waf or a routing trace or something I can enable to check how those rule files are loaded?
I am running into random issues on some of our proxy instances where they are firing on SecRules that depend on variables set in another file (part of OWASP CRS 4.0/dev). The rules are hitting because the variables are empty. I can only speculate this is because the variables were not set because they loaded after the file(s) with the SecRules.
I opened a case TS012534448 on this problem I believe may be related to rule ordering. It's starting to get a lot of attention here because we were getting ready to switch from using WCP in blocking to using ModSec in blocking. However, after a trough review of the logs, we found random inconsistencies on our instances. I cannot seem to get it to reproduce in my lab. Simple rules are firing are firing every single request, even with the curl container health checks, such as SecRule id 920430 (HTTP PROTOCOL) or SecRule id 911100 (HTTP METHOD), because the variables that drive them are empty. An FYI, the variables are set in the initialization rule file for OWASP, and left as default so they are not overridden in crs-setup.conf (I believe the variables would likely be defined if they were defined in crs-setup.conf, however the rule ordering still may not be correct).
If it is indeed a file load ordering issue, the only thing I can think without changing the ordering is for me to have our automated builds concatenate all the rules files in alphabetical order into one giant file and just have one giant rule file on the environment. I need to get it resolved so we can continue moving forward with enabling ModSec for blocking.
Thanks for any thoughts!
------------------------------
Matt Jenkins
Original Message:
Sent: Mon December 19, 2022 03:44 PM
From: Scott Exton
Subject: ModSecurity Details
Matt,
This issue has been resolved in 10.0.5. Unfortunately there is no way to avoid the memory leak in 10.0.4.
Thanks.
Sent from my iPhone
Original Message:
Sent: 12/19/2022 10:11:00 AM
From: Matt Jenkins
Subject: RE: ModSecurity Details
@Scott Exton Hey in v10.0.4.0 IF1 virtual appliances we are seeing severe memory usage, in fact likely going to have to disable ModSecurity until we can upgrade to v10.0.5.0.
I presume from what you said we should be better in v10.0.5.0 regarding memory leaks being resolved?
Also, is there anything in v10.0.4.0 that can be done to resolve the issue without upgrading to get us by for now?
Whatever it is seems to be load based. The proxies that get more load start consuming memory more aggressively. Restarting the webseal instance frees up the memory, but then it starts consuming again.
------------------------------
Matt Jenkins
Original Message:
Sent: Tue November 29, 2022 06:45 PM
From: Scott Exton
Subject: ModSecurity Details
Matt,
In answer to your questions:
- 10.0.4 is using ModSecurity v3.
- I don't foresee any issues with using the v4 Beta ruleset, but this is not something which we have tested yet.
- I don't have any additional tips. There was a memory leak in the 10.0.4 implementation, but this has been fixed for the upcoming 10.0.5 release.
I hope that this helps.
Scott A. Exton
Senior Software Engineer
Chief Programmer - IBM Security Verify Access
IBM Master Inventor
Original Message:
Sent: 11/29/2022 2:52:00 PM
From: Matt Jenkins
Subject: ModSecurity Details
@Scott Exton A few questions for you (or anyone else using the new ModSecurity technical preview):
1. What version of ModSecurity is in ISVA v10.0.4.0? I just want to make sure I am referring to the correct documentation for either v2 or v3.
2. I see ISVA includes the v3.3.2 OWASP ModSecurity Core Rule Set. Do you foresee any issues of using the v4 beta ruleset? The v3.3.2 doesn't have log4j rules and hence I suspect a lot of other things may be missing that we would want. Granted we could just add log4j in a rule, but I'd rather use the OWASP rules if possible.
3. Do you have any tips/suggestions outside of what configuration comes shipped with ISVA v10.0.4.0 IF1?
Thanks!
Matt
------------------------------
Matt Jenkins
------------------------------