Ubuntu 18.04 - Nginx crashing

[sn0rcha]

Hey Guys,

I just did a fresh install of Ubuntu 18.04 and Nginx 1.17.4 and ModSecurity. Whenever I activate any rules (OWASP set for example), Nginx crashes every request with the following error:

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid

Anyone have any ideas?

[bartoszx]

Hi. I've got same problem.
I thought I messed something with my Dockerfile but every build with any version of nginx cause the same error.

[martinhsv]

I have been able to reproduce this as well. However, it does not occur with all rulesets.

For example, I was not able to reproduce this with a slightly older version (~four weeks ago) of CRS3 -- which means that using an older version of CRS3 may be used as a near-term mitigation of this issue.

[sn0rcha]

I have been able to reproduce this as well. However, it does not occur with all rulesets.

For example, I was not able to reproduce this with a slightly older version (~four weeks ago) of CRS3 -- which means that using an older version of CRS3 may be used as a near-term mitigation of this issue.

Any chance you can provide the link to that ruleset so we don't have to try all the old ones :)

[Rondore]

I'm seeing this same issue on Ubuntu 16.04 with nginx 1.16.1 and OWASP rules. I rolled back the latest commit (beedddd) by running git reset --hard d5b93c1013ca9c9a96c75369576b54832e9a2b48 and now I'm not getting the error anymore.

[ei-ke]

I can confirm this for the latest commit to v3/master of ModSec with the current OWASP CRS and the ModSec nginx connector. Using ModSec v3.0.3 doesn't produce that error.

System:

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
Kernel: 4.15.0-65-generic

nginx

nginx version: nginx/1.17.4
built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) 
built with OpenSSL 1.1.1  11 Sep 2018 (running with OpenSSL 1.1.1c  28 May 2019)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.17.4/debian/debuild-base/nginx-1.17.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

2019/10/05 01:40:17 [notice] 13853#13853: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:46:53 [alert] 13855#13855: worker process 13856 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:46:53 [alert] 13855#13855: worker process 13857 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:46:56 [alert] 13855#13855: worker process 13858 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:47:01 [alert] 13855#13855: worker process 13859 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:47:01 [alert] 13855#13855: worker process 14387 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:47:05 [alert] 13855#13855: worker process 14388 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:47:05 [alert] 13855#13855: worker process 14389 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:47:09 [alert] 13855#13855: worker process 14390 exited on signal 6
2019/10/05 01:48:26 [notice] 14443#14443: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:48:58 [alert] 14444#14444: worker process 14445 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:48:59 [alert] 14444#14444: worker process 14446 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:49:02 [alert] 14444#14444: worker process 14447 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:49:02 [alert] 14444#14444: worker process 14448 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:49:07 [alert] 14444#14444: worker process 14476 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:49:07 [alert] 14444#14444: worker process 14477 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:49:08 [alert] 14444#14444: worker process 14478 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:49:38 [alert] 14444#14444: worker process 14480 exited on signal 6
2019/10/05 01:50:05 [notice] 14522#14522: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
2019/10/05 01:50:15 [notice] 14541#14541: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 01:50:17 [alert] 14543#14543: worker process 14544 exited on signal 6
2019/10/05 01:53:14 [notice] 14748#14748: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
2019/10/05 02:04:02 [notice] 927#927: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:04:18 [alert] 1096#1096: worker process 1099 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:04:27 [alert] 1096#1096: worker process 1097 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:04:31 [alert] 1096#1096: worker process 1098 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:05:05 [alert] 1096#1096: worker process 1100 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:14:00 [alert] 1096#1096: worker process 1819 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:15:11 [alert] 1096#1096: worker process 1820 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:15:11 [alert] 1096#1096: worker process 1821 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:15:11 [alert] 1096#1096: worker process 1840 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:15:16 [alert] 1096#1096: worker process 8878 exited on signal 6
2019/10/05 02:15:39 [notice] 8918#8918: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:15:45 [alert] 8920#8920: worker process 8921 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:16:51 [alert] 8920#8920: worker process 8922 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:16:51 [alert] 8920#8920: worker process 8923 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:16:56 [alert] 8920#8920: worker process 8924 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 02:17:26 [alert] 8920#8920: worker process 8926 exited on signal 6
2019/10/05 02:17:46 [notice] 8972#8972: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
2019/10/05 02:18:30 [notice] 9000#9000: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
2019/10/05 12:08:24 [notice] 14682#14682: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 12:09:01 [alert] 14683#14683: worker process 14684 exited on signal 6
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
2019/10/05 12:09:21 [alert] 14683#14683: worker process 14685 exited on signal 6
2019/10/05 12:12:52 [emerg] 15587#15587: dlopen() "/etc/nginx/modules/ngx_http_modsecurity_module.so" failed (libmodsecurity.so.3: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:1
2019/10/05 12:22:12 [notice] 27356#27356: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
2019/10/05 12:22:15 [notice] 27362#27362: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)
2019/10/05 12:30:24 [notice] 27467#27467: ModSecurity-nginx v1.0.0 (rules loaded inline/local/remote: 0/1818/0)

PS: dlopen() "/etc/nginx/modules/ngx_http_modsecurity_module.so" failed (libmodsecurity.so.3: cannot open shared object file: No such file or directory) can be ignored, I was updating the module.

[airween]

Could you try to reproduce this issue with this fix?

[bartoszx]

I cant reproduce this issue with above fix.
It is working ok right now.

[ei-ke]

@airween I can confirm that #2179 solves this problem. Thank you!

[airween]

You don't need to say thanks for me :), the author of patch is @defanator.

And hope that these information helps to developers.

[ei-ke]

@airween whoops, then thanks to @defanator :)

[zimmerle]

Duplicate of: #2178; Fixed by @defanator at: #2179