Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wazuh-manager 3.6.1 segfault ossec-anaysisd #1526

Closed
fgrosse19 opened this issue Sep 14, 2018 · 2 comments
Closed

wazuh-manager 3.6.1 segfault ossec-anaysisd #1526

fgrosse19 opened this issue Sep 14, 2018 · 2 comments

Comments

@fgrosse19
Copy link

Hey,

over the last week we migrated from OSSEC 2.8.3 to wazuh 3.6.1. For this we installed wazuh on a 64bit Ubuntu 18.04 (kernel 4.15.0) machine.
For the migration we followed the instructions in the wazuh documentation and used the wazuh-puppetmodule (modified for 3.6.1) to deploy the agents.
Everything was working fine but since yesterday evening we are receiving segfaults from ossec-analysisd on the manager.
After analysisd exited, ossec-remoted also stops working because "/queue/ossec/queue" is not accessible and the agents lose the connection to the manager.
Yesterday we added the last few agents to wazuh, so now we 97 active Agents.
The only other thing we changed were the agent ids because with the puppet module some of our agents got the same id, so we had to change the function in the puppet module and every agent got a new id.
But after that change the manager ran for over 6 hours without a problem.

After we start the wazuh-manager it takes between 5 and 30 minutes until the segfault appears.

Can you tell us what the problem is or how we can fix it?
Just tell me if you need more information.
Thanks in advance.

Here are some logfiles from our server:

ossec.log

2018/09/14 14:29:56 ossec-syscheckd: INFO: Syscheck scan frequency: 79200 seconds
2018/09/14 14:30:11 ossec-syscheckd: INFO: Starting syscheck scan (forwarding database).
2018/09/14 14:30:11 ossec-syscheckd: INFO: Starting syscheck database (pre-scan).
2018/09/14 14:30:48 ossec-syscheckd: INFO: Initializing real time file monitoring engine.
2018/09/14 14:30:51 ossec-syscheckd: INFO: Real time file monitoring engine started.
2018/09/14 14:30:51 ossec-syscheckd: INFO: Finished creating syscheck database (pre-scan completed).
2018/09/14 14:30:56 ossec-syscheckd: INFO: Ending syscheck scan (forwarding database).
2018/09/14 14:31:06 ossec-syscheckd: INFO: Starting syscheck real-time monitoring.
2018/09/14 14:32:21 ossec-remoted: ERROR: socketerr (not available).
2018/09/14 14:32:21 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/14 14:32:21 ossec-remoted: ERROR: socketerr (not available).
2018/09/14 14:32:21 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'.
2018/09/14 14:32:21 ossec-remoted: ERROR: socketerr (not available).
2018/09/14 14:32:21 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'.
2018/09/14 14:32:21 ossec-remoted: ERROR: socketerr (not available).
2018/09/14 14:32:21 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'.
2018/09/14 14:32:22 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:22 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:22 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:23 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:24 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:24 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/14 14:32:24 ossec-remoted: CRITICAL: (1211): Unable to access queue: '/queue/ossec/queue'. Giving up..
2018/09/14 14:32:25 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:26 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:27 ossec-logcollector: ERROR: socketerr (not available).
2018/09/14 14:32:28 ossec-logcollector: ERROR: socketerr (not available).

kern.log

Sep 14 14:32:21 varys kernel: [ 169.153276] ossec-analysisd[524]: segfault at 7f0920089000 ip 00007f09290c6963 sp 00007fff3bbc08a8 error 6 in libc-2.27.so[7f092900b000+1e7000]

gdb ossec-analysisd

2018/09/14 11:45:25 ossec-analysisd: INFO: Total rules enabled: '2258'
2018/09/14 11:45:25 ossec-analysisd: DEBUG: Chrooted to directory: /var/ossec, using user: ossec
2018/09/14 11:45:25 ossec-analysisd: INFO: No IP in the white list for active response.
2018/09/14 11:45:25 ossec-analysisd: INFO: No Hostname in the white list for active response.
2018/09/14 11:45:25 ossec-analysisd: INFO: Started (pid: 11132).
2018/09/14 11:45:25 ossec-analysisd: DEBUG: SyscheckInit completed.
2018/09/14 11:45:25 ossec-analysisd: DEBUG: RootcheckInit completed.
2018/09/14 11:45:25 ossec-analysisd: DEBUG: SyscollectorInit completed.
2018/09/14 11:45:25 ossec-analysisd: DEBUG: CiscatInit completed.
2018/09/14 11:45:25 ossec-analysisd: DEBUG: OS_CreateEventList completed.
2018/09/14 11:45:25 ossec-analysisd: DEBUG: FTSInit completed.
2018/09/14 11:45:25 ossec-analysisd: DEBUG: Accumulator Init completed.
2018/09/14 11:45:28 ossec-analysisd: DEBUG: (unix_domain) Maximum send buffer set to: '212992'.
2018/09/14 11:45:28 ossec-analysisd: INFO: Connected to '/queue/alerts/ar' (active-response queue)
2018/09/14 11:45:28 ossec-analysisd: DEBUG: (unix_domain) Maximum send buffer set to: '212992'.
2018/09/14 11:45:28 ossec-analysisd: INFO: Connected to '/queue/alerts/execq' (exec queue)
2018/09/14 11:45:28 ossec-analysisd: DEBUG: Active response Init completed.
[New Thread 0x7ffff63b5700 (LWP 11241)]
2018/09/14 11:45:28 ossec-analysisd: DEBUG: Startup completed. Waiting for new messages..
2018/09/14 11:45:28 ossec-analysisd: DEBUG: Input message handler thread started.

Thread 1 "ossec-analysisd" received signal SIGSEGV, Segmentation fault.
__memset_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
200 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0 __memset_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
#1 0x00007ffff6ca0a18 in __strncpy_sse2 (s1=0x37c1de0 "", s2=0x7ffff0161dc2 "", n=)
at ../string/strncpy.c:31
#2 0x0000000000452778 in ?? ()
#3 0x0000000000419e43 in ?? ()
#4 0x0000000000412196 in ?? ()
#5 0x0000000000406b5b in ?? ()
#6 0x00007ffff6c19b97 in __libc_start_main (main=0x406480, argc=2, argv=0x7fffffffe528, init=,
fini=, rtld_fini=, stack_end=0x7fffffffe518) at ../csu/libc-start.c:310
#7 0x0000000000406c75 in ?? ()
#8 0x00007fffffffe518 in ?? ()
#9 0x000000000000001c in ?? ()
#10 0x0000000000000002 in ?? ()
#11 0x00007fffffffe787 in ?? ()
#12 0x00007fffffffe7a6 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb) where
#0 __memset_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
#1 0x00007ffff6ca0a18 in __strncpy_sse2 (s1=0x37c1de0 "", s2=0x7ffff0161dc2 "", n=)
at ../string/strncpy.c:31
#2 0x0000000000452778 in ?? ()
#3 0x0000000000419e43 in ?? ()
#4 0x0000000000412196 in ?? ()
#5 0x0000000000406b5b in ?? ()
#6 0x00007ffff6c19b97 in __libc_start_main (main=0x406480, argc=2, argv=0x7fffffffe528, init=,
fini=, rtld_fini=, stack_end=0x7fffffffe518) at ../csu/libc-start.c:310
#7 0x0000000000406c75 in ?? ()
#8 0x00007fffffffe518 in ?? ()
#9 0x000000000000001c in ?? ()
#10 0x0000000000000002 in ?? ()
#11 0x00007fffffffe787 in ?? ()
#12 0x00007fffffffe7a6 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb) list
195 in ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
(gdb) bt full
#0 __memset_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
No locals.
#1 0x00007ffff6ca0a18 in __strncpy_sse2 (s1=0x37c1de0 "", s2=0x7ffff0161dc2 "", n=)
at ../string/strncpy.c:31
size = 0
#2 0x0000000000452778 in ?? ()
No symbol table info available.
#3 0x0000000000419e43 in ?? ()
No symbol table info available.
#4 0x0000000000412196 in ?? ()
No symbol table info available.
#5 0x0000000000406b5b in ?? ()
No symbol table info available.
#6 0x00007ffff6c19b97 in __libc_start_main (main=0x406480, argc=2, argv=0x7fffffffe528, init=,
fini=, rtld_fini=, stack_end=0x7fffffffe518) at ../csu/libc-start.c:310
self =
__self =
result =
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -3680630955903089276, 4222028, 140737488348448, 0, 0,
3680631486955324804, 3680611509687879044}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0,
0x7ffff7de5733 <_dl_init+259>, 0x7ffff7963018}, data = {prev = 0x0, cleanup = 0x0,
canceltype = -136423629}}}
not_first_call =
#7 0x0000000000406c75 in ?? ()
No symbol table info available.
#8 0x00007fffffffe518 in ?? ()
No symbol table info available.
#9 0x000000000000001c in ?? ()
No symbol table info available.
#10 0x0000000000000002 in ?? ()
No symbol table info available.
#11 0x00007fffffffe787 in ?? ()
No symbol table info available.
#12 0x00007fffffffe7a6 in ?? ()
No symbol table info available.
#13 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) cont
Continuing.
Couldn't get registers: No such process.
Couldn't get registers: No such process.
(gdb) [Thread 0x7ffff63b5700 (LWP 11241) exited]
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
@ddpbsd
Copy link
Member

ddpbsd commented Sep 14, 2018

Wazuh issues should be posted to the Wazuh github: https://github.com/wazuh/wazuh
Apologies if that's the wrong repository.

@ddpbsd ddpbsd closed this as completed Sep 14, 2018
@fgrosse19
Copy link
Author

Sorry, too many browser tabs...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ddpbsd @fgrosse19