Replace OSSEC with Wazuh

[ageis]

Feature request

Description

Wazuh is the upgraded fork of OSSEC; from what I hear OSSEC HIDS themselves have even started recommending it instead. It is being more actively maintained and has a wider community of users. If you recall what was on their website two years ago, it has vastly improved since then. So, my argument is you should earmark this instead before you get going on #2134, #2136 or #1756 .

The alerting rulesets in Wazuh are also better. From their website:

"We have modified the existing OSSEC ruleset to increase threat detection capabilities, add functionality and expand OSSEC scope. It includes, among many others, compliance mapping with PCI DSS v3.1, CIS security controls and additional decoders and rules... Our team enforce and review all out-of-the box OSSEC rules, adapting, improving and augmenting their detection capabilities. Empowering generic rules and ensuring they are prepared to face modern technologies and environments... We create new rules and rootchecks periodically that are added to OSSEC scope so they can be used by the users community."

They also have repositories available and so you won't have to build packages anymore. Just apply whatever customization and hardening to your post-installation.

User Stories

The catch is, there's an ELK stack involved, which I know you're familiar with @conorsch , and will expand the resource requirements of servers a bit. Many people complain and find the OSSEC e-mails frustrating, don't they? This way, admins can just look at Kibana, which I would guess they have already pre-loaded with nice vizs/dashs/searchs of log data. GUI alerts.

If you want to keep some e-mail stuff going and the Postfix setups still useful, then Kibana has the Reporting Plugin in X-Pack which can send out a PDF attachment by e-mail of what a dashboard looks like on a custom schedule( via ElasticSearch's watcher plugin. Then e-mail credentials/server details for the scheduled alert reports would go in elasticsearch.yml.

And, at that point, since you have Filebeat, Logstash and ES X-Pack then you could extend it to introduce additional, traditional forms of non-security monitoring (e.g. resource/cpu/memory/disk) that have been lacking on both machines AFAIK just by adding Metricbeat into the mix. You can write a watcher trigger where if a saved Kibana search (for say, any OSSEC alert above level 8) will trigger the e-mail action.

X-Pack is a plus anyway because it requires you to set up security and authentication on ES (unless explicitly disabled), and it has monitoring, alerting and notification and machine learning features.

They use PhantomJS, but I'm not sure if they have the capability to export regular PNG/image screenshots yet. I complained about it in a ticket because I had to develop some really hacky solutions to get a PNG instead of PDF. I mean inotify-cron watching /tmp for when PhantomJS would write its screenshot temporarily and grabbing that instead of PDF, lol.

And, guess what: because there's Filebeat, then in theory it becomes possible for FPF to directly get secure visibility into SecureDrop instance alerts/logs (without revealing source-related activity or metadata of course), instead of having to troubleshoot remotely or have admins paste them over, by adding your Logstash server as an extra output to the Filebeat config.

I would actually use host a Tor hidden service for that idea, since Filebeat has a proxy_url directive where you can use SOCKS5.

I'm just becoming familiar with Wazuh. I haven't had a chance to try it yet, since I only recently learned that it was being preferred to OSSEC. There's a Docker container available for trying it out quickly.

[ageis]

If you take a look at the architecture diagram, you could just replace the components on the Monitor server in the first example. ELK+Wazuh Webapp/API on Monitor Server, Wazuh agent on Application Server.

But alternatively, check out the distributed architecture below that. There might be a chance that FPF could host the ELK stack and Wazuh app for all SecureDrops and ease the burden on the Mon Server, but ONLY if there's a way to have explicit login users and segmentation between sources from Wazuh agents corresponding to each organization in both the Kibana and Wazuh App. Admins logging into your server to check their alerts. Then you'd want the Wazuh manager/Filebeat/API on the Monitor Server, and Wazuh agent on the Application Server. (not sure if manager persistently stores logs from agent or just generates the alerts and discards them)

Of course, there are hard conversations to be had before making such a decision, whether it's safe to do and do securely while protecting sources and whether orgs are comfortable with it, etc. But I know that we previously considered the ability for an SD admin to have a command which opens a temporary debugging terminal tunnel for staff to troubleshoot their instance directly.

On the other hand, the Monitor Server seems underutilized anyway so it should handle the whole shebang fine.

[ageis]

In terms of planning on how to harden parts of this stack, then you'd probably want to look at just running the Wazuh Docker containers on hub, combined with the default Docker seccomp profiles... I looked around and I couldn't find many good AppArmor profiles for ElasticSearch or Logstash.

Anyways, please take some time to review this info over the next few days and whether it could be conceivably be taken seriously, or if you'd rather stick with OSSEC. I'm going to be playing with it anyway when I get some more free time soon and will report back.

[ageis]

Well shit, I just remembered that X-Pack only works for 30 days and then they want a license. When I talked to an Elastic representative, they quoted me at something like $5,500 per year for a start-up subscription on 1-5 nodes (with no support). I felt that was a little exorbitant, but maybe a non-profit would receive a better price on a license. Or just disregard everything I said about X-Pack. Sorry about that!

[ageis]

Very impressed with @wazuh and the work done by @santiagobassett's team. If are using OSSEC, it is time to upgrade to it.

— Daniel Cid (@DanielCid) August 16, 2017

I'd just like to show off some more of Wazuh's magic and how they've built their own-looking, powerful app on Kibana, and attempt to make the case that it's the next generation of OSSEC, since it's just better, both in rule coverage for so more programs, additional features, (I don't remember OSSEC paying attention to syscalls) and user experience.

@redshiftzero @conorsch @dachary @heartsucker @msheiny @garrettr please take a close look at each single image. The title of the views you're observing should be shown in the alt="" text by hovering over.

Just to make a quick point...

All of these companies who work with OSSEC (now owned by Trend Micro), plus Wazuh + AtomicCorp who provide expert support for the former and Sucuri, are contribute and support each other a bit. So it's worth looking at what each is up to differently. AtomicCorp uses OSSEC in their secure Linux product and adds thread intelligence feeds. You'll find LOTS of informative and useful info on @dcid's blog website. The bottom of OSSEC's official site even mentions Wazuh:

"has developed an OSSEC ruleset, to improve detection capabilities...maintaing installers and providing an Open Source ruleset to improve OSSEC detection and log analysis capabilities. Wazuh has developed its own modules for OSSEC integration with Log management systems like Splunk or Elasticsearch. "

I'd need to ask my buddy @compoterhacker about AlienVault another product based on OSSEC and whether it's any good. I haven't touched it and don't know much.

A few things I want to emphasize:

• Even though you can only use X-Pack plugin for 30 days, you'll get by a lot easier without it anyway. I'll maybe edit it out later, but in fact I will perhaps e-mail and ask them whether they have better deals available for non-profits (without representing you) and if they do, I'll get the salesperson's info to give to someone at FPF.
• The fact that you are updating or re-writing the OSSEC rulesets and fixing noise in several tickets during 0.4.x work is no problem at all @redshiftzero, or at least it shouldn't be much of one. Rules in etc/rules between Wazuh and OSSEC-HIDS are relatively similar, just some different author and license fields and filenames, and the Wazuh ruleset includes some more contributions from the community.

So, just to be aware, the Wazuh ruleset has 15 more rules than OSSEC by default, and it covers many of the the same services and whatnot, but they have prepended numbers to the filenames. I'll demonstrate what I mean via Meld (directory diff):

• I'd suggest creating a new repo called securedrop-ossec-rules. So just take what you're starting from (all of OSSEC's core rules) then make a branch to contain which you've specifically changed or customized or added. And then slowly merge on top of them (using an adept Git gunslinger), and then you should be ready to commit & push.
• The Wazuh Ruleset is available as a remote which one can pull/merge in from, or a .zip, but remember that the parent etc/ is missing. And I'm not sure if vanilla OSSEC has a repo for its core rules.

• However, they provide a real useful script for locally updating your entire ruleset automatically!

• I can't imagine that you would have to worry about ever updating OSSEC upstream rules after that. Just keep merging in wazah-ruleset upstream changes to securedrop-ossec-rules which contain your modifications, and then explicitly apply that to the rules folder

(TODO: I'm imagining upstream packages instead of building your own, so might have to make sure whether any updates would overwrite your SecureDrop-customized rules) I could check this quickly but I've already written so much and I'm getting a bit lazy lol.

• Note, OSSEC 2.9.2 just came out 10 days ago and has several new rules contained. Check the changelog. If you are updating this stuff soon anyway, then might as well roll this release in.

• While we're on that topic, it's important to know that there is another fairly well-respected OSSEC fork called "Sucuri" by @dcid. It doesn't seem updated as often, and I'm yet not aware of the substantive differences, and this could be mean more be work or time consuming, but it could be worth diffing the the OSSEC-HIDS Sucuri Fork against the regular release of OSSEC_HIDS or even that of Wazuh, just to see if there is anything worthwhile pulling in from Sucuri. But ideally, I think the only custom things should go in your in your /etc configs, plus rules maintained in a separate securedrop-ossec-rules repository, with some scripts to make it easy to pull in and sync with other remotes. I have to refamiliarize on what's in src/ and /contrib ...

• Let's get more specific. The rules that come bundled in Wazuh are listed here in PDF:
  OSSEC Ruleset

Here's a diagram showing the data flow:

Here's the excellent documentation, starting from the page which explains how to install, update and contribute to the ruleset:

New stuff

• This tool helps people obtain PCI DSS v3.2 compliance. (OSSEC is only v1.2/2.0 I believe) which was implied by Compliance, vuln assessment, security policies and the like? #2145.
• You also get SCAP compliance! With vulnerability assessment and scanning process OpenSCAP, that identifies and classifies vulnerabilities on a system. There's security policies specifically for Ubuntu xenial and trusty.
• There are more, and easier methods of agent registration available to you.
• A RESTful API. Lets you run syscheck or rootcheck, add and delete agents, and more. Very sweet.

Integrator

Integrator allows Wazuh to talk to external APIs and alerting tools:

• It integrates with Slack.
• It integrates with PagerDuty. This is a main app (can send phone calls, SMS, is a website and a mobile app) that I use to receive/acknowledge/resolve high priority alerts in SRE teams, as it lets you manage schedules, escalations, overrides, roles and cycled shifts, etc.
• I'm sure that can't be all and there must be more coming.

Upgrading

Thankfully, upgrading from OSSEC is well-documented and as you saw above the architecture is flexible. You can even make it a gradual rollout if you'd like to simply replace the OSSEC Manager on the Monitor Server first, since Wazuh manager is fully compatible with an OSSEC agent. (though you would lose syscheck there)

Unfortunately, the preferred automation is in Puppet. There are a few Ansible OSSEC roles out there (one of them was started by me), and many of them have an issue request to upgrade to Wazuh. There were one or two for Ansible that covered Wazuh, but seemed primitive and I tend not to trust third-party stuff and always like to write it myself instead. So it's just a matter of converting Puppet to Ansible, unless you want to stick to their well-maintained Puppet stuff as complementary and not have to rewrite your Ansible on occasion during updates. There's also the alternative of Dockerization which they have readily prepared out of the box. It's a plus that the majority of your engineers are already familiar with ELK.

Installation

This is extremely well documented. In terms of ELK, you'll have Filebeat --(uses SSL)-->Logstash --> ElasticSearch will be from the various Elastic repos (artifacts.elastic.co) and relying on openjdk-8-jre.

Every other component link to list
is contained within the Wazuh repos at packages.wazuh.com which is absolutely packaged for Ubuntu trusty and xenial among others.

The Wazuh app is actually just a Kibana plugin, e.g. /usr/share/kibana/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp.zip

Another option to quickly try all of this out is the pre-built virtual machine image on CentOS 7 which you can import into VirtualBox.

What I ask of the SecureDrop devs with this proposal in general:

• Everyone prefers to be taken seriously rather than dismissed, and I've put a lot of research into this already... So, at least read the ticket, main website and documentation. And try the container or VM demos, and I'd like the Lead Developer to get back to me about whether it could conceivably end up on your roadmap, but there's no rush.
• Perhaps place this in milestone #29 which talked about alternatives to OSSEC, and ELK was mentioned. This is literally ELK with OSSEC which makes it so much better. Besides, in my opinion ELK is not really a serious OSSEC alternative. If Wazuh isn't what you want, then we need to go back to the drawing board once again: OSSEC alternatives #1123

[ageis]

Impact on remote support capability

As I pointed out in #973 and earlier, this has a chance to make remote support much easier since Filebeat is present, and thus you could add a second Logstash output listening on a Tor hidden service. (either stealth authenticated or HTTPS over Tor w/ CA client/server keys). Eventually I'll get a chance to look at the Wazuh Logstash instance to see how it tags different agents. You'd either need the filter to tag them per-SecureDrop or put them in separate indicies.

Requirements

• Also, this might not scale well when you get like, even just 30 nodes shipping into one L+ES. Have seen it happen. So, my idea would be, get the latest, best 1U server, harden it, and disk encrypted, and install it in a secret colo that only one or two people can get into (talk to me if you want suggestions where to go). Assign a lot of workers to Logstash input and output, using every core available, and a big heap to multiple ES nodes. Then just load balance Logstash by putting HAProxy in front of it.
• I suggest a physical colo with tier 3 security and whatnot since you may not trust your VPS provider in the event that any logs shipped to you do end up containing things that shouldn't be in them; or, by the inverse, if you do it securely, then analyzing them on a bare metal server you practically treat like an App Server in terms of what security rules are applied to it can be done more safely.
• You may need to modify pfSense Firewall rules, for one thing.

[ageis]

Good comment from Wazuh dev explaining when they forked from OSSEC and their relation to the upstream project:

Wazuh had forked at OSSEC version 2.9 beta. Since then, we have been bringing the new code from OSSEC to the Wazuh project once for each Wazuh release at less. In fact we are working in importing the new rules to our ruleset project and the functional code to this repository.

Regarding to the JSON output, both OSSEC and Wazuh produce alerts in JSON format, and are able to output them via Syslog. The Syslog client (ossec-csyslogd) read the plain-text alerts and transformed them into JSON objects in order to send them:

Log analysis engine (analysisd) → alerts.json
Log analysis engine (analysisd) → alerts.log → Syslog client (csyslogd)

We have introduced some improvements in the alerts, like full agent information (agent ID, name and IP) or dynamic fields. The thing is that the Syslog clients always reads the plain-text alerts no matter if the JSON alert output is enabled, and the Syslog client's plain-text alert parser was outdated.

This enhancement makes the Syslog client read the JSON alert output from the analysis engine and send the data directly to the Syslog server, bringing the new alert capabilities and improving the performance (since there is no alert re-parsing).

Answering your last question, we are talking to the OSSEC guys about moving to Wazuh and we hope so, but it does not depend on us.

How they have introduced full agent info (agent ID, name and IP) makes me wonder whether you could get rid of the Monitor Server server entirely in the future and centralize alert collection with FPF running the Wazuh manager+App/ELK stack (since this component might be difficult for some admins to manage/update/deploy), the App Server becomes an agent. But there are a variety of hard problems and concerns with doing something like that, both securely, and providing proper instance segmentation... And the fact that if an attacker got access, they could just turn the agent off (which itself should be an alert managed by a different monitoring service -- "OSSEC not running")

[ghost]

@ageis What about we start by trying a replacement, to the extent it is possible. I.e. keep just email notifications https://documentation.wazuh.com/current/user-manual/manager/manual-email-report/index.html and use their packages if possible https://documentation.wazuh.com/current/installation-guide/packages-list/index.html#ubuntu . Using the ossec migration documentation you suggested. It would make absolutely no difference/improvement from the SecureDrop admin point of view. It would just migrate to wazuh. Does that sound like something sane ?

Things to check:

• How are the Debian GNU/Linux packages built ? https://documentation.wazuh.com/current/installation-guide/packages-list/index.html only has .deb not .dsc
• Can the wazuh manager be installed and operate with just email notification and nothing else ?

[ageis]

Oh, nice.. There is an official Ansible playbook for Wazuh now: https://github.com/wazuh/wazuh-ansible

[ageis]

Anecdotally, I have used the official Ansible roles to setup Wazuh manager and agent(s) with great success. Conveniently, there are actually two possible avenues for new agent registration: authd (same as OSSEC, optionally using a shared password) or via the Wazuh API (which potentially can be HTTPS and protected via basic auth).

Again, in terms of alerting, logging, configuration, port numbers and file paths, and forgetting about the ELK stack components (e.g. Wazuh Kibana app plus Filebeat and/or Logstash to pipe the alert stream into wazuh-alerts-3.x index) Wazuh is virtually or nearly a complete drop-in replacement for OSSEC. Running /var/ossec/bin/ossec-control status shows just a few extra pieces (I am not sure how this output compares/contrasts with the latest OSSEC):

Agent:

ossec-logcollector is running...
ossec-syscheckd is running...
ossec-agentd is running...
ossec-execd is running...

Manager:

wazuh-clusterd not running...
wazuh-modulesd is running...
ossec-monitord is running...
ossec-logcollector is running...
ossec-remoted is running...
ossec-syscheckd is running...
ossec-analysisd is running...
ossec-maild not running...
ossec-execd is running...
wazuh-db is running...
ossec-authd is running...
ossec-agentlessd not running...
ossec-integratord not running...
ossec-dbd not running...
ossec-csyslogd not running...

Additionally, I've found Wazuh's Debian repositories to be usable, all LTS variants of Ubuntu are supported. E-mail notifications are off by default, but can be turned on via vars which influence the ossec.conf template. Installing from wazuh-ansible, the alerting rules which get installed are customized for your particular Linux distribution.

OpenSCAP

Now here's another cool thing I discovered. Wazuh can be fully integrated (it's disabled by default) with OpenSCAP, which means automated standards compliance, auditing and vulnerability/CVE assessment. IMO this is one of the key advantages of the improved Wazuh ruleset.