Poor experience for replicaset configuration with opsworks

[jonapich]

Hi,

I have struggled to use this cookbook in order to setup a mongo 2.6 server in AWS Opsworks and thought I'd share what I did to make it work.

First of all, the doc still mentions the 10gen recipe. The recipe doesn't exist and mongodb_org_repo must be used instead.

mongodb-org must be specified as the package name in the node attributes if you want anything higher than mongo 2.4.

A ton of default paths don't work and create various issues. edelight/chef-mongodb#262 has a workaround which worked for me.

In Opsworks, I had to separate the recipes in different lifecycles. For instance, you cannot play the mongodb_org_repo recipe and then install, it will trigger an apt error. The mongodb_org_repo recipe didn't work when put in the Setup lifecycle, not sure if it conflicts with some other recipes (either aws's builtin recipes or my own). This is not something I ever encountered with cookbooks, so I prefer to mention it. I was able to make it work if I put the repo recipe in "deploy" and the replicaset recipe in "configure".

Finally, I had to modify the recipe to make it work in Opsworks since it doesn't provide the same databag (chef zero?) and the search() command to find replicaset members yield no results.

This is very specific to my use case, but here's the patch anyway (please excuse my n00b ruby skills). The replicaset is created using all of the instances that belong to the same layer as the node being configured. If an instance belongs to multiple layers, all those instances are considered (which may or may not fit one's scenario...)

  if new_resource.is_replicaset && new_resource.auto_configure_replicaset
    rs_nodes = []
    Chef::Log.info("Using OpsWorks mode")
    # fake the resource for the sake of configure_replicaset method in opsworks.
    dn = node[:stack_state_sync][:ds_domain]  # this is our domain name suffix for DNS resolution
    Chef::Log.info("Domain Name is #{dn}")
    layer_ids = search("aws_opsworks_instance", "self:true").first[:layer_ids]
    search("aws_opsworks_instance").each do |instance|
      if instance[:status] == 'online'
        instance[:layer_ids].each do |layer_id|
          if layer_ids.include? layer_id
            Chef::Log.info("#{instance[:hostname]} is part of the replicaset.")
            # this is where I fake the object that is typically passed to configure_replicaset()
            n = {
              'name' => instance[:hostname],
              'fqdn' => "#{instance[:hostname]}.#{dn}",
              'hostname' => instance[:hostname],
              'ipaddress' => instance[:private_ip],
              'mongodb' => node[:mongodb]
            }
            a = OpenStruct.new n
            rs_nodes << a
            break
          end
        end
      end
    end

[shortdudey123]

I am currently in the middle of rewriting the cookbook for a better experience. However, we aren't supporting < 3.0 anymore due to 2.6 being EOL Oct 31, 2016

However, we should be able to support opsworks. (Though i personally am not familiar with it and its differences.) It looks like aws_opsworks_instance is a data bag that has the instances info, correct? Does opsworks not store the node data correctly to use the search that is in this cookbook?

[jonapich]

@shortdudey123 the rs_nodes object that is passed to configure_replicaset is empty and the code reaches the line Chef::Log.warn("Cannot configure replicaset '#{name}', no member nodes found"). I have not traced/debug this, I just assumed it was not present since I remembered opsworks used chef-solo for chef 11. It uses chef zero https://docs.chef.io/ctl_chef_client.html#about-chef-zero when you use chef 12 (i'm in chef 12, of course).

[shortdudey123]

So it looks like calling aws_opsworks_instance for a search makes calls the the AWS API so it gets around the chef nodes not being in the chef-zero instance.

What if you disable auto_configure_replicaset and call the config library (MongoDB.configure_replicaset(new_resource.replicaset, replicaset_name, rs_nodes)) in a wrapper cookbook that passes the rs_nodes that you find from your search?

[jonapich]

@shortdudey123 you know better than I if that would work, I guess that's a nice way to use the cookbook without modifications, for anyone else reading this thread. As far as I'm concerned though, my workaround fulfills my use case so I'll keep it like that. This is just a short term setup while we phase out mongo.

[shortdudey123]

Gotcha, since opsworks doesn't actually provide a chef server, its a bit harder to deal with
I am going to leave this open so i can look at opsworks after i finish the rewrite to see how it would work to integrate that searching / discovery into the cookbook in a more thorough and testable way.

[tayeke]

Thank for this thread @jonapich, almost have my configure recipe set up. I turned off auto configure as suggested.

[github-actions]

Marking stale due to inactivity. Remove stale label or comment or this will be closed in 7 days. Alternatively drop by the #sous-chefs channel on the Chef Community Slack and we'll be happy to help! Thanks, Sous-Chefs.