deploying redis docker image to bosh-lite, state remains in CLAIMED forever

[shalako]

following instructions for veritas, trying to deploy redis to diego on bosh-lite.

https://github.com/pivotal-cf-experimental/veritas

root@dd83d975-2942-4170-8322-e7f768d3ac5d:~# cat redis.lrp
{
    "process_guid":"82bcf571-630f-4ad3-bfa6-146afd40bded",
    "domain": "redis",
    "rootfs":"docker:///redis",
    "instances":1,
    "ports":[
        6379
    ],
    "action":{
        "run":{
            "path":"/entrypoint.sh",
            "args":[
                "redis-server"
            ],
            "dir":"/data",
            "user":"root"
        }
    }
}

root@dd83d975-2942-4170-8322-e7f768d3ac5d:~# BBS_ENDPOINT=http://bbs.service.cf.internal:8889 veritas desire-lrp redis.lrp
Desiring:
{
  "process_guid": "82bcf571-630f-4ad3-bfa6-146afd40bded",
  "domain": "redis",
  "rootfs": "docker:///redis",
  "instances": 1,
  "env": null,
  "action": {
    "run": {
      "path": "/entrypoint.sh",
      "args": [
        "redis-server"
      ],
      "dir": "/data",
      "user": "root"
    }
  },
  "start_timeout": 0,
  "disk_mb": 0,
  "memory_mb": 0,
  "cpu_weight": 0,
  "privileged": false,
  "ports": [
    6379
  ],
  "log_source": "",
  "log_guid": "",
  "metrics_guid": "",
  "annotation": ""
}

root@dd83d975-2942-4170-8322-e7f768d3ac5d:~# BBS_ENDPOINT=http://bbs.service.cf.internal:8889 veritas dump-store
Tasks
LRPs
  522884f4-c3b0-413a-b588-3c0e56b5702f-11d7bf6c-1dc9-49be-b932-390825871a47
    1 preloaded:cflinuxfs2 PRIVILEGED (256 MB, 1024 MB, 1 CPU)
      8080 => hello-routing.10.244.0.34.xip.io
       0: 57083a59-a64b-46ef-63e7-8ef3feb40c74 cell_z1-0 [RUNNING for 27m51.050250015s]
  82bcf571-630f-4ad3-bfa6-146afd40bded
    1 docker:///redis (0 MB, 0 MB, 0 CPU)
       0: 161d9c5d-34d4-461a-5d93-904a6412b100 cell_z1-0 [CLAIMED for 3.823272526s]
  92bcf571-630f-4ad3-bfa6-146afd40bded
    UNDESIRED
       0: [CRASHED (3 - Container stopped by user) for 5m18.878019295s]
Domains
  cf-apps

lrp state never changes from CLAIMED to RUNNING

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/105056906.

[Amit-PivotalLabs]

It looks like it was claimed for 3.8s, not that long. It might be cycling
through pending, claimed, crashed, pending.... If you "watch veritas
dump-store" do you see the docker:///redis app cycle around the state
machine?

On Tue, Oct 6, 2015 at 4:08 PM, cf-gitbot notifications@github.com wrote:

We have created an issue in Pivotal Tracker to manage this. You can view
the current status of your issue at:
https://www.pivotaltracker.com/story/show/105056906.

—
Reply to this email directly or view it on GitHub
#80 (comment)
.

[shalako]

tried setting properties.diego.executor.allow_privileged: true as was recommended in the past, as redis must run as root. this did not appear to fix the problem.

looks like the executor disappeared, anyway

[emalm]

Hi, @shalako,

What versions of CF and Diego are you running on your BOSH-lite? I'll try to reproduce this on my local instance, once I get it bootstrapped. It may be that the image is taking an unusual time to download, or that garden-linux for some reason can't download it from inside the BOSH-lite VM.

The diego.executor.allow_privileged property is no longer present in the release, as the executor now always allows privileged (run-as-root) actions. The executor is still present, but as a logical process running inside the rep instead of as a separate OS process.

Thanks,
Eric

[emalm]

Hey, @shalako,

I was able to get the redis image running on my BOSH-lite with that same LRP JSON, using BOSH-lite 9000.55.0 and recent builds of cf-release and diego-release, so I suspect the issue might be with garden-linux 0.307.0 and how it's interacting with the system on your 9000.36.0 BOSH-lite image.

I think at one point we made sure to set the garden.mount_btrfs_loopback property set to true in your Diego deployment manifest, to work around these kinds of issues. Is it still set that way?

Also, since we were seeing errors from garden-linux about the /sys filesystem, let's also check whether sysfs is correctly mounted on that directory.

Thanks,
Eric

[shalako]

@ematpl

Thanks so much for trying it out. I destroyed and upgraded the box. Now on 9000.56.0 and everything working so far. Redis came right up. Didn't have to change garden.mount_btrfs_loopback, though now that you mention I remember we had to alter that to get things working before.

In the past I was doing a git pull in the bosh-lite dir, but this time on bringing the box up I noticed a warning suggesting vagrant box upgrade. That appears to have done the trick. Looks like I may not have had to destroy the box either; I'll try an in-place upgrade next time.