Cannot deploy diego with latest code and stemcell 3126

[StanleyShen]

Hello all

I am following the page and tried to deploy latest code.

It keeps failing for

Started updating job database_z1 > database_z1/0 (canary). Failed: database_z1/0' is not running after update (00:02:20) Error 400007:database_z1/0' is not running after update.

bosh-warden-boshlite-ubuntu-trusty-go_agent | ubuntu-trusty | 3126* | dfd5572b-0fce-4b44-4395-82259620ec3e

I downloaded the latest stemcell 3126, not the one via "bosh public stemcells, and download the latest Warden Trusty Go-Agent stemcell "
I deployed it several days ago with stencil 389 and it's fine.
I am not sure if it's related to stencil 3126 or the latest code.

when the error occurs, I can see something like:

[2015-11-24 01:42:06 #29789] [canary_update(database_z1/0)] DEBUG -- DirectorJobRunner: SENT: agent.4ab59606-8979-4258-a4ea-3897e3d59817 {"protocol":2,"method":"get_state","arguments":[],"reply_to":"director.68a99df4-b87c-4230-8cd5-b320bd44d09f.802f5615-63db-4dba-9757-0a0575953339"}
D, [2015-11-24 01:42:06 #29789] [] DEBUG -- DirectorJobRunner: RECEIVED: director.68a99df4-b87c-4230-8cd5-b320bd44d09f.802f5615-63db-4dba-9757-0a0575953339 {"value":{"properties":{"logging":{"max_log_file_size":""}},"job":{"name":"database_z1","release":"","template":"etcd","version":"3037015104836fd4969854a15165cce9bfb8acfd","sha1":"5e2e432b550ce8aef336a7bb01570165621603f1","blobstore_id":"33c2ad4c-0065-4a8f-95c6-5b226ca5652e","templates":[{"name":"etcd","version":"3037015104836fd4969854a15165cce9bfb8acfd","sha1":"5e2e432b550ce8aef336a7bb01570165621603f1","blobstore_id":"33c2ad4c-0065-4a8f-95c6-5b226ca5652e"},{"name":"bbs","version":"bd3d9c68a7a3bbe9a93884d7c1f266e8f1ef9cac","sha1":"cff260fcead9cccb18953ee7e354982695ed225a","blobstore_id":"d89aec85-5a8d-4b6c-b84a-df4a55746651"},{"name":"consul_agent","version":"df7504712a02bb465a917f100ff40c51a0da32cf","sha1":"3ba66eeda24320ca91e300cc40a33144bdd2e6cd","blobstore_id":"e463bd72-3362-4ef4-80d3-4979a1430a88"},{"name":"metron_agent","version":"c009de792027d65e490e5c9bd0b09c9add020ed7","sha1":"96b473324fda5d5ed0d12f1af5785ff58d01f7fb","blobstore_id":"0175b11d-8948-4323-ae3c-fa8ed7db22f8"}]},"packages":{"bbs":{"name":"bbs","version":"325bef21f3f7e7cc4d1bc55a06cd518eca8dffed.1","sha1":"94bbff6b653b6c96dcbabc4c19820a79fe8634e5","blobstore_id":"43399447-dd00-4bc1-7af1-acbda052c4e5"},"common":{"name":"common","version":"e401816a4748292163679fafcbd8f818ed8154a5.1","sha1":"d5fc7b5d0c0bf68975425ff879fed4f03f6af399","blobstore_id":"5f75ad05-e410-4573-7d05-0c7132eac8ed"},"consul":{"name":"consul","version":"14b83378b30a2b55a25e641e835af3e5c87a0d41.1","sha1":"6cf08a72ac0805398af033d46294b6b57a9e7c0c","blobstore_id":"f267949c-9619-4db7-4d95-df6673dca130"},"consul-common":{"name":"consul-common","version":"ffab9ae7bea8a053aacca8816681e241b0fab30b.1","sha1":"bf5c7f918578023e671bb1080b5e30cc299777b5","blobstore_id":"7f29003e-a438-43f5-6f02-1fb13d64983a"},"etcd":{"name":"etcd","version":"b38616a5b7e6cff0f1db95facb6b2e729fba0c30.1","sha1":"489bf4c713e2b4c65253981d70c78925a84429a6","blobstore_id":"8f50f27e-a391-49f0-598a-dba2a008d27b"},"etcd-common":{"name":"etcd-common","version":"a5492fb0ad41a80d2fa083172c0430073213a296.1","sha1":"da095d3281ca6b8b0215a1edcd98b19151e5a3d2","blobstore_id":"bf3e565b-5a7b-4a6c-af40-01401d0a7e17"},"metron_agent":{"name":"metron_agent","version":"e50a1584f6163481e1627ec3a7a05d2950dbcc5f.1","sha1":"c3df649bc57735110c1df91b40ae0159cedd1558","blobstore_id":"ca44ea37-dd78-4a7f-726b-650dffd120ac"},"pid_utils":{"name":"pid_utils","version":"81f185e4c02f2e4cb7b73a8f243d1918c6408e50.1","sha1":"b5b4a444b19446e935c6e766ea4776f71afff1f3","blobstore_id":"4245a240-1dec-447c-6eb0-15f4577c8384"}},"configuration_hash":"e134bff8ac0f3f24c19fd0cf3bfd8c7394dad70f","networks":{"diego1":{"cloud_properties":{},"default":["dns","gateway"],"dns_record_name":"0.database-z1.diego1.cf-warden-diego.bosh","ip":"10.244.16.130","netmask":"255.255.255.252"}},"resource_pool":{"cloud_properties":{},"name":"database_z1","stemcell":{"name":"bosh-warden-boshlite-ubuntu-trusty-go_agent","version":"3126"}},"deployment":"cf-warden-diego","index":0,"id":"","persistent_disk":1024,"rendered_templates_archive":{"sha1":"77578a3922d011e1399b77d50fa92d2bb1b561df","blobstore_id":"6779316f-abbf-4f0a-963a-1f44c4f64ac3"},"agent_id":"4ab59606-8979-4258-a4ea-3897e3d59817","bosh_protocol":"1","job_state":"starting","processes":[{"name":"etcd","state":"unknown"},{"name":"bbs","state":"starting"},{"name":"consul_agent","state":"starting"},{"name":"metron_agent","state":"starting"}],"vm":{"name":"3c562360-ba5d-4e97-4c72-e7c89eaaebcc"},"ntp":{"message":"file missing"}}}
I, [2015-11-24 01:42:06 #29789] [canary_update(database_z1/0)] INFO -- DirectorJobRunner: Waiting for 12.777777777777777 seconds to check database_z1/0 status

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/108828892.

[emalm]

Hi, @StanleyShen,

It appears that the consul_agent job from cf-release won't function correctly on the 3126 BOSH-Lite stemcell. The BOSH team is aware of the issue, and will fix it as part of https://www.pivotaltracker.com/story/show/107958688. For now, we recommend you use version 2776 of the BOSH-Lite stemcell. I'll fix the README documentation for the time being to point out that that version of the stemcell should be used instead of 3126.

Thanks,
Eric

[emalm]

I've posted to the cf-dev mailing list about this issue at https://lists.cloudfoundry.org/archives/list/cf-dev@lists.cloudfoundry.org/message/2DFNBZCVHL52D6OTVRB4GBQFKXU34UF2/ and made the README change in diego-release, so I'll close this out. Thanks again!

Best,
Eric