Insufficient resources on memory

[nsharmacovs]

Summary

We have a custom autoscaling job which scales the VM based on memory consumption.
We are calculating the used memory and based on the output of "free -m" which matches with the output of bosh vm --vitals .

Today our scaling job scaled down the VMS. Even though we had enough capacity in term of memory the application started complaining about insufficient memory when starting up on remaining VMs.

I would like to understand how Available memory in REP is calculated by Diego.

I came across https://github.com/cloudfoundry/rep/blob/320f5e9ff0ba2a7bb1294927cbe31ce5af40c987/resources.go#L63-L68

func (c *CellState) ResourceMatch(res *Resource) error {
switch {
case !c.MatchRootFS(res.RootFs):
return ErrorIncompatibleRootfs
case c.AvailableResources.MemoryMB < res.MemoryMB:
return ErrorInsufficientResources
case c.AvailableResources.DiskMB < res.DiskMB:
return ErrorInsufficientResources
case c.AvailableResources.Containers < 1:
return ErrorInsufficientResources
default:
return nil
}
}
It looks like the data is fetched using curl http://10.96.166.55:1800/state Which provides me with
AvailableResources":
{"MemoryMB":0,"DiskMB":139264,"Containers":237},"TotalResources":{"MemoryMB":30720,"DiskMB":163840,"Containers":249}

When I execute bosh VMS --vitals I am getting

| vp_diego_cell_internal_stack_red_z1/1 (9bab400f-0848-4fd4-98cf-116a5360269a) | running | n/a | runner_z1 | 10.96.166.55 | 0.30, 0.30, 0.36 | 0.6% | 0.1% | 0.0% | 72% (21.7G) | 2% (576.2M) | 39% | 47% | 0% |

On the VM i have
vp_diego_cell_internal_stack_red_z1/9bab400f-0848-4fd4-98cf-116a5360269a:~$ free -m
total used free shared buffers cached
Mem: 30662 30336 325 1 1741 6381
-/+ buffers/cache: 22212 8449
Swap: 30663 576 30086

i should be able to push an application of 4 GB easily based on VM capacity.

[Describe the result that you expected to happen.]
I should not get insufficient.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/166887975

The labels on this github issue will be updated when the story is started.

[heyjcollins]

Each app running on a cell will have some memory allocated to it.
Even if an app is not utilizing all of the memory allocated, that memory is "reserved" and not included in the rep's calculation of available memory.

You can use cfdot on your diego-cells to view the resources available on each cell:

• bosh -d <YOUR FOUNDATION NAME> ssh diego-cell/0
• sudo -i
• cfdot cell-states | jq '{cell_id, TotalResources, AvailableResources}'

Hopefully this will help.

[mariash]

@nsharmacovs as @heyjcollins mentioned checking the actual memory on a VM is not correct because Diego will advertise a total amount of memory and space and it reserves it against that total for each app running on a Cell. It doesn’t matter if the app uses that memory or not, it’s still reserved. This can cause the free reserved memory to differ quite a bit from the actual free memory.

Please see these articles:

https://community.pivotal.io/s/article/Starting-or-Staging-an-Application-Results-in-an-Insufficient-Resources-Error
https://community.pivotal.io/s/article/how-to-get-remaining-memory-on-diego-cells

[mariash]

Please feel free to re-open this issue if you still have questions.

[rluan]

Please be aware of this value for DISK SIZE 163840.
"TotalResources": {
"Containers": 249,
"DiskMB": 163840,
"MemoryMB": 30720
},

but when I ssh into the server and get the actual is:

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 3G 0 disk
└─xvda1 202:1 0 3G 0 part /
xvdb 202:16 0 74G 0 disk
├─xvdb1 202:17 0 30G 0 part [SWAP]
└─xvdb2 202:18 0 44G 0 part /var/vcap/data
xvdf 202:80 0 10G 0 disk
└─xvdf1 202:81 0 10G 0 part /var/vcap/store
loop0 7:0 0 1G 0 loop /var/vcap/data/garden/graph/aufs/diff/e7ddc903654daf9986c56a866ab93b1b9104230687b70b5a4cc6c39b657741f4
loop1 7:1 0 2G 0 loop /var/vcap/data/garden/graph/aufs/diff/e272657f48d07f91b9c946c7315ede8fc2765489d04033969347797f704a38b1
loop2 7:2 0 3G 0 loop /var/vcap/data/garden/graph/aufs/diff/57253dfc198e342a8ffd664e1c15447ab4c03ccc94151711e3a94a1ba9d91e20
loop3 7:3 0 2G 0 loop /var/vcap/data/garden/graph/aufs/diff/30731b46059b4e9f57b6f424a9ea85e1fc6999d95431544062f72b1de7965602

df -hm

Filesystem 1M-blocks Used Available Use% Mounted on
udev 15327 1 15327 1% /dev
tmpfs 3068 1 3068 1% /run
/dev/xvda1 2960 1245 1549 45% /
none 1 0 1 0% /sys/fs/cgroup
none 5 0 5 0% /run/lock
none 15340 2 15338 1% /run/shm
none 100 0 100 0% /run/user
/dev/xvdb2 44260 21978 20012 53% /var/vcap/data
tmpfs 1 1 1 2% /var/vcap/data/sys/run
/dev/xvdf1 9949 23 9398 1% /var/vcap/store
tmpfs 3 0 3 0% /var/vcap/data/executor/instance_identity
/dev/loop0 1008 310 632 33% /var/vcap/data/garden/graph/aufs/diff/e7ddc903654daf9986c56a866ab93b1b9104230687b70b5a4cc6c39b657741f4
none 1008 310 632 33% /var/vcap/data/garden/graph/aufs/mnt/e7ddc903654daf9986c56a866ab93b1b9104230687b70b5a4cc6c39b657741f4
/dev/loop3 2016 1366 533 72% /var/vcap/data/garden/graph/aufs/diff/30731b46059b4e9f57b6f424a9ea85e1fc6999d95431544062f72b1de7965602
none 2016 1366 533 72% /var/vcap/data/garden/graph/aufs/mnt/30731b46059b4e9f57b6f424a9ea85e1fc6999d95431544062f72b1de7965602
/dev/loop2 3024 2124 732 75% /var/vcap/data/garden/graph/aufs/diff/57253dfc198e342a8ffd664e1c15447ab4c03ccc94151711e3a94a1ba9d91e20
none 3024 2124 732 75% /var/vcap/data/garden/graph/aufs/mnt/57253dfc198e342a8ffd664e1c15447ab4c03ccc94151711e3a94a1ba9d91e20
/dev/loop1 2016 919 980 49% /var/vcap/data/garden/graph/aufs/diff/e272657f48d07f91b9c946c7315ede8fc2765489d04033969347797f704a38b1
none 2016 919 980 49% /var/vcap/data/garden/graph/aufs/mnt/e272657f48d07f91b9c946c7315ede8fc2765489d04033969347797f704a38b1

so, where is this number 163840 from ? seems like a default value from some hard coding...