Fix dangling directories in /var/vcap/data/rep/shared/garden/download_cache/

[yangwenyanibm]

cachedownloader leaving dangling directories in cell's ephemeral disk

Summary

Some old folders exist in /var/vcap/data/rep/shared/garden/download_cache/ occupy pretty much ephemeral disk space

/:/var/vcap/data/rep/shared/garden/download_cache# ls -Art
08e216965c1b52a0935f894c304a3efc-1610186769820162404-1.d    08e216965c1b52a0935f894c304a3efc-1610188672213112665-222.d	3412840c404781c0090939e948fb94b4-1610189250050810845-292
e5ae9c977194ba23ad109cdd27446f8b-1610186787613538643-4.d    f45f9f4410ea7cc4870a8e4ac65af0d6-1610188682297496841-223	5bb49226b476bfa8927510eada572ac8-1610189265469824698-294
08e216965c1b52a0935f894c304a3efc-1610186854261952639-8.d    08e216965c1b52a0935f894c304a3efc-1610188684627167716-224.d	52b97d296aa4eee53c49168a47b46e87-1610189271069112279-296
8d57cad7f3c96d37dd98b83ecec2090a-1610186868197720018-12.d   d4bd3c59903057d7b7e66798766c6849-1610188697199906564-225	08e216965c1b52a0935f894c304a3efc-1610189288601641319-299.d
97b9219dd62b4bbd1015b55bc1405d4c-1610186895752414909-15.d   b8453b3d993ace332220991e1d39a9b8-1610188722201199299-226	9b5411071ca1975421052efba9d506f8-1610189289845036272-300
08e216965c1b52a0935f894c304a3efc-1610186902097463479-18.d   a901806fe40a346b64b194677f7176a2-1610188725560756657-227	ff1611934d832cc8cb6e3d4ff199aaa0-1610189293120453945-301
5d7d548c1c5bd9992104f3af5faa5c82-1610186945123247910-26.d   8798193240e2aae0f4c9c07b4bfa1ea7-1610188733898481456-228	08e216965c1b52a0935f894c304a3efc-1610189301365721327-303.d
08e216965c1b52a0935f894c304a3efc-1610186961613578245-28.d   ecd135f74fe26fd760008698f898aed4-1610188747161896901-229	5b299eaa27c78c54f8051dd48ef37cd1-1610189302392982599-304
08e216965c1b52a0935f894c304a3efc-1610187011990900615-32.d   08e216965c1b52a0935f894c304a3efc-1610188758356679813-231.d	c82bb2b43f6a3e14ea0d3c61c11777fc-1610189307698614001-305
08e216965c1b52a0935f894c304a3efc-1610187072595176904-44.d   8f971d626996340635b3929384afaa23-1610188767006430365-232	efe40f2bd6e5c1c314d6d612c6eb4f12-1610189355801411465-306
08e216965c1b52a0935f894c304a3efc-1610187098177928618-47.d   08e216965c1b52a0935f894c304a3efc-1610188768182445715-233.d	08e216965c1b52a0935f894c304a3efc-1610189379416497936-308.d
08e216965c1b52a0935f894c304a3efc-1610187107143628792-48.d   f12fe54c8a8a109c58c95a0ede8bfd51-1610188771627322322-234	d0d1866374ded59912164bffbc2697f8-1610189381478467002-309
08e216965c1b52a0935f894c304a3efc-1610187123662794698-51.d   a1b17324b2d96a47d7358e671ea586cb-1610188773285968088-235	73f71fe15c09ccd608a3fba4437b2278-1610189423273158600-312
08e216965c1b52a0935f894c304a3efc-1610187157489525637-55.d   a6b65905c16ac004b1496e73c267a7fe-1610188783593729558-236	08e216965c1b52a0935f894c304a3efc-1610189428920760918-313.d
08e216965c1b52a0935f894c304a3efc-1610187216095149923-61.d   08e216965c1b52a0935f894c304a3efc-1610188798883955547-237.d	c6d9e4dd223ba9488b559078146e6621-1610189433066957393-314
81a9a1987207f7e71fe7a37911848ea4-1610187236191872022-64.d   faba9c71de2f42f7601db5ee6ff9d403-1610188800975158894-238	65982c7e5a86accad095cbcb309cb2eb-1610189438209053024-315
08e216965c1b52a0935f894c304a3efc-1610187258921286227-65.d   08e216965c1b52a0935f894c304a3efc-1610188812649802950-239.d	d307ccc978c763c462952655cd20d291-1610189457876859311-317
08e216965c1b52a0935f894c304a3efc-1610187319496517865-74.d   7dcca057df995d36b8c02cffef6e008d-1610188814331883953-240	08e216965c1b52a0935f894c304a3efc-1610189468217947821-319.d
08e216965c1b52a0935f894c304a3efc-1610187364768357693-80.d   9f86a79a52f4e31ec22023b9b5e00ed7-1610188823222168383-241	08e216965c1b52a0935f894c304a3efc-1610189478283934864-320.d
08e216965c1b52a0935f894c304a3efc-1610187402224863069-84.d   ee275c1a61c4a53be24812f54c70c138-1610188848737496803-242	6b67b64c6492933303761ba981b8fe7d-1610189481593151672-321
555c0bf1a1383235e759cc9eec6c8184-1610187512210743253-101.d  b20ae8ba013006cffa06587ab5c6feaf-1610188877359858729-243	751601ffe3ad8932a5a7a9003a47e2f2-1610189482342190252-322
08e216965c1b52a0935f894c304a3efc-1610187531782729783-103.d  4f6e0b647d7e40588ac31cefeaa324d6-1610188881770121071-244	5f877cfbf73daeb0085d16e2fc9ee358-1610189495302660708-323
08e216965c1b52a0935f894c304a3efc-1610187556498415756-106.d  08e216965c1b52a0935f894c304a3efc-1610188901301815832-246.d	a926bcb136ed6cb61033227528029128-1610189504463801166-325
08e216965c1b52a0935f894c304a3efc-1610187575987467623-109.d  5372fe25ea9dfc6916561ea1a680447a-1610188901805298741-247	49cc6251b24a956b77ac55595fec8f7a-1610189530425407550-328
08e216965c1b52a0935f894c304a3efc-1610187680428383923-120.d  b294a807314df7f1cd7edf48da8d2b4d-1610188912067911714-248	b8fccc06d89d73f51689bb6b9aecc522-1610189557692829756-329
08e216965c1b52a0935f894c304a3efc-1610187728260821143-126.d  fc412a1bda755edc82d0c2a5c659848d-1610188920900105296-249	0c86070f49b9e440ee98eb89d5993598-1610189572484319126-331
08e216965c1b52a0935f894c304a3efc-1610187759579570647-130.d  17a466aa7b34f49a11727d64478f5055-1610188924797337607-250	08e216965c1b52a0935f894c304a3efc-1610189573403469017-332.d
08e216965c1b52a0935f894c304a3efc-1610187801981468436-136.d  08e216965c1b52a0935f894c304a3efc-1610188928835533019-251.d	431b3ab6845ff6e8d2df31e8e6eb484a-1610189583942109766-333
08e216965c1b52a0935f894c304a3efc-1610187898546006417-143.d  b5f05c28fd521d95ba2107e7ede3a5cf-1610188941318392355-252	46360f6698d25f91752aa1aeb48b13d2-1610189616026630743-335
08e216965c1b52a0935f894c304a3efc-1610187924023262692-147.d  1a039a796cf75e2673f856c734e49773-1610188955410694764-253	08e216965c1b52a0935f894c304a3efc-1610189647026772157-336.d
08e216965c1b52a0935f894c304a3efc-1610187947462663163-150.d  08e216965c1b52a0935f894c304a3efc-1610188962003498455-254.d	a9a0fbef4cb46596ad7166188a0cfbed-1610189649506949090-337
08e216965c1b52a0935f894c304a3efc-1610187952787772794-152.d  9ba85231575ac0ac558a0bff279931d3-1610188967189530160-255	175bab733f2d90e421bde5853afa9d74-1610189656366850000-338
08e216965c1b52a0935f894c304a3efc-1610188017281578539-158.d  c6781a53aad13d25189dfbe6b72ede7e-1610188976035682697-256	3162aa58dc57a32fb16ce3ec411a18ae-1610189686591337861-339
08e216965c1b52a0935f894c304a3efc-1610188031968198672-161.d  51f574712987fa524c7565336a2416cc-1610188990000279099-258	b0dc4f4aa8e60909537b97a7cd46cc4e-1610189699597807168-340
08e216965c1b52a0935f894c304a3efc-1610188064061645409-164.d  fe72af7a6fb8ee505e82bcea130cce14-1610189007228075271-260	4748784121761e562dea87ccb8707414-1610189707483538249-342
08e216965c1b52a0935f894c304a3efc-1610188176954117680-171.d  08e216965c1b52a0935f894c304a3efc-1610189007988618811-261.d	08e216965c1b52a0935f894c304a3efc-1610189710412258343-343.d
08e216965c1b52a0935f894c304a3efc-1610188214382356536-176.d  bb449f808796915afc77eb493a1def1d-1610189016876700705-262	d4ed9e8cb1f6ebcd50b28b31c9ad4a9b-1610189714758246166-344
08e216965c1b52a0935f894c304a3efc-1610188225917930831-179.d  439cc1212749015faf86e88abdf67c02-1610189020995922640-263	27ba98c574bf7e42573f2aa97e0d1cb7-1610189736971395596-345
08e216965c1b52a0935f894c304a3efc-1610188350621955970-188.d  57a3e215ec85c5130e147c10aef42ac3-1610189036444585758-265	6ef73c3004ddcbd8de5a37246415e85a-1610189739563600404-346
08e216965c1b52a0935f894c304a3efc-1610188411838410535-194.d  d2917aa1c966cc2c17aee929f2b18e95-1610189039290725376-266	0d2d07eb5581ec9852127de19166ab26-1610189749777840780-347
08e216965c1b52a0935f894c304a3efc-1610188469739333911-202.d  06770b3d20e85e0d0c6ce6fd79f32150-1610189058454590741-268	08e216965c1b52a0935f894c304a3efc-1610189785539431572-349.d
08e216965c1b52a0935f894c304a3efc-1610188542848529618-206.d  08e216965c1b52a0935f894c304a3efc-1610189061104731405-269.d	cd1b08aa027b439b10fd8ad14bac9b83-1610189790100140504-350
88a8815ffbcb34611bc2bf6946c0504b-1610188544430589389-207    dc2896db1cae17872cd82af06cf63603-1610189065853943103-270	c46baab8cede06ca1042a4bc964c03e6-1610189826306395254-353
dd842457f872366666f87c6ff97cb953-1610188545131341643-208    424618bdb067a58d0535c82030cc1f83-1610189084641323020-273	600e43ef3ee77554344f08fd7fdf9fb3-1610189828340251193-354
1da6d3507f96df657092eb8fbfc83fc0-1610188554541434371-209    08e216965c1b52a0935f894c304a3efc-1610189086955012783-274.d	5be22b22b813ebd7811c74f37f2b2c5d-1610189838359030778-355
c9a6eb81f3994aeb2fd1d404eed91441-1610188556989099672-210    eeb28d83e010fcebd50f0d3a60d6fc17-1610189089620832233-275	08e216965c1b52a0935f894c304a3efc-1610189851669165540-356.d
3bfc3a38e9dc1d4fd98134ea16351681-1610188575040263782-211    e4ee7dc54ea5cce50d7946e479166c8e-1610189121038410111-277	f1e8c03925f8518a8d35d92217a74d3d-1610189857944798436-357
08e216965c1b52a0935f894c304a3efc-1610188609785976934-213.d  08e216965c1b52a0935f894c304a3efc-1610189143444393612-278.d	9fbfeb78fd409de9c466d58087ba9130-1610189859511928463-358
60f9f9ed972b331b8d13b0abcead7258-1610188615754508072-214    e32cbcce4b4a1eac698971b7a7757332-1610189150087933586-279	08e216965c1b52a0935f894c304a3efc-1610189871098649290-359.d
04f38558451a16532d0e0f9519b76e11-1610188628799788373-215    e588e45a993ba4be90ab40bab57e8f5f-1610189159971335854-280	7ec17cac349dc66e86b96cc6b45fc2bd-1610189874623408059-360
1d408e4d2c03159a71684514e1d45ab5-1610188642623650352-216    e99d696f0e2a96481c8aaae0724e1225-1610189177132843621-282	0745a7b8529fcf9b7b6818cced00251d-1610189880493845942-361
08e216965c1b52a0935f894c304a3efc-1610188651905614154-217.d  e8c4658d12053d6eac3f5770587eea41-1610189190258997677-284	86d432c825c156586847a32731ca2b53-1610189893724257102-362
d1ef485b38748d6b256b43fd623c6e93-1610188657794687436-218    449d4a589c7bf597c7207776b86bb070-1610189209559161297-286	f5b299128bb168f99f3e88ebd4db96c7-1610189900265172771-363
08e216965c1b52a0935f894c304a3efc-1610188660628275999-219.d  4a9859320f805760e2959d18e714f296-1610189224412554234-289	08e216965c1b52a0935f894c304a3efc-1610189910048121323-364.d
9a675ca7e1326f33a1edaabb870f4aff-1610188665072490446-220    08e216965c1b52a0935f894c304a3efc-1610189231561449401-290.d	temp
fad02b46a10af48762cae0a6272b9acf-1610188666078021449-221    ddc93c37b26238229ea2c117075fb651-1610189233371033844-291

Steps to Reproduce

• I have 6 Diego cells each with 100G ephemeral disk.
• Running about 90 application instances in the environment.
• Push 12 applications with about 1 second interval periodically.
• Some of the cells' ephemeral disk usage will grow to 90% in a few hours.

Diego repo

cachedownloader

Environment Details

• diego-release version: 2.48.0
• related config in /var/vcap/jobs/rep/config/rep.json:

    "max_cache_size_in_bytes": 10000000000,
  

Some Investigations

• I have adding logs in cacheddownloader/file_cache.go#366 as bellow:

  func (c *FileCache) updateOldEntries(logger lager.Logger, cacheKey string, entry *FileCacheEntry) {
  	if entry != nil {
  		logger = logger.Session("file-cache.updateOldEntries", lager.Data{"cache_key": cacheKey, "in_use": entry.inUse(), "ExpandedDirectoryPath": entry.ExpandedDirectoryPath})
  		if !entry.inUse() && entry.ExpandedDirectoryPath != "" {
  			// put it in the oldEntries Cache since somebody may still be using the directory
  			logger.Info("Add to OldEntries")
  			c.OldEntries[cacheKey+entry.ExpandedDirectoryPath] = entry
  		} else {
  			// We need to remove it from oldEntries
  			logger.Info("Delete from OldEntries")
  			delete(c.OldEntries, cacheKey+entry.ExpandedDirectoryPath)
  		}
  	}
  }
  

• Logs output

  {"timestamp":"2021-01-09T10:24:42.328966708Z","level":"info","source":"rep","message":"rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-create.node-create.downloader.file-cache.add-directory.file-cache.updateOldEntries.Delete from OldEntries","data":{"ExpandedDirectoryPath":"/var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187801981468436-136.d","cache_key":"08e216965c1b52a0935f894c304a3efc","container-guid":"88cf1ae6-fa12-460d-7ceb-a0d0","container-state":"reserved","guid":"88cf1ae6-fa12-460d-7ceb-a0d0","in_use":true,"lrp-instance-key":{"instance_guid":"88cf1ae6-fa12-460d-7ceb-a0d0","cell_id":"shard02-diego-cell-z1-0"},"lrp-key":{"process_guid":"91530181-0084-4c75-8612-0a19c5c0d4ad-77908893-af25-4e79-aa46-ca2c67cc371b","index":0,"domain":"cf-apps"},"session":"978.1.1.3.2.1.1.5.1","size":103936000,"source_path":"/var/vcap/data/rep/shared/garden/download_cache/temp/transformed685275020"}}
  {"timestamp":"2021-01-09T10:24:56.265447236Z","level":"info","source":"rep","message":"rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-create.node-create.downloader.file-cache.add-directory.file-cache.updateOldEntries.Add to OldEntries","data":{"ExpandedDirectoryPath":"/var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187882328851372-140.d","cache_key":"08e216965c1b52a0935f894c304a3efc","container-guid":"bed54682-f1da-46ff-7dd0-6cb2","container-state":"reserved","guid":"bed54682-f1da-46ff-7dd0-6cb2","in_use":false,"lrp-instance-key":{"instance_guid":"bed54682-f1da-46ff-7dd0-6cb2","cell_id":"shard02-diego-cell-z1-0"},"lrp-key":{"process_guid":"81553e5b-ca56-4f6d-9e02-f3a47e04e4f5-ad6bb947-d504-411e-9b01-a1e1ded943a4","index":0,"domain":"cf-apps"},"session":"994.1.1.3.2.1.1.5.1","size":103936000,"source_path":"/var/vcap/data/rep/shared/garden/download_cache/temp/transformed006108032"}}
  {"timestamp":"2021-01-09T10:24:58.546140196Z","level":"info","source":"rep","message":"rep.executing-container-operation.task-processor.run-container.containerstore-create.node-create.downloader.file-cache.add-directory.file-cache.updateOldEntries.Add to OldEntries","data":{"ExpandedDirectoryPath":"/var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187896265337184-142.d","cache_key":"08e216965c1b52a0935f894c304a3efc","container-guid":"cf31371b-6706-4449-a65b-f90ecd0eb411","container-state":"reserved","guid":"cf31371b-6706-4449-a65b-f90ecd0eb411","in_use":false,"session":"999.1.3.2.1.2.5.1","size":103936000,"source_path":"/var/vcap/data/rep/shared/garden/download_cache/temp/transformed451402930"}}
  

• Verify the directories refered in logs:
  • The entries been adding to OldEntries, the folders are not exist.
  • The entry been removed from OldEntries, the folder still exists and turns into dangling directories, cause cachedownloader will not never clean it.

  /:/var/vcap/data/rep/shared/garden/download_cache# ls /var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187801981468436-136.d
  builder  builder.exe  diego-sshd  diego-sshd.exe  getenv.exe  healthcheck  healthcheck.exe  launcher  launcher.exe  shell  winpty-agent.exe  winpty.dll
  /:/var/vcap/data/rep/shared/garden/download_cache# ls /var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187882328851372-140.d
  ls: cannot access '/var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187882328851372-140.d': No such file or directory
  /:/var/vcap/data/rep/shared/garden/download_cache# ls /var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187896265337184-142.d
  ls: cannot access '/var/vcap/data/rep/shared/garden/download_cache/08e216965c1b52a0935f894c304a3efc-1610187896265337184-142.d': No such file or directory
  /:/var/vcap/data/rep/shared/garden/download_cache#
  

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[aminjam]

Thanks @yangwenyanibm for reporting this issue. I see you have linked a PR with this issue as well. I am not sure If I understand the failing case. Can you please revise your PR with a test case that would be failing in the current version (mimicking this behavior) and passing after applying the PR?

[aminjam]

@yangwenyanibm Have you had chance to add a failing test so that we can better understand the problem you were seeing?

[ameowlia]

Hi all,

I am closing this issue due to inactivity. If you are still having this issue please provide a failing test case so we can better understand the issue you are trying to fix.

Thank you!