Skip to content

Improve error handling on process start #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 12, 2024
Merged

Improve error handling on process start #91

merged 3 commits into from
Mar 12, 2024

Conversation

@klapkov
Copy link
Contributor

@klapkov klapkov commented Jan 18, 2024

Related to cloudfoundry/diego-release#892. With this improvement, if there is a failure while starting a process, lets say for example the env variable is too large, here is how the app logs will look now:

 2024-01-18T00:49:08.59+0000 [API/0] OUT Added process: "web"
   2024-01-18T00:49:08.59+0000 [API/0] OUT Created app with guid 043a7ccf-8838-44ab-897e-597047e87ac3
   2024-01-18T00:49:08.61+0000 [API/0] OUT Applied manifest to app with guid 043a7ccf-8838-44ab-897e-597047e87ac3 (---
   2024-01-18T00:49:08.61+0000 [API/0] OUT applications:
   2024-01-18T00:49:08.61+0000 [API/0] OUT - name: mentos-cola
   2024-01-18T00:49:08.61+0000 [API/0] OUT disk-quota: 4G
   2024-01-18T00:49:08.61+0000 [API/0] OUT memory: 500M
   2024-01-18T00:49:08.61+0000 [API/0] OUT random-route: true
   2024-01-18T00:49:08.61+0000 [API/0] OUT buildpack: https://github.com/cloudfoundry/go-buildpack.git
   2024-01-18T00:49:08.61+0000 [API/0] OUT env: "[PRIVATE DATA HIDDEN]"
   2024-01-18T00:49:08.61+0000 [API/0] OUT )
   2024-01-18T00:49:10.88+0000 [API/0] OUT Scaling process: "web"
   2024-01-18T00:49:10.95+0000 [API/0] OUT Updated app with guid 043a7ccf-8838-44ab-897e-597047e87ac3 ({"lifecycle"=>{"data"=>{"buildpacks"=>["https://github.com/cloudfoundry/go-buildpack.git"]}}})
   2024-01-18T00:49:11.08+0000 [API/0] OUT Updated app with guid 043a7ccf-8838-44ab-897e-597047e87ac3 ({"environment_variables"=>"[PRIVATE DATA HIDDEN]"})
   2024-01-18T00:49:11.97+0000 [API/0] OUT Uploading app package for app with guid 043a7ccf-8838-44ab-897e-597047e87ac3
   2024-01-18T00:49:15.17+0000 [API/0] OUT Creating build for app with guid 043a7ccf-8838-44ab-897e-597047e87ac3
   2024-01-18T00:49:15.60+0000 [STG/0] OUT Cell fcb7c7db-8a2c-4216-95d3-a7c6500c8e9c creating container for instance b9e62f25-1be6-438a-8da3-094aa5516775
   2024-01-18T00:49:16.07+0000 [STG/0] OUT Security group rules were updated
   2024-01-18T00:49:16.12+0000 [STG/0] OUT Cell fcb7c7db-8a2c-4216-95d3-a7c6500c8e9c successfully created container for instance b9e62f25-1be6-438a-8da3-094aa5516775
   2024-01-18T00:49:16.82+0000 [STG/0] OUT Downloading app package...
   2024-01-18T00:49:16.88+0000 [STG/0] OUT Downloaded app package (1.1K)
   2024-01-18T00:49:17.02+0000 [STG/0] OUT failed-creating-process: runc exec: exit status 255: exec failed: unable to start container process: exec /tmp/lifecycle/builder: argument list too long
   2024-01-18T00:49:18.37+0000 [STG/0] OUT Cell fcb7c7db-8a2c-4216-95d3-a7c6500c8e9c stopping instance b9e62f25-1be6-438a-8da3-094aa5516775
   2024-01-18T00:49:18.37+0000 [STG/0] OUT Cell fcb7c7db-8a2c-4216-95d3-a7c6500c8e9c destroying container for instance b9e62f25-1be6-438a-8da3-094aa5516775
   2024-01-18T00:49:18.43+0000 [API/0] ERR Failed to stage build: staging failed
   2024-01-18T00:49:18.88+0000 [STG/0] OUT Cell fcb7c7db-8a2c-4216-95d3-a7c6500c8e9c successfully destroyed container for instance b9e62f25-1be6-438a-8da3-094aa5516775


failed-creating-process: runc exec: exit status 255: exec failed: unable to start container process: exec /tmp/lifecycle/builder: argument list too long

@klapkov klapkov requested a review from a team as a code owner January 18, 2024 00:51
@klapkov klapkov mentioned this pull request Jan 18, 2024
Closed
PlamenDoychev
PlamenDoychev reviewed Jan 23, 2024
View reviewed changes
PlamenDoychev added a commit to PlamenDoychev/community that referenced this pull request Mar 9, 2024
@PlamenDoychev
Add klapkov as ARP Reviewer
448a2ff 
[DIEGO-RELEASE] Add locket client keepalive time and timeout to jobs: cloudfoundry/diego-release#722
[DIEGO-RELEASE] Make max_containers prop configurable: cloudfoundry/diego-release#876
[BBS] Add request metrics for BBS - still open: cloudfoundry/bbs#80
[BBS] Use scheduling info instead of the whole desiredLRP: cloudfoundry/bbs#79
[BBS] Add routing info endpoint: 
cloudfoundry/bbs#66
[BBS] Remove cpu_weight limit: 
cloudfoundry/bbs#81
[EXECUTOR] Improve error handling on process start - still open: cloudfoundry/executor#91
[LOCKET]  Add a keepalive timeout on the locket client: cloudfoundry/locket#12
[GROOTFS] UsedVolumesSize and UsedStoreInBytes metrics: cloudfoundry/grootfs#155
[ROUTE-EMITTER] Use routing info bbs endpoint when syncing: cloudfoundry/route-emitter#23
[ROUTE-EMITTER]  Use routing_info for desired_lrp's when there are missing actual_lrp's: 
cloudfoundry/route-emitter#26
[SILK-RELEASE] Deduplicate Iptables Rules with Dynamic ASG's: cloudfoundry/silk-release#101
[SILK-RELEASE] Make container_metadata_file_check_timeout on silk-shutdown configurable: 
cloudfoundry/silk-release#111
This was referenced Mar 9, 2024
Closed
Merged
ameowlia pushed a commit to cloudfoundry/community that referenced this pull request Mar 12, 2024
@PlamenDoychev @ameowlia
Add klapkov as ARP Reviewer
3b4e19d 
[DIEGO-RELEASE] Add locket client keepalive time and timeout to jobs: cloudfoundry/diego-release#722
[DIEGO-RELEASE] Make max_containers prop configurable: cloudfoundry/diego-release#876
[BBS] Add request metrics for BBS - still open: cloudfoundry/bbs#80
[BBS] Use scheduling info instead of the whole desiredLRP: cloudfoundry/bbs#79
[BBS] Add routing info endpoint:
cloudfoundry/bbs#66
[BBS] Remove cpu_weight limit:
cloudfoundry/bbs#81
[EXECUTOR] Improve error handling on process start - still open: cloudfoundry/executor#91
[LOCKET]  Add a keepalive timeout on the locket client: cloudfoundry/locket#12
[GROOTFS] UsedVolumesSize and UsedStoreInBytes metrics: cloudfoundry/grootfs#155
[ROUTE-EMITTER] Use routing info bbs endpoint when syncing: cloudfoundry/route-emitter#23
[ROUTE-EMITTER]  Use routing_info for desired_lrp's when there are missing actual_lrp's:
cloudfoundry/route-emitter#26
[SILK-RELEASE] Deduplicate Iptables Rules with Dynamic ASG's: cloudfoundry/silk-release#101
[SILK-RELEASE] Make container_metadata_file_check_timeout on silk-shutdown configurable:
cloudfoundry/silk-release#111

wip
ameowlia pushed a commit to cloudfoundry/community that referenced this pull request Mar 12, 2024
@PlamenDoychev @ameowlia
Add klapkov as ARP Reviewer
18591a5 
[DIEGO-RELEASE] Add locket client keepalive time and timeout to jobs: cloudfoundry/diego-release#722
[DIEGO-RELEASE] Make max_containers prop configurable: cloudfoundry/diego-release#876
[BBS] Add request metrics for BBS - still open: cloudfoundry/bbs#80
[BBS] Use scheduling info instead of the whole desiredLRP: cloudfoundry/bbs#79
[BBS] Add routing info endpoint:
cloudfoundry/bbs#66
[BBS] Remove cpu_weight limit:
cloudfoundry/bbs#81
[EXECUTOR] Improve error handling on process start - still open: cloudfoundry/executor#91
[LOCKET]  Add a keepalive timeout on the locket client: cloudfoundry/locket#12
[GROOTFS] UsedVolumesSize and UsedStoreInBytes metrics: cloudfoundry/grootfs#155
[ROUTE-EMITTER] Use routing info bbs endpoint when syncing: cloudfoundry/route-emitter#23
[ROUTE-EMITTER]  Use routing_info for desired_lrp's when there are missing actual_lrp's:
cloudfoundry/route-emitter#26
[SILK-RELEASE] Deduplicate Iptables Rules with Dynamic ASG's: cloudfoundry/silk-release#101
[SILK-RELEASE] Make container_metadata_file_check_timeout on silk-shutdown configurable:
cloudfoundry/silk-release#111

wip
@winkingturtle-vmw winkingturtle-vmw merged commit db9758c into cloudfoundry:main Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PlamenDoychev PlamenDoychev PlamenDoychev left review comments

@winkingturtle-vmw winkingturtle-vmw winkingturtle-vmw approved these changes

Assignees

No one assigned

Labels

None yet

Projects

Application Runtime Platform Working Group
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@klapkov @PlamenDoychev @winkingturtle-vmw