Bug 1894918: Panic output due to timeouts in kube-apiserver

[p0lyn0mial]

Backports 96217, 95725, 95002 and 96424

Default timeout (10 s) for the TokenReview client (96217)
Default timeout (10 s0) for the SubjectAccessReview client (95725)
No stack trace of aborted requests in the logs (95002 and 96424)

Note: moving 95002, 96424 wasn't clean and needs review.
Note: UPSTREAM: 95002: klog v1 prints to stderr is a new commit

[openshift-ci-robot]

@p0lyn0mial: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1894918: Panic output due to timeouts in kube-apiserver

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[p0lyn0mial]

/assign @sttts

[openshift-ci-robot]

@p0lyn0mial: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1894918: Panic output due to timeouts in kube-apiserver

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[p0lyn0mial]

/retest

[p0lyn0mial]

/retest

[sttts]

/lgtm
/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: p0lyn0mial, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is invalid:

• expected dependent Bugzilla bug 1894916 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: An error was encountered adding this pull request to the external tracker bugs for bug 1894918 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.

JSONRPC error 100500: DBD::Pg::db do failed: ERROR: deadlock detected DETAIL: Process 104684 waits for ShareLock on transaction 2764664229; blocked by process 104869. Process 104869 waits for ExclusiveLock on tuple (94510,5) of relation 527931 of database 16384; blocked by process 104684. HINT: See server log for query details. [for Statement "UPDATE bugs SET delta_ts = ? WHERE bug_id = ?"]

 at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1516.

at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1516.
at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1516.
Bugzilla::Bug::update('Bugzilla::Bug=HASH(0x55958aa43730)') called at /loader/0x55957a7d51d8/Bugzilla/Extension/ExternalBugs/WebService.pm line 88
Bugzilla::Extension::ExternalBugs::WebService::add_external_bug('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x559589ac8db0)') called at (eval 3281) line 1
eval ' $procedure->{code}->($self, @params)
;' called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 220
JSON::RPC::Legacy::Server::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55958aa1a5e8)') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 297
Bugzilla::WebService::Server::JSONRPC::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55958aa1a5e8)') called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 126
JSON::RPC::Legacy::Server::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 70
Bugzilla::WebService::Server::JSONRPC::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/jsonrpc.cgi line 31
ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_jsonrpc_2ecgi::handler('Apache2::RequestRec=SCALAR(0x559589828160)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207
eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207
ModPerl::RegistryCooker::run('Bugzilla::ModPerl::ResponseHandler=HASH(0x55958c6c2710)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 173
ModPerl::RegistryCooker::default_handler('Bugzilla::ModPerl::ResponseHandler=HASH(0x55958c6c2710)') called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 32
ModPerl::Registry::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x559589828160)') called at /var/www/html/bugzilla/mod_perl.pl line 139
Bugzilla::ModPerl::ResponseHandler::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x559589828160)') called at (eval 3281) line 0
eval {...} called at (eval 3281) line 0

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: An error was encountered adding this pull request to the external tracker bugs for bug 1894918 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.

JSONRPC error 100500: DBD::Pg::db do failed: ERROR: deadlock detected DETAIL: Process 36773 waits for ShareLock on transaction 2768180572; blocked by process 36473. Process 36473 waits for ExclusiveLock on tuple (94510,7) of relation 527931 of database 16384; blocked by process 36773. HINT: See server log for query details. [for Statement "UPDATE bugs SET delta_ts = ? WHERE bug_id = ?"]

 at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1516.

at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1516.
at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1516.
Bugzilla::Bug::update('Bugzilla::Bug=HASH(0x55f44260a2c8)') called at /loader/0x55f433307660/Bugzilla/Extension/ExternalBugs/WebService.pm line 88
Bugzilla::Extension::ExternalBugs::WebService::add_external_bug('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55f443142730)') called at (eval 2573) line 1
eval ' $procedure->{code}->($self, @params)
;' called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 220
JSON::RPC::Legacy::Server::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55f43e2b3a50)') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 297
Bugzilla::WebService::Server::JSONRPC::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55f43e2b3a50)') called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 126
JSON::RPC::Legacy::Server::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 70
Bugzilla::WebService::Server::JSONRPC::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/jsonrpc.cgi line 31
ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_jsonrpc_2ecgi::handler('Apache2::RequestRec=SCALAR(0x55f43de10da8)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207
eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207
ModPerl::RegistryCooker::run('Bugzilla::ModPerl::ResponseHandler=HASH(0x55f44330c9d8)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 173
ModPerl::RegistryCooker::default_handler('Bugzilla::ModPerl::ResponseHandler=HASH(0x55f44330c9d8)') called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 32
ModPerl::Registry::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x55f43de10da8)') called at /var/www/html/bugzilla/mod_perl.pl line 139
Bugzilla::ModPerl::ResponseHandler::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x55f43de10da8)') called at (eval 2573) line 0
eval {...} called at (eval 2573) line 0

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

[openshift-ci-robot]

@openshift-bot: This pull request references Bugzilla bug 1894918, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.5.z) matches configured target release for branch (4.5.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
• dependent bug Bugzilla bug 1894916 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
• dependent Bugzilla bug 1894916 targets the "4.6.z" release, which is one of the valid target releases: 4.6.0, 4.6.z
• bug has dependents

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci-robot]

@p0lyn0mial: Some pull requests linked via external trackers have merged:

• openshift/oauth-server#65

The following pull requests linked via external trackers have not merged:

• openshift/openshift-apiserver#163 is open

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state. Once unlinked, request a bug refresh with /bugzilla refresh.

Bugzilla bug 1894918 has not been moved to the MODIFIED state.

Details

In response to this:

Bug 1894918: Panic output due to timeouts in kube-apiserver

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.