Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] ImagePullBackOff and JAVA_HOME Not Defined in WSO2 APK Deployments #2653

Open
mudasirhaji opened this issue Nov 13, 2024 · 0 comments
Open

[Bug] ImagePullBackOff and JAVA_HOME Not Defined in WSO2 APK Deployments #2653

mudasirhaji opened this issue Nov 13, 2024 · 0 comments
Labels
Type/Bug

Comments

@mudasirhaji
Copy link

Description

Issue Summary
Installed WSO2 API Manager using Helm following the official documentation, but certain deployments are encountering errors. Notably, two pods (apk-wso2-apk-config-ds-deployment and apk-wso2-apk-gateway-runtime-deployment) are failing to initialize properly.

Description
We are facing issues with specific pods in the wso2 namespace related to image pulling errors and JAVA_HOME not being set correctly in the configuration. This is blocking the deployment and impacting the initialization of several components within the WSO2 APK environment.

Deployment Details
Namespace: wso2
Environment: Kubernetes
Kubernetes Version: 1.21

Observed Errors
ImagePullBackOff Errors: Multiple deployments are in ImagePullBackOff status due to image pulling issues. Specifically:

Pods such as apk-wso2-apk-adapter-deployment, apk-wso2-apk-config-ds-deployment, and apk-wso2-apk-gateway-runtime-deployment are failing due to image pull errors.
JAVA_HOME Error:

In apk-wso2-apk-config-ds-deployment pod, we encounter a JAVA_HOME error preventing Java from starting:

Error: JAVA_HOME is not defined correctly.
Admin cannot execute /opt/java/openjdk/bin/java

GRPCRoute Version Error:

In logs for some deployment pods:

ERRO [api_controller.go:150] - [dp.NewAPIController] [-] Error adding indexes: no matches for kind "GRPCRoute" in version "gateway.networking.k8s.io/v1" [error_code=2612 severity=Blocker]

Steps Taken
Attempted to pull logs for deeper troubleshooting.
Verified JAVA_HOME settings in the deployment but could not locate explicit definitions.
Steps to Reproduce
Deploy the WSO2 APK components in the wso2 namespace.
Check the status of deployments using kubectl get all -n wso2.

kubectl get all -n wso2
NAME READY STATUS RESTARTS AGE
pod/apk-cert-manager-95c6fcbbd-d4gpq 1/1 Running 0 2d11h
pod/apk-cert-manager-cainjector-6c5f449445-27kn6 1/1 Running 0 2d11h
pod/apk-cert-manager-webhook-867bf4c77b-kv4tt 1/1 Running 0 2d11h
pod/apk-wso2-apk-adapter-deployment-77bbfc94f5-gnzm8 0/1 ImagePullBackOff 72 44h
pod/apk-wso2-apk-common-controller-deployment-5bc97d9c4f-n5hzg 1/1 Running 0 2d11h
pod/apk-wso2-apk-config-ds-deployment-79dfbb578-cwlsv 0/1 ImagePullBackOff 105 2d11h
pod/apk-wso2-apk-gateway-runtime-deployment-94f95b9c-75qpb 0/2 ImagePullBackOff 226 2d11h
pod/apk-wso2-apk-idpds-deployment-585fdfd6f4-r72hv 0/1 Init:0/1 0 2d11h
pod/apk-wso2-apk-idpui-deployment-7bff7f5cc7-scv6j 1/1 Running 0 2d11h
pod/apk-wso2-apk-ratelimiter-deployment-5cfd655f69-fc8n8 0/1 ImagePullBackOff 67 2d11h
pod/gateway-api-admission-lq5jh 0/1 Completed 0 2d11h
pod/gateway-api-admission-patch-qj4bf 0/1 Completed 0 2d11h
pod/gateway-api-admission-server-7fd9cbfb6d-dgd7d 1/1 Running 0 2d11h
pod/redis-master-0 0/1 CrashLoopBackOff 701 2d11h
pod/wso2apk-db-service-0 0/1 Error 698 2d11h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/apk-cert-manager ClusterIP 10.101.23.208 9402/TCP 2d11h
service/apk-cert-manager-cainjector ClusterIP 10.109.146.151 9402/TCP 2d11h
service/apk-cert-manager-webhook ClusterIP 10.111.119.78 443/TCP,9402/TCP 2d11h
service/apk-wso2-apk-adapter-service ClusterIP 10.106.66.135 8765/TCP,18000/TCP,443/TCP,18005/TCP,6060/TCP 2d11h
service/apk-wso2-apk-common-controller-service ClusterIP 10.109.150.185 18005/TCP,18002/TCP,443/TCP,18003/TCP,9543/TCP 2d11h
service/apk-wso2-apk-config-ds-service ClusterIP 10.102.27.163 9443/TCP 2d11h
service/apk-wso2-apk-enforcer-service ClusterIP 10.99.128.33 9092/TCP 2d11h
service/apk-wso2-apk-gateway-service LoadBalancer 10.101.67.169 10.70.107.206 9095:32372/TCP 2d11h
service/apk-wso2-apk-idp-ds-service ClusterIP 10.103.147.110 9443/TCP 2d11h
service/apk-wso2-apk-idp-ui-service ClusterIP 10.99.53.217 9443/TCP 2d11h
service/apk-wso2-apk-ratelimiter-service ClusterIP 10.105.7.37 8090/TCP,8091/TCP,6070/TCP 2d11h
service/gateway-api-admission-server ClusterIP 10.107.194.29 443/TCP 2d11h
service/redis-headless ClusterIP None 6379/TCP 2d11h
service/redis-master ClusterIP 10.110.42.252 6379/TCP 2d11h
service/wso2apk-db-service ClusterIP 10.106.37.131 5432/TCP 2d11h
service/wso2apk-db-service-hl ClusterIP None 5432/TCP 2d11h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/apk-cert-manager 1/1 1 1 2d11h
deployment.apps/apk-cert-manager-cainjector 1/1 1 1 2d11h
deployment.apps/apk-cert-manager-webhook 1/1 1 1 2d11h
deployment.apps/apk-wso2-apk-adapter-deployment 0/1 1 0 2d11h
deployment.apps/apk-wso2-apk-common-controller-deployment 1/1 1 1 2d11h
deployment.apps/apk-wso2-apk-config-ds-deployment 0/1 1 0 2d11h
deployment.apps/apk-wso2-apk-gateway-runtime-deployment 0/1 1 0 2d11h
deployment.apps/apk-wso2-apk-idpds-deployment 0/1 1 0 2d11h
deployment.apps/apk-wso2-apk-idpui-deployment 1/1 1 1 2d11h
deployment.apps/apk-wso2-apk-ratelimiter-deployment 0/1 1 0 2d11h
deployment.apps/gateway-api-admission-server 1/1 1 1 2d11h

NAME DESIRED CURRENT READY AGE
replicaset.apps/apk-cert-manager-95c6fcbbd 1 1 1 2d11h
replicaset.apps/apk-cert-manager-cainjector-6c5f449445 1 1 1 2d11h
replicaset.apps/apk-cert-manager-webhook-867bf4c77b 1 1 1 2d11h
replicaset.apps/apk-wso2-apk-adapter-deployment-77bbfc94f5 1 1 0 2d11h
replicaset.apps/apk-wso2-apk-common-controller-deployment-5bc97d9c4f 1 1 1 2d11h
replicaset.apps/apk-wso2-apk-config-ds-deployment-79dfbb578 1 1 0 2d11h
replicaset.apps/apk-wso2-apk-gateway-runtime-deployment-94f95b9c 1 1 0 2d11h
replicaset.apps/apk-wso2-apk-idpds-deployment-585fdfd6f4 1 1 0 2d11h
replicaset.apps/apk-wso2-apk-idpui-deployment-7bff7f5cc7 1 1 1 2d11h
replicaset.apps/apk-wso2-apk-ratelimiter-deployment-5cfd655f69 1 1 0 2d11h
replicaset.apps/gateway-api-admission-server-7fd9cbfb6d 1 1 1 2d11h

NAME READY AGE
statefulset.apps/redis-master 0/1 2d11h
statefulset.apps/wso2apk-db-service 0/1 2d11h

NAME COMPLETIONS DURATION AGE
job.batch/gateway-api-admission 1/1 20s 2d11h
job.batch/gateway-api-admission-patch 1/1 18s 2d11h

Expected Outcome
All pods should start successfully without ImagePullBackOff errors.
Java should initialize correctly without JAVA_HOME errors.
Actual Outcome
Several pods are stuck in ImagePullBackOff or CrashLoopBackOff.
Java initialization fails in the apk-wso2-apk-config-ds-deployment pod due to JAVA_HOME not being defined.

Requested Support
Guidance on resolving the ImagePullBackOff and Java initialization issues.
Assistance in setting JAVA_HOME in a way that the deployments recognize it correctly.
Fix for the GRPCRoute error message.

Steps to Reproduce

Steps to Reproduce
Deploy the WSO2 APK components in the wso2 namespace.
Check the status of deployments using kubectl get all -n wso2.

Affected Component

Adapter

Version

Kubernetes version 1.21

Environment Details (with versions)

No response

Relevant Log Output

kubectl logs apk-wso2-apk-adapter-deployment-77bbfc94f5-gnzm8 -n wso2
2024-11-13 05:47:03 INFO [logger.go:42] - [config.UpdateLoggers] [-] Updated loggers []
2024-11-13 05:47:03 INFO [logger.go:83] - [loggers.UpdateLoggers] [-] Updated loggers []
2024-11-13 05:47:03 INFO [logger.go:65] - [loggers.UpdateLoggers] [-] Updated loggers []
2024-11-13 05:47:03 INFO [logger.go:42] - [configs.UpdateLoggers] [-] Updated loggers []
2024-11-13 05:47:03 INFO [logger.go:83] - [loggers.UpdateLoggers] [-] Updated loggers []
2024-11-13 05:47:03 INFO [eventPublisher.go:178] - [controlplane.init.0.func1] [-] Adapter control plane is not enabled. Not starting agent worker. []
2024-11-13 05:47:04 INFO [main.go:28] - [main.main] [-] Starting the Adapter []
2024-11-13 05:47:04 INFO [adapter.go:229] - [adapter.Run] [-] Starting adapter .... []
2024-11-13 05:47:04 INFO [adapter.go:132] - [adapter.runManagementServer] [-] port: 18000 management server listening []
2024-11-13 05:47:04 INFO [server.go:450] - [xds.UpdateEnforcerConfig] [-] New Config cache update for the label: commonEnforcerLabel version: 931576438 []
2024-11-13 05:47:04 INFO [adapter.go:134] - [adapter.runManagementServer.func1] [-] Starting XDS GRPC server. []
2024-11-13 05:47:04 INFO [gateway_controller.go:152] - [dp.NewGatewayController] [-] Gateway Controller successfully started. Watching Gateway Objects.... []
2024-11-13 05:47:04 ERRO [api_controller.go:150] - [dp.NewAPIController] [-] Error adding indexes: no matches for kind "GRPCRoute" in version "gateway.networking.k8s.io/v1" [error_code=2612 severity=Blocker]

#########################################

kubectl logs apk-wso2-apk-config-ds-deployment-79dfbb578-cwlsv -n wso2
Error: JAVA_HOME is not defined correctly.
 Admin cannot execute /opt/java/openjdk/bin/java

Related Issues

No response

Suggested Labels

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type/Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mudasirhaji