API authentication failure due to Unclassified Authentication Failure

[prithvi66]

Description:
WSO2 IOTS 3.3.0 has an intermittent issue when the server crashes and stops working every 2 weeks. The issues starts with a Warning about connection timeout which is continuously thrown followed by API authentication failure due to Unclassified Authentication Failure. The full Stack trace is given below

TID: [-1] [] [2022-09-29 04:40:10,839] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5215340 Socket Timeout : 60000 Remote Address : /31.4.180.20:19683 {org.apache.synapse.transport.passthru.SourceHandler} TID: [-1] [] [2022-09-29 04:40:11,046] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5215337 Socket Timeout : 60000 Remote Address : /178.139.224.138:17876 {org.apache.synapse.transport.passthru.SourceHandler} TID: [-1] [] [2022-09-29 04:40:11,046] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5215341 Socket Timeout : 60000 Remote Address : /31.4.208.55:65353 {org.apache.synapse.transport.passthru.SourceHandler} TID: [-1] [] [2022-09-29 04:40:11,076] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5214642 Socket Timeout : 60000 Remote Address : /31.4.218.56:9797 {org.apache.synapse.transport.passthru.SourceHandler} TID: [-1] [] [2022-09-29 04:40:11,076] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5215335 Socket Timeout : 60000 Remote Address : /77.208.168.108:18687 {org.apache.synapse.transport.passthru.SourceHandler} TID: [-1234] [] [2022-09-29 04:40:11,914] ERROR {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - API authentication failure due to Unclassified Authentication Failure {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Error while accessing backend services for API key validation at org.wso2.carbon.apimgt.gateway.handlers.security.keys.WSAPIKeyDataStore.getAllURITemplates(WSAPIKeyDataStore.java:77) at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getAllURITemplates(APIKeyValidator.java:791) at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.doGetAPIInfo(APIKeyValidator.java:639) at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.findMatchingVerb(APIKeyValidator.java:573) at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getResourceAuthenticationScheme(APIKeyValidator.java:357) at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:127) at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate(APIAuthenticationHandler.java:210) at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:158) at org.apache.synapse.rest.API.process(API.java:325) at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:90) at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:69) at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:303) at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:92) at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:337) at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:383) at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:151) at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Error while accessing backend services for API key validation at org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient.getAllURITemplates(APIKeyValidatorClient.java:190) at org.wso2.carbon.apimgt.gateway.handlers.security.keys.WSAPIKeyDataStore.getAllURITemplates(WSAPIKeyDataStore.java:75) ... 20 more Caused by: org.apache.axis2.AxisFault: connect timed out at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203) at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:81) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:441) at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:227) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) at org.wso2.carbon.apimgt.keymgt.stub.validator.APIKeyValidationServiceStub.getAllURITemplates(APIKeyValidationServiceStub.java:237) at org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient.getAllURITemplates(APIKeyValidatorClient.java:174) ... 21 more Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Suggested Labels:
Connection time out after request is read: http-incoming-5215335 Socket Timeout : 60000 Remote Address
API authentication failure due to Unclassified Authentication Failure
Error while accessing backend services for API key validation

Suggested Assignees:
WSO2

Affected Product Version:
WSO2 IoTS 3.3.0, carbon-apimgt v6.2.201

OS, DB, other environment details and versions:
Redhat OS, MySQL 5.7, Java 1.8.0_144

Steps to reproduce:
Intermittent Issue that happens about once in 2 weeks

Related Issues:
Similar Issue is observed in product-apim
https://github.com/wso2/product-apim/issues?q=is%3Aissue+is%3Aopen+Unclassified+Authentication+Failure

Any help on this would be really appreciated