DynamoDB DependencyException #153
Labels
Comments
|
How often are you getting this error? |
|
It happened March 8th, 9th, 14th and 25th. It is not a daily issue, but every time it happens it causes an Error alarm. |
|
Hi, it has happened again and caused an alarm. Any update? |
|
Is this resolved?if yes, How? |
|
Hi @pfifer I see the same issue with my eventconsumer while renewing a lease. @timestamp: 2017-05-31T18:02:34.541Z
@version: 1
_id: AVxfrG9jiydF8RCgVyWX
_index: customerorder-prod-eventconsumer-2017.05.31
_type: log
beat: {
"hostname": "ip-172-28-14-145.us-west-2.compute.internal",
"name": "ip-172-28-14-145.us-west-2.compute.internal",
"version": "5.3.0"
}
class: com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable
exception: {
"exception_class": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException",
"exception_message": "Encountered an exception while renewing leases. The number of leases which might not have been renewed is 1",
"stacktrace": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException: Encountered an exception while renewing leases. The number of leases which might not have been renewed is 1\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:130)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator.runRenewer(LeaseCoordinator.java:255)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable.run(LeaseCoordinator.java:179)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.util.concurrent.ExecutionException: com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: N0H4CKHA8KBSD3PLPGPQC7SEMFVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:192)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:111)\n\t... 9 more\nCaused by: com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: N0H4CKHA8KBSD3PLPGPQC7SEMFVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:577)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:391)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:176)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:159)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.access$000(LeaseRenewer.java:49)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:151)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:137)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\t... 3 more\nCaused by: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: N0H4CKHA8KBSD3PLPGPQC7SEMFVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1579)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1249)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1835)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1811)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1661)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:382)\n\t... 9 more"
}
file: LeaseCoordinator.java
input_type: log
level: ERROR
line_number: 181
logger_name: com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator
mdc: {}
message: LeasingException encountered in lease renewing thread
method: run
num_hits: 2
num_matches: 1
offset: 595138
related_events: [
{
"@timestamp": "2017-05-31T18:02:34.541000+00:00",
"@version": 1,
"_id": "AVxfrG9jiydF8RCgVyWW",
"_index": "customerorder-prod-eventconsumer-2017.05.31",
"_type": "log",
"beat": {
"hostname": "ip-172-28-14-145.us-west-2.compute.internal",
"name": "ip-172-28-14-145.us-west-2.compute.internal",
"version": "5.3.0"
},
"class": "com.amazonaws.services.kinesis.leases.impl.LeaseRenewer",
"exception": {
"exception_class": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException",
"exception_message": "com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: N0H4CKHA8KBSD3PLPGPQC7SEMFVV4KQNSO5AEMVJF66Q9ASUAAJG)",
"stacktrace": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: N0H4CKHA8KBSD3PLPGPQC7SEMFVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:577)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:391)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:176)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:159)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.access$000(LeaseRenewer.java:49)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:151)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:137)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: N0H4CKHA8KBSD3PLPGPQC7SEMFVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1579)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1249)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1835)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1811)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1661)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:382)\n\t... 9 more"
},
"file": "LeaseRenewer.java",
"input_type": "log",
"level": "ERROR",
"line_number": "119",
"logger_name": "com.amazonaws.services.kinesis.leases.impl.LeaseRenewer",
"mdc": {},
"message": "Encountered an exception while renewing a lease.",
"method": "renewLeases",
"offset": 590078,
"source": "/var/log/customerorder/customerorder-consumer.log",
"source_host": "ip-172-28-14-145.us-west-2.compute.internal",
"thread_name": "LeaseCoordinator-2",
"type": "log"
}
]
source: /var/log/customerorder/customerorder-consumer.log
source_host: ip-172-28-14-145.us-west-2.compute.internal
thread_name: LeaseCoordinator-2
type: log
ExecutionException is thrown at line 199 in following code 109 for (Future<Boolean> renewLeaseTask : renewLeaseTasks) {
110 try {
111 if (!renewLeaseTask.get()) {
112 lostLeases++;
113 }
114 } catch (InterruptedException e) {
115 LOG.info("Interrupted while waiting for a lease to renew.");
116 leasesInUnknownState += 1;
117 Thread.currentThread().interrupt();
118 } catch (ExecutionException e) {
119 LOG.error("Encountered an exception while renewing a lease.", e.getCause());
120 leasesInUnknownState += 1;
121 lastException = e;
122 }
123 } |
|
This is indicating that sending the request to DynamoDB failed. I would need to determine if this is before or after retry, and the KCL retries. If there are retries it should be possible to drop the error message to warn, unless it's happening repeatedly. |
|
@pfifer I did not understand when you say retry? I don't have trace level logs but following is sequence of logs in INFO level {"@timestamp":"2017-06-13T21:20:59.969Z","source_host":"ip-172-21-63-243.us-west-2.compute.internal","file":"Worker.java","method":"info","level":"INFO","line_number":"717","thread_name":"Thread-8","@version":1,"logger_name":"com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker","message":"Sleeping ...","class":"com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker$WorkerLog","mdc":{}}
{
"exception": {
"stacktrace": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 4P7OSJM7II68L0LSMH17QJQIHNVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:577)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:391)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:176)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:159)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.access$000(LeaseRenewer.java:49)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:151)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:137)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 4P7OSJM7II68L0LSMH17QJQIHNVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2089)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2065)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeUpdateItem(AmazonDynamoDBClient.java:1811)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1787)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:382)\n\t... 9 more",
"exception_class": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException",
"exception_message": "com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 4P7OSJM7II68L0LSMH17QJQIHNVV4KQNSO5AEMVJF66Q9ASUAAJG)"
},
"source_host": "ip-172-21-63-243.us-west-2.compute.internal",
"method": "renewLeases",
"level": "ERROR",
"message": "Encountered an exception while renewing a lease.",
"mdc": {},
"@timestamp": "2017-06-13T21:21:40.829Z",
"file": "LeaseRenewer.java",
"line_number": "119",
"thread_name": "LeaseCoordinator-2",
"@version": 1,
"logger_name": "com.amazonaws.services.kinesis.leases.impl.LeaseRenewer",
"class": "com.amazonaws.services.kinesis.leases.impl.LeaseRenewer"
}
{
"exception": {
"stacktrace": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException: Encountered an exception while renewing leases. The number of leases which might not have been renewed is 1\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:130)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator.runRenewer(LeaseCoordinator.java:255)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable.run(LeaseCoordinator.java:179)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.util.concurrent.ExecutionException: com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 4P7OSJM7II68L0LSMH17QJQIHNVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:192)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:111)\n\t... 9 more\nCaused by: com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 4P7OSJM7II68L0LSMH17QJQIHNVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:577)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:391)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:176)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:159)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.access$000(LeaseRenewer.java:49)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:151)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseRenewer$RenewLeaseTask.call(LeaseRenewer.java:137)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\t... 3 more\nCaused by: com.amazonaws.services.dynamodbv2.model.InternalServerErrorException: Internal server error (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalServerError; Request ID: 4P7OSJM7II68L0LSMH17QJQIHNVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2089)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2065)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeUpdateItem(AmazonDynamoDBClient.java:1811)\n\tat com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1787)\n\tat com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:382)\n\t... 9 more",
"exception_class": "com.amazonaws.services.kinesis.leases.exceptions.DependencyException",
"exception_message": "Encountered an exception while renewing leases. The number of leases which might not have been renewed is 1"
},
"source_host": "ip-172-21-63-243.us-west-2.compute.internal",
"method": "run",
"level": "ERROR",
"message": "LeasingException encountered in lease renewing thread",
"mdc": {},
"@timestamp": "2017-06-13T21:21:40.830Z",
"file": "LeaseCoordinator.java",
"line_number": "181",
"thread_name": "LeaseCoordinator-2",
"@version": 1,
"logger_name": "com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator",
"class": "com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable"
}
{
"@timestamp": "2017-06-13T21:22:01.034Z",
"source_host": "ip-172-21-63-243.us-west-2.compute.internal",
"file": "Worker.java",
"method": "info",
"level": "INFO",
"line_number": "717",
"thread_name": "Thread-8",
"@version": 1,
"logger_name": "com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker",
"message": "Current stream shard assignments: shardId-000000000000",
"class": "com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker$WorkerLog",
"mdc": {}
}
{
"@timestamp": "2017-06-13T21:22:01.034Z",
"source_host": "ip-172-21-63-243.us-west-2.compute.internal",
"file": "Worker.java",
"method": "info",
"level": "INFO",
"line_number": "717",
"thread_name": "Thread-8",
"@version": 1,
"logger_name": "com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker",
"message": "Sleeping ...",
"class": "com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker$WorkerLog",
"mdc": {}
} |
|
Any updates on this? |
|
This is happening a LOT for me now... |
|
Hey @pfifer, any updates on this? |
|
There isn't much this is indicating something went wrong talking to DynamoDB. If it's happening a lot inspect the underlying cause. If it's an internal service error we aren't able to do much, and in that case the best option is to contact DynamoDB and have them investigate the table. You can also investigate the metrics for your table to ensure that calls aren't getting throttled. DynamoDB also has a metric for system errors that I would check. This exception occurs it indicates that the KCL was unable to renew the lease due to an issue calling UpdateItem on the application's lease table. When this occurs the KCL will assume its lost the lease and will shutdown the record processor with |
|
Any updates on how to try/catch this error or other way to suppress this? we are still getting Error logs for this issue.... |
|
I'm having the same problem, any solution/workaround to this? Silencing all the errors from that class / package is not a solution as we need to be alerted in case of DynamoDB issues. I just observed 4x within the same second: Instantly followed by: In the Cloudwatch metrics of the affected DynamoDB has been 19 SystemErrors on UpdateItem method. |
|
Also experiencing this issue under (relatively) heavy load. |
@pfifer is it safe to ignore it? You have added a label as an enhancement. Any update on that? It seems an old issue that no one could manage to fix it and still noise... Thank you! |
|
Hello team, I'm experiencing the same issue as described above. In our logs we have: followed by: After a couple of such exceptions it seems that no records were processed from some shards. After a restart of the workers, we saw a spike in the max iterator age ms. The consumers started to process the records from the shards and things came back to normal. We've also found log messages for throttling the producers for these shards: which I believe is expected as no records were processed for quite some time from them. The versions of the libraries that we're using are: |
We highly recommend you to immediately upgrade to amazon-kinesis-client:1.14.1 due to known issues in 1.14.0 - #778 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi, I am getting a dependency error from DynamoDB and as it is logged as ERROR and is thrown from KCL lease manager and I am unable to catch it separately to ignore. It seems like a transient issue from DynamoDB. Any idea how to handle / ignore this error? Thank you so much.
The text was updated successfully, but these errors were encountered: