error in adduserecord with kinesalite and kpl

[vikrantch-hk]

Hi Team,

I am using kinesalite with KPL and getting below error message

com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2018-04-04 18:11:54.990304] [0x00000694] [error]   [shard_map.cc:152] Shard map update for stream "sample" failed. Code:  Message: Unable to connect to endpoint;

I am using below configuration

amazon-kinesis-client 1.8.8
amazon-kinesis-producer 0.12.8

System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");
		System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY, "true");
	    System.setProperty("USE_SSL", "true");
		
		KinesisProducerConfiguration config = new KinesisProducerConfiguration();
        config.setRegion(Region.getRegion(Regions.US_EAST_1).getName());
        config.setCredentialsProvider(new EnvironmentVariableCredentialsProvider());
        config.setKinesisEndpoint("192.168.70.107");
        config.setKinesisPort(4567);
        config.setLogLevel("warning");
        config.setVerifyCertificate(false);
        producer = new com.amazonaws.services.kinesis.producer.KinesisProducer(config);

I am able to put record using kinesis rest api. Please let me know what configuration I am missing.

[mhart]

Please try with v1.13.0 and see if this fixes your problem

[oriy]

Encountered the same issue
KPL calls ListShards api before connecting the stream
v1.13.0 includes listShards and should work properly

[vikrantch-hk]

@mhart @oriy sorry I am not at node. I am using npm install -g kinesalite and I guess it's installing v1.13.0 only however I am seeing below logs during install

gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/release/.nvm/versions/node/v5.1.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:270:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Linux 2.6.32-431.el6.x86_64
gyp ERR! command "/home/release/.nvm/versions/node/v5.1.0/bin/node" "/home/release/.nvm/versions/node/v5.1.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/release/.nvm/versions/node/v5.1.0/lib/node_modules/kinesalite/node_modules/leveldown
gyp ERR! node -v v5.1.0
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
npm WARN install:[email protected] [email protected] install: `prebuild-install || node-gyp rebuild`
npm WARN install:[email protected] Exit status 1

is it installing some other version because of above error logs or it's v1.13.0 only

[vikrantch-hk]

i have updated node and level versions and after that there is not error logs in npm install kinesalite, but I am still facing same issue while trying to push record using kinesalite and kpl getting

com.amazonaws.services.kinesis.producer.UserRecordFailedException

and below error in logs

LogInputStreamReader - [2018-04-05 12:03:47.972517] [0x00001b64] [error]   [shard_map.cc:152] Shard map update for stream "sample" failed

[vikrantch-hk]

just to update I am able to put and get records using kpl 0.10.2 and setting ssl true in kinesalite, aren't we supporting kpl latest versions?

[vikrantch-hk]

@mhart @oriy please help

[mhart]

@vikrantch-hk I don't know much about KPL to be honest – I don't use it

[vikrantch-hk]

@mhart ok so on which version of aws-sdk kinesalite is

[mhart]

This project is intended to implement the API as documented here: https://docs.aws.amazon.com/kinesis/latest/APIReference/Welcome.html

If there's a specific API call missing (I know that UpdateShardCount is for example), that's much easier to debug/understand rather than an external system like trying to debug KPL.

[vikrantch-hk]

@mhart ok then PutRecord is returning "unable to connect endpoint"

[mhart]

Could be that you're not connecting to kinesalite correctly? Is CBOR turned off (kinesalite doesn't support CBOR). https://github.com/mhart/kinesalite#cbor-protocol-issues-with-the-java-sdk

Or is it an SSL connection problem?

[vikrantch-hk]

@mhart yes CBOR is disabled and verfiycertificate false

[vikrantch-hk]

@mhart does comment on awslabs/amazon-kinesis-producer#197 helps

[Raghav2211]

Same issue facing :( Any Update on that