Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API-543 Add bigint serializer #975

Merged
merged 51 commits into from
Aug 4, 2021
Merged
Show file tree
Hide file tree
Changes from 50 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
254bf91
Add datetime classes back, add default serializers, change year in co…
srknzl Jun 30, 2021
4983154
Update tests and serialization
srknzl Jun 30, 2021
9a99dd1
Update binary compat test
srknzl Jun 30, 2021
532c0f6
Add date/time to default serializers test
srknzl Jun 30, 2021
13c2629
Fix tests
srknzl Jun 30, 2021
65c9267
Adjust tests according to back compat
srknzl Jul 5, 2021
a9b9aac
Rename sql service getter to getSql
srknzl Jul 5, 2021
5345542
Adjust timezone string
srknzl Jul 5, 2021
dc9f3f8
Add jet tests
srknzl Jul 5, 2021
5098d82
Add bigdecimal serializer and use it
srknzl Jul 6, 2021
99a37c7
Add portable methods boilerplate
srknzl Jul 6, 2021
6841a85
Update default serializers test
srknzl Jul 8, 2021
67516ed
Adjust bigdecimal
srknzl Jul 8, 2021
33e3137
Adjust apidoc
srknzl Jul 8, 2021
b286473
Move props out of constr to have a apidoc for them with private const…
srknzl Jul 8, 2021
a6c3f4d
typo fix
srknzl Jul 8, 2021
f577754
Update apidoc
srknzl Jul 8, 2021
9fb1104
Add some tests
srknzl Jul 8, 2021
187641f
Optimize bigdecimal serializer
srknzl Jul 8, 2021
fe07136
Some changes
srknzl Jul 8, 2021
7f05d2b
Add portable methods boilerplate
srknzl Jul 6, 2021
61eab57
revert whitespace changes
srknzl Jul 8, 2021
6c98945
Merge branch 'add-portable-methods' of github.com:srknzl/hazelcast-no…
srknzl Jul 8, 2021
9fd13b3
Remove whitespace change
srknzl Jul 8, 2021
7f7cbb8
Add somethings
srknzl Jul 8, 2021
b40e1f4
Changes
srknzl Jul 9, 2021
34c0347
wip
srknzl Jul 9, 2021
7714dc7
Update class def tests
srknzl Jul 9, 2021
3c21f50
Update rc script to download sql separately, and not download all jar…
srknzl Jul 9, 2021
a7d1713
Merge branch 'sql-5.0' into add-portable-methods
srknzl Jul 9, 2021
3e51e19
Update tests
srknzl Jul 12, 2021
bf3e857
Add constructors for datetime wrappers
srknzl Jul 12, 2021
de6564c
Merge
srknzl Jul 13, 2021
60a07a2
Fix tests
srknzl Jul 13, 2021
a2c1dc4
Merge branch 'sql-5.0' into add-portable-methods
srknzl Jul 13, 2021
930e777
Add config checks (#959)
srknzl Jul 6, 2021
059e9b0
Add bigint serializer
srknzl Jul 13, 2021
e909b45
Update targer version and add simple live test
srknzl Jul 13, 2021
6e1c120
Rename datetime classes
srknzl Jul 13, 2021
3c8a6ac
Onelinify
srknzl Jul 13, 2021
846a05d
Rename adjustment
srknzl Jul 13, 2021
18d5af1
Adjust datetime rename
srknzl Jul 13, 2021
5423351
Change names
srknzl Jul 27, 2021
c0fa741
revert unnecessary changes
srknzl Jul 27, 2021
ff4ae92
merge
srknzl Jul 27, 2021
94b9cbd
Add live serialization tests
srknzl Jul 27, 2021
c936abe
Test fix
srknzl Jul 28, 2021
e77a52e
merge
srknzl Jul 30, 2021
f256d78
Delete parser option
srknzl Jul 30, 2021
23d46b9
Revert eslint changes
srknzl Jul 30, 2021
076a8e5
Fix bigint registration
srknzl Aug 4, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -582,6 +582,7 @@ Hazelcast serializes all your objects before sending them to the server. Certain
|-----------------|---------------------------------------|
| boolean | Boolean |
| number | Byte, Short, Integer, Float, Double |
| BigInt | BigInteger |
| string | String |
| Long | Long |
| Buffer | byte[] |
Expand All @@ -591,7 +592,7 @@ Hazelcast serializes all your objects before sending them to the server. Certain

> **NOTE: A `number` is serialized as `Double` by default. You can configure this behavior using the `defaultNumberType` serialization config option. See [API Documentation](http://hazelcast.github.io/hazelcast-nodejs-client/api/current/docs/) for more information.**

Arrays of the `boolean`, `number`, `string`, and `Long` types can be serialized as `boolean[]`, `byte[]`, `short[]`, `int[]`, `float[]`, `double[]`, `string[]`, and `long[]` for the Java server side, respectively.
Arrays of the `boolean`, `number`, `BigInt`, `string`, and `Long` types can be serialized as `boolean[]`, `BigInteger[]', `byte[]`, `short[]`, `int[]`, `float[]`, `double[]`, `string[]`, and `long[]` for the Java server side, respectively.

**Serialization Priority**

Expand Down
16 changes: 16 additions & 0 deletions src/serialization/DefaultSerializers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -559,3 +559,19 @@ export class BigDecimalSerializer implements Serializer<BigDecimal> {
output.writeInt(bigDecimal.scale);
}
}

/** @internal */
export class BigIntSerializer implements Serializer<BigInt> {

id = -26;

read(input: DataInput): BigInt {
const body = input.readByteArray();
return BigDecimalUtil.bufferToBigInt(body);
}

write(output: DataOutput, bigint: BigInt): void {
output.writeByteArray(BigDecimalUtil.bigIntToBuffer(bigint));
}
}

2 changes: 2 additions & 0 deletions src/serialization/SerializationService.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {Serializer, IdentifiedDataSerializableFactory} from './Serializable';
import {
ArrayListSerializer,
BigDecimalSerializer,
BigIntSerializer,
BooleanArraySerializer,
BooleanSerializer,
ByteArraySerializer,
Expand Down Expand Up @@ -291,6 +292,7 @@ export class SerializationServiceV1 implements SerializationService {
this.registerSerializer('linkedList', new LinkedListSerializer());
this.registerSerializer('uuid', new UuidSerializer());
this.registerSerializer('bigDecimal', new BigDecimalSerializer());
this.registerSerializer('bigInt', new BigIntSerializer());
this.registerIdentifiedFactories();
this.registerSerializer('!portable', new PortableSerializer(this.serializationConfig));
if (this.serializationConfig.jsonStringDeserializationPolicy === JsonStringDeserializationPolicy.EAGER) {
Expand Down
2 changes: 2 additions & 0 deletions src/util/Util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ export function getType(obj: any): string {
return 'offsetDateTime';
} else if (obj instanceof BigDecimal) {
return 'bigDecimal';
} else if (typeof obj === 'bigint') {
mdumandag marked this conversation as resolved.
Show resolved Hide resolved
return 'bigInt';
} else {
const t = typeof obj;
if (t !== 'object') {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,13 @@ describe('DefaultSerializersLiveTest', function () {

it('string', async function () {
await map.put('testStringKey', 'testStringValue');
const response = await RC.executeOnController(cluster.id, generateGet('testStringKey'), 1);
const response = await RC.executeOnController(cluster.id, generateGet('testStringKey'), Lang.JAVASCRIPT);
expect(response.result.toString()).to.equal('testStringValue');
});

it('utf8 sample string test', async function () {
await map.put('key', 'Iñtërnâtiônàlizætiøn');
const response = await RC.executeOnController(cluster.id, generateGet('key'), 1);
const response = await RC.executeOnController(cluster.id, generateGet('key'), Lang.JAVASCRIPT);
expect(response.result.toString()).to.equal('Iñtërnâtiônàlizætiøn');
});

Expand All @@ -80,7 +80,7 @@ describe('DefaultSerializersLiveTest', function () {

it('array', async function () {
await map.put('a', ['a', 'v', 'vg']);
const response = await RC.executeOnController(cluster.id, generateGet('a'), 1);
const response = await RC.executeOnController(cluster.id, generateGet('a'), Lang.JAVASCRIPT);
expect(response.result.toString()).to.equal(['a', 'v', 'vg'].toString());
});

Expand Down Expand Up @@ -111,19 +111,19 @@ describe('DefaultSerializersLiveTest', function () {

it('emoji string test on RC', async function () {
await map.put('key', '1⚐中💦2😭‍🙆😔5');
const response = await RC.executeOnController(cluster.id, generateGet('key'), 1);
const response = await RC.executeOnController(cluster.id, generateGet('key'), Lang.JAVASCRIPT);
expect(response.result.toString()).to.equal('1⚐中💦2😭‍🙆😔5');
});

it('utf8 characters test on RC', async function () {
await map.put('key', '\u0040\u0041\u01DF\u06A0\u12E0\u{1D306}');
const response = await RC.executeOnController(cluster.id, generateGet('key'), 1);
const response = await RC.executeOnController(cluster.id, generateGet('key'), Lang.JAVASCRIPT);
expect(response.result.toString()).to.equal('\u0040\u0041\u01DF\u06A0\u12E0\u{1D306}');
});

it('utf8 characters test on RC with surrogates', async function () {
await map.put('key', '\u0040\u0041\u01DF\u06A0\u12E0\uD834\uDF06');
const response = await RC.executeOnController(cluster.id, generateGet('key'), 1);
const response = await RC.executeOnController(cluster.id, generateGet('key'), Lang.JAVASCRIPT);
expect(response.result.toString()).to.equal('\u0040\u0041\u01DF\u06A0\u12E0\u{1D306}');
});

Expand All @@ -142,7 +142,7 @@ describe('DefaultSerializersLiveTest', function () {
'\\"value\\": \\"" + new String(value) + "\\"}"\n';

await map.put('key', restValue);
const response = await RC.executeOnController(cluster.id, script, 1);
const response = await RC.executeOnController(cluster.id, script, Lang.JAVASCRIPT);
const result = JSON.parse(response.result.toString());
expect(result.contentType).to.equal(restValue.contentType);
expect(result.value).to.equal(restValue.value);
Expand All @@ -157,7 +157,7 @@ describe('DefaultSerializersLiveTest', function () {
'result = "\\"" + uuid.toString() + "\\"";\n';

await map.put('key', uuid);
const response = await RC.executeOnController(cluster.id, script, 1);
const response = await RC.executeOnController(cluster.id, script, Lang.JAVASCRIPT);
const result = JSON.parse(response.result);
expect(result).to.equal(uuid.toString());
});
Expand All @@ -171,7 +171,7 @@ describe('DefaultSerializersLiveTest', function () {
'list.add(3);\n' +
'map.set("key", list);\n';

await RC.executeOnController(cluster.id, script, 1);
await RC.executeOnController(cluster.id, script, Lang.JAVASCRIPT);

const actualValue = await map.get('key');
expect(actualValue).to.deep.equal([1, 2, 3]);
Expand All @@ -186,7 +186,7 @@ describe('DefaultSerializersLiveTest', function () {
'list.add(3);\n' +
'map.set("key", list);\n';

await RC.executeOnController(cluster.id, script, 1);
await RC.executeOnController(cluster.id, script, Lang.JAVASCRIPT);

const actualValue = await map.get('key');
expect(actualValue).to.deep.equal([1, 2, 3]);
Expand Down Expand Up @@ -545,4 +545,47 @@ describe('DefaultSerializersLiveTest', function () {
}
}
});

const bigIntParams = [
['1111', 1111n],
['-1111', -1111n],
['9999999999999999999999999', 9999999999999999999999999n],
['-9999999999999999999999999', -9999999999999999999999999n],
['0', 0n],
['1', 1n],
['-7', -7n],
];

it('should deserialize BigInt', async function () {
TestUtil.markClientVersionAtLeast(this, '5.0');

let script = 'var map = instance_0.getMap("' + map.getName() + '");\n';

bigIntParams.forEach((values, index) => {
const bigIntString = values[0];
script += `map.set("${index}", new java.math.BigInteger("${bigIntString}"));\n`;
});

await RC.executeOnController(cluster.id, script, Lang.JAVASCRIPT);

for (let i = 0; i < bigIntParams.length; i++) {
const actualValue = await map.get(i.toString());
const expectedBigIntValue = bigIntParams[i][1];

expect(actualValue).to.be.equal(expectedBigIntValue);
}
});

it('should serialize BigInt correctly', async function () {
TestUtil.markClientVersionAtLeast(this, '5.0');
for (let i = 0; i < bigIntParams.length; i++) {
const bigintValue = bigIntParams[i][1];
await map.put(i.toString(), bigintValue);
}

for (let i = 0; i < bigIntParams.length; i++) {
const responseString = await getMapValueAsString(i);
expect(responseString).to.be.equal(bigIntParams[i][0]);
}
});
});
Binary file modified test/unit/serialization/1.serialization.compatibility.binary
Binary file not shown.
6 changes: 4 additions & 2 deletions test/unit/serialization/DefaultSerializersTest.js
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,14 @@ describe('DefaultSerializersTest', function () {
new LocalTime(11, 22, 41, 123456789),
new LocalDateTime(new LocalDate(2022, 7, 29), new LocalTime(12, 23, 42, 123456789)),
new OffsetDateTime(new LocalDateTime(new LocalDate(2022, 7, 29), new LocalTime(12, 23, 42, 123456789)), -64800),
BigDecimal.fromString('1.11111111111111111111111111')
BigDecimal.fromString('1.11111111111111111111111111'),
BigInt('111111111111111111111111111'),
];

parameters.forEach((obj) => {
it('type: ' + typeof obj + ', isArray: ' + Array.isArray(obj) + ', value: '
+ (obj instanceof BigDecimal ? `BigDecimal ${obj.toString()}` : JSON.stringify(obj)), function () {
+ (obj instanceof BigDecimal ? `BigDecimal ${obj.toString()}` :
(typeof obj === 'bigint' ? `BigInt ${obj}` : JSON.stringify(obj))), function () {
const config = new SerializationConfigImpl();
const serializationService = new SerializationServiceV1(config);
const serialized = serializationService.toData(obj);
Expand Down
4 changes: 3 additions & 1 deletion test/unit/serialization/ReferenceObjects.java
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import com.hazelcast.internal.serialization.Data;
import com.hazelcast.nio.serialization.Portable;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.nio.CharBuffer;
import java.time.LocalTime;
import java.time.LocalDate;
Expand Down Expand Up @@ -141,6 +142,7 @@ class ReferenceObjects {
aOffsetDateTime = OffsetDateTime.of(aLocalDateTime, ZoneOffset.ofHours(18));
}

static BigInteger aBigInteger = new BigInteger("1314432323232411");
static BigDecimal aBigDecimal = new BigDecimal(31231);
static Class aClass = BigDecimal.class;

Expand All @@ -150,6 +152,6 @@ class ReferenceObjects {
booleans, bytes, chars, doubles, shorts, floats, ints, longs, strings,
aCustomStreamSerializable, aCustomByteArraySerializable,
anIdentifiedDataSerializable, aPortable,
aDate, aLocalDate, aLocalTime, aLocalDateTime, aOffsetDateTime, aBigDecimal, aClass
aDate, aLocalDate, aLocalTime, aLocalDateTime, aOffsetDateTime, aBigInteger, aBigDecimal, aClass
};
}
1 change: 1 addition & 0 deletions test/unit/serialization/ReferenceObjects.js
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ to.APortable = new APortable(
to.aCustomByteArraySerializable, exports.aData
);
to.aDate = new Date(Date.UTC(1990, 2, 1, 0, 0, 0, 0));
to.aBigInteger = BigInt('1314432323232411');
to.aBigDecimal = BigDecimal.fromString('31231');
to.aClass = 'java.math.BigDecimal';

Expand Down
6 changes: 3 additions & 3 deletions tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
"strictBindCallApply": true,
"strictFunctionTypes": true,
"noImplicitThis": true,
"target": "es2016",
"target": "es2018",
mdumandag marked this conversation as resolved.
Show resolved Hide resolved
"outDir": "lib",
"preserveConstEnums": true,
"lib": [
"es2016",
"es2017.string"
"es2018",
"es2020.bigint"
],
"removeComments": false,
"sourceMap": true,
Expand Down