Add RPC for `server_definitions` to rippled #4703

mvadari · 2023-09-13T19:41:21Z

High Level Overview of Change

Add a new RPC / WS call: server_definitions, which returns an SDK-compatible definitions.json generated from the rippled instance currently running. This will alleviate definitions juggling, especially with side chains which may have exotic features and serialized fields. Software will be able to query server_definitions against a node from the network they want to connect to and immediately know how to speak that node's language, even if new features are added to it in the future (as long as there are no new serialized types that the software doesn't know how to serialize/deserialize).

Example:

> {"command": "server_definitions"}
< {
    "result": {
        "FIELDS": [
            [
                "Generic",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": 0,
                    "type": "Unknown"
                }
            ],
            [
                "Invalid",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": -1,
                    "type": "Unknown"
                }
            ],
            [
                "ObjectEndMarker",
                {
                    "isSerialized": false,
                    "isSigningField": true,
                    "isVLEncoded": false,
                    "nth": 1,
                    "type": "STObject"
                }
            ],
        ...

Context of Change

I took #4402 and modified it to avoid using magic_enum. Closes #3657.

Type of Change

New feature (non-breaking change which adds functionality)

Test Plan

I haven't written tests yet - I wanted to get this out first, because I'm sure there will be plenty of other issues.

src/ripple/rpc/handlers/ServerInfo.cpp

src/ripple/protocol/KnownFormats.h

nbougalis · 2023-09-14T12:02:22Z

src/ripple/protocol/SField.h

+#pragma push_macro("XMACRO")
+#undef XMACRO
+
+#define XMACRO(STYPE)             \


Yuck. Consider using Boost.Describe at least.

I can't figure out how to properly use Boost.Describe - I keep getting a ton of errors. The documentation is very unclear, since the API has changed a lot in the past few versions. Even ChatGPT/Bard isn't able to provide much help.

nbougalis · 2023-09-14T12:05:45Z

src/ripple/rpc/handlers/ServerInfo.cpp

+    bool
+    hashMatches(uint256 hash) const
+    {
+        return defsHash && *defsHash == hash;


Not needed: you can directly compare an std::optional<T> with a T and it will do the right thing: return hash == defsHash; is enough.

Although, as I said before, it's unclear why defsHash is an std::optional or even a variable at all. You could just implement this as:

return to_string(hash) == defs[jss::value].asString()

I removed the std::optional in 5fc2ee5, IMO it's a bit clearer and slightly more performant to have the separate variable.

src/ripple/rpc/handlers/ServerInfo.cpp

src/ripple/protocol/SField.h

nbougalis · 2023-09-14T19:02:07Z

src/ripple/rpc/handlers/ServerInfo.cpp

+            return RPC::invalid_field_error(jss::hash);
+    }
+
+    static const Definitions defs{};


Minor nit, but I don't know that you actually need an instance of a class for this: you may as well just have something like inline const Json::Value definitions = []() { ... }();

IMO things like the hashMatches method are a bit neater with a class as opposed to something inline. If you think I should refactor to remove the class though, I can.

sublimator · 2023-09-22T04:55:24Z

Nice. Been wanting this for a while:
#1972

src/ripple/app/main/Main.cpp

src/ripple/rpc/impl/Handler.cpp

src/ripple/protocol/SField.h

src/ripple/protocol/jss.h

src/ripple/app/main/Main.cpp

src/ripple/rpc/handlers/ServerInfo.cpp

ckeshava · 2023-10-18T18:58:33Z

The PR looks nice. I've still left some suggestions, but they are my subjective opinions and shouldn't stop the PR from going into production

mvadari · 2023-10-18T20:44:55Z

@intelliot this is ready to merge 👍

mvadari · 2023-10-18T20:56:50Z

I don't have the powers to retry actions but it looks like something failed with the container setup in the dependency test.

intelliot · 2023-10-18T20:59:33Z

I clicked the button to retry the CI job
I've asked @nbougalis to confirm that he doesn't have any remaining blockers (i.e. comments/concerns that still must be addressed)

"If others have signed off, please use their sign off instead."

intelliot

@ckeshava has approved the latest code in this PR (aside from the merge commit)

intelliot · 2023-10-18T22:38:47Z

Suggested commit message:

feat(rpc): add `server_definitions` method (#4703)

Add a new RPC / WS call for `server_definitions`, which returns an
SDK-compatible `definitions.json` (binary enum definitions) generated by
the server. This enables clients/libraries to dynamically work with new
fields and features, such as ones that may become available on side
chains. Clients query `server_definitions` on a node from the network
they want to work with, and immediately know how to speak that node's
binary "language", even if new features are added to it in the future
(as long as there are no new serialized types that the software doesn't
know how to serialize/deserialize).

Example:

```js
> {"command": "server_definitions"}
< {
    "result": {
        "FIELDS": [
            [
                "Generic",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": 0,
                    "type": "Unknown"
                }
            ],
            [
                "Invalid",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": -1,
                    "type": "Unknown"
                }
            ],
            [
                "ObjectEndMarker",
                {
                    "isSerialized": false,
                    "isSigningField": true,
                    "isVLEncoded": false,
                    "nth": 1,
                    "type": "STObject"
                }
            ],
        ...
\```

Close #3657

---------

Co-authored-by: Richard Holland <[email protected]>

kennyzlei · 2023-10-24T17:04:31Z

For Clio, we are considering adding this support in 2.2 release as it needs some time to investigate support for this API. This means for s1/s2, rippled 2.0 release would mean we would need to wait for Clio 2.2 to have this RPC accessible

XRPLF/clio#942

mvadari · 2023-10-24T17:06:30Z

For Clio, we are considering adding this support in 2.2 release as it needs some time to investigate support for this API. This means for s1/s2, rippled 2.0 release would mean we would need to wait for Clio 2.2 to have this RPC accessible

XRPLF/clio#942

@kennyzlei is it possible to just forward to rippled in 2.1? It should be pretty performant as-is and shouldn't need to be called too often.

kennyzlei · 2023-10-24T17:11:25Z

@mvadari I believe this will depend on the timing of our Clio releases. We are starting to release more frequently with 2.1 slotted to release in 2 weeks, so 2.2 should not be delayed by much. We need the investigation time to decide if Clio should forward these requests or do something, which may be tight to complete within 2 weeks

Add support for new command added in XRPLF/rippled#4703

SaxenaKaustubh · 2023-11-09T15:01:14Z

Worked on this with @mvadari on this change. After a few iterations, we have this working the way we want. This concludes the manual testing.

Add a new RPC / WS call for `server_definitions`, which returns an SDK-compatible `definitions.json` (binary enum definitions) generated by the server. This enables clients/libraries to dynamically work with new fields and features, such as ones that may become available on side chains. Clients query `server_definitions` on a node from the network they want to work with, and immediately know how to speak that node's binary "language", even if new features are added to it in the future (as long as there are no new serialized types that the software doesn't know how to serialize/deserialize). Example: ```js > {"command": "server_definitions"} < { "result": { "FIELDS": [ [ "Generic", { "isSerialized": false, "isSigningField": false, "isVLEncoded": false, "nth": 0, "type": "Unknown" } ], [ "Invalid", { "isSerialized": false, "isSigningField": false, "isVLEncoded": false, "nth": -1, "type": "Unknown" } ], [ "ObjectEndMarker", { "isSerialized": false, "isSigningField": true, "isVLEncoded": false, "nth": 1, "type": "STObject" } ], ... ``` Close XRPLF#3657 --------- Co-authored-by: Richard Holland <[email protected]>

RichardAH and others added 9 commits September 13, 2023 12:40

Add rpc call server_definitions to rippled.

439179d

capitalization fix for server_definitions with URIToken

c8f73bd

remove magic_enum

ede9634

get STypes, TxTypes, LedgerEntryTypes, Fields working

e8ba1ef

get TER codes working

38db68b

clean up

18cecad

run clang-format

2ac883e

fix bugs + clean up

d36cb79

add to CLI help list

54f19e3

mvadari mentioned this pull request Sep 13, 2023

Add rpc call server_definitions to rippled. #4402

Closed

fix header issue

1607c55

ckeshava reviewed Sep 13, 2023

View reviewed changes

src/ripple/rpc/handlers/ServerInfo.cpp Outdated Show resolved Hide resolved

Merge branch 'develop' into server-definitions

2ca2bb7

nbougalis previously requested changes Sep 14, 2023

View reviewed changes

mvadari added 4 commits September 18, 2023 16:51

use KnownFormats iterator instead of custom map

ff4f55d

add comments back into macro

166ab29

merge develop

fd2da62

Merge branch 'develop' into server-definitions

e22e3b3

intelliot requested review from seelabs and ckeshava September 25, 2023 05:24

ckeshava reviewed Sep 28, 2023

View reviewed changes

src/ripple/app/main/Main.cpp Show resolved Hide resolved

ckeshava reviewed Sep 28, 2023

View reviewed changes

src/ripple/rpc/impl/Handler.cpp Show resolved Hide resolved

ckeshava reviewed Oct 2, 2023

View reviewed changes

intelliot assigned seelabs and ckeshava Oct 2, 2023

mvadari added 4 commits October 3, 2023 16:49

Merge branch 'develop' into server-definitions

423bf60

make defsHash not optional

5fc2ee5

add hash support to rippled CLI

5d36478

fix XChainBridge capitalization

f430626

fix build issue

cd69226

ckeshava approved these changes Oct 18, 2023

View reviewed changes

Merge branch 'develop' into server-definitions

008f092

intelliot modified the milestones: 2024 release, 2.0 Oct 18, 2023

intelliot approved these changes Oct 18, 2023

View reviewed changes

intelliot merged commit 078bd60 into XRPLF:develop Oct 18, 2023
17 checks passed

intelliot added Passed Passed code review & PR owner thinks it's ready to merge. Perf sign-off may still be required. Feature Request Used to indicate requests to add new features Will Need Documentation Added to API Changelog labels Oct 18, 2023

cindyyan317 mentioned this pull request Oct 19, 2023

New RPC :server_definitions XRPLF/clio#942

Closed

mvadari mentioned this pull request Oct 20, 2023

feat: add support for server_definitions RPC XRPLF/xrpl.js#2535

Merged

3 tasks

intelliot added the Testable label Oct 20, 2023

intelliot mentioned this pull request Oct 23, 2023

Proposed 2.0.0-b4 #4782

Merged

1 task

kennyzlei added the Clio Reviewed label Oct 24, 2023

mvadari mentioned this pull request Oct 24, 2023

feat: add support for server_definitions RPC XRPLF/xrpl-py#659

Merged

3 tasks

ckniffen pushed a commit to XRPLF/xrpl.js that referenced this pull request Oct 31, 2023

feat: add support for server_definitions RPC (#2535)

14f40f1

Add support for new command added in XRPLF/rippled#4703

intelliot mentioned this pull request Dec 1, 2023

Consider automatically generate enum definitions during compilation #3657

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add RPC for `server_definitions` to rippled #4703

Add RPC for `server_definitions` to rippled #4703

mvadari commented Sep 13, 2023 •

edited

Loading

nbougalis Sep 14, 2023

mvadari Oct 2, 2023 •

edited

Loading

nbougalis Sep 14, 2023

mvadari Oct 17, 2023

nbougalis Sep 14, 2023

mvadari Oct 3, 2023

sublimator commented Sep 22, 2023

ckeshava commented Oct 18, 2023

mvadari commented Oct 18, 2023

mvadari commented Oct 18, 2023

intelliot commented Oct 18, 2023

intelliot left a comment

intelliot commented Oct 18, 2023 •

edited

Loading

kennyzlei commented Oct 24, 2023

mvadari commented Oct 24, 2023

kennyzlei commented Oct 24, 2023

SaxenaKaustubh commented Nov 9, 2023

Add RPC for server_definitions to rippled #4703

Add RPC for server_definitions to rippled #4703

Conversation

mvadari commented Sep 13, 2023 • edited Loading

High Level Overview of Change

Context of Change

Type of Change

Test Plan

nbougalis Sep 14, 2023

Choose a reason for hiding this comment

mvadari Oct 2, 2023 • edited Loading

Choose a reason for hiding this comment

nbougalis Sep 14, 2023

Choose a reason for hiding this comment

mvadari Oct 17, 2023

Choose a reason for hiding this comment

nbougalis Sep 14, 2023

Choose a reason for hiding this comment

mvadari Oct 3, 2023

Choose a reason for hiding this comment

sublimator commented Sep 22, 2023

ckeshava commented Oct 18, 2023

mvadari commented Oct 18, 2023

mvadari commented Oct 18, 2023

intelliot commented Oct 18, 2023

intelliot left a comment

Choose a reason for hiding this comment

intelliot commented Oct 18, 2023 • edited Loading

kennyzlei commented Oct 24, 2023

mvadari commented Oct 24, 2023

kennyzlei commented Oct 24, 2023

SaxenaKaustubh commented Nov 9, 2023

Add RPC for `server_definitions` to rippled #4703

Add RPC for `server_definitions` to rippled #4703

mvadari commented Sep 13, 2023 •

edited

Loading

mvadari Oct 2, 2023 •

edited

Loading

intelliot commented Oct 18, 2023 •

edited

Loading