Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Less contentious restructure #4966

Merged
merged 11 commits into from
Apr 19, 2024
Merged

Conversation

thejohnfreeman
Copy link
Collaborator

@thejohnfreeman thejohnfreeman commented Mar 25, 2024

This PR implements half of the changes planned for the physical redesign. These changes, described below, are in files touched infrequently by other contributors, so I expect them to be less disruptive or contentious. (The second half of the changes will come in a later PR that I expect to be more contentious.)

  • Remove several unused files, including CMake modules and years-old scripts.
  • Remove Builds/containers, which I've been helping @legleux over the last couple of weeks exfiltrate to a separate project, to decouple its development cadence from that of rippled. As far as I know, only Ripple uses these files, and only to build and test packages.
  • Consolidate all third-party libraries within external/ by moving secp256k1 and ed25519-donna out of src/. From now on, src/ will contain only first-party sources.
  • Simplify the generation of protobufs by leveraging the standard CMake function for it. Put the generation commands next to each other in the same file. Package protobufs into a library separate from libxrpl, to avoid issues with unity builds.
  • Standardize on .h as the only extension for first-party headers. This could require an importer to change its include paths, but the only affected headers are part of beast/{unit_,}test/, so I don't think any other projects are affected (validator-keys-tool is not).

Before this PR is merged, we should:

  • Confirm it has at least two approvals from qualified reviewers.
  • Rebase the commits on top of develop. The commit sequence that predates the review should be preserved. Fixes made since the review opened should be merged into earlier commits.
  • Rewrite non-substantive (e.g. formatting) commits as authored by "Pretty Printer [email protected]".
  • Add rewritten non-substantive commit IDs to .git-blame-ignore-revs.
  • Confirm the branch still passes all tests after making these changes.

@codecov-commenter
Copy link

codecov-commenter commented Mar 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.9%. Comparing base (676aae2) to head (b84f7e7).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           develop   #4966     +/-   ##
=========================================
- Coverage     71.0%   70.9%   -0.0%     
=========================================
  Files          796     796             
  Lines        66727   66727             
  Branches     10973   10978      +5     
=========================================
- Hits         47348   47336     -12     
- Misses       19379   19391     +12     
Files Coverage Δ
src/ripple/app/main/Main.cpp 47.0% <ø> (ø)
src/ripple/app/paths/AMMLiquidity.h 100.0% <ø> (ø)
src/ripple/overlay/impl/TxMetrics.cpp 16.2% <ø> (ø)
src/ripple/overlay/impl/TxMetrics.h 100.0% <ø> (ø)
src/ripple/protocol/impl/SecretKey.cpp 83.4% <100.0%> (ø)

... and 5 files with indirect coverage changes

Impacted file tree graph

Copy link
Collaborator

@godexsoft godexsoft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the changes 👍
Just one thing that i don't necessarily agree with is renaming to .h instead of going the other way and renaming everything to .hpp. Do you have C code in the project? Or why was .h preferred?
To me it feels like a bit of a step backwards.

One side-effect of using .h is that github will count and report some files as C code if the header is simple enough that it could pass for a C header.

Copy link
Collaborator

@scottschurr scottschurr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This restructure looks great to me.

FWIW, unlike @godexsoft, I think changing the few *.hpp files to *.h files is a good approach. The code base has a ton of *.h files. If we're going for uniformity (and I think that's a good idea) this smaller change is better. Switching all of the *.h files in the build to *.hpp would be much more disruptive.

Almost everything in this pull request built for me out of the box. But I had to make two small changes to the source to get the build to complete. Neither file that I had to change is currently in this change set. So I'll describe the issues here.

  1. In src/ripple/protocol/impl/SecretKey.cpp:194 I get the following error:
error: 'secp256k1_ec_privkey_tweak_add' is deprecated: Use secp256k1_ec_seckey_tweak_add instead [-Werror,-Wdeprecated-declarations]
            if (secp256k1_ec_privkey_tweak_add(
                ^

If I change the secp256k1_ec_privkey_tweak_add on that line to secp256k1_ec_seckey_tweak_add then that file builds and links.

  1. I get the following error on src/test/protocol/MultiApiJson_test.cpp:55:
error: unused variable 'index' [-Werror,-Wunused-const-variable]
    constexpr static auto index =
                          ^

And, indeed, if I remove the five lines that declare this static lambda then everything builds and links just fine.

Once those small things are patched up everything builds, links, passes unit tests, and syncs on my Intel Mac.

@godexsoft
Copy link
Collaborator

@scottschurr I have no objections either way. If moving to .h is significantly easier then that's a valid strategy 👍
I'm wondering however who is it disruptive for? Does anyone but Clio actually consume libxrpl? From Clio's perspective, where we already have everything as .hpp it would look better if rippled's headers were also .hpp but that's very minor.

@scottschurr
Copy link
Collaborator

I have no objections either way. If moving to .h is significantly easier then that's a valid strategy 👍
I'm wondering however who is it disruptive for?

That's a fair question, @godexsoft. In terms of disruption I was thinking of folks, like me, who work in the rippled code base. Renaming fewer files means I don't have to retrain that part of my brain. I also frequently have trouble tracing history through renamed files in git. Reducing the number of renamed files will increase the likelihood that looking at history using git continues to work for more files.

But neither of those are killer arguments. It's mostly about my personal convenience. So, yeah, I'm still leaning toward the current approach that @thejohnfreeman has taken.

@thejohnfreeman
Copy link
Collaborator Author

I prefer .hpp to .h too, stylistically, but the .hpp files are in the extreme minority in this codebase, to the point that they are overlooked: they have been mistakenly excluded from the formatting requirements for years.

The next PR is about to rename every file in src/ripple. If we want to adopt .hpp, that would be the time, but I don't feel strongly enough to fight for it.

@ximinez
Copy link
Collaborator

ximinez commented Apr 2, 2024

Does anyone but Clio actually consume libxrpl?

Two that I know about are:

Copy link
Collaborator

@ximinez ximinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a couple of little tweaks, but otherwise this looks good.

external/README.md Outdated Show resolved Hide resolved
external/README.md Outdated Show resolved Hide resolved
src/ripple/beast/test/yield_to.h Outdated Show resolved Hide resolved
src/ripple/protocol/impl/SecretKey.cpp Show resolved Hide resolved
Copy link
Collaborator

@ximinez ximinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good.

Comment on lines 1 to 11
These directories come from outside.
The subdirectories in this directory contain either copies or Conan recipes
of external libraries used by rippled.
The Conan recipes include patches we have not yet pushed upstream.

| Folder | Upstream Repo | Description |
| Folder | Upstream | Description |
|:----------------|:---------------------------------------------|:------------|
| `ed25519-donna` | https://github.com/floodyberry/ed25519-donna | [Ed25519](http://ed25519.cr.yp.to/) digital signatures |
| `rocksdb` | https://github.com/conan-io/conan-center-index/tree/master/recipes/rocksdb | Fast key/value database. (Supports rotational disks better than NuDB.) |
| `secp256k1` | https://github.com/bitcoin-core/secp256k1 | ECDSA digital signatures using the **secp256k1** curve |
| `snappy` | https://github.com/conan-io/conan-center-index/tree/master/recipes/snappy | "Snappy" lossless compression algorithm. |
| `soci` | https://github.com/conan-io/conan-center-index/tree/master/recipes/soci | Abstraction layer for database access. |
| `ed25519-donna` | [Project](https://github.com/floodyberry/ed25519-donna) | [Ed25519](http://ed25519.cr.yp.to/) digital signatures |
| `rocksdb` | [Recipe](https://github.com/conan-io/conan-center-index/tree/master/recipes/rocksdb) | Fast key/value database. (Supports rotational disks better than NuDB.) |
| `secp256k1` | [Project](https://github.com/bitcoin-core/secp256k1) | ECDSA digital signatures using the **secp256k1** curve |
| `snappy` | [Recipe](https://github.com/conan-io/conan-center-index/tree/master/recipes/snappy) | "Snappy" lossless compression algorithm. |
| `soci` | [Recipe](https://github.com/conan-io/conan-center-index/tree/master/recipes/soci) | Abstraction layer for database access. |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like how you updated this!

Copy link
Collaborator

@Bronek Bronek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that the following commits do exactly what they claim to do, and can be added to .git-blame-ignore-revs

fa3e44cc07 Consolidate external libraries
95491c0cce Rename .hpp to .h
4de37264d4 Format formerly .hpp files
ccacc94aba Rewrite includes

I would like to know for sure that the removal of packaging files won't hurt the users - I assumed they are used to build .deb and .rpm but I guess was wrong ?

src/ripple/protocol/impl/SecretKey.cpp Show resolved Hide resolved
src/ripple/json/impl/LICENSE Show resolved Hide resolved
@thejohnfreeman
Copy link
Collaborator Author

@Bronek the packaging files have been moved to a separate, internal (Ripple GitLab) project. They were never used outside of Ripple. They drive our internal package pipelines. As a separate project, they can be developed outside of the rippled review and release process. I've already worked with @legleux to ensure that the new project can successfully package this branch.

@thejohnfreeman
Copy link
Collaborator Author

Now that this PR has two approvals, I will work on the finishing steps outlined at the end of the PR description.

@thejohnfreeman thejohnfreeman merged commit b84f7e7 into XRPLF:develop Apr 19, 2024
17 checks passed
@thejohnfreeman thejohnfreeman deleted the backport branch April 19, 2024 18:42
@ximinez ximinez mentioned this pull request Apr 26, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants