Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to elasticsearch v8.7.0 #867

Merged

Conversation

MarvinZeising
Copy link
Contributor

@MarvinZeising MarvinZeising commented Apr 11, 2023

I'm proposing an upgrade to elasticsearch v8.7.0, as requested in #868.

The tests didn't finish locally (even before my changes), so I'm interested to see what the CI action says.

@MarvinZeising MarvinZeising force-pushed the upgrade-to-elasticsearch-8.7.0 branch 2 times, most recently from b5a8d6f to d6c0acb Compare April 11, 2023 18:40
@codecov-commenter
Copy link

codecov-commenter commented Apr 12, 2023

Codecov Report

Patch coverage: 87.69% and project coverage change: -0.10 ⚠️

Comparison is base (97f7ed9) 85.87% compared to head (06d401f) 85.77%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #867      +/-   ##
==========================================
- Coverage   85.87%   85.77%   -0.10%     
==========================================
  Files          50       52       +2     
  Lines        2350     2460     +110     
==========================================
+ Hits         2018     2110      +92     
- Misses        173      182       +9     
- Partials      159      168       +9     
Impacted Files Coverage Δ
preset/vault/preset.go 77.41% <77.41%> (ø)
preset/k3s/preset.go 80.39% <94.11%> (-0.95%) ⬇️
preset/cassandra/preset.go 93.10% <100.00%> (ø)
preset/cockroachdb/preset.go 81.53% <100.00%> (ø)
preset/elastic/preset.go 68.70% <100.00%> (+0.24%) ⬆️
preset/k3s/options.go 100.00% <100.00%> (ø)
preset/mariadb/preset.go 88.57% <100.00%> (ø)
preset/memcached/preset.go 89.74% <100.00%> (ø)
preset/mongo/preset.go 74.73% <100.00%> (ø)
preset/mssql/preset.go 85.07% <100.00%> (ø)
... and 5 more

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Owner

@orlangure orlangure left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this!

The tests indeed fail now because the container never becomes healthy. I guess something is changed in the API so we can't use the same method to test container readiness. If you still want to debug this, I suggest starting with WithLogWriter and WithDebugMode options to look into what happens inside the container, and where does the flow get stuck.

If there was a change in the API, we'll have to support multiple health checks depending on preset version.

preset/elastic/preset.go Show resolved Hide resolved
preset/elastic/preset_test.go Outdated Show resolved Hide resolved
@orlangure
Copy link
Owner

Hey @MarvinZeising,
Turns out elastic tests were garbage and only tested a single version (7), as you just pointed out in your PR.

It means that only one major version was supported, and versions 5 and 6 were not. As I don't use Elastic in my current projects, I'm not even sure how important is this support. I think that maybe version 6 is somewhat important to support, but 5 can be dropped...

Anyway, if you decide to work on supporting 7 and 8 side by side (in code and in tests), maybe you could look into supporting version 5 and 6, or only 6, at the same time. For now, I modified the tests to only run on version 7, and removed 5 and 6 from the list of supported versions in readme.

@MarvinZeising
Copy link
Contributor Author

Thanks for your review @orlangure.

Versions 5 and 6 of elasticsearch are already at end of life. If they're not supported officially and no one is using them (because they didn't work with gnomork before), I don't think we should add this.
Version 7.9.3 is also at end of life, so I updated it to the last v7 which is 7.17.9. That should be backwards-compatible with the previous minor versions.

The container wasn't getting healthy because the endpoint is secure. Disabling TLS fixed. Since in my company, we use that setting in v7 as well (because it's a sensible default setting) I didn't even add a version check.

I tried to combine the tests for the different versions like you did. But that proved to be difficult since there's no common interface for the clients. So I figured the cleanest approach would just be to separate these tests and use the respective client library.
Is that okay for you?

@orlangure
Copy link
Owner

I don't mind separating the tests, if this is easier.

But something confuses me. We now use client v8 in the preset regardless of versions (7 or 8). Does this client know how to connect and work with both Elastic 7 and 8? Because a server test for example fails because it sends v7.9.2 as the version, and Elastic client v8 can't connect to it. I would expect that existing tests continue to work if they use v7. So something wasn't backwards compatible between 7.9 and 7.17. What do you think we should do with this issue?

@MarvinZeising
Copy link
Contributor Author

Thanks for your comment, @orlangure!

I missed two places to update the default elasticsearch version in. I fixed that and now the tests succeed locally.

My idea was that preset_test.go tests the default version (currently 8.7.0) and preset_<version>_test.go tests a specific supported older version. That's why the first one uses the v8 client, while the other test uses the v7 client.

I also tried the server test with 7.17.9 (by specifying that in elastic.json > preset.version) and the v8 client is indeed backwards-compatible with that version, just not with 7.9 as that seems to be too old.

Copy link
Owner

@orlangure orlangure left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, thank you for the contribution!

@orlangure orlangure merged commit cc815d7 into orlangure:master Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants