Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Importer(s) are not communicating import done correctly #26046

Closed
Tracked by #26058
ChrisKujawa opened this issue Dec 13, 2024 · 9 comments · Fixed by #26124
Closed
Tracked by #26058

Importer(s) are not communicating import done correctly #26046

ChrisKujawa opened this issue Dec 13, 2024 · 9 comments · Fixed by #26124
Assignees
@EuroLew
Labels
chaos-day Marks an issue or PR as related to an experiment performed during a chaos day component/zeebe Related to the Zeebe component/team kind/bug Categorizes an issue or PR as a bug severity/high Marks a bug as having a noticeable impact on the user with no known workaround version:8.7.0-alpha3 version:8.8.0-alpha1 Marks an issue as being completely or in parts released in 8.8.0-alpha1

Comments

@ChrisKujawa
Copy link
Member

Describe the bug

When running a benchmark, we observed that the Camunda Exporter failed after restart, because the Importer doesn't seem to be completed (where it should be).

INFO 2024-12-12T21:20:50.478899335Z Waiting for importers to finish, cached record with key 2251799814997670 but did not flush

The importer seems to not complete all documents.

import-completed
@ChrisKujawa ChrisKujawa added kind/bug Categorizes an issue or PR as a bug component/zeebe Related to the Zeebe component/team labels Dec 13, 2024
@ChrisKujawa ChrisKujawa added the severity/high Marks a bug as having a noticeable impact on the user with no known workaround label Dec 13, 2024
@ChrisKujawa ChrisKujawa mentioned this issue Dec 13, 2024
Closed
@ChrisKujawa
Copy link
Member Author

https://zeebe-io.github.io/zeebe-chaos/2024/12/12/News-from-Camunda-Exporter-project

@ChrisKujawa ChrisKujawa added the chaos-day Marks an issue or PR as related to an experiment performed during a chaos day label Dec 13, 2024
@ChrisKujawa ChrisKujawa mentioned this issue Dec 13, 2024
Open
@EuroLew EuroLew mentioned this issue Dec 16, 2024
Merged
1 task
github-merge-queue bot pushed a commit that referenced this issue Dec 16, 2024
@ChrisKujawa
Importers not communicated completion (#26124)
61079ca 
## Description

The importers were not correctly marking their import position documents
as completed to signify the record readers were done. This was due to
missing zeebe indices causing a `NoSuchIndex` exception which means the
path through which record reader completion was checked and set was
missed.

The fix involves treating a no such index exception as an empty batch
with regards to record reader completion check as they signify the same
thing (no records to export).

## Checklist

- [x] Given missing zeebe indices record readers can still get marked as
completed.

## Related issues

closes #26046
@michaldytko
Copy link

@ChrisKujawa I can still reproduce it with 8.7.0-alpha3. I'm using docker-compose with CamundaExporter for Zeebe.

@EuroLew
Copy link
Contributor

EuroLew commented Feb 4, 2025

@ChrisKujawa I can still reproduce it with 8.7.0-alpha3. I'm using docker-compose with CamundaExporter for Zeebe.

Can you give me the exact steps to reproduce? Is it consistent or non-deterministic?

@michaldytko
Copy link

Hi @EuroLew, I can reliably reproduce it. I used docker-compose to start alpha3 with CamundaExporter. The issue occurs every time I restart the whole docker compose.

@ChrisKujawa
Copy link
Member Author

@EuroLew please take a look at this, this week.

As discussed here https://camunda.slack.com/archives/C06F0GLJNFM/p1738926629778839?thread_ts=1738775414.411889&cid=C06F0GLJNFM

We agreed on the following: we should do the check always (except if it is disabled manually), we are checking for importers completed and additionally, whether there are any Zeebe ES indices at all (because if not we don't need to wait for importers). If no Zeebe ES indices, we can safely assume a green field installation (or that we already catched up to the end of the log) and start.

@ChrisKujawa ChrisKujawa reopened this Feb 10, 2025
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes added the version:8.8.0-alpha1 Marks an issue as being completely or in parts released in 8.8.0-alpha1 label Feb 11, 2025
@EuroLew
Copy link
Contributor

EuroLew commented Feb 12, 2025

@EuroLew please take a look at this, this week.

As discussed here https://camunda.slack.com/archives/C06F0GLJNFM/p1738926629778839?thread_ts=1738775414.411889&cid=C06F0GLJNFM

We agreed on the following: we should do the check always (except if it is disabled manually), we are checking for importers completed and additionally, whether there are any Zeebe ES indices at all (because if not we don't need to wait for importers). If no Zeebe ES indices, we can safely assume a green field installation (or that we already catched up to the end of the log) and start.

will do

@EuroLew EuroLew mentioned this issue Feb 13, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 17, 2025
@EuroLew
Camunda exporter startup bug (#28105)
e46c010 
## Description

There is a bug where the camunda exporter can enter an error loop in a
new cluster where it is waiting for importers to complete which never
will (as importers will not do any work in 8.8). This fix will as per
the slack thread referred to here
#26046 (comment),
assume that a lack of existence of zeebe indices (which the importer
reads from), means that it is a fresh cluster, thus the exporter does
not need to wait and we avoid the error loop.

Questions
- There is a requirement that the new `zeebeIndexPrefix` match the zeebe
index prefix defined for the ES exporter, I have documented this in the
yaml but I feel like this should be explicitly stated elsewhere
otherwise the 8.8 camunda exporter can enter boot loops. (Although it
will work work with default configuration).
- Would this need to get backported for people running benchmarks? (to
which version?)

## Related issues

relates to #26046
@ChrisKujawa
Copy link
Member Author

Should be resolved with #28105

@ChrisKujawa ChrisKujawa reopened this Feb 28, 2025
@ChrisKujawa
Copy link
Member Author

Actually we need to validate the version of the Zeebe indices, like 8.7. IF 8.8 only is in the zeebe indices, then it is still fine to assume greenfield.

@ChrisKujawa
Copy link
Member Author

Will be covered with #28833

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EuroLew EuroLew

Labels
chaos-day Marks an issue or PR as related to an experiment performed during a chaos day component/zeebe Related to the Zeebe component/team kind/bug Categorizes an issue or PR as a bug severity/high Marks a bug as having a noticeable impact on the user with no known workaround version:8.7.0-alpha3 version:8.8.0-alpha1 Marks an issue as being completely or in parts released in 8.8.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Importers not communicated completion camunda/camunda
5 participants
@ChrisKujawa @camundait @michaldytko @rodrigo-lourenco-lopes @EuroLew