Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New computations is starving in init stage #1637

Closed
renjiezh opened this issue Jun 1, 2024 · 2 comments
Closed

New computations is starving in init stage #1637

renjiezh opened this issue Jun 1, 2024 · 2 comments
Labels
bug Something isn't working
Milestone

Comments

@renjiezh
Copy link
Contributor

renjiezh commented Jun 1, 2024

Describe the bug
While duchy mills are busy on computing, new computations will stuck in init stage until all old computations are complete. This prevent EDPs from fulfilling requisitions of new computations.

Steps to reproduce

  1. Create a number of computations so that all mills are occupied .
  2. Wait until all computations are in computing stage. Create another batch of computations.

Component(s) affected
Duchy, EDP

Version
v0.5.5

Environment
halo-cmm-qa

Additional context
According to experiments, new computations only stuck at aggregator but not at worker 1.

@renjiezh renjiezh added the bug Something isn't working label Jun 1, 2024
@renjiezh
Copy link
Contributor Author

renjiezh commented Jun 3, 2024

A workaround is to force mills to override the Computation order and pick up init Computations by changing the creation time.

Use the spanner sql query against all Duchies:

update Computations
set CreationTime = TIMESTAMP_SUB(CreationTime, INTERVAL 3650 DAY)
where CreationTime > "2024-05-31T17:47:00.000000Z" and ComputationStage = 1

The where clause should be modified accordingly to help select the new computations.

To revert the modification of CreationTime:

update Computations
set CreationTime = TIMESTAMP_ADD(CreationTime, INTERVAL 3650 DAY)
where CreationTime <= "2015-01-01T00:00:00.000000Z" and ComputationStage != 1

@renjiezh
Copy link
Contributor Author

renjiezh commented Aug 6, 2024

Resolved by #1673

@renjiezh renjiezh closed this as completed Aug 6, 2024
@SanjayVas SanjayVas added this to the v0.5.7 milestone Aug 6, 2024
ple13 pushed a commit that referenced this issue Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants