Skip to content

Add translog files age to Translog Stats (#28613) #28613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 16, 2018
Merged

Add translog files age to Translog Stats (#28613) #28613

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
948614e
#28189 Implemented earliest_generation_last_modified_age as a translo…
justinwyer Feb 9, 2018
056710a
changes per review
justinwyer Feb 12, 2018
e3586d1
Merge branch 'master' of https://github.com/elastic/elasticsearch int…
justinwyer Feb 16, 2018
4066b42
changes per second review
justinwyer Feb 16, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions rest-api-spec/src/main/resources/rest-api-spec/test/indices.stats/20_translog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ setup:
# non empty generation with one op may be smaller or larger than that.
# - gt: { indices.test.primaries.translog.uncommitted_size_in_bytes: $creation_size }
- match: { indices.test.primaries.translog.uncommitted_operations: 1 }
- gte: { indices.test.primaries.translog.earliest_last_modified_age: 0 }

- do:
indices.flush:
Expand All @@ -46,6 +47,7 @@ setup:
## creation translog size has some overhead due to an initial empty generation that will be trimmed later
- lt: { indices.test.primaries.translog.uncommitted_size_in_bytes: $creation_size }
- match: { indices.test.primaries.translog.uncommitted_operations: 0 }
- gte: { indices.test.primaries.translog.earliest_last_modified_age: 0 }

- do:
indices.put_settings:
Expand All @@ -67,3 +69,4 @@ setup:
- match: { indices.test.primaries.translog.operations: 0 }
- lte: { indices.test.primaries.translog.uncommitted_size_in_bytes: $creation_size }
- match: { indices.test.primaries.translog.uncommitted_operations: 0 }
- gte: { indices.test.primaries.translog.earliest_last_modified_age: 0 }
2 changes: 1 addition & 1 deletion server/src/main/java/org/elasticsearch/index/translog/Translog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ static long findEarliestLastModifiedAge(long currentTime, Iterable<TranslogReade
for (BaseTranslogReader r : readers) {
earliestTime = Math.min(r.getLastModifiedTime(), earliestTime);
}
return currentTime - Math.min(earliestTime, writer.getLastModifiedTime());
return Math.max(0, currentTime - Math.min(earliestTime, writer.getLastModifiedTime()));
}

/**
Expand Down
20 changes: 12 additions & 8 deletions server/src/test/java/org/elasticsearch/index/translog/TranslogTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,9 +369,16 @@ protected TranslogStats stats() throws IOException {
}

public void testFindEarliestLastModifiedAge() throws IOException {
long fixedTime = System.currentTimeMillis();
long[] periods = new long[10];
for (int i = 0; i < 9; i++) {
final int numberOfReaders = scaledRandomIntBetween(1, 10);
Copy link
Contributor

@bleskes bleskes Feb 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make this go down to 0?

Copy link
Contributor Author

@justinwyer justinwyer Feb 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explicitly test the 0 readers case in that test as well as a random number of readers between 1 and 10 I can do the change of you prefer however.

Copy link
Contributor

@bleskes bleskes Feb 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I see. I missed it.

long fixedTime = randomLongBetween(0, 10000000000000000L);
long[] periods = new long[numberOfReaders + 1];
long period = randomLongBetween(10000, 1000000);
periods[numberOfReaders] = period;
TranslogWriter w = mock(TranslogWriter.class);
stub(w.getLastModifiedTime()).toReturn(fixedTime - period);
assertThat(Translog.findEarliestLastModifiedAge(fixedTime, new ArrayList<>(), w), equalTo(period));

for (int i = 0; i < numberOfReaders; i++) {
periods[i] = randomLongBetween(10000, 1000000);
}
List<TranslogReader> readers = new ArrayList<>();
Expand All @@ -380,11 +387,8 @@ public void testFindEarliestLastModifiedAge() throws IOException {
stub(r.getLastModifiedTime()).toReturn(fixedTime - l);
readers.add(r);
}
long period = randomLongBetween(10000, 1000000);
periods[9] = period;
TranslogWriter w = mock(TranslogWriter.class);
stub(w.getLastModifiedTime()).toReturn(fixedTime - period);
assertThat(Translog.findEarliestLastModifiedAge(fixedTime, readers, w), equalTo(LongStream.of(periods).max().orElse(0L)));
assertThat(Translog.findEarliestLastModifiedAge(fixedTime, readers, w), equalTo
(LongStream.of(periods).max().orElse(0L)));
}

public void testStats() throws IOException {
Expand Down