Help with slow logs getting cut off

[nik9000]

One of our key features for debugging is the slow log - but when folks have very large _searches we can end up with cut off slow logs. Something chops them. I recall that we had something to do that but I can't find it at the moment. Maybe its some external tool doing the chopping

Anyway! Maybe we could make the slow logs shorter by not outputting default parameters when we toXContent the search's source. Or something. I dunno. I've just been frustrated lately when looking at slow logs from folks that are cut off and want to do something about it.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[pgomulka]

@nik9000 I guess you are talking about plaintext slow logs in version before 7.x? or the config is stale?
In versions <7.0 we were truncating size of the message to 1000
#36833

for indexing slow log we still truncate and it is configurable by index.indexing.slowlog.source

can you confirm the version and the provide the logging config?

[jpountz]

We discussed it in a fix-it meeting. Having requests cut off in the slowlog is a big deal for troubleshooting production issues, and even if we had it entirely by raising the maximum length of a slowlog entry, the fact that it's so verbose makes it hard to read by a human. So the preferred route would be to stop serializing parameters when they have the default value.

We couldn't think of a component that would break if we stopped serializing default values. @nik9000 will try out by no longer serializing the size parameter of search requests to see if it breaks anything. In the case where some other components rely on having default values, we could still rely on the context that is passed to xcontent serialization so that default values would only be omitted when serializing requests to the slowlog.

[nik9000]

@nik9000 will try out by no longer serializing the size parameter of search requests to see if it breaks anything. In the case where some other components rely on having default values, we could still rely on the context that is passed to xcontent serialization so that default values would only be omitted when serializing requests to the slowlog.

It looks like it's been omitting the default size for a long, long time. So I picked a common query, match, and tried to omit it's defaults. That's #83338.