[ML] GET _transform only returns 100

[sophiec20]

Found in 7.2.0-BC5

GET _data_frame/transforms/_all will only return up to 100 documents.

GET _data_frame/transforms/_all?size=200 will return up to 200 documents.

If I had 105 data frame transforms, and I ran GET _data_frame/transforms/_all without specifying the size, then unfortunately the response does not give any indication that I need to page through the results. As a consequence, the UI only ever supports 100 transforms.

There is a hard-coded defensive upper limit for 10k data frame transforms (same as for anomaly detection). Consider how we can improve usability for cases where there are more than 100 transforms.

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

A couple of thoughts:

• The UI should support pagination (made note in the related issue)
• We should return the total count of the transforms that match the id pattern supplied. The count field in the response is actually supposed to be the total number of IDs matched, not the length of the array of objects (if code comments are to be believed) https://github.com/elastic/elasticsearch/blob/ebd6997da4acb79105c89b3957903d0f0598338a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/util/QueryPage.java#L22..L26

[droberts195]

• We should return the total count of the transforms that match the id pattern supplied

Yes, agreed, this is how to solve the problem of knowing whether it's worth trying to get another page.

[peteharverson]

Note that the data frame _stats endpoint also uses a default size of 100:

GET _data_frame/transforms/_stats
GET _data_frame/transforms/_all/_stats

For a system with over 100 transform stats docs, both the above return 100 docs, with count returned as 100.

[benwtrent]

Cool, I will get a PR together for fixing this behavior in 7.3. Returning the total count of docs makes sense :)

[benwtrent]

#43241 <- PR so that the count is the total count given the id pattern.

[droberts195]

#43241 fixed many of the problems, but I don't think it fixed the HLRC entirely. GetDataFrameTransformStatsResponse in the HLRC still assumes that count is the number of hits returned. However, please don't work on this immediately because #44350 is making major changes to the stats response and any fix to the reporting of count will clash horribly.

[hendrikmuhs]

#62809 fixed HLRC, I wonder if this issue can be closed now.

@elastic/ml-qa would one of you be able to verify that we haven't missed anything? If so, we can close this issue.