Make normalizeLimit configurable instead of hard-coded

[vipulmathur]

Experience Report

While using the @normalize directive in queries, I get the error message: "Couldn't evaluate @normalize directive - to many results" (note that there seems to be a typo there: should be "too many results"). I need to get normalized results that are more in number than the hard-coded limit of 10000: https://github.com/dgraph-io/dgraph/blob/master/query/outputnode.go#L38

I understand that there may be a need to impose a limit, but like other limits, this should be made configurable too.

What you wanted to do

Get normalized results, when the number of results is greater than 10000.

What you actually did

Ran the query with @normalize and got the error above. Ran the query without @normalize and got the results (about 14K in number, but without the normalization of course).

Why that wasn't great, with examples

The query works without normalize, and returns about 14K items. However, adding the @normalize directive throws the error mentioned above. I am okay with the query taking a bit longer with normalization but currently the limit is hard-coded.

When using the @normalize directive, it is not always possible to ensure that the number of results will always be less than 10000. How does one ensure that? Is that documented somewhere? The forum post I refer to below seems to suggest that pagination + offset will help but it did not help the person in the forum post.

Any external references to support your case

This forum post seems to talk about the same issue. https://discuss.dgraph.io/t/limit-on-normalize/3118

I brought this up on Twitter and was asked to open a feature request: https://twitter.com/dgraphlabs/status/1122149007902425089

[martinmr]

Thanks for pointing out the typo. PR to fix it: #3350

I'll work on making these parameter configurable.

[vipulmathur]

@martinmr any estimate on when the configurable parameter might land?

Could someone please explain how increasing this number impacts Dgraph? That way we will be able to work with a manual change to the number. I want to know what range is safe to use in production.

[martinmr]

sorry for the delay. I have some cycles today so I'll try to get it in today. It should be included starting with version 1.0.15.

With respect to the performance impact we haven't done much testing. I'd say if your query without the normalize keyword works fine. there shouldn't be too much trouble. I'd start by setting the value to 15K and work up from there.