k=100K, but much less is returned

[sanikolaev]

Bug Description:

Take this file:

snikolaev@dev2:~$ ls -lah /home/snikolaev/lisdocument1.sql.gz
-rw-rw-r-- 1 snikolaev snikolaev 362M Dec  8 18:02 /home/snikolaev/lisdocument1.sql.gz

Load 100K docs with embeddings to Manticore:

snikolaev@dev2:/tmp$ time gzip -cd lisdocument1.sql.gz | mysql -P9306 -h0

real	0m53.528s
user	0m21.327s
sys	0m1.000s

There're 2 disk chunks + a ram chunk now. Merge to one disk chunk:

mysql> flush ramchunk lisdocument1;
mysql> optimize table lisdocument1 option cutoff=1, sync=1;

And the following query now returns ~80K docs even though the k is 100K and all the limits are also set to 100K and the cutoff is off. And sometimes it returns significantly less:

snikolaev@dev2:/tmp$ for n in `seq 1 10`; do gzip -cd lisdocument1.sql.gz | mysql -P9306 -h0; mysql -P9306 -h0 -e "flush ramchunk lisdocument1; optimize table lisdocument1 option sync=1, cutoff=1; SELECT count(*) FROM lisdocument1 WHERE  knn (vector, 100000, (0.31006125,-0.0003656157,0.081459515,0.515771,0.014484076,-0.66133505,0.32237425,-0.55777,-0.386141,-0.023305604,0.11610652,-0.67190164,-0.08708035,0.08902938,-0.6518018,-0.051225036,-1.1461623,-0.41078997,0.17181408,-0.4735955,0.03203327,0.23325141,-0.9909842,-0.3320622,0.201117,0.6546775,-0.7582774,0.9626067,0.7869749,0.7003234,0.5161592,-0.47969586,1.1994319,0.28759456,-0.26597247,-0.22219379,0.33459783,-0.97726834,-0.4739887,0.43206725,0.01194832,-0.05791433,0.4811258,-0.8271685,0.3951732,0.24713722,0.11192199,1.0213652,0.19014879,-0.6778019,0.50749946,-1.049119,0.4575615,0.9339654,0.58290714,-0.097028166,-0.48701167,-1.0065265,-0.13599022,-0.40211567,-0.5544309,0.1297585,0.19523889,-0.77102447,0.64259815,-0.30415413,-0.23215108,0.67778605,0.618211,0.7600791,0.23732586,-0.7648567,0.062301278,0.13431896,0.66374356,0.40523198,0.46663165,0.51943946,-0.73183507,-0.57254523,0.50741565,-0.14492488,-0.026737308,-0.63081896,-0.041068904,0.43559575,0.1266691,0.68689376,0.013397118,0.53886193,0.20262197,0.29941437,-0.6999152,0.28189236,0.9438162,0.6245068,-0.2915007,-0.08586646,-0.39370182,0.6360048,-0.38291323,0.7843485,-0.54510206,-0.52077717,1.2116083,-0.12180229,0.13172503,-0.10871028,0.41399533,-0.47680008,0.9621055,-0.54643613,0.21518388,-0.54834914,-0.1906337,-0.09179794,-0.590677,-0.52437264,-0.7151974,-0.02695119,0.21971804,-0.024934726,0.05605529,0.44665658,-0.12020654,0.26943594,-0.10920261,0.20266409,0.012066222,0.19158714,-0.12457689,-0.4856409,0.16108908,-0.20201308,0.041395552,-0.34803805,-1.1102498,-0.74369335,-1.6364297,1.0657281,0.07174268,0.010433137,0.7891052,0.60267335,-0.2899389,0.4021704,-0.20868495,-0.003395542,-0.05169468,-0.17775702,0.21387385,0.0991693,-0.9152813,0.19724046,0.35551172,-0.4999967,-0.8879677,0.16530086,-1.1837602,-0.59362537,0.12462508,-0.34771776,0.9998891,0.8668287,-0.6882197,0.2541907,0.49837124,0.15233028,0.336384,-0.038960453,-0.7221209,-0.071910635,0.04892005,-0.21825159,0.16531892,-0.8357419,0.5070645,0.1569018,0.313003,-0.17091025,-0.33268282,1.1826873,0.24373363,-0.15638427,0.27422035,0.39059705,-0.34179664,-0.35423273,-0.7033785,0.21909295,1.1341084,0.27519116,1.1161767,-0.56932676,-0.9429059,0.40243575,-0.5576759,-0.21010822,-0.4341065,1.2042234,-0.82608193,0.4289299,-0.14796726,-0.027835188,-0.26135963,-0.5276095,-0.88423705,0.04380098,-0.43928683,0.41062203,-0.34576598,0.13919765,-0.01392624,-0.5553707,-0.45019755,0.91757697,0.4029668,0.1176647,-0.15419589,0.73095477,0.40260544,0.43138027,0.21548833,-0.5267257,-0.30721363,-0.25096813,-0.6048828,0.7789119,0.15235308,0.17003383,0.8087661,-0.38634503,-0.29575384,0.3041153,-0.0025674396,-0.12190996,0.5652693,0.123609096,-0.41633,-0.075612135,0.5379093,0.47812945,-0.7582835,0.48872802,0.17062183,-0.41053674,0.24030015,0.43435127,0.40152794,0.006724952,-0.32310417,-0.35114315,-0.37304744,-0.53361934,-0.14794493,0.5957107,0.48854348,0.22275516,0.28737432,-0.55769044,0.19703211,-0.43264365,0.70282584,0.8103258,0.4166922,0.18818198,-0.6881402,-0.8271788,0.33227536,0.44830915,-0.21777764,-0.3315875,-0.18762241,-0.5730435,1.2522116,0.05436271,0.5053515,0.08252778,-0.7671772,-0.67160285,0.7309791,-0.6450694,-0.23141778,0.081859656,-0.31854483,-0.66024923,0.14364149,0.30650187,0.14491636,0.09296628,0.29341727,-0.28662416,-0.8733087,0.73710185,0.6832434,-1.0337256,-0.36529627,-1.185272,0.70611227,-0.21201886,-0.4519033,0.35380524,0.24411415,0.7839299,-1.0546187,-0.5108705,-0.040125087,0.0038510442,0.16296358,0.14024405,-0.29560274,-0.36276716,1.3071771,0.086237356,-0.791835,0.11178309,-0.12588637,-0.4115165,0.30388063,-0.51085454,-0.090283126,-1.5626551,0.53435034,-0.22315815,-0.31446335,0.48053166,0.64726245,-1.073468,0.56286776,0.10031773,0.39174485,0.5910364,-0.27479896,-0.25149876,-0.064312816,-0.07417552,0.8132626,0.07798847,0.25687593,-0.60261244,-0.45493522,0.54915696,-0.053107496,-1.2380457,-0.21066967,-0.1940282,-0.43841827,0.17368084,-0.14619267,-0.2789112,0.2749565,0.4712215,0.3926232,-0.32814994,-1.0122043,-1.0696679,-0.16326195,-0.975733,0.39950857,0.19773667,-0.16035733,-0.058070622,-0.118425414,0.18218455,0.6603359,0.54957753,0.44137517,0.21397805,-0.22320518,-0.91974175,0.48958048,-0.084736824,-0.426293,0.8708147,-0.2713833,-0.39039102,0.30332446,-0.7585659,-0.12065592,-0.15349118,0.8288996,0.23025775,-0.35495567,0.21538107,0.86950296,-0.00998487,0.0017167316,0.120606296,0.4414424,0.4505997,0.829459,0.7318415,-0.8386215,0.2715878,0.65265125,-0.07971094,0.48587146,-0.5186518,0.55407375,0.2426369,0.21575326,-0.044891708,-1.2336338,-0.10642625,-0.5177275,-0.6177509,0.8570399,1.043228,-0.2713039,0.29890174,-0.47719136,0.09048428,0.81275654,0.22632912,-0.27022663,0.10290417,0.4140533,0.17572246,0.48987168,0.6703758,-0.46391836,0.075863644,15.550138,0.62152815,-0.04971984,0.10588257,0.17923146,0.15616268,-0.46352684,-0.08553317,-0.8809511,-0.13167156,0.88224876,-0.1254104,0.08551519,0.41633347,-0.39136684,-0.5745892,0.059034057,0.27425095,0.6071214,-1.021161,0.40765795,0.41800106,-0.2145601,0.020918911,0.60376966,0.48886603,0.23570696,-0.07289946,0.7897558,-0.38197488,0.8834917,-0.48737484,0.72564256,1.0258603,-0.7900168,-0.1589608,-0.5894068,-1.2211249,0.21977632,-0.032760903,-0.73974997,0.027299361,-0.40298098,-0.59684885,0.18560478,0.22460128,-0.25463927,0.6956338,0.04916962,-0.05426601,0.4170774,0.4209522,0.69340205,-0.5526944,0.13108079,0.028241986,0.27075502,0.058803745,-0.12544855,-0.6035113,-1.2496368,-0.6891652,-0.21734844,0.88274825,0.21550162,0.900132,0.23280652,0.022997925,0.070107326,0.2929837,-0.9167534,-0.44679806,0.6751843,0.26168746,-0.9102508,-0.53386253,-0.11613186,-0.33997464,-0.65545017,0.34133926,0.5415031,-0.8402537,-0.4420601,0.9791182,0.3558302,-1.3223187,0.58805984,-0.9981405,-1.0075716,-0.10522475,-0.4694484,-0.88297,-0.073731676,-0.40783596,-0.44489437,-0.35490802,0.5813389,-0.48236555,-0.8025031,0.14774305,-0.07733792,-0.04009475,-0.19666764,-0.13940135,0.56877077,0.89890414,-0.34569648,0.79776573,-0.091007896,-0.69942564,-1.1672308,0.30055726,-0.84184074,-1.0750866,0.53956366,-0.9331158,-0.29834008,0.0052645435,0.44171295,-0.4195513,0.8877479,-0.058045257,0.05742934,-0.16210446,-1.0204247,0.44277644,0.13323916,-1.1805303,-0.21072462,0.12990586,-0.044168193,-1.2782251,-0.25332808,0.40048394,0.2744026,-0.6884139,0.75036937,-0.8379228,0.1622562,0.1323186,-0.027955009,-0.42298132,-0.17523514,-0.64205575,0.44626942,0.11984201,0.61406684,-0.5398223,0.13014658,1.4530864,0.072602026,-0.34999475,-0.7632572,-0.3052821,-0.3685159,-0.30019602,-0.26202905,0.065592654,0.1356203,-0.23066288,-0.06168315,1.18814,0.18889207,-0.496132,0.71889156,0.3918985,-0.1458961,0.12719785,-0.8727122,-0.33598074,-0.20535125,-0.48016322,-0.40368298,-0.5138646,-0.62812066,0.455146,-0.23726645,-0.35210988,0.6264089,-0.35540456,-0.8420332,-0.4773195,-0.043095402,0.5031531,0.31352416,-0.56513035,0.6253261,-0.22605664,-0.022685153,0.56507564,0.13309203,-0.2219282,-0.74726635,-1.0282854,0.097224504,-0.030685902,-0.053559277,-1.3350763,0.6840824,-0.21796289,0.54228777,-0.1751819,0.185303,-1.3930091,0.28261814,0.55813426,0.058485728,-0.2205351,0.5061636,-0.45415583,-0.41728288,0.21399207,0.10460134,-0.5251125,-0.05460708,0.12028494,-0.8888363,-0.5616349,1.0626166,-0.57732487,-0.71718526,-0.3703574,0.18429625,0.5347522,0.20444115,0.57214594,0.37266445,-1.2059611,0.8612643,0.48821098,-0.43664938,-0.82340217,0.67264736,0.22491321,-0.74951804,0.0871341,-0.27833295,0.46309146,-0.78231233,-0.21181802,0.42298067,-0.07849596,0.5937467,1.0871675,0.019744089,-0.7014219,0.20075509,-0.64872456,-0.6484867,-0.48085892,0.7048896,0.37797463,-0.19823772,0.009384685,-0.44905084,0.47751105,0.07683447,-0.7515542,-0.14501087,0.17405201,0.17612083,0.54188806,-0.2535289,0.3439073,-0.6030962,0.38111544,-0.651172,-0.96390456,-0.030525658,-0.26229542,-0.07045684,0.28682828,0.4695437,0.48653972,-0.021991393,-0.19629684,-0.104265794,0.72771174,1.1777867,-0.42338756,-0.28041443,0.9536649,1.531987,-0.29626778,0.4382578,-0.715345,-0.8170925,0.94427216,1.243451,0.17655466,0.62224877,0.20158736,0.16209148,0.09094697,0.054510932,0.18709274,0.43218136,0.750771,0.52881354,-0.05870044,0.15825632,0.85584885,0.435063,0.10643143,0.95394486,1.6185488,-0.46994826,-0.0854546,-0.5466821,0.57927006,-0.428444,0.26448515,-0.6947298,1.218788,-0.14955972,0.18937686,0.12341008,-1.0414394,1.0318452,0.18007237,0.32144284,-0.41614947,-0.08107099,-0.24519788,-0.10084439,-0.5396372,0.36262137,-0.26675776,-0.21520412,0.14965774,-0.058672633,0.50249416,0.24154651,0.49916118,0.3448778,0.12129112,0.23797168,-1.1313815,-0.55008906,-0.7546291,-0.34192833,-1.1170193,-0.27139792,-1.0538054,-0.16249998,-0.90182513)) limit 100000 OPTION cutoff = 0, max_matches = 100000;"; done;

+----------+
| count(*) |
+----------+
|    82176 |
+----------+
+----------+
| count(*) |
+----------+
|    80652 |
+----------+
+----------+
| count(*) |
+----------+
|       79 |
+----------+
+----------+
| count(*) |
+----------+
|    82176 |
+----------+
+----------+
| count(*) |
+----------+
|    81921 |
+----------+
+----------+
| count(*) |
+----------+
|    82176 |
+----------+
+----------+
| count(*) |
+----------+
|     1224 |
+----------+
+----------+
| count(*) |
+----------+
|    82176 |
+----------+
+----------+
| count(*) |
+----------+
|    82176 |
+----------+
+----------+
| count(*) |
+----------+
|    82176 |
+----------+

Notes

• Reducing the docs count to 10K solves both issues.

Manticore Search Version:

Manticore 6.3.9 dd29aca47@24120423 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219) (knn 2.3.1 edadc69@24112219)

Operating System Version:

Ubuntu Jammy (dev2)

Have you tried the latest development version?

No

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated