slow-running queries

[iburunat]

When running helm-do-ag on a folder with many org files (~5k files, not code, but text), I am experiencing long retrieval times (in the order of 4 minutes).

Usually it retrieves one result very quickly. After 4-5 minutes I see the rest.

Is this behaviour normal?
Thanks for your feedback!

[toothbrush]

Hey @iburunat, i just had the same issue. I went digging and for me the issue was that helm-ag actually will rewrite a query such as module Statsd into (?=.*module.*)(?=.*Statsd.*) (depending on some circumstances). I found that out by noticing my CPU being pegged during a search and having a look at what ps aux said my Emacs was actually executing. By poking around in the helm-ag source, i found that in some circumstances this rewrite gets done by helm-ag--join-patterns:

helm-ag/helm-ag.el

Lines 989 to 996 in c16d7a7

	(cond ((memq 'pcre helm-ag--command-features)
	(cl-loop for s in patterns
	if (helm-ag--convert-invert-pattern s)
	concat (concat "(?=" it ")")
	else
	concat (concat "(?=.*" s ".*)")))
	((memq 're2 helm-ag--command-features)
	(string-join patterns ".*"))

My first attempt was to just use Ripgrep, because that rewrite isn't done for Ripgrep as mentioned in the README:

helm-ag/README.md

Lines 279 to 285 in c16d7a7

	#### NOTE: For pt and rg users
	When using `ag` or `ack`, `helm-do-ag` convert pattern from `foo bar` to `"(?=.*" foo ".*)(?=.*" bar ".*)"`
	which pattern matches line which contains both `foo` and `bar`. But when using `pt` or `rg`, `helm-do-ag`
	does not convert the pattern because Golang `regexp`(`pt` is written in Golang)
	and rust's `regex` (`rg` is written in rust) does not support look-a-head pattern.
	So using `pt` or `rg` behaves differently from `ag` when you use such pattern.

However i failed to get helm-ag correctly configured to use ripgrep (i guess that's a separate issue - rg works well for me on the CLI).

My next idea was to convince helm-ag to treat ag as if it has a "basic" RE2 capacity rather than PCRE with lookahead, which is extremely slow in some cases. I did that by looking at what helm-ag--set-command-features does and overriding the default helm-ag--command-features for Ag (which is '(pcre ag)) with '(re2 ag) to trigger a different code path in helm-ag--join-patterns. I also had to override the helm-ag--set-command-features function with a no-op so that it doesn't override my tweaks.

This is extremely hacky but in the absence of a knob to tweak helm-ag's behaviour, this seemed the easiest way to unblock myself. Hope it helps you! Here's my final init.el snippet:

(use-package helm-ag
  :config
  (setq helm-ag--command-features '(re2 ag))

  (defun helm-ag--set-command-features ()
    "Intentional no-op."))