Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only some ignores applied when searching a sub-directory #1757

Closed
SimonSapin opened this issue Dec 8, 2020 · 2 comments
Closed

Only some ignores applied when searching a sub-directory #1757

SimonSapin opened this issue Dec 8, 2020 · 2 comments
Labels
bug A bug. gitignore Bugs related to gitignore problems.

Comments

@SimonSapin
Copy link

What version of ripgrep are you using?

ripgrep 12.1.1
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

How did you install ripgrep?

System package https://www.archlinux.org/packages/community/x86_64/ripgrep/

What operating system are you using ripgrep on?

Arch Linux

Describe your bug.

When giving a sub-directory of the current directory to search as a second command-line argument to ripgrep, only some patterns of .ignore in the current directory are applied.

What are the steps to reproduce the behavior?

mkdir tmp
cd tmp
mkdir rust
mkdir rust/target
echo needle > rust/source.rs
echo needle > rust/target/rustdoc-output.html
echo rust/target > .ignore
rg --files-with-matches needle
rg --files-with-matches needle rust
echo target >> .ignore
rg --files-with-matches needle rust

What is the actual behavior?

The first and third rg commands only print rust/source.rs as expected. However the second one prints both rust/source.rs and rust/target/rustdoc-output.html.

With --debug added, and -j1 for more readable output:

$ rg --files-with-matches needle --debug -j1
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(needle)], limit_size: 250, limit_class: 10 }
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 1 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1730: ignoring ./.ignore: Ignore(IgnoreMatch(Hidden))
rust/source.rs
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1730: ignoring ./rust/target: Ignore(IgnoreMatch(Gitignore(Glob { from: Some("./.ignore"), original: "rust/target", actual: "rust/target", is_whitelist: false, is_only_dir: false })))

$ rg --files-with-matches needle rust --debug -j1
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(needle)], limit_size: 250, limit_class: 10 }
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 1 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
rust/source.rs
rust/target/rustdoc-output.html

After echo target >> .ignore:

$ rg --files-with-matches needle rust --debug -j1
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(needle)], limit_size: 250, limit_class: 10 }
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 1 literals, 1 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
rust/source.rs
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1730: ignoring rust/target: Ignore(IgnoreMatch(Gitignore(Glob { from: Some("/tmp/tmp/.ignore"), original: "target", actual: "**/target", is_whitelist: false, is_only_dir: false })))

What is the expected behavior?

Each command should only print rust/source.rs, as the HTML file should be ignored.

One might imagine that when searching a sub-directory only .ignore files under that sub-directory are considered, but the third command shows that is not the case. (The more general ignore pattern is not the desired one, it only serves to show that the .ignore file is indeed read.)

Work-around

I’ve managed to find a set up that does what I want by creating a .ignore file in the rust sub-directory of the project instead, with a /target pattern.

Other suggestion

After trying a few things and seeing unexpected results I wanted to find out details of the syntax of .ignore files. For example, is an initial slash meaningful?

I had to dig to find the answer in https://docs.rs/ignore/0.4.17/ignore/struct.WalkBuilder.html:

.ignore files have the same semantics as gitignore files

and

.gitignore files have match semantics as described in the gitignore man page.

It would be good to repeat these two facts in ripgrep’s own docs.

@BurntSushi BurntSushi added the bug A bug. label Dec 9, 2020
@goto-engineering
Copy link

goto-engineering commented Dec 10, 2020

This works for me with rg 12.1.1 on macOS:
image

edit: nevermind, I realized that you're fixing the problem as part of the recreate steps. If I only execute the first 2 rg commands, I'm getting the same thing.

@SimonSapin
Copy link
Author

Ah yes, sorry, I should have separated that last step some more.

To me it’s not really a fix since it also ignore directories or files named target at any other location, which is not necessarily desirable. I only included it to show that .ignore in the current directory is read at all when searching a sub-directory. (It might make some sense if it weren’t even though that’s nat what I’d want in this case.)

sengaya added a commit to sengaya/ripgrep that referenced this issue Aug 29, 2022
When searching subdirectories the path was not correctly build and
included duplicate parts. This fix will remove the duplicate part if
possible. It also contains a small optimization that will skip the
second for loop if there is already a match.

Fixes BurntSushi#1757
@BurntSushi BurntSushi added the gitignore Bugs related to gitignore problems. label Jul 8, 2023
sengaya added a commit to sengaya/ripgrep that referenced this issue Jul 14, 2023
When searching subdirectories the path was not correctly build and
included duplicate parts. This fix will remove the duplicate part if
possible. It also contains a small optimization that will skip the
second for loop if there is already a match.

Fixes BurntSushi#1757
BurntSushi pushed a commit that referenced this issue Sep 20, 2023
When searching subdirectories the path was not correctly built and
included duplicate parts. This fix will remove the duplicate part if
possible.

Fixes #1757, Closes #2295
ink-splatters pushed a commit to ink-splatters/ripgrep that referenced this issue Oct 25, 2023
When searching subdirectories the path was not correctly built and
included duplicate parts. This fix will remove the duplicate part if
possible.

Fixes BurntSushi#1757, Closes BurntSushi#2295
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Nov 28, 2023
14.0.2 (2023-11-27)
===================
This is a patch release with a few small bug fixes.

Bug fixes:

* [BUG #2654](BurntSushi/ripgrep#2654):
  Fix `deb` release sha256 sum file.
* [BUG #2658](BurntSushi/ripgrep#2658):
  Fix partial regression in the behavior of `--null-data --line-regexp`.
* [BUG #2659](BurntSushi/ripgrep#2659):
  Fix Fish shell completions.
* [BUG #2662](BurntSushi/ripgrep#2662):
  Fix typo in documentation for `-i/--ignore-case`.


14.0.1 (2023-11-26)
===================
This a patch release meant to fix `cargo install ripgrep` on Windows.

Bug fixes:

* [BUG #2653](BurntSushi/ripgrep#2653):
  Include `pkg/windows/Manifest.xml` in crate package.


14.0.0 (2023-11-26)
===================
ripgrep 14 is a new major version release of ripgrep that has some new
features, performance improvements and a lot of bug fixes.

The headlining feature in this release is hyperlink support. In this release,
they are an opt-in feature but may change to an opt-out feature in the future.
To enable them, try passing `--hyperlink-format default`. If you use [VS Code],
then try passing `--hyperlink-format vscode`. Please [report your experience
with hyperlinks][report-hyperlinks], positive or negative.

[VS Code]: https://code.visualstudio.com/
[report-hyperlinks]: BurntSushi/ripgrep#2611

Another headlining development in this release is that it contains a rewrite
of its regex engine. You generally shouldn't notice any changes, except for
some searches may get faster. You can read more about the [regex engine rewrite
on my blog][regex-internals]. Please [report your performance improvements or
regressions that you notice][report-perf].

[report-perf]: BurntSushi/ripgrep#2652

Finally, ripgrep switched the library it uses for argument parsing. Users
should not notice a difference in most cases (error messages have changed
somewhat), but flag overrides should generally be more consistent. For example,
things like `--no-ignore --ignore-vcs` work as one would expect (disables all
filtering related to ignore rules except for rules found in version control
systems such as `git`).

[regex-internals]: https://blog.burntsushi.net/regex-internals/

**BREAKING CHANGES**:

* `rg -C1 -A2` used to be equivalent to `rg -A2`, but now it is equivalent to
  `rg -B1 -A2`. That is, `-A` and `-B` no longer completely override `-C`.
  Instead, they only partially override `-C`.

Build process changes:

* ripgrep's shell completions and man page are now created by running ripgrep
with a new `--generate` flag. For example, `rg --generate man` will write a
man page in `roff` format on stdout. The release archives have not changed.
* The optional build dependency on `asciidoc` or `asciidoctor` has been
dropped. Previously, it was used to produce ripgrep's man page. ripgrep now
owns this process itself by writing `roff` directly.

Performance improvements:

* [PERF #1746](BurntSushi/ripgrep#1746):
  Make some cases with inner literals faster.
* [PERF #1760](BurntSushi/ripgrep#1760):
  Make most searches with `\b` look-arounds (among others) much faster.
* [PERF #2591](BurntSushi/ripgrep#2591):
  Parallel directory traversal now uses work stealing for faster searches.
* [PERF #2642](BurntSushi/ripgrep#2642):
  Parallel directory traversal has some contention reduced.

Feature enhancements:

* Added or improved file type filtering for Ada, DITA, Elixir, Fuchsia, Gentoo,
  Gradle, GraphQL, Markdown, Prolog, Raku, TypeScript, USD, V
* [FEATURE #665](BurntSushi/ripgrep#665):
  Add a new `--hyperlink-format` flag that turns file paths into hyperlinks.
* [FEATURE #1709](BurntSushi/ripgrep#1709):
  Improve documentation of ripgrep's behavior when stdout is a tty.
* [FEATURE #1737](BurntSushi/ripgrep#1737):
  Provide binaries for Apple silicon.
* [FEATURE #1790](BurntSushi/ripgrep#1790):
  Add new `--stop-on-nonmatch` flag.
* [FEATURE #1814](BurntSushi/ripgrep#1814):
  Flags are now categorized in `-h/--help` output and ripgrep's man page.
* [FEATURE #1838](BurntSushi/ripgrep#1838):
  An error is shown when searching for NUL bytes with binary detection enabled.
* [FEATURE #2195](BurntSushi/ripgrep#2195):
  When `extra-verbose` mode is enabled in zsh, show extra file type info.
* [FEATURE #2298](BurntSushi/ripgrep#2298):
  Add instructions for installing ripgrep using `cargo binstall`.
* [FEATURE #2409](BurntSushi/ripgrep#2409):
  Added installation instructions for `winget`.
* [FEATURE #2425](BurntSushi/ripgrep#2425):
  Shell completions (and man page) can be created via `rg --generate`.
* [FEATURE #2524](BurntSushi/ripgrep#2524):
  The `--debug` flag now indicates whether stdin or `./` is being searched.
* [FEATURE #2643](BurntSushi/ripgrep#2643):
  Make `-d` a short flag for `--max-depth`.
* [FEATURE #2645](BurntSushi/ripgrep#2645):
  The `--version` output will now also contain PCRE2 availability information.

Bug fixes:

* [BUG #884](BurntSushi/ripgrep#884):
  Don't error when `-v/--invert-match` is used multiple times.
* [BUG #1275](BurntSushi/ripgrep#1275):
  Fix bug with `\b` assertion in the regex engine.
* [BUG #1376](BurntSushi/ripgrep#1376):
  Using `--no-ignore --ignore-vcs` now works as one would expect.
* [BUG #1622](BurntSushi/ripgrep#1622):
  Add note about error messages to `-z/--search-zip` documentation.
* [BUG #1648](BurntSushi/ripgrep#1648):
  Fix bug where sometimes short flags with values, e.g., `-M 900`, would fail.
* [BUG #1701](BurntSushi/ripgrep#1701):
  Fix bug where some flags could not be repeated.
* [BUG #1757](BurntSushi/ripgrep#1757):
  Fix bug when searching a sub-directory didn't have ignores applied correctly.
* [BUG #1891](BurntSushi/ripgrep#1891):
  Fix bug when using `-w` with a regex that can match the empty string.
* [BUG #1911](BurntSushi/ripgrep#1911):
  Disable mmap searching in all non-64-bit environments.
* [BUG #1966](BurntSushi/ripgrep#1966):
  Fix bug where ripgrep can panic when printing to stderr.
* [BUG #2046](BurntSushi/ripgrep#2046):
  Clarify that `--pre` can accept any kind of path in the documentation.
* [BUG #2108](BurntSushi/ripgrep#2108):
  Improve docs for `-r/--replace` syntax.
* [BUG #2198](BurntSushi/ripgrep#2198):
  Fix bug where `--no-ignore-dot` would not ignore `.rgignore`.
* [BUG #2201](BurntSushi/ripgrep#2201):
  Improve docs for `-r/--replace` flag.
* [BUG #2288](BurntSushi/ripgrep#2288):
  `-A` and `-B` now only each partially override `-C`.
* [BUG #2236](BurntSushi/ripgrep#2236):
  Fix gitignore parsing bug where a trailing `\/` resulted in an error.
* [BUG #2243](BurntSushi/ripgrep#2243):
  Fix `--sort` flag for values other than `path`.
* [BUG #2246](BurntSushi/ripgrep#2246):
  Add note in `--debug` logs when binary files are ignored.
* [BUG #2337](BurntSushi/ripgrep#2337):
  Improve docs to mention that `--stats` is always implied by `--json`.
* [BUG #2381](BurntSushi/ripgrep#2381):
  Make `-p/--pretty` override flags like `--no-line-number`.
* [BUG #2392](BurntSushi/ripgrep#2392):
  Improve global git config parsing of the `excludesFile` field.
* [BUG #2418](BurntSushi/ripgrep#2418):
  Clarify sorting semantics of `--sort=path`.
* [BUG #2458](BurntSushi/ripgrep#2458):
  Make `--trim` run before `-M/--max-columns` takes effect.
* [BUG #2479](BurntSushi/ripgrep#2479):
  Add documentation about `.ignore`/`.rgignore` files in parent directories.
* [BUG #2480](BurntSushi/ripgrep#2480):
  Fix bug when using inline regex flags with `-e/--regexp`.
* [BUG #2505](BurntSushi/ripgrep#2505):
  Improve docs for `--vimgrep` by mentioning footguns and some work-arounds.
* [BUG #2519](BurntSushi/ripgrep#2519):
  Fix incorrect default value in documentation for `--field-match-separator`.
* [BUG #2523](BurntSushi/ripgrep#2523):
  Make executable searching take `.com` into account on Windows.
* [BUG #2574](BurntSushi/ripgrep#2574):
  Fix bug in `-w/--word-regexp` that would result in incorrect match offsets.
* [BUG #2623](BurntSushi/ripgrep#2623):
  Fix a number of bugs with the `-w/--word-regexp` flag.
* [BUG #2636](BurntSushi/ripgrep#2636):
  Strip release binaries for macOS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A bug. gitignore Bugs related to gitignore problems.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants