Cquery reports ast error when precompiled header is used

[pdavydov108]

Hello, I use cquery build from source from latest master (commit 82c438c). I have a c++ project where I use precompiled headers. It is compiled with clang, OS is linux, x86_64. Cquery is built with libclang from llvm.org (clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-14.04). I use it with vim via vim-lsp plugin. When precompiled headers are disabled, (compile_commands.json has no -include /path/to/header.h compiler flag) cquery works just fine, it's fast and quite reliable. However, if I enable pch, cquery reports an ast error from libclang. Here is the full log produced by vim-lsp plugin:

Wed 21 Mar 2018 01:11:30 PM MSK:["lsp#register_server","server registered","rls"]
Wed 21 Mar 2018 01:11:30 PM MSK:["lsp#register_server","server registered","cquery"]
Wed 21 Mar 2018 01:11:30 PM MSK:["lsp#register_server","server registered","pyls"]
Wed 21 Mar 2018 01:11:30 PM MSK:["vim-lsp signs enabled"]
Wed 21 Mar 2018 01:11:30 PM MSK:["s:on_text_document_did_open()",1,"cpp","/home/pablo/test","file:///home/pablo/test/main.cpp"]
Wed 21 Mar 2018 01:11:30 PM MSK:[{"response":{"data":{"__data__":"vim-lsp","lsp_id":1,"server_name":"cquery"},"message":"started lsp server successfully"}}]
Wed 21 Mar 2018 01:11:30 PM MSK:["--->",1,"cquery",{"method":"initialize","params":{"rootUri":"file:///home/pablo/test","initializationOptions":{"cacheDirectory":"/home/pablo/.cquery","cacheFormat":"msgpack"},"capabilities":{},"rootPath":"/home/pablo/test","trace":"off"}}]
Wed 21 Mar 2018 01:11:30 PM MSK:["<---",1,"cquery",{"response":{"method":"$cquery/progress","jsonrpc":"2.0","params":{"indexRequestCount":0,"onIndexedCount":0,"doIdMapCount":0,"onIdMappedCount":0,"loadPreviousIndexCount":0,"activeThreads":0}}}]
Wed 21 Mar 2018 01:11:30 PM MSK:["<---",1,"cquery",{"response":{"id":1,"jsonrpc":"2.0","result":{"capabilities":{"executeCommandProvider":{"commands":[]},"documentHighlightProvider":true,"hoverProvider":true,"definitionProvider":true,"referencesProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"codeActionProvider":true,"textDocumentSync":2,"codeLensProvider":{"resolveProvider":false},"documentLinkProvider":{"resolveProvider":true},"workspaceSymbolProvider":true,"documentRangeFormattingProvider":false,"documentFormattingProvider":false,"documentSymbolProvider":true,"renameProvider":true,"completionProvider":{"resolveProvider":false,"triggerCharacters":[".",":",">","#","<","\"","/"]}}}},"request":{"method":"initialize","jsonrpc":"2.0","id":1,"params":{"rootUri":"file:///home/pablo/test","initializationOptions":{"cacheDirectory":"/home/pablo/.cquery","cacheFormat":"msgpack"},"capabilities":{},"rootPath":"/home/pablo/test","trace":"off"}}}]
Wed 21 Mar 2018 01:11:30 PM MSK:["--->",1,"cquery",{"method":"textDocument/didOpen","params":{"textDocument":{"uri":"file:///home/pablo/test/main.cpp","version":1,"languageId":"cpp","text":"#include <iostream>\n#include <boost/optional.hpp>\n#include <boost/filesystem.hpp>\n\nstruct Data {\n    int a;\n    int b;\n};\n\nint main(int argc, char *argv[]) {\n    Data data;\n    std::cout << data.a << std::endl;\n    return 0;\n}"}}}]
Wed 21 Mar 2018 01:11:30 PM MSK:[{"response":{"data":{"path":"file:///home/pablo/test/main.cpp","__data__":"vim-lsp","filetype":"cpp","server_name":"cquery"},"message":"textDocument/open sent"}}]
Wed 21 Mar 2018 01:11:30 PM MSK:[{"response":{"data":{"path":"file:///home/pablo/test/main.cpp","__data__":"vim-lsp","server_name":"cquery"},"message":"not dirty"}}]
Wed 21 Mar 2018 01:11:30 PM MSK:["<---(stderr)",1,"cquery",["2018-03-21 13:11:30.980 (   0.871s) [indexer2     ]clang_translation_unit.cc:110   libclang had ast read error for /home/pablo/test/main.cpp. Please try running the following, identify which flag causes the issue, and report a bug. cquery will then filter the flag for you  automatically:","$ /usr/bin/clang++ -working-directory=/home/pablo/test/Debug -include /home/pablo/test/Debug/Precompiled.h -g /home/pablo/test/main.cpp -resource-dir=/home/pablo/cquery/build/release/lib/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-14.04/lib/clang/6.0.0 -Wno-unknown-warning-option -fparse-all-comments -fsyntax-only",""]]
Wed 21 Mar 2018 01:11:30 PM MSK:["<---(stderr)",1,"cquery",["2018-03-21 13:11:30.981 (   0.872s) [indexer0     ]clang_translation_unit.cc:110   libclang had ast read error for /home/pablo/test/main.cpp. Please try running the following, identify which flag causes the issue, and report a bug. cquery will then filter the flag for you  automatically:","$ /usr/bin/clang++ -working-directory=/home/pablo/test/Debug -include /home/pablo/test/Debug/Precompiled.h -g /home/pablo/test/main.cpp -resource-dir=/home/pablo/cquery/build/release/lib/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-14.04/lib/clang/6.0.0 -Wno-unknown-warning-option -fparse-all-comments -fsyntax-only",""]]
Wed 21 Mar 2018 01:11:30 PM MSK:["<---",1,"cquery",{"response":{"id":1,"jsonrpc":"2.0","error":{"code":-32603,"message":"Failed to index /home/pablo/test/main.cpp"}}}]
Wed 21 Mar 2018 01:11:31 PM MSK:["<---",1,"cquery",{"response":{"method":"$cquery/progress","jsonrpc":"2.0","params":{"indexRequestCount":0,"onIndexedCount":0,"doIdMapCount":0,"onIdMappedCount":0,"loadPreviousIndexCount":0,"activeThreads":1}}}]
Wed 21 Mar 2018 01:11:31 PM MSK:["<---",1,"cquery",{"response":{"method":"$cquery/progress","jsonrpc":"2.0","params":{"indexRequestCount":0,"onIndexedCount":0,"doIdMapCount":0,"onIdMappedCount":0,"loadPreviousIndexCount":0,"activeThreads":0}}}]
Wed 21 Mar 2018 01:11:31 PM MSK:["<---",1,"cquery",{"response":{"method":"$cquery/progress","jsonrpc":"2.0","params":{"indexRequestCount":0,"onIndexedCount":0,"doIdMapCount":0,"onIdMappedCount":0,"loadPreviousIndexCount":0,"activeThreads":0}}}]
Wed 21 Mar 2018 01:11:35 PM MSK:["s:on_text_document_did_close()",1]

And this is my compile_commands.json file:

[
{
  "directory": "/home/pablo/test/Debug",
  "command": "/usr/bin/clang++     -include /home/pablo/test/Debug/Precompiled.h  -g   -o CMakeFiles/main.dir/main.cpp.o -c /home/pablo/test/main.cpp",
  "file": "/home/pablo/test/main.cpp"
}
]

I tried running the command with compiler suggested by cquery ($ /usr/bin/clang++ -working-directory=/home/p...) but it runs without issues, or at least it doesn't report anything. If this helps, I can provide a small test project where this case can be reproduced. For me it would be ok if cquery would just filter out -include flag, that is basically what I do now to make it work, though I'm not sure if it is the best way to fix it.

p.s. Thanks for your work on cquery, it has issues, but in general it's cool and fast and have a lot of useful features!

[jacobdufault]

Does switching include to -include-pch fix it? I'm a bit hesitant to blacklist this flag since it seems like it could be used beyond just precompiled headers.

Assuming include-pch does not work a small test project would be great.

[pdavydov108]

@jacobdufault Please check this repo. I tried using -include-pch instead of -include, but this didn't help. Also, I think gcc doesn't support -include-pch, so I usually avoid it to do less configuration.
Also, similar topic is discussed in this ycmd issue. In that thread people came to a conclusion that the reason of error is that .pch is generated with a different compiler version that was used within ycmd. I'll check if this is the case here as soon as I'll have time.

[pdavydov108]

@jacobdufault Maybe one possible solution would be to add some command line key (for example --ignore-pch) that will remove all --include and --include-pch flags from the list that is passed to libclang, so that some users can enable it optionally? If this error really happens because libclang used in cquery and host compiler are of different versions and builds then it is hard for users to avoid this issue. I thought that I can build cquery with system libclang (I know it's not recommended) but it is not obvious from ./waf configure --help how to do it. I wonder if it is even possible to force it, all flags there refer to it as 'fallback configuration'.

[jacobdufault]

If the right solution here is to ignore -include-pch and -include then we can make project.cc do that for you, a PR to do so would be greatly appreciated :)

[pdavydov108]

@jacobdufault Sure, I can send a PR, however I'm not sure how is it better to do it - always remove this keys or provide some option that users can set?

[jacobdufault]

Always remove the keys sgtm, we can add an option if another use files a bug saying they need them.