C-C++ layer cleanup

[quicknir]

I'm glad to see that C-C++ layer has gotten some much needed TLC, especially rtags support. However, it's also become a bit of a mess with many packages supporting the same functionality, sometimes to no purpose. I've maintained my own fairly detailed C++ layer for a while, so I have some thoughts that I think could improve the layer, and also really reduce confusion for people using it.

First, is to eliminate redundant packages. In particular, there are multiple packages that all do the same exact thing as ycmd, but worse. ycmd is a really well maintained, polished project, and alternatives to error checking and completion that are a hundred lines of elisp calling to the clang binary are not going to match it. These packages (or, in some cases, they are parts of other packages like flycheck or company), are: flycheck-clang, company-clang, and company-clang-headers. I'm sure someone will argue in favor of them, but ycmd does all of what these do, and significantly better, and ycmd is far more performant on large projects. ycmd is very easy to install (it's even in some package managers). We should encourage people to use by far the best tool in the semantic completion/error checking space, simplifying the layer in the process.

Second, there are some other packages that I think can't really carry their weight:

• srefactor: to have this work properly, you need significant additional setup so that semantic properly indexes your entire project. In addition, the semantic parser isn't very good and this isn't likely to work reliably on a modern codebase. You are much better off using rtags refactor; it only does renaming but it will actually do it reliably.
• realgud: I've played around with this in the past, and it offers no functionality that gdb-mi doesn't already offer (it actually uses gdb-mi). It doesn't support all the windows that gdb-mi offers with multiple windows. We should only use gdb-mi, and work on making small improvements rather than bothering with realgud.
• google style: with all respect, this really shouldn't be there. Most of us don't use google style. If you want to provide a layer variable that sets up a C style, with a default that also serves as an example, and that example happens to be google's style, I'm fine with that (and it would be very helpful, as a beginner it took me forever to figure out how to configure style).

Third, there are a lot of issues with the keybindings. All of the rtags stuff is bound under g, even if it makes no sense. g is supposed to be for "goto", things that navigate you in the code. rename-symbol, reparse buffer, fixits, etc, none of these are examples of going anywhere. Also, it is not necessary to bind every single rtags functions under the sun. Some of these are useless; location stack backward is already handled by C-o, for example. rtags-goto-offset. Etc. Let's look and see what's actually useful before binding it.

Fourth, there are some issues with how clang-format is setup. First, the add-hook call that adds to the save hook should pass the add-hook option to make it buffer local. Currently I end up with clang format hook in all my buffers (including python). Second, the hook should always be added, not conditionally on the option of whether or not to reformat on save. The conditional is handled in the function call itself, checking this conditional in two points is redundant, and it also means that if format on save is off and you turn it on, you need to close and reopen buffers for it to work.

Fifth, the documentation is insufficient.

Sixth, the clang-format bindings are more complicated than needs to be. clang-format is not something that people use selectively, it's almost always used as a required auto formatter. , = should format the whole buffer, and that's all, just like yapf does in python mode. I'm also not very clear why so many functions are written from scratch when clang-format-region is already provided. But If you test out clang-format-region, you'll quickly see that clang-format just isn't very good at formatting part of a buffer; that's not what it's designed to do, and you get very weird results sometimes.

After getting rid of all these layers, instead of having a million variables enabling/disabling each component, it should just boil down to whether you have a compilation database. If you do, use rtags and ycmd. If you don't, use gtags. Maybe a second option to use ycmd even if you don't have a compilation database (sometimes the default flags work ok).

My layer addresses most of these points: https://github.com/quicknir/config/tree/master/spacemacs.d/layers/cpp2. It is missing some layers that should be supported (like gtags, ivy-rtags, etc), and it doesn't have everything bound. But most of the things I've discussed above are better handled here. It also sets up the jump handlers correctly so that it tries to use rtags, and if rtags server is not running, it will fall back to ycmd's goto def, which is decent.

I think some merge of the current layer and my layer would result in an ideal C-C++ layer that I'd be happy to use and I think would improve things for other layer users.

@syl20bnr

[myrgy]

also we'd like to export cake support as dedicated layer. Improve integration with helm-make and add helm-test support.

[MaskRay]

It'll alse be nice to integrate cquery and lsp-mode #10134

[dvzubarev]

@quicknir Just wanted to say a word of defense for realgud. gdb-mi is not usable for me, since I work on relatively large codebase (260k loc) with heavy templates. I keep constantly getting those errors https://stackoverflow.com/questions/31206568/how-to-fix-stack-overflow-in-regexp-matcher-in-emacs
I don't think this will be fixed in the emacs (since it requires rewriting gdb-mi), so my sole tool for debugging is realgud.

[quicknir]

@dvzubarev Unless I'm misunderstanding, doesn't this problem only come up in the locals window? Which realgud doesn't even have? If that's the case you can "solve" this problem simply turning turning off multiple windows, which IIRC will cause gdb-mi to only spawn the source and command window, similar to realgud. Also, it seems like a fix to your issue has been merged: http://lists.gnu.org/archive/html/bug-gnu-emacs/2017-12/msg01046.html though I don't know how long it will take to become live.

Please let me know whether what I wrote is accurate or not.

[myrgy]

guys, how do you show structures content? I've added cxx pretty printers, but still variables are printer as "complex value". (that was with gdb-mi). reargued doesn't work at all 😟
UPD: hm, from third launch it works, a bit odd...
UPD2: realgud works as well, but there it no watch and breakpoints window.

[MaskRay]

Regarding rtags-location-stack-{forward,backward} (though I am using lsp-ui-peek.el),
A jump list specific to xref (definitions/references and other types of xref jumps) are useful and cannot be replaced by C-o C-i. See emacs-lsp/lsp-ui#27

[myrgy]

@MaskRay , lsp looks pretty interesting. Do u have layer for that? I'd like to give a try.

[MaskRay]

@myrgy https://github.com/MaskRay/Config/blob/master/home/.emacs.d/private/%2Bmy/my-code/packages.el https://github.com/MaskRay/Config/blob/master/home/.emacs.d/private/%2Bmy/my-cquery/packages.el

[jvillasante]

What's the practical difference of using ycmd instead of irony-mode? There's now also lsp. Which has the potential to become the better default for a c-c++ layer?

[quicknir]

@jvillasante ycmd in my experience has a much larger user base, developer base, more polished implementation and better performance. They do pretty well exactly the same thing. lsp is different, it is like rtags in that it is more heavyweight and indexes your whole project, allowing things like find references which ycmd and irony do not.

@MaskRay I think lsp looks great, but I think it's a separate issue. The point here is to try to eject deadweight packages and simplify the layer as much as possible, fix some of the bugs/issues/bindings discussed above. lsp also seems to be rather "fresh", e.g. there are features it claims to support (like syntax highlighting) that aren't really practical for performance reasons. I'd like to keep this issue on that topic, and I'd suggest opening a separate issue, or even PR for lsp. Since for now it seems like rtags and lsp are comparably viable, I'd probably start with a PR that introduces optional support for it, and also more step by step instructions on how to get it going. You've sent me lots of random links but it's been rather confusing tbh.

Anyway, let's keep focused: if there are concerns with some of the changes I'm proposing at the top, let's talk about those. @dvzubarev , can you confirm whether gdb-mi with multiple-windows off, works as well for you as realgud?

[MaskRay]

+1 for simplifying the c-c++ layer.

I actually use realgud but the default keymap is a bit weird, s for -next while i for -step, and also o for -finish. This is because n f are taken by evilified maps.

(evilified-state-evilify-map realgud:shortkey-mode-map
        :eval-after-load realgud
        :mode realgud-short-key-mode
        :bindings
        "s" 'realgud:cmd-next
        "i" 'realgud:cmd-step
        "b" 'realgud:cmd-break
        "B" 'realgud:cmd-clear
        "o" 'realgud:cmd-finish
        "c" 'realgud:cmd-continue
        "e" 'realgud:cmd-eval
        "r" 'realgud:cmd-restart
        "q" 'realgud:cmd-quit
        "S" 'realgud-window-cmd-undisturb-src)

[dvzubarev]

@quicknir sorry for the late response, I will try emacs master as soon as I can. I used gud without multiple-windows (this is command gud-gdb right?) and it worked well, but it didn't even highlight my breakpoints. I do not know whether can i turn highlighting on. Also I missed realgud:cmd-eval-dwim. So realgud is better than gud-gdb in my opinion.

[quicknir]

@MaskRay spacemacs bindings generally stay under , so probably all bindings of the layer will stay behind , d. Glad you are pro simplifying. The idea I think will be to get rid of many extraneous packages, and when you are ready, a PR that has the layer conditional use either rtags or cquery as the "database level" backend (I'm inclined to keep ycmd for auto completion and syntax checking for the near future, for a few reasons).

@dvzubarev I will try to use gdb-mi and see whether I can verify this. I'm still pretty skeptical about realgud tbh even if both your points are correct; many of us use multiple windows without problem and realgud has no equivalent for that. We'll see whether it's worth supporting both; it's a real shame if the benefits are split between these two, especially (again) since realgud is built on gdb-mi, they could have just upstreamed their improvements...

If I don't hear too much more I'll submit a PR that fixes all of the issues I listed above over the next week or so.

[MaskRay]

@quicknir realgud:shortkey-mode-map is a special minor mode enabled in the source code buffer mimicking gdb tui or cgdb, I think. They are triggered by a single key instead of a , prefix. I'm still using cgdb much but the source code viewer is not so good and evaluating expressions require a lot of typing (no features like eval at point) I asked them to support easy eval at point but it looks it requires some efforts cgdb/cgdb#141

Having LSP is definitely a separate issue. I mainly use C++ but it's apparent we should also hear from Haskell, Rust, Python, .....

[quicknir]

@MaskRay It may be triggered by a single key by default, but this isn't a good approach and not consistent with spacemacs. The bindings you give above overwrite a pile of evil bindings that people may want to use in the source code buffer. The best we can do is write a hydra (transient state) that has those bindings). Probably bound to , d .. Maybe to you it makes sense to override evil bindings when debugging in a source buffer, but for others it will not; e.g. some might want to immediately stick in a TODO for something in one function before they forget, or may want to use e for navigation, etc. Spacemacs with rare exceptions uses the leader keys to avoid these controversies.

As for lsp for other major modes, yeah that is a good point. It probably makes sense for there to be a separate lsp layer that takes care of all "generic" lsp related stuff, and then have C++ specific config only happen here (the same as ycmd has its own layer instead of being owned by this layer, because ycmd can be used for other languages).

[MaskRay]

If you are interested at lsp + cquery combo, I cordially invite you to check out

• lsp Add +tools/lsp layer for lsp-mode & lsp-ui #10211
• cquery Add +tools/cquery layer #10236

[emacsun]

Hi guys,
one more thing: please consider all the platforms(*nix, mac,windows) during designing your tools for C/C++.

sometimes, one have to work under windows. However, it seems that all the tools are designed for *nix

[dvzubarev]

The best we can do is write a hydra (transient state) that has those bindings)

Yep, I ended up writing such a hydra.

I wish I could use gdb-mi but I can't. I tried master and I didn't get those errors anymore but it was very slow. I don't mind if c++ will not use realgud (I think I can hack something around to use it anyway), but I think realgud is useful for spacemacs since it supports ruby, python, bash, perl, etc. Layers for these languages may use realgud as the main debugger. So I think realgud should be in the separate layer, as is done now and it should be maintained independently from language layers.

[myrgy]

We can extract realgud and gdb-mi as +tools/+debuggers. And then any other language layer can use as a small component.

[dvzubarev]

@myrgy Yes, I thought it is already done in devel branch, but this PR (#9246) is not merged yet.

[myrgy]

Awesome! I've missed that.

[myrgy]

@emacsun, unfortunately I dont use windows for development now, so if you will have any issues, please, let us know!

[emacsun]

@myrgy I totally understand you. Unfortunately, I have to work under Windows. Until now, I did not find a straightforward method to support c/c++ completion. So sad.

[myrgy]

@emacsun, as far as I know rtags supports windows, so it should be just a matter of compiling it under windows. I dont know if its possible under cygwin, but it might be a solution.

[MaskRay]

cquery also builds on Windows. But help required from someone to package it..

[myrgy]

Just to create references
c-c++ Layer needs improvement #7628
add debug layer #9246
New debug layer #7355

[quicknir]

To all who have taken an interest, I've started this work. I really hope that its able to go in, some of my changes are fairly dramatic. I've eliminated anything that fully overlaps with ycmd: ycmd is honestly just the better tool. Some of the flags were (I'm sorry) rather inane, things like c-c++-use-c++-11 or whatever it it was. That includes eliminating piles of elisp code that tried to get compilation flags for company-clang or company-c-headers (which are both gone). As part of my PR I'm also going to replace the existent ycmd file: it doesn't have a good set of default flags (important for first impressions and quickies), it doesn't get system headers automatically (this is important because libclang doesn't have the same default include paths that the clang binary does). Support for clang_complete will be dropped (since we are using ycmd only, let's stick with compile_commands.json).

I've also simplified and improved the clang-format stuff.

To be done:

• replace ycmd global config for finding flags (ycmd layer, not c-c++, but related, will be in same PR)
• improve rtags bindings
• eliminate realgud and just stick with gdb-mi (it seems like people are ok with this in the end, realgud will live in another layer)
• improve docs
• introduce a default c-c++ style variable (instead of all the google style crap). This sets a more reasonable default (I'm flexible to modifying it) than emacs (which is insane), and as a bonus it really helps first time users setup their basic formatting and get going
• more careful setup of jump handlers (i have to double check how the layer does it now, it's a bit tricky: ideally you want to give rtags priority, then ycmd).

My focus is basically going to be to allow people to get up and running with ycmd in a toy file ASAP so they can observe the magic, with good links/explanations on how to proceed with compile_commands.json and rtags, and even some debugging tips.

My branch with work so far is here: https://github.com/quicknir/spacemacs/tree/cpp.

[myrgy]

@quicknir ,

1. I think we shouldn't remove c-c++-enable-clang-format-on-save parameter. In your branch check were lost.
2. company-cmake should be initialized in cmake layer, not in C++.

[MaskRay]

Nice to see these are removed:

-    (when c-c++-enable-c++11
-      (setq company-clang-arguments '("-std=c++11")))

It might be harsh to mention cquery again... But has anyone tried #10211 + #10236

It can be made an optional dependency with a

(defvar c-c++-enable-cquery-support nil ......

I gave a bit more detail in jacobdufault/cquery#363 (comment) and compared it with rtags

[quicknir]

@myrgy Are you sure? I see the parameter still: https://github.com/quicknir/spacemacs/blob/cpp/layers/%2Blang/c-c%2B%2B/config.el. Can you clarify? As for 2, sure, I didn't touch the cmake layer, but happy to make the adjustment.

@MaskRay It's not harsh, it just deserves its own issue. I don't really have time at the moment to get into it. I'd suggest perhaps waiting until my PR is merged and then submitting a fresh PR with the cquery stuff.

[myrgy]

@quicknir , just propsal to make PR to you repository to your branch - it will allow us to comment.
c-c++-enable-clang-format-on-save is there, but there is no check in packages.el

(defun c-c++/init-clang-format ()
  (use-package clang-format
    :init
    (progn
 >> flag check was there...
      (spacemacs/add-to-hooks 'spacemacs/clang-format-on-save c-c++-mode-hooks)
      (dolist (mode c-c++-modes)
        (spacemacs/set-leader-keys-for-major-mode mode
"=" 'spacemacs/clang-format-region-or-buffer)))))

[quicknir]

@myrgy yup, that's intentional. The check happens in the hook. I discussed this in my very first post on this issue.

[myrgy]

My fault, I missed that. @quicknir , do you think it will be possible to enable clang-format selectively for different projects using .dir-locals.el ?

[quicknir]

@myrgy I'm not sure, it maybe be necessary to make enable clang-format-on-save a buffer local variable. But in any case I still think always adding the hook is the right thing to do (though you have me doubting myself). Do you want to help out with this? You can this pretty easily; if it doesn't work try making the variable buffer local.

[myrgy]

@quicknir , yep, I'll try to take a look.

[cormacc]

Had reason to look for an alternative to rtags after my linux development machine imploded and I was left setting up a dev environment on a windows machine I normally use for office/qms duties. Set up the cquery and lsp-server layers last night based on the links posted by @MaskRay and very impressed with it so far. Feels much more responsive than rtags (and the stand-in windows laptop is pretty underpowered compared to the machine it's replacing), and pretty easy to set up (on both that windows machine and another laptop running os x). The overlays provide a lot of information -- could maybe be a bit intrusive, but there's probably some configuration I haven't got to yet -- chasing a deadline here ,so have just scratched the surface far. But looks promising, as a solution that's potentially more performant and portable than the status quo? It's working very well with a largeish embedded codebase in plain C with heavy use of a macro-based DSL, but haven't done any CPP work with it yet.

[MaskRay]

@cormacc Thanks! The thread is for clean-up of existing packages as mentioned by @quicknir many comments above. But I'd like to know how owners think of adding LSP to other language layers, which might replace packages in other layers. There may be some drastic changes, but I don't know as I do not do much dev in other languages. And if that is the concern of owners to integrate LSP.

[steve-lorimer]

@quicknir what's the status of this please?

[haydenflinner]

I've been using this layer for about a month and it's been awesome, definitely a way zippier and smoother experience than the current c-c++ layer.

I also like that it's basically got two options -- works about the same as the current layer but easier if you don't have a compile_commands.json set up, but once you set one up, rtags works like a charm and with great discoverability.

Going to try the LSP-layer now if it's not too hard to set up, but very excited to see such progress for CPP in spacemacs. Hope we can decide on a replacement for the current layer soon.

[quicknir]

@skebanga @cormacc Thought I would comment here to respond some of the pings I got. Sorry for the long absence. Not too surprisingly (at this point), I haven't worked on this for a while. The issue is that I already have a more opinionated c++ layer that i use locally, that definitively uses rtags + ycmd (i.e. there are not options to select other backends like irony, ggtags, cquery, etc), that works perfectly. In terms of trying to make this work available upstream:

1. I didn't feel confident that maintainers actually agreed with my vision for the layer, and that they would actually merge whatever I came up with. I realize that often maintainers also feel like they can't approve/agree with things until they see the final product, so I'm not trying to make this black and white. But this lowered my motivation considerably.
2. As I started working on this, almost all of the actual effort ended up just figuring out how exactly to provide multiple "backends" for functionality would work, e.g. whether rtags or ggtags should be used to find references, but then should it be a global choice, a per project choice, etc. It seems simple at first but it became a headache quickly, and since I had no interest in e.g. using ggtags, there was a ton of inertia to overcome to actually test out how things are working. I don't think this is a good situation; I think what makes my rtags/ycmd based cpp layer good is that I'm using it every single day, I've made dozens of small tweaks and improvements as I've encountered issues.

So, I'd like to come full circle with this and try to offer a solution that I think I am realistically willing to work on and that I think will help people. One of my original ideas was to simply allow multiple layers for the same language. Instead of having a c++ layer which awkwardly tries to blend the combinatorial explosion of possible backend choices (ycmd, irony, ggtags, rtags, cquery, etc etc), I'd like to offer a separate layer (cpp2; name of course is negotiable) that is opinionated and simply uses rtags and ycmd. There are pros and cons to this approach like anything.

Anyhow, if one of the maintainers, perhaps @sdwolfz , directly tells me they like this and is willing to have a second layer for a specific language with specific backend choices, provided it is well documented, has consistent keybindings, etc, then I'll do the work that remains to put cpp2 in suitable shape, including dealing with any feedback. If this isn't the route that Spacemacs wants to take, if they strictly want a single layer per language with lots of options controlling the backends used for component, then I'll probably bow out as I'm not willing the time to work through it. In that case of course, anyone who is so willing is welcome to use any of the code I've written for the ycmd/rtags components.

I think my rtags/ycmd approach is very well honed and offers a lot of value to Spacemacs users, and I think that C++ is an exceptionally both complex and popular beast, so I think there is ample justification to have more than one layer for it. But if the project doesn't see it that way, I understand. Just let me know.

[cormacc]

Thanks @quicknir -- I hear you. A lot to be said for simplicity.
That said, I'll still have a bash at integrating cquery/ccls with the c-c++ layer, having taken it this far. Good to know I won't be stepping on your toes by doing so. Must take a look at the cpp2 layer too when I get a chance.

[quicknir]

@cormacc Do you want me to push the work I have so far to a branch somewhere? There are some things that are worth integrating in any case, e.g. bugfixes for clang-format on save.

[cormacc]

Sure - that would be great.

…

On Thu, 23 Aug 2018 at 13:30, quicknir ***@***.***> wrote: @cormacc <https://github.com/cormacc> Do you want me to push the work I have so far to a branch somewhere? There are some things that are worth integrating in any case, e.g. bugfixes for clang-format on save. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10143 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmzi5cU2EWpnHBlJjxnW_2d2G0nBK7Yks5uTqBJgaJpZM4RcdUT> .

[sdwolfz]

Personally I would prefer having a single c-c++ layer and have the functionality controlled by a backed variable in that layer, from a user's perspective I imagine choosing a backend would be a less confusing task than evaluating the differences between 2 layers.

But I understand why you would want to have a separate layer as well, the current version is really difficult to dive into and change without the fear of breaking something, reason why I've stayed clear of most of the non trivial c-c++ PRs until now.

What I propose for now is we try to improve the current c-c++ layer as much as we can and try to figure out what exactly makes it difficult to have the configuration in one place, maybe we can work around the issue step by step, or we eliminate a problematic package completely. If there are any incremental changes you can cherry pick from what you've done so far please open up PRs and I will prioritize them (the smaller the PR the better). That way we can /cc more people to require feedback on specific changed and if you let the ycmd/rtags integration of your work to someone else they will have an easier time figuring stuff out.

[lovrolu]

@quicknir I'm interested in the cpp2 layer you mentioned. Did you end up pushing it anywhere? I found this but I'm not sure if that's the latest version you have.

Appreciate the work you put into this -- great job! :-)

[myrgy]

I think it's more about: #11242

[kgfly]

Please keep real-gud alive because it is going to support lldb. And LLDB is the default debugger on Mac now. It's tricky (and steps are constantly changing) to make GDB working on Mac.

(Emacs default gud can never support lldb, because the boss of GNU does not allow Emacs officially supports LLDB since he thinks LLDB is a "systematic attack" to Emacs/GNU. If you "emacs lldb" online, you will find more.)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please let us know if this issue is still valid!