Include global UserSettings.h file

[NicoHood]

If we could have a global UserSettings.h file which is included at the top of the Arduino.h file we could:

• Let the user add custom settings to an .ino file for libraries and the core itself
• This way we can easily overwrite things like RX_BUFFER_SIZE as well. (without a boards.txt mod)
• We could also add options to disable the USB-Core and other components if not needed
• We could add an option to disable the SerialEvent function. You can keep it by default for compatibility, but set a few parameters to tweak the Arduino code and save some flash
• We could use this for libraries, such as PinChangeInterrupt to effectively make use of direct port access with pre defined pins. This also works for other libraries. A known pin at runtime is way faster and also saves a lot of time.
• Global debug options could be enabled this way.
• Many more things I can imagine of and you as well.

This requires every file to see the UserSettings.h file, it needs to be included in the Arduino.h file and it needs to be placed in the .ino folder, and if not it should be created as an empty file. Also make sure if you delete it, an empty dummy will be added again.

Imported from: arduino/Arduino#3757
Also fixes (and maybe more): arduino/Arduino#3717 arduino/Arduino#3635

[stevemarple]

I think I have a commit which successfully allows macros (#defines) across all compilation units. My approach is to test for a build_props.txt file in the sketch folder. It has the same key=value format found in the board files. Global macros can be defined with

compiler.c.extra_flags=-D NDEBUG
compiler.cpp.extra_flags=-D NDEBUG -D TESTLIBRARY_BUFSIZE=100

Followers of the DRY principle can take advantage of the property expansion method in arduino-builder:

compiler.c.extra_flags=-D NDEBUG
compiler.cpp.extra_flags={compiler.c.extra_flags} -D TESTLIBRARY_BUFSIZE=100

A global UserSettings.h file couldcan be included across all compilation units:

compiler.c.extra_flags=-include {build.source.path}/UserSettings.h
compiler.cpp.extra_flags={compiler.c.extra_flags}

A property defining the sketch folder appears to be missing (#24) requiring the path to UserSettings.h to be hard-coded.
[Code snippet above updated now that build.source.path is correct.]

I'm not ready to submit a pull request since I have not added any unit tests but I would appreciate feedback and testing to know if this is useful. The commit can be found at stevemarple@a5fc12b

[cmaglie]

The idea of having a specially-named .h file in the sketch be included during compilation of the core and libraries is a long-standing one: https://code.google.com/p/arduino/issues/detail?id=27

I've some concerns about opening this possibility to developers, I think that @damellis has very well explained the reasons here:

It may or may not be a good idea but it is, I think, a big change and one that should be considered carefully on the developers list. This would open up lots of potential tweaks to the core and libraries, greatly increasing the effective API of those files (i.e. previously-internal details of the files that people will now be able to rely on and therefore will become more difficult to change). It will also add a lot of potential complexity / features that people may see in sketches they get from others. In addition, by not having a way to provide pre-processor macros / compile-time constants to the core and libraries, I think we've forced people to find easier-to-use alternatives in the form of run-time configuration (which is more consistent with other parts of the Arduino API). With this option, I worry that people will instead take the path-of-least-resistant and use compile-time configuration, complicating things for users.

In 4 years I've seen cases where this rationale applies for true, the most significant are:

• Problem: HardwareSerial1/2/3/4 objects always takes memory even if not used -> Path-of-least-resistant solution: Add defines to disable/enable HardwareSerial -> Current solution: Put objects in different compilation units, so they are automatically optimized by the compiler.
• Problem: extending the core's USB-HID classes takes too much flash/ram if all the improvements are taken together (Overhaul USB HID for Leonardo and Micro Arduino#1803) -> Path-of-least-resistant solution: add core's defines to selectively disable them -> Current solution (still work-in-progress): Pluggable USB/HID
• Problem: some configuration values may be optimized saving some bytes -> Path-of-least-resistant solution: add defines for the configuration values -> Better solution: Sacrifice a bunch of bytes and make them runtime-values so the user doesn't have to remember yet another #define
• Defines could be "abused" and used even when they are not needed, with the result to provide a more complex API for the end user. The following is an emblematic example: Library configuration file management (issue #1734) Arduino#1808

To conclude, is this feature really worth it? the things that are absolutely not possible without #defines and #ifdefs are really compelling for end users?

[ffissore]

I agree with all these reasons and experience shows that better/easier to maintain solutions have been found without relying on #define.
However, there are cases in which a lib dev is trying to understand which problem one of her user is having: enabling -DDEBUG in one place could allow a non-dev to help a dev
What about yet-another property in library.properties: debug=true ? If present, -DDEBUG will be added to library compilation command lines

[NicoHood]

Nah. I really do need this file inclusion, not only a library.properties. I need a setting which can be added to the sketch as additional .h file tab. Its not just about enabling debug. If i'd want this I can just put it in the librarys header itself.

Its about the time after you release stuff and the user wants to customize stuff. You dont need to tell him which source file he needs to find and what to edit. And the next time he needs a different setting and needs to search it again. This way we can have settings per sketch.

I suggest two options at the same time:

• Add a UserSetting.h file which is included by Arduino.h. This way we can add #defines to disable different hardwareserial etc (see list above, thx christian)
• Add the sketch folder to the general inclusion path. This way the library maintainer can force the user to add this file to its sketch. This is needed for more complex or improved libraries. This should normally solved with the option above, but this way you can force the user to use it and no other libraries/core will see this file if it has a unique name. This implementation is required to make the solution above work anyways.

You could also do a separate CoreSettings.h (which is included by the Arduino.h file) and a LibrarySettings.h file (which may be included by the library). A Wrapper for including both could be to just use #include UserSettings.h to just include both .h files.

Having 2 separate Settings files would be required for deactivating the USB-Core (how it is right now). In the CoreSettings.h you just #undef USBCON and in the LibrarySettings you do not do this, but maybe some other settings. This way one could write a whole new USB-Core, even without using the Pluggable stuff. Same goes for Serial deactivation etc.

[stevemarple]

I agree that in some circumstances #define's could be considered a lazy method, for instance to define internal buffer sizes. An alternative I use in my MicroNMEA and IniFile libraries is to make the
user pass a pointer to the buffer and its size. I'm not convinced it's easier to use, and I'm not sure that requiring novice users pass pointers is a good idea either. At least in this case an alternative
exists. There are things inside libraries which could be genuinely useful in some circumstances but consideration of memory requirements in common usage prevents it. This affects cases which would need additional member variables as I don't think that compiler/linker optimizations can recover
that memory. However conditional compilation would prevent its allocation.

There are many useful contributed libraries which require users to modify the library header files to comment/uncomment the appropriate #define. This is terrible practice and I suspect it has continued because the Arduino IDE lacks the ability to define sketch-specific macros. Adding a method to globally define macros would make using these libraries much less painful.

The effect of inserting a library.properties.debug property is limited and doesn't solve the problem I'm trying to address. Moreover it doesn't seem useful unless the option is exposed by the IDE - the developer would need to restart the IDE just to enable debugging. Changing a build_options.txt file would however be easy and it does correctly force a complete recompilation.

Whilst I accept you must consider the environment for novice users (that was me a few years ago) the Arduino ecosystem has grown. As the existence of the Arduino at Heart program shows some of that growth is towards more advanced/industrial users. Please consider their needs too.

Would a new pull request which only read build_options.txt when build.custom_sketch_properties=true be accepted? That would discourage novice use whilst enabling more experienced programmers greater customisation in the way every other common IDE allows.

[NicoHood]

Am I think we are talking about two different features here.

• Including a header file as global config
• Passing custom compiler flags to the IDE.

I think both cases are really helpful. I really like your idea, but you should probably discuss this in the PR you opened. I can imagine things like: preserving some registers from being used for very special libraries etc. If this makes sense in all cases is out of question here.

However I'd rather put those compiler flags in the libraries.properties file. The dot_a_linkage setting is placed there as well, and why do we need another file? Also I think an option like build.custom_sketch_properties=true is not needed. Because if you dont want to use it, just dont (do not add this file). As simple as that.

[stevemarple]

@NicoHood Pull request #29 is a generalisation of this feature request. It enables a global config file to be included with gcc's -include command line option or by using the -D macro=definition option macros can be defined without an extra file. Your request is not the first for this feature but as far as I could see it is the first in relation to arduino-builder, hence the discussion here.
I took the approach I did since arduino-builder already accepts 'customisations' at a platform and board level, and from the command line. Tweaking the build in this way doesn't require any changes to platform or board recipes, or to https://github.com/arduino/Arduino.

[NicoHood]

I was not aware that you can add -include as optional parameter. This sounds pretty nice and partly solves the issue here I think.

Is there a way to point to the sketch folder directly? Is there a way to ignore the include if no file was found? I am still not 100% if this solves the whole issue, but sure it enables us a wide range of new possibilities.

And finally how can I test this build (your PR)?

[stevemarple]

@NicoHood Instructions are in my first post in this thread, including how to reference the sketch source folder.

You need a very recent version of the Arduino IDE (nightly build or built from the latest sources). Replace the arduino-builder executable with one built from from dev-global-macros branch, https://github.com/stevemarple/arduino-builder/tree/dev-global-macros. Alternatively download a 64bit version from https://www.dropbox.com/sh/oypoojbqodwkc05/AACZwX4uSJdoH5AcMgFMq88ea?dl=0

[stevemarple]

In the end I worked round this by defining my own boards.txt file with a menu option to include custom_include.h. I needed a boards file anyway to support my Calunium ATmega128P clone so it was the route of least resistance for me. I'd still like to see support added generally.

[robjen]

really could do with this feature, injecting #defines into a library at compile time allow me to write code that is stripped back to optimal, saving space and allowing for things like endianess without switch/if statments based on variables passed at 'begin' (extra code and kills performance on every access)

and this sooo nearly works .... I notice from looking at building other simple project that the environment is pulling ALL files into the temp area including the copies of the libraries and the sketch but there is a preprocessing step that fails on the project first build (it seems to build a library in place at the library path first) ....

so I did the following

wrote my own library put it in the proper place
wrote a stetch which in its directory contains mylib_config.h
library has a #include "../sketch/mylib_config.h"

1. compile sketch and it fails missing file in library also note none of the libraries were copied ;-(
2. comment out the #include in my library and build it once
   works fine (my library has defaults for anything not defined)
3. now add back my #include line and build again
   now the build is using the cached files in the temp area (even copied over my mod from the library area) and the config files has its desired effect on the library now

it seems to me ... if the environment is going to cache the files in temp anyway ... why does it not do that step FIRST and pre-compile the library with that copy there ? since its going to be doing everything else in there until I close the envirnment. If the copy was made first then everything would work as people desire

[ricardojlrufino]

Since IDE does not support the concept of PROJECTS nor the definition of environment variables. That would be the least to do.

Most IDE has a configuration file to specify this type of configuration.

[xcvista]

@ricardojlrufino It might be a better idea to keep the name "Project" out of Arduino, and try to achieve the goals by tweaking the existing sketch format.

For the #define's, my idea would be, as stated above, convert all simple #define's in the main sketch (.ino) file into -D for all calls to the compiler.

For other IDE settings, I would prefer #pragma arduino in main sketch file: whatever comes after this in the line is treated as part of boards.txt for this sketch.

[PaulStoffregen]

Everyone recently commenting on this issue should re-read Cristian's comment posted on Oct 5, 2015.

[ConnorRigby]

here is that post. That solution would be fine had it been merged.

[sleemanj]

When avr-gcc 5.0 arrives in Arduino land then the __has_include macro will be available.

That would mean that a UserSettings.h optionally being provided and with no impact if not provided should be as simple as adding -DUSER_SETTINGS_FILE=\"/path/to/UserSettings.h\" and then library (or core) authors can use....

#if defined(__has_include) &&  __has_include(USER_SETTINGS_FILE)
    #include USER_SETTINGS_FILE
#endif

For alternative core developers it will be trivially simple to add this to their core, but without the Arduino core itself doing so there risks to be a proliferation of differing standards "UserSettings.h", "Configuration.h", USER_SETTINGS_FILE, LIBRARY_SETTINGS....

Indeed, if even the sketch source folder is available to the GCC preprocessor, it would probably be possible to construct include file paths in macros with some magic somehow, the quote marks necessary for #include are a bit of an issue it seems but I think a correct incantation of #define magic could be found that would allow constructing arbitrary candidate include files if the sketch folder is exposed and to include them if they exist, or not if they don't.

My interest is mostly in tiny processors where every byte counts. Currently I'm writing a library for sleep management which for some cases uses the WDT, but not all use-cases employ it, for example sleeping "forever" doesn't need the WDT, sleeping "deeply" does, sleeping "idle" doesn't necessarily.

The compiler has no way to optimize out an unused ISR like the WDT, so even if the library-user never calls thing_that_needs_to_use_the_wdt() the WDT ISR is still included. The only way (as far as I know) to not have that WDT ISR in, is to not compile it at all. On a chip with 1k flash even 50 bytes of unused ISR is hugely wasteful.

[matthijskooijman]

The compiler has no way to optimize out an unused ISR like the WDT

Slightly off-topic, but there is a trick with using a .a archive file for linking, which is used by HardwareSerial. See this comment (which only applies for files linked through .a files, but you can specify some property in library.properties that makes this happen). If you have additional questions, feel free to drop me a mail, rather than continuing this as an off-topic discussion here.

[stevemarple]

@sleemanj Core developers can add the desired functionality now, no need for avr-gcc 5.0. See #15 (comment) and https://github.com/stevemarple/Calunium/blob/master/software/arduino-1.6/calunium/avr/boards.txt

[sleemanj]

@stevemarple doesn't that (-include) cause an error if the file does not exist? It does in my avr-gcc (4.8.2)

$ avr-gcc -include foo.h test.c
cc1: fatal error: foo.h: No such file or directory

With __has_include no error would happen if the file doesn't exist.

Edit: I see you have set that up to only be used if you select the appropriate menu option, in which case, yeah you could do that certainly, accepting that if the user picks that option and attempts to compile without actually having the required file created then they will get an error.

[PaulStoffregen]

Again, everyone here should re-read Cristian's comment near the top of this issue, written Oct 5, 2015.

Arduino has historically rejected this idea (in a variety of forms) not based on technical issues with its implementation, but based on the massive effective expansion of the API it would have, and the anticipated long-term effect that could have on the Arduino ecosystem.

Focusing on the technical implementation details is missing the "big picture".

[sleemanj]

Focusing on the technical implementation details is missing the "big picture".

Which is, that this is probably going to happen anyway (and as above with @stevemarple 's core already is) with or without Arduino officially endorsing it with a one-true-way (of having compile-time configuration passed into cores and libraries).

I think it would be better with a one-true-way personally, rather than having each core doing their own thing.

This ticket is still open, so right now it has not been rejected flatly as WONTFIX, I think it's reasonable to continue discussion at least until it is, and part of that is technical detail.

[EtienneGameSeed]

So finally is there any way to do this ?
I need this too, in order to customize the audio features of the Teensy Core.
I've tried to add -prefs="build.extra_flags=-DAUDIO_INPUT_NB_CHANNEL=2" to the arduino builder command line, but it doesn't work.
arduino_builder seems to understand the command and takes it into account, it adds it to the customBuildProperties in build.options.json, but the compiler still throws an error like "error: 'AUDIO_INPUT_NB_CHANNELS' was not declared in this scope"
Have I missed something ?

[facchinm]

@EtienneGameSeed it looks like you are missing a minus sign.
The command should read -prefs="build.extra_flags=-DAUDIO_INPUT_NB_CHANNEL=2".

[EtienneGameSeed]

Thanks, sorry it's a typo error that was in my message (I corrected it).
What I have in the build.options.json file is :
"customBuildProperties": "build.extra_flags=-DAUDIO_INPUT_NB_CHANNEL=2"
But it doesn't work...

[facchinm]

Which board/IDE version/builder version are you targeting? I just tested that flag and it works as intended, but it also overrides the existing field (eg, if build.extra_flags was already set at -Dsomething this disappears) so you must find an unused hook.

[EtienneGameSeed]

I'm using IDE 1.8.7 and TeensyDuino 1.44.
I found why it didn't work and corrected it.
It was because there was no "build.extra_flags" in Teensy's platform.txt.
I've added it and now it works !
Thanks a lot Martino.

Here's the corrected file : platform.txt

[facchinm]

Closing this as the discussion about the metadata (or configuration) file is going to take place in arduino-cli repo and in the mailing list

[ricardojlrufino]

Closing this as the discussion about the metadata (or configuration) file is going to take place in arduino-cli repo and in the mailing list

can you leave the link here?

[daanvV]

@sleemanj Core developers can add the desired functionality now, no need for avr-gcc 5.0. See #15 (comment) and https://github.com/stevemarple/Calunium/blob/master/software/arduino-1.6/calunium/avr/boards.txt

@stevemarple , that sounds like a good idea! I've tried this for teensy 3.2 but no success sadly.

I added the following lines to the boards.txt file in C:\Users\XXXX\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.57.2:

teensy31.menu.sketchinclude.none=None

teensy31.menu.sketchinclude.custominclude=custom_include.h
teensy31.menu.sketchinclude.custominclude.compiler.c.extra_flags=-include {build.source.path}/custom_include.h
teensy31.menu.sketchinclude.custominclude.compiler.cpp.extra_flags=-include {build.source.path}/custom_include.h

Then I added the file custom_include.h to my sketch folder, and then I added the line #include "custom_include.h" to my main .ino file in the sketch folder. I also added that same line to a library in the libraries folder. I then include that library in my main .ino file.

However when I compile via the Arduino IDE (version 2) I get an error, originating from the library file custom_include.h: No such file or directory.

Any idea what could be going wrong? Am I supposed to select something from a menu maybe? If so, what menu?
Many thanks.

N.B. I realise this issue is closed, but wasn't sure where else to post this question.