Library cross-linking doesn't work correctly

[aphelps]

Sorry about the continuing issues as I work on converting my stuff from using the Arduino IDE to platformio, but the main reason I'm trying to move it is that it was getting to complex for the IDE and I need some more flexibility (particularly the ability to pass environment variables into compilation), so I probably have a more complicated code structure than folks are usually using with an arduino.

My next issue I think boils down to libraries not being cross linked correctly. For instance if library1 uses code from library2 I'm seeing linker errors. I've made a minimal example demonstrating the problem here:
https://github.com/aphelps/platformio_test/tree/master/test_library_crosslink

With the code as is, the compilation fails:

$ platformio run
Processing arduino_nano environment:
avr-g++ -o .pioenvs/arduino_nano/firmware.elf -Os -Wl,--gc-sections -mmcu=atmega328p .pioenvs/arduino_nano/src/main.o -L.pioenvs/arduino_nano .pioenvs/arduino_nano/libTestLib1.a .pioenvs/arduino_nano/libTestLib2.a .pioenvs/arduino_nano/libFrameworkArduino.a -lm
.pioenvs/arduino_nano/libTestLib2.a(TestLib2.o): In function `TestLib2_Test()':
/Users/amp/Dropbox/Arduino/platformio_tests/test_library_crosslink/.pioenvs/arduino_nano/TestLib2/TestLib2.cpp:6: undefined reference to `TestLib1_Test()'
collect2: error: ld returned 1 exit status
scons: *** [.pioenvs/arduino_nano/firmware.elf] Error 1

This fails because the function TestLib2_Test() is in library TestLib2 which calls TestLib1_Test() in library TestLib1. However if you uncomment a call to TestLib1_Test() in main.ino that function will be compiled and the linker finds it: https://github.com/aphelps/platformio_test/blob/master/test_library_crosslink/src/main.ino#L9

[ivankravets]

particularly the ability to pass environment variables into compilation)

PlatformIO is flexible tool. You can define own build flags within 3 options:

1. build_flags
2. srcbuild_flags
3. Define global environment variable PIOSRCBUILD_FLAGS.

P.S: I'm working on the new finder for dependent library with nested logic. It will analyze all source files and find for the libs automatically and include them.

[aphelps]

Yeah, I was already playing with the build flags. Thanks for the pointer on the PIOSRCBUILD_FLAGS variable, that'll be useful for scripting what I have in mind. I have a slowly growing batch of arduino-based modules sharing similar hardware but with different configurations beyond that stored in EEPROM, so my ultimate goal is to have a build script that reads the EEPROM, sets build flags, and then builds & uploads from a single source tree (rather than the git branches I have to deal with now).

From all I've seen platformio should be great for doing this sort of thing, once the other complexities of my setup are worked out. Thanks for looking into the nested library logic. Hopefully at some point I'll get the time to fix my own issues here, I just haven't had the time to thoroughly dig through your code base yet.

[ivankravets]

@aphelps can you retest it with the latest develop?

$ pip uninstall platformio
$ pip install https://github.com/ivankravets/platformio/archive/develop.zip

Thanks a lot! If you have any problems, please reopen this issue.

[aphelps]

I did the uninstall/install and tried 'platformio run -t clean; platformio run' with the simple git repo I linked to above and it hits the same error. Can you get that example code running?

It looks like I can't re-open this issue directly, it appears only project collaborators can reopen and issue once its closed.

[ivankravets]

This issue is linked with gcc-linker and static library approach. PlatformIO builds all user/system libs to static libraries and then includes them in firmware building step. If PlatformIO didn't build source code of libs to static libs, we couldn't have a problem with "cross-linking" (because we would use -I flag).

I spent a lot of time to implement virtual "Library Dependency Finder" which has behaviour close to C Preprocessor. It allows to be PlatformIO extremely FAST! PlatformIO recompiles only changed files from your src files, not "all includes" (like Arduino IDE or etc) or libs. Try to run twice platformio run and then change some files and again platformio run.

So... If we use "static" libraries, we have to control the order for them manually. I mean under "libs order", the "order" for #include. I've just mentioned "Library Dependency Finder"... This finder can remember the correct order if you don't interfere in his work. In your example, you tried to sort includes, but didn't use anything from #include "TestLib1.h".

I recommend to read this great explanation about gcc-linker.

If any [static] library A depends on symbols defined in library B, then library A should appear first in the list supplied to the linker.

In your case, library TestLib2 depends on TestLib1. In this situation we have several solutions:

1. Just comment first include #include "TestLib1.h" in main.ino
2. Change order between includes to

#include <Arduino.h>

#include "TestLib2.h"
#include "TestLib1.h"

But... We can skip #include "TestLib1.h", because it's included inside #include "TestLib2.h"
3. Tell gcc-linker that we want enable "cyclic dependencies" between all libs (I didn't recommend to use it. Use 1 or 2 step). See modified platformio.ini

[env:arduino_nano]
platform = atmelavr
framework = arduino
board = nanoatmega328
build_flags = -Wl,--start-group

---

I understand that this sounds complicated for you after Arduino IDE. We should relay on Arduino IDE like "right behaviour". Arduino IDE has a few "non-standart" issues which break generic logic of source code building. For example, a utility folder from "libs" is automatically added to "include path" within Arduino IDE. We have 2 "include dirs":

1. Root lib dir
2. Root lib dir / utility

Sorry, I don't want to blame Arduino IDE because they simplify a lot of things. PlatformIO also tries to simplify cross-platform code building.

If you have any questions, don't hesitate to discuss here.

[aphelps]

Yeah, I'm actually very familiar with non-Arduino C development, though I admit its been a while since I've dealt directly with the linker. I'll see if I can rearrange include order in my actual project (rather than this trivial example I set up) to get it building on both the IDE and platformio. I can't guarantee that's possible though, as some of the libraries I'm using are 3rd party ones from Adafruit which have some inter-dependencies.

I think it should be a goal to be able to take a project that builds in the Arduino IDE directly to platformio with as little hassle as possible, but I do understand that the IDE does builds in weird and aggravating ways. If I was the only user of this thing I'm working on I wouldn't care as much about compatibility, but I'm going to need to be able to share this with people that may have never used the MacOS command line.

[ivankravets]

I'll see if I can rearrange include order in my actual project

You shouldn't rearrange anything. Your existing libs should work with PlatformIO without any changes.

I'm using are 3rd party ones from Adafruit which have some inter-dependencies.

Which libs do you use? Are they located within PlatformIO Library Manager? If no, I can add them.

---

1. You should include just library which is used in your project.
2. You shouldn't include "library dependencies" in your source code if they are included in "main" library. PlatformIO will scan all your code and all libs which it uses. Then, PlatformIO will automatically built lib's source code to static lib and include them in right order.
3. I understand why you bother, because you had problem with the example which is described above. I explained you why your example shouldn't work. You included "TestLib1", but never used it. Then gcc-ld decided that you no need this lib, and it "skip" it.

---

I want to promise you that Arduino-based libs are compatible with PlatformIO in 99% 😄
Let's use PlatformIO with your libs and tell me where it doesn't work.

Thanks a lot! 👍

[ivankravets]

@aphelps I've just enabled "cyclic dependencies" by default for atmelavr platform. I hope you will never have problems wit your libs :) Please re-install develop version of PlatformIO.

[aphelps]

With the "cyclic dependencies" change my actual project now compiles and uploads, thanks! This is looking awesome.

As for the 3rd party libraries, the big one is https://github.com/FastLED/FastLED. Now that I can build I can work on getting the libraries I've been using for this added to the library manager.

[ivankravets]

With the "cyclic dependencies" change my actual project now compiles and uploads, thanks! This is looking awesome.

Thanks a lot for testing 👍 I hope, I will release 0.11.0 in next week.

As for the 3rd party libraries, the big one is https://github.com/FastLED/FastLED.

Done. http://platformio.org/#!/lib/show/126/FastLED
See pull-request FastLED/FastLED#128