SQLITE DB format question

[tstern-masten]

Hello,

I am attempting to utilize juicer and the xtce_generator to parse a cFS binary and generate a XTCE XML.

I was able to complete the auto-yamcs tutorial and I was able to create a sqlite database from juicer using my binaries but running that created database through xtce_generator has not been successful.

The error I am currently seeing is that the select module from telemetry call is failing due to no such table: telemetry which makes me think I have something formatted incorrectly.

Is there additional post processing expected for the juicer output or is this likely user error?

For reference I am on Ubuntu 20.04 and here is the sqlite db I am using newdb.zip

Let me know if any additional information is needed.

[lorenzo-gomez-windhover]

Hi there,

Yes, there is some extra steps. Our documentation should be more explicit about this, sorry about that. I'll be opening a ticket on this and update it accordingly.

Essentially the xtce tool assumes the sqlite database has been processed by https://github.com/WindhoverLabs/tlm_cmd_merger. Not sure if you noticed that is a submodule auto-yamcs just like xtce_generator.

This tool essentially merges all of the message ids(commands and telemetry) into the sqlite database. All of the messages are defined in a yaml file that is given to tlm_cmd_merger. You can find an example yaml file here for this tool.

Do notice that a lot of these tools are driven by yaml, this is intentional. We essentially capture the configuration of Flight Software such as CFS in yaml files.

So now that you have a sqlite database with all of your structs, you should do the following:

1. Take the example yaml file and modify it for your needs. This will likely mean changing message ids. Since you are using cfs, the module structure should remain similar.
2. Run the tool as stated in documentation python3 src/tlm_cmd_merger.py --yaml_path [PATH_TO_YAML] --sqlite_path [PATH_TO_SQLITE] PATH_TO_YAML will be your modified yaml file and PATH_TO_SQLITE will be the sqlite file you posted. This step modifies your database, so I suggest make a copy of it. That way if something is misconfigured, you can add your fixes to the yaml file and re-run the tool.
3. Once your database has all of the message ids for commands and telemetry, then you can run xtce_generator.

This is a bit of work upfront, but it's worth it. Once you've got a yaml that is correct (and like I said it shouldn't change too much since you are using CFS), it might require small changes every once in a while such as adding a new message id but nothing too bad. But once that is all set, you can generate xtce files in seconds. And most importantly, it is highly unlikely for your xtce definitions to be incorrect since the definitions are extracted(by juicer) directly from the binary that will run on target. Writing xtce files by hand is very time-consuming and(most important) it is error-prone since it is a person writing hundreds of bit patterns.

Anyway, hope all of this helps. I know this is a lot of information, so please reach out if you get stuck somewhere or I did not explain something clearly.

Thanks
Lorenzo

[tstern-masten]

Thank you so much for the response.

I have been configuring my YAML for our version of cFS and have made some significant headway thanks to your response.

One thing I noticed is I was lacking some symbols that I expected from structs that were never declared directly and were instead referenced in another struct. I verified that my -g flag was enabled and used during build and nm showed no knowledge of these symbols so it wouldn't appear to a problem with Juicers output.

Upon digging into the airliner code base I found a file called "to_symbols.c" for the telemetry output application which appears to declare the same structs that I am having trouble with, but are then never used further. I am curious if this was done to ensure these symbols would be generated and thus picked up by Juicer during the initial parse.

Again thank you for your response and please let me know if any additional information is needed,
Tim

[lorenzo-gomez-windhover]

Upon digging into the airliner code base I found a file called "to_symbols.c" for the telemetry output application which appears to declare the same structs that I am having trouble with, but are then never used further. I am curious if this was done to ensure these symbols would be generated and thus picked up by Juicer during the initial parse.

Yes, you are correct: these files are specifically written for that purpose. The way juicer works is that it is looking at object files, so there must be a compilation unit(which is just a .so/.cpp/.c file) which has those structs such as TO_NoArgCmd_t.

Again thank you for your response and please let me know if any additional information is needed,

Not a problem 👍 . It's really cool to see others use these tools. So we've used these tools for a while now, and there have been edge cases we've run into. One of them is having something like typdef int16 CFE_SB_MsgId_t in source code. There is a concrete example of how to fix this here https://github.com/WindhoverLabs/auto-yamcs#how_to_use_it_tuning under the section "7. Remapping Database Symbols". And all it is you have to remap the symbol via a yaml file(like this one) with https://github.com/WindhoverLabs/auto-yamcs/blob/develop/src/remap_symbols.py.

Another thing to note is that we have only tested these tools with CFS 6.5. This is not going to matter that much for the most part except for cases where you have to do that manual remapping with things like typdef int16 CFE_SB_MsgId_t as mentioned above. And it will only matter if and only if the name of symbol CFE_SB_MsgId_t has changed in the newer versions after 6.5. Which is unlikely, but plausible.

[tstern-masten]

Awesome thank you,

We are going to attempt to use v6.7 so once I have that working I will try and post that override file here for future use.

[tstern-masten]

Just wanted to say a quick thank you. I was able to get auto-yamcs to work with cFS 6.7 including the CI/TO_Lab application with your help. We plan on using this tool further in the coming months so I will be sure to document and contribute anything we can. Thank you again.