Query oper state

[choppsv1]

In FRR we're building a centralize mgmt daemon.. Config for the system is stored in the daemon and sent to the protocol daemons based on matching a path prefix of interest. For oper-state however it's not workable to store operational state in a database as it's constantly changing (and may be quite large). So we need to query for it.

We've got a basic system setup using path prefixes as with config to send xpath queries to the protocol daemons, and in the protocol daemon we have meta-data we attach to the libyang schema nodes that contain callbacks to fetch data. So we walk the query xpath and use a combination of the callbacks get_elem() (some leaf value) and get_next(), get_keys() (get list elmeents and each elements keys) to build a operational data tree up.

It would be nice if there were a way to integrate this sort of thing into libyang though, the primary reason being to better handle xpath queries. We currently only support simple "path" style queries, no wildcards, no functions etc..

The minimum help I think we could use, and it would be very helpful, is some help with xpath parsing and predicate evaluation. So when e.g., we reach a list and the user has specified some predicate we can walk the elements of the list and only use those which match the predicate. This is going to be very important e.g., when we have 1M routes and the user is trying to look at a subset.

Currently the xpath functionality is not public, it would be useful if part of it were made public, and specifically support for the above predicate testing. Maybe also walking the schema tree based on the xpath query would be nice too.

Does any of this sound doable? Has it already been done somewhere else (e.g., sysrepo/netopeer?)

[michalvasko]

I think I got the general idea but am not sure about the exact functionality you need, maybe try to illustrate on an example. Regarding the XPath functionality, it is all private on purpose but once I understand the use-case some new public functions may be an option.

Naturally, some XPath parsing was implemented mainly in sysrepo but having an internal implementation seemed like the better and cleaner solution than adapting libyang API. You can also look at its XPath API. It really simplifies parsing and is nice and easy to use but I have kept it in the API only for backwards compatibility because it is quite inefficient so it is not used by any of our projects. Just another reason why implementing XPath parsing tailored to your needs may be the best solution.

[choppsv1]

So, list schema nodes have lookup(keys) to get a specific entry (returned as an opaque object ptr representing the entry), get_next(parent, last_entry) which iterates the list by passing the parent object ptr and the previous object ptr, it returns the next opaque object ptr in the list or NULL for done, get_keys(entry) which takes the opaque entry ptr and returns the keys values that it represents. Finally, leaf schema nodes have a get_elem(entry) callback which take the opaque object ptr for a list entry and return the value for a leaf data node .

The list get_next(parent, entry) is given 2 values the previous opaque object ptr it returned and also the last parent list object ptr if there was one (so a list inside a list)

So I have a user input xpath. I use that to obtain schema node set using lys_find_xpath. For each terminal schema node then I can start at the root and walk back to that schema node. Let's walk the schema path then:

• For container nodes I can just build the data node myself, and descend into the child schema node.
• For a list node I use get_next(parent, entry) to obtain the next entry value (think an interface object, or a route object)
• For each list entry I descend into it's child schema nodes
• Repeat if chils is container or list
• Otherwise it's a leaf node and I use the schema node's get_elem(args) to obtain the leaf value.

That's a generic walk. Now what I'd like to have is a way at each list node to take the opaque entry ptr, the schema node and an xpath parsed object (I guess?) and determine if this entry should be included based on the xpath object and it's position in path (given by schema node). I have get_keys(entry) that will give me the key values for the entry that I could pass to a xpath matching routine, but I also could use the walking algorithm from above to resolve sub-xpaths too if the user xpath utilized complex predicates (e.g., functions or value comparisons for non-key leaf nodes, or relative paths?)

Currently we use the user xpath as a prefix to start iterating downward from. So we use lyd_new_path2() to obtain a data node tree and build a path from root to the node using that. We walk that data node path using our schema node callbacks, but for list entries we have the key values so we use lookup(keys) to get a specific list entry object. When we reach the end of that path if it's a leaf we are done if it's a container or list entry we use the walking algorithm from above to add all the child values to the data tree.

To support real user xpaths now all I can think to do is recurse with the lyd_new_path2() above, I start with the full user xpath but if lyd_new_path2() doesn't resolve I chop off the top element of the user xpath and try again, I repeat this until i get a resolution to a node. I then use that and my walking algorithm to build a data tree (which might be huge for something like routing table with 1M routes). Then I use lyd_find_xpath() and the users original xpath on the resulting data tree to extract a ly_set of matching nodes. Then I use this ly_set to trim the data tree according to that set, then I can JSON print this trimmed tree to give the user their results. This is not going to scale though.

[choppsv1]

Perhaps there could be a lyd_xpath_element_is_present(xpath, tree, scnode, keys, value, add_dnodes_cb, userdata) and I pass it the xpath object, the current data tree I've built, the schema node of the element (leaf or list entry) I'm working on, and the key of a given entry or value if a leaf, plus a callback function ptr add_dnodes_cb(). lyd_xpath_element_is_present() would return true or false based if the element should be included, it could use the add_dnodes_cb(xpath, userdata) callback to passing it another xpath to have more nodes added to the tree that it might need to make the true/false decision.

[choppsv1]

In fact, unless I've missed something, this lyd_xpath_element_is_present function could be used starting from the root schema node to actually drive the entire construction of the resulting data tree based on the xpath.

[michalvasko]

So you have described your solution, which I may have vaguely understood but I am missing the high-level description of the problem, add that please. Only that way I can make certain there is no solution using the existing API or some simple proprietary algorithm.

Well, you have described something in the original post but it seems complex and I just do not understand all the details that I need to, you can try describing the input (XPath?) and the expected output.

[choppsv1]

OK not sure which part is the confusing part so I'll describe our current process, again and maybe you can tell me where you're losing it.

All schema nodes have private FRR data which includes callbacks I mentioned previously get_elem for non-list and lookup(keys), get_next(), and get_keys(), which obtain actual data from the process. The arguments they are passed include object pointers from parent list entry callbacks. These object pointers are basically the actual concrete objects in FRR like interface objects or route objects. so e.g., the get_elem() callback for the "speed" leaf of an interface object is going to be passed the interface object pointer that it's parents get_next() returned

Ok so far? Just try and imagine how to build state using callbacks from schema nodes and a path.. there's really no other way to do this.

Ok so the process: We get an xpath from the user it points at a node call it D

First we need to get the nodes on the branch leading to D so:

1 - We call lyd_new_path() - this creates a small data tree. We walk back from D to the root gathering up the key values from each list node we encounter.

2 - We walk the schema nodes from the root back to D using callbacks to obtain the internal object pointers (described at the top) for the list entries along that branch.

3 - we now are back at D and we start a full walk of the schema nodes down to the leafs,, using the callbacks to obtain the data to return to the user.

This is what we currently have, is that good enough to understand the process? If not, please ask a targeted question on what doesn't make sense. :)

[michalvasko]

What I understand from this is that you have a solution that splits a full YANG data tree into individual schema nodes with their private data that allows creating the actual data nodes. The problem you are now having is that when you have an XPath, you somehow need to traverse all the relevant schema nodes and get their data nodes that ultimately create the full data tree selected by the XPath that you can return.

Assuming this is correct, my question is why have you chosen this solution? Why are all the data fragmented based on their schema nodes? You have initially mentioned a centralized daemon and that is probably what needs to collect all the operational data chunks from all the managed daemons to satisfy some query.

I am still missing the actual use-case, I can only think of a NETCONF (RESTCONF, or whatever) get RPC with an XPath filter and you are trying to provide the data for the reply or something similar. You have mentioned only an XPath and maybe it is not actually relevant. In any case, the first thing that comes to my mind is using sysrepo for this. You have all the daemons subscribed for providing their operational data and when you call sr_get_data() with the XPath filter, it is up to sysrepo to collect them all and provide the correct data. Now, without the high-level details about the problem I do not know whether this is feasible and why not in case it is not.

[choppsv1]

What I understand from this is that you have a solution that splits a full YANG data tree into individual schema nodes with their private data that allows creating the actual data nodes. The problem you are now having is that when you have an XPath, you somehow need to traverse all the relevant schema nodes and get their data nodes that ultimately create the full data tree selected by the XPath that you can return.

Yes. The entire point is to obtain the state that matches the xpath, and to do so efficiently (i.e., not by getting all the state and then filtering, but just calling the callbacks for the data that will match the xpath). So basically have an xpath driven walk through the schema tree obtaining the data nodes that correspond.

Assuming this is correct, my question is why have you chosen this solution? Why are all the data fragmented based on their schema nodes?

I don't understand this question, FRR management is based on YANG models (config and oper-state) how else would we organize it than by using the YANG model?

You have initially mentioned a centralized daemon and that is probably what needs to collect all the operational data chunks from all the managed daemons to satisfy some query.

That's correct, mgmtd is the centralized management daemon. It has path prefixes it associate with each daemon to know where to send YANG config as well as how to query for oper-state. mgmtd calls those daemons "backend" clients (e.g., ospfd, ripd, bgpd...). mgmtd` also has "frontend" clients like a CLI and eventually NETCONF and RESTCONF servers that modify the configuration and query for operational state.

I am still missing the actual use-case, I can only think of a NETCONF (RESTCONF, or whatever) get RPC with an XPath filter and you are trying to provide the data for the reply or something similar.

Or CLI, but yes some (frontend) client is requesting state from our YANG models.

You have mentioned only an XPath and maybe it is not actually relevant. In any case, the first thing that comes to my mind is using sysrepo for this. You have all the daemons subscribed for providing their operational data and when you call sr_get_data() with the XPath filter, it is up to sysrepo to collect them all and provide the correct data. Now, without the high-level details about the problem I do not know whether this is feasible and why not in case it is not.

We are sysrepo, I mean that mgmtd is doing the exact same job. Are you saying that se_get_data() queries other processes for YANG modeled data, whenever a new request is made? I couldn't find this when I looked. It seemed to me that it was more of a push than a pull system.

Concrete example that works currently:

root@r1:/# vtysh

Hello, this is FRRouting (version 9.1-dev).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

r1# show mgmt get-data-tree /frr-interface:lib/interface[name="r1-eth0"]
{
  "frr-interface:lib": {
    "interface": [
      {
        "name": "r1-eth0",
        "vrf": "default",
        "state": {
          "if-index": 2,
          "mtu": 1500,
          "mtu6": 1500,
          "speed": 10000,
          "metric": 0,
          "phy-address": "72:4b:24:9b:72:5e"
        },
        "frr-zebra:zebra": {
          "state": {
            "up-count": 0,
            "down-count": 0
          }
        }
      }
    ]
  }
}
r1#

This is based on the model https://github.com/FRRouting/frr/blob/master/yang/frr-interface.yang which the zebra model augmenting https://github.com/FRRouting/frr/blob/master/yang/frr-zebra.yang

This works fine and is very efficient using the process I described previously.

What we don't have is if that xpath query is using anything more the explicit keys in the predicates, then that xpath doesn't point at anything specific node we can instantiate and grab the keys from the branch and walk downwards from, we need the xpath itself to help drive the data collection.

BTW I mentioned maybe a single function earlier that might drive this process, but this won't work b/c we have to break the iteration up so other code in the daemon can run in between.

Again, I think what we need is a function I can pass a partially built data tree to (what i've built so far), a specific schema node from that tree (where I last stopped), and it then tells me which schema nodes I need to collect data from next in order to fulfill the xpath query.

[choppsv1]

BTW mgmtd is sending the xpath query string to each "backend" client. The walking of schema tree and calling callbacks to get data is all happening inside the backend (daemon) process, it then returns the YANG modeled data to mgmtd which merges all the replies it gets.

I only mention this to make sure you're not thinking mgmtd is trying to walk the tree or something weird like that.

[choppsv1]

Here's a concrete example of something we don't support but want to: "show all interfaces in the "default" vrf"

r1# show mgmt get-data-tree /frr-interface:lib/interface[./vrf="default"]

vrf is not a key it's just a leaf value under the interface node.

[michalvasko]

We are sysrepo, I mean that mgmtd is doing the exact same job. Are you saying that se_get_data() queries other processes for YANG modeled data, whenever a new request is made? I couldn't find this when I looked. It seemed to me that it was more of a push than a pull system.

Well, you copying the functionality of sysrepo and requiring some new functions to do so does not make me too keen on adding them. And yes, you have described what it is doing exactly, sysrepo has lots of features and specifically for operational data, it supports both push and pull data (mentioned clearly in the docs).

Okay, so I think I understand what you are trying to do and sysrepo attempts to do something similar, it also faces the problem of how to effectively collect the operational data without getting filtered-out subtrees. You can look at the function yourself but despite being only text-based, I think it supports a few more use-case than what your solution does. The idea is that whenever a new common XPath is found that does not work efficiently, support for it can be added. Additionally, it relies on the applications to subscribe to small subtrees of data because the only decision that can be made is whether to get data for a subscribed subtree or not.

My belief is that what you are attempting (supporting "all" the XPaths) is pretty much impossible and that is the reason we have taken the opposite approach (support only common XPaths and iteratively add to them). XPath is incredibly complex and attempts at algorithms similar to what you described can only end in failure, I have enough experience from libyang. In other words, I will not provide any assistance nor new functions in libyang.

So my suggestion is for you to explore sysrepo and try using it instead of your current solution and provide feedback. I see no reason to implement the same thing again when we can rather improve what exists already.

[choppsv1]

sysrepo is not a solution for us. Indeed, support for it was even added to frr and it rotted on the vine.

So if one wants to do something that sysrepo does, but needs support from libyang, the answer is to take a hike?

I do not think this is impossible but clearly from your reponse we have to look elsewhere for a solution.

[I mean I describe a function that can implement this functionality, so clearly I don't see it as impossible...sigh]

[choppsv1]

I mean this answer is really frustrating. Our mgmtd is a huge project with a lot development time invested in it it's not just a sysrepo clone, it's constructed specifically for our project needs. Being told "that sounds like our sysrepo so just use that" is really frustrating blatant ignoring of what that actually would mean to us.

[michalvasko]

Could you at least elaborate, why sysrepo is not a solution for you and what your mgmtd does in addition to it? Also, unless you can justify adding complex functionality you must not be surprised it will not be added, it is going to be us to maintain it further and it is our time that will be spent on this, impossible to know how much.

And if you are so certain you have described the new function so clearly and is so simple that go ahead and implement it, if the PR looks clear enough I will have no problem merging it.

[choppsv1]

I've spent a lot of time describing mgmtd in this thread.. As you aren't interested in helping at this point I don't want to keep doing design presentations. mgmtd is a solution for us that allows us to transition over to it in steps, utilizing our existing YANG infrastructure, supporting our legacy CLIs, and optimized exactly for our use case.

I didn't start off asking for anything complex I: Currently the xpath functionality is not public, it would be useful if part of it were made public, and specifically support for the above predicate testing. asked for the APIs to be made public, exactly so I could use them to implement the function I needed.

[michalvasko]

We have stopped moving towards any solution and I have too much other work to continue this way. So, based on your description, you want to learn what schema nodes are referenced in XPath predicates, so just get the standalone predicates and call lys_find_xpath() or lys_find_xpath_atoms(). If those are not suitable then like I said, provide a PR with the functionality you need.