Restore phandles from binary representations #151

ukleinek · 2024-10-21T07:45:40Z

device trees generated with options -@ and -L contain information about the label names used originally and which values are phandle values. This information can be reused when compiling back to dts format to make the result much more human readable.

Also dtdiff is adapted to make use of this to generate smaller diffs without hunks e.g. for a different phandle allocation.

This superseeds pull request #93.

If the input has a __symbols__ node, restore the named labels for the respective nodes. Signed-off-by: Uwe Kleine-König <[email protected]>

If -L is given to dtc, use the information about phandle references in the __local_fixup__ node to restore phandles. Signed-off-by: Uwe Kleine-König <[email protected]>

Signed-off-by: Uwe Kleine-König <[email protected]>

With the phandles restored a difference between two device trees that only consists of a different allocation of phandle values is shown without all the followup changes that usually are not interesting. Signed-off-by: Uwe Kleine-König <[email protected]>

ukleinek · 2024-10-21T08:02:34Z

A possible followup improvement would be to restore phandles from __fixup__ .

ukleinek · 2024-10-24T06:11:09Z

Just a quick headsup: I found a dtbo file where my code crashes with -@ -L. So don't merge yet please.

ukleinek · 2024-10-24T06:29:09Z

OK, I think that's unrelated to my changes and just presents a situation to the decompiler that didn't happen before. Here is a reproducer:

$ git diff
diff --git a/treesource.c b/treesource.c
index 067647b60a17..fa1a48579489 100644
--- a/treesource.c
+++ b/treesource.c
@@ -104,6 +104,12 @@ static void write_propval_string(FILE *f, const char *s, size_t len)
 static void write_propval_int(FILE *f, const char *p, size_t len, size_t width)
 {
        const char *end = p + len;
+
+       if (len % width) {
+               fprintf(stderr, "Huh, len = %zu, width = %zu\n", len, width);
+               exit(1);
+       }
+
        assert(len % width == 0);
 
        for (; p < end; p += width) {

uwe@taurus:~/dts-decompile-labels$ cat test3.dts
/dts-v1/;

/ {
	tralala = <&somelabel>, "text making the next label unaligned",
		<&somelabel>, "some more text";

	somelabel: somenode {

	};
};
$ dtc -@ -L -I dts -O dtb test3.dts > test3.dtb
$ dtc -@ -L -I dtb -O dts test3.dtb 
/dts-v1/;

/ {
Huh, len = 37, width = 4
	tralala = <&somelabel

So I think some heuristic considers the property "tralala" an array of ints but that doesn't handle that it ends at offset 41 which is unaligned.

dgibson

I think this is a great idea. The implementation looks mostly sound but there are a few minor issues that need addressing.

dgibson · 2024-10-28T09:02:40Z

dtc.c

 		generate_label_tree(dti, "__symbols__", true);
+		generate_labels_from_tree(dti, "__symbols__");


I'd not immediately sure if it make sense to only do this when generate_symbols is set.

dgibson · 2024-10-28T09:05:52Z

livetree.c

+		if (labeled_node)
+			add_label(&labeled_node->labels, p->name);
+		else if (quiet < 1)
+			fprintf(stderr, "Warning: Path %s referenced in %s missing",


I'd suggest suggest an explicit "referenced in symbol %s" to make it clearer what's going on without context.

dgibson · 2024-10-28T09:06:48Z

dtc.c

@@ -344,6 +344,7 @@ int main(int argc, char *argv[])
 	if (generate_fixups) {
 		generate_fixups_tree(dti, "__fixups__");
 		generate_local_fixups_tree(dti, "__local_fixups__");
+		local_fixup_phandles(dti, "__local_fixups__");


Again, not sure if it only makes sense to do this with -L.

dgibson · 2024-10-28T09:08:33Z

livetree.c

@@ -3,6 +3,8 @@
 * (C) Copyright David Gibson <[email protected]>, IBM Corporation.  2005.
 */

+#include <libfdt/libfdt.h>
+


Oof. We don't currently use libfdt within dtc, which is sort-of deliberate (they provide a cross-check to each other). There's a case to be made to drop that policy and instead use libfdt for much of the flat tree logic within dtc. However, I'd really prefer to do that as an overall change, using it for as much as we can, rather than adding small pieces of libfdt usage ad-hoc.

..also, I can't actually see what you're using this for.

Maybe this is only a development relict, will try without.

dgibson · 2024-10-28T09:12:23Z

livetree.c

+		}
+
+		/*
+		 * Each property in the __local_fixup__ tree is a concatination


Nits: 'concatenation' and __local_fixups__ .

dgibson · 2024-10-28T09:15:15Z

livetree.c

+		}
+
+		for (i = 0; i < lfp->val.len; i += sizeof(fdt32_t))
+			add_phandle_marker(dti, p, fdt32_ld((fdt32_t *)(lfp->val.val + i)));


Not a big deal, but I'd prefer to use i += 1 with the loop limit decreased correspondingly. Then you can cast lfp->val.val to fdt32_t * once and just index into it with i.

dgibson · 2024-10-28T09:16:18Z

livetree.c

+
+		if (!subnode) {
+			if (quiet < 1)
+				fprintf(stderr, "Warning: subnode %s in %s referenced in __local_fixups__ missing\n",


For properties you do need to give the property name and path separately. However for subnodes you might as well just stick them together "node %s/%s referenced in local_fixups ...".

dgibson · 2024-10-28T09:19:28Z

treesource.c

+			m->ref = refn->fullpath;
+	} else if (quiet < 1) {
+		fprintf(stderr, "Warning: node referenced by phandle 0x%x in property %s not found\n",
+			phandle, prop->name);


I think you do need to consider __fixups__ as well. Without that, this error will be triggered if you're decompiling a dtbo with any external phandle references.

dgibson · 2024-10-28T09:20:57Z

dtdiff

@@ -17,7 +17,7 @@ source_and_sort () {
 	    *.dts)
 		IFORMAT=dts
 		;;
-	    *.dtb)
+	    *.dtb|*.dtbo)


Even something this simple needs some sort of commit message: why is this change desirable? (I can see why, but someone reading the commit log in the future might not).

dgibson · 2024-10-28T09:22:59Z

A possible followup improvement would be to restore phandles from __fixup__ .

I think you do need to implement this. For one thing processing but __local_fixups__ but not __fixups__ would just be surprising behaviour to a user, I think. Secondly, it could cause some bogus errors in some cases (see detailed comments).

ukleinek · 2024-10-28T09:29:26Z

Do you have an idea for the unaligned phandle problem?

dgibson · 2024-10-28T09:43:17Z

Do you have an idea for the unaligned phandle problem?

Sorry, I'm not sure exactly what problem you mean.

ukleinek · 2024-10-28T14:08:06Z

Do you have an idea for the unaligned phandle problem?

Sorry, I'm not sure exactly what problem you mean.

I mean the crash reported in #151 (comment) with the follow up in the next comment.

dgibson · 2024-10-30T02:07:48Z

Do you have an idea for the unaligned phandle problem?

Sorry, I'm not sure exactly what problem you mean.

I mean the crash reported in #151 (comment) with the follow up in the next comment.

Ah, right, sorry.

I think the basic problem is that previously the only case where we'd have phandle markers during decompile is with -I dts -O dts, in which case we'd also have type markers helping us figure out how to print each part of each property. The new code is putting in phandle markers, but without type markers. We should handle that, of course, but looks like there are some edge cases.

More specifically, it looks like write_propval() calls guess_value_type() to figure out a missing type only once for the whole property. Having a phandle marker does tell us that those specific 4 bytes should be in integer < ... > context. But we'll need to separately guess the type for each chunk between phandles. I suspect that might be irritatingly fiddly, but I'm afraid it's going to be a prerequisite for implementing this.

Uwe Kleine-König added 4 commits October 21, 2024 09:44

dtc: Restore labels from __symbols__ node

5636db0

If the input has a __symbols__ node, restore the named labels for the respective nodes. Signed-off-by: Uwe Kleine-König <[email protected]>

dtc: Restore phandle references from __local_fixup__ node

4486645

If -L is given to dtc, use the information about phandle references in the __local_fixup__ node to restore phandles. Signed-off-by: Uwe Kleine-König <[email protected]>

dtdiff: Use input format dtb for dtbo files

4e296a2

Signed-off-by: Uwe Kleine-König <[email protected]>

ukleinek mentioned this pull request Oct 21, 2024

Device tree overlays cleanup analogdevicesinc/linux#2624

Merged

ukleinek mentioned this pull request Oct 21, 2024

Some phandle related improvements and some bug fixes #93

Closed

dgibson requested changes Oct 28, 2024

View reviewed changes

dgibson mentioned this pull request Oct 30, 2024

Improve dtdiff by avoiding or nulling the phandles #145

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore phandles from binary representations #151

Restore phandles from binary representations #151

ukleinek commented Oct 21, 2024

ukleinek commented Oct 21, 2024

ukleinek commented Oct 24, 2024

ukleinek commented Oct 24, 2024

dgibson left a comment

dgibson Oct 28, 2024

dgibson Oct 28, 2024

dgibson Oct 28, 2024

dgibson Oct 28, 2024

ukleinek Oct 28, 2024

dgibson Oct 28, 2024

dgibson Oct 28, 2024

dgibson Oct 28, 2024

dgibson Oct 28, 2024

dgibson Oct 28, 2024

dgibson commented Oct 28, 2024

ukleinek commented Oct 28, 2024

dgibson commented Oct 28, 2024

ukleinek commented Oct 28, 2024

dgibson commented Oct 30, 2024

		generate_label_tree(dti, "__symbols__", true);
		generate_labels_from_tree(dti, "__symbols__");

Restore phandles from binary representations #151

Are you sure you want to change the base?

Restore phandles from binary representations #151

Conversation

ukleinek commented Oct 21, 2024

ukleinek commented Oct 21, 2024

ukleinek commented Oct 24, 2024

ukleinek commented Oct 24, 2024

dgibson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgibson commented Oct 28, 2024

ukleinek commented Oct 28, 2024

dgibson commented Oct 28, 2024

ukleinek commented Oct 28, 2024

dgibson commented Oct 30, 2024