Side by side comparison of two models

[yassersouri]

Is there a way to get a side by side view comparing two models on the same tasks?
For example seeing which instances both models got wrong or when one got something wrong while the other didn't, etc.

Thanks for the great library.

[dragonstyle]

Right now there isn't an automated way to do this (though it is on our todo list). You can sort of set this up using the Inspect VSCode extension by just opening two logs side by side and visually comparing them (tedious but perhaps helpful).

[tadamcz]

We would also find this a very interesting feature for the log viewer. I had kind of assumed this would be too much work to be worth it, as it seems to invalidate a fundamental assumption of the log viewer. But great to hear it's on the todo list! Maybe it's easier than I thought.

[dragonstyle]

It isn't easy, unfortunately, but I think we'd still like to do it!

[yassersouri]

If you wanted to approach this problem how would you do it? Can you break down this feature into 4-5 high-level tasks so that an outsider can contribute?

…

On Tue, Feb 18, 2025 at 5:37 AM Charles Teague ***@***.***> wrote: It isn't easy, unfortunately, but I think we'd still like to do it! — Reply to this email directly, view it on GitHub <#1327 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGSB7ONVEY2LIMPIPGVCD2QMZSRBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE3TCOBWGY2TAMJZQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSHA2TKOBZHE4DOMVHORZGSZ3HMVZKMY3SMVQXIZI> . You are receiving this email because you authored the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[dragonstyle]

Thanks, this is a great suggestion. I'm right in the middle of quite a bit of refactoring of the main application logic which is necessary to tackle a feature like this (and will cause a lot of merge issues as I work through it). Let me get through this work and then I'll revert here with some suggestions as I see a good path!