Add writeable virtual mounting of BDRI filesystems

[aspargas2]

This replaces the old brute-force parsing system for "mounting" ticket.db files, but maintains backwards compatibility with any scripts and tutorials assuming the old directory structure of the T: drive, aside from the hidden directory. These changes are fully functional as is; this PR is only a draft PR because there are still some remaining features that use the old brute-force ticket.db parsing method, and I wish to move away from that completely before this is merged. Feel free to go ahead and review the changes I've already pushed, as those parts shouldn't be changing any further from here.

[aspargas2]

This should now be ready for final review and merge.

[aspargas2]

Oh, I guess I should also mention this fixes #596.

[d0k3]

@aspargas2 sorry about the radio silence. I'll look though your changes this week and will hopefully be able to merge it.

[TurdPooCharger]

Does this PR rejects or filters out nonlegit or fake tickets?

If they're visible, is it possible to distinguish and separate these from system and eshop/update tickets like this?

• 000400000123ABC0.FFFFFFFF.tik

[aspargas2]

It adds a homebrew directory to the virtual ticket db mount where everything that would've gone into the eshop directory but had a bad signature goes. The old system just rejected anything with a bad signature, so the files shown in the eshop directory should be the same as with the old system, save for this code finding tickets the old system missed entirely. However, regarding system tickets, this PR simply sorts everything with a common keyY index of 1 into the system directory without checking its signature. Though I can't think of any way this could happen without intentional creation of this situation, if there was a case of there being tickets with ck_idk==1 and invalid signatures, there would be no way to distinguish them from signed system tickets just by filename. Hope that answered the question.

[TurdPooCharger]

@aspargas2, there is a specific scenario that I am worried about regarding system tickets.

Let's say there's a noob who's clueless about not knowing that it's a bad idea to randomly delete system titles or tickets off from FBI. The thought process behind this action is that those supposedly random red colored TitleIDs are not needed (haha! yikes..) because they kinda remind him of preinstalled bloatware on Windows.

Maybe this noob knows a thing or two about firmware or data recovery techniques, such as faketik app being able to restore missing tickets...

Anyway, noob keeps noobing to the point he softbricks his 3DS firmware. And to make matters worse, he never backed up a SysNAND *.bin image because (surprise, surprise!) his 3DS was hacked using a YouTube tutorial that didn't cover that subject.

If said noob was to visit Nintendo Homebrew discord, this scenario would likely play out where the locals there suggest that he fixes his 3DS with CTRTransfer.

However, if the noob was to look for help on GBAtemp, there is a good chance I might be the one he encounters where he's instead told to fix the 3DS with CTRTransfer (Type D9).

---

The reason for my asking is that there's a potential pitfall in how CTRTransfer (Type D9) currently works. Unlike standard CTRTransfer, CTRTransfer (Type D9) leaves in place the user's original ticket.db. System tickets (*.00000000.tik) from the donor ctrtransfer image are first dumped and then system tickets from the original ticket.db are dumped on top of those. These tickets later go on to be installed in FBI. Any fake system ticket that's presented to the CDN server will result in a failed system update. Not good for the end user. :/

---

I found out when system titles with missing tickets are restored with faketik, the Index to the common KeyY at offset 0x1F1 is set to 00 instead of 01.

Here's a mini tutorial on how to produce a fresh ticket.db restored with fake system tickets.

1. In the 1:/dbs, rename ticket.db → ticket.bak.
2. Boot to HOME Menu. System titles will be missing.
3. Rosalina menu inject Homebrew Launcher like this.
4. Save the Rosalina settings, reboot, and try again.
5. If injection is successful, access to Homebrew Launcher → faketik should be gained.
6. Go back to GM9, dump the ticket.db's partitionA.bin, and hex examine the fake system tickets. Note the values for Index to the common keyY.
7. Delete the (fake) ticket.db. Rename ticket.bak → ticket.db.

---

I want to be certain those changes to how ticket.db is handled retains the ability to distinguish legitimate vs fake tickets, even for system titles. This is in case tweaks to the script are required to work in tune with this PR.

When the PR encounters fake system tickets, are these also separately sorted to homebrew instead of system?

Side Note: I don't expect GodMode9 to sort out fake tickets that look convincingly real, or are mostly real but off edited by one or few points. Anyone who bothers to put that much effort purposely trying to install and pass off fake system tickets as real ones to the CDN probably at that point deserves to keep their softbrick.

[aspargas2]

@TurdPooCharger Yeah, that story you told of the noob deleting system titles is painfully familiar. The only thing about a ticket that this code uses to decide whether a ticket is said system ticket or an eshop/homebrew ticket is the Index to the common KeyY (can I just call that cky_idx?) field. This was the case with the old ticket.db code as well. When considering the potential implications of only distinguishing between legit and non-legit tickets for a cky_idx of 0, I had thought of this specific case. As you mentioned, faketik sets the cky_idx of these fake system tickets to 0, so this PR thinks they aren't system tickets at all, checks their signatures, and sorts them into the homebrew folder. I just tested the exact steps you described, and all the tickets appeared in the homebrew folder as expected.

However, the more I think about it, the more I think it makes more sense to sort all unsigned tickets (aside from those with a cky_idx > 1) into the homebrew directory, regardless of whether or not we can see it causing any issues in any practical circumstances. This would be more consistent with the behavior of the original ticket.db mounting system in any case, and would ensure no scripts would need adjustment. I would like to hear @d0k3's take on this, but if you and I are in agreement, I'll go ahead and change this behavior and push the commit, as it's a very simple logic change.

Regarding your side note, this PR will actually be able to sort out tickets whose signatures are invalid in any way, no matter how small the edit from a ticket's legit form was. GodMode9 already had a function implemented which uses the 3DS's RSA hardware to verify a ticket's signature (ValidateTicketSignature), so this PR just calls that to decide if a ticket is legit.

On an unrelated note, I've just found another bug in this PR, so be expecting a commit to fix that.

[TurdPooCharger]

Sounds good to me. I have no further questions with the cky_idx and (ValidateTicketSignature) regarding the read aspect of tickets. It is very reassuring those two measures and this PR's sorting nonvalid tickets to homebrew will keep them separate from the other legit eshop, hidden, system, and unknown. @aspargas2, thank you.

---

Now about the write aspect. Hadn't had time to compile your PR fork yet and try this on my n3DSXL, but does [CLIPBOARD] selecting *.tik files with the (Y) button, navigating to the [T:] TICKET.DB IMAGE, and pressing (Y) again in one of those subfolders allows for installing tickets?

Or, is installing tickets done by doing something like pressing (A) on the *.tik file → Ticket options... → Install ticket?

[aspargas2]

@TurdPooCharger I'm not sure if you saw this, but I removed the hidden directory in this PR, as per discussion with d0k3. Although it technically would have been possible to keep it, it didn't lend itself to being done very easily, and we didn't see a very common practical use for it.

Regarding the write aspect, you can indeed install tickets by using the Y button to copy them into the T: drive. It expects the ticket to be named in the same format as other tickets in the mount: <TitleID>.<ConsoleID>.tik. However, the console ID field in the filename is ignored and immediately changed to the console ID in the ticket itself. Similarly, it doesn't matter what folder you paste the ticket into (you can even paste it into the root of the mount); it will immediately be sorted to the appropriate folder. I did not add any additional options to the Ticket Options menu.

Here's a build of commit 226ea9c if you'd like to try any of this.

[aspargas2]

OK, that should finally be the last commit unless anyone else has things that need to be fixed or changed. Here's a build of 519f2ea.

[d0k3]

Reviewed it and found no issues - it's working great, too. So I already merged it to master. Thanks a lot @aspargas2, and thanks a lot @TurdPooCharger for always helpful comments, too.

I of course agree about all tickets with invalid signature being shown inside the homebrew folder. system tickets shouldn't be the exception.

[TurdPooCharger]

Sorry, late reporting back. Was still undergoing testing while coming up with ideas of unintended usage. I wanted to present potential pitfalls that were found and possible solutions to noob proof them.

Here's an informal write-up of build 519f2ea.

Issues Found

1. When copying backing eshop tickets, they lose their Console/DeviceID tag appearance. These will restore after unmounting and remounting the ticket.db.
   Ex: Paste as 0004000E00021800.4F81D8C7.tik → shows up as 0004000E00021800.00000000.tik
   Unmount and remount ticket.db → reappears as 0004000E00021800.4F81D8C7.tik

2. The name of the ticket <TID_HIGH><TID_LOW>.<DEVICEID>.tik should not be trusted to match what it really is.
   Renamed file name of 0004001000021200.00000000.tik → 0004001000021300.00000000.tik.
   Copy and paste into [T:] BDRI IMAGE. Name is kept as 0004001000021300.00000000.tik.

3. Following up on issue # 2, when not renaming the ticket, It is possible to double install the same ticket.
   Installed (actual) 0004001000021200.00000000.tik. Sneaky 212 renamed 213 also there too.
   Applies for any category of tickets including homebrew.

4. It is possible to install tickets named with characters that are not hex values (0-9, A-F, a-f).
   Ex: 000400000003W100.00000000.tik gets added as 0000000400000003.00000000.tik
   Solution: Only accept installing tickets that match between file names and raw hex values in those fields shown in the Structure of the Ticket diagram.

5. It is possible to inject someone else's private legit tickets.
   Ex: My n3DSXL has the DeviceID of 4F81D8C7. I can dump and install my cousin's o3DSXL tickets that are *.FB027FC8.tik.
   Discussion topics: (1) How this differs from FBI installing tickets. (2) eShop piracy and ban risks.
   Note: The DeviceIDs of my n3DSXL and cousin's o3DSXL presented here are not really those.

Strong Points / Noted observation

1. If you try to paste something renamed like trollol.tik, GodMode9 will prevent you and warn you.
   T:/trollol.tik
   Error: Cannot open destination file.

2. When trying to inject a garbage fake ticket (848 bytes filled with random hex values), the injection will fail with this message.
   Failed copying path:
   <TID_HIGH><TID_LOW>.00000000.tik

3. How does it handle when the ticket file is TOO BIG?
   When trying to inject a (mostly) real ticket but with one extra byte (849 bytes), the injection will fail with this message.
   T:/<TID_HIGH><TID_LOW>.00000000.tik
   Error: Cannot open destination file.

4. How does it handle when the ticket file is TOO SMALL?
   T:/<TID_HIGH><TID_LOW>.00000000.tik
   Error: Cannot open destination file.

5. How does it handle injecting tickets of the same <TID_HIGH><TID_LOW> but different <DEVICE_ID>?
   Already installed in the ticket.db - 0004000000021800.4F81D8C7.tik
   Try to inject this in the ticket.db - 0004000000021800.FB027FC8.tik
   T:/0004000000021800.FB027FC8.tik
   Error: Cannot open destination file.

To be tested / unanswered

1. What happens if you inject arbitrary ticket A into already installed ticket B?

• garbage (not valid format) tickets
• universal legit tickets {system titles & prebundled games from special editions}
• private legit tickets {eShop purchases, update titles}
• fake/nonlegit (valid format) tickets {faketik, TIKdevil, freeshop/related CDN downloaders, GM9, homebrew generated}

2. Following the above point 1, can ticket.db be broken from badly hex editing?

3. Test gm9 script commands and stability.

Comment / suggestion

1. That previously mentioned idea about installing tickets this way,

• <ticketname>.tik file → Ticket options... → Install ticket

, should not be implemented or considered at this time. If considered, does ease of installation outweigh the risk of bricking ticket.db? Maybe put restrictions for TitleIDs of known system tickets?

2. Talk about something along the lines of making the hex editor read only mode when viewing the tickets already installed in ticket.db. Limit allowable edits to only copying and deleting. No injecting ticket A → ticket B at this moment; trying to come up with how injection could be made to work smoothly.

[aspargas2]

@TurdPooCharger

Issues 1 and 4 are certainly bugs; I'm not sure why either of those happen, but I will look into them and get them sorted out ASAP.
My take on issues 2, 3 and 5 is this: ticket.db and title.db/variants are fully-fledged FAT filesystems, so I'm trying to treat them as such. When deciding what to allow and what to disallow for safety, I thought of it in terms of what gm9 protects you from wrongly doing to important files in ctrnand, which is basically nothing. The title ID is like a filename in this filesystem, so similarly to how you can, for example, modify the name of a .app file to not match its internal content ID or make another copy of the exact same .app file with a different filename, you can have a ticket under an improper name or two of the same ticket installed with different names. Similarly to how you can replace your LocalFriendCodeSeed_A/B or any other unique files with that of another system's, you can install a ticket that's only meant for another system. I think of copying something into the T: drive less as installing a ticker or title info entry and more as copying a file into a filesystem just like one would to any file. I understand wanting noob-proofing, but I would argue that it's no more likely that a noob will do something improper with a mounted ticket.db than with any other important system files.

With that said, there are some restrictions that disallow copying literally anything into there, as you have observed. Going through the noted strong points/observations:
1 happens because it needs some way to know what title ID to give the ticket as a "name" in the filesystem. In hindsight, I could've pulled this from the ticket data itself, but I don't see much point to allowing malformed ticket names in.
2, 3 and 4 all happen for the same reason. This code doesn't only accept one size of ticket, but it expects the second u32 of the ticket's content index to be appropriate for the size of ticket being imported.
5 happens because the console the code doesn't allow two tickets of with the same title ID "name," and the console ID isn't actually part of the name within the filesystem; it's just in the virtual name to maintain backwards compatibility.

There's no reason that injecting one ticket to another to replace the ticket shouldn't work fine, but please do let me know if you find any problems with it. At the present time, I see no reason to disallow it.
The worst hex editing one ticket could do is break that one ticket in the same way you can break any file anywhere. I should note that because of the aforementioned reason about ticket size, hexediting the second u32 of the content index is disallowed. I don't see any reason to disallow hexediting on files in the T: drive any more than there would be a reason to disallow hexediting on any other system file.
I agree that more testing of gm9 scripting functionality would be good. Though I can't see any way it could go wrong, I didn't do a comprehensive test.

I actually like the idea of also adding a context menu "install" option on ticket files. I don't know what you mean regarding potential bricking of the entire title.db. Setting aside the fact that I can't see anything one could do with one ticket that would affect any other ticket (again, this is a proper FAT filesystem), I would consider it more appropriate to put more FBI-like restrictions/warnings on such an install option. I'm all for having noob-proofing in what the noobs are going to be instructed to use; I just don't feel like it needs to be in the virtual mounting layer.

Of course, none of this up to me anyway. I'll go ahead and start working on issues 1 and 4, and if @d0k3 wants anything else changed, I'll do that too.

PS: Sorry that my posts aren't as nicely markdown-formatted as yours; I'm just bad at organizing things like that.