A nbd client instead of custom piece

[bignaux]

I still not able to do a proper read but i've not error when i use a fake read like in hdd_fake driver.
Since i get no answer on #410, i take the roomspace of incomplete hdldsvr to write this prototype.
I'd like to publish lwNBD in upstream LWIP, so i choose same licence as there.
There rooms for improvement of course, and performance gain could be done via completing the protocol, like trim option NBD_CMD_WRITE_ZEROES ... ) , rewrite as a non-blocking event loop and so.

I really think this kind of standardized server would be a great enhancement and fun place for next development.

Pull Request checklist

Note: these are not necessarily requirements

• I reformatted the code with clang-format
• I checked to make sure my submission worked
• I am the author of submission or have permission from the original author
• Requires update of the PS2SDK
• Requires update of the gsKit
• Others (please specify below)

Pull Request description

[AKuHAK]

@bignaux Dont forget to run make format before pushing the PR, to keep formatting unified. Now formatting is a bit messy: https://github.com/ps2homebrew/Open-PS2-Loader/pull/415/checks?check_run_id=2464648228

[rickgaiser]

Hi @bignaux , I see you're making great progress on your NBD server project.

I'd like to publish lwNBD in upstream LWIP

That's a great idea. I see you've already created a standard interface to the block devices: struct nbd_context. I think the interface should be defined inside a header, and all implementations (hdd_fake, hdd_atad, hdd_dev9, etc....) should be in separate files.

I've been working on block devices for a long time (usbmass_bd, IEEE1394_bd, mx4sio_bd), all using a standard block device interface: struct block_device. Note how similar these two interfaces are. I think an NBD -> BDM_block_device implementation would be great for accessing all BDM based block devices from a PC.

From Linux you can mount NBD devices, and use them as if they are local hdd. But this does require Linux to know the file system that is used on the PS2. So for USB, IEEE1394 and mx4sio this would be easy, as they all use fat32. But for the HDD, you'll need to know PFS, so special tools (or a linux PFS file system driver) are required.

I really think this kind of standardized server would be a great enhancement and fun place for next development.

I do too ;).

EDIT: PR changed to draft becouse its still a work in progress.

[bignaux]

@AKuHAK : I just posted there to try to get help for the read issue, it was not really an attempt to get things merge. Btw, i'd like to keep for example nbd-protocol.h as verbatim (nbdkit reference).

@rickgaiser : i'm active contributor on pfsshell & hdl_dump . We need this kind of server on the new toolchain ( see ps2homebrew/hdl-dump#33 ). I hope this non-platfom specific server will find its user outside PS2 world, RPI pico for example use LWIP stack and could enjoy NBD to export usb mass storage instead of an heavier protocol.

Concerning my issue, that's a view of first 50MB of my hdd, compare from localdump and netdump. bad block in red (using jessek/colorize)

reproduice :

touch myps2
nbdfuse myps2 nbd://192.168.1.45 &
dd if=myps2 | pv | dd of=dump count=10000
fusermount -u myps2

[bignaux]

Thanks to @rickgaiser , the bug was found and now READ is fixed. Let's go on WRITE.

[bignaux]

I think we have something mergeable now, please test and review.

[rickgaiser]

I've tested the NBD server. From linux I can access the PS2 internal hdd, using nbdfuse. With nbdfuse I can access the drive (blocks of data) but I cannot access the files. I remember you saying you transferred a game, what tools can I use to transfer games to/from hdd using NBD?

The speed is currently around 950kb/s. So transferring a 4GB game would require more than 1 hour. Filling a 2TB drive would require 23 days... I hope this can be increased a lot, becouse I'm not that patient ;-). The lwIP stack can do at least 4MB/s, some have gone to 6MB/s but I think that's with UDP (not TCP).

Anyway, for this PR to be approved it only needs to be usable, not fast. So if there's some client users can use to transfer games then I think it's ready for merge.

[uyjulian]

hdl_dump and pfsshell could be used on the exposed block device…

[bignaux]

Yeah it's slow to copy an entire collection of save games. But that's not the only purpose of such server. I'm writing an OPL manager tool, to have multi-platform batcher, you can see an example of script to update cover https://gist.github.com/bignaux/49259df0a9bb99e964891f9e62f7af8b#file-opl-update-art . Someone is working on the mc driver for NBD, and most are 8MB ... I'll use nbd to expose some part of the hdd (with dmsetup) and can format it with mkfs and use it as a NAS ... For now, it's only a replacement of hdldsvr and hdl-dump server, and that's already good to close a bunch of bugs.

[bignaux]

Close :

• WIP: hdl_server now compiles on ps2dev:latest hdl-dump#33
• hdl_svr_093.elf looks for hard ipconfig.dat paths from both Memory Cards Slots 1/2, but not the HDD, ever. hdl-dump#27
• can' build the server with latest ps2sdk (from git) hdl-dump#2
• hdldsvr : status and perspective #410
• [FR] udpbd support #325

[rickgaiser]

Great! I'm looking forward to all new possibilities this will bring.

[Apocalypse612]

Thank you!