piolib: support transfer lengths that don't fit in 16 bits

[jepler]

struct rp1_pio_sm_xfer_data_args {
    uint16_t sm;
    uint16_t dir;
    uint16_t data_bytes;
    void *data;
};

The structure used for data transfers is limited to a 16-bit count of data_bytes.

Our application would like to do larger transfers. Right now we are working around this limitation by submitting several buffers of data sequentially, but it would be preferable if we could submit all the data in one go:

constexpr size_t MAX_XFER = 32768;

void pio_sm_xfer_data_large(PIO pio, int sm, int direction, size_t size, uint32_t *databuf) {
    while(size) {
        size_t xfersize = std::min(size_t{MAX_XFER}, size);
        int r = pio_sm_xfer_data(pio, sm, direction, xfersize, databuf);
        if (r) {
            perror("pio_sm_xfer_data (reboot may be required)");
            abort();
        }
        size -= xfersize;
        databuf += xfersize / sizeof(*databuf);
    }
}```

[pelwell]

That's a lot of data - what's your use case?

I'm not against in principle, it's just a matter of not breaking anything in the process. It might be that, due to the alignment requirements, switching to uint32_t would just work.

[jepler]

I'm driving HUB75 matrices with PIO. Ideally I'd submit a whole frame with one ioctl, but even a 64x32 matrix with 10 bitplanes exceeds 64kB in the encoding I'm using. And folks like to use several of these matrices in daisy chain mode.

Right now I'm just submitting 32kB at a time and trying to determine whether some artifacts I'm seeing are due to a flaw in my code or are due to a pause in the bitstream between ioctls. (a smaller 32x16 display that just needs one data submission per frame doesn't have these artifacts)

[jepler]

(@ladyada ping for interest)

[pelwell]

There are now two PRs for this: one for piolib (#108), and one for the kernel driver (raspberrypi/linux#6543). Both are needed for larger transfers to work. The library should simply fail to do larger transfers with a non-updated kernel.

[jepler]

Thanks!

[pelwell]

That's all been merged now. The driver change missed yesterdays rpi-update release, but you can install a kernel containing the updated driver using either sudo rpi-update pulls/6543 or sudo rpi-updates rpi-6.6.y.

[jepler]

This appears to be working, thanks! I don't know if this issue should be closed now, or if it ought to wait for a released kernel to support it?

[pelwell]

The kernel will be released as sure as night follows day, so let's close this now.

[henryjliu]

Is there something else I need to do to make this work besides sudo rpi-updates rpi-6.6.y and compiling?

I made a sample:
#define BUF_SIZE 32768*2
uint8_t databuf[BUF_SIZE];
pio_sm_config_xfer(pio, sm, PIO_DIR_TO_SM, BUF_SIZE, 1);
pio_sm_xfer_data(pio, sm, PIO_DIR_TO_SM, BUF_SIZE, databuf);

This works fine for BUF_SIZE 32768 but BUF_SIZE 32768*2 seg faults first time and locks the os requiring power cycle the 2nd time.

[jepler]

I think pio_sm_config_xfer is still limited to values that fit in 16 bits. Try with your xfer configured for 32kB as before, but then when it comes to the actual xfer send your whole BUF_SIZE.

[henryjliu]

This works now if I set pio_sm_config_xfer(pio, sm, PIO_DIR_TO_SM, 32768, 2);//this is max size but
pio_sm_xfer_data(pio, sm, PIO_DIR_TO_SM, 32768*4, databuf);

though I'm confused what pio_sm_config_xfer(pio, sm, PIO_DIR_TO_SM, 32768, 2) does exactly. Does it just basically split into multiple 32768 transitions or is this number just ignored?

[jepler]

The kernel code handling this API call is https://github.com/raspberrypi/linux/blob/rpi-6.6.y/drivers/misc/rp1-pio.c#L606

(but be aware there's some translation of arguments between the userspace pio_sm_config_xfer call and the arguments here, where the truncation to 16 bits of the config_xfer length argument occurs)

From what I understand reading it, this allocates DMA buffers in the Linux kernel's address space. The numbers given are the buffer size & the count of buffers, and there can be up to 4 buffers.

When the transmission actually occurs, data is copied from userspace to the buffer, and then the dma is submitted (https://github.com/raspberrypi/linux/blob/a18d9ced4965462cb7b3b4252ada440395105308/drivers/misc/rp1-pio.c#L726). When the number of buffers is greater than 1, more an one dma can be submitted, which likely reduces the gaps between the dma transmissions.

Reasons for using fewer or smaller buffers? I'm not sure. Linux DMA-capable memory might be limited, and the returns for larger buffer sizes seemed to be modest in my own testing. However, I was throughput-oriented in my testing (though I would have expected 5ms gaps as you noted in #117 to be apparent in my application)

[henryjliu]

I've been thinking about how to do it to implement a ping pong buffer and I guess way one way could be use 2 sm. First reads in number of bytes to shift with mov x, num_bytes and then does decrement until end reached. Afterwards can wait on irq 0 and sets irq1. Second sm waits on irq 1 and does same thing. Irq is not available in userspace but seems assembly could should still work for interprocess communication shouldn't it?

[pelwell]

There are two PRs - one in the kernel (raspberrypi/linux#6640) and one here (#119) - that together support larger DMA buffers. You will still end up with small gaps between the buffers - I'm not aware of a way of adding DMA descriptors to a chain while the controller is running - but you get to decide how frequent they are. Another approach would be to use cyclic transfers, which is how ALSA manages transfers to I2S soundcards, but that's another level of complexity.