Skip to content
This repository has been archived by the owner on Apr 1, 2024. It is now read-only.

ISSUE-8974: Peeking at compressed messages throws an exception #1893

Closed
sijie opened this issue Dec 16, 2020 · 0 comments
Closed

ISSUE-8974: Peeking at compressed messages throws an exception #1893

sijie opened this issue Dec 16, 2020 · 0 comments

Comments

@sijie
Copy link
Member

sijie commented Dec 16, 2020

Original Issue: apache#8974


Earlier in the 2.6.0 branch, the compression library was swapped for https://github.com/airlift/aircompressor.

When messages are published with LZ4 compression and you try to use the Peek API, https://pulsar.apache.org/admin/v2/persistent/{tenant}/{namespace}/{topic}/subscription/{subName}/position/{messagePosition} on the message, the broker throws an exception, causing a 500 return code. This breaks the dashboard.

Here is the exception:


14:54:25.006 [pulsar-web-43-1] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [superuser] Failed to get message at position 1 from persistent://xxxxxx
java.lang.IllegalArgumentException: Unsupported input ByteBuffer implementation java.nio.HeapByteBufferR
        at io.airlift.compress.lz4.Lz4Decompressor.decompress(Lz4Decompressor.java:58) ~[io.airlift-aircompressor-0.16.jar:0.16]
        at org.apache.pulsar.common.compression.CompressionCodecLZ4.decode(CompressionCodecLZ4.java:100) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1]
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.generateResponseWithEntry(PersistentTopicsBase.java:1977) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar
:2.6.1]
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalPeekNthMessage(PersistentTopicsBase.java:1921) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.
6.1]

If I recall correctly, this also fails for SNAPPY compression, but works fine for other compression types (ZLIB).

Here is the pull request where this change was introduced: apache#5390

I am 100% certain this is the cause of the issue. Reverting this PR resolves the issue.

Some potentially useful references:

See:
https://github.com/airlift/aircompressor/blob/24b4f18dcb8a70d273ba1d1cd4c0715f15909573/src/main/java/io/airlift/compress/lz4/Lz4Decompressor.java#L58
real-logic/agrona#66 (comment)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant