Breaking Down korge-core into Different Modules #2088
Replies: 8 comments 6 replies
-
I'm fine with it. Before splitting
That's already supported: fun Deflate(windowBits: Int): CompressionMethod
class GZIP : CompressionMethod
suspend fun CompressionMethod.uncompress(i: AsyncInputStream, o: AsyncOutputStream): UnitAnd to use synchronously if the source and destination is synchronous in the end: fun <T : Any> runBlockingNoSuspensions(callback: suspend () -> T): T |
Beta Was this translation helpful? Give feedback.
-
Ok, I will try my best to contribute there.
Sorry, I missed that. Thanks for pointing it out. I think By the way, I was checking and found a few things missing for my scenario:
Here is an example of Java code that I'm trying to achieve in Kotlin using val stream = GZIPInputStream(FileInputStream(file)) // InputStream
stream.mark
stream.reset
stream.skip
val inputStreamReader = InputStreamReader(input, Charset.forName(charsetName))
inputStreamReader.read(charArray: CharArray, offset: Int, len: Int)
inputStreamReader.mark
inputStreamReader.reset
inputStreamReader.skipI've also made modifications to internal class CharReaderSyncStream(
private val stream: SyncStream,
private val charset: Charset,
private val chunkSize: Int
) : CharReader {
private var temp = ByteArray(chunkSize)
private val buffer = ByteArrayDeque()
private var tempStringBuilder = StringBuilder()
private var markedTempStringBuilder: StringBuilder? = null
private var currentState: ReaderState = ReaderState(consumedBytes = 0, consumedBuffer = 0, lastBytesRead = 0)
private var markedState: ReaderState? = null
override fun clone(): CharReader = CharReaderFromSyncStream(stream.clone(), charset, chunkSize)
init {
stream.mark(SharedConstants.DefaultBufferSize)
}
private fun bufferUp() {
while (buffer.availableRead < temp.size) {
val readCount = stream.read(temp)
if (readCount <= 0) break
currentState = currentState.copy(
consumedBytes = currentState.consumedBytes + readCount,
lastBytesRead = readCount
)
buffer.write(temp, 0, readCount)
if (currentState.applyMarkedState) {
currentState = currentState.copy(applyMarkedState = false)
if (currentState.consumedBuffer > 0) {
buffer.skip(currentState.consumedBuffer)
}
} else {
currentState = currentState.copy(consumedBuffer = 0)
}
}
}
override fun read(out: StringBuilder, count: Int): Int {
bufferUp()
while (tempStringBuilder.length < count) {
val readCount = buffer.peek(temp)
val consumed = charset.decode(tempStringBuilder, temp, 0, readCount)
if (consumed <= 0) break
currentState = currentState.copy(consumedBuffer = currentState.consumedBuffer + consumed)
buffer.skip(consumed)
}
val slice = tempStringBuilder.substring(0, kotlin.math.min(count, tempStringBuilder.length))
tempStringBuilder = StringBuilder(slice.length).append(tempStringBuilder.substring(slice.length))
out.append(slice)
return slice.length
}
fun mark(readLimit: Int) {
this.markedTempStringBuilder = StringBuilder(tempStringBuilder)
this.markedState = this.currentState
}
fun reset() {
if (this.markedState != null) {
if (this.markedTempStringBuilder != null) {
this.tempStringBuilder = StringBuilder(this.markedTempStringBuilder!!)
this.markedTempStringBuilder = null
}
buffer.clear()
temp = ByteArray(chunkSize)
val skipedBytes = this.markedState!!.consumedBytes - this.markedState!!.lastBytesRead
this.currentState = this.markedState!!.copy(consumedBytes = skipedBytes, applyMarkedState = true)
stream.reset()
stream.mark(SharedConstants.DefaultBufferSize)
if (skipedBytes > 0) {
stream.skip(skipedBytes)
}
this.markedState = null
}
}
fun skip(count: Int) {
this.read(count)
}
}
private data class ReaderState(
val consumedBytes: Int,
val consumedBuffer: Int,
val lastBytesRead: Int,
val applyMarkedState: Boolean = false
)Lastly, I've identified some issues in I genuinely appreciate all your hard work on this. It has been tremendously helpful for me, and I hope the broader community will also benefit from these improvements. I apologize if I have initiated several discussions here. Thank you once again for your understanding and assistance. |
Beta Was this translation helpful? Give feedback.
-
|
Let me write one separate message for each topic, so we can reply to each thing individually in a separate thread. |
Beta Was this translation helpful? Give feedback.
-
Why do you need it? I mean, can't you just duplicate the AsyncStrean to emulate a mark? |
Beta Was this translation helpful? Give feedback.
-
It works like that, because I/O in korio is all asynchronous. There is a separate API for sync I/O but the point on korio is to also work on JS where only async I/O is typically available and also make it difficult to have bottle necks related to I/O. Typically when reading binary stuff, you read asynchronously the header, and with the header, you get the size of the dynamical header information you need and read a full block asynchronously, then you process that block synchronously having that in memory for performance sake. |
Beta Was this translation helpful? Give feedback.
-
What's your scenario for an |
Beta Was this translation helpful? Give feedback.
-
|
I am currently developing an HTML & XML parse ksoup. In this parser, users have the option to load large files for parsing. However, I aim to avoid loading the entire file into memory, especially in scenarios where users only need to read the header of a large HTML file. In such cases, the parser loads just the data within the head tag from the file and parses it. The parser primarily operates with Therefore, I need to pass a stream to |
Beta Was this translation helpful? Give feedback.
-
|
Would this work for you? (untested) vfsFile.openUse { stream ->
val deque = DequeSyncStream()
val charReader = CharReaderFromSyncStream(deque)
while (deque.hasMore || !stream.eof) {
if (deque.available < 512) {
deque.write(stream.readBytes(512))
}
// ...
println(charReader.read(8))
}
}
|
Beta Was this translation helpful? Give feedback.
-
Yeah, because Char is the Kotlin type, not code points. The binary representation of characters change depending on the Charset. I understand in any case that that name can be misleading. But that API is for reading arrays, not strings. If you have ideas on APIs to improve this, feel free to share your thoughts. |
Beta Was this translation helpful? Give feedback.
-
Not implemented AFAIK. Can you put a real example with a case where you would need that mark and reset in a more specific way so I can understand the use-case?
We can make adjustments so it is efficient. But first I want to understand the use-case to know if that's possible to do it with the current API, without adding extra complexity to the implementation. |
Beta Was this translation helpful? Give feedback.
-
|
In the parser, reading characters instead of streams is the requirement. There are several use cases for this approach, including:
The Input data size: 8, ChunkSize: 3, User reads: 2 characters |
Beta Was this translation helpful? Give feedback.
-
|
Wouldn't a small layer on top of that work? Like buffering and/or a CharDeque? You can read from an asynchronous source like 1024 bytes and keep a deque filled. Then you can peek characters and only consume them when you are sure you can to consume them. Or consume them by default and readding them to the start of the deque. For templates you would typically have everything in memory. It is true that parsing big XMLs or JSONs might require streaming. But in the case of XMLs or JSONs, I don't see specific reasons for a parser wanting to rewind something already consumed. |
Beta Was this translation helpful? Give feedback.
-
|
@soywiz Why not exclusively concentrate on the In java there is one base class for Stream that is We might require a class similar to |
Beta Was this translation helpful? Give feedback.
-
|
From my experience the asynchronou stuff introduces an important overhead, specially for performance scenarios dealing with binary formats. In fact there is a kind of stream that is completely crafted for in-memory ByteArrays for stuff where having the overhead of SyncStreams was too big for the scenario. Used for example in TTF parsing that might include a lot of stuff on it. I'm open to introduce Async stuff as required if you need it and makes sense, but I would like to keep the synchronous variants that currently exists at least. |
Beta Was this translation helpful? Give feedback.
-
Hi, I'm testing
korge-corein one of my Kotlin Multiplatform HML & XML Parser projects ksoup, and it has been really helpful. However, I have a few concerns aboutkorge-corethat could greatly benefit the community. I would like to use charset encoding, streaming API, and gzip compression only from korge-core, but the library is coupled with a lot of extra functionalities.It would be helpful if the library could be divided into modules, such as korge-core (containing only the core without network, compression, or parser code), and create other separate modules like korge-network, korge-compression, korge-parser, etc. This modular approach would make the library more versatile for the Kotlin Multiplatform community.
Additionally, do you have any plans for implementing gzip compression with a streaming API, similar to GZIPInputStream, to decompress files on the fly as only the necessary bytes decompressed?"
Beta Was this translation helpful? Give feedback.
All reactions