[KYUUBI #1022] Add basic EngineStatusStore for events #1023
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.kyuubi.engine.spark.events | ||
|
|
||
| import java.util.concurrent.ConcurrentHashMap | ||
|
|
||
| import scala.collection.JavaConverters.collectionAsScalaIterableConverter | ||
|
|
||
| import org.apache.kyuubi.config.KyuubiConf | ||
| import org.apache.kyuubi.config.KyuubiConf.ENGINE_UI_SESSION_LIMIT | ||
|
|
||
| /** | ||
| * A memory store that tracking the number of statements and sessions, it provides: | ||
| * | ||
| * - stores all events. | ||
| * - cleanup the events when reach a certain threshold: | ||
| * 1). remove the finished events first. | ||
| * 2). remove the active events if still reach the threshold. | ||
| * | ||
| * // TODO KYUUBI #983 this store will be used in the third task. | ||
| */ | ||
| class EngineEventsStore(conf: KyuubiConf) { | ||
|
|
||
| /** | ||
| * The number of SQL client sessions kept in the Kyuubi Query Engine web UI. | ||
| */ | ||
| private val retainedSessions: Int = conf.get(ENGINE_UI_SESSION_LIMIT) | ||
|
|
||
| /** | ||
| * store all session events. | ||
| */ | ||
| val sessions = new ConcurrentHashMap[String, SessionEvent] | ||
|
|
||
| /** | ||
| * get all session events order by startTime | ||
| */ | ||
| def getSessionList: Seq[SessionEvent] = { | ||
| sessions.values().asScala.toSeq.sortBy(_.startTime) | ||
| } | ||
|
|
||
| def getSession(sessionId: String): Option[SessionEvent] = { | ||
| Option(sessions.get(sessionId)) | ||
| } | ||
|
|
||
| /** | ||
| * save session events and check the capacity threshold | ||
| */ | ||
| def saveSession(sessionEvent: SessionEvent): Unit = { | ||
| sessions.put(sessionEvent.sessionId, sessionEvent) | ||
| checkSessionCapacity() | ||
| } | ||
|
|
||
| /** | ||
| * cleanup the session events if reach the threshold | ||
| */ | ||
| private def checkSessionCapacity(): Unit = { | ||
|
Contributor
Author
There was a problem hiding this comment. Can't find a suitable opensource library to supports following characteristic:
If we implement a new Map which extends AbstractMap, it seems will be complex. If we keep 200 events in memory, efficiency may not be affected. |
||
| var countToDelete = sessions.size - retainedSessions | ||
|
|
||
| val reverseSeq = sessions.values().asScala.toSeq.sortBy(_.startTime).reverse | ||
|
Member
There was a problem hiding this comment. we sort the value set for evevy single event？...
Contributor
Author
There was a problem hiding this comment. if we use treemap, and let startTime or endTime as key, it will remove events if key repeat.
Contributor
There was a problem hiding this comment. here you should use asyn, by this you can sort the value only once when countToDelete reached the set value.
Contributor
Author
There was a problem hiding this comment. if we use a new thread, thinghs will be more complex |
||
|
|
||
| // remove finished sessions first. | ||
| for (event <- reverseSeq if event.endTime != 0L && countToDelete > 0) { | ||
| sessions.remove(event.sessionId) | ||
| countToDelete -= 1 | ||
| } | ||
|
|
||
| // remove active event if still reach the threshold | ||
| for (event <- reverseSeq if countToDelete > 0) { | ||
| sessions.remove(event.sessionId) | ||
| countToDelete -= 1 | ||
| } | ||
| } | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.kyuubi.engine.spark.events | ||
|
|
||
| import org.apache.kyuubi.KyuubiFunSuite | ||
| import org.apache.kyuubi.config.KyuubiConf | ||
| import org.apache.kyuubi.config.KyuubiConf.ENGINE_UI_SESSION_LIMIT | ||
|
|
||
| class EngineEventsStoreSuite extends KyuubiFunSuite { | ||
|
|
||
| test("ensure that the sessions are stored in order") { | ||
| val store = new EngineEventsStore(KyuubiConf()) | ||
|
|
||
| val s1 = SessionEvent("a", "ea", "test1", "1.1.1.1", 1L) | ||
| val s2 = SessionEvent("c", "ea", "test2", "1.1.1.1", 3L) | ||
| val s3 = SessionEvent("b", "ea", "test3", "1.1.1.1", 2L) | ||
|
|
||
| store.saveSession(s1) | ||
| store.saveSession(s2) | ||
| store.saveSession(s3) | ||
|
|
||
| assert(store.getSessionList.size == 3) | ||
| assert(store.getSessionList.head.sessionId == "a") | ||
| assert(store.getSessionList.last.sessionId == "c") | ||
| } | ||
|
|
||
| test("test drop sessions when reach the threshold ") { | ||
| val conf = KyuubiConf() | ||
| conf.set(ENGINE_UI_SESSION_LIMIT, 3) | ||
|
|
||
| val store = new EngineEventsStore(conf) | ||
| for (i <- 1 to 5) { | ||
| val s = SessionEvent(s"b$i", "ea", s"test$i", "1.1.1.1", 2L) | ||
| store.saveSession(s) | ||
| } | ||
|
|
||
| assert(store.getSessionList.size == 3) | ||
| } | ||
|
|
||
| test("test drop sessions when reach the threshold, and try to keep active events.") { | ||
| val conf = KyuubiConf() | ||
| conf.set(ENGINE_UI_SESSION_LIMIT, 3) | ||
|
|
||
| val store = new EngineEventsStore(conf) | ||
|
|
||
| store.saveSession(SessionEvent("s1", "ea", "test1", "1.1.1.1", 1L, 0L)) | ||
| store.saveSession(SessionEvent("s2", "ea", "test1", "1.1.1.1", 2L, 0L)) | ||
| store.saveSession(SessionEvent("s3", "ea", "test1", "1.1.1.1", 3L, 1L)) | ||
| store.saveSession(SessionEvent("s4", "ea", "test1", "1.1.1.1", 4L, 0L)) | ||
|
|
||
| assert(store.getSessionList.size == 3) | ||
| assert(store.getSessionList(2).sessionId == "s4") | ||
| } | ||
|
|
||
| test("test check session after update session") { | ||
| val store = new EngineEventsStore(KyuubiConf()) | ||
| val s = SessionEvent("abc", "ea", "test3", "1.1.1.1", 2L) | ||
| store.saveSession(s) | ||
|
|
||
| val finishTimestamp: Long = 456L | ||
| s.endTime = finishTimestamp | ||
| store.saveSession(s) | ||
|
|
||
| assert(store.getSession("abc").get.endTime == finishTimestamp) | ||
| } | ||
|
|
||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.