This document contains information about the different methods to integrate new libraries into your Kotlin Kernel for Jupyter notebooks. It also describes the supported integration features a library can provide when integrated into your Kotlin Kernel.
Click here to expand the table of contents.
- Integrate new libraries
- Library integration methods
- Supported integration features
- List of supported integration features
- Dependencies
- Repositories
- Initial imports
- Callbacks after library loading (called once)
- Callbacks before each cell execution
- Callbacks after each cell execution
- Callbacks on cell execution interruption
- Callbacks right before kernel shutdown
- Callbacks on color scheme change
- Results renderers
- Results text renderers
- Throwables renderers
- Variables handling
- Annotated classes handling
- File annotations handling
- Code preprocessing
- Library static resources loading
- Variables reporting
- Internal variables markers
- Typename rules for transitively loaded integration classes
- Minimal kernel version supported by the library
- Library options
- Link to the library site
- Library description
- Creating a library descriptor
- Integration using the Kotlin API
There are two main methods for integrating a new library into your Kotlin Kernel for notebooks:
-
Creating a JSON library descriptor: It's an easy-to-go solution that does not require you changing the library. You create a JSON file defining the most frequent library features, such as properties, renderers, and initial imports. The exact syntax depends on where the descriptor is located. You can make the new library available via the
%use
line magic. -
Using the Kotlin API: This method requires modifying the library code to include integration logic. You can define an integration class in your library code, or create a separate project for integration if it's a library you don't maintain. The library is automatically integrated when ist JAR containing the
META-INF/kotlin-jupyter-libraries/libraries.json
file (with the integration class name) is added to the notebook classpath. You can add the integration class name with the@file:DependsOn
annotation or with a descriptor (see above) that defines the corresponding dependency. Additionally, it is possible to write tests for this kind of integration.Once you have defined the integration class, you can use all available integration features.
Regardless of the integration method, library integrations can define dependencies
and callbacks to interact with the notebook environment. The dependencies can contain Kotlin-API-based integrations, and the callbacks
can contain the %use
line magic, which means that library integrations can load other libraries, and so on. Don't
hesitate to rely on this feature.
Supported integration features are functionalities a library can provide when integrated into your Kotlin Kernel for notebooks.
To use the supported integration features, you can:
-
Use the descriptor API: If the feature is supported in the descriptor API, you can create a JSON file containing this feature description. This JSON file is loaded into the notebook via the
%use
line magic.If the feature is supported in the descriptor API, you can load the corresponding JSON string directly using the
loadLibraryDescriptor
method inside a notebook cell. -
Use the JupyterIntegration API: You can add the feature directly from a notebook cell using the
USE {}
method and adding the feature method. Here's an example of theimport
method. For more feature methods, see the list of supported integration features.USE { import("my.awesome.Clazz") }
-
Use the LibraryDefinition API: You can add the feature directly from the notebook cell if you have a
LibraryDefinition
instance. This instance can be created using thelibraryDefinition {}
method. Use the following syntax:USE(libraryDefinition)
Inside a Kotlin JVM library, you can create a class implementing the LibraryDefinition
or LibraryDefinitionProducer
interfaces in one of the following ways:
- Create a direct implementor of
LibraryDefinition
. Override the properties defined in the "LibraryDefinition API" column from the following table. - Extend the
LibraryDefinitionImpl
interface. Set its properties defined in the "LibraryDefinition API" column from the following table. - Define a class that implements the
JupyterIntegration
interface. Override theBuilder.onLoaded
method and use methods from the "JupyterIntegration API" column. This class is loaded into the notebook via the%use
line magic along with the whole library artifact. To let the notebook know about this class, adjust the build correspondingly. If you don't adjust the build, the class is not loaded. However, you can still load this class from the notebook using theloadLibraryDefinitions()
orloadLibraryProducers()
methods.
Here's a list of the supported integration features. See interactive examples in this API guide notebook.
Feature | Descriptor API | LibraryDefinition API | JupyterIntegration API |
---|---|---|---|
Dependencies | dependencies |
dependencies |
dependencies() |
Repositories | repositories |
repositories |
repositories() addRepository() repository() |
Initial imports | imports |
imports |
import() importPackage() |
Callbacks after library loading (called once) | init |
init |
onLoaded{} |
Callbacks before each cell execution | initCell |
initCell |
beforeCellExecution{} |
Callbacks after each cell execution | - | afterCellExecution |
afterCellExecution{} |
Callbacks on cell execution interruption | - | interruptionCallbacks |
onInterrupt{} |
Callbacks right before kernel shutdown | shutdown |
shutdown |
onShutdown{} |
Callbacks on color scheme change | - | colorSchemeChangedCallbacks |
onColorSchemeChange{} |
Results renderers | renderers |
renderers |
addRenderer() render<T>{} renderWithHost<T>{} |
Results text renderers | - | textRenderers |
addTextRenderer() |
Throwables renderers | - | throwableRenderers |
addThrowableRenderer() renderThrowable<T>{} |
Variables handling | - | converters |
addTypeConverter() onVariable{} updateVariable{} onVariableByRuntimeType{} updateVariableByRuntimeType{} |
Annotated classes handling | - | classAnnotations |
addClassAnnotationHandler() onClassAnnotation<T>{} |
File annotations handling | - | fileAnnotations |
addFileAnnotationHanlder() onFileAnnotation<T>{} |
Code preprocessing | - | codePreprocessors |
addCodePreprocessor() preprocessCodeWithLibraries{} preprocessCode{} |
Library static resources loading | resources |
resources |
resource() |
Internal variables markers | - | internalVariablesMarkers |
markVariableInternal() |
Typename rules for transitively loaded integration classes | integrationTypeNameRules |
integrationTypeNameRules |
addIntegrationTypeNameRule() acceptIntegrationTypeNameIf{} discardIntegrationTypeNameIf{} |
Minimal kernel version supported by the library | minKernelVersion |
minKernelVersion |
setMinimalKernelVersion() |
Library options | properties |
options |
addOption() addOptions() |
Link to the library site, used to generate the README | link |
website |
setWebsite() |
Library description, used to generate the README | description |
description |
setDescription() |
Regardless of the API you use for adding dependencies, notebook dependencies are expressed as Kotlin strings. The supported formats are:
- Coordinates of Maven dependencies in form of
<group>:<artifact>:<version>
- Absolute paths to the local JAR files
- Absolute paths to the local directories containing classes
Mind the following:
- The
compile
andruntime
scopes of dependencies are resolved transitively, but added to both compile and runtime classpath. That's why you may see undesired variants offered in completion. - In Kotlin Notebook, sources of the dependencies are resolved and included in the response metadata.
In other clients, they do not. To control this behavior, use the
SessionOptions.resolveSources
option. - MPP libraries are usually not resolved by the Maven resolver. You should either use the
jvm
variants of these artifacts or enable experimental multiplatform resolution with theSessionOptions.resolveMpp
option. - To show the current notebook classpath, use the
:classpath
command.
Here's an example of how to use the dependencies
feature via the Descriptor API:
{
"dependencies": [
"<dependency1>",
"<dependency2>"
]
}
Here's an example of how to use the dependencies
feature via the JupyterIntegration API:
USE {
dependencies("<dependency1>", "<dependency2>")
// or
dependencies {
implementation("<dependency1>")
}
}
Repositories are strings describing where the dependencies come from:
- Maven repositories containing URLs and credentials (if applicable)
- Local directories, relatively to which local dependencies are resolved
Here's an example of how to use the repositories
feature via the Descriptor API:
{
"repositories": [
"<repo1-url>",
{
"path": "<repo2-url>",
"username": "auth-username",
"password": "auth-token"
}
]
}
Here's an example of how to use the repositories
feature via the JupyterIntegration API:
USE {
repositories("<repo1>", "<repo2>")
// or
repositories {
maven {
url = "<repo1-url>"
credentials {
username = "auth-username"
password = "auth-token"
}
}
}
}
Imports are import declarations used by the rest of the following cells. The imports syntax can also be star-ended.
Here's an example of how to use the imports
feature via the Descriptor API:
{
"imports": [
"my.package.*",
"my.package.Clazz"
]
}
Here's an example of how to use the imports
feature via the JupyterIntegration API:
USE {
imports("my.package.*", "my.package.Clazz")
// or
import<Clazz>()
importPackage<Clazz>()
}
This callback type comprises code executed (or compiled in case of descriptor API) right after loading the library. In the Descriptor API, the code pieces are executed separately and not merged into one snippet.
Here's an example of how to use this type of callback via the Descriptor API:
{
"init": [
"val x = 3",
"%use dataframe"
]
}
Here's an example of how to use this type of callback via the JupyterIntegration API:
USE {
onLoaded {
println("Integration loaded")
// Makes the variable visible inside the notebook
scheduleExecution("val x = 3")
}
}
This callback type comprises code executed (or compiled in case of the Descriptor API) right before each user-initiated cell execution. In the descriptor API, codes pieces are executed separately and not merged into one snippet.
Here's an example of how to use this type of callback via the Descriptor API:
{
"initCell": [
"val y = x + 1",
"println(\"abc\")"
]
}
Here's an example of how to use this type of callback via the JupyterIntegration API:
USE {
beforeCellExecution {
println("Before cell execution")
// Variable x will be visible inside the notebook
scheduleExecution("val x = 3")
}
}
This callback type comprises code executed right after each user-initiated cell execution.
Here's an example of how to use this type of callback via the JupyterIntegration API:
USE {
afterCellExecution { snippetInstance, resultField ->
println("After cell execution: ${resultField.name} = ${resultField.value}")
// Variable x will be visible inside the notebook
scheduleExecution("val x = 3")
}
}
This callback type comprises code executed after the cell execution is interrupted by the user.
Here's an example of how to use this type of callback via the JupyterIntegration API:
USE {
onInterrupt {
println("Execution was interrupted...")
}
}
This callback type comprises code executed when the user initiates a kernel shutdown.
Here's an example of how to use this type of callback via the Descriptor API:
{
"shutdown": [
"val y = x + 1",
"println(\"abc\")"
]
}
Here's an example of how to use this type of callback via the JupyterIntegration API:
USE {
onShutdown {
println("Bye!")
}
}
This callback type comprises code executed when the user changes the color scheme in the IDE with the notebook opened, and the session started. This callback doesn't work this way for clients other than Kotlin Notebook, but it's safe for any other client.
Here's an example of how to use this type of callback via the JupyterIntegration API:
var isDark = notebook.currentColorScheme == ColorScheme.DARK
USE {
println("Dark? - $isDark")
onColorSchemeChange { colorScheme ->
isDark = colorScheme == ColorScheme.DARK
println("Color scheme is changed")
}
}
Rendering is the procedure of transforming a value to a form that is appropriate for displaying it in the Jupyter client. The Kotlin Kernel for Jupyter notebooks supports various features and mechanisms for rendering values. For more information, see Rendering.
Here's an example of how to use the renderers
feature via the Descriptor API:
{
"renderers": {
"org.jetbrains.letsPlot.intern.Plot": "HTML(frontendContext.getHtml($it as org.jetbrains.letsPlot.intern.Plot))"
}
}
Here's an example of how to use the render
feature via the JupyterIntegration API:
USE {
render<Plot> {
HTML(frontendContext.getHtml(it))
}
}
Results text renderers are a feature used to customize how data structures from your library are displayed in a text format within the Kotlin Jupyter notebook. For more information, see Rendering.
Here's an example of how to use text renderers via the JupyterIntegration API:
USE {
addTextRenderer { processor, table ->
(table as? Table)?.let { frontendContext.getPlainText(it) }
}
}
Throwables renderers are a feature in Kotlin Jupyter that allows you to customize how exceptions (errors) thrown by your library code are displayed within the notebook. For more information, see Rendering.
Here's an example of how to use throwables renderers via the JupyterIntegration API:
USE {
renderThrowable<NullPointerException> { npe ->
"Isn't Kotlin null-safe?"
}
}
Variables handlers are run for each property of the
executed snippets, if applicable. They also give access to the KotlinKernelHost
object so that it's possible to execute code there.
Here's an example of how to handle variables via the JupyterIntegration API:
USE {
updateVariable<MyType> { value, kProperty ->
// MyWrapper class should be previously defined in the notebook
execute("MyWrapper(${kProperty.name})").name
}
onVariable<MyType2> { value, kProperty ->
println("Variable ${kProperty.name}=$value executed!")
}
}
If you have an annotation with runtime retention, you can mark a cell's class with this annotation. Then, marked classes can be processed.
Annotation arguments are not available in this type of callback, but this API should become more consistent in future kernel versions.
Here's an example of how to handle annotated classes via the JupyterIntegration API:
// Should have runtime retention
annotation class MyAnnotation
USE {
onClassAnnotation<MyAnnotation> { classifiersList -> println("Annotated classes: $classifiersList") }
}
@MyAnnotation
class MyClass
You can add file-level annotations to the code snippets. Examples of such annotations are @file:DependsOn()
and
@file:Repository()
, which the kernel uses to add dependencies to the notebook.
In the callback, you have access to the file annotation object and assigned annotation properties.
Here's an example of how to handle file annotations via the JupyterIntegration API:
// Might have any retention, but files should be a valid target
annotation class MyAnnotation
USE {
onFileAnnotation<MyAnnotation> {
val myAnno = it.first() as MyAnnotation
println("My annotation object: $myAnno")
}
}
The user's code can be amended in any way before execution. One such transformation is magic preprocessing, which involves cutting off the code and specifically processing it. It's possible to write your own preprocessor: it gets the code and should return the amended code. Preprocessors are applied one after another, depending on their priority and order.
Here's an example of how to use code preprocessors via the JupyterIntegration API:
USE {
preprocessCode { code -> generateRunBlocking(code) }
}
Static resources such as JS and CSS files can be used by the library producing an HTML file. Generally, some specific wrappers should be written to load resources correctly. You can do it yourself or let the kernel infrastructure do it for you. The resource bundles builder DSL is defined and documented here.
Here's an example of how to use library static resources via the JupyterIntegration API:
USE {
resources {
js("plotly") {
//...
}
}
}
You can see the variables defined in the notebook in both plain text and HTML formats.
To ignore some variables in the variable report, mark these variables as internal using the JupyterIntegration API:
USE {
markVariableInternal { prop ->
prop.name.startsWith("_")
}
}
As mentioned before, libraries can load other libraries transitively, either by executing the %use
line magic as
a part of the initialization code or by including a dependency that contains an integration.
By default, all integration classes found in the dependencies are loaded. However, you can turn off the loading functionality of some integrations by using typename rules to skip them. At the same time, the library can load its integration class forcefully by specifying "accepting" as a typename rule. In this case, even if the typename is disabled by the loader library, the corresponding class will be loaded.
Here's an example of how to use typename rules via the Descriptor API:
{
"integrationTypeNameRules": [
"-:org.jetbrains.kotlinx.dataframe.**",
//"+:org.jetbrains.kotlinx.dataframe.**",
]
}
Here's an example of how to use typename rules via the JupyterIntegration API:
USE {
discardIntegrationTypeNameIf {
it.startsWith("org.jetbrains.kotlinx.dataframe.")
}
// acceptIntegrationTypeNameIf {
// it.startsWith("org.jetbrains.kotlinx.dataframe.")
// }
}
You can define the minimal kernel version to be supported by the library integration.
In the JupyterIntegration API, it's also possible to set notebook.kernelVersion
.
Here's an example of how to define the minimal kernel version via the Descriptor API:
{
"minKernelVersion": "0.11.0.1"
}
Here's an example of how to define the minimal kernel version via the JupyterIntegration API:
USE {
setMinimalKernelVersion("0.11.0.1")
}
Library options are useful for different purposes:
- To extract some frequently updatable parts of the library descriptors (such as library versions).
- To assist the Renovate GitHub app to update library versions.
- To pass some values transitively in library loading so that libraries might know through what other libraries they were loaded.
Note: Give unique names for the options of your library because these options can override some other options, and it may lead to unexpected quirks.
Here's an example of how to use library options via the Descriptor API.
Options ending with -renovate-hint
are ignored in descriptors and shouldn't be visible:
{
"properties": [
{
"name": "api",
"value": "4.4.1"
},
{
"name": "api-renovate-hint",
"value": "update: package=org.jetbrains.lets-plot:lets-plot-kotlin-kernel"
}
],
"dependencies": [
"org.company:library:$api"
]
}
Here's an example of how to use library options via the JupyterIntegration API:
USE {
addOption("api", "4.4.1")
}
Options in JupyterIntegration API could be only used when this library loads some other integration transitively,
and the integration class' constructor has two arguments, the second of which is of type Map<String, String>
.
All previously loaded options are put into the map and passed as an argument.
The library integration can have a link to the library's site.
These links are embedded into the README. Links can also be embedded into the :help
command, but only for descriptors.
Here's an example of adding links to the library site via the Descriptor API:
{
"link": "https://github.com/Kotlin/kandy"
}
Here's an example of adding links to the library site via the JupyterIntegration API:
USE {
setWebsite("https://github.com/Kotlin/kandy")
}
The library integration can have a description.
The description is embedded into the README and the :help
command, but only for descriptors.
Here's an example of adding a library description via the Descriptor API:
{
"description": "Kotlin plotting DSL for Lets-Plot"
}
Here's an example of adding a library description via the JupyterIntegration API:
USE {
setDescription("Kotlin plotting DSL for Lets-Plot")
}
To support a new JVM
library for your Kotlin Kernel for notebooks and make it available via the %use
line magic, you need to create a library descriptor for it.
For examples of library descriptors, see the libraries repository.
A library descriptor is a <libName>.json
file with the following fields:
properties
: a dictionary of properties that are used within the library descriptor. Library properties can be used in any part of the library descriptor as$property
.description
: a short library description used to generate a library list in the README.link
: a link to the library website. This link will be displayed through the:help
REPL command.minKernelVersion
: a minimal version of the Kotlin kernel to be used with this descriptor.repositories
: a list of Maven or Ivy repositories to search for dependencies.dependencies
: a list of library dependencies.imports
: a list of default imports for the library.init
: a list of code snippets to be executed when the library is included.initCell
: a list of code snippets to be executed before executing any cell.shutdown
: a list of code snippets to be executed on the kernel shutdown. Any cleanup code should be placed here.renderers
: a mapping from fully qualified names of types to be rendered to the Kotlin expression returning output value. The source object is referenced as$it
.resources
: a list of JS/CSS resources. For examples, see this descriptor.integrationTypeNameRules
: a list of rules for integration classes that are about to be loaded transitively. Each rule has the form[+|-]:<pattern>
where+
or-
denotes if this pattern is accepted or declined. The pattern may consist of any characters. Special combinations are allowed. For example:?
, any single character or no character;*
, any character sequence excluding dot;**
, any character sequence.
Note: All fields of the library descriptor are optional.
For the most relevant specification, see the org.jetbrains.kotlinx.jupyter.libraries.LibraryDescriptor
class.
The name of the library JSON file should have the <name>.json
syntax, where <name>
is the argument for the %use
line magic.
To register a new library descriptor:
- For private usage: create it anywhere on your computer and reference it using file syntax.
- Alternative way for private usage: create a descriptor in the
.jupyter_kotlin/libraries
folder and reference it using the default syntax. - For sharing with the community: Commit it to the libraries repository and create a pull request.
If you are maintaining a library and want to update your library descriptor, create a pull request with your update. Once your request is accepted,
the new version of your library will become available to all Kotlin Jupyter users upon their next kernel startup,
provided they use the %useLatestDescriptors
magic command. If they do not use this command, a kernel update will be necessary to access the updated library version.
You can also add a Kotlin kernel integration to your library using a Gradle plugin. To do so, you must add the plugin dependency to your build script.
For build.gradle
:
plugins {
id "org.jetbrains.kotlin.jupyter.api" version "<jupyterApiVersion>"
}
For build.gradle.kts
:
plugins {
kotlin("jupyter.api") version "<jupyterApiVersion>"
}
From the snippets above, <jupyterApiVersion>
is one of the published versions.
It's preferred to use the latest stable version.
This Gradle plugin adds the following dependencies to your project:
Artifact | Gradle option to exclude/include | Enabled by default | Dependency scope | Method for adding dependency manually |
---|---|---|---|---|
kotlin-jupyter-api |
kotlin.jupyter.add.api |
yes | compileOnly |
addApiDependency(version: String?) |
kotlin-jupyter-api-annotations |
kotlin.jupyter.add.scanner |
no | compileOnly |
addScannerDependency(version: String?) |
kotlin-jupyter-test-kit |
kotlin.jupyter.add.testkit |
yes | testImplementation |
addTestKitDependency(version: String?) |
You can turn on and off the dependency with its default version (version of the plugin)
by setting the corresponding Gradle option to true
or false
.
If the corresponding option is set to false
(by default or in your setup), you
can still add the dependency manually by using the method from the kotlinJupyter
extension:
kotlinJupyter {
// Uses the default version
addApiDependency()
// Uses a custom artifact version
addApiDependency("0.10.0.1")
}
If you use the KSP plugin, you can utilize annotations to mark integration classes:
-
Enable
kotlin-jupyter-api-annotations
dependency by adding the following line to yourgradle.properties
:kotlin.jupyter.add.scanner = true
-
Add one of these implementations:
org.jetbrains.kotlinx.jupyter.api.libraries.LibraryDefinitionProducer
ororg.jetbrains.kotlinx.jupyter.api.libraries.LibraryDefinition
. -
Mark the implementation with the
JupyterLibrary
annotation:
package org.my.lib
import org.jetbrains.kotlinx.jupyter.api.annotations.JupyterLibrary
import org.jetbrains.kotlinx.jupyter.api.*
import org.jetbrains.kotlinx.jupyter.api.libraries.*
@JupyterLibrary
internal class Integration : JupyterIntegration() {
override fun Builder.onLoaded() {
render<MyClass> { HTML(it.toHTML()) }
import("org.my.lib.*")
import("org.my.lib.io.*")
}
}
For more examples, see the integration of the DataFrame library.
For further information, see the docs for:
org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration
org.jetbrains.kotlinx.jupyter.api.libraries.LibraryDefinitionProducer
org.jetbrains.kotlinx.jupyter.api.libraries.LibraryDefinition
You can also choose not to use the KSP plugin to detect implementations. In this scenario, you have to reference your implementations directly in your build script. Be aware that this approach does not include any existence checks, so you need to ensure that all referenced implementations are correctly defined.
The following example shows how to refer the Integration
class in your buildscript (in this case,
you shouldn't mark it with the JupyterLibrary
annotation).
For build.gradle
:
processJupyterApiResources {
libraryProducers = ["org.my.lib.Integration"]
}
For build.gradle.kts
:
tasks.processJupyterApiResources {
libraryProducers = listOf("org.my.lib.Integration")
}
You can automatically check if your library integrates correctly into the kernel. To achieve this, inherit your
test class from the org.jetbrains.kotlinx.jupyter.testkit.JupyterReplTestCase
class and use its methods to execute cells.
Your library integration descriptors should be already in the classpath and will be loaded automatically by the test logic.
You don't need to use the %use
line magic or the DependsOn
annotation to switch on your library.
The artifact containing test templates is included automatically into the testImplementation
configuration if you
use the Gradle plugin. You can turn this behavior off by setting the kotlin.jupyter.add.testkit
Gradle property
to false
. If you want to manually include this artifact in your build, see the instructions
here.
For examples of integration testing, see JupyterReplTestingTest
in
this repository
or related tests in DataFrame.
If you don't use Gradle as a build system, there is an alternative to integrate a library with the Kotlin Kernel for Jupyter notebooks:
-
Add the
org.jetbrains.kotlinx:kotlin-jupyter-api:<jupyterApiVersion>
dependency as a compile dependency. For configuration instructions for different build systems, see the documentation. -
Add one or more integration classes. Integration classes are derived from the
LibraryDefinitionProducer
orLibraryDefinition
interfaces. In this scenario, you don't need the@JupyterLibrary
annotation. -
Add the file
META-INF/kotlin-jupyter-libraries/libraries.json
to the JAR resources. This file should contain FQNs of all integration classes in the JSON form:
{
"definitions": [],
"producers": [
{
"fqn": "org.jetbrains.kotlinx.jupyter.example.GettingStartedIntegration"
}
]
}
Classes derived from the LibraryDefinition
interface should be added to the definitions
array.
Classes derived from the LibraryDefinitionProducer
interface should be added to the producers
array.
For more information, see: