[Java] "Language server crashed 5 times within the last 3 minutes"

[Sandared]

TLDR: Lombok sets a java agent in the user settings. When you open a workspace without lombok extension, the LS crashes, because the agent is not there.

---

Describe the bug
When I try to open this GitHub repository with GitPod then I get the error message
"Language server crashed x times within the last x minutes" and I can't code anymore.

To Reproduce
Go to https://github.com/DataDog/dd-trace-java, click the Gitpod button in GitHub and wait for the error message.

Expected behavior
Language server should not crash?

Additional Information

[akosyakov]

I cannot reproduce it with the given repo. Could you share the workspace id please with which it happened I would like to investigate Theia logs?

[Sandared]

The workspaces I got this error in are:
a95dbd4e-c3e9-44ce-b57d-ff18773307b2
fa8e10ef-2d04-40dc-80bb-3481f52c8f87
f2f84478-d944-42b0-9de8-c96b1e279a87

[akosyakov]

@Sandared thank you, btw does it works for you again?

[Sandared]

@akosyakov
Unfortunately no. I can open the repo in Gitpod but the Java language server does not work.

[akosyakov]

@Sandared if you apply this configuration: #1018 (comment)

Does it work?

[akosyakov]

Unfortunately no. I can open the repo in Gitpod but the Java language server does not work.

Could you specify what you do? Which editor do you open an so on in https://github.com/DataDog/dd-trace-java?

[Sandared]

No unfortunately not :( But now the error messagen about the crashing language server pops up almost instantly

[akosyakov]

@Sandared or take a workspace snapshot and share it here, just don't delete the original workspace.

Please also have a look into View -> Output -> Language Support for Java. If there is nothing try to enable in settings "java.trace": "verbose" and "java.server.trace": "verbose" and reload the page. Maybe you will get some errors in the output.

[Sandared]

@akosyakov here you go: https://gitpod.io#snapshot/12eb7107-4690-4837-a3eb-50ac73643112

There is no Language Suport for Java option for mw, just for Output. I also tried to set the properties you mentioned, but that didn't change anything :/

[akosyakov]

Unfortunately, it runs for me as well :(

@Sandared Can we have a call to debug it together?

[Sandared]

Uh I finally managed to see some output for the language server: It complains about some missing lombok.jar

[Trace - 12:44:08 PM] Sending request 'initialize - (0)'.
Params: {
    "processId": 4028,
    "rootPath": "/workspace/dd-trace-java",
    "rootUri": "file:///workspace/dd-trace-java",
    "capabilities": {
        "workspace": {
            "applyEdit": true,
            "workspaceEdit": {
                "documentChanges": true,
                "resourceOperations": [
                    "create",
                    "rename",
                    "delete"
                ],
                "failureHandling": "textOnlyTransactional"
            },
            "didChangeConfiguration": {
                "dynamicRegistration": true
            },
            "didChangeWatchedFiles": {
                "dynamicRegistration": true
            },
            "symbol": {
                "dynamicRegistration": true,
                "symbolKind": {
                    "valueSet": [
                        1,
                        2,
                        3,
                        4,
                        5,
                        6,
                        7,
                        8,
                        9,
                        10,
                        11,
                        12,
                        13,
                        14,
                        15,
                        16,
                        17,
                        18,
                        19,
                        20,
                        21,
                        22,
                        23,
                        24,
                        25,
                        26
                    ]
                }
            },
            "executeCommand": {
                "dynamicRegistration": true
            },
            "configuration": true,
            "workspaceFolders": true
        },
        "textDocument": {
            "publishDiagnostics": {
                "relatedInformation": true,
                "tagSupport": true
            },
            "synchronization": {
                "dynamicRegistration": true,
                "willSave": true,
                "willSaveWaitUntil": true,
                "didSave": true
            },
            "completion": {
                "dynamicRegistration": true,
                "contextSupport": true,
                "completionItem": {
                    "snippetSupport": true,
                    "commitCharactersSupport": true,
                    "documentationFormat": [
                        "markdown",
                        "plaintext"
                    ],
                    "deprecatedSupport": true,
                    "preselectSupport": true
                },
                "completionItemKind": {
                    "valueSet": [
                        1,
                        2,
                        3,
                        4,
                        5,
                        6,
                        7,
                        8,
                        9,
                        10,
                        11,
                        12,
                        13,
                        14,
                        15,
                        16,
                        17,
                        18,
                        19,
                        20,
                        21,
                        22,
                        23,
                        24,
                        25
                    ]
                }
            },
            "hover": {
                "dynamicRegistration": true,
                "contentFormat": [
                    "markdown",
                    "plaintext"
                ]
            },
            "signatureHelp": {
                "dynamicRegistration": true,
                "signatureInformation": {
                    "documentationFormat": [
                        "markdown",
                        "plaintext"
                    ],
                    "parameterInformation": {
                        "labelOffsetSupport": true
                    }
                }
            },
            "definition": {
                "dynamicRegistration": true,
                "linkSupport": true
            },
            "references": {
                "dynamicRegistration": true
            },
            "documentHighlight": {
                "dynamicRegistration": true
            },
            "documentSymbol": {
                "dynamicRegistration": true,
                "symbolKind": {
                    "valueSet": [
                        1,
                        2,
                        3,
                        4,
                        5,
                        6,
                        7,
                        8,
                        9,
                        10,
                        11,
                        12,
                        13,
                        14,
                        15,
                        16,
                        17,
                        18,
                        19,
                        20,
                        21,
                        22,
                        23,
                        24,
                        25,
                        26
                    ]
                },
                "hierarchicalDocumentSymbolSupport": true
            },
            "codeAction": {
                "dynamicRegistration": true,
                "codeActionLiteralSupport": {
                    "codeActionKind": {
                        "valueSet": [
                            "",
                            "quickfix",
                            "refactor",
                            "refactor.extract",
                            "refactor.inline",
                            "refactor.rewrite",
                            "source",
                            "source.organizeImports"
                        ]
                    }
                }
            },
            "codeLens": {
                "dynamicRegistration": true
            },
            "formatting": {
                "dynamicRegistration": true
            },
            "rangeFormatting": {
                "dynamicRegistration": true
            },
            "onTypeFormatting": {
                "dynamicRegistration": true
            },
            "rename": {
                "dynamicRegistration": true,
                "prepareSupport": true
            },
            "documentLink": {
                "dynamicRegistration": true
            },
            "typeDefinition": {
                "dynamicRegistration": true,
                "linkSupport": true
            },
            "implementation": {
                "dynamicRegistration": true,
                "linkSupport": true
            },
            "colorProvider": {
                "dynamicRegistration": true
            },
            "foldingRange": {
                "dynamicRegistration": true,
                "rangeLimit": 5000,
                "lineFoldingOnly": true
            },
            "declaration": {
                "dynamicRegistration": true,
                "linkSupport": true
            },
            "selectionRange": {
                "dynamicRegistration": true
            },
            "callHierarchy": {
                "dynamicRegistration": true
            }
        },
        "window": {
            "workDoneProgress": true
        }
    },
    "initializationOptions": {
        "bundles": [
            "/tmp/vscode-extensions/[email protected]/extension/server/com.microsoft.java.debug.plugin-0.23.0.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/com.microsoft.java.test.plugin-0.22.0.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.eclipse.jdt.junit4.runtime_1.1.1000.v20190510-0840.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.eclipse.jdt.junit5.runtime_1.0.500.v20190510-0840.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.jupiter.api_5.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.jupiter.engine_5.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.jupiter.migrationsupport_5.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.jupiter.params_5.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.vintage.engine_5.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.opentest4j_1.1.1.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.platform.commons_1.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.platform.engine_1.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.platform.launcher_1.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.platform.runner_1.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.junit.platform.suite.api_1.4.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/org.apiguardian_1.0.0.v20190212-2109.jar",
            "/tmp/vscode-extensions/[email protected]/extension/server/com.microsoft.jdtls.ext.core-0.5.1.jar"
        ],
        "workspaceFolders": [
            "file:///workspace/dd-trace-java"
        ],
        "settings": {
            "java": {
                "debug": {
                    "logLevel": "warn",
                    "settings": {
                        "showHex": false,
                        "showStaticVariables": false,
                        "showQualifiedNames": false,
                        "showLogicalStructure": true,
                        "showToString": true,
                        "maxStringLength": 0,
                        "hotCodeReplace": "manual",
                        "enableRunDebugCodeLens": true,
                        "forceBuildBeforeLaunch": true,
                        "console": "integratedTerminal"
                    }
                },
                "home": "/home/gitpod/.sdkman/candidates/java/current",
                "jdt": {
                    "ls": {
                        "vmargs": "-noverify -Xmx1G -XX:+UseG1GC -XX:+UseStringDeduplication -javaagent:\"/tmp/vscode-extensions/[email protected]/extension//server/lombok.jar\""
                    }
                },
                "errors": {
                    "incompleteClasspath": {
                        "severity": "ignore"
                    }
                },
                "configuration": {
                    "checkProjectSettingsExclusions": true,
                    "updateBuildConfiguration": "interactive",
                    "maven": {
                        "userSettings": null
                    }
                },
                "trace": {
                    "server": "verbose"
                },
                "import": {
                    "maven": {
                        "enabled": true
                    },
                    "gradle": {
                        "enabled": true,
                        "wrapper": {
                            "enabled": true
                        },
                        "version": null,
                        "arguments": null,
                        "jvmArguments": null,
                        "home": null
                    },
                    "exclusions": [
                        "**/node_modules/**",
                        "**/.metadata/**",
                        "**/archetype-resources/**",
                        "**/META-INF/maven/**"
                    ]
                },
                "maven": {
                    "downloadSources": false,
                    "updateSnapshots": false
                },
                "referencesCodeLens": {
                    "enabled": false
                },
                "signatureHelp": {
                    "enabled": false
                },
                "implementationsCodeLens": {
                    "enabled": false
                },
                "format": {
                    "enabled": true,
                    "settings": {
                        "url": null,
                        "profile": null
                    },
                    "comments": {
                        "enabled": true
                    },
                    "onType": {
                        "enabled": true
                    }
                },
                "saveActions": {
                    "organizeImports": false
                },
                "contentProvider": {
                    "preferred": null
                },
                "autobuild": {
                    "enabled": true
                },
                "maxConcurrentBuilds": 1,
                "completion": {
                    "enabled": true,
                    "overwrite": true,
                    "guessMethodArguments": false,
                    "favoriteStaticMembers": [
                        "org.junit.Assert.*",
                        "org.junit.Assume.*",
                        "org.junit.jupiter.api.Assertions.*",
                        "org.junit.jupiter.api.Assumptions.*",
                        "org.junit.jupiter.api.DynamicContainer.*",
                        "org.junit.jupiter.api.DynamicTest.*",
                        "org.mockito.Mockito.*",
                        "org.mockito.ArgumentMatchers.*",
                        "org.mockito.Answers.*"
                    ],
                    "filteredTypes": [
                        "java.awt.*",
                        "com.sun.*"
                    ],
                    "importOrder": [
                        "java",
                        "javax",
                        "com",
                        "org"
                    ]
                },
                "foldingRange": {
                    "enabled": true
                },
                "progressReports": {
                    "enabled": true
                },
                "codeGeneration": {
                    "hashCodeEquals": {
                        "useJava7Objects": false,
                        "useInstanceof": false
                    },
                    "useBlocks": false,
                    "generateComments": false
                },
                "selectionRange": {
                    "enabled": true
                },
                "showBuildStatusOnStart": {
                    "enabled": false
                },
                "test": {
                    "report": {
                        "showAfterExecution": "onFailure",
                        "position": "sideView"
                    },
                    "editor": {
                        "enableShortcuts": true
                    },
                    "log": {
                        "level": "info"
                    },
                    "message": {
                        "hintForDeprecatedConfig": true,
                        "hintForSetingDefaultConfig": true
                    },
                    "defaultConfig": "",
                    "config": {}
                },
                "dependency": {
                    "showOutline": true,
                    "syncWithFolderExplorer": true,
                    "autoRefresh": true,
                    "packagePresentation": "flat"
                }
            }
        },
        "extendedClientCapabilities": {
            "progressReportProvider": true,
            "classFileContentsSupport": true,
            "overrideMethodsPromptSupport": true,
            "hashCodeEqualsPromptSupport": true,
            "advancedOrganizeImportsSupport": true,
            "generateToStringPromptSupport": true,
            "advancedGenerateAccessorsSupport": true,
            "generateConstructorsPromptSupport": true,
            "generateDelegateMethodsPromptSupport": true,
            "advancedExtractRefactoringSupport": true,
            "moveRefactoringSupport": true,
            "clientHoverProvider": true
        },
        "triggerFiles": [
            "file:///workspace/dd-trace-java/dd-trace-ot/src/main/java/datadog/trace/common/writer/DDAgentWriter.java"
        ]
    },
    "trace": "verbose",
    "workspaceFolders": [
        {
            "uri": "file:///workspace/dd-trace-java",
            "name": "dd-trace-java"
        }
    ]
}
[Error - 12:44:08 PM] Connection to server is erroring. Shutting down server.
[Trace - 12:44:08 PM] Sending request 'shutdown - (1)'.
[Error - 12:44:08 PM] Connection to server is erroring. Shutting down server.
[Error - 12:44:08 PM] Connection to server is erroring. Shutting down server.
Picked up JAVA_TOOL_OPTIONS: -Xmx2254m
Error opening zip file or JAR manifest missing : /tmp/vscode-extensions/[email protected]/extension//server/lombok.jar

[Sandared]

I added the lombok plugin to the non working workspaces and after a restart they do work again...

I did use the lombok plugin for VSCode in one of my other projects. May this somehow influence other workspaces? Are the used plugins not per workspace but per user?

[akosyakov]

Uh I finally managed to see some output for the language server: It complains about some missing lombok.jar

Could you sum up how you did for others? It seems that Java extension by default does not log any errors, so nothing left in logs. We need some instruction to collect error logs.

I did use the lombok plugin for VSCode in one of my other projects. May this somehow influence other workspaces? Are the used plugins not per workspace but per user?

No, we don't have user extensions yet. For each workspace extensions are installed according to .gitpod.yml file. Could it be that you added and removed lombok vscode extension in broken workspaces in the past.

[dhavalmshah]

A workaround as suggested by @Sandared seems to get the work space going.
I ran the following commands:

mkdir -p /tmp/vscode-extensions/[email protected]/extension//server
curl https://projectlombok.org/downloads/lombok.jar -o /tmp/vscode-extensions/[email protected]/extension/server/lombok.jar

and after that just refresh the browser page, and it should start working
however this has to be done every time you start the workspace :(

[akosyakov]

@dhavalmshah You can install [email protected] VS Code extension for your repo once, then it will be persisted into .gitpod.yml and installed for each workspace.

[akosyakov]

@Sandared Could you check please your user settings for something about java jvm args and lombok? Maybe it was inserted by java extension in one workspace, and now picked up in another.

[Sandared]

🤦‍♂️ my user settings contained this line: "java.jdt.ls.vmargs": "-noverify -Xmx1G -XX:+UseG1GC -XX:+UseStringDeduplication -javaagent:\"/tmp/vscode-extensions/[email protected]/extension//server/lombok.jar\"",

I'm pretty sure I never set this manually, so maybe the Lombok extension did this when I installed it in one of my workspaces? ... sorry for the inconvenience.

[akosyakov]

I'm pretty sure I never set this manually, so maybe the Lombok extension did this when I installed it in one of my workspaces? ... sorry for the inconvenience.

Yeah, but it is our fault that we don't remove such settings if extension is not installed as well.

[akosyakov]

There is uninstall hook https://code.visualstudio.com/api/references/extension-manifest#extension-uninstall-hook which is used by lombok extension: https://github.com/GabrielBB/vscode-lombok/blob/f7021aa249475311261e3993b2caf434ee1f109b/src/uninstall.ts#L2

Unfortunately vscode api cannot be used in such hook, so lombok extension changes code settings file. It looks hacky and does not help us since we store it other place. There is already an issue in VS Code to handle it somehow: microsoft/vscode#45474

It won't work in VS Code insider and also there are similar issues about lombok on redhat.java.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[amentler]

I have the same Problem with a lombok plugin . Im happy to have found this issue here but i dont know how to really fix it, except adding the plugin to my workspace, though i dont need it there!
i used the plugin within another workspace and perhaps i installed it for me as a user and not just for the workspace (i deinstalled it later).
the repository i am using works for other people so its propably not the repositorys fault.

the id of the not working workspace is: e7a2a741-fb3c-4584-920b-aa98544f0f66

is it possible for me to repair it myself? there must be remains somewhere. im currently trying to grep everything lombok related but until now i found nothing but the log file.

{ message: 'Picked up JAVA_TOOL_OPTIONS: -Xmx2254m\nError opening zip file or JAR manifest missing : /tmp/vscode-extensions/[email protected]/extension/server/lombok.jar\n', level: 'info', timestamp: '2020-10-08 15:24:49.224' } { message: 'The Language Support for Java server crashed 5 times in the last 3 minutes. The server will not be restarted.', level: 'error', timestamp: '2020-10-08 15:24:49.225' } { message: '[Error - 3:24:49 PM] Connection to server got closed. Server will not be restarted.', level: 'info', timestamp: '2020-10-08 15:24:49.226' } { message: 'Picked up JAVA_TOOL_OPTIONS: -Xmx2254m\n', level: 'info', timestamp: '2020-10-08 15:24:49.227' } { message: 'Error opening zip file or JAR manifest missing : /tmp/vscode-extensions/[email protected]/extension/server/lombok.jar\n', level: 'info', timestamp: '2020-10-08 15:24:49.228' } { message: 'The Language Support for Java (Syntax Server) server crashed 5 times in the last 3 minutes. The server will not be restarted.', level: 'error', timestamp: '2020-10-08 15:24:49.229' } { message: '[Error - 3:24:49 PM] Connection to server got closed. Server will not be restarted.', level: 'info', timestamp: '2020-10-08 15:24:49.229' }

[Sandared]

Hey :)
If you are not using Lombok, but you still get the error, then you can remove this part from your settings.json:
-javaagent:\"/tmp/vscode-extensions/[email protected]/extension//server/lombok.jar\"" and be fine.

[amentler]

where is that file?
in the root i did a
find -name "settings.json" -exec grep -i "lombok" '{}' \;
and a
find -name "settings.json" -exec cat {} \;

there is no settings.json that contains something similar to that line.

[Sandared]

I think you can only Open that File directly in a running Gitpod workspace via Ctrl+;

[amentler]

thanks! i had been in the settings before but i hadnt found the right setting or the button to go into text mode.

the solution was to either edit the settings ( ctrl+; )in the text mode using that tiny little button on the top right of that window showing {} (bad usability) and then deleting -javaagent:\"/tmp/vscode-extensions/[email protected]/extension//server/lombok.jar\"" (what sandared suggested).
or to search for

Jdt
Jdt Ls Vmargs
Specifies extra VM arguments used to launch the Java Language Server. Eg. use `-XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xmx1G -Xms100m ` to optimize memory usage for container environments with the parallel garbage collector

deleting what has been said.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.