Skip to content

Conversation

@JaroslavTulach
Copy link
Member

@JaroslavTulach JaroslavTulach commented Sep 23, 2025

Pull Request Description

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

@JaroslavTulach JaroslavTulach self-assigned this Sep 23, 2025
@JaroslavTulach JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label Sep 23, 2025
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Sep 23, 2025

  • @AdRiley, I am trying to verify this change is correct. However doing:
corepack pnpm i
corepack pnpm compile
corepack pnpm dev:gui
image
  • and then I can run (and fail) the tests:
enso$ ENSO_LAUNCHER=native sbt buildEngineDistribution

    - [FAILED] can read multiple sheets - error case - With_New_Column - no valid sheets [297ms]
        Reason: An unexpected panic was thrown: (Unsupported_Argument_Types.Error [] 'Cannot convert \'CredentialReference[secretId=secret-3360Yd1PR0XaCGaifwwFOJrqb3d, serviceName
    - [FAILED] can read multiple sheets - error case - With_New_Column - no valid sheets [297ms]
        Reason: An unexpected panic was thrown: (Unsupported_Argument_Types.Error [] 'Cannot convert \'CredentialReference[secretId=secret-3360Yd1PR0XaCGaifwwFOJrqb3d, serviceName=Google]\'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap) to Java type \'org.enso.base.enso_cloud.ExternalLibraryCredentialHelper$CredentialReference\': Unsupported target type.')
        at <enso> case_branch(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:177:13-93)
        at <enso> Google_Sheets_Workbook._wrap_credentials(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:175-182)
        at <enso> Google_Sheets_Workbook.type.new<arg-1>(/home/devel/NetBeansPro
    - [FAILED] can read multiple sheets - error case - With_New_Column - no valid sheets [297ms]
        Reason: An unexpected panic was thrown: (Unsupported_Argument_Types.Error [] 'Cannot convert \'CredentialReference[secretId=secret-3360Yd1PR0XaCGaifwwFOJrqb3d, serviceName=Google]\'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap) to Java type \'org.enso.base.enso_cloud.ExternalLibraryCredentialHelper$CredentialReference\': Unsupported target type.')
        at <enso> case_branch(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:177:13-93)
        at <enso> Google_Sheets_Workbook._wrap_credentials(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:175-182)
        at <enso> Google_Sheets_Workbook.type.new<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:47-75)
        at <enso> Google_Sheets_Workbook.type.new(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:19-76)
        at <enso> Google_Sheets.read(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets.enso:15:9-57)
        at <enso> Main.add_specs.Main.add_specs<arg-1>(/home/devel/NetBeansProjects/enso/enso/test/Google_Test/src/Main.enso:311:18-115)
        at <enso> case_branch.case <internal-1178>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Group.enso:37:61-64)
        at <enso> Helpers.execute_spec_code<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Helpers.enso:56:18-34)
   jects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:47-75)
        at <enso> Google_Sheets_Workbook.type.new(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:19-76)
        at <enso> Google_Sheets.read(/home/devel/NetBeansProjects/enso/enso/distrib
    - [FAILED] can read multiple sheets - error case - With_New_Column - no valid sheets [297ms]
        Reason: An unexpected panic was thrown: (Unsupported_Argument_Types.Error [] 'Cannot convert \'CredentialReference[secretId=secret-3360Yd1PR0XaCGaifwwFOJrqb3d, serviceName=Google]\'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap) to Java type \'org.enso.base.enso_cloud.ExternalLibraryCredentialHelper$CredentialReference\': Unsupported target type.')
        at <enso> case_branch(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:177:13-93)
        at <enso> Google_Sheets_Workbook._wrap_credentials(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:175-182)
        at <enso> Google_Sheets_Workbook.type.new<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:47-75)
        at <enso> Google_Sheets_Workbook.type.new(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:19-76)
        at <enso> Google_Sheets.read(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets.enso:15:9-57)
        at <enso> Main.add_specs.Main.add_specs<arg-1>(/home/devel/NetBeansProjects/enso/enso/test/Google_Test/src/Main.enso:311:18-115)
        at <enso> case_branch.case <internal-1178>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Group.enso:37:61-64)
        at <enso> Helpers.execute_spec_code<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Helpers.enso:56:18-34)
   ution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets.enso:15:9-57)
        at <enso> Main.add_specs.Main.add_specs<arg-1>(/home/devel/NetBeansProjects/enso/enso/test/Google_Test/src/Main.enso:311:18-115)
        at <enso> case_branch.case <internal-1178>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Group.enso:37:61-64)
        at <enso> Helpers.execute_spec_code<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Helpers.enso:56:18-34)
   =Google]\'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap) to Java type \'org.enso.base.enso_cloud.ExternalLibraryCredentialHelper$CredentialReference\': Unsupported target type.')
        at <enso> case_branch(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:177:13-93)
        at <enso> Google_Sheets_Workbook._wrap_credentials(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:175-182)
        at <enso> Google_Sheets_Workbook.type.new<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:47-75)
        at <enso> Google_Sheets_Workbook.type.new(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets_Workbook.enso:45:19-76)
        at <enso> Google_Sheets.read(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Google/0.0.0-dev/src/Google_Sheets.enso:15:9-57)
        at <enso> Main.add_specs.Main.add_specs<arg-1>(/home/devel/NetBeansProjects/enso/enso/test/Google_Test/src/Main.enso:311:18-115)
        at <enso> case_branch.case <internal-1178>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Group.enso:37:61-64)
        at <enso> Helpers.execute_spec_code<arg-1>(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Test/0.0.0-dev/src/Helpers.enso:56:18-34)

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Oct 7, 2025

I have all the setup to execute the tests in regular JVM mode:

enso$ sbt
sbt:enso> runEngineDistribution --run test/Google_Test
48 tests succeeded.
0 tests failed.
0 tests skipped.
0 groups skipped.
[success] Total time: 102 s (01:42), completed 7. 10. 2025 11:12:50

now onto running the same tests in dual JVM mode. With bc03db6 there is eleven remaining failures:

sbt:enso> runEngineDistribution --run test/Google_Test 
  --vm.D=polyglot.enso.classLoading=Standard.Google:guest,enso_dev.Google_Test:guest,hosted

37 tests succeeded.
11 tests failed.
0 tests skipped.
0 groups skipped.
[success] Total time: 105 s (01:45), completed 7. 10. 2025 17:18:47

we are on track to get the dual JVM mode working for Standard.Google library!

@JaroslavTulach JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Oct 7, 2025
var emptySheetError = emptySheetType.invokeMember("Error");
var errorType = EnsoMeta.getType("Standard.Base.Error", "Error");
var error = errorType.invokeMember("throw", emptySheetError);
throw error.throwException();
Copy link
Member Author

@JaroslavTulach JaroslavTulach Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • cdbf8fb is a solution to dual exceptions problem, @AdRiley
  • rather than throwing Java exception and then trying to convert it to Empty_Sheet.Error
  • let's directly throw Empty_Sheet.Error!
  • the getSheetRange method returns Table - as such we need to throw it as Panic (hence error.throwException()
    • otherwise just return error; to propagate the value as (dataflow) Error
  • because we throw Panic we have Panic.recover in the Enso code
  • if we returned (dataflow) Error - we could delete the Enso conversion code completely!

Copy link
Member Author

@JaroslavTulach JaroslavTulach Oct 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • we can go even further with this approach, @jdunkerley
  • if we just return error; then there is no need to do anything special on the Enso side
  • as de343f3 illustrates, it is possible to remove Empty_Sheet.handle_java_exception
  • the only drawback of the de343f3 is the change of function return type to Object
    • we need to return either Table or Value and Object is the only common superclass
  • the throw error.throwException() solution workarounds this by returning via an exception (at the cost of necessary handler on the Enso side)

@JaroslavTulach
Copy link
Member Author

Right now we have:

  • failing JUnit tests because of changes to DataflowError.throwException():
[error] (runtime-language-epb / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] (runtime-integration-tests / Test / test) sbt.TestsFailedException: Tests unsuccessful
  • one API change because of adding default Java class argument to as_credential_reference
  • intermittent(?) stalled(?) test on Windows

Other than that we are ready to move on. Heuréka!

error!("Current API vs Old API: {}", err);
error!("If you wish to overwrite the current API in the directory {}, run the following command {},
error!("If you wish to overwrite the current API in the directory {}, run the following command
sbt \"runEngineDistribution --no-ir-caches --docs=api --in-project {}\"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change, @AdRiley, @GregoryTravis, @jdunkerley, will print:

index 8b63b3d24d..f2c90e184d 100644
--- home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Table/0.0.0-dev/docs/api/Errors.md
+++ home/devel/NetBeansProjects/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Table/2025.3.1-dev/docs/api/Errors.md
@@ -43,7 +43,7 @@
     - to_display_text self -> Standard.Base.Any.Any
 - type Empty_Sheet
     - Error
-    - handle_java_exception -> Standard.Base.Any.Any
+    - handle_java_exception ~action:Standard.Base.Any.Any -> Standard.Base.Any.Any
     - to_display_text self -> Standard.Base.Any.Any
 - type Existing_Column
     - Error column_name:Standard.Base.Any.Any

ERROR main_internal: enso_build::engine::context: If you wish to overwrite the current API in the directory distribution/lib/Standard/Table/0.0.0-dev/docs/api, run the following command 
  sbt "runEngineDistribution --no-ir-caches --docs=api --in-project distribution/lib/Standard/Table/0.0.0-dev"
  and commit the modified files
ERROR main_internal: enso_build_cli: error=API check failed for library Standard.Table
 INFO main_internal: enso_build_cli: close
Error: API check failed for library Standard.Table

in case of API check failure. Which I can then directly copy and run the regenerator:

enso$ sbt "runEngineDistribution --no-ir-caches --docs=api --in-project distribution/lib/Standard/Table/0.0.0-dev"

Please note the --no-ir-caches flag added for your convenience as the actionable applied after recent complains.

@JaroslavTulach JaroslavTulach force-pushed the wip/jtulach/DualGoogle branch from c1f5e6e to 53f1b1a Compare October 8, 2025 06:57
mergify bot pushed a commit that referenced this pull request Nov 10, 2025
- [Problems upgrading](#14040 (comment)) to most recent develop
- likely caused by [allowCoreThreadsTimeout](https://github.com/enso-org/enso/pull/13907/files#r2503655230)
- require us to allow access to `Channel` from multiple threads.
- technically we would disable `allowCoreThreadsTimeout`, but
- we will multi threaded access in the future anyway...
- this PR enables such an access.

# Important Notes
- only master can initialize connection on a new thread
- then slave can reply on such a thread or any other thread which was initiated previously by the master
- if slave tries to reply on a new thread created in its own JVM, such a message will yield an `IllegalStateException`
- the relation between master and slave is **asymmetric** with respect to **threading**
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Nov 12, 2025

  • Support for GoogleSheets credentials has been restored
  • @AdRiley sample project seems to work for me both in enso-linux-x86_64-2025.3.1-nightly.2025.11.9.AppImage and the version in this PR
obrazek

`database-polyglot-root`
.listFiles("*.jar")
.map(_.getAbsolutePath()) ++
`google-polyglot-root`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The main change of this PR
  • std-bits/google are no longer included when building enso NI

@JaroslavTulach
Copy link
Member Author

Conversions between Table from different JVMs

obrazek
* the location/stacktrace of `Cannot convert 'org.enso.table.data.table.Table@8d8f754'(language: Enso Polyglot Bridge, type: org.enso.table.data.table.Table) to Java type 'org.enso.table.data.table.Table': Unsupported target type.`  isn't printed anywhere...
  • these conversion problems seem to be gone
  • I believe @jdunkerley's Adding DuckDB type mapping and InDB support #14263 plays a significant role here
  • by associating a Table with its JVM and selecting the right classes to invoke from the right JVM
  • the convertions are no longer happening and the correct JVM is consulted to provide the result

Convert a Java problem into its Enso equivalent.
translate_problem p = case p of
_ : InvalidAggregation ->
translate_problem p = case p.problemType of
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I like this approach, and wonder if each Problem should be responsible for making an Enso object.

@github-project-automation github-project-automation bot moved this from 🌟 Q/A review to 🔴 Changes requested in Issues Board Nov 12, 2025
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Nov 12, 2025

Re. this comment:

I really don't like making these classes Enso dependent - we debug them a lot in pure Java.
Particularly the Excel code if complex and hard to work with.

  • debugging in "pure Java" is possible even when referencing Enso
  • however the cost of duplicating every error and exception cannot really be addressed
    • EmptySheetException & Empty_Sheet isn't the only example
    • there is a ton of such "exception/error pairs" all over the codebase
  • I don't think it is a good coding pattern
    • it is very verbose - tons of lines in Java and many in Enso
    • it spreads handle_xyz_exception methods all over unrelated places
    • chance of forgetting to catch the Java exception somewhere isn't zero
    • simpler to just create the Enso Error and/or Panic directly
  • moreover Error.throw or Panic.throw will play much more nicely with dual JVM mode

Copy link
Member

@jdunkerley jdunkerley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the libs changes.

@jdunkerley
Copy link
Member

I think with your solution we can move in this direction.

We can still keep the JUnit tests within the std-bits libraries and keep the tight code cycle I've been there when needed.
I agree having error/exception is not nice code pattern and we end up having it everywhere as this is the only solution we had.

I'll try out the approach and see how it goes going forward.

@JaroslavTulach JaroslavTulach moved this from 🔴 Changes requested to 🌟 Q/A review in Issues Board Nov 13, 2025
@JaroslavTulach JaroslavTulach merged commit d2cafb7 into develop Nov 13, 2025
102 of 104 checks passed
@JaroslavTulach JaroslavTulach deleted the wip/jtulach/DualGoogle branch November 13, 2025 17:28
@github-project-automation github-project-automation bot moved this from 🌟 Q/A review to 🟢 Accepted in Issues Board Nov 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

-compiler -libs Libraries: New libraries to be implemented -libs-API-change-Base Marks a PR that changes the public API of Standard.Base CI: Clean build required CI runners will be cleaned before and after this PR is built. CI: No changelog needed Do not require a changelog entry for this PR.

Projects

Status: 🟢 Accepted

Development

Successfully merging this pull request may close these issues.

5 participants