Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Scala Native #1172

Open
3 of 5 tasks
olafurpg opened this issue May 10, 2018 · 36 comments
Open
3 of 5 tasks

Support Scala Native #1172

olafurpg opened this issue May 10, 2018 · 36 comments
Assignees

Comments

@olafurpg
Copy link
Member

olafurpg commented May 10, 2018

It would be awesome to support the scalafmt cli on native to avoid JVM startup time. I managed to get it working with publishLocal a while ago and experienced ~30x speedup formatting a single file (1.5s on JVM vs 50ms on Native)

https://twitter.com/olafurpg/status/857559907876433920

The following milestones are now complete

We are only a few steps away from being able to properly support Scala Native. The missing pieces are

@virusdave
Copy link

Any updates on this? Any remaining blockers people can help with?

@olafurpg
Copy link
Member Author

@virusdave No update. Native support has been merged into paiges but it's not yet released, scalafmt doesn't depend on paiges directly but it's brought in transitively.

Biggest blocker is that the pure Scala configuration parser is not 100% compatible with the standard HOCON parser. I'm not sure what's the best move there, I don't like the idea of having slightly incompatible parsers on Native/JVM. We could use the pure Scala parser for JVM+Native but that would mean using a non-standard configuration syntax and impose a breaking change on everybody (including non-Native users).

My favorite option at the moment is to manually port the 11,000 line Java HOCON implementation to Scala. It should be a line-by-line translation, drop-in replacement. Related discussion lightbend/config#552

Another alternative is to support .scalafmt.json (several JSON parsers support Native already) and declare that as the official syntax moving forward.

@ssaavedra
Copy link
Contributor

To be fair, some improvements are already in place: paiges is already available for scala-native and metaconfig seems to work for native too although it's not in Central.

I tried something different, that is AOT compilation with GraalVM's native-image, which would also avoid the VM warmup penalty and we would still enjoy Lightbend's config for HOCON.

Compilation worked fine (I did an sbt cli/assembly and then native-image -jar scalafmt.jar) but when running the ELF with something like scalafmt $PWD/SomeClass.scala execution halts due to a com.oracle.svm.core.jdk.UnsupportedFeatureError: Code that was considered unreachable by closed-world analysis was reached with this stacktrace.

I'm not sure if it's a faster approach to go towards scala-native or graalvm, but I'm raising the issue to consider it.

@olafurpg
Copy link
Member Author

Thanks for sharing @ssaavedra. How would graalvm native-image work in terms of distribution and testing? What I find compelling about Scala Native is that I can

  • run the full test suite in native via sbt test, just like on the JVM
  • publish native artifacts to maven central like normal and users link the binary for their platform with a single command coursier bootstrap com.geirsson:scalafmt-cli_native:VERSION --native. Users will not need to compile scalafmt from source.

@ssaavedra
Copy link
Contributor

Also, if you could give steps to reproduce a working usage of scalafmt under scala-native it would be much appreciated: I have a publishLocal version of metaconfig (native) and paiges (native) and even disabled testing, but I'm still unable to make things compile. When I create a cliNative project, and build it interactively setting set scalaVersion in ThisBuild := "2.11.12" and cliNative/nativeLink, I get the following error:

[info] Done compiling.
[info] Linking (5210 ms)
[error] cannot link: @java.lang.Thread::init
[error] cannot link: @java.lang.Thread::join_unit
[error] cannot link: @java.lang.Thread::setDaemon_bool_unit
[error] cannot link: @java.lang.Thread::start_unit
[error] cannot link: @java.util.concurrent.Future
[error] cannot link: @java.util.concurrent.atomic.AtomicReference::getAndUpdate_java.util.function.UnaryOperator_java.lang.Object
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinPool
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinPool::execute_scala.concurrent.forkjoin.ForkJoinTask_unit
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinPool::getParallelism_i32
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinTask
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinTask::fork_scala.concurrent.forkjoin.ForkJoinTask
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinTask::join_java.lang.Object
[error] cannot link: @scala.concurrent.forkjoin.ForkJoinWorkerThread
[error] cannot link: @scala.concurrent.forkjoin.RecursiveAction
[error] cannot link: @scala.concurrent.forkjoin.RecursiveAction::init
[error] unable to link
[error] (cliNative / Compile / nativeLink) unable to link
[error] Total time: 9 s, completed Aug 30, 2018 4:23:39 PM

@olafurpg
Copy link
Member Author

The scalafmt cli currently uses parallel collections which I suspect use JVM concurrency features. To link the cli in native it probably needs a bit of refactoring.

@olafurpg
Copy link
Member Author

For people interested in native scalafmt, please give the branch in #1266 a try and report back how it works :)

@ekrich
Copy link

ekrich commented Jan 7, 2019

Some update on this. I have ported the config library and published for JVM and have a few branches to add to a potential Scala Native 0.3.9 release. I have successfully run the bit of code used to parse scalafmt config files in Scala Native.

https://github.com/ekrich/sconfig

PR to use this in metaconfig which is used by scalafmt - I think the default still uses lightbend/config for JVM.
olafurpg/metaconfig@b2363df

@ekrich
Copy link

ekrich commented Mar 22, 2019

We now have the changes in Scala Native 0.3.x to allow sconfig to cross compile to Scala Native and run the code metaconfig uses to parse the scalafmt HOCON config file.

@larsrh
Copy link

larsrh commented Mar 31, 2019

Paiges available now.

@He-Pin
Copy link

He-Pin commented Jan 19, 2021

Scala native 0.4.0 is just released https://scala-native.readthedocs.io/en/v0.4.0/changelog/0.4.0.html

@ekrich
Copy link

ekrich commented Jan 19, 2021

Status of sconfig - ekrich/sconfig#139

@ekrich
Copy link

ekrich commented Feb 25, 2021

I have released a version of java.time, sjavatime, and a new version of sconfig that should be ready and not be blocking this anymore.

@tgodzik
Copy link
Contributor

tgodzik commented Feb 25, 2021

I think biggest blocking issue is scalameta currently :/ The scala native support for removed some time ago and we need to add it back.

@tgodzik
Copy link
Contributor

tgodzik commented Mar 3, 2021

Ok, I almost got something working : scalameta/scalameta#2261

@tgodzik
Copy link
Contributor

tgodzik commented Mar 3, 2021

There is a compilation error though, but we are looking into this.

@ekrich
Copy link

ekrich commented Mar 4, 2021

I wanted to point out this although I am not sure if it has been solved or not yet - scala-native/scala-native#1575

The compilation error should be fixed in master - scala-native/scala-native@0ded408

I see you have provided a workaround in scalameta here: scalameta/scalameta@2db5f93#diff-865573f4d2cf3a17bdfd191d2665095bf22d743bc9433628d09b677103786d54L270-L281

@tgodzik tgodzik self-assigned this May 24, 2021
@ekrich
Copy link

ekrich commented Jun 4, 2021

@tgodzik Let me know if you need any help.

@tgodzik
Copy link
Contributor

tgodzik commented Jun 5, 2021

I am down to to errors:

[error] Found 2 missing definitions while linking
[error] Not found Top(scopt.Read$)
[error]         at /home/tgodzik/Documents/scalafmt/scalafmt-cli/shared/src/main/scala/org/scalafmt/cli/CliArgParser.scala:61
[error]         at /home/tgodzik/Documents/scalafmt/scalafmt-cli/shared/src/main/scala/org/scalafmt/cli/CliArgParser.scala:64
....
[error] Not found Top(org.scalafmt.interfaces.Scalafmt$)
[error]         at /home/tgodzik/Documents/scalafmt/scalafmt-cli/shared/src/main/scala/org/scalafmt/cli/ScalafmtDynamicRunner.scala:26

Second one is something I need to rewrite to Scala, since the default interfaces are written in Java, however I can't figure out why scopt is problematic since that is released for native. I do have to work on couple of things still, but I should get something compilable soon.

@tgodzik
Copy link
Contributor

tgodzik commented Jun 6, 2021

Weird thing, if I do import scopt.Read by hand it works, but I get:

[info] Linking (2383 ms)
[error] missing symbols:
[error] * M12java.net.URID5toURLL12java.net.URLEO
[error]   - from M34org.ekrich.config.impl.PlatformUriD5toURLL12java.net.URLEO
[error]   - from M33org.ekrich.config.impl.Parseable$D10relativeToL12java.net.URLL16java.lang.StringL12java.net.URLEO
[error]   - from M45org.ekrich.config.impl.Parseable$ParseableURLD10relativeToL16java.lang.StringL33org.ekrich.config.ConfigParseableEO
[error]   - from M33org.ekrich.config.impl.Parseable$D6newURLL12java.net.URLL36org.ekrich.config.ConfigParseOptionsL32org.ekrich.config.impl.ParseableEO
[error]   - from M32org.ekrich.config.ConfigFactory$D8parseURLL12java.net.URLL36org.ekrich.config.ConfigParseOptionsL24org.ekrich.config.ConfigEO
[error]   - from M38org.ekrich.config.impl.SimpleIncluder$D25includeURLWithoutFallbackL38org.ekrich.config.ConfigIncludeContextL12java.net.URLL30org.ekrich.config.ConfigObjectEO

and a bunch of other things :O

@tgodzik
Copy link
Contributor

tgodzik commented Jun 6, 2021

My current code is at https://github.com/scalameta/scalafmt/compare/master...tgodzik:add-scala-native?expand=1

There are still some other minor stuff to work out, but I am completely stumped by this one.

@ekrich
Copy link

ekrich commented Jun 6, 2021

I’ll have to look at that later today. I could have missed something there or something changed in Scala Native.

@ekrich
Copy link

ekrich commented Jun 7, 2021

Scala Native still has an empty URL class. Do you mind trying nativeLinkStubs := true in the build. The URI.toURL method is a stub in Scala Native.

  @scalanative.annotation.stub
  def toURL(): java.net.URL = ???

When working with Olafur previously, the code did not use the ConfigFactory.parseFile code path. Unfortunately, I was concentrating on having a smooth transition to Scala 3 and refactoring the tests for more code coverage was further down on the list for sconfig. The tested path was how the scalafmt code worked before for Scala Native and is shown here: https://github.com/ekrich/sconfig/blob/main/docs/SCALA-NATIVE.md We may be hitting an unknown code path or something else is going on.

After looking further at this it looks like PlatformUri in sconfig should not be shared with the JVM. We are just kicking the link stubs one layer deeper to more link stubs.

Wow, you had to bring the difflib across from munit too.

@tgodzik
Copy link
Contributor

tgodzik commented Jun 7, 2021

Scala Native still has an empty URL class. Do you mind trying nativeLinkStubs := true in the build. The URI.toURL method is a stub in Scala Native.

That helped. I am down to two issuses:

[error] * M26java.text.SimpleDateFormatRL16java.lang.StringE
...
[error] * T26java.text.SimpleDateFormat

Is SimpleDateFormat implemented in sjavatime or do we need to add it there?

[error] * M38java.util.concurrent.ConcurrentHashMapRE
...
[error] * M40java.util.concurrent.LinkedBlockingDequeRE
...

The concurrent collections is something that I can probably replace.

[error] * M18scopt.OptionParserRL16java.lang.StringE
...
[error] * T15java.io.Console

No idea about console and scoopt yet.

Wow, you had to bring the difflib across from munit too.

I might try to release it as a separate library.

@tgodzik
Copy link
Contributor

tgodzik commented Jun 7, 2021

Ok, got it all compiling at least for now, but getting an exception:

scala.NotImplementedError: an implementation is missing
	at java.lang.Throwable.fillInStackTrace(Unknown Source)
	at scala.Predef$.???(Unknown Source)
	at java.net.URI.toURL(Unknown Source)
	at org.ekrich.config.impl.PlatformUri.toURL(Unknown Source)
	at org.ekrich.config.impl.SimpleConfigOrigin$.newFile(Unknown Source)
	at org.ekrich.config.impl.Parseable$ParseableFile.createOrigin(Unknown Source)
	at org.ekrich.config.impl.Parseable.postConstruct(Unknown Source)
	at org.ekrich.config.impl.Parseable$.newFile(Unknown Source)
	at org.ekrich.config.ConfigFactory$.parseFile(Unknown Source)
	at org.ekrich.config.ConfigFactory$.parseFile(Unknown Source)
	at metaconfig.sconfig.SConfig2Class$.$anonfun$gimmeConfFromFile$1(Unknown Source)
	at metaconfig.sconfig.SConfig2Class$$$Lambda$2.apply(Unknown Source)
	at metaconfig.sconfig.SConfig2Class$.gimmeSafeConf(Unknown Source)
	at metaconfig.sconfig.SConfig2Class$.gimmeConfFromFile(Unknown Source)
	at metaconfig.sconfig.package$.metaconfig$sconfig$package$$$anonfun$sConfigMetaconfigParser$1(Unknown Source)
	at metaconfig.sconfig.package$$anonfun$1.fromInput(Unknown Source)
	at metaconfig.Input$InputImplicits.parse(Unknown Source)
	at metaconfig.Input$InputImplicits.$anonfun$parse$1(Unknown Source)
	at metaconfig.Input$InputImplicits$$Lambda$1.apply(Unknown Source)
	at scala.Option.fold(Unknown Source)
	at metaconfig.Input$InputImplicits.parse(Unknown Source)
	at org.scalafmt.config.Config$.$anonfun$hoconFileToConf$2(Unknown Source)
	at org.scalafmt.config.Config$$$Lambda$2.apply(Unknown Source)
	at metaconfig.Configured.andThen(Unknown Source)
	at org.scalafmt.config.Config$.hoconFileToConf(Unknown Source)
	at org.scalafmt.cli.CliOptions.$anonfun$hoconOpt$3(Unknown Source)
	at org.scalafmt.cli.CliOptions$$Lambda$8.apply(Unknown Source)
	at scala.Option.map(Unknown Source)
	at org.scalafmt.cli.CliOptions.hoconOpt$lzycompute(Unknown Source)
	at org.scalafmt.cli.CliOptions.getHoconValueOpt(Unknown Source)
	at org.scalafmt.cli.CliOptions.getVersionIfDifferent(Unknown Source)
	at org.scalafmt.cli.Cli.findRunner(Unknown Source)
	at org.scalafmt.cli.Cli.run(Unknown Source)
	at org.scalafmt.cli.Cli.mainWithOptions(Unknown Source)
	at org.scalafmt.cli.Cli$.main(Unknown Source)
	at <none>.main(Unknown Source)
	at <none>.__libc_start_main(Unknown Source)
	at <none>._start(Unknown Source)

@ekrich I think that's something we should fix in sconfig?

@ekrich
Copy link

ekrich commented Jun 7, 2021

@tgodzik if you could only call ConfigFactory.parseString, I think it would work. parseFile can read URLs from the file and URL is a mine field in Scala Native and Scala.js - no support basically. Please do something like this if possible.

import java.nio.file.{Files, Paths}
val bytes = Files.readAllBytes(Paths.get("src/main/resources/myapp.conf"))
val configStr = new String(bytes)
val conf = ConfigFactory.parseString(configStr)

We don't have java.text support so maybe java.util.Formatter?

I put together sjavatime as a stop gap until https://github.com/cquiroz/scala-java-time could be updated but nobody has managed to get this library updated yet.

Hope this helps.

@tgodzik
Copy link
Contributor

tgodzik commented Jun 7, 2021

I will do with toString for now, it's not that bad. Turns out though there is a bunch of other issues:

java.lang.UnsupportedOperationException
	at java.lang.Throwable.fillInStackTrace(Unknown Source)
	at java.nio.file.PathMatcherImpl$.apply(Unknown Source)
	at scala.scalanative.nio.fs.UnixFileSystem.getPathMatcher(Unknown Source)
	at org.scalafmt.config.ProjectFiles$FileMatcher$.$anonfun$nio$1(Unknown Source)
	at org.scalafmt.config.ProjectFiles$FileMatcher$$$Lambda$2.apply(Unknown Source)
	at scala.collection.immutable.List.map(Unknown Source)
	at scala.collection.immutable.List.map(Unknown Source)
	at org.scalafmt.config.ProjectFiles$FileMatcher$.create(Unknown Source)
	at org.scalafmt.config.ProjectFiles$FileMatcher$.nio(Unknown Source)
	at org.scalafmt.config.ProjectFiles$FileMatcher$.apply(Unknown Source)
	at org.scalafmt.cli.ScalafmtCoreRunner$.$anonfun$run$2(Unknown Source)
	at org.scalafmt.cli.ScalafmtCoreRunner$$$Lambda$2.apply(Unknown Source)
	at metaconfig.Configured$ConfiguredImplicit.fold(Unknown Source)
	at org.scalafmt.cli.ScalafmtCoreRunner$.run(Unknown Source)
	at org.scalafmt.cli.Cli.runWithRunner(Unknown Source)
	at org.scalafmt.cli.Cli.run(Unknown Source)
	at org.scalafmt.cli.Cli.mainWithOptions(Unknown Source)
	at org.scalafmt.cli.Cli$.main(Unknown Source)
	at <none>.main(Unknown Source)
	at <none>.__libc_start_main(Unknown Source)
	at <none>._start(Unknown Source)

Scalafmt uses getPathMatcher which seems to be not implemented and that is used in a bunch of places. I also found some failing regexes, which I asked Wojciech to investigate.

I also found another issue with running git:

        sys.process
          .Process(cmd, workingDirectory.jfile)
          .!!(swallowStderr)
          .trim

Seems that it's not possible currently to run a process?

There is one last issue that deals with Threads, but I guess I can work around it for Scala Native.

@ekrich
Copy link

ekrich commented Jun 7, 2021

We do support java.lang.Process and java.lang.ProcessBuilder. I looked and it seems that scala.sys.process.ProcessImpl requires threads.

@tgodzik
Copy link
Contributor

tgodzik commented Jun 7, 2021

Ach ok, that fixed it. The remaining issues are:

  • scala.scalanative.nio.fs.UnixFileSystem.getPathMatcher maybe there is a way to work around it
  • failing regexes - I don't think there is a way around those currently
  • non-thread implementation of UpdateDisplayThread

@ekrich
Copy link

ekrich commented Jun 7, 2021

What is the issue with scala.scalanative.nio.fs.UnixFileSystem.getPathMatcher?

cc/ Our regex expert @LeeTibbert

@tgodzik
Copy link
Contributor

tgodzik commented Jun 7, 2021

What is the issue with scala.scalanative.nio.fs.UnixFileSystem.getPathMatcher?

cc/ Our regex expert @LeeTibbert

It's not currently implemented for Scala Native, we could try turning the glob patterns into regexes aka https://stackoverflow.com/questions/1247772/is-there-an-equivalent-of-java-util-regex-for-glob-type-patterns
but that is a bit hacky.

Anyway, I am trying out that solution (seems to do the trick for most cases), but another thing popped up after implementing that:

org.scalafmt.cli.FailedToFormat: /home/tgodzik/Documents/metals/tests/unit/src/test/scala/tests/worksheets/WorksheetNoDecorationsLspSuite.scala
        <no stack trace available>
Caused by: java.lang.NullPointerException
        at java.lang.Throwable.fillInStackTrace(Unknown Source)
        at scala.scalanative.runtime.package$.throwNullPointer(Unknown Source)
        at <none>.(Unknown Source)
        at org.scalafmt.internal.State$.getColumnsWithStripMargin(Unknown Source)
        at org.scalafmt.internal.State$.getColumns(Unknown Source)
        at org.scalafmt.internal.State.next(Unknown Source)

I pushed the current state to the branch if anyone wants to take a look.

Edit:

That seems to be an issue with the regexes also (run a file separately):

Caused by: java.util.regex.PatternSyntaxException: Illegal/unsupported escape sequence near index 3

(\h*+\|)?([^
]*+)
   ^

@tgodzik
Copy link
Contributor

tgodzik commented Jun 8, 2021

Ok, we worked around the Regexes thanks to @sjrd - we might need to fix some issues with the workaround still - but everything finally compiles and runs!

I've put a draft PR here: https://github.com/tgodzik/scalafmt/pull/13/files

I still need to:

  • Make TermDisplay work for Native
  • Add tests and fix everything
  • Tidy up

This might still be a bit of work. Next step would be the publishing story, I think we could try the same as in Bloop, publish artifact to github and link to them from coursier, which would allow us to install binaries using cs

@jchyb
Copy link
Contributor

jchyb commented Nov 3, 2021

Some updates. I took over from @tgodzik to finish the PR and I mostly did, it's pretty much done (jchyb#1). Testing on a my small unrelated repo yielded very promising results. However. I decided to do some real benchmarks on the scala-native repo itself, and... there are some problems. Some of the bigger files take extremely long to process (from about 1s sys time on JVM to 10s on Native). I have reasons to believe it's related to scala-native/scala-native#1713, but I need to investigate further. I don't think it makes much sense to merge before fixing the underlying issue.

EDIT: I just realized I should check how it works with another GC to confirm the issue.
EDIT 2: Neither commix GC or boehm GC fixed anything so it might be another issue after all.

@ekrich
Copy link

ekrich commented Jan 13, 2022

@jchyb I know @densh took a look at the performance issues recently and posted on the Scala Native discord.

I am wondering too if it might be a good idea to introduce changes made in the Scala Native PR branch here that can be merged. It seems reasonable to have the code support Scala Native even if the usability is sub par. I feel that runtime or other issues are somewhat tangential to this issue. It might make it easier for others to get involved as well.

Currently, 3 of the 5 check boxes are completed.

@jchyb
Copy link
Contributor

jchyb commented Jan 14, 2022

@ekrich I have been meaning to get back to this. For now I would like to properly implement glob support on the SN side. Should not be too difficult and the regex wrapping system used before was not great and on further consideration I would not want to introduce too many SN-specific considerations for the scalafmt team. Additionally @kitbellew review (for which I am very grateful) revealed some issues with f.e. the lack of test cases there.

In terms of performance, I will also try testing scalafmt with the newest SN version and report my findings today in the issue. The recent changes to Strings seem substantial.

As for the check boxes they look slightly outdated, I was under the impression that sconfig was already used on the metaconfig side (but I don't remember exactly at this point)

@ekrich
Copy link

ekrich commented Jan 14, 2022

@jchyb Sounds like a plan - sconfig was merged for the Scala Native cross project as far as I know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants