Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'ng scalafmt --stdin' throws exception 'java.io.IOException: Unknown stream type: H' #1115

Closed
hacker-works opened this issue Feb 27, 2018 · 19 comments

Comments

@hacker-works
Copy link

hacker-works commented Feb 27, 2018

  • Version: scalafmt 1.4.0, vim 8, nailgun 0.9.3
  • Integration: Vim 8, Chiel92/vim-autoformat, nailgun 0.9.3
  • Configuration:

I'm using scalafmt through nailgun:

" Enable vim-autoformat for Scala
function! StartNailgunScalafmt()
    silent execute '!scalafmt_ng 2>/dev/null 1>/dev/null &'
    sleep 1
    silent execute '!ng ng-alias scalafmt org.scalafmt.cli.Cli'
    execute(':redraw!')
endfunction

call StartNailgunScalafmt()

noremap <F5> :Autoformat<CR>
" let g:autoformat_verbosemode=1
let g:formatdef_scalafmt = "'ng scalafmt --stdin'"
let g:formatters_scala = ['scalafmt']
let g:formatters_sbt = ['scalafmt']

Steps

Given code like this:

import $ivy.{
  `org.apache.spark::spark-core:2.2.1`,
  `org.apache.spark::spark-sql:2.2.1`
}
import ammonite.ops._
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.sql.SQLContext

object TestSpark {
  def test(): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("spike")
    val sc = new SparkContext(conf)

    sc.setLogLevel("WARN")

    sc.textfile("/Users/vladi/Work/pioneers-core/docs/README.md")
    val matchingLineAndLineNumberTuples = textFile.zipWithIndex().filter({ case (line, lineNumber) => line.toLowerCase.contains("scala") }).collect
    matchingLineAndLineNumberTuples.sortBy(_._2).foreach {
      case (l, i) => println(s"LINE ${i}:\t${l}")
    }
  }
}

When I run scalafmt like this:

cat spark.scala | scalafmt --stdin

The output on stdin is just as expected:

import $ivy.{
  `org.apache.spark::spark-core:2.2.1`,
  `org.apache.spark::spark-sql:2.2.1`
}
import ammonite.ops._
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext

object TestSpark {
  def test(): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("spike")
    val sc = new SparkContext(conf)

    sc.setLogLevel("WARN")

    sc.textfile("/Users/vladi/Work/pioneers-core/docs/README.md")
    val matchingLineAndLineNumberTuples = textFile
      .zipWithIndex()
      .filter({ case (line, lineNumber) => line.toLowerCase.contains("scala") })
      .collect
    matchingLineAndLineNumberTuples.sortBy(_._2).foreach {
      case (l, i) => println(s"LINE ${i}:\t${l}")
    }
  }
}

But when I run scalafmt with nailgun:

scalafmt_ng 2>/dev/null 1>/dev/null &
[1] 58892

ng ng-alias scalafmt org.scalafmt.cli.Cli
[1]+  Done                    scalafmt_ng 2> /dev/null > /dev/null

cat spark.scala | ng scalafmt --stdin

Problem

I'm getting an exception that I didn't got before (my setup used to work just fine a week ago):

java.io.IOException: Unknown stream type: H
	at com.martiansoftware.nailgun.NGInputStream.readHeader(NGInputStream.java:73)
	at com.martiansoftware.nailgun.NGInputStream.read(NGInputStream.java:120)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:70)
	at scala.collection.Iterator.foreach(Iterator.scala:929)
	at scala.collection.Iterator.foreach$(Iterator.scala:929)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
	at scala.collection.TraversableOnce.addString(TraversableOnce.scala:357)
	at scala.collection.TraversableOnce.addString$(TraversableOnce.scala:353)
	at scala.collection.AbstractIterator.addString(Iterator.scala:1417)
	at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:323)
	at scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:322)
	at scala.collection.AbstractIterator.mkString(Iterator.scala:1417)
	at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:325)
	at scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:325)
	at scala.collection.AbstractIterator.mkString(Iterator.scala:1417)
	at org.scalafmt.cli.InputMethod$StdinCode$.apply(InputMethod.scala:26)
	at org.scalafmt.cli.Cli$.getInputMethods(Cli.scala:106)
	at org.scalafmt.cli.Cli$.run(Cli.scala:156)
	at org.scalafmt.cli.Cli$.mainWithOptions(Cli.scala:66)
	at org.scalafmt.cli.Cli$.nailMain(Cli.scala:32)
	at org.scalafmt.cli.Cli.nailMain(Cli.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.martiansoftware.nailgun.NGSession.run(NGSession.java:280)

Expectation

My expectation is that instead of exception it'll output the formatted file.

Workaround

On the other hand ng scalafmt spark.scala formats the file as expected.

Notes

I have looked at the nailgun code indicated in the exception. This code is found only in nailgun 0.9.1. Mac OS X brew is installing only nailgun 0.9.3

Currently, Scalafmt is using nailgun 0.9.1 as a build dependency:

"com.martiansoftware" % "nailgun-server" % "0.9.1",

@olafurpg
Copy link
Member

What happens if you include --no-stderr? I have not used the nailgun integration myself but there was a related discussion in the vim-autoformat issue tracker iirc

@hacker-works
Copy link
Author

cat spark.scala | ng scalafmt --stdin --no-stderr
java.io.IOException: Unknown stream type: H
	at com.martiansoftware.nailgun.NGInputStream.readHeader(NGInputStream.java:73)
	at com.martiansoftware.nailgun.NGInputStream.read(NGInputStream.java:120)
...

Throws the same exception.

@olafurpg
Copy link
Member

Anything in vim-autoformat/vim-autoformat#184 that could help?

@hacker-works
Copy link
Author

@olafurpg: I have basically followed the instructions there... and it used to work just fine. But a week ago I have updated the Mac OS X with the recommended updates and in addition I ran brew upgrade

@olafurpg
Copy link
Member

olafurpg commented Feb 27, 2018 via email

@hacker-works
Copy link
Author

Maven repos provide only nailgun 0.9.1. So to build with 0.9.3 we need to build it locally and install it in the local Maven repository. :/

@hacker-works
Copy link
Author

OK, so what I did was:

git clone [email protected]:facebook/nailgun.git
cd nailgun
mvn clean install

Now I have 0.9.3-SNAPSHOT in my local maven repository.

I've changed in the build.sbt to be able to use dependencies from maven local:

resolvers ++= Seq(Resolver.mavenLocal, Resolver.sonatypeRepo("releases"))

sbt compile passes without errors.

@olafurpg: So, how should I test the build?

@hacker-works
Copy link
Author

hacker-works commented Feb 27, 2018

Summary of what I've tried so far:

My changes to build.sbt:

line 13: resolvers ++= Seq(Resolver.mavenLocal, Resolver.sonatypeRepo("releases")),

line 61: "com.martiansoftware" % "nailgun-server" % "0.9.3-SNAPSHOT",

I have built and installed locally scalafmt-cli by running sbt cli/publishLocal

Then I have reinstalled scalafmt_ng with:

sudo coursier bootstrap --standalone com.geirsson:scalafmt-cli_2.12:1.4.0 \
    -r ivy2local \
    -o /usr/local/bin/scalafmt_ng -f --main com.martiansoftware.nailgun.NGServer

And I've restarted nailgun:

scalafmt_ng 2>/dev/null 1>/dev/null &
[1] 70424

ng ng-alias scalafmt org.scalafmt.cli.Cli

And then tested it again:

cat spark.scala | ng scalafmt --stdin
java.io.IOException: Unknown stream type: H
	at com.martiansoftware.nailgun.NGInputStream.readHeader(NGInputStream.java:73)
	at com.martiansoftware.nailgun.NGInputStream.read(NGInputStream.java:120)
        ...

So, seems changing the nailgun version to 0.9.3 is not fixing the issue.

NOTE: What is strange to me is that readHeader is no longer in the source of NGInputStream.java in nailgun 0.9.3 and still it is in the stack trace...

@olafurpg
Copy link
Member

What does ng --nailgun-version print for you? It's 0.9.0 for me. I would be surprised if there was a breaking change in the protocol between 0.9.1 and 0.9.3

@olafurpg
Copy link
Member

Just tried following installation nailgun installation for scalafmt and I am able to reproduce the problem. Not sure what to do. @pjrt do you still use scalafmt through vim?

@pjrt
Copy link
Collaborator

pjrt commented Feb 27, 2018

I do use it through vim, but not as part of the vim-autoformat plugin (was having some issues with it, but I never investigated it). Currently I just run it as :!scalafmt -i -f %.

I never tried to get nailgun running, though this issue makes me want to try it..

@hacker-works
Copy link
Author

ng --nailgun-version
NailGun client version 0.9.0

Same here!

@olafurpg
Copy link
Member

@jonmeredith
Copy link

FWIW, checking out tag nailgun-all-0.9.1 of nailgun, building and installing the ng client fixed it for me.

@olafurpg
Copy link
Member

Any actionable item that be done to close this ticket? I personally don't use the nailgun integration, but I am able to reproduce the bug and it would be great to have working nailgun.

OTOH, I am optimistic we will have Scala Native support in the coming few months #1172

@vladimir-tsvetkov
Copy link

@olafurpg: I don't think there is.
I still have to find the time to reproduce @jonmeredith's suggestion, but intuition tells me it'll work this way.

Thank you for the help, everyone!

@olafurpg
Copy link
Member

OK, I'll close this ticket then in favor of #1172. I am optimistic scala native support will avoid the need for nailgun altogether

mwhittaker added a commit to mwhittaker/nailgun that referenced this issue Jun 16, 2018
If you clone the nailgun repository, it builds just fine. However, if
you try to use this version of nailgun with scalafmt [1], it doesn't
actually work [2]. In [2], they suggest building the nailgun-all-0.9.1
tag. But, this tag doesn't build because of javadoc errors. This commit
fixes those errors so that the code builds again.

[1]: https://scalameta.org/scalafmt/#Nailgun
[2]: scalameta/scalafmt#1115
@ahakanbaba
Copy link

This is not a solution but another workaround.

Setting the following in vim-autoformat settings worked for me.

let g:formatdef_scalafmt = '"ng org.scalafmt.cli.Cli ".expand("%:p")." --stdout 2> /dev/null"'

Works without using the --stdin mode. Vim function called expand(..) returns the full path of the file and the --stdout prints the formatted code to the output.

Probably vim-autoformat pipes the file contents to the stdin of this call. It looks like that gets ignored without problems.

@olafurpg
Copy link
Member

For people using the nailgun integration, I recommend trying out the graalvm native image in #1266 Please report back how it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants