Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stackoverflow in typechecker.Typers$Typer in scala 2.12.3/2.12.4 #10604

Closed
fanf opened this issue Nov 12, 2017 · 25 comments
Closed

Stackoverflow in typechecker.Typers$Typer in scala 2.12.3/2.12.4 #10604

fanf opened this issue Nov 12, 2017 · 25 comments
Labels

Comments

@fanf
Copy link

fanf commented Nov 12, 2017

I'm observing a stackoverflow in typechecker with ScalaIDE 4.7.0 when using scalac 2.12.3 or 2.12.4, which is not present with scala 2.12.2. The stackoverflow does not happen when I'm compiling from the command line using maven.
I saw a lot of variation of the problem (call site initialization exception and java.lang.UnsupportedOperationException: Position.point on NoPosition) but these other problem seems to be consequences of the stackoverflow.

Of course I tried to set a bigger Xss, but it does not change anything (even with 256m), and the fact that the maven compilation works or that it works on 2.12.2 with normal Xss tends to let me think that it is a genuine bug.

I don't have at all a minimal reproducer, and the stacktraces are not very telling, but the problem happen with an open source project: https://github.com/Normation/rudder/ (on branch master).

It may or not may be related to Daniel "infinite macro expansion in scala 2.12.3/2.12.4" #10584 or perhaps both have same root cause (rudder-core uses doobie and so macro expansion).

Nonetheless, here are the SOE and other exception I saw:

2017-10-26 18:33:47,710 ERROR [main] - org.scala-ide.sdt.core - org.scala-ide.sdt.core - org.scala-ide.sdt.core - 0 - Error in Scala compiler
java.lang.StackOverflowError
    at scala.tools.nsc.typechecker.Contexts$Context.make(Contexts.scala:463)
    at scala.tools.nsc.typechecker.Contexts$Context.makeNewScope(Contexts.scala:516)
    at scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:5511)
    at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5546)
    at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5553)
    at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5589)
    at scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5619)
    at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5563)
    at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5567)
    at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5670)
    at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5676)
    at scala.tools.nsc.typechecker.Typers$Typer.typedSelectOrSuperCall$1(Typers.scala:4999)
    at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5537)
    at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5553)
    at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5589)
    at scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5619)
    at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5563)
    at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5567)
    at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$38(Typers.scala:4697)
    at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:699)
    at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4699)
    at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4746)
    at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5536)
    at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5553)
    at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5589)
    at scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5619)
    at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5563)
   .... and on and on...
@fanf
Copy link
Author

fanf commented Nov 23, 2017

It may, perhaps, not sure anymore at anything, that the problem might be link with the use of the "-verbose" option. Removing that flag seems to make it disapear (but as I don't have a clean way to trigger it in the first place, I'm not sure of anything).

@tpolecat
Copy link

Yeah we're seeing java.lang.UnsupportedOperationException: Position.point on NoPosition.

@olafurpg
Copy link
Member

olafurpg commented Dec 7, 2017

Upgrading from 2.12.3 to 2.12.4 introduced a StackOverflowError in inductive shapeless derivation in the scalafix repo scalacenter/scalafix#477 (comment) Increasing -Xss8m fixed the problem. This regression took a while to track down since the SO was caught by the compiler during implicit expansion and only reported under -Xlog-implicits. I'm not sure if it's directly related to this PR, but wanted to leave a note somewhere. At least, I'm not the only one hitting on SO while upgrading in 2.12.x

@olafurpg
Copy link
Member

olafurpg commented Dec 7, 2017

I took the liberty to open another issues about the fact that the SO was getting swallowed and not reported without -Xlog-implicits #10649

@fanf
Copy link
Author

fanf commented Dec 14, 2017

It may be related to #10552

@fanf
Copy link
Author

fanf commented Jan 22, 2018

I've a variant of that one on IntelliJ (I believe I also got it from time to time on scala IDE) with a NPE:

Error:scala: ## Exception when compiling 55 sources to /home/fanf/java/workspaces/rudder-project/rudder/rudder-core/target/test-classes
null
scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2486)
scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$96(Typers.scala:5505)
scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typerWithLocalContext$1(Typers.scala:486)
scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:486)
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5540)
scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5547)
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5584)
scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5616)
scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5557)
scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5562)
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5667)
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5673)
scala.tools.nsc.typechecker.Typers$Typer.typedSelectOrSuperCall$1(Typers.scala:5008)
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5531)
scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5547)
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5584)
scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5616)
scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5557)
scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5562)
scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$38(Typers.scala:4708)
scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:698)
scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4710)
scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4757)
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5530)
scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5547)
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5584)
scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5616)
scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5557)
scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5562)
scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2457)
scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$96(Typers.scala:5505)
scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typerWithLocalContext$1(Typers.scala:486)
scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:486)
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5540)
scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5547)
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5584)
scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5616)

...... and again and again and again

@thestuckster
Copy link

thestuckster commented Mar 8, 2018

Is this issue being actively worked on by anyone? We suddenly started getting the same problem on my project at work. We bumped the stack size which has fixed the issue but, that seems more like a band-aid rather than an actual fix.

We are seeing this issue when compiling with Maven on the command line instead of just the IDE.

@SethTisue SethTisue added the typer label Mar 8, 2018
@joan38
Copy link

joan38 commented Mar 21, 2018

Same here with 2.12.4

@normana400
Copy link

bump

@normana400
Copy link

fwiw, Im having this issue w/ shapeless / slickless in Scala 2.11.11

@SethTisue
Copy link
Member

SethTisue commented Mar 27, 2018

some of y'all know this already, but maybe not everyone does: it's considered normal in Scala that you may have to increase JVM stack size in order to compile some kinds of especially involved or nested Scala code.

does anyone know a way to reproduce the problem they're having on Scala 2.12.5 (yes, I mean 2.12.5 specifically, now that it's out), with a minimal example and without needing access to your project's entire source tree? without that, it's really hard to know whether this ticket even identifies a specific issue, or whether it's just a miscellaneous collection of people reporting a miscellaneous collection of StackOverflowErrors in miscellaneous versions of Scala, which could have miscellaneous causes.

(I'm not criticizing anyone who's reported or commented here, but in order to actually do anything about this, we need to converge on something more specific and actionable.)

note that 2.12.5 fixes #10552, so trying again in 2.12.5 may produce different error logs that may continue additional clues that we didn't have before.

@rafalmag
Copy link

I had the same issue and I can confirm that increasing of the stack size from default one (1MB to 5MB) helped.

@SethTisue
Copy link
Member

closing since there isn't currently anything actionable here, unless new information comes to light.

@mycaule
Copy link

mycaule commented Jul 12, 2018

Hello,

We had the same kind of problem today at work.
We wanted to write a very big case class but encountered these compilation errors.
The errors were fixed with the magical -Xss2048K flag.

case class VeryBigRecord(
    x001: Int,
    x002: Int,
    x003: Int,
    x004: Int,
    x005: Int,
    x006: Int,
    ...
    x178: Int
)

I have reproduced the bug in a minimal reproducible way.
https://github.com/mycaule/scalabug10604-testcase

It might be the same stuff as the 22 fields case class limit earlier.

@SethTisue
Copy link
Member

@mycaule unrelated to the 22 limit. it's:

it's considered normal in Scala that you may have to increase JVM stack size in order to compile some kinds of especially involved or nested Scala code.

@er1c
Copy link

er1c commented Oct 31, 2018

I'm getting this same error with 2.12.7 and I bumped the memory from 1GB to 4GB to no effect:

$ sbt -v                                                                                                                                                                                                                   ✔  8106  07:48:28
[process_args] java_version = '11'
[copyRt] java9_rt = '/Users/eric/.sbt/0.13/java9-rt-ext-oracle_corporation_11_0_1/rt.jar'
# Executing command line:
java
-Xms16384m
-Xmx16384m
-XX:ReservedCodeCacheSize=512m
-XX:MaxMetaspaceSize=1024m
-XX:+UseParallelGC
-Dscala.ext.dirs=/Users/eric/.sbt/0.13/java9-rt-ext-oracle_corporation_11_0_1
-jar
/usr/local/Cellar/sbt/1.2.6/libexec/bin/sbt-launch.jar
[error] ## Exception when compiling 22 sources to /Users/eric/Work/ta-webservice-api/ws-generator/target/scala-2.12/classes
[error] null
[error] scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:670)
[error] scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:789)
[error] scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5617)
[error] scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$38(Typers.scala:4746)
[error] scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:693)
[error] scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4748)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4776)
[error] scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5571)
[error] scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:789)
[error] scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5617)
[error] scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:714)
[error] scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:789)
[error] scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5617)
[error] scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$38(Typers.scala:4746)
[error] scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:693)

Java 11 issue?

$ java -version
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)

I also have the same issue in scala 2.12.2, but it has a much more interesting stack trace:

java.lang.AssertionError: assertion failed: class Any: class Definitions
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parseMethod(ClassfileParser.scala:570)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parseClass$4(ClassfileParser.scala:470)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.queueLoad$1(ClassfileParser.scala:470)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parseClass$5(ClassfileParser.scala:480)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:485)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$1(ClassfileParser.scala:156)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:127)
	at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:316)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:218)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1521)
	at scala.tools.nsc.backend.jvm.BCodeHelpers.completeSilentlyAndCheckErroneous(BCodeHelpers.scala:235)
	at scala.tools.nsc.backend.jvm.BTypesFromSymbols.$anonfun$classBTypeFromSymbol$5(BTypesFromSymbols.scala:122)
	at scala.collection.MapLike.getOrElse(MapLike.scala:128)
	at scala.collection.MapLike.getOrElse$(MapLike.scala:126)
	at scala.collection.concurrent.TrieMap.getOrElse(TrieMap.scala:631)
	at scala.tools.nsc.backend.jvm.BTypesFromSymbols.classBTypeFromSymbol(BTypesFromSymbols.scala:118)
	at scala.tools.nsc.backend.jvm.BTypesFromSymbols.$anonfun$setClassInfo$17(BTypesFromSymbols.scala:440)
	at scala.tools.nsc.backend.jvm.BTypesFromSymbols.setClassInfo(BTypesFromSymbols.scala:440)
	at scala.tools.nsc.backend.jvm.BTypesFromSymbols.$anonfun$classBTypeFromSymbol$5(BTypesFromSymbols.scala:126)
	at scala.collection.MapLike.getOrElse(MapLike.scala:128)
	at scala.collection.MapLike.getOrElse$(MapLike.scala:126)
	at scala.collection.concurrent.TrieMap.getOrElse(TrieMap.scala:631)
	at scala.tools.nsc.backend.jvm.BTypesFromSymbols.classBTypeFromSymbol(BTypesFromSymbols.scala

Full log: https://gist.github.com/er1c/4650e34b02c92a1a4c00bdaf1889f598

It works successfully on 2.12.3

@Jasper-M
Copy link
Member

@er1c You should increase the stack size with the -Xss flag. e.g. -Xss5m.

@er1c
Copy link

er1c commented Oct 31, 2018

@Jasper-M yeah I was basically playing with multiple options if you see above, sbt is running with 16GB, are you simply talking about javaOptions in (Compile, run) ++= Seq("-Xms4G", "-Xmx4G"), sbt launcher memory via /usr/local/etc/sbtopts, or .jvmopts? What's interesting is it compiles in 2.12.3 but not 2.12.7 or 2.12.2

@er1c
Copy link

er1c commented Oct 31, 2018

Ahh, I missed/overlooked the -Xms vs -Xss, and that isn't part of the /usr/local/etc/sbtopts mem options.

.jvmopts file with:

-Xss4m

Does work fine with 2.12.7

@lrytz
Copy link
Member

lrytz commented Oct 31, 2018

👍 also note that the scala compiler runs in the same JVM as sbt, so you need to start sbt with enough Xss

@er1c
Copy link

er1c commented Oct 31, 2018

@lrytz the .jvmopts file gets picked up by the sbt launcher:

$ sbt -v
[process_args] java_version = '11'
[copyRt] java9_rt = '/Users/eric/.sbt/0.13/java9-rt-ext-oracle_corporation_11_0_1/rt.jar'
# Executing command line:
java
-Xms1024m
-Xmx1024m
-XX:ReservedCodeCacheSize=128m
-XX:MaxMetaspaceSize=256m
-XX:+UseParallelGC
-Xss4m
-Dscala.ext.dirs=/Users/eric/.sbt/0.13/java9-rt-ext-oracle_corporation_11_0_1
-jar
/usr/local/Cellar/sbt/1.2.6/libexec/bin/sbt-launch.jar

@Mr-HHP
Copy link

Mr-HHP commented Dec 3, 2021

你好,

我们今天在工作中遇到了同样的问题。 我们想写一个非常大的case class但是遇到了这些编译错误。 错误已通过魔法-Xss2048K标志修复。

case class VeryBigRecord(
    x001: Int,
    x002: Int,
    x003: Int,
    x004: Int,
    x005: Int,
    x006: Int,
    ...
    x178: Int
)

我以最小的可重现方式重现了该错误。 https://github.com/mycaule/scalabug10604-testcase

它可能与之前的 22 个字段案例类限制相同。

Hello, I have the same problem as you. I created a very large case class and encountered these errors while compiling. I solved the problem by increasing stack memory (-Xss2048m), but I don't understand the cause, do you know the cause?

@er1c
Copy link

er1c commented Dec 3, 2021

@Mr-HHP it's the "-Xss5m" that will do the trick

@Mr-HHP
Copy link

Mr-HHP commented Dec 3, 2021

@Mr-HHP是“-Xss5m”可以解决问题

This solves the problem, but I'm not sure why. What does "Scala" do with compiling "case Class"

@som-snytt
Copy link

som-snytt commented Dec 3, 2021

@Mr-HHP I don't know in particular, but I was just looking at how it typechecks a function application, and a case class's synthetic unapply will involve two apply similar to

Some.apply[(Int, Int)](scala.Tuple2.apply[Int, Int](x$0.x, x$0.y))

Each apply requires checking first the Some.apply itself (typedQualifier in the stack trace) and then the application of the args. Possibly it consumes stack inferring those type args, if they are not specified in the synthetic code. The other stack trace is in erasure, which I don't know about.

Edit: see #12397

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests