Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of AndTypes on the LHS of subtype comparisons #18235

Merged
merged 6 commits into from
Jul 19, 2023

Conversation

odersky
Copy link
Contributor

@odersky odersky commented Jul 18, 2023

Fixes #18226
Fixes #12077

Might fix some other reported issues with AndTypes as well.

Fixes scala#18226

Might fix some other reported issues with AndTypes as well.
@odersky odersky requested a review from smarter July 18, 2023 09:46
@odersky
Copy link
Contributor Author

odersky commented Jul 18, 2023

The PR currently causes zio to fail.

[error] -- [E007] Type Mismatch Error: /__w/dotty/dotty/community-build/community-projects/zio/test/shared/src/main/scala/zio/test/MutableRunnableSpec.scala:82:34 
[error] 82 |      )((spec, aspect) => spec @@ aspect)
[error]    |                                  ^^^^^^
[error]    |Found:    (aspect :
[error]    |  zio.test.TestAspect[R & zio.test.environment.TestEnvironment,
[error]    |    R & zio.test.environment.TestEnvironment, MutableRunnableSpec.this.Failure,
[error]    |    MutableRunnableSpec.this.Failure]
[error]    |)
[error]    |Required: zio.test.TestAspect[
[error]    |  zio.Has[zio.test.environment.TestClock.Service] &
[error]    |    zio.Has[zio.test.environment.TestRandom.Service²] &
[error]    |    zio.Has[zio.test.environment.TestSystem.Service³] & (
[error]    |    zio.Has[zio.test.environment.Live.Service⁴] &
[error]    |    zio.Has[zio.test.Sized.Service⁵] & (zio.Has[zio.system.System.Service⁶] &
[error]    |    zio.Has[zio.random.Random.Service⁷])) & (
[error]    |    zio.Has[zio.test.TestConfig.Service⁸] &
[error]    |    zio.Has[zio.test.environment.TestConsole.Service⁹] &
[error]    |    zio.Has[zio.clock.Clock.Service¹⁰] & (zio.Has[zio.console.Console.Service¹¹]
[error]    |     & zio.Has[zio.blocking.Blocking.Service¹²] & (R &
[error]    |    zio.Has[zio.test.Annotations.Service¹³]))),
[error]    |  zio.Has[zio.test.environment.TestClock.Service] &
[error]    |    zio.Has[zio.test.environment.TestRandom.Service²] &
[error]    |    zio.Has[zio.test.environment.TestSystem.Service³] & (
[error]    |    zio.Has[zio.test.environment.Live.Service⁴] &
[error]    |    zio.Has[zio.test.Sized.Service⁵] & (zio.Has[zio.system.System.Service⁶] &
[error]    |    zio.Has[zio.random.Random.Service⁷])) & (
[error]    |    zio.Has[zio.test.TestConfig.Service⁸] &
[error]    |    zio.Has[zio.test.environment.TestConsole.Service⁹] &
[error]    |    zio.Has[zio.clock.Clock.Service¹⁰] & (zio.Has[zio.console.Console.Service¹¹]
[error]    |     & zio.Has[zio.blocking.Blocking.Service¹²] & (R &
[error]    |    zio.Has[zio.test.Annotations.Service¹³]))),
[error]    |Any, Nothing]
[error]    |
[error]    |where:    R         is a type in class MutableRunnableSpec with bounds <: zio.Has[?]
[error]    |          Service   is a trait in object TestClock
[error]    |          Service²  is a trait in object TestRandom
[error]    |          Service³  is a trait in object TestSystem
[error]    |          Service¹² is a trait in object Blocking
[error]    |          Service¹³ is a trait in object Annotations
[error]    |          Service¹¹ is a trait in object Console
[error]    |          Service¹⁰ is a trait in object Clock
[error]    |          Service⁴  is a trait in object Live
[error]    |          Service⁵  is a trait in object Sized
[error]    |          Service⁶  is a trait in object System
[error]    |          Service⁷  is a trait in object Random
[error]    |          Service⁸  is a trait in object TestConfig
[error]    |          Service⁹  is a trait in object TestConsole
[error]    |
[error]    | longer explanation available when compiling with `-explain`
[error] one error found
[error] (testJVM / Compile / compileIncremental) Compilation failed
[error] Total time: 63 s (01:03), completed Jul 18, 2023 1:13:58 PM

This is a pretty horrible error message, and I will not have the time to debug this further.

I can't reproduce the problem either on my M1. When I try, it crashes before I get to the previous error message with

set current project to root (in build file:/Users/odersky/workspace/dotty/community-build/community-projects/scalatest/)
Generated /Users/odersky/workspace/dotty/community-build/community-projects/scalatest/dotty/scalactic/target/scala-3.3.2-RC1-bin-SNAPSHOT/src_managed/main/org/scalactic/Every.scala
Generated /Users/odersky/workspace/dotty/community-build/community-projects/scalatest/dotty/scalactic/target/scala-3.3.2-RC1-bin-SNAPSHOT/src_managed/main/org/scalactic/ColCompatHelper.scala
Generated /Users/odersky/workspace/dotty/community-build/community-projects/scalatest/dotty/scalactic/target/scala-3.3.2-RC1-bin-SNAPSHOT/src_managed/main/org/scalactic/ArrayHelper.scala
[error] java.util.ConcurrentModificationException
[error] 	at java.base/java.util.TreeMap.callMappingFunctionWithCheck(TreeMap.java:750)
[error] 	at java.base/java.util.TreeMap.computeIfAbsent(TreeMap.java:558)
[error] 	at aQute.bnd.osgi.Jar.putResource(Jar.java:259)
[error] 	at aQute.bnd.osgi.Jar$1.visitFile(Jar.java:186)
[error] 	at aQute.bnd.osgi.Jar$1.visitFile(Jar.java:167)
[error] 	at java.base/java.nio.file.Files.walkFileTree(Files.java:2811)
[error] 	at aQute.bnd.osgi.Jar.buildFromDirectory(Jar.java:166)
[error] 	at aQute.bnd.osgi.Jar.<init>(Jar.java:109)
[error] 	at aQute.bnd.osgi.Jar.<init>(Jar.java:138)
[error] 	at aQute.bnd.osgi.Analyzer.setClasspath(Analyzer.java:1465)
[error] 	at com.typesafe.sbt.osgi.Osgi$.bundleTask(Osgi.scala:57)
[error] 	at com.typesafe.sbt.osgi.SbtOsgi$.$anonfun$defaultOsgiSettings$1(SbtOsgi.scala:55)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] 	at sbt.Execute.work(Execute.scala:291)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:833)
[error] (scalacticDotty / osgiBundle) java.util.ConcurrentModificationException
[error] Total time: 0 s, completed 18 Jul 2023, 15:24:08
[error] Test dotty.communitybuild.CommunityBuildTestA.zio failed: java.lang.RuntimeException: Publish command exited with code 1 for project scalatest. Project details:
[error] SbtCommunityProject(scalatest,scalacticDotty/clean; scalacticDottyJS/clean; set scalatestTestDotty / Test / managedSources ~= (_.filterNot(_.getName == "GeneratorSpec.scala").filterNot(_.getName == "FrameworkSuite.scala").filterNot(_.getName == "WaitersSpec.scala").filterNot(_.getName == "TestSortingReporterSpec.scala").filterNot(_.getName == "JavaFuturesSpec.scala").filterNot(_.getName == "ParallelTestExecutionSpec.scala").filterNot(_.getName == "TimeLimitsSpec.scala").filterNot(_.getName == "DispatchReporterSpec.scala").filterNot(_.getName == "TestThreadsStartingCounterSpec.scala").filterNot(_.getName == "SuiteSortingReporterSpec.scala").filterNot(_.getName == "CommonGeneratorsSpec.scala").filterNot(_.getName == "PropCheckerAssertingSpec.scala").filterNot(_.getName == "ConductorMethodsSuite.scala").filterNot(_.getName == "EventuallySpec.scala")); set scalacticTestDotty / Test / managedSources ~= (_.filterNot(_.getName == "NonEmptyArraySpec.scala")); set genRegularTests4 / Test / managedSources ~= (_.filterNot(_.getName == "FrameworkSuite.scala").filterNot(_.getName == "GeneratorSpec.scala").filterNot(_.getName == "CommonGeneratorsSpec.scala").filterNot(_.getName == "ParallelTestExecutionSpec.scala").filterNot(_.getName == "DispatchReporterSpec.scala").filterNot(_.getName == "TestThreadsStartingCounterSpec.scala").filterNot(_.getName == "EventuallySpec.scala")); scalacticTestDotty/test; scalatestTestDotty/test; scalacticDottyJS/compile; scalatestDottyJS/compile,List(),List(SbtCommunityProject(scala-xml,xml/test,List(),List(),dotty.communitybuild.SbtCommunityProject$$$Lambda$53/0x0000000800ccdfc0@7b0ce58,xml/publishLocal,xml/doc,List(-Xcheck-macros, -Ysafe-init),false)),dotty.communitybuild.projects$$$Lambda$57/0x0000000800cd9b78@57a7b450,scalacticDotty/publishLocal; scalatestDotty/publishLocal; scalacticDottyJS/publishLocal; scalatestDottyJS/publishLocal,;scalacticDotty/doc,List(-Xcheck-macros, -Ysafe-init),false), took 56.263 sec
[error]     at dotty.communitybuild.CommunityProject.publish(projects.scala:64)
[error]     at dotty.communitybuild.CommunityProject.publish$(projects.scala:35)
[error]     at dotty.communitybuild.SbtCommunityProject.publish(projects.scala:104)
[error]     at dotty.communitybuild.CommunityProject.publishDependencies$$anonfun$1(projects.scala:52)
[error]     at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
[error]     at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
[error]     at scala.collection.immutable.List.foreach(List.scala:333)
[error]     at dotty.communitybuild.CommunityProject.publishDependencies(projects.scala:52)
[error]     at dotty.communitybuild.CommunityProject.publishDependencies$(projects.scala:35)
[error]     at dotty.communitybuild.SbtCommunityProject.publishDependencies(projects.scala:104)
[error]     at dotty.communitybuild.CommunityProject.publish(projects.scala:58)
[error]     at dotty.communitybuild.CommunityProject.publish$(projects.scala:35)
[error]     at dotty.communitybuild.SbtCommunityProject.publish(projects.scala:104)
[error]     at dotty.communitybuild.CommunityBuildRunner$.run$$anonfun$1(CommunityBuildRunner.scala:23)
[error]     at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
[error]     at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
[error]     at scala.collection.immutable.List.foreach(List.scala:333)
[error]     at dotty.communitybuild.CommunityBuildRunner$.run(CommunityBuildRunner.scala:23)
[error]     at dotty.communitybuild.CommunityBuildTestA.zio(CommunityBuildTest.scala:33)
[error]     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[error]     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]     at java.lang.reflect.Method.invoke(Method.java:568)
[error]     ...

So, what do we do?

It looks like the change fixes a clear problem in subtyping and type inference, but zio relied in some way on the previous behavior.

I won't be able to sort this out further. If someone else could take it over this would be great!

@guizmaii
Copy link

@odersky I see some zio.Has in the logs. Is the community build still using ZIO1? Because ZIO1 is not maintained anymore. That's not really interesting for you to test Dotty on it. If it's still ZIO1, I'd advise you guys to migrate to ZIO2 🙂

@odersky
Copy link
Contributor Author

odersky commented Jul 18, 2023

Thanks for the info! It would be great to get a PR submitting zio2 to the community build!

@guizmaii
Copy link

I asked if anyone has some time to make this PR in the ZIO Discord. Let's see if someone picks up this task. If not, I can give it a try but not before this weekend, sorry.

@smarter
Copy link
Member

smarter commented Jul 18, 2023

Minimized:

class Has[A]
trait Foo

class TestAspect[+LowerR, -UpperR]

class Spec[-R] {
  def foo[R1 <: R](aspect: TestAspect[R1, R1]): Unit = {}
}

class SuiteBuilder[R <: Has[_]] {
  def toSpec(
    spec: Spec[R & Has[Foo]],
    aspect: TestAspect[
      R & Has[Foo],
      R & Has[Foo]
    ]
  ) =
    spec.foo(aspect)
}
-- [E007] Type Mismatch Error: try/MRS.scala:18:13 -----------------------------
18 |    spec.foo(aspect)
   |             ^^^^^^
   |         Found:    (aspect : TestAspect[R & Has[Foo], R & Has[Foo]])
   |         Required: TestAspect[R1, R1]
   |
   |         where:    R1 is a type variable with constraint <: R & Has[Foo]

@NavidJalali
Copy link

Thanks for the info! It would be great to get a PR submitting zio2 to the community build!

I'm gonna be honest, I just realized what the community build even is, but I might give it a shot, are these the steps? https://dotty.epfl.ch/docs/contributing/community-build.html

Fixes a new problem exhibited in the ZIO 1 code base that got uncovered
by the previous fix to AndTypes.
@bishabosha
Copy link
Member

bishabosha commented Jul 18, 2023

but I might give it a shot, are these the steps? https://dotty.epfl.ch/docs/contributing/community-build.html

yeah that's correct, you can also look at an old PR that added a project e.g. #14911

@odersky
Copy link
Contributor Author

odersky commented Jul 18, 2023

Thanks for the minimization @smarter. That illustrated another problem that was previously hidden by the wrong baseType handling.

def tryBaseType(cls2: Symbol) =

def computeBase(tp: Type): Type = tp.widenDealias match
case tp @ AndType(tp1, tp2) =>
Copy link
Member

@smarter smarter Jul 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call the operands tp11 and tp12 to avoid shadowing the existing tp1/tp2.

@odersky
Copy link
Contributor Author

odersky commented Jul 19, 2023

That fixed it! In light of the problem it found we can still keep ZIO 1 in frozen state in the community build, until there's something new that would have to be changed in ZIO itself. But it would be good to also get ZIO 2.

@odersky odersky assigned odersky and smarter and unassigned odersky Jul 19, 2023
Copy link
Member

@dwijnand dwijnand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. I'd love a #how-i-fixed-it, if possible.

@odersky
Copy link
Contributor Author

odersky commented Jul 19, 2023

How I fixed it

I compiled with -explain, which gave me a subtype trace, but that one came too late since the problematic type variable R in #18226 had already been instantiated to Any. So I turned all traces on in Typer and TypeComparer by changing occurrences of trace to trace.force in the source. That gave me a long log, and I searched for when the variable x was first typed and adapted. That pointed me to the interesting subtype trace that should have passed but did not. The goal that should have succeeded but did not was

 R & Row[Int] <: Row[String]

with type variable R uninstantiated. The goal should have succeeded in fourthTry by going through an either of two alternative tests. But it failed before getting there. I then added some debug printlns to TypeComparer to figure out which branches it took and after a while I realized that because the RHS is a named type Row[String] it tried to compute the base type of the LHS with class Row. That base type exists; it is Row[Int]. The logic then continued with that base type and failed since Row[Int] !<: Row[String]. I then tried to turn the base type logic off if the (widened, dealiased) LHS is an AndType. That made the test pass.

But I knew that change would be too restrictive. We sometimes need the base type computation, as in here:

class Row[+X]
class A
class B
class C extends Row[A]
class D extends Row[B]
C & D <: Row[A & B]

Here we compute the base type of Row for C & D, which is Row[A & B], and succeed. But neither C nor D alone is a subtype of Row[A & B].

So I then tried to identified exactly those cases where the base type would lose information, and proceed to fourthTry in those cases. I missed a case, which was exhibited in ZIO 1 and which @smarter minimized. That case was fixed by a slight change in 3a252a7.

@odersky
Copy link
Contributor Author

odersky commented Jul 19, 2023

Writing my previous comment showed me that the logic can be simplified: We can always try to do the basetype computation, but if that fails we can see whether LHS was an AndType and proceed to fourthTry.

Thinking more about it ... On the other hand, if we do that, the computation with the approximated base type might instantiate type variables on the right, which then could be instantiated to something less good than otherwise. So it's not a try base type, if that fails fourthTry, but a try both base type and fourthTry and pick the one that succeeds while instantiating the least amount of variables (or most amount for GADTs). Which is either.

Copy link
Member

@smarter smarter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but it's not clear to me if this change can go in the LTS since the risk of regression is hard to estimate. In this week's dotty meeting we were leaning towards disallowing type inference changes in the LTS.

@odersky
Copy link
Contributor Author

odersky commented Jul 19, 2023

I also think backporting this to the LTS would be too risky.

@odersky odersky merged commit 816bd5e into scala:main Jul 19, 2023
17 checks passed
@odersky odersky deleted the fix-18226 branch July 19, 2023 16:57
@smarter
Copy link
Member

smarter commented Jul 19, 2023

The main branch is still 3.3.x, we haven't made a branch for the lts yet

@Kordyjan Kordyjan added this to the 3.4.0 milestone Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants