Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory consumption #97

Merged
merged 1 commit into from
Jul 27, 2021
Merged

Reduce memory consumption #97

merged 1 commit into from
Jul 27, 2021

Conversation

julienrf
Copy link
Collaborator

Creating chains of settings was consuming an exponential amount of memory. This did not scale when used with builds with dozens of modules.

By using dynamic settings, the chain of dependencies is not kept, resulting in less memory consumption.

Creating chains of settings was consuming an exponential amount of memory. This did not scale when used with builds with dozens of modules.

By using dynamic settings, the chain of dependencies is not kept, resulting in less memory consumption.
@julienrf julienrf merged commit 516315d into main Jul 27, 2021
@julienrf julienrf deleted the eat-less-memory branch July 27, 2021 11:10
@adpi2
Copy link
Member

adpi2 commented Jul 27, 2021

Oh that's a nice bug!

Have you tried with join rather than foldLeft?

Something like this maybe (I didn't try to compile it):

      val modules: Seq[Setting[(String, String)]]] = allProjectRefs.map { case (projectRef, _) =>
          Def.setting {
            val projectOrganization       = (projectRef / organization).value
            val projectName               = (projectRef / moduleName).value
            val projectVersion            = (projectRef / version).value
            val projectCrossVersion       = (projectRef / crossVersion).value
            val projectScalaVersion       = (projectRef / scalaVersion).value
            val projectScalaBinaryVersion = (projectRef / scalaBinaryVersion).value
            if (versionRegex.findFirstMatchIn(projectVersion).isDefined) {
              val nameWithBinarySuffix =
                CrossVersion(projectCrossVersion, projectScalaVersion, projectScalaBinaryVersion)
                  .fold(projectName)(_(projectName))
              Some(projectOrganization -> nameWithBinarySuffix)
            } else {
              // Don’t include the module if its version does not match the regex
              None
            }
          }
        }
    modules.join.map(_.flatten.toSet)

@julienrf
Copy link
Collaborator Author

Interesting, I didn’t know about join. Your code does not compile, but I could use Def.uniform instead (which is a generalization of join, AFAIU), and it works fine. Are there any benefits of doing that instead of what I have in the PR?

@adpi2
Copy link
Member

adpi2 commented Jul 27, 2021

It feels more idiomatic to me. At least you don't need those settingDyn that are hard to understand and that could be removed by mistake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants