[CORE] Add Bloop integration for faster Scala incremental compilation#11645
Merged
liuneng1994 merged 1 commit intoapache:mainfrom Feb 26, 2026
Merged
[CORE] Add Bloop integration for faster Scala incremental compilation#11645liuneng1994 merged 1 commit intoapache:mainfrom
liuneng1994 merged 1 commit intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
liuneng1994
force-pushed
the
bloop-maven-plugin
branch
from
e5d9019 to
4f4abda
Compare
|
Run Gluten Clickhouse CI on x86 |
- Add bloop-maven-plugin (v2.0.3) to pom.xml with version property and pluginManagement entry - Add -Pbloop profile to skip style checks during config generation - Add dev/bloop-setup.sh script to generate Bloop configuration with JVM test options injection and -release option removal - Add dev/bloop-test.sh convenience wrapper with SPARK_ANSI_SQL_MODE=false default for Spark 4.x test compatibility - Add docs/developers/bloop-integration.md with usage guide Bloop eliminates ~52s Zinc analysis reload overhead by maintaining a persistent JVM with warm compiler state, enabling faster iterative development workflows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
baibaichen
force-pushed
the
bloop-maven-plugin
branch
from
4f4abda to
7d32a8a
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
approved these changes
Feb 26, 2026
| </build> | ||
| </profile> | ||
| <profile> | ||
| <id>bloop</id> |
Contributor
There was a problem hiding this comment.
skipCheck seems more clear
baibaichen
added a commit
that referenced
this pull request
Feb 26, 2026
…d introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in #11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes #11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3 tasks
baibaichen
added a commit
that referenced
this pull request
Feb 26, 2026
…d introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in #11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes #11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
added a commit
that referenced
this pull request
Feb 26, 2026
…d introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in #11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes #11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
added a commit
that referenced
this pull request
Feb 26, 2026
…d introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in #11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes #11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
added a commit
that referenced
this pull request
Feb 27, 2026
…lugin 4.9.2 scala-maven-plugin 4.9.2 introduced a behavior change in ScalaMojoSupport.computeBytecodeVersionOptions() [1]: when maven.compiler.target is set (e.g. "1.8"), it now injects '-release <target>' into scalac args for Scala >= 2.12. The '-release' flag in scalac is only supported when running on Java 9+. Building under JDK 8 with recompileMode=all therefore fails with: scalac error: -release is only supported on Java 9 and higher This was not visible with recompileMode=incremental (the previous default set by #11645) because Zinc's SbtIncrementalCompiler takes a different code path that bypasses computeBytecodeVersionOptions. Fix: in the java-8 profile (auto-activated via <jdk>1.8</jdk>), override scala-maven-plugin's <target> with an empty string. This makes isNotEmpty(target)=false [2], so -release is not injected. Scalac 2.12 running under JDK 8 then uses its own default bytecode target (JVM 1.8), which is correct. Java 9/11/17/21 environments are unaffected: for those, -release N is a valid scalac argument and no override is needed. [1] https://github.com/davidB/scala-maven-plugin/blob/4.9.2/src/main/java/scala_maven/ScalaMojoSupport.java#L364-L392 [2] https://github.com/davidB/scala-maven-plugin/blob/4.9.2/src/main/java/scala_maven/ScalaMojoSupport.java#L154-L156 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
added a commit
to baibaichen/gluten
that referenced
this pull request
Feb 27, 2026
…d introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in apache#11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes apache#11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
added a commit
to baibaichen/gluten
that referenced
this pull request
Feb 27, 2026
Problem: scala-maven-plugin 4.9.2 changed behavior for Scala >= 2.12 builds. When maven.compiler.target is set (e.g. "1.8"), it now passes the value to computeBytecodeVersionOptions() [1] which injects '-release 1.8' into scalac args. The scalac '-release' flag is only supported on Java 9+, so building under JDK 8 with recompileMode=all (restored by the parent commit) fails with: scalac error: -release is only supported on Java 9 and higher Root cause: PR apache#11645 (which introduced scala-maven-plugin 4.9.2) also set recompileMode=incremental, which routes through Zinc's SbtIncrementalCompiler and bypasses computeBytecodeVersionOptions(). Restoring recompileMode=all exposes the 4.9.2 regression for Java 8 builds. Fix: in the java-8 profile, pin scala.compiler.version to 4.8.0, which does not have this regression. Java 9+ builds continue to use 4.9.2 unaffected. [1] https://github.com/davidB/scala-maven-plugin/blob/4.9.2/src/main/java/scala_maven/ScalaMojoSupport.java#L620-L648 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
added a commit
that referenced
this pull request
Feb 27, 2026
…d introduce fast-build profile (#11659) * [GLUTEN-11658][CORE] Restore scala.recompile.mode default to 'all' and introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in #11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes #11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix Java 8 build failure caused by scala-maven-plugin 4.9.2 Problem: scala-maven-plugin 4.9.2 changed behavior for Scala >= 2.12 builds. When maven.compiler.target is set (e.g. "1.8"), it now passes the value to computeBytecodeVersionOptions() [1] which injects '-release 1.8' into scalac args. The scalac '-release' flag is only supported on Java 9+, so building under JDK 8 with recompileMode=all (restored by the parent commit) fails with: scalac error: -release is only supported on Java 9 and higher Root cause: PR #11645 (which introduced scala-maven-plugin 4.9.2) also set recompileMode=incremental, which routes through Zinc's SbtIncrementalCompiler and bypasses computeBytecodeVersionOptions(). Restoring recompileMode=all exposes the 4.9.2 regression for Java 8 builds. Fix: in the java-8 profile, pin scala.compiler.version to 4.8.0, which does not have this regression. Java 9+ builds continue to use 4.9.2 unaffected. [1] https://github.com/davidB/scala-maven-plugin/blob/4.9.2/src/main/java/scala_maven/ScalaMojoSupport.java#L620-L648 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds Bloop build server integration to accelerate Scala development workflows. Bloop maintains a persistent JVM with warm Zinc compiler state, dramatically reducing incremental compilation times.
Key Changes
-Pbloopprofile to skip style checks during config generation-releaseoption removal for compatibilitySPARK_ANSI_SQL_MODE=falsedefault for Spark 4.x test compatibilityBenchmark Results
Usage
Test plan
./dev/bloop-setup.sh -Pjava-17,spark-4.1,scala-2.13,backends-veloxbloop compile gluten-core- passesbloop test gluten-core- 6 suites, 24 tests passSPARK_ANSI_SQL_MODE=false bloop test backends-velox -o '*VeloxHashJoinSuite*'- 7 tests pass🤖 Generated with Claude Code