Open
Conversation
- Hive version 1.2.2 → 2.3.9 in dependencies.gradle - Added hive-serde (split from hive-metastore in 2.3.9) - Added DataNucleus/Derby/javax.jdo test deps (consolidated in root build.gradle) - Added calcite-druid test dep (Hive 2.3.9 references DruidQuery at runtime) - Pentaho exclusion (not in Maven Central) - Hive CBO disabled via systemProperty (CalcitePlanner incompatible with Calcite 1.21.0.265) - Excluded problematic Hive 2.3.9 transitives from coral-common (log4j-core, javax.servlet from Jetty orbit, slf4j-log4j12) to prevent conflicts in downstream consumers like Trino - coral-service: use deps map instead of hardcoded Hive/Hadoop versions - StaticHiveFunctionRegistry: disabled TRANSLATE3, split substr/substring - MetastoreProvider: getProxy(conf) → getProxy(conf, true) (API change in 2.3.9) - Disabled 3 tests: testEnumUnionString, testTranslateFunction (x2)
YogeshKothari26
force-pushed
the
pr2-hive-239
branch
from
f0a9c98 to
c3f2fc1
Compare
Resolve conflict in gradle/dependencies.gradle: keep both spark3.5 entries from master and Hive 2.3.9 DataNucleus/Derby deps from PR2.
coral-spark-catalog (PR linkedin#584) tests fail on the Hive 2.3.9 branch because: 1. avatica-1.8.0.jar (via calcite-druid) bundles jackson-databind 2.6.3 un-relocated, shadowing Spark 3.5's jackson-databind 2.15.2 and causing NoSuchMethodError on JsonMappingException(Closeable, String) which was added in jackson-databind 2.7.0. 2. Hive 2.3.9 embedded metastore requires DataNucleus + Derby + hive-serde (separate from hive-metastore in 2.3.x). 3. Hive 2.3.9 CalcitePlanner is incompatible with Calcite 1.21.0.265 (requires CBO disable). All changes are test-only (testImplementation / test block).
YogeshKothari26
force-pushed
the
pr2-hive-239
branch
from
fe06489 to
1912407
Compare
- Re-enable addFunctionEntry("translate", TRANSLATE3) and
addFunctionEntry("translate3", TRANSLATE3) — these were disabled
preemptively during the Hive 2.3.9 upgrade but are not affected
by the Calcite version mismatch (verified by unit tests and
production DAG regression against 4027 views on Holdem)
- Re-enable testTranslateFunction in CoralSparkTest and
HiveToTrinoConverterTest (both pass)
- Revert substr/substring from split 2-arg/3-arg registrations back
to original optionalOrd(2) — the split was unnecessary as other
functions using the same pattern (round, bround, rand) were not
affected
- Update testEnumUnionString comment to reflect actual Hive 2.3.9
error (AssertionError in UnparseTranslator, not "stricter type
checking")
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request, and why are they necessary?
This PR upgrades Coral's Hive dependency from 1.2.2 to 2.3.9.
Part 2 of 3: #580 Gradle → this PR (Hive) → Java 17 (WIP). Stays on Java 8.
Dependencies
hive-serde(split fromhive-metastorein 2.3.9)build.gradlesystemProperty(CalcitePlanner incompatible with Calcite 1.21.0.265)Source fixes
StaticHiveFunctionRegistry: No changes to function registrations. All existing functions (including TRANSLATE3, substr/substring) are compatible with Hive 2.3.9.MetastoreProvider:getProxy(conf)→getProxy(conf, true)(single-arg overload removed in 2.3.9)coral-spark-catalog compatibility
After merging master (which added
coral-spark-catalogvia #584), the new module's tests needed Hive 2.3.9 compatibility fixes:hive-metastoreandhive-exec-core(Hive 2.3.9 brings Jackson 2.6.x which conflicts with Spark 3.5's Jackson 2.15)Tests disabled
testEnumUnionString— Hive 2.3.9SemanticAnalyzerthrowsAssertionErrorinUnparseTranslator.addTranslationduring CREATE VIEW with UNION ALL between Avro enum and string columnsHMS API Changelog: 1.2.2 → 2.3.9
Core read APIs — UNCHANGED:
getTable,getDatabase,getTables,getAllDatabases,getAllTables,getFields,getSchema,listPartitions,getPartitionsByNames,listPartitionNames,listPartitionsByFilter,tableExists— all identical signatures.Breaking changes (4, all handled internally by Coral):
RetryingMetaStoreClient.getProxy(HiveConf)— removed, replaced bygetProxy(HiveConf, boolean)getProxy()Map param → ConcurrentHashMapdropPartitions()ignoreProtection param removedgetPartition()param order swapped (tblName, dbName → dbName, tblName)New methods: 25 additive (constraints, bulk metadata, partition values, etc.)
Thrift: 0.9.2 → 0.9.3 (wire compatible)
Full source comparison
Transitive dependency analysis
Compared
coral-commoncompile classpath: master (146 artifacts) → this PR (221 artifacts).+102 new (from Hive 2.3.9 dependency tree), -27 removed (from Hive 1.2.2).
Excluded transitives — verified NOT leaking in published POM:
org.apache.logging.log4j:log4j-coreorg.eclipse.jetty.orbit:javax.servletorg.slf4j:slf4j-log4j12Shadow JAR:
coral-trino-parserverified identical to master — 3,450 entries, zero differences.How was this patch tested?
PR2 branch (
pr2-hive-239— Java 8, Hive 2.3.9)./gradlew clean build— all tests passCombined branch (
java17-hive239-migration— Java 17, Hive 2.3.9, Gradle 8)Phase 3 — Full Regression (4028 production views, same-day comparison)
Test Plan (LinkedIn Internal) | Test Evidence (LinkedIn Internal)