[KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine by LamiumAmplexicaule · Pull Request #7349 · apache/kyuubi

LamiumAmplexicaule · 2026-03-12T05:25:50Z

Why are the changes needed?

The engine and the service are separate components, so their authentication principal and keytab should also be separated.
Close: #7348

How was this patch tested?

Unit tests.

./build/mvn clean install -Dtest=none -DwildcardSuites=org.apache.kyuubi.ha.client.zookeeper.EmbeddedZookeeperDiscoveryClientSuite,org.apache.kyuubi.ha.client.zookeeper.ZookeeperClientProviderSuite,org.apache.kyuubi.engine.spark.SparkProcessBuilderSuite

Was this patch authored or co-authored using generative AI tooling?

No.

…eper authentication between the service and the engine

aajisaka

Thank you @LamiumAmplexicaule. Started CI workflow

aajisaka · 2026-03-12T05:52:54Z

docs/configuration/settings.md

+| kyuubi.ha.zookeeper.engine.auth.principal      | &lt;undefined&gt;                                              | Kerberos principal name that is used for the engine's ZooKeeper authentication.                                                                                                                                          | string   | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab         | &lt;undefined&gt;                                              | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication.                                                                                                                           | string   | 1.12.0 |


If they are unset, what are used by default? Would you document?

Thank you for your review.
I have added the document for fallback of kyuubi.ha.zookeeper.engine.auth.principal and kyuubi.ha.zookeeper.engine.auth.keytab in 20b6f0d.

I realized that instead of updating it manually, I should use dev/gen/gen_all_config_docs.sh, so I’ve update it.

https://kyuubi.readthedocs.io/en/master/contributing/code/developer.html#append-descriptions-of-new-configurations-to-settings-md

aajisaka · 2026-03-12T06:03:13Z

kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/zookeeper/ZookeeperClientProvider.scala

+            })
+          Configuration.setConfiguration(jaasConf)
+        case _ =>
+      }


The indents look incorrect. Would you keep the current indents so that reviewers can easily review the diff.

I set it up according to IntelliJ IDEA Setup Guide — Apache Kyuubi, so I understand that this indentation is the result of automatic formatting. (It’s possible that I misconfigured something, though.)
Do I need to intentionally disable this and adjust it manually?

…keeper.engine.auth.principal` and `kyuubi.ha.zookeeper.engine.auth.keytab`

aajisaka

Minor comments

kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala

aajisaka · 2026-03-13T03:57:16Z

Hi @LamiumAmplexicaule would you fix the test failure https://github.com/apache/kyuubi/actions/runs/22996851002/job/66896343535?pr=7349#step:7:429? It looks related to the patch.

- set up zookeeper auth for engine *** FAILED ***
  java.lang.IllegalArgumentException: Can't get Kerberos realm
  at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:71)
  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
  at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
  at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled(KerberizedTestHelper.scala:154)
  at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled$(KerberizedTestHelper.scala:142)
  at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.tryWithSecurityEnabled(ZookeeperDiscoveryClientSuite.scala:66)
  at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.$anonfun$new$5(ZookeeperDiscoveryClientSuite.scala:142)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  ...
  Cause: java.lang.IllegalArgumentException: KrbException: Cannot locate default realm
  at javax.security.auth.kerberos.KerberosPrincipal.<init>(KerberosPrincipal.java:159)
  at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:120)
  at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:69)
  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
  at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
  at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled(KerberizedTestHelper.scala:154)
  at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled$(KerberizedTestHelper.scala:142)
  at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.tryWithSecurityEnabled(ZookeeperDiscoveryClientSuite.scala:66)
  at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.$anonfun$new$5(ZookeeperDiscoveryClientSuite.scala:142)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  ...

codecov-commenter · 2026-03-13T04:27:13Z

Codecov Report

❌ Patch coverage is 0% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (f319921) to head (b862221).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
.../ha/client/zookeeper/ZookeeperClientProvider.scala	0.00%	26 Missing ⚠️
...la/org/apache/kyuubi/ha/HighAvailabilityConf.scala	0.00%	10 Missing ⚠️
...ache/kyuubi/engine/spark/SparkProcessBuilder.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #7349   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         698     698           
  Lines       43657   43667   +10     
  Branches     5896    5896           
======================================
- Misses      43657   43667   +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

LamiumAmplexicaule · 2026-03-16T02:12:50Z

It turned out that the Cannot locate default realm error occurs for the following reason.

In org.apache.kyuubi.KerberizedTestHelper#tryWithSecurityEnabled, System.clearProperty("java.security.krb5.conf") is called in the finally block.

kyuubi/kyuubi-common/src/test/scala/org/apache/kyuubi/KerberizedTestHelper.scala

Line 159 in 52b038b

System.clearProperty("java.security.krb5.conf")

In org.apache.kyuubi.ha.client.zookeeper.ZookeeperClientProvider, org.apache.hadoop.security.authentication.util.JaasConfiguration or org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration is used,

kyuubi/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/zookeeper/ZookeeperClientProvider.scala

Lines 124 to 144 in 52b038b

    
           val jaasConf = jaasConfigurationCache.computeIfAbsent( 
        
             (principal, keytab), 
        
             _ => { 
        
               // HDFS-16591 makes breaking change on JaasConfiguration 
        
               DynConstructors.builder() 
        
                 .impl( // Hadoop 3.3.5 and above 
        
                   "org.apache.hadoop.security.authentication.util.JaasConfiguration", 
        
                   classOf[String], 
        
                   classOf[String], 
        
                   classOf[String]) 
        
                 .impl( // Hadoop 3.3.4 and previous 
        
                   // scalastyle:off 
        
                   "org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration", 
        
                   // scalastyle:on 
        
                   classOf[String], 
        
                   classOf[String], 
        
                   classOf[String]) 
        
                 .build[Configuration]() 
        
                 .newInstance("KyuubiZooKeeperClient", zkClientPrincipal, keytab) 
        
             }) 
        
           Configuration.setConfiguration(jaasConf)

and inside it there is a line:

options.put("refreshKrb5Config", "true");

Because of this, sun.security.krb5.Config#refresh is invoked before com.sun.security.auth.module.Krb5LoginModule#login.

At that point, since java.security.krb5.conf does not exist,

it attempts to read /etc/krb5.conf.

As /etc/krb5.conf does not exist, nothing is loaded, and the singleton instance maintained by sun.security.krb5.Config is overwritten (the stanzaTable becomes empty).

https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L152

As a result, get("libdefaults", "default_realm") returns null, which causes a KrbException("Cannot locate default realm") to be thrown.

https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L1196

For unit tests that use JAAS, we need to save the current javax.security.auth.login.Configuration before the test and restore it afterward.

…iguration` on zookeeper auth test

[KYUUBI apache#7348] Separate the principal and keytab used for ZooKe…

b2dc88c

…eper authentication between the service and the engine

github-actions bot added kind:documentation Documentation is a feature! module:server module:ha labels Mar 12, 2026

aajisaka requested changes Mar 12, 2026

View reviewed changes

LamiumAmplexicaule force-pushed the separate-princiapl-and-keytab branch from f2c2693 to 84da062 Compare March 12, 2026 07:00

[KYUUBI apache#7348] Add documentation for fallback of `kyuubi.ha.zoo…

20b6f0d

…keeper.engine.auth.principal` and `kyuubi.ha.zookeeper.engine.auth.keytab`

LamiumAmplexicaule force-pushed the separate-princiapl-and-keytab branch from 84da062 to 20b6f0d Compare March 12, 2026 09:01

aajisaka requested changes Mar 12, 2026

View reviewed changes

kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala Show resolved Hide resolved

kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala Outdated Show resolved Hide resolved

[KYUUBI apache#7348] Fix spacing between sentences

b862221

[KYUUBI apache#7348] Save and restore `javax.security.auth.login.Conf…

47ade1b

…iguration` on zookeeper auth test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine#7349

[KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine#7349
LamiumAmplexicaule wants to merge 4 commits intoapache:masterfrom
LamiumAmplexicaule:separate-princiapl-and-keytab

LamiumAmplexicaule commented Mar 12, 2026

Uh oh!

aajisaka left a comment •

edited

Loading

Uh oh!

aajisaka Mar 12, 2026

Uh oh!

LamiumAmplexicaule Mar 12, 2026 •

edited

Loading

Uh oh!

LamiumAmplexicaule Mar 12, 2026 •

edited

Loading

Uh oh!

aajisaka Mar 12, 2026

Uh oh!

LamiumAmplexicaule Mar 12, 2026

Uh oh!

aajisaka left a comment

Uh oh!

Uh oh!

Uh oh!

aajisaka commented Mar 13, 2026

Uh oh!

codecov-commenter commented Mar 13, 2026 •

edited

Loading

Uh oh!

LamiumAmplexicaule commented Mar 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		\| kyuubi.ha.zookeeper.engine.auth.principal \| <undefined> \| Kerberos principal name that is used for the engine's ZooKeeper authentication. \| string \| 1.12.0 \|
		\| kyuubi.ha.zookeeper.engine.auth.keytab \| <undefined> \| Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. \| string \| 1.12.0 \|

Conversation

LamiumAmplexicaule commented Mar 12, 2026

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

aajisaka left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aajisaka Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

LamiumAmplexicaule Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LamiumAmplexicaule Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aajisaka Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

LamiumAmplexicaule Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aajisaka commented Mar 13, 2026

Uh oh!

codecov-commenter commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

LamiumAmplexicaule commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aajisaka left a comment •

edited

Loading

LamiumAmplexicaule Mar 12, 2026 •

edited

Loading

LamiumAmplexicaule Mar 12, 2026 •

edited

Loading

codecov-commenter commented Mar 13, 2026 •

edited

Loading

LamiumAmplexicaule commented Mar 16, 2026 •

edited

Loading