[SPARK-54685][SQL][CONNECT] Remove redundant observed-metrics responses

[heyihong]

What changes were proposed in this pull request?

This PR removes the observed metrics response generation from SparkConnectPlanExecution in Spark Connect server since creating the observed metrics response in ExecuteThreadRunner is sufficient.

Changes:

• Removed the call to createObservedMetricsResponse() in the main query execution flow after generating metrics response
• Removed the private method createObservedMetricsResponse() that was responsible for creating observed metrics responses from query execution

Why are the changes needed?

The createObservedMetricsResponse() method is being removed because:

1. The response is always empty: In Spark Connect, all observations must be registered first using executeHolder.observations, and the method specifically filters out already-registered observations (!executeHolder.observations.contains(name)). This means the collected observed metrics will always be empty.
2. Misleading coverage: The method only handles plans in the query execution path but doesn't cover commands, which can also produce observed metrics. This incomplete coverage could be misleading to users expecting observed metrics from commands. Removing this code simplifies the execution flow and eliminates potential confusion about observed metrics handling.

Does this PR introduce any user-facing change?

No. Since the method always returns an empty response due to the observation registration requirement, removing it has no functional impact on users.

How was this patch tested?

build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor 2.1.47

[heyihong]

@ueshin @hvanhovell @viirya

[hvanhovell]

@heyihong can you please do this the other way around and remove the dependency on Observation in ExecuteThreadRunner.

[allisonwang-db]

cc @ueshin

[heyihong]

@heyihong can you please do this the other way around and remove the dependency on Observation in ExecuteThreadRunner.

@hvanhovell Yes, I am working on that. This pull request is a code cleanup and is orthogonal to your ask if I understand correctly. This piece of code can be removed even if we remove the dependency on Observation in ExecuteThreadRunner.

cc: @allisonwang-db @ueshin

[allisonwang-db]

cc @ueshin

[HyukjinKwon]

Merged to master.

[yaooqinn]

@heyihong @HyukjinKwon, can you check the JIRA-ID which seems to be wrong?

[heyihong]

@yaooqinn Good catch — this is the right JIRA ticket: https://issues.apache.org/jira/browse/SPARK-54685
cc: @HyukjinKwon