Skip to content

FLINK-5911: Orchestrator cleanup in status.py (PR 4/4)#4286

Merged
nleigh merged 8 commits intomasterfrom
u/nathanleigh/FLINK-5911/orchestrator-cleanup
Apr 14, 2026
Merged

FLINK-5911: Orchestrator cleanup in status.py (PR 4/4)#4286
nleigh merged 8 commits intomasterfrom
u/nathanleigh/FLINK-5911/orchestrator-cleanup

Conversation

@nleigh
Copy link
Copy Markdown
Contributor

@nleigh nleigh commented Apr 13, 2026

Ticket: https://jira.yelpcorp.com/browse/FLINK-5911

Part 4 of 4. Final cleanup — all Flink logic now lives in `flink_tools.py`; `status.py` is a thin orchestrator.

Also addresses feedback from PR 3 (#4285):

  • `checkpoint_data` typed as `Dict[str, Union[FlinkCheckpointStatus, BaseException]]` (was `Dict[str, Any]`) in both `format_flink_jobs_table` and the call site in `status.py`
  • `_fetch_flink_job_details` / `fetch_flink_job_checkpoints` renamed to drop the `` prefix — they are called cross-module so the private prefix was misleading
  • `should_job_info_be_shown` uses a set literal instead of a tuple
  • `format_flink_jobs_table` signature simplified: `job_ids` removed (was never used inside the function; job IDs are accessed via `job["jid"]`), `checkpoint_data` made an optional keyword arg defaulting to `None`
  • `TestFormatFlinkJobsTable` test class added: verbose job ID display, 3-job non-verbose limit, verbose shows all jobs, failed state color, checkpoint/restart display, `None` dashboard URL guard

Changes

`paasta_tools/flink_tools.py`

  • `format_flink_jobs_table` signature: `(jobs, dashboard_url, verbose, checkpoint_data=None)`
  • `fetch_flink_job_details` / `fetch_flink_job_checkpoints`: public names (no `_` prefix)
  • `should_job_info_be_shown`: set literal
  • `checkpoint_data` typed as `Dict[str, Union[FlinkCheckpointStatus, BaseException]]`

`paasta_tools/cli/cmds/status.py`

  • Removed direct imports of `get_flink_config/overview/jobs_from_paasta_api_client` — now called as `flink_tools.*`
  • Removed `FlinkJobs` import and `FlinkJobs()` fallback; `jobs`/`job_ids`/`overview`/`checkpoint_data` initialized upfront
  • All API fetches (config, overview, jobs, checkpoints) consolidated into a single `if state == "running":` block
  • `checkpoint_data` type annotation updated to match `fetch_flink_job_checkpoints` return type

`tests/test_flink_tools.py`

  • `TestFormatFlinkJobsTable`: 6 tests covering job table rendering, job limit, verbose mode, state colors, checkpoints/restarts, missing dashboard URL

nleigh added 5 commits April 13, 2026 08:12
…, BaseException]]

Addresses nemacysts review comment on PR #4285.
Removes direct imports of get_flink_config/overview/jobs_from_paasta_api_client
and calls them via flink_tools.* — consistent with all other flink helpers.
These are called cross-module from status.py so the _ prefix is misleading.
Addresses nemacysts review comment on PR #4285.
Addresses nemacysts review comment on PR #4285.
…mport

Initializes dashboard_url/overview/jobs/job_ids/checkpoint_data upfront and
groups all API fetch calls (config, overview, jobs, checkpoints) inside a
single if-running block. Removes FlinkJobs import and FlinkJobs() fallback.
nemacysts
nemacysts previously approved these changes Apr 13, 2026
nleigh added 2 commits April 13, 2026 08:38
…oint_data optional

job_ids was never used inside the function (job IDs are accessed via job["jid"]).
checkpoint_data defaults to None so callers don't need to pass an empty dict.
Covers verbose job ID display, 3-job non-verbose limit, verbose shows all jobs,
failed state color, checkpoint/restart display, and None dashboard_url guard.
@nleigh nleigh marked this pull request as ready for review April 13, 2026 16:38
@nleigh nleigh requested a review from a team as a code owner April 13, 2026 16:38
@nleigh nleigh requested review from Copilot and nemacysts April 13, 2026 16:38
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Final cleanup for FLINK-5911 (part 4/4): completes the refactor so status.py is primarily orchestration while Flink-specific logic (formatting + async fetch helpers) lives in flink_tools.py.

Changes:

  • Made Flink job detail/checkpoint fetch helpers public (fetch_flink_job_details, fetch_flink_job_checkpoints) and adjusted call sites accordingly.
  • Simplified format_flink_jobs_table() signature (removed unused job_ids, made checkpoint_data optional with a None default).
  • Added focused unit tests for format_flink_jobs_table() rendering behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
paasta_tools/flink_tools.py Publicizes fetch helpers; simplifies jobs table formatter signature; minor refactor in should_job_info_be_shown.
paasta_tools/cli/cmds/status.py Consolidates Flink API calls behind flink_tools.* and simplifies orchestration/variable initialization.
tests/test_flink_tools.py Adds a new test class covering job table formatting behaviors and edge cases.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

except Exception:
pass # checkpoints are informational, don't fail status

job_details = flink_tools.collect_flink_job_details(status, overview, [])
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collect_flink_job_details() accepts a jobs list and returns it in the details dict, but this call always passes [] even though jobs has just been fetched. Consider passing the jobs variable here (or, if jobs are intentionally unused in the details struct, remove the jobs parameter from collect_flink_job_details to avoid misleading call sites).

Copilot uses AI. Check for mistakes.
The jobs field in FlinkJobDetailsDict was stored but never read back —
format_flink_jobs_table receives jobs directly from status.py.
Addresses nemacysts review comment on PR #4286.
Copy link
Copy Markdown
Member

@nemacysts nemacysts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

less code in status.py is always nice :D

@nleigh nleigh merged commit 56ae2b2 into master Apr 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants