Support executor #6792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

pvditt wants to merge 17 commits into v2 from support-executor

Contributor

pvditt commented Dec 9, 2025 •

edited

Loading

What changes were proposed in this pull request?

idl changes:

copy over Node/Task/Metadata changes needed for initial cloud merge
copy over ExecutionEnv
remove datacatalog

plugins changes:

copy over plugin change to make resources garbage collectable https://github.com/unionai/flyte/commit/007d2e675431c366fea6e0766a8ce9363b4791a8
remove un-needed EventsRecorder from TaskExecutionContext
remove EmbeddedSecretManager (might re-add later?)

flytestdlib changes:

copy over GRPC metrics added in https://github.com/unionai/flyte/commit/00cf6135d5e7001d6b0c9d7e85f2588979616420

How was this patch tested?

running in staging clusters at union

Labels

Please add one or more of the following labels to categorize your PR:

added: For new features.
changed: For changes in existing functionality.
deprecated: For soon-to-be-removed features.
removed: For features being removed.
fixed: For any bug fixed.
security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

pvditt added 6 commits

December 8, 2025 19:39


          add RawOutputDataConfig

9e3de0b

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          add ExecutionEnv idl

26fbae9

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          add Node related idl

fa2a678

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          remove datacatalog

7fc9845

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          re-add mocks

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          remove dynamic job spec

90ee3fa

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

Collaborator

flyte-bot commented Dec 9, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.


          update cache client

1da8038

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

Collaborator

flyte-bot commented Dec 9, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.


          Merge branch 'v2' into support-executor

9e79ab8

Collaborator

flyte-bot commented Dec 13, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

pvditt added 2 commits

December 13, 2025 23:18


          gen storage metadata mock

c74fb0a

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          Merge branch 'v2' into support-executor

8d7734c

Collaborator

flyte-bot commented Dec 14, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

pvditt added 3 commits

December 16, 2025 20:43


          set v2 scope for prom request metrics

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          copy over grpcutils metrics

f339c81

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>


          move secret manager impl to executor

903e74d

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

Collaborator

flyte-bot commented Dec 17, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.


          copy over garbage collectable changes

4fb06fb

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

Collaborator

flyte-bot commented Dec 18, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.


          Merge branch 'v2' into support-executor

96cf77c

pvditt marked this pull request as ready for review

December 23, 2025 01:14

gen

24c88b5

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

pvditt requested review from machichima, popojk and wild-endeavor

December 23, 2025 01:30

machichima reviewed

View reviewed changes

flyteidl2/datacatalog/datacatalog.proto

Member

machichima Dec 23, 2025

Curious, why do we want to remove this? Replaced by cacheservice?

Contributor Author

pvditt Dec 23, 2025

yup replaced by cacheservice

Member

machichima commented Dec 23, 2025

Got same error when doing make lint locally. Could you please merge v2? there's a fix for this: #6810 (comment)


          Merge branch 'v2' into support-executor

e855d8e

Collaborator

flyte-bot commented Dec 23, 2025

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

wild-endeavor reviewed

View reviewed changes

Contributor

wild-endeavor left a comment

But I don't understand the new messages like TaskNode, TaskNodeOverrides, NodeMetadata, etc. Are these going to be used in perpetuity for v2? Feels weird. Or will they only be used temporarily? If temporarily, we should merge these in as deprecated. And delete them soon after.

flyteidl2/core/tasks.proto

    
              // A Workflow graph Node. One unit of execution in the graph. Each node can be linked to a Task, a Workflow or a branch

              // node.

              message Node {

Contributor

wild-endeavor Dec 23, 2025

@pvditt don't Nodes not exist anymore?

Contributor Author

pvditt Dec 23, 2025 •

edited

Loading

@wild-endeavor will be deprecated in follow-ups. Wanted to have as few logical changes in the initial migration (first massive code change) as possible to support better reviews

Contributor Author

pvditt commented Dec 23, 2025

But I don't understand the new messages like TaskNode, TaskNodeOverrides, NodeMetadata, etc. Are these going to be used in perpetuity for v2? Feels weird. Or will they only be used temporarily? If temporarily, we should merge these in as deprecated. And delete them soon after.

@wild-endeavor Wanted to have as few logical changes in the initial migration (first massive code change) as possible to support better reviews. Can update to have these be deprecated as part of this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet