[Proposal] Opt-in Tilt-based local development workflow with Kind#5060
[Proposal] Opt-in Tilt-based local development workflow with Kind#5060hajnalmt wants to merge 2 commits intovolcano-sh:masterfrom
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
volcano-sh-bot
added
the
size/XXL
Summary of ChangesHello @hajnalmt, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the local development experience for Volcano contributors by introducing a Tilt-based workflow. It addresses the slow feedback loop inherent in traditional Kubernetes development by enabling live-reloading of core components within a local Kind cluster. The new setup streamlines environment provisioning, image building, and deployment, making iterative development more efficient and accessible, while also providing a safer environment for AI-assisted coding. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive Tilt-based local development workflow, which is a great improvement for developer productivity. The changes include a detailed design document, new Makefiles, Dockerfiles, and Tilt configuration to orchestrate the local environment. My review focuses on ensuring consistency, correctness, and efficiency in the new files. I've identified a few issues, such as inconsistencies in the Go version used, an invalid Kubernetes version for downloading kubectl, and opportunities to improve Docker layer caching. Addressing these points will make the new workflow more robust and reliable.
hajnalmt
changed the title
volcano-sh-bot
added
the
do-not-merge/work-in-progress
There was a problem hiding this comment.
Pull request overview
This PR introduces a Tilt + Kind + ctlptl based local development workflow for Volcano, including a design proposal, developer documentation updates, and the supporting Tilt/Make/Docker/Helm plumbing needed for live-reload development against a local Kind cluster.
Changes:
- Add a full Tilt-based local dev environment under
hack/tilt/(Tiltfile, Make targets, Dockerfiles, entrypoint, ctlptl cluster config, Helm override values). - Update the Helm chart to support a dedicated admission-init image override.
- Add/extend developer docs with quick-start instructions and a detailed design proposal.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| installer/helm/chart/volcano/values.yaml | Adds admission_init_image_name default value to support separate init image configuration. |
| installer/helm/chart/volcano/templates/admission-init.yaml | Switches admission-init Job image to use admission_init_image_name. |
| hack/tilt/values-tilt-override.yaml | Adds dev-oriented Helm overrides for Tilt (pull policy, tolerations, log levels, scheduler config). |
| hack/tilt/entrypoint.sh | Adds in-container rebuild/restart loop triggered by Tilt live_update. |
| hack/tilt/ctlptl-kind-registry.yaml | Declares the Kind cluster + local registry via ctlptl. |
| hack/tilt/Tiltfile | Defines image builds, Helm deploy, resource grouping, and test resources for Tilt. |
| hack/tilt/Makefile | Adds dev-* lifecycle and tooling install targets (Tilt/kind/ctlptl). |
| hack/tilt/Dockerfile.tilt | Provides a Go-based dev image for scheduler/controller/webhook with live-update support. |
| hack/tilt/Dockerfile.admission-init.tilt | Adds a lightweight admission-init image with kubectl + secret generation script. |
| hack/lib/install.sh | Enhances kind install/version checking and makes cluster creation idempotent. |
| docs/development/prepare-for-development.md | Adds Tilt quick-start section and pointers to the new workflow/design doc. |
| docs/development/development.md | Documents Tilt workflow usage, tips, logging/status commands, and targets. |
| docs/design/tilt-based-development.md | Adds a detailed design proposal covering architecture and rationale. |
| Makefile | Adds dev workflow integration (OUTPUT_DIR/BIN_DIR refactor, includes hack/tilt/Makefile, exposes kind version). |
| .dockerignore | Excludes _output from Docker build contexts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
/kind feature I have been using tilt succefully for nearly 5 month now to debug volcano issues and understand various behaviours. (originally I have used k3d, but as kind is volcano's main testing framework, I have ported the feature to kind filed this proposal.) If the proposal is not accepted, it can still be a good baseline for people who want to use it on their clusters (I do that too) 😊 As tilt fully supports remote development, and used by several big projects like ClusterAPI, Crossplane and Cilium. |
volcano-sh-bot
added
kind/feature
hajnalmt
changed the title
|
It's a great enhancement! I haven't used ctlptl or tilt before, they look great. I often get frustrated with having to recompile the image and run |
hajnalmt
force-pushed
the
tilt-based-development-kind-compact
branch
5 times, most recently
from
df6b1a8 to
1c1c4fd
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
hajnalmt
changed the title
volcano-sh-bot
removed
the
do-not-merge/work-in-progress
hajnalmt
changed the title
volcano-sh-bot
added
the
do-not-merge/work-in-progress
hajnalmt
force-pushed
the
tilt-based-development-kind-compact
branch
from
1c1c4fd to
ca3c48c
Compare
hajnalmt
changed the title
volcano-sh-bot
removed
the
do-not-merge/work-in-progress
hajnalmt
force-pushed
the
tilt-based-development-kind-compact
branch
from
ca3c48c to
bc56170
Compare
Add design document covering motivation, architecture, detailed design, tool comparison (Tilt vs Skaffold, DevSpace, Telepresence), and agentic workflow safety considerations for the Kind + ctlptl + Tilt local development workflow. Update prepare-for-development.md and development.md with Tilt-based quick start and full workflow reference. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Add Tilt-based local development environment using Kind (via ctlptl) with a local container registry. Provides single-command setup (make dev-up) with live-reload for scheduler, controller-manager, and webhook-manager components. Key features: - ctlptl-managed Kind cluster with local Docker registry - Tilt live-update with in-container Go rebuilds via entrypoint.sh - Multi-architecture support (amd64/arm64) for tool downloads, Docker builds, and in-container rebuilds - Integrated monitoring (Prometheus, Grafana) with port-forwards - On-demand unit and e2e test triggers via Tilt resources - All dev tooling self-contained in _output/bin Includes Helm chart fix: use dedicated admission_init_image_name variable for admission init Job image. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
hajnalmt
force-pushed
the
tilt-based-development-kind-compact
branch
from
bc56170 to
5b65f6e
Compare
|
I have finished every work on this one @JesseStutler It works like charm 😊 moved it from WIP, I can demonstrate it at any meeting. |
Summary
Resolves #5059
Design
See
docs/design/tilt-based-development.mdfor the full proposal covering motivation, architecture, file layout, image building strategy, live-reload mechanism, and comparison with alternative tools (Skaffold, DevSpace, manual scripts).What's Included
Commit 1:
docs: add Tilt-based local development design proposaldocs/design/tilt-based-development.mddocs/development/development.md(Tilt workflow section with tips)docs/development/prepare-for-development.mdCommit 2:
tilt: add Kind-based local dev workflow with live-reloadhack/tilt/Tiltfile— Tilt orchestration for 4 images + Helm deploy + resource groupinghack/tilt/Dockerfile.tilt— Multi-component Dockerfile with Go build + live-update supporthack/tilt/Dockerfile.admission-init.tilt— Lightweight admission init imagehack/tilt/Makefile—dev-up,dev-down,dev-cleantargets with tool auto-installhack/tilt/entrypoint.sh— File-watch loop for in-container rebuild on synchack/tilt/ctlptl-kind-registry.yaml— Declarative Kind + registry cluster confighack/tilt/values-tilt-override.yaml— Dev-specific Helm value overridesMakefileintegration andhack/lib/install.shversion-aware Kind installerQuick Start
E2E Testing
The Tiltfile includes on-demand e2e test resources that run directly against the live Kind cluster. All test resources use
TRIGGER_MODE_MANUALwithauto_init=False— they never run automatically.Available Test Suites
e2e-tests-allinstall-kwok,install-ginkgoe2e-tests-jobpinstall-ginkgoe2e-tests-jobseqinstall-ginkgoe2e-tests-schedulingbaseinstall-ginkgoe2e-tests-schedulingactioninstall-ginkgoe2e-tests-vcctlinstall-ginkgoe2e-tests-drainstall-ginkgoe2e-tests-hypernodeinstall-ginkgoe2e-tests-cronjobinstall-ginkgoDependency Workflow
Test dependencies (
install-ginkgo,install-kwok) are alsoTRIGGER_MODE_MANUAL— they must be triggered manually once per Tilt session before running tests. Triggering a test while its dependency is not yet Ready will block until the dependency is manually triggered and completes.install-ginkgo(andinstall-kwokif runninge2e-tests-all)Focused Test Execution
A pre-configured focused example resource is included as a template:
E2E_FOCUSE2E_SKIPe2e-tests-schedulingbase-focused-exampleGang scheduling\[sig-.*\]|Full OccupiedThis resource can be modified directly in the Tiltfile or duplicated as a base to add more focused test resources targeting specific test cases during development.
Override the defaults by setting
E2E_FOCUSandE2E_SKIPenvironment variables before starting Tilt:All test resources set
E2E_LOCAL_ONLY=1, which mirrors the CI filtering used for local-only test suites.Images