Skip to content

Comments

feat(artifacts): add support for artifacts key#672

Open
timhuynh94 wants to merge 70 commits intomainfrom
feat_artifacts
Open

feat(artifacts): add support for artifacts key#672
timhuynh94 wants to merge 70 commits intomainfrom
feat_artifacts

Conversation

@timhuynh94
Copy link
Contributor

@timhuynh94 timhuynh94 commented Jan 19, 2026

Overview

This branch wires the worker into the new artifacts + object storage pipeline by:


Architecture

1. Worker config + storage hookup

  • cmd/vela-worker/flags.go, run.go, worker.go
    • CLI now includes storage flags (driver, endpoint, access/secret key, bucket) and passes them into storage.Setup.
    • Worker gains fields for Storage *storage.Setup (config) and Storage storage.Storage (runtime client).
  • cmd/vela-worker/operate.go
    • On startup, the worker calls VelaClient.Storage.GetInfo() to fetch storage configuration from the server.
    • If storage is enabled, it:
      • Fills in any missing worker config (endpoint, bucket, access/secret).
      • Builds a storage client via storage.New(...).
      • Stores the client on w.Storage and logs driver/bucket/endpoint.
    • If storage is disabled, it explicitly clears storage config and sets w.Storage = nil.

2. Executor setup (linux/local)

  • executor.Setup now includes a Storage storage.Storage field.
  • cmd/vela-worker/exec.go
    • If w.Storage was successfully initialized, it is passed into the executor setup.
    • The executor is created from a populated executor.Setup struct instead of an inline literal.
  • executor/executor_test.go
    • Tests construct enabled/disabled storage setups and ensure:
      • Linux and local executors can be created with storage.
      • The top‑level executor respects the Storage field in executor.Setup.

3. Linux / local executor behavior for artifacts

  • executor/linux/{build.go,stage.go,outputs.go,report.go,artifacts.go}
  • executor/local/{build.go,stage.go} and related tests

These pieces:

  • Thread the storage engine into the linux/local executor implementations.
  • When a pipeline step is configured with the artifacts key, they:
    • Collect artifacts file paths from the runtime.
    • Use the storage client (when enabled) to move those files into the configured bucket.
    • Prepare the metadata so the server (via the Vela SDK) can associate uploads with artifacts records.

4. Runtime integration

  • runtime/docker/artifacts.go (+ artifacts_test.go)
    • Adds Docker‑specific helpers for:
      • Discovering artifacts files in a container given paths from the pipeline definition.
      • Pulling those files out so the executor can upload them to storage / report them to the server.
  • runtime/engine.go
    • Wires the new test report helper into the runtime lifecycle so it runs at the appropriate point in a build.
  • runtime/kubernetes/container.go
    • Introduces PollFileNames as a no‑op for Kubernetes: artifacts file polling is not supported for dynamic pod environments yet.

5. Step skipping, middleware, and wiring

  • internal/step/skip.go (+ tests)
    • Updated to account for new behavior around test‑related steps and storage, so pipelines behave sensibly when storage is disabled or unavailable.
  • router/middleware/executor/executor_test.go
    • Ensures the executor wiring path continues to work with the new Storage field in executor.Setup.

6. Dependencies

  • go.mod, go.sum
    • Bump / align dependencies (including sdk-go, server storage packages, and Docker) to versions that include the new artifacts and storage APIs.

@timhuynh94 timhuynh94 requested a review from a team as a code owner January 19, 2026 17:05
@timhuynh94 timhuynh94 changed the title Feat artifacts Feat (artifacts): artifacts key Jan 19, 2026
@codecov
Copy link

codecov bot commented Feb 3, 2026

Codecov Report

❌ Patch coverage is 2.89855% with 134 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.95%. Comparing base (f816a53) to head (c8dc836).

Files with missing lines Patch % Lines
runtime/docker/artifact.go 0.00% 72 Missing ⚠️
executor/linux/outputs.go 0.00% 40 Missing ⚠️
cmd/vela-worker/exec.go 0.00% 9 Missing ⚠️
runtime/kubernetes/container.go 0.00% 6 Missing ⚠️
executor/linux/build.go 0.00% 3 Missing and 1 partial ⚠️
executor/linux/driver.go 0.00% 2 Missing ⚠️
cmd/vela-worker/flags.go 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #672      +/-   ##
==========================================
- Coverage   53.21%   51.95%   -1.26%     
==========================================
  Files         124      125       +1     
  Lines        5420     5551     +131     
==========================================
  Hits         2884     2884              
- Misses       2323     2453     +130     
- Partials      213      214       +1     
Files with missing lines Coverage Δ
executor/linux/linux.go 100.00% <ø> (ø)
executor/local/local.go 100.00% <ø> (ø)
executor/setup.go 100.00% <100.00%> (ø)
mock/docker/docker.go 0.00% <ø> (ø)
runtime/docker/docker.go 75.00% <ø> (ø)
runtime/kubernetes/kubernetes.go 87.87% <ø> (ø)
cmd/vela-worker/flags.go 0.00% <0.00%> (ø)
executor/linux/driver.go 50.00% <0.00%> (-50.00%) ⬇️
executor/linux/build.go 68.66% <0.00%> (-0.67%) ⬇️
runtime/kubernetes/container.go 74.83% <0.00%> (-3.10%) ⬇️
... and 3 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timhuynh94 timhuynh94 changed the title Feat (artifacts): artifacts key feat (artifacts): artifacts key Feb 12, 2026
@wass3r wass3r changed the title feat (artifacts): artifacts key feat(artifacts): add support for artifacts key Feb 16, 2026
//
//nolint:revive // ignore unexported type as it is intentional

func New(opts ...Opt) (*client, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func New returns unexported type *linux.client, which can be annoying to use (revive)

//
//nolint:revive // ignore unexported type as it is intentional

func New(opts ...Opt) (*client, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func New returns unexported type *local.client, which can be annoying to use (revive)

//
//nolint:revive // ignore unexported type as it is intentional

func New() (*mock, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func New returns unexported type *docker.mock, which can be annoying to use (revive)

//
//nolint:revive // ignore returning unexported client

func New(opts ...ClientOpt) (*client, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func New returns unexported type *docker.client, which can be annoying to use (revive)

//
//nolint:revive // ignore returning unexported client

func NewMock(opts ...ClientOpt) (*client, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func NewMock returns unexported type *docker.client, which can be annoying to use (revive)

//
//nolint:revive // ignore returning unexported client

func New(opts ...ClientOpt) (*client, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func New returns unexported type *kubernetes.client, which can be annoying to use (revive)

//
//nolint:revive // ignore returning unexported client

func NewMock(_pod *v1.Pod, opts ...ClientOpt) (*client, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
unexported-return: exported func NewMock returns unexported type *kubernetes.client, which can be annoying to use (revive)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants