Auto-push Kubeflow deployments corresponding to different Kubeflow versions.

[jlewi]

It would be nice if we could auto-push dev.kubeflow.org or perhaps create another environment nightly-dev.kubeflow.org.

There are certain changes like UI that are difficult to test manually. It would be useful if we had an up todate environment.

I think we could easily adapt our E2E tests to do this and then just run it as a cron job.

[jlewi]

It might be interesting to look at using Weave Flux
https://www.weave.works/blog/gitops-operations-by-pull-request

Weave Flux supports keeping K8s infrastructure in sync with config defined in a git repository.

Not sure if ksonnet is supported yet.

[jlewi]

Here's an idea for how to make progress

1. We could use cron job to regularly run recreate_app.sh
2. Create a PR using hub CLI for GitHub
3. Manually approve the PR

Either use argo-cd or a cron job to call redeploy_app.sh to keep the deployed instance in sync with what's checked in.

[jlewi]

kubeflow/kubeflow#1100

Added a script for upgrading the ksonnet application.

[jlewi]

We'd like to maintain a pool of Kubeflow deployments corresponding to different Kubeflow versions.

e.g

kf-vX.Y-n00
...
kf-vX.Y-n05

The reason for having a pool of deployments for a given major release (X.Y) is that we want to run tests against these clusters. We don't want to interrupt the tests when deploying an updated version. So by maintaining a pool and cycling through them we can create a new version for new tests while letting already running tests run to completion.

We will want to recycle the names because we want to have a fixed set of endpoints e.g

https://kf-v0-4-n00.endpoints.kubeflow-ci.cloud.goog/

The reason for having a fixed set of endpoints is that we have to manually set the redirect URIs on the OAuth credentials used for IAP. So by recycling the endpoints we don't have to manually update the OAuth credential with new redirect URIs.

#269 created a python script to deploy Kubeflow.

The next steps would be

• Update the script in A script to automate creation of Kubeflow clusters for testing. #269 to add logic to recycle the deployments
  * Check if there is an unused name kf-vX.Y-n00,...kf-vX.Y-n05 and if there is use it
  * Otherwise delete the oldest deployment and create a new one using its name.

• Create a cron job to run the above script and redeploy using the latest commit for each release

• Create a simple flask app to provide information about all the different deployments
  * E.g. so we can see when each endpoint was last updated

• Recycle SSL certificates for each endpoint

  • We will hit quota issues with LetsEncrypt if we can minting SSL certs each time we reuse an existing DNS name
  • https://github.com/kubeflow/kubeflow/blob/master/testing/test_deploy_app.py has code for
    storing certs in GCS and reusing them.

[jlewi]

Boskos might be useful for managing clusters.