Feature: reuse e2e tests on productized code#237
Feature: reuse e2e tests on productized code#237cardil wants to merge 4 commits intoknative:masterfrom
Conversation
Also adding some extra OWNER files and test-infra/devstats from vendor directory that shouldn't be commited
This commit let change test flags from outside. It can be done either by: * declaring environmental variables: `TEST_NAMESPACE`, and `TEST_RESOURCE`, * using `-ldflags "-X knative.dev/serving-operator/test.ServingOperatorNamespace=knative-serving"` Changing those flags is needed to perform tests on RH operators.
knative-prow-robot
left a comment
knative-prow-robot
left a comment
There was a problem hiding this comment.
@cardil: 8 warnings.
Details
In response to this:
Proposed Changes
- Rely on PR #223
- Expose e2e test as a compliance suite (3f6aac3)
- Add a test context to fine tune compliance suite testing (72c0d7f)
Proposed changes details
Expose e2e test as a compliance suite (3f6aac3)
As a knative vendor we would like to execute tests from upstream knative in out productized code's test plans.
To do that, I propose this minor change that would result in type safe way of reusing upstream test code, by vendoring it in our code. It involves moving e2e test to regular
.gofile, instead of_test.gofile. In that way we can call that code from our test suite.Add a test context to fine tune compliance suite testing (72c0d7f)
Sometimes upstream tests could not pass yet. That's natural as vendors will always be behind upstream code. In that case we need a mechanism that will let us skip parts of the tests.
That was implemented in
test.Contextobject.Example usage
There is already created a PR openshift-knative/serverless-operator#32 that uses this concept. Below I highlighted a key test file from that PR (note that we are using 0.9 version of this change):
package e2e import ( "testing" "github.com/openshift-knative/serverless-operator/test" upstreame2e "knative.dev/serving-operator/test/e2e" upstreamtest "knative.dev/serving-operator/test" ) func TestUpstreamKnativeServingOperator(t *testing.T) { upstreamtest.ServingOperatorNamespace = "knative-serving" suite := upstreame2e.ComplianceSuite() ctx := test.SetupClusterAdmin(t) defer test.CleanupAll(ctx) test.CleanupOnInterrupt(t, func() { test.CleanupAll(ctx) }) createSubscriptionAndWaitForCSVtoSucceed(t, ctx) // <1> skipConfigureSubTest := upstreamtest.Skip( "TestKnativeServingDeployment/configure", "Skip due to SRVKS-241") // <2> upstreamtest. NewContext(t). WithOverride(skipConfigureSubTest). RunSuite(suite) // <3> undeployServerlessOperatorAndCheckDependentOperatorsRemoved(t, ctx) // <4> }
<1>setup of our Serverless operator<2>registering a skip forTestKnativeServingDeployment/configurethat we are not supporting at this point<3>actual compliance suite execution<4>teardown of our Serverless operatorRelease Note
NONE
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
| */ | ||
|
|
||
| // This is a dummy buildable source file to satisfy `dep` | ||
| package knative_serving |
There was a problem hiding this comment.
Golint naming: don't use an underscore in package name. More info.
|
Hi @cardil. Thanks for your PR. I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: cardil The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
As a knative vendor we would like to execute tests from upstream knative in out productized code's test plans. To do that, I popose this minor change that would result in type safe way of reusing upstream test code, by vendoring it in out code.
Sometimes upstream tests could not pass yet. In that case we need a mechanism that will let us skip parts of the tests. That was implemented in test.Context object
cardil
force-pushed
the
feature/reuse-tests-on-productized
branch
from
72c0d7f to
5d95dd6
Compare
|
I do think this concept spans wide enough to be proposed and discussed in the productivity WG. It only really makes sense if something like this is adopted or at least acknowledged across the entirety of Knative. |
|
What's also not entirely clear to me yet: Why is this better than just using |
|
To further clarify (based on a Slack discussion), referencing the bullets in your example.
|
|
@markusthoemmes proposing another way of reusing tests. That's to create a bash script that create test context and execute tests. My approach is different. Writing everything as a Go test will be better of ways:
The last point is biggest for me. That's what I would like to see. To be able to run everything just from IDE. Just as you run Spring Test or Arquillian! If you think of e2e testing, that might be written in every language that you want. Any! That's that it simulates user operations on live system. But, writing those in the same language as production code and unit tests code is beneficial. You can use fewer languages to do the job. You are more proficient with language that you're writing the production code in and can use the same tools to do linting, static analysis. |
You can't exclude tests with |
|
/cc @chaodaiG |
|
Related: knative/serving#4263 |
|
Is this still being worked on? |
|
Possibly related: knative/serving#4263 |
|
@cardil: PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
This is PoC of solution to be discussed. |
|
@cardil how should we proceed with this one? anything I can do to help? |
|
It should be discussed on TOC meeting. It was already pitched there. |
|
Close this PR, because this repo is deprecated. |
11 participants
Proposed Changes
Proposed changes details
Expose e2e test as a compliance suite (dbfbf2c)
As a knative vendor we would like to execute tests from upstream knative in out productized code's test plans.
To do that, I propose this minor change that would result in type safe way of reusing upstream test code, by vendoring it in our code. It involves moving e2e test to regular
.gofile, instead of_test.gofile. In that way we can call that code from our test suite.Add a test context to fine tune compliance suite testing (5d95dd6)
Sometimes upstream tests could not pass yet. That's natural as vendors will always be behind upstream code. In that case we need a mechanism that will let us skip parts of the tests.
That was implemented in
test.Contextobject.Example usage
There is already created a PR openshift-knative/serverless-operator#32 that uses this concept. Below I highlighted a key test file from that PR (note that we are using 0.9 version of this change):
<1>setup of our Serverless operator<2>registering a skip forTestKnativeServingDeployment/configurethat we are not supporting at this point<3>actual compliance suite execution<4>teardown of our Serverless operatorRelease Note