Threescale 6735 - Reconcile Secrets (replace PR 845)

[valerymo]

What

THREESCALE-6735

• This PR is based on originnal PR 845 ; The description - copied; validation section was updated. PR 845 was closed.
• Current PR is also fixes the Legacy issue with additionalPodAnnotations. It has been split to StagingAdditionalPodAnnotations and ProductionAdditionalPodAnnotations

How

• The APIManager referencing the secret will have labels with the UID of the secrets being referenced.
• The controller will watch secrets with the apimanager.apps.3scale.net/watched-by=apimanager label. The operator will get notified of the changes in the secret only when this label is present.
• Controller implements a custom mapper that:
  • Lists all the APIManager CRs with the secrets referencing the label
  • Creates a reconciliation request for each element in the list
• The deployment will be notified of the update to the secret through annotations in the pod template. The annotations in the pod template will specify the resource version of the secret so each time the secret is updated the resource version changes and it then updates the annotation in the pod template causing a redeployment of the pod. When the pod is redeployed it will read the updated content of the secret.

Verification

1. Run the following commands to install the 3scale operator

go mod vendor 
go mod tidy 
make install
export NAMESPACE=3scale-test
oc new-project $NAMESPACE
make download
make run

2. For the operator to create routes we need to mock the s3 credentials secret, run the following command.

kubectl apply -f - <<EOF
---
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: UkVQTEFDRV9NRQ==
  AWS_BUCKET: UkVQTEFDRV9NRQ==
  AWS_REGION: UkVQTEFDRV9NRQ==
  AWS_SECRET_ACCESS_KEY: UkVQTEFDRV9NRQ==
type: Opaque
EOF

3. Preparing for Policy secrets creation:

These are steps to prepare for policy secrets creation:

• Create apicast-policy.json file

{
  "$schema": "http://apicast.io/policy-v1/schema#manifest#",
  "name": "APIcast Example Policy",
  "summary": "This is just an example.",
  "description": "This policy is just an example how to write your custom policy.",
  "version": "0.1",
  "configuration": {
    "type": "object",
    "properties": { }
  }
}

• Create example.lua file

local setmetatable = setmetatable

local _M = require('apicast.policy').new('Example', '0.1')
local mt = { __index = _M }

function _M.new()
  return setmetatable({}, mt)
end

function _M:init()
  -- do work when nginx master process starts
end

function _M:init_worker()
  -- do work when nginx worker process is forked from master
end

function _M:rewrite()
  -- change the request before it reaches upstream
    ngx.req.set_header('X-CustomPolicy', 'customValue')
end

function _M:access()
  -- ability to deny the request before it is sent upstream
end

function _M:content()
  -- can create content instead of connecting to upstream
end

function _M:post_action()
  -- do something after the response was sent to the client
end

function _M:header_filter()
  -- can change response headers
end

function _M:body_filter()
  -- can read and change response body
  -- https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#body_filter_by_lua
end

function _M:log()
  -- can do extra logging
end

function _M:balancer()
  -- use for example require('resty.balancer.round_robin').call to do load balancing
end

return _M

• Create init.lua file

return require('example')

At this point we are ready to create customPolicy secrets

4. Create apicast customPolicy secrets

   oc create secret generic custom-policy1-secret \
  --from-file=./apicast-policy.json \
  --from-file=./init.lua \
  --from-file=./example.lua

 oc create secret generic custom-policy2-secret \
  --from-file=./apicast-policy.json \
  --from-file=./init.lua \
  --from-file=./example.lua

 oc create secret generic custom-policy3-secret \
  --from-file=./apicast-policy.json \
  --from-file=./init.lua \
  --from-file=./example.lua

5. Add this label to the secrets apimanager.apps.3scale.net/watched-by: apimanager

oc label secret custom-policy1-secret apimanager.apps.3scale.net/watched-by=apimanager
oc label secret custom-policy2-secret apimanager.apps.3scale.net/watched-by=apimanager
oc label secret custom-policy3-secret apimanager.apps.3scale.net/watched-by=apimanager

6. Deploy the APIManager CR

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: apps.vmogilev01.lpi0.s1.devshift.org
  apicast:
    stagingSpec:
      customPolicies:
        - name: custom-policy1
          version: "0.1"
          secretRef:
             name: custom-policy1-secret
    productionSpec:
      customPolicies:
        - name: custom-policy2
          version: "0.1"
          secretRef:
              name: custom-policy2-secret
        - name: custom-policy3
          version: "0.1"
          secretRef:
              name: custom-policy3-secret

7. Check APIManager , it should have a label with the UIDs of the three secrets

 $ oc get apimanager -o=jsonpath='{range .items[*]}{@.metadata.name}{"\n"}{@.metadata.labels}{"\n"}{end}'

example-apimanager
{"secret.apimanager.apps.3scale.net/a8617f96-7478-4652-afcb-3c8e709293a4":"true","secret.apimanager.apps.3scale.net/cc248ccd-d165-45e9-8389-77ec23a978c5":"true","secret.apimanager.apps.3scale.net/f581cccf-5cc4-4df1-a5c3-35d182ad8f15":"true"}

8. Check Deployment Configurations (DCs) Annotations and Pod Template's annotations
   check annotations in DCs and compare with CR confuguration:

• apicast-staging dc will have 1 annotation (+ pod template annotation)- for custom-policy1-secret
• apicast-production dc will have annotations (+ pod template annotations) - for custom-policy2-secret and custom-policy3-secret

Examples of DCs:

• apicast-production

$ oc describe dc apicast-production
Name:		apicast-production
....
Annotations:	apps.3scale.net/apicast-policy-volume-9a76abac4835ec9155836c1d6924b523=policy-01-custom-policy3
		apps.3scale.net/apicast-policy-volume-a8453ae999de5bd7ab810907d589c3d1=policy-01-custom-policy2
...
Pod Template:
  .....
  Annotations:		apimanager.apps.3scale.net/custompolicy-secret-resource-version-custom-policy2-secret: 349550
			apimanager.apps.3scale.net/custompolicy-secret-resource-version-custom-policy3-secret: 349572

• apicast-staging

$ oc describe dc apicast-staging
Name:		apicast-staging
....
Annotations:	apps.3scale.net/apicast-policy-volume-758608052572255606d48d206a59e240=policy-01-custom-policy1
.....
Pod Template:
  .....
  Annotations:		apimanager.apps.3scale.net/custompolicy-secret-resource-version-custom-policy1-secret: 349544
			apps.3scale.net/env-configmap-hash: 1046001186

• The following commands also could be used to check annotations of DCs:

oc get dc apicast-production -o=jsonpath='{@.metadata.annotations}'
oc get dc apicast-production -o=jsonpath='{@.spec.template.metadata.annotations}'
oc get dc apicast-staging -o=jsonpath='{@.metadata.annotations}'
oc get dc apicast-staging -o=jsonpath='{@.spec.template.metadata.annotations}'

10. Add new labels to secrets

• check revision of DCs:

$ oc get dc |grep apicast
apicast-production   1          1         1         config,image(amp-apicast:2.14)
apicast-staging      1          1         1         config,image(amp-apicast:2.14)

• add label to secret custom-policy1-secret used for apicast-staging

oc label secret custom-policy1-secret newlabel=newlabel   

oc get deploymentconfig apicast-production -o=jsonpath='{@.spec.template.metadata.annotations}'

• check that apicast-staging DC has new Revision and apicast-production has not changed:

$ oc get dc |grep apicast
apicast-production   1          1         1         config,image(amp-apicast:2.14)
apicast-staging      2          1         1         config,image(amp-apicast:2.14)

• add label to one of secrets used for apicast-production.
  Expected: apicast-production version has increased, but apicast-staging version has not changed.

$ oc get dc |grep apicast
apicast-production   2          1         1         config,image(amp-apicast:2.14)
apicast-staging      2          1         1         config,image(amp-apicast:2.14)

Notes In this test we could see that DCs of apicast-production and apicast-staging updated independently. It's working this way if they are using different custom policies/secrets.
If apicast-production and apicast-staging will have same policies / same secrets, as in example below -
both DCs will be updated in case one of secrets changed.

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: apps.vmogilev01.41x3.s1.devshift.org
  apicast:
    stagingSpec:
      customPolicies:
        - name: custom-policy1
          version: "0.1"
          secretRef:
             name: custom-policy1-secret
        - name: custom-policy2
          version: "0.1"
          secretRef:
              name: custom-policy2-secret
    productionSpec:
      customPolicies:
        - name: custom-policy1
          version: "0.1"
          secretRef:
             name: custom-policy1-secret
        - name: custom-policy2
          version: "0.1"
          secretRef:
              name: custom-policy2-secret

[valerymo]

Hi @MStokluska , @austincunningham, @eguzki . This PR is based on PR of Patryk - #845 and fixes issues with DCs update. additionalPodAnnotations has been split to StagingAdditionalPodAnnotations and ProductionAdditionalPodAnnotations. Validation notes - updated

[valerymo]

attaching more test logs, adding more secrets and updating CR.
testLogs.txt

[austincunningham]

Needs a rebase

[valerymo]

Needs a rebase

@austincunningham
rebase done. Test done after rebase. Thanks for review

[qlty-cloud-legacy]

Code Climate has analyzed commit 9605838 and detected 9 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Duplication	6
Style	3

View more on Code Climate.

[MStokluska]

Looks good to me, once e2e passes it's ready to merge, good job!
/lgtm

[austincunningham]

/approve