THREESCALE-9564 redis 6 #839

eguzki · 2023-07-07T14:29:00Z

What

Support Redis 6 for internal databases. System and Backend redis are upgraded to Redis 6

Fixes https://issues.redhat.com/browse/THREESCALE-9564

The upgrade should only happen when redis is used as internal database, either for backend or system.

Migrating from 5.0 to 6.0
=========================

Redis 6.0 is mostly a strict superset of 5.0, you should not have any problem
upgrading your application from 5.0 to 6.0. However this is a list of small
non-backward compatible changes introduced in the 6.0 release:

* The SPOP <count> command no longer returns null when the set key does not
  exist. Now it returns the empty set as it should and as happens when it is
  called with a 0 argument. This is technically a fix, however it changes the
  old behavior.

Command level compatibility should not be an issue for 3scale, as there is support for Redis 6 since 3scale 2.11.

Verification Steps: Fresh install

Requires openshift (oc CLI) session
Create new project

oc new-project 3scale-redis-6

Checkout this PR's branch and run the operator

make install
make run

Deploy 3scale
- no need to specify S3 credentials
- wildcardDomain must be something valid (only if it is going to be used later). Usually ${PROJECT_NAME}.apps.${CLUSTER_DOMAIN}

k apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: aws-auth
stringData:
  AWS_ACCESS_KEY_ID: testID
  AWS_SECRET_ACCESS_KEY: testkey
  AWS_BUCKET: testbucket
  AWS_REGION: us-east-1
type: Opaque
EOF

k apply -f - <<EOF
---
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: apimanager1
spec:
  wildcardDomain: example.net
  resourceRequirementsEnabled: false
  system:
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: aws-auth
EOF

wait for deployment to be ready

oc wait --for=condition=available apimanager/apimanager1 --timeout=-1s
apimanager.apps.3scale.net/apimanager1 condition met

The above "wait" ensures the pods are up and running.

Check the backend redis imagestream for the redis image based on quay.io/centos7/redis-6-centos7

❯ k get is backend-redis -o jsonpath='{.status.tags}' | yq e -P
- items:
    - created: "2023-07-07T14:44:09Z"
      dockerImageReference: quay.io/centos7/redis-6-centos7@sha256:b549dc0714dd1307e2b6be9cc28adceac4d2d1b6bf032898d88f425b7c8939d0
      generation: 2
      image: sha256:b549dc0714dd1307e2b6be9cc28adceac4d2d1b6bf032898d88f425b7c8939d0
  tag: "2.14"

Use 3scale's default product, default application to run a request against 3scale echo api. The request should be validated by backend. Backend listener logs should show the following line

 ❯ k logs $(oc get pods --selector deploymentconfig=backend-listener -o name)
.....
0.129.0.160 - - [07/Jul/2023 14:03:19 UTC] "GET /transactions/authrep.xml?service_token=a7e833147f4e834d007f1f44ef72b45602d246d06ef414c8641c7eb0770bc51d&service_id=2&usage%5Bhits%5D=1&user_key=f088e04d70d4434688208c125b1cf3ff HTTP/1.1" 200 - 0.031655315 0 0 0 12 19 7 - "rejection_reason_header=1&limit_headers=1&no_body=1"
....

Verification Steps: Upgrade from 2.13

Requires openshift (oc CLI) session
Create new project

oc new-project 3scale-redis-6-upgrade

Install upstream (community) 3scale Operator with a subscription to the 2.13 channel
Deploy 3scale
- no need to specify S3 credentials
- wildcardDomain must be something valid (only if it is going to be used later). Usually ${PROJECT_NAME}.apps.${CLUSTER_DOMAIN}

k apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: aws-auth
stringData:
  AWS_ACCESS_KEY_ID: testID
  AWS_SECRET_ACCESS_KEY: testkey
  AWS_BUCKET: testbucket
  AWS_REGION: us-east-1
type: Opaque
EOF

k apply -f - <<EOF
---
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: apimanager1
spec:
  wildcardDomain: example.net
  resourceRequirementsEnabled: false
  system:
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: aws-auth
EOF

wait for deployment to be ready

oc wait --for=condition=available apimanager/apimanager1 --timeout=-1s
apimanager.apps.3scale.net/apimanager1 condition met

The above "wait" ensures the pods are up and running.

Check the backend redis imagestream for the redis image based on centos/redis-5-centos7

❯ k get is backend-redis -o jsonpath='{.status.tags}' | yq e -P
- items:
    - created: "2023-07-07T14:57:12Z"
      dockerImageReference: centos/redis-5-centos7@sha256:7e12e362a6430f8831a445b26181285c3d36df29b1fdd2abe1247d5068905129
      generation: 2
      image: sha256:7e12e362a6430f8831a445b26181285c3d36df29b1fdd2abe1247d5068905129
  tag: "2.13"

Use 3scale's default product, default application to run a request against 3scale echo api.
Remove the running operator 2.13 (remove subscription or from "Installed Operators" in the catalog UI.
Run Upgrade: Checkout this PR's branch and run the operator

make install
make run

During upgrade, some pods my restart and report error. That is expected as redis is being replaced and living pods will see connections being dropped.

Wait until all pods are up&running. oc wait does not work during upgrade because all the deploymentconfigs will report as available (they have at least one available pod up&running) during the upgrade.

Check new redis pod runs image based on quay.io/centos7/redis-6-centos7

❯ k get $(oc get pods --selector deploymentconfig=backend-redis -o name) -o jsonpath='{.spec.containers[0].image}' | yq e -P
quay.io/centos7/redis-6-centos7@sha256:b549dc0714dd1307e2b6be9cc28adceac4d2d1b6bf032898d88f425b7c8939d0

Check redis new pod logs

❯ k logs $(oc get pods --selector deploymentconfig=backend-redis -o name) 
---> 15:13:07     Processing Redis configuration files ...
---> 15:13:07     WARNING: setting REDIS_PASSWORD is recommended
---> 15:13:07     Sourcing post-init.sh ...
---> 15:13:07     Cleaning up environment variable REDIS_PASSWORD ...
---> 15:13:07     Running final exec -- Only Redis logs after this point
1:C 07 Jul 2023 15:13:07.403 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Jul 2023 15:13:07.403 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Jul 2023 15:13:07.403 # Configuration loaded
1:M 07 Jul 2023 15:13:07.405 * Running mode=standalone, port=6379.
1:M 07 Jul 2023 15:13:07.405 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 07 Jul 2023 15:13:07.405 # Server initialized
1:M 07 Jul 2023 15:13:07.405 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
1:M 07 Jul 2023 15:13:07.422 * DB loaded from append only file: 0.016 seconds
1:M 07 Jul 2023 15:13:07.422 * Ready to accept connections

Note on the line "DB loaded from append only file: 0.016 seconds". It seems that redis just loads the append only file as if it was generated by another redis 6 instance. Seems that the format has not change since redis 5, hence updating data file is not even needed.

Run the same request it was done before the upgrade. The request is returning 200OK and 3scale backend is validating the request.

 ❯ k logs $(oc get pods --selector deploymentconfig=backend-listener -o name)
.....
0.129.0.160 - - [07/Jul/2023 14:03:19 UTC] "GET /transactions/authrep.xml?service_token=a7e833147f4e834d007f1f44ef72b45602d246d06ef414c8641c7eb0770bc51d&service_id=2&usage%5Bhits%5D=1&user_key=f088e04d70d4434688208c125b1cf3ff HTTP/1.1" 200 - 0.031655315 0 0 0 12 19 7 - "rejection_reason_header=1&limit_headers=1&no_body=1"
....

openshift-ci · 2023-07-07T14:29:05Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

austincunningham · 2023-07-19T11:21:31Z

Verification

Install

check out this branch
create namespace 3scale-redis-6 and create s3 secret and apimanger cr
Wait for install to complete

oc wait --for=condition=available apimanager/apimanager1 --timeout=-1s
apimanager.apps.3scale.net/apimanager1 condition met

check the backend redis after the install

oc get is backend-redis -o jsonpath='{.status.tags}' | yq e -P
- items:
    - created: "2023-07-19T10:20:11Z"
      dockerImageReference: quay.io/centos7/redis-6-centos7@sha256:57c885ef27fac78209e23828e2ab30558518dc543a99149e12dbd8745568cbda
      generation: 2
      image: sha256:57c885ef27fac78209e23828e2ab30558518dc543a99149e12dbd8745568cbda
  tag: "2.14"

login to 3scale-admin route and run through the wizard
check for the transactions in the backend-listener

oc logs $(oc get pods --selector deploymentconfig=backend-listener -o name) | grep /transactions
10.131.2.47 - - [19/Jul/2023 10:27:35 UTC] "GET /transactions/authrep.xml?service_token=73c09af519e51a45c2c941913d1b630d694aa44c8a4a762c7ed409a0c36afae9&service_id=2&usage%5Bhits%5D=1&user_key=64399c2b3b48c3a0f15238264c72bc3f HTTP/1.1" 200 - 0.009157371 0 0 0 12 19 7 - "rejection_reason_header=1&limit_headers=1&no_body=1"

Upgrade

create namespace 3scale-redis-6-upgrade and create s3 secret and apimanger cr
install the 3scale-operator from the community edition on 2.13
Wait for install to complete

oc wait --for=condition=available apimanager/apimanager1 --timeout=-1s
apimanager.apps.3scale.net/apimanager1 condition met

check the redis version in backend-redis

oc get is backend-redis -o jsonpath='{.status.tags}' | yq e -P
- items:
    - created: "2023-07-19T10:45:58Z"
      dockerImageReference: centos/redis-5-centos7@sha256:7e12e362a6430f8831a445b26181285c3d36df29b1fdd2abe1247d5068905129
      generation: 2
      image: sha256:7e12e362a6430f8831a445b26181285c3d36df29b1fdd2abe1247d5068905129
  tag: "2.13"

log onto 3scale-admin route and run through the wizard
Check the backend-listener logs for /transactions

oc logs $(oc get pods --selector deploymentconfig=backend-listener -o name) | grep /transactions
10.131.2.47 - - [19/Jul/2023 10:27:35 UTC] "GET /transactions/authrep.xml?service_token=73c09af519e51a45c2c941913d1b630d694aa44c8a4a762c7ed409a0c36afae9&service_id=2&usage%5Bhits%5D=1&user_key=64399c2b3b48c3a0f15238264c72bc3f HTTP/1.1" 200 - 0.009157371 0 0 0 12 19 7 - "rejection_reason_header=1&limit_headers=1&no_body=1"

removed the operator
Install this branch with make install and make run
Wait for the install to complete
check the redis image in backend-redis

oc get $(oc get pods --selector deploymentconfig=backend-redis -o name) -o jsonpath='{.spec.containers[0].image}' | yq e -P
quay.io/centos7/redis-6-centos7@sha256:57c885ef27fac78209e23828e2ab30558518dc543a99149e12dbd8745568cbda

Check the redis logs

oc logs $(oc get pods --selector deploymentconfig=backend-redis -o name) 
---> 10:59:11     Processing Redis configuration files ...
---> 10:59:11     WARNING: setting REDIS_PASSWORD is recommended
---> 10:59:11     Sourcing post-init.sh ...
---> 10:59:11     Cleaning up environment variable REDIS_PASSWORD ...
---> 10:59:11     Running final exec -- Only Redis logs after this point
1:C 19 Jul 2023 10:59:11.956 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 19 Jul 2023 10:59:11.956 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 19 Jul 2023 10:59:11.956 # Configuration loaded
1:M 19 Jul 2023 10:59:11.957 * Running mode=standalone, port=6379.
1:M 19 Jul 2023 10:59:11.957 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 19 Jul 2023 10:59:11.957 # Server initialized
1:M 19 Jul 2023 10:59:11.957 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
1:M 19 Jul 2023 10:59:11.958 * DB loaded from append only file: 0.001 seconds
1:M 19 Jul 2023 10:59:11.958 * Ready to accept connections

check the backend-listner logs for `/transactions

oc logs $(oc get pods --selector deploymentconfig=backend-listener -o name)| grep /transaction
# This returned nothing guessing its because the pod restated, so I curl the endpoint and looks like that worked
curl "https://api-3scale-apicast-staging.apps.aucunnin-ccs.edy6.s1.devshift.org:443/?user_key=0b176e8c876412ca46259e155c2d9c10"
{
  "method": "GET",
  "path": "/",
  "args": "user_key=0b176e8c876412ca46259e155c2d9c10",
  "body": "",
  "headers": {
    "HTTP_VERSION": "HTTP/1.1",
    "HTTP_HOST": "echo-api.3scale.net",
    "HTTP_X_REAL_IP": "10.128.6.35",
    "HTTP_X_3SCALE_PROXY_SECRET_TOKEN": "Shared_secret_sent_from_proxy_to_API_backend_a26851cbd2a98244",
    "HTTP_USER_AGENT": "curl/7.85.0",
    "HTTP_ACCEPT": "*/*",
    "HTTP_X_FORWARDED_HOST": "api-3scale-apicast-staging.apps.aucunnin-ccs.edy6.s1.devshift.org",
    "HTTP_X_FORWARDED_PORT": "443",
    "HTTP_X_FORWARDED_PROTO": "https",
    "HTTP_FORWARDED": "for=185.167.180.230;host=api-3scale-apicast-staging.apps.aucunnin-ccs.edy6.s1.devshift.org;proto=https",
    "HTTP_X_FORWARDED_FOR": "185.167.180.230,13.39.92.158",
    "HTTP_X_ENVOY_EXTERNAL_ADDRESS": "13.39.92.158",
    "HTTP_X_REQUEST_ID": "af2938c4-81bb-4a8a-964f-f4fe4fe85e4c",
    "HTTP_X_ENVOY_EXPECTED_RQ_TIMEOUT_MS": "15000"
  },
  "uuid": "db49efdc-91e1-4352-8449-b3c4aad7ea9e"
}%                                                                                                                                                                                                                                

oc logs $(oc get pods --selector deploymentconfig=backend-listener -o name)| grep /transaction                                 
10.131.2.57 - - [19/Jul/2023 11:04:17 UTC] "GET /transactions/authrep.xml?service_id=2&service_token=cfc1b75591f9d69caeafde7d5093c5c2d35e7f839d893320970ce6a880587ab2&usage%5Bhits%5D=1&user_key=0b176e8c876412ca46259e155

/lgtm

qlty-cloud-legacy · 2023-07-20T14:29:55Z

Code Climate has analyzed commit 8e66442 and detected 4 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Duplication	4

View more on Code Climate.

openshift-ci bot added the do-not-merge/work-in-progress label Jul 7, 2023

eguzki marked this pull request as ready for review July 7, 2023 15:17

eguzki requested a review from a team as a code owner July 7, 2023 15:17

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 7, 2023

eguzki force-pushed the redis-6 branch from e4201ce to d1abeae Compare July 13, 2023 09:32

austincunningham approved these changes Jul 19, 2023

View reviewed changes

austincunningham added the verified label Jul 19, 2023

eguzki added 3 commits July 20, 2023 16:28

THREESCALE-9564 redis 6

dabcb9e

bundle update

d3c1d75

little fix

8e66442

eguzki force-pushed the redis-6 branch from d1abeae to 8e66442 Compare July 20, 2023 14:28

eguzki merged commit 2acedc8 into master Jul 20, 2023

eguzki deleted the redis-6 branch July 20, 2023 15:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

THREESCALE-9564 redis 6 #839

THREESCALE-9564 redis 6 #839

Uh oh!

eguzki commented Jul 7, 2023 •

edited

Loading

Uh oh!

openshift-ci bot commented Jul 7, 2023

Uh oh!

austincunningham commented Jul 19, 2023 •

edited

Loading

Uh oh!

qlty-cloud-legacy bot commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

THREESCALE-9564 redis 6 #839

THREESCALE-9564 redis 6 #839

Uh oh!

Conversation

eguzki commented Jul 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Verification Steps: Fresh install

Verification Steps: Upgrade from 2.13

Uh oh!

openshift-ci bot commented Jul 7, 2023

Uh oh!

austincunningham commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification

Install

Upgrade

Uh oh!

qlty-cloud-legacy bot commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eguzki commented Jul 7, 2023 •

edited

Loading

austincunningham commented Jul 19, 2023 •

edited

Loading