exposing the kvstore (etcd) via ingress with TLS seems to not work

[jklare]

When installing voltha via the helm chart and exposing etcd via an ingress controller as documented here:
https://github.com/opencord/voltha-helm-charts/tree/master#using-an-ingress-controller
It seems that voltctl respects the "stacks.tls.useTLS: true" for the server connection, but not for the kvstore (etcd) connection.
Using ingresses like described in a pipeline like here:
https://jenkins.opencord.org/view/vip/job/build_berlin-community-pod-1-gpon_1T8GEM_voltha_DT_master_test/57/console
shows errors like the one below:

14:10:42  + voltctl log level list
14:10:47  ERROR: Unable to retrieve list of voltha components : context deadline exceeded 
14:10:47  Is ETCD available at voltha-infra.local:443?
14:10:48  + voltctl log level set WARN read-write-core#github.com/opencord/voltha-go/db/model
14:10:53  COMPONENTNAME      PACKAGENAME                               STATUS     ERROR
14:10:53  read-write-core    github.com/opencord/voltha-go/db/model    Failure    context deadline exceeded
14:10:53  + voltctl log level set WARN read-write-core#github.com/opencord/voltha-lib-go/v3/pkg/kafka
14:10:58  COMPONENTNAME      PACKAGENAME                                       STATUS     ERROR
14:10:58  read-write-core    github.com/opencord/voltha-lib-go/v3/pkg/kafka    Failure    context deadline exceeded
14:10:58  + voltctl log level set WARN adapter-open-olt#github.com/opencord/voltha-lib-go/v3/pkg/db
14:11:02  COMPONENTNAME       PACKAGENAME                                    STATUS     ERROR
14:11:02  adapter-open-olt    github.com/opencord/voltha-lib-go/v3/pkg/db    Failure    context deadline exceeded
14:11:02  + voltctl log level set WARN adapter-open-olt#github.com/opencord/voltha-lib-go/v3/pkg/probe
14:11:07  COMPONENTNAME       PACKAGENAME                                       STATUS     ERROR
14:11:07  adapter-open-olt    github.com/opencord/voltha-lib-go/v3/pkg/probe    Failure    context deadline exceeded
14:11:07  + voltctl log level set WARN adapter-open-olt#github.com/opencord/voltha-lib-go/v3/pkg/kafka
14:11:13  COMPONENTNAME       PACKAGENAME                                       STATUS     ERROR
14:11:13  adapter-open-olt    github.com/opencord/voltha-lib-go/v3/pkg/kafka    Failure    context deadline exceeded

Further debugging on the node, where this pipeline showed the errors, with etcdctl showed that quering the log level directly from etcd by using etcdctl is throwing an error like shown below:

jenkins@node-1:/tmp/etcd/etcd-v3.5.6-linux-amd64$ ./etcdctl --endpoints=voltha-infra.local:443 get "service/voltha/voltha_voltha/config/global/loglevel/default"
{"level":"warn","ts":"2023-10-02T14:23:55.829Z","logger":"etcd-client","caller":"v3@v3.5.6/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00024c000/voltha-infra.local:443","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection closed before server preface received"}
Error: context deadline exceeded

While using the same command and disabling insecure-transport (aka enabling TLS for transport and ignoring that those certs are self-signed) is showing the correct log level like below:

jenkins@node-1:/tmp/etcd/etcd-v3.5.6-linux-amd64$ ./etcdctl --insecure-transport=false --insecure-skip-tls-verify=true --endpoints=voltha-infra.local:443 get "service/voltha/voltha_voltha/config/global/loglevel/default"
service/voltha/voltha_voltha/config/global/loglevel/default
WARN

Grepping for the "UseTls" config flag in the current code base shows, that it is used in a function called NewConnection(), which seems to be used for implementing connections in most sub commands like "adapter, devices, flows, groups and logicaldevices", but not for "log", which is the sub command to contact etcd.
WIthout having enough knowledge to fix this issue or fully understand the current code base, it still looks to me like the internal/pkg/commands/log.go code needs to be revised to respect the "UseTls" config flag when establishing connections towards etcd (or a separate configuration flag needs to be added).

I hope there is enough debug information in this ticket. If now, please let me know how i can provide more and what exactly is needed.

[joey-onf]

From voltha-tst discussion in slack:

Amit Ghosh
I think it's a good catch and should be fixed. I do not see a reason why we should have a separate security configuration for log vs the other commands. @ALL any thoughts?

Mahir Gunyel
Agree. Should be same for log as well. On the other hand we are better move this to Jira.

[jklare]

I think this issue belongs here, since it is an issue directly related to the opencord/voltctl code in this repository. The README of this repo does not mention an issue tracking system outside of this repo (e.g. in JIRA) and therefore people running into this issue will most likely have a hard time finding any issues tracked outside of this repo. If we want to move this issue to JIRA, we should probably mention that we are not using github issues, but JIRA to track issues for this repo in the README.md.

[joey-onf]

fyi> Repositories are hosted on gerrit, github is simply a mirror:

• https://gerrit.opencord.org/plugins/gitiles/voltctl
• https://github.com/opencord/voltctl

[joey-onf]

Also bulk issues are already being tracked in jira:

• https://jira.opencord.org/secure/Dashboard.jspa

Jenkins is aware of how update tickets with job status when ticket numbers are mentioned in commit messages.

https://docs.voltha.org/master/overview/contributing.html

• Submitting code, commit mesage

https://docs.voltha.org/master/overview/contributing/jira_tickets.html?highlight=jira%20tickets