Error creating rollback entry - Resource SecObjectProtection not found

[ksyblast]

hello!

Urgent help needed!

OS: Oracle Linux 8
Kernel: UEK 5.15.0-308.179.6.el8uek.x86_64
piraeus-operator: v2.8.0
velero: v1.15.2

Got an error during backup of ANY volume:

time="2025-05-29T15:24:24Z" level=warning msg="VolumeSnapshotContent snapcontent-5fefd56c-f43b-45cf-8998-848f9b119661 has error: Failed to check and update snapshot content: failed to take snapshot of the volume pvc-e60b0f01-dfe4-4237-b4aa-b59aa70828b6: \"rpc error: code = Internal desc = failed to delete temporary snapshot ID: Message: 'Exception thrown.'; Details: 'com.linbit.linstor.transaction.TransactionException: Error creating rollback entry'; Reports: '[6822F3BD-00000-001896]'\"" backup=velero/ksy-20250529-02 cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:744" pluginName=velero

ERROR REPORT 6822F3BD-00000-001896

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Controller
Version:                            1.30.4
Build ID:                           bef74a44609cb592c5efad2e707b50e696623c61
Build time:                         2025-02-03T15:48:28+00:00
Error time:                         2025-05-29 15:23:46
Node:                               linstor-controller-5dd47c5b84-7rftt
Thread:                             grizzly-http-server-8

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiTransactionException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiTransactionException
Generated at:                       Method 'commit', Source file 'CtrlTransactionHelper.java', Line #30

Error message:                      com.linbit.linstor.transaction.TransactionException: Error creating rollback entry

Call backtrace:

    Method                                   Native Class:Line number
    commit                                   N      com.linbit.linstor.core.apicallhandler.controller.CtrlTransactionHelper:30
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlKvsApiCallHandler:182
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:1139
    lambda$modifyKVStore$2                   N      com.linbit.linstor.api.rest.v1.KeyValueStore:104
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:235
    modifyKVStore                            N      com.linbit.linstor.api.rest.v1.KeyValueStore:97
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor204:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:569
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:840

Caused by:
==========

Category:                           RuntimeException
Class name:                         TransactionException
Class canonical name:               com.linbit.linstor.transaction.TransactionException
Generated at:                       Method 'commit', Source file 'ControllerK8sCrdTransactionMgr.java', Line #157

Error message:                      Error creating rollback entry

ErrorContext: 

Call backtrace:

    Method                                   Native Class:Line number
    commit                                   N      com.linbit.linstor.transaction.ControllerK8sCrdTransactionMgr:157
    commit                                   N      com.linbit.linstor.core.apicallhandler.controller.CtrlTransactionHelper:26
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlKvsApiCallHandler:182
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:1139
    lambda$modifyKVStore$2                   N      com.linbit.linstor.api.rest.v1.KeyValueStore:104
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:235
    modifyKVStore                            N      com.linbit.linstor.api.rest.v1.KeyValueStore:97
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor204:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:569
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:840

Caused by:
==========

Category:                           LinStorException
Class name:                         DatabaseException
Class canonical name:               com.linbit.linstor.dbdrivers.DatabaseException
Generated at:                       Method 'createRollbackEntry', Source file 'ControllerK8sCrdRollbackMgr.java', Line #84

Error message:                      Resource SecObjectProtection/b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686 not found

ErrorContext:


Call backtrace:

    Method                                   Native Class:Line number
    createRollbackEntry                      N      com.linbit.linstor.transaction.ControllerK8sCrdRollbackMgr:84
    commit                                   N      com.linbit.linstor.transaction.ControllerK8sCrdTransactionMgr:152
    commit                                   N      com.linbit.linstor.core.apicallhandler.controller.CtrlTransactionHelper:26
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlKvsApiCallHandler:182
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:1139
    lambda$modifyKVStore$2                   N      com.linbit.linstor.api.rest.v1.KeyValueStore:104
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:235
    modifyKVStore                            N      com.linbit.linstor.api.rest.v1.KeyValueStore:97
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor204:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:569
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:840


END OF ERROR REPORT.

I have checked here it's recommended to create the problematic resource, but with which contents should I do it?
I don't find any mentions of this resource in etcd and it's triggered for ANY volume we are trying to backup with the same SecObjectProtection/b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686.

[ghernadi]

Hello, sorry for the delayed response.

Try this content:

apiVersion: "internal.linstor.linbit.com/v1-27-1"
kind: "SecObjectProtection"
metadata:
  name: "b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686"
spec:
  object_path: "/keyvaluestores/CSI-BACKUP-MAPPING"
  creator_identity_name: "PUBLIC"
  owner_role_name: "PUBLIC"
  security_type_name: "PUBLIC"

If there are also other entries missing, I'd like to ask you to send a k8s dump to my email so I can see if I can fix this for you.

[ksyblast]

Hello and thanks for the answer!
Unfortunately the suggested fix did not help. I have created the object but still get the same error

time="2025-06-02T14:31:23Z" level=info msg="Created VolumeSnapshot namespace/velero-test-pvc-xfs-wgpb6" backup=velero/ksy-20250602-02 cmd=/velero logSource="pkg/backup/actions/csi/pvc_action.go:187" pluginName=velero
time="2025-06-02T14:31:23Z" level=info msg="Waiting for CSI driver to reconcile volumesnapshot namespace/velero-test-pvc-xfs-wgpb6. Retrying in 5s" backup=velero/ksy-20250602-02 cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:712" pluginName=velero
time="2025-06-02T14:31:28Z" level=info msg="Waiting for VolumeSnapshotContents snapcontent-fe1ccd23-40fa-46f9-91ea-888955e6f96d to have snapshot handle. Retrying in 5s" backup=velero/ksy-20250602-02 cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:739" pluginName=velero
time="2025-06-02T14:31:28Z" level=warning msg="VolumeSnapshotContent snapcontent-fe1ccd23-40fa-46f9-91ea-888955e6f96d has error: Failed to check and update snapshot content: failed to take snapshot of the volume pvc-e60b0f01-dfe4-4237-b4aa-b59aa70828b6: \"rpc error: code = Internal desc = failed to delete temporary snapshot ID: Message: 'Exception thrown.'; Details: 'com.linbit.linstor.transaction.TransactionException: Error creating rollback entry'; Reports: '[6822F3BD-00000-002865]'\"" backup=velero/ksy-20250602-02 cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:744" pluginName=velero

 kubectl -n piraeus-datastore exec deploy/linstor-controller -- linstor error-reports show 6822F3BD-00000-002865
...
Category:                           LinStorException
Class name:                         DatabaseException
Class canonical name:               com.linbit.linstor.dbdrivers.DatabaseException
Generated at:                       Method 'createRollbackEntry', Source file 'ControllerK8sCrdRollbackMgr.java', Line #84

Error message:                      Resource SecObjectProtection/b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686 not found

ErrorContext:

...

kubectl get secobjectprotection.internal.linstor.linbit.com/b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686 -o yaml 
apiVersion: internal.linstor.linbit.com/v1-15-0
kind: SecObjectProtection
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"internal.linstor.linbit.com/v1-15-0","kind":"SecObjectProtection","metadata":{"annotations":{},"name":"b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686"},"spec":{"creator_identity_name":"PUBLIC","object_path":"/keyvaluestores/CSI-BACKUP-MAPPING","owner_role_name":"PUBLIC","security_type_name":"PUBLIC"}}
  creationTimestamp: "2025-06-02T14:31:01Z"
  generation: 1
  name: b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686
  resourceVersion: "2065499381"
  uid: 5f038f8b-e159-447c-8151-db3c8aa56bd0
spec:
  creator_identity_name: PUBLIC
  object_path: /keyvaluestores/CSI-BACKUP-MAPPING
  owner_role_name: PUBLIC
  security_type_name: PUBLIC

[ghernadi]

Sorry, I forgot that LINSTOR is internally caching a few things. That would at least explain the discrepancy between LINSTOR'S "not found" vs kubectl get ... "exists" status.

However, restarting the controller could lead to the problem that if the database is not consistent right now, the controller might not start properly. So either try to clone your k8s database to some playground project and try to start the controller there, or just send me a dump of your k8s database to my email address so I can verify the database consistency.

If you want to try it yourself, please make sure that the dummy controller cannot reach the actual satellites to avoid unnecessary reconnects and OFFLINE(Other controller) status messages from the actual controller.

If it turns out that there are other issues in the database, I guess you will need to send me the database anyways, since it can be quite cumbersome to figure out what is missing.

[ksyblast]

Yes I'm also afraid to restart the controller. We will discuss it in our team and I will try what you suggested or send you the dump later on, thank you again for your help.

[ksyblast]

Hi!
The database was OK so we restarted the controller, the error disappeared but now there's a new error:

2025-06-05 05:55:59.231 [DeviceManager] ERROR LINSTOR/Satellite/2fcb4d SYSTEM - Generated resource file for resource 'pvc-918df6b7-f2ac-4edb-8a3c-0fd5fe8e5e19' is invalid. [Report number 68406749-5C801-000023]```

The external command sent the following error information:
/var/lib/linstor.d/pvc-918df6b7-f2ac-4edb-8a3c-0fd5fe8e5e19.res_tmp:103: in resource pvc-918df6b7-f2ac-4edb-8a3c-0fd5fe8e5e19
    ipv4:10.1.1.18:7023 is also used /var/lib/linstor.d/pvc-3279d999-4a6d-4c8e-bf6b-2536f091b8ab.res:71 (resource pvc-3279d999-4a6d-4c8e-bf6b-2536f091b8ab)
/var/lib/linstor.d/pvc-918df6b7-f2ac-4edb-8a3c-0fd5fe8e5e19.res_tmp:104: in resource pvc-918df6b7-f2ac-4edb-8a3c-0fd5fe8e5e19
    ipv4:10.1.1.15:7023 is also used /var/lib/linstor.d/pvc-3279d999-4a6d-4c8e-bf6b-2536f091b8ab.res:72 (resource pvc-3279d999-4a6d-4c8e-bf6b-2536f091b8ab)
command sh-nop exited with code 10

Both PVC-s were created during velero backup.

[ghernadi]

Can you check if both PVCs still exist in linstor rd l and also if both really have the same TCP port? If so, I guess we will need to fix this in the DB, but I'd be curious how you ended up with duplicated TCP ports..

[ksyblast]

We have already fixed it I hope.

I can't say exactly if there were both resources in 'r l' but definitely at least one of them was hanging in Connecting state. When the backup was still running I could see that these both PVCs were temporary PVCs created by velero. When the backup was finished the resource was still in Connecting state and there were 3 rd in the database not belonging to any PVC. The problematics PVCs were no longer listed in the API. We deleted the dangling rd-s manually, one of them was failing until we found and also deleted the corresponding PV which still was seen in K8s API. Now the number of rd-s and resources corresponds to the number of PVCs.

Maybe I will also recheck in the database if something related to them is still left.

P.S. Once or twice we already had this issue with a duplicate port after backups, we had to find all the corresponding objects in the db and delete them manually.

[ghernadi]

Deleting entries from the database manually is always a bit risky since you can overlook some entries and end up with orphaned data. The issue here is not just cluttering the database but also new entries might accidentally relate to those entries, making it nearly impossible to figure out how one got into this state.

This is why I usually try to figure out if there is really the same port in the database itself or if we only have some orphaned .res file laying around and drbdadm finds that when checking the validity of .res files. In the latter case cleaning up the .res file should be preferred than trying to manipulate the database manually.

[ksyblast]

Thank you @ghernadi . So could you please confirm that the most appropriate action in case we face the situation again is to delete the problematic .res file from the satellite? Could you please give some recommendations how we can trace the same port re-use scenario? I can collect the logs if needed.

The last backup partially failed with the error we have already faced: piraeusdatastore/piraeus-operator#824

[ghernadi]

Yes. To my understanding there are just 2 possibilities how you can end up in a duplicated TCP port scenario (or duplicated minor number).

1. LINSTOR made a mistake and the same TCP port is actually in the database. You should be able to check this using rd l and check for duplicated ports (or in future this will be moved to r l --show-drbd-ports or something like that)
2. There is some other .res file lying around on your satellite that LINSTOR does not know about (or maybe "no longer knows about"). When running any drbdadm command, drbdadm scans all available .res files and runs some uniqueness checks, for example for TCP ports, minor numbers, etc... That means drbdadm does indeed take .res files into account that are not known by LINSTOR, maybe not even managed by LINSTOR, so that drbdadm can complain about duplicated settings.

In the first case, there is not much you can do besides trying to delete the resource if the controller is still up and running. If the controller is already offline, it may not start properly due to the duplicated entry. In this case you will need to clean up the DB itself (which is quite cumbersome as mentioned earlier)

In the latter case, you are correct with your assumption. You can try restart the satellite (not the controller). When the satellite starts, it first deletes all .res files in /var/lib/linstor.d (unless LS_KEEP_RES is used). Once it receives the data from the controller, the satellite rewrites all needed .res files and checks if the DRBD resources need to be adjusted.

Now that I have mentioned this orphaned .res file scenario: In 1.31.1 we released a fix for a bug where if you had a DRBD resource manually downed (drbdadm down $rsc) and deleted the LINSTOR resource while the corresponding DRBD resource was down, LINSTOR did not properly delete the .res file. LINSTOR would in this case re-use the TCP port for the next created resource since the previous resource was deleted from the LINSTOR database. The still remaining orphaned .res file on the satellite could cause the issue described in 2.

So if it is possible that you had such an incident, you might want to upgrade LINSTOR to avoid this in future.

[ksyblast]

Today we have faced the situation with the duplicate port again after velero backup. I can confirm the duplicate port was not in rd l. There was only one record with the mentioned port in rd l and it was some dangling velero PVC left after the backup, the PVC was no longer in the cluster, only the PV was left.

I checked the .res files at the satellites and there were some res_tmp files using the same port. PVC-s corresponding to these resources were no longer in the cluster. I made some combination of deleting the res_tmp resources, restarting the satellites and so on and finally the issue was resolved.

We are planing the upgrade to the version 1.31.1. I will let you know if we still face backup issues with velero afterwards.

[ghernadi]

Glad to hear you could fix this on your own!

However, I am a bit surprised that a res_tmp should cause this issue. To my understanding drbdadm is explicitly only scanning *.res files, not *.res_tmp. The latter file is usually created by LINSTOR in order to check the generated file before replacing the corresponding .res file with it. For that LINSTOR runs commands like

drbdadm --config-to-test /var/lib/linstor.d/rsc.res_tmp --config-to-exclude /var/lib/linstor.d/rsc.res sh-nop

In other words, this "config-to-test" needs to explicitly ignore the "real" [...]/rsc.res file and needs to be given explicitly the [...]/rsc.res_tmp file to test. Otherwise I'd expect drbdadm to happily ignore all files that do not match *.res...

Which means, that my best guess would still be some orphaned .res file still occupying the TCP port - which should have been cleaned up by restarting the satellite.

We will make a few changes in the near future, some of the regarding the TCP port handling itself, others regarding the .res_tmp files, so this issue might be addressed in future releases even more (hopefully in a positive way 😁 )

[bernardgut]

Hello, after going through #451, and "fixing" it by deleting the problematic linstor CRDs, I am now experiencing this exact same issue, down to ID:

ERROR REPORT 684A99A9-00000-005154

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Controller
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T07:51:57+00:00
Error time:                         2025-06-28 08:11:59
Node:                               linstor-controller-7496884d58-2xthr
Thread:                             grizzly-http-server-24

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiTransactionException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiTransactionException
Generated at:                       Method 'commit', Source file 'CtrlTransactionHelper.java', Line #30

Error message:                      com.linbit.linstor.transaction.TransactionException: Error creating rollback entry

Call backtrace:

    Method                                   Native Class:Line number
    commit                                   N      com.linbit.linstor.core.apicallhandler.controller.CtrlTransactionHelper:30
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlKvsApiCallHandler:182
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:1139
    lambda$modifyKVStore$2                   N      com.linbit.linstor.api.rest.v1.KeyValueStore:104
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:235
    modifyKVStore                            N      com.linbit.linstor.api.rest.v1.KeyValueStore:97
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor232:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:569
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:840

Caused by:
==========

Category:                           RuntimeException
Class name:                         TransactionException
Class canonical name:               com.linbit.linstor.transaction.TransactionException
Generated at:                       Method 'commit', Source file 'ControllerK8sCrdTransactionMgr.java', Line #157

Error message:                      Error creating rollback entry

ErrorContext: 

Call backtrace:

    Method                                   Native Class:Line number
    commit                                   N      com.linbit.linstor.transaction.ControllerK8sCrdTransactionMgr:157
    commit                                   N      com.linbit.linstor.core.apicallhandler.controller.CtrlTransactionHelper:26
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlKvsApiCallHandler:182
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:1139
    lambda$modifyKVStore$2                   N      com.linbit.linstor.api.rest.v1.KeyValueStore:104
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:235
    modifyKVStore                            N      com.linbit.linstor.api.rest.v1.KeyValueStore:97
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor232:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:569
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:840

Caused by:
==========

Category:                           LinStorException
Class name:                         DatabaseException
Class canonical name:               com.linbit.linstor.dbdrivers.DatabaseException
Generated at:                       Method 'createRollbackEntry', Source file 'ControllerK8sCrdRollbackMgr.java', Line #84

Error message:                      Resource SecObjectProtection/b6a849d48698fe9940ba0d46833827435bd8c66911bf4735b623d1d0dfa74686 not found

ErrorContext:


Call backtrace:

    Method                                   Native Class:Line number
    createRollbackEntry                      N      com.linbit.linstor.transaction.ControllerK8sCrdRollbackMgr:84
    commit                                   N      com.linbit.linstor.transaction.ControllerK8sCrdTransactionMgr:152
    commit                                   N      com.linbit.linstor.core.apicallhandler.controller.CtrlTransactionHelper:26
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlKvsApiCallHandler:182
    modifyKvs                                N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:1139
    lambda$modifyKVStore$2                   N      com.linbit.linstor.api.rest.v1.KeyValueStore:104
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:235
    modifyKVStore                            N      com.linbit.linstor.api.rest.v1.KeyValueStore:97
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor232:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:569
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:840


END OF ERROR REPORT.

I wanted to delete the velero namespace and all backup data because I do not have time to deal with this and my prod can live without backups for a bit, but unfortunately I am seeing that k linstor s l returns all snapshots stuck in DELETING. I think I can solve this by deleting all snapshots and starting over... @ghernadi any idea how to do that ? Right now all I have is the error-reports above and all snapshots stuck in DELETING.

How do I get out of this in a clean way ? I just want to start over (without deleting any actual PVC data)