Eng 1740 fallback to database identities if identities have expired from redis

[JadeCara]

Ticket ENG-1740

Description Of Changes

🎯 Whenever we try to read identities or custom privacy request fields from Redis, fallback to getting the data from the database if the values have already expired.

Refer to the updates to get_cached_identity_data and get_cached_custom_privacy_request_fields in this closed PR Privacy request cache refresh by galvana · Pull Request #4610 · ethyca/fides

Code Changes

• Added fallback query for identities and custom fields to privacy request
• Added new function to verify if identity is cached for existing endpoint call
• Added some tests for this fallback functionality.

Steps to Confirm

1. Run fidesplus pointed at this branch.
2. To allow us to manipulate the cache before our privacy request completes - Set auto approve to false using PATCH /api/v1/config with:

{
  "execution": {
    "require_manual_request_approval": true
  }
}

3. Create a privacy request and grab the id
4. using the local redis login redis-cli -a your_password
5. delete the identity keys DEL id-<<privacy_request_id here>>-identity-email
6. Verify its gone by running KEYS * to verify that key is gone
7. approve the privacy request and verify it runs and that the logs contain:

fidesplus-worker-generic-1  | 2025-11-04 03:14:02.788 | DEBUG    | fides.api.models.privacy_request.privacy_request:get_cached_identity_data:710 - Cache miss for request pri_08f0df9c-6cb7-44aa-9228-3f662e19657c, falling back to DB | {'privacy_request_id': 'pri_08f0df9c-6cb7-44aa-9228-3f662e19657c', 'privacy_request_source': 'Request Manager'}

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
fides-plus-nightly	Ready	Preview	Comment	Nov 5, 2025 3:25pm

1 Skipped Deployment

Project	Deployment	Preview	Comments	Updated (UTC)
fides-privacy-center	Ignored			Nov 5, 2025 3:25pm

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.28%. Comparing base (951dbbe) to head (f9bd0c0).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6896   +/-   ##
=======================================
  Coverage   87.27%   87.28%           
=======================================
  Files         523      523           
  Lines       34251    34267   +16     
  Branches     3943     3945    +2     
=======================================
+ Hits        29893    29909   +16     
  Misses       3499     3499           
  Partials      859      859           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Overview

Greptile Summary

Implements fallback logic to retrieve identity and custom privacy request fields from the database when Redis cache expires, preventing privacy requests from failing when cached data is no longer available.

Key changes:

• Added verify_cache_for_identity_data() method to check if identity data exists in cache without triggering fallback
• Enhanced get_cached_identity_data() to automatically fetch from DB and re-cache when cache miss occurs
• Enhanced get_cached_custom_privacy_request_fields() with similar fallback behavior
• Updated restart endpoint to use verification method instead of retrieval method for cache checking
• Added comprehensive test coverage for fallback scenarios

Test improvements:

• Removed manual cache deletion in test fixtures, following best practices (f4371da8)
• Added tests for both successful fallback and edge cases (no persisted identity)

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The implementation is straightforward and defensive - adding fallback logic that prevents failures when cache expires. The changes are well-tested with multiple test cases covering edge cases. No breaking changes or risky refactoring. The endpoint change correctly delegates to a new verification method that doesn't trigger side effects.
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
src/fides/api/models/privacy_request/privacy_request.py	4/5	Added fallback logic to retrieve identity and custom fields from DB when Redis cache expires, plus helper method to verify cache presence
src/fides/api/api/v1/endpoints/privacy_request_endpoints.py	5/5	Updated restart endpoint to use new verify_cache_for_identity_data() method instead of get_cached_identity_data() for cache verification
tests/ops/models/privacy_request/test_privacy_request.py	5/5	Added comprehensive tests for cache fallback behavior for identity and custom privacy request fields

_{4 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[erosselli]

maybe we can move these 3 lines to a private method that we can reuse in this verify_cache_for_identity_data function and also in get_cached_identity_data ?

[erosselli]

ship it!