Preserve customer saas dataset changes

[Linker44]

Ticket ENG-1822

Description Of Changes

Customer changes are now merged with the upcoming integration update instead of being overwritten.

When customer changes take priority:

• adding a new field
• deleting a field that is only present in their dataset
• editing a fields fides_meta or data_categories

When customer changes dont take priority:

• The customer deleted dataset fields that are present in our official integration.
• customer added/deleted/changed collections or changed collection names. The change will not be registered and we will use the officially defined collection.
• customer changed a field name, this will be detected as a new field effectively making the dataset have the new field (field which name was changed by the customer) + the old field (the field present in the ofifical dataset)

When Integration updates take priority:

• the field being updated is not detected as changed by the customer
• adding new collections

Customers can:

• Add fields

• Add nested fields
• change fields metadata and data categories

Customers cannot


• delete fields

  Customers can only delete fields that were created by them.
• change field names
(a customer changing a field name will result in a new field being detected and preserved as well as the 'changed' field also remaining present with no changes)
• add/delete/change collections
  Only officially added collections are taking into account when performing the merge.

Code Changes

• we now preserve customer changes to datasets
• once the merge is performed we update the stored dataset with the upcoming one. more information on this decision here

Steps to Confirm

You may use #7053 to test.

1. set up an integration
2. make changes to the dataset through /api/v1/connection/{connection_key}/datasetconfig or through the dataset editor ui if using the testing branch. Add new fields, new collection, change a fields meta or data_category and delete a field.
3. update the dataset on the codebase by adding a new field and changing a fields data category.
4. upon server restart verify that the dataset config for the connector now includes the changes made through the datasetconfig and the changes made officially through the codebase (except the field deletion which shouldnt be preserved).

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
fides-plus-nightly	Ready	Preview	Comment	Dec 11, 2025 5:40pm

1 Skipped Deployment

Project	Deployment	Preview	Comments	Updated (UTC)
fides-privacy-center	Ignored			Dec 11, 2025 5:40pm

[codecov]

Codecov Report

❌ Patch coverage is 85.45455% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.30%. Comparing base (a9b9ccc) to head (eb384ec).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/fides/service/connection/merge_configs_util.py	83.33%	10 Missing and 5 partials ⚠️
src/fides/service/connection/connection_service.py	95.00%	0 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (85.45%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7002      +/-   ##
==========================================
- Coverage   87.31%   87.30%   -0.02%     
==========================================
  Files         532      533       +1     
  Lines       34982    35088     +106     
  Branches     4051     4067      +16     
==========================================
+ Hits        30543    30632      +89     
- Misses       3560     3570      +10     
- Partials      879      886       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Overview

Greptile Summary

This PR implements dataset merging functionality to preserve customer customizations to SaaS connector datasets during integration updates.

Key changes:

• Added merge_datasets, _merge_collection, _merge_fields, and _merge_field methods to ConnectionService to handle three-way merging of datasets (stored template, customer modifications, upcoming template)
• Customer field additions, metadata changes, and data category modifications are now preserved during updates
• Integration deletions of fields take priority over customer modifications
• Customer collection additions are intentionally ignored (only official collections are merged)
• The stored template dataset in SaasTemplateDataset is updated after each merge to track the latest official version

Issues found:

• Minor typo: normilzed should be normalized (5 occurrences)

Confidence Score: 4/5

• This PR is safe to merge with minor typo fixes recommended
• The implementation logic is sound with comprehensive test coverage. The three-way merge strategy correctly handles customer modifications, integration updates, and field deletions. Only a minor typo was found which doesn't affect functionality.
• No critical files require special attention. The typo in connection_service.py should be fixed but is cosmetic.

Important Files Changed

File Analysis

Filename	Score	Overview
src/fides/service/connection/connection_service.py	4/5	Adds dataset merging functionality to preserve customer changes during SaaS template updates. Contains a minor typo (normilzed) but logic is sound.
tests/service/test_connection_service.py	5/5	Comprehensive test coverage for dataset merging including field changes, deletions, collection additions, and preservation scenarios.

[greptile-apps]

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[JadeCara]

Overall looks good! a few suggestions, but mostly nits. I do think greptile made a good call out thats worth reviewing.

[Linker44]

@greptileai

changes implemented

[greptile-apps]

_{2 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[galvana]

The logic looks good but I think the merge logic should live in its own util. Something like src/fides/service/connection/dataset_merge_util.py. The ConnectionService is getting pretty big so let's keep this merging logic out of it.

changes implemented.

[galvana]

Good work on a complex feature, ship it!