Updated files and patched dependencies #49

carloscumpian · 2025-11-20T00:32:17Z

Changes:

Python 3.12 Upgrade ✅

Upgraded from Python 3.9.21 to Python 3.12.12
- Dockerfiles/Dockerfile.dss
- .travis.yml
- Pipfile
Fixed async/await syntax (removed deprecated @asyncio.coroutine)
- detect_secrets_stream/scan_worker/app.py
- detect_secrets_stream/scan_worker/diffscanworker.py
Updated 26+ packages for CVE fixes
- Pipfile
- Dockerfiles/Dockerfile.dss
All services running with Python 3.12
- tested w/ steps in: https://github.com/IBM/detect-secrets-stream/tree/main/kustomize_envs/dev#local-end-2-end-test

Package Updates ✅

Pipfile
- ibm-db: 3.2.6 → 3.2.7
- gevent: 23.9.1 → 24.11.1
- pipenv: 2018.11.26 → 2024.4.0
- dss: 0.13.1+ibm.62.dss → 0.13.1+ibm.64.dss
detect-secrets pre-commit: 0.13.1+ibm.55.dss → 0.13.1+ibm.64.dss

Platform Compatibility ✅

⚠️ Added FROM --platform=linux/amd64 to Dockerfile (We can remove this for testing in .travis.yml and GHA as I don't think its needed outside of local tests due to M1/M2 chips)
- Resolves ibm-db Linux ARM64 incompatibility
- Works on Mac M1/M2 (via Rosetta 2), Travis CI, GitHub Actions, and production

Deployment Verification ✅

All 5 pods running (gd-ingest, gd-revoker, postgres, scan-worker, vault)
Container structure tests: PASSED (2/2)
Health checks: All services operational
Vault: Initialized, unsealed, read/write working
Port forwarding: Active on all services

Pre-commit Hook Fixed ✅

Updated .pre-commit-config.yaml to use detect-secrets 0.13.1+ibm.64.dss
Version warning eliminated
Pre-commit hooks working correctly

Vault Configuration ✅

sleep 3 kept in vault_init.sh - was timing out when testing, but should be able to be removed if needed (only 3 secs)
added echos for testing, but can also be removed if not wanted/needed

Documentation ✅

Updated kustomize_envs/dev/README.md with additional testing procedures
Added verification commands for all services

Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

- Pin pipenv to version 2024.4.0 (same as Dockerfile) - Update Trivy skip-dirs from python3.9 to python3.12 - Fixes ImportError: cannot import name 'dedup' from 'pipenv.vendor.pythonfinder.utils' Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

- Updated cryptography from 44.0.0 to 44.0.1 - Fixes CVE-2024-12797 (NULL pointer dereference vulnerability) - Regenerated Pipfile.lock Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

Convert f-strings from single-quoted outer strings with double-quoted dict keys to double-quoted outer strings with single-quoted dict keys. This satisfies the pre-commit double-quote-string-fixer hook which was failing in Travis CI. Changes to detect_secrets_stream/util/secret_util.py: - Lines 991-1002: Fixed f-strings in ingest_commit function - Lines 1112-1124: Fixed f-strings in rescan_commit function Changes to .flake8: - Added E231 ignore for secret_util.py to suppress false positives on URL strings containing '://' which flake8 incorrectly flags as needing whitespace after the colon The double-quote-string-fixer hook was incorrectly converting f'text {dict["key"]}' to f'text {dict['key']}' which is a syntax error. The proper format is f"text {dict['key']}" with double quotes outside and single quotes for dict key access inside. Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

Changes to .flake8: - Added global ignore for E231 (missing whitespace after ':') - E231 produces many false positives for URL strings with '://' and f-strings with dictionary key access - This affects 80+ lines across multiple files in the codebase Changes to detect_secrets_stream/notification/tests/org_set_controller_test.py: - Fixed JSON string formatting broken by add-trailing-comma hook - Moved closing ']' bracket back to end of JSON string on line 79 - Removed duplicate closing parenthesis - Ensured proper JSON array closure: '}}]' These changes resolve all remaining pre-commit hook failures in Travis CI. Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

…ing-comma Changes to .flake8: - Expanded ignore list to include E126, E223, E226, E241, E272, E702, W503, W504, N818 - These are either false positives or legacy code style issues - Prevents 20+ flake8 errors across the codebase Changes to .pre-commit-config.yaml: - Excluded org_set_controller_test.py from add-trailing-comma hook - The hook incorrectly reformats JSON string literals, breaking syntax - This file contains multi-line JSON strings that should not be modified Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

carloscumpian · 2025-11-20T16:57:16Z

.flake8

.flake8 - added ignore = E231,E126,E223,E226,E241,E272,E702,W503,W504,N818 since it was giving out errors about whitespace that wasn't needed.

example https:// it was giving warning due it wanting a space between : and /

carloscumpian · 2025-11-20T16:58:56Z

.pre-commit-config.yaml

added an exclude to org_set_controller_test.py under hooks: id: add-trailing comma since it was trying to add a comma that wasn't needed

carloscumpian · 2025-11-20T16:59:53Z

Dockerfiles/Dockerfile.dss

added --platform=linux/amd64 - this was mostly due to a local issue I was having - doesn't seem that it hurts Travis/GHA, but from looking around we can take it off since it should build with either one

added python3-dev libev-dev libc-ares-dev && \ - newer version of gevent needed additional packages, but haven't tested removing them

added RUN PIP_USER=1 PIP_IGNORE_INSTALLED=1 pip install certifi==2024.12.14 typing-extensions==4.12.2 packaging==24.2 zope.interface==7.2 zope.event==5.0
The above packages were being skipping when going through line 30

carloscumpian · 2025-11-20T17:03:20Z

detect_secrets_stream/util/secret_util.py

Added double quotes due to Travis CI complaining about them
The hook was incorrectly converting f'text {dict[\"key\"]}' to f'text {dict['key']}' which is a syntax error. The proper format is f\"text {dict['key']}\" with double quotes outside and single quotes for dict key access inside the f-string." python format doc

carloscumpian · 2025-11-20T17:03:44Z

skaffold.yaml

same as dockerfile above, local issue with my M1 chip
buildArgs: BUILDPLATFORM: linux/amd64

krkazmier

as long as the local tests passed with ingesting and building, this looks good to me! i don't know how much adding linux/amd64 in places will affect this though. After this is merged these changes will need to staged against the ibm-cloud-secrets-manager branch and we can work together to do local testing for that before proceeding with creating a new release/image

Updated files and patched dependencies

48460dc

Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

carloscumpian force-pushed the caleb-dss-patching branch from 604a146 to 48460dc Compare November 20, 2025 00:45

carloscumpian added 5 commits November 19, 2025 18:52

Upgrade cryptography to 44.0.1 to fix CVE-2024-12797

0350f5d

- Updated cryptography from 44.0.0 to 44.0.1 - Fixes CVE-2024-12797 (NULL pointer dereference vulnerability) - Regenerated Pipfile.lock Signed-off-by: Caleb Cumpian <Carlos.Cumpian@ibm.com>

carloscumpian commented Nov 20, 2025

View reviewed changes

skaffold.yaml

Copy link

Author

carloscumpian Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as dockerfile above, local issue with my M1 chip
buildArgs: BUILDPLATFORM: linux/amd64

carloscumpian marked this pull request as ready for review November 24, 2025 15:07

krkazmier approved these changes Dec 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Updated files and patched dependencies #49

Updated files and patched dependencies #49

Uh oh!

carloscumpian commented Nov 20, 2025

Uh oh!

carloscumpian Nov 20, 2025

Uh oh!

carloscumpian Nov 20, 2025

Uh oh!

carloscumpian Nov 20, 2025

Uh oh!

carloscumpian Nov 20, 2025

Uh oh!

carloscumpian Nov 20, 2025

Uh oh!

krkazmier left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Updated files and patched dependencies #49

Are you sure you want to change the base?

Updated files and patched dependencies #49

Uh oh!

Conversation

carloscumpian commented Nov 20, 2025

Changes:

Python 3.12 Upgrade ✅

Package Updates ✅

Platform Compatibility ✅

Deployment Verification ✅

Pre-commit Hook Fixed ✅

Vault Configuration ✅

Documentation ✅

Uh oh!

carloscumpian Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

carloscumpian Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

carloscumpian Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

carloscumpian Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

carloscumpian Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

krkazmier left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants