perf: Major performance optimizations - memory leak fix, SQLite tuning, and cache improvements

[Chesyre]

User description

This PR addresses critical performance issues causing high RAM consumption and database errors during concurrent downloads

Key Changes:

• Fix memory leak in WebsocketsUpdater by properly disposing DbContext scopes after each iteration
• Add adaptive WebSocket polling interval (1s with clients, 5s idle) to reduce unnecessary CPU usage
• Configure SQLite with WAL mode, connection pooling, and optimized PRAGMAs for better concurrency
• Remove aggressive cache invalidation calls (13 locations) that caused cache thrashing
• Add AsSplitQuery() optimization for more efficient database queries

---

PR Type

Enhancement

---

Description

• Remove aggressive cache invalidation calls (13 locations) causing cache thrashing

• Fix memory leak in WebsocketsUpdater by properly disposing DbContext scopes

• Add adaptive WebSocket polling interval (1s active, 5s idle) reducing CPU usage

• Configure SQLite with WAL mode, connection pooling, and optimized PRAGMAs

• Add AsSplitQuery() optimization for more efficient database queries

---

Diagram Walkthrough

flowchart LR
  A["Cache Invalidation Removal"] --> B["Reduced Cache Thrashing"]
  C["DbContext Scope Management"] --> D["Fixed Memory Leak"]
  E["Adaptive Polling Interval"] --> F["Lower CPU Usage"]
  G["SQLite Configuration"] --> H["Better Concurrency"]
  I["AsSplitQuery Optimization"] --> J["Efficient Queries"]

Loading

File Walkthrough

	Relevant files
Performance optimization	DownloadData.cs Remove aggressive cache invalidation calls                              server/RdtClient.Data/Data/DownloadData.cs Removed 13 aggressive TorrentData.VoidCache() calls from download update methods Eliminated cache invalidation after Add, UpdateUnrestrictedLink, UpdateFileName, UpdateDownloadStarted, UpdateDownloadFinished, UpdateUnpackingQueued, UpdateUnpackingStarted, UpdateUnpackingFinished, UpdateCompleted, UpdateError, UpdateRetryCount, DeleteForTorrent, and Reset operations Reduces cache thrashing and improves performance during concurrent operations +0/-26    TorrentData.cs Add AsSplitQuery optimization for queries                                server/RdtClient.Data/Data/TorrentData.cs Added .AsSplitQuery() to the torrent retrieval query Optimizes database query execution for better performance with related entities +1/-0
Configuration changes	DiConfig.cs Configure SQLite connection pooling and caching                    server/RdtClient.Data/DiConfig.cs Enhanced SQLite connection string with Cache=Shared;Pooling=True;Command Timeout=30 parameters Enables connection pooling and shared cache for better concurrency handling +1/-1      Startup.cs Configure SQLite WAL mode and PRAGMAs                                        server/RdtClient.Service/BackgroundServices/Startup.cs Added SQLite PRAGMA configuration for WAL mode and performance tuning Configured journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, and cache_size=-64000 Improves database concurrency and reduces lock contention +6/-0
Bug fix	WebsocketsUpdater.cs Fix scope leak and add adaptive polling                                    server/RdtClient.Service/BackgroundServices/WebsocketsUpdater.cs Fixed memory leak by moving DbContext scope creation inside the loop with using statement Added adaptive polling interval: 1 second when clients connected, 5 seconds when idle Reduces unnecessary CPU usage and properly disposes resources after each iteration Added logging for service startup +9/-4

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢	No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Missing audit logs: New operational actions (WebSocket updater cycles and remote updates) do not add any audit trail for critical actions; only generic info/error logs are present, making it unclear if sensitive state changes are recorded elsewhere. Referred Code logger.LogInformation("WebsocketsUpdater started."); while (!stoppingToken.IsCancellationRequested) { try { using var scope = serviceProvider.CreateScope(); var remoteService = scope.ServiceProvider.GetRequiredService<RemoteService>(); await remoteService.Update(); } catch (Exception ex) { logger.LogError(ex, $"Unexpected error occurred in WebsocketsUpdater: {ex.Message}"); } var delay = RdtHub.HasConnections ? TimeSpan.FromSeconds(1) : TimeSpan.FromSeconds(5); await Task.Delay(delay, stoppingToken); ... (clipped 4 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Broad catch block: The broad exception handler logs errors but does not implement backoff or safeguards to prevent tight error loops, and removed cache invalidations may introduce edge cases not explicitly handled. Referred Code catch (Exception ex) { logger.LogError(ex, $"Unexpected error occurred in WebsocketsUpdater: {ex.Message}"); } var delay = RdtHub.HasConnections ? TimeSpan.FromSeconds(1) : TimeSpan.FromSeconds(5); await Task.Delay(delay, stoppingToken); Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: DB config risk: New SQLite settings (WAL, pooling, shared cache, PRAGMAs) may affect isolation and concurrency; without visibility into connection safeguards and access controls, security impact cannot be fully assessed from the diff. Referred Code public static void Config(IServiceCollection services, AppSettings appSettings) { if (String.IsNullOrWhiteSpace(appSettings.Database?.Path)) { throw new("Invalid database path found in appSettings"); } var connectionString = $"Data Source={appSettings.Database.Path};Cache=Shared;Pooling=True;Command Timeout=30"; services.AddDbContext<DataContext>(options => options.UseSqlite(connectionString)); services.AddScoped<DownloadData>(); services.AddScoped<SettingData>(); Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	Re-evaluate the cache invalidation strategy The PR's removal of all cache invalidation calls in DownloadData.cs will result in a stale cache. A more sophisticated caching strategy, like targeted updates or time-based expiration, should be implemented to ensure data consistency. Examples: server/RdtClient.Data/Data/DownloadData.cs [47-248] await dataContext.SaveChangesAsync(); return download; } public async Task UpdateUnrestrictedLink(Guid downloadId, String unrestrictedLink) { var dbDownload = await dataContext.Downloads .FirstOrDefaultAsync(m => m.DownloadId == downloadId); ... (clipped 192 lines) Solution Walkthrough: Before: // In RdtClient.Data/Data/DownloadData.cs public async Task UpdateDownloadFinished(Guid downloadId, DateTimeOffset? dateTime) { var dbDownload = await dataContext.Downloads .FirstOrDefaultAsync(m => m.DownloadId == downloadId); if (dbDownload == null) { return; } dbDownload.DownloadFinished = dateTime; await dataContext.SaveChangesAsync(); // The call to TorrentData.VoidCache() was removed here. } After: // In RdtClient.Data/Data/DownloadData.cs public async Task UpdateDownloadFinished(Guid downloadId, DateTimeOffset? dateTime) { var dbDownload = await dataContext.Downloads .FirstOrDefaultAsync(m => m.DownloadId == downloadId); if (dbDownload == null) { return; } dbDownload.DownloadFinished = dateTime; await dataContext.SaveChangesAsync(); // A more nuanced cache invalidation is needed instead of complete removal. // e.g., invalidate only the specific item or use time-based expiration. await TorrentData.InvalidateTorrent(dbDownload.TorrentId); } Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical flaw where removing all TorrentData.VoidCache() calls without a replacement will cause the torrent cache to become permanently stale, leading to incorrect data being served.	High
Possible issue	Handle potential task cancellation exceptions Move the Task.Delay call into the try-catch block and add a catch for OperationCanceledException to ensure graceful shutdown of the background service. server/RdtClient.Service/BackgroundServices/WebsocketsUpdater.cs [19-38] while (!stoppingToken.IsCancellationRequested) { try { - using var scope = serviceProvider.CreateScope(); - var remoteService = scope.ServiceProvider.GetRequiredService<RemoteService>(); - - await remoteService.Update(); + using (var scope = serviceProvider.CreateScope()) + { + var remoteService = scope.ServiceProvider.GetRequiredService<RemoteService>(); + await remoteService.Update(); + } + + var delay = RdtHub.HasConnections + ? TimeSpan.FromSeconds(1) + : TimeSpan.FromSeconds(5); + + await Task.Delay(delay, stoppingToken); + } + catch (OperationCanceledException) + { + // This is expected on shutdown, so we can just ignore it. } catch (Exception ex) { logger.LogError(ex, $"Unexpected error occurred in WebsocketsUpdater: {ex.Message}"); } - - var delay = RdtHub.HasConnections - ? TimeSpan.FromSeconds(1) - : TimeSpan.FromSeconds(5); - - await Task.Delay(delay, stoppingToken); } Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that an OperationCanceledException from Task.Delay is unhandled, which would cause an ungraceful shutdown of the background service. Applying this change improves the service's robustness.	Medium
More

[rogerfar]

Why remove these?

[Chesyre]

Since the cache is managed automatically, it empties itself.
So I didn't find any use in forcing it to empty.