richard-churchman
bf0faba627
This commit consolidates extensive refactoring across the engine, orchestrators, background tasks, JSON pipelines, caching layers, messaging, and infrastructure. Major improvements include async correctness, cancellation propagation, architectural restructuring, performance optimizations, parser and engine enhancements, UI updates, deployment improvements, and critical bug fixes. Asynchronous, Cancellation, and Concurrency Improvements: * Converted all repository and controller methods to fully asynchronous implementations; removed remaining synchronous database calls. * Introduced centralized cancellation via a `CancellationTokenProvider` in DI and propagated tokens through database, engine, cache, and background processes. * Implemented graceful SIGTERM handling with coordinated shutdown. Most tasks are subject to immediate cancellation, while: - Draining AMQP consumers - Draining Kestrel in-flight requests - Draining async Archive writes - Draining async Case Creation processing * Added middleware to track active requests and emit `Connection: close` during shutdown. * Enhanced exception handling: - Cancellation exceptions logged as INFO - Improved logging for async call failures * Added `/ready` endpoint: - Returns 200 if the engine is ready, 503 otherwise - Integrated readiness signaling from TaskController and ActivationWatcher * Fixed concurrency/timing issues, including LRU cache updates using a semaphore instead of locks. * Ensured proper propagation of cancellation tokens across async call stacks, except where drainage is required. Engine, Orchestrator, and Pipeline Refactoring: * Rebuilt Invoke Orchestrator using a Context–Strategy–Pipeline hybrid pattern. * Extracted JSON parsing and database record parsing into dedicated helper classes. * Standardized instance object state management using a centralized Context object; removed private mutable state from orchestrators and engine. * Reorganized namespaces and moved background tasks to dedicated classes. * Removed god-class, organic-growth code and streamlined dependency flow; dependencies are now passed via Context only. * Added reference date integrity checks: - Prevent processing of future `ReferenceDate` values to prevent purge of records through bad data. - Updating of reference date ignored for records older than the latest entry to prevent invalid purges. * Introduced nested classes for large models and properties to improve readability. Moved very large property sets to nested classes for better context in engine pipelines. * Implemented rich extension functions library for parser and rule coder functionality: - Parser Scans new extensions assembly to add parser security tokens, enabling centralized development of extension functions to enhance rule capabilities. - Documented new tokens under Rule Compilation Tokens and Extensions. * Simplified `Invoke` data extraction logic via helper extension methods. * Centralized Context creation for JSON settings and contract resolvers. * Dropped legacy Response Payload approach, pending improved serialization to support pruning response payload. * Renamed `Report Keys` to `Archive Keys` for consistency with POCOs and database entities. * Reorganized engine files and namespaces to reflect new software architecture. * Implemented a bytes limit for payloads received over the invoke controller, controlled via the `MaxInvokeControllerRequestBytes` environment variable. Cache, Redis, TTL, and Background Task Enhancements: * Replaced Postgres `NOTIFY` in callbacks with Redis-based signaling given Postgres should not be interacted with during real-time flows — only Redis. * Improved callback timeout/removal logic and reduced initial invocation payload sizes. * Added environment variables for SearchKey cache job interval control: `SearchKeyCacheServerIntervalType` and `SearchKeyCacheServerIntervalValue`. * Refactored cache purge with observability tables and supporting migrations to enhance confidence in high-risk processing. * Converted TTL counters to `long` type with corresponding storage updates in the `ArchiveKeys` table. * Ensured only one TTL increment per invocation, irrespective of activation rules. * Updated Redis background scans from `HGETALL` to incremental `HSCAN` to prevent deadlocks and improve performance. - Removed obsolete payload-first cache tracking. * Improved initial LRU cache fill visibility via TaskController integration. * Refactored repository access in the Cache Service and audited DbContext disposal to prevent connection leaks. * Fixed counter metrics in the LRU cache. * Batched SortedSetRemoveAsync and HashDeleteAsync as it was causing long operations in Redis. AMQP, Threading, and Background Workflow Improvements: * Refactored AMQP messaging to resolve deadlocks in async consumers. * Moved RabbitMQ channel creation to consumer tasks and improved heartbeat handling to avoid deadlocks given `IModel` is not thread-safe. * Offloaded expensive case-creation processing to scalable background threads: - Reduces pressure on main transaction threads - Eliminates blocking in Archive storage and reduces memory leak risk (backlogged case creation has a much smaller footprint than backlogged Archive payload) - Threads controlled via the `CaseCreationThreads` environment variable * Introduced `TaskController` to manage all background (non-transactional) tasks. TaskController allows visibility of all background tasks. * Centralized cancellation and graceful shutdown across engine, background tasks, and web host. * On SIGTERM, if a task is not terminated within two seconds — perhaps due to drainage — it is written to console to give feedback on blocking tasks. * Implemented heartbeats in the connection to handle brittle connections and unforgiving .NET RabbitMQ clients, controlled by the `AMQPHeartbeatSeconds` environment variable. JSON, Parser, and Serialization Fixes: * Updated JSON settings, contract resolvers, and DI wiring. * Reduced payload response size on async HTTP invocation to only return the EntityAnalysisModelInstanceEntry GUID for callback, notwithstanding head check and parse still taking place. * Added `DBNull` checks for database led reprocessing deserialization. * Ensured the DictionaryNoBoxing converter is properly added to serialization settings to parse dates using the round-trip format. Performance, Stability, and General Cleanup: * Updated PooledDictionary to fix hash collisions and improve initial sizing. * Return null from PooledDictionary.cs instead of exception. * Ensured DB connections are disposed properly during model sync to prevent pool starvation in the new patterns. * Refactored SMTP client usage and added authentication/connection validations. * Removed unnecessary AutoMapper list mappings and streamlined repository configurations. * Renamed illogical methods and variables for clarity, as came across them. * Applied `ConfigureAwait` consistently across all projects. * Added `Async` suffix to async method names. * Replaced async unsupported `out` variables with tuples UI, Infrastructure, and Deployment Updates: * Updated UI to show case-creation, missing tagging and notification queue balance counters. * Removed obsolete test Blazor files (this project will go Angular). * Added production GC environment variables to `docker-compose.yml`. * Added `Jube.HealthCheck.csproj` for lightweight HTTP health checks in container (avoids installing `curl` as root). * Updated `docker-compose.yml` to bind port `5001` for job process readiness checks, as was already the case with the other process. * Implemented frame-busting on the login page to prevent the authentication page from being loaded in a frame after expiration. Critical Bug Fixes: * Fixed transaction buffer flushes on commit. * Fixed bulk write attempts on empty buffers. Miscellaneous Improvements: * Moved engine and model manager into a unified hybrid context with strategy & pipeline patterns. * Shifted thread pool configuration to startup pipeline. * Reorganized files and namespaces to align with the new architecture. * Improved observability and versioning for `Archive` and `ArchiveKey` tables to support reprocessing. * Dropped unused/complex payload-first cache tracking. * Many additional small bug fixes and refinements discovered during the grand refactor. * The Sandpit had some telemetry to a Chatbot and Google Analytics. Although it was disabled by default — and the Sandpit was intended for Jube use only — it should not have been in the public build. * Updated UI to display case-creation counters and missing tagging/notification fields. * Added a filter to remove instances from the synchronization page that have not sent a heartbeat. * Fixed invoke counter and removed the condition that counters are only written on invoke for more consistent system observability. * Updated the parser to support dates in inline functions, which was previously broken, which meant that dates not available. * Improved thread pool startup parameters to be a factor of two times logical cores for worker threads and four time logical cores for io threads. * Set command timout to zero for archive persistence given sensitivity. In time will create a dead letter archive for total failure, as on failure the records dropped. **Notes:** Functionality remains substantially the same where intended; most changes focus on stability, maintainability, async correctness, and operational robustness.
36 lines
1.2 KiB
C#
36 lines
1.2 KiB
C#
/* Copyright (C) 2022-present Jube Holdings Limited.
|
|
*
|
|
* This file is part of Jube™ software.
|
|
*
|
|
* Jube™ is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License
|
|
* as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
|
|
* Jube™ is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
|
|
* of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
|
|
|
|
* You should have received a copy of the GNU Affero General Public License along with Jube™. If not,
|
|
* see <https://www.gnu.org/licenses/>.
|
|
*/
|
|
|
|
namespace Jube.TaskCancellation
|
|
{
|
|
using Interfaces;
|
|
|
|
public class CancellationTokenProvider : ICancellationTokenProvider
|
|
{
|
|
private readonly CancellationTokenSource cts = new CancellationTokenSource();
|
|
|
|
public CancellationToken Token
|
|
{
|
|
get
|
|
{
|
|
return cts.Token;
|
|
}
|
|
}
|
|
|
|
public void Cancel()
|
|
{
|
|
cts.Cancel();
|
|
}
|
|
}
|
|
}
|