master
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 richard-churchman
|
4481615382 |
Implemented Redis Resiliency and improved failover capabilities:
* Replaced the StackOverflow Redis client in the Cache library with the Resilient Redis Library. * Improved policy wrappers by introducing two distinct policies: one for idempotent operations and one for increment operations. Added a circuit breaker to the Redis Resiliency layer to prevent thundering herd on re-instantiation, along with shared state tracking for known-down Redis instances. * Updated the Redis resilient command wrapper to proxy all commands through the appropriate policy (e.g. based on idempotency). * Implemented certain other Redis policy state detection. |
||
|
richard-churchman
|
62f33221b6 |
Resiliency, Database Security, Bug Fixes and Observability:
* Implemented a wrapper library for Redis and Postgres clients using Polly retry to enhance durability during failover. Removed all direct client references across projects, replacing them with the new resilient libraries. Bulk changes were largely confined to the database context constructor, given the application's broad use of the Repository pattern and LINQ2DB. * Implemented a WAL log in Redis that stores JSON payloads pending commit to the Archive table as part of an asynchronous process. The WAL entry is removed once the commit completes. During graceful drain on SIGTERM, this provides an observability and recovery mechanism for transactions lost due to host failure or fatal termination. * Implemented a read-only reporting connection for all case management query functions and other backend functions with significant read overhead or security implications, enabling targeting of read replicas under read-only database users. Dynamic query functions have been handled carefully to ensure parameterised variables are always used. * Implemented a `MigrationConnectionString` environment variable to support a dedicated migration user, allowing the application and reporting to run under minimal permissions while still supporting the online upgrade process used in Jube Herd (Docker Swarm). * Added evaluation counters to Gateway and Activation Rule Counters, and ensured they are displayed in the Gateway Rules, Activation Rules, and Activation Rules pages — an important missing observability feature for gradual rule rollout assessment prior to taking decline or case creation steps. Also introduced model total response time counters throughout the invocation pipeline and updated the administration page accordingly. Made all response times throughout the system record in microseconds for consistency. * Updated `CookieOptions` to include the `Secure` flag having been flagged on CodeQL. * Fixed major bugs in fan-out queries across all Case Management entities caused by the recently released permissions functionality, where entities were duplicated when more than one user was present. * Fixed a long-standing issue with performance counter increments to use `Interlocked`, or custom locks in the case of date-based counters. * Fixed a bug in preservation import introduced during the `CacheIndexId` functionality, which failed due to a nested transaction. The import now bypasses `CacheIndexId` rekeying and reads it directly from the `.jemp` file, which was itself the correct original behaviour. * Fixed Payload Responses not being included in the response payload when response payload was selected in Request XPath. * Fixed the Report Table switch in Request XPath having no effect and not writing to the `ArchiveKeys` table. * Fixed the inability to edit Case Workflows in the user interface because of a minor JS typo in the key name. * Refactored notification dispatch to use async methods for all database calls. * Refactored the Postgres Query class to make better use of `using`/dispose patterns, moved connection creation to the constructor, and ensured consistent disposal. Extended the use of `using` dispose throughout all LINQ2DB queries outside the Repository pattern. * Some refactoring to use byte arrays rather than `MemoryStream` in serialisation of JSON Archive and response payloads. Eliminated duplication in Archive payload creation for case creation, given the payload now also exists in the WAL, and is created during invocation (not asynchronously as was the case). * Updated the composite index for `AddArchiveKeyTableIndex` in Baseline to use covering indexes ONLY, avoiding Postgres page lookups (noting this table exists solely as an access path to the Archive table via predication and join on `EntityAnalysisModelInstanceEntryGuid`). IMPORTANT NOTE: No migration provided for existing data. Any such index will need to be rebuilt manually to obtain this benefit. * Updated the completions controller and query to order fields by name ascending, with new repository methods to apply ordering server-side. * Updated the date format for Created Dates in the user interface to a more verbose and readable format, consistent with the rest of the system. * Removed legacy billing aggregation counter for response elevation, which is no longer a relevant concept. Renamed remaining billing-related counter fields for clarity. * Updated Dockerfile for the new resilience projects. * Fixed typo in TTL Counter Resolution page where hours was duplicated instead of minutes. * Implemented RedisBackplane environment variable to support SignalR with Redis in clustered environments. * Fixed loss of password, password expiry, and password created date during User Administration updates; added empty password check in authentication controller to raise an exception treated as user not found; fixed inability to change password on expiry; fixed password expiry date evaluation bug; moved increment password failure to capture all code paths. * Updated ExhaustiveSearchInstanceTrialInstanceVariable repository to use logical deletes instead of physical deletes, which are unsupported by the database user; updated MockArchive loading to occur only once since the data is static. * Updated default Log4NetLogLevel environment variable to WARN and updated documentation accordingly. * Fixed missing reference for Archive Keys during payload creation, as while they were being populated, the reference to it was missing, so they did not get stored. |
||
|
richard-churchman
|
2f32c72d69 |
Features to exercise greater control over cache burden and improve cache warming:
* Added `Cache` flag to `EntityAnalysisRequestXPath`, updating POCOs, DTOs, and synchronisation mappings. Included migrations and updated pages and JS for new fields. * Updated the Completions Endpoint to only return values if Cache is enabled in the EntityAnalysisModelRequestXPath for EntityAnalysisModelAbstractionRule configuration. * Updated the parser to ignore fields if the parse processing type is Abstraction Rule, so to ensure that an Abstraction Rule does not compile anywhere, in the controller or syncronisation, if it trys to use a key that exists but is marked as Cache. * Created migrations for the cache flag and added indexes to cache key columns. * Implemented a new serialisation pattern for message pack payloads to reduce byte storage of names, replacing them with a `CacheIndexId` serialised as a single byte. * Implemented indexing logic for `CacheIndexId`, which increments within a transaction per `EntityAnalysisModel`. Created `CacheIndexId` values for legacy data. * Created migrations for legacy Redis cache payload data to adopt the new `CacheIndexId` format. Added a unique composite index to enforce sequencing and eliminate conflict risks. * Implemented a new LRU cache warming strategy that populates the LRU cache based on journal usage. During warming, cache entries are populated in order of most recently used journals, exiting when byte limits are reached. This replaces a full cursor scan of payload data, reducing container startup time and mitigating Redis herd-stampede risks. Each journal recall writes to a sorted set (`LruJournals`) that maintains an ordered activity list for cache population. On payload entry removal, the corresponding journal is checked and the `LruJournal` key is removed if no further data exists. Added a `SortedSetLruJournalRemove` enum for task tracking. Implemented an `IsFull` method on the local LRU cache to exit warming once full. * Updated TTL Counters to process `double` values instead of `int` to support summing of integer and float payload types, making Volume and Velocity rules available to TTL Counters and long-term state. Updated POCOs, DTOs, and synchronisation mappings, applied logic in invocation processes, and included migrations. Updated pages and JS for new fields. * Updated TTL Counters to specify date resolution for TTL Counter entries, reducing memory footprint by allowing aggregation at broader date roundings (e.g. for long-term state, a daily entry expiry is sufficient). Updated VB.NET code wrappers for the new generic. * Updated TTL Counters to support an `EnableSum` flag, which uses a `TtlCounterDataValue` field to specify the value to sum. Implemented a dropdown in the page and JS that returns only integer and float data types for selection. * Updated the efficient no-boxing dictionary to instantiate on the basis of generic type, enabling use with both `int` (message pack) and `string` (core processing and legacy payload migrations). This touched a large number of files and is the primary reason for the size of this commit. Updated the dictionary to support addition of internal values via the interface for serialisation and mapping. Updated JSON serialisers to support the generic. * Implemented invocation mapping logic for insert and recall between the indexed Redis payload and the string-keyed payload required for processing. * Bumped the message pack envelope version flag to version 2 to reflect the new payload formats (noting that all data has been migrated and version 1 no longer exists in Redis). * Updated documentation to cover `LruJournals` in the Redis section, the `Cache` field on the `EntityAnalysisModelsRequestXPath` page, and the `ResolutionInterval` and `TtlCounterDataValue` fields on the `EntityAnalysisTtlCounters` page. |
||
|
richard-churchman
|
c1bb677026 |
Improved Case Automation POST body conext and other minor issues:
* Pushed Case Workflow Display HTML rendering logic down into a controller for improved security and consistent tokenisation of the Jube token format against context. Included a migration to update the default demo entry to use proper tokenisation. Updated the repository to support fetching CaseWorkflowDisplay with appropriate security. * Implemented improved tokenisation of the EntityAnalysisModelInstanceEntryPayload model via a common extension method on EntityAnalysisModelInstanceEntryPayload, used consistently across the system for serialisation and storage. * Included a migration to update the tokenisation format for full CaseWorkflowDisplay support. * Fixed an issue where CaseWorkflowFilter was incorrectly filtering on CaseWorkflowStatus.Guid instead of CaseWorkflowStatus.Name, causing the GUID to be returned rather than the display name on the template/demo model. (The remainder was fixed in GitHub issue #6.) Included a migration with a corrective SQL statement. * Fixed an issue where tokens were not being stored alongside CaseWorkflowFilter definition configuration. It now behaves identically to CaseSearch. Note: migrating missing tokens is not feasible as it requires JS-side parsing via JQueryBuilder. * Reduced the size of the CaseWorkflowForm invocation to include only the required fields given the new POST body context being rendered server side. Improved API serialisation by using a dictionary for form payload keys and values. * Consolidated JSON serialisation settings into DI and updated usages throughout. * Updated EntityAnalysisModelInstanceEntryPayload write serialisers to handle deserialisation, which is required to support tokenisation of new Case Automation POST bodies. * Created an extension method on the EntityAnalysisModelInstanceEntryPayload to centralise tokenisation of strings across Activation and Case Automation on a consistent basis. Simplified Case Notification and Case HTTP dispatch by removing tokenisation logic from those classes. * Removed a redundant field from the CaseWorkflowFormEntry POCO (database left unchanged for now). * Fixed a bug in Case session SQL rebuilding where the rebuild date was not being correctly checked for null or default value, which caused a failure in the last migration. * Made minor modifications to the Create Case process and method signature to skip serialisation when the payload object required for tokenisation is already available on the thread (e.g. during synchronous reprocessing). * Updated Case Workflow Macro processing to fetch the archive payload for the case from the database, improving security of that invocation endpoint. * Removed what appeared to be a duplicate EntityAnalysisModelInstanceEntryPayload model introduced during refactoring. * Fixed a but in Case Worfkflow Display Role assigment owing to volatile Guid in its update. |
||
|
richard-churchman
|
bf0faba627 |
Comprehensive Refactoring, Async/Cancellation Overhaul, and Engine Enhancements:
This commit consolidates extensive refactoring across the engine, orchestrators, background tasks, JSON pipelines, caching layers, messaging, and infrastructure. Major improvements include async correctness, cancellation propagation, architectural restructuring, performance optimizations, parser and engine enhancements, UI updates, deployment improvements, and critical bug fixes. Asynchronous, Cancellation, and Concurrency Improvements: * Converted all repository and controller methods to fully asynchronous implementations; removed remaining synchronous database calls. * Introduced centralized cancellation via a `CancellationTokenProvider` in DI and propagated tokens through database, engine, cache, and background processes. * Implemented graceful SIGTERM handling with coordinated shutdown. Most tasks are subject to immediate cancellation, while: - Draining AMQP consumers - Draining Kestrel in-flight requests - Draining async Archive writes - Draining async Case Creation processing * Added middleware to track active requests and emit `Connection: close` during shutdown. * Enhanced exception handling: - Cancellation exceptions logged as INFO - Improved logging for async call failures * Added `/ready` endpoint: - Returns 200 if the engine is ready, 503 otherwise - Integrated readiness signaling from TaskController and ActivationWatcher * Fixed concurrency/timing issues, including LRU cache updates using a semaphore instead of locks. * Ensured proper propagation of cancellation tokens across async call stacks, except where drainage is required. Engine, Orchestrator, and Pipeline Refactoring: * Rebuilt Invoke Orchestrator using a Context–Strategy–Pipeline hybrid pattern. * Extracted JSON parsing and database record parsing into dedicated helper classes. * Standardized instance object state management using a centralized Context object; removed private mutable state from orchestrators and engine. * Reorganized namespaces and moved background tasks to dedicated classes. * Removed god-class, organic-growth code and streamlined dependency flow; dependencies are now passed via Context only. * Added reference date integrity checks: - Prevent processing of future `ReferenceDate` values to prevent purge of records through bad data. - Updating of reference date ignored for records older than the latest entry to prevent invalid purges. * Introduced nested classes for large models and properties to improve readability. Moved very large property sets to nested classes for better context in engine pipelines. * Implemented rich extension functions library for parser and rule coder functionality: - Parser Scans new extensions assembly to add parser security tokens, enabling centralized development of extension functions to enhance rule capabilities. - Documented new tokens under Rule Compilation Tokens and Extensions. * Simplified `Invoke` data extraction logic via helper extension methods. * Centralized Context creation for JSON settings and contract resolvers. * Dropped legacy Response Payload approach, pending improved serialization to support pruning response payload. * Renamed `Report Keys` to `Archive Keys` for consistency with POCOs and database entities. * Reorganized engine files and namespaces to reflect new software architecture. * Implemented a bytes limit for payloads received over the invoke controller, controlled via the `MaxInvokeControllerRequestBytes` environment variable. Cache, Redis, TTL, and Background Task Enhancements: * Replaced Postgres `NOTIFY` in callbacks with Redis-based signaling given Postgres should not be interacted with during real-time flows — only Redis. * Improved callback timeout/removal logic and reduced initial invocation payload sizes. * Added environment variables for SearchKey cache job interval control: `SearchKeyCacheServerIntervalType` and `SearchKeyCacheServerIntervalValue`. * Refactored cache purge with observability tables and supporting migrations to enhance confidence in high-risk processing. * Converted TTL counters to `long` type with corresponding storage updates in the `ArchiveKeys` table. * Ensured only one TTL increment per invocation, irrespective of activation rules. * Updated Redis background scans from `HGETALL` to incremental `HSCAN` to prevent deadlocks and improve performance. - Removed obsolete payload-first cache tracking. * Improved initial LRU cache fill visibility via TaskController integration. * Refactored repository access in the Cache Service and audited DbContext disposal to prevent connection leaks. * Fixed counter metrics in the LRU cache. * Batched SortedSetRemoveAsync and HashDeleteAsync as it was causing long operations in Redis. AMQP, Threading, and Background Workflow Improvements: * Refactored AMQP messaging to resolve deadlocks in async consumers. * Moved RabbitMQ channel creation to consumer tasks and improved heartbeat handling to avoid deadlocks given `IModel` is not thread-safe. * Offloaded expensive case-creation processing to scalable background threads: - Reduces pressure on main transaction threads - Eliminates blocking in Archive storage and reduces memory leak risk (backlogged case creation has a much smaller footprint than backlogged Archive payload) - Threads controlled via the `CaseCreationThreads` environment variable * Introduced `TaskController` to manage all background (non-transactional) tasks. TaskController allows visibility of all background tasks. * Centralized cancellation and graceful shutdown across engine, background tasks, and web host. * On SIGTERM, if a task is not terminated within two seconds — perhaps due to drainage — it is written to console to give feedback on blocking tasks. * Implemented heartbeats in the connection to handle brittle connections and unforgiving .NET RabbitMQ clients, controlled by the `AMQPHeartbeatSeconds` environment variable. JSON, Parser, and Serialization Fixes: * Updated JSON settings, contract resolvers, and DI wiring. * Reduced payload response size on async HTTP invocation to only return the EntityAnalysisModelInstanceEntry GUID for callback, notwithstanding head check and parse still taking place. * Added `DBNull` checks for database led reprocessing deserialization. * Ensured the DictionaryNoBoxing converter is properly added to serialization settings to parse dates using the round-trip format. Performance, Stability, and General Cleanup: * Updated PooledDictionary to fix hash collisions and improve initial sizing. * Return null from PooledDictionary.cs instead of exception. * Ensured DB connections are disposed properly during model sync to prevent pool starvation in the new patterns. * Refactored SMTP client usage and added authentication/connection validations. * Removed unnecessary AutoMapper list mappings and streamlined repository configurations. * Renamed illogical methods and variables for clarity, as came across them. * Applied `ConfigureAwait` consistently across all projects. * Added `Async` suffix to async method names. * Replaced async unsupported `out` variables with tuples UI, Infrastructure, and Deployment Updates: * Updated UI to show case-creation, missing tagging and notification queue balance counters. * Removed obsolete test Blazor files (this project will go Angular). * Added production GC environment variables to `docker-compose.yml`. * Added `Jube.HealthCheck.csproj` for lightweight HTTP health checks in container (avoids installing `curl` as root). * Updated `docker-compose.yml` to bind port `5001` for job process readiness checks, as was already the case with the other process. * Implemented frame-busting on the login page to prevent the authentication page from being loaded in a frame after expiration. Critical Bug Fixes: * Fixed transaction buffer flushes on commit. * Fixed bulk write attempts on empty buffers. Miscellaneous Improvements: * Moved engine and model manager into a unified hybrid context with strategy & pipeline patterns. * Shifted thread pool configuration to startup pipeline. * Reorganized files and namespaces to align with the new architecture. * Improved observability and versioning for `Archive` and `ArchiveKey` tables to support reprocessing. * Dropped unused/complex payload-first cache tracking. * Many additional small bug fixes and refinements discovered during the grand refactor. * The Sandpit had some telemetry to a Chatbot and Google Analytics. Although it was disabled by default — and the Sandpit was intended for Jube use only — it should not have been in the public build. * Updated UI to display case-creation counters and missing tagging/notification fields. * Added a filter to remove instances from the synchronization page that have not sent a heartbeat. * Fixed invoke counter and removed the condition that counters are only written on invoke for more consistent system observability. * Updated the parser to support dates in inline functions, which was previously broken, which meant that dates not available. * Improved thread pool startup parameters to be a factor of two times logical cores for worker threads and four time logical cores for io threads. * Set command timout to zero for archive persistence given sensitivity. In time will create a dead letter archive for total failure, as on failure the records dropped. **Notes:** Functionality remains substantially the same where intended; most changes focus on stability, maintainability, async correctness, and operational robustness. |
||
|
Richard ChurchmanandGitHub
|
f695da9c47 |
Merge pull request #88
Refactors startup and DI, implements a local LRU cache to reduce Redis load, introduces high-performance dictionaries, and optimizes Invoke routing for parallelism. Includes Redis and library upgrades, schema migrations, logging improvements, and overall code cleanup for better performance, readability, and maintainabilit |