Task 47 — Retry, timeout, circuit breaker

Phase: 4 Initiative: 19 Status: done Owner: — Slug: retry-timeout-breaker

Goal

Peer-вызовы защищены deadline-aware retries, exponential backoff с jitter, per-peer circuit breaker’ами и bulkheads. Non-idempotent операции не ретраятся вслепую и требуют idempotency keys.

Acceptance criteria

• ResilienceConfig section in MeshConfig
• Jitter in RetryPolicy.GetDelay() + MaxDelay enforcement
• ResilientTransport wrapper: CB check → retry loop with jitter → bulkhead
• CircuitBreaker configured from ResilienceConfig.FailureThreshold
• RetryPolicy configured from ResilienceConfig
• DI registration in MeshServiceExtensions
• CircuitBreaker integrated into FanOutOrchestrator (peer filtering)
• appsettings.json updated with Resilience section
• Unit tests for CircuitBreaker, RetryPolicy, ResilientTransport
• dotnet build + dotnet test pass

Sub-tasks

• Add ResilienceConfig to MeshConfig in AppConfig.cs
• Add jitter to RetryPolicy.GetDelay() (deterministic + random)
• Configure CircuitBreaker from ResilienceConfig in DI
• Configure RetryPolicy from ResilienceConfig in DI
• Create ResilientTransport wrapper (src/agent/Mesh/Resilience/ResilientTransport.cs)
• Implement bulkhead pattern via per-peer semaphore in ResilientTransport
• Integrate CircuitBreaker into FanOutOrchestrator peer filtering
• Add Resilience section to appsettings.json
• Write unit tests in tests/Hercules.Agent.Tests/Mesh/Resilience/
• dotnet build src/agent/Hercules.csproj — succeed
• dotnet test --filter "FullyQualifiedName~Resilience" — pass

Scope / Likely files

src/agent/Mesh/Resilience/

Dependencies

• блокирует / опирается на: task_037 — transports
• блокирует / опирается на: task_040 — trust-admission

Risks / Rollback

Retry storm; per-peer rate limit + наблюдаемое состояние breaker’ов.

Validation

dotnet build src/agent/Hercules.csproj                          → 0 errors
dotnet build tests/Hercules.Agent.Tests/Hercules.Agent.Tests.csproj → 0 errors
dotnet test --filter "FullyQualifiedName~Resilience" --no-build  → 18/18 pass
dotnet test tests/Hercules.Agent.Tests/Hercules.Agent.Tests.csproj → 1307/1316 pass
                                                              (9 pre-existing failures:
                                                               OtelServiceTests ×5, NumericValidatorTests ×2,
                                                               BudgetGuardTests ×1, WasmToolTests ×1)

Links

• Backlog: ../backlog.md
• Roadmap (EN): ../../ROADMAP-EN.md
• Roadmap (RU): ../../ROADMAP-RU.md