Building a Reliable Workflow Engine in Java | Worker Leases & Recovery | Part 9

Continuing to build my own n8n-inspired workflow automation engine using Java and Spring Boot — Part 9. In this part, I continued improving the reliability of the execution engine by working on worker leases, ownership-aware execution completion, stale-lease recovery, and cancellation when a worker loses ownership. A background worker system needs to handle more than successful execution. It must also account for workers that crash, lose ownership, or leave executions unfinished. WHAT I WORKED ON Worker Lease Renewal Implemented a heartbeat mechanism using ScheduledExecutorService. The worker renews its lease every 30 seconds, extending the lease by two minutes. Ownership-Aware Execution Completion Added a conditional database operation that checks the execution ID, RUNNING status, worker ownership, and lease validity before recording the terminal execution status. Stale-Lease Recovery Implemented an atomic recovery operation to mark eligible executions with expired leases as FAILED, record the recovery reason, and clear the previous worker ownership. Periodic Recovery Scheduling Updated the execution recovery service to check for expired leases periodically using Spring's @Scheduled annotation, with scheduling enabled through SchedulingConfig. Cooperative Cancellation Connected the worker's lease-validity flag to the ExecutionEngine so that the engine can stop scheduling additional nodes after lease loss. CURRENT ARCHITECTURE Workflow API → WorkflowService → PostgreSQL → Execution Queue → ExecutionWorker → Atomic Claim and Lease → ExecutionEngine → Ownership-Safe Completion Recovery services help reconcile persisted execution state with the in-memory queue and worker processing. WHY THIS MATTERS A worker should not be allowed to complete an execution after losing ownership. Expired leases also need to be handled so that abandoned executions do not remain RUNNING indefinitely. These mechanisms are important building blocks for a more reliable background processing system. WHAT'S NEXT? The next priorities are preventing duplicate queue entries, completing periodic queued-execution recovery, closing the database-to-queue submission gap, and testing worker crashes, lease expiration, and competing workers. The latest changes still need a full project test run, so the next step is to validate the implementation and fix any regressions. TECH STACK Java, Spring Boot, PostgreSQL, Hibernate, JPA, ExecutorService, ScheduledExecutorService, LinkedBlockingQueue, REST APIs. I'm building this project in public to learn backend development through practical implementation and understand how real software systems handle concurrency, scheduling, worker ownership, and failure recovery. Follow along as I continue building, testing, debugging, and improving the workflow engine. #Java #SpringBoot #BackendDevelopment #WorkflowAutomation #JavaConcurrency #PostgreSQL #DistributedSystems #SystemDesign #SoftwareEngineering #BuildInPublic