Building a Reliable Workflow Engine in Java | Atomic Claims & Worker Recovery | Part 8

Continuing to build my own n8n-inspired workflow automation engine using Java and Spring Boot — Part 8. In this part, I focused on making the workflow execution system more reliable by introducing database-backed execution ownership, atomic claiming, and startup recovery. A queue can deliver execution IDs to workers, but what happens when multiple workers try to process the same execution? What happens when the application restarts while executions are waiting in the queue? These are the reliability problems I started addressing in this part. WHAT I WORKED ON 1. Atomic Execution Claiming Implemented a conditional database update so a worker can claim an execution only when its status is QUEUED. Workers skip executions that have already been claimed or are no longer eligible for processing. 2. Database-Backed Execution Ownership Added workerId and leaseExpiresAt fields to the Execution entity to begin tracking which worker owns an execution and when its lease expires. 3. Lease-Based Worker Design Started implementing lease renewal using ScheduledExecutorService, with a planned 30-second heartbeat and a two-minute lease. Lease renewal still needs complete implementation and testing. 4. Startup Recovery Created QueuedExecutionRecoveryService to find queued executions in PostgreSQL when the application starts and re-enqueue their IDs for processing. 5. Queue-Full Handling Added handling for a full execution queue. Rejected executions are marked FAILED, the failure timestamp and result are recorded, and the API returns HTTP 503 Service Unavailable. 6. Worker and Execution Engine Architecture Continued separating execution responsibilities between the bounded LinkedBlockingQueue, ExecutionWorker, and ExecutionEngine while preserving concurrent node execution. ARCHITECTURE Workflow API → WorkflowService → PostgreSQL → Execution Queue → ExecutionWorker → Atomic Execution Claim → ExecutionEngine WHY THIS MATTERS Building a background execution system involves more than running tasks asynchronously. It also requires coordinating ownership, preventing duplicate claims, handling application restarts, and managing failures. This work moves my project toward a more reliable distributed-worker design, although ownership-safe completion and complete lease recovery still need to be implemented and tested. WHAT'S NEXT? Finish lease renewal, ensure only the owning worker can mark an execution SUCCESS or FAILED, recover expired leases safely, and address the race between saving an execution and adding it to the in-memory queue. TECH STACK Java, Spring Boot, PostgreSQL, Hibernate, JPA, ExecutorService, ScheduledExecutorService, LinkedBlockingQueue, REST APIs. I'm building this project in public to learn backend development through practical implementation and understand how real software systems handle concurrency, scheduling, and reliability. Follow along as I continue building, testing, debugging, and improving the workflow engine. #Java #SpringBoot #BackendDevelopment #WorkflowAutomation #JavaConcurrency #PostgreSQL #DistributedSystems #SystemDesign #SoftwareEngineering #BuildInPublic