Retries Are Not Enough: Designing Async Job Reconciliation in PHP

Asynchronous jobs decouple imports, synchronizations, notifications, document generation, and integrations. However, the fact that a consumer has processed a message does not necessarily prove that the business outcome is correct. Success may have been recorded before an external effect was confirmed, a failure may have occurred between two steps, or the same operation may have run more than once.

Asynchronous job reconciliation in PHP addresses that gap: it compares what the system expected to achieve with evidence of what happened, detects missing outcomes or discrepancies, and triggers a controlled correction. It does not replace the queue, retries, or idempotency; it complements them with an independent verification.

A technical execution is not the same as a business outcome

A task can finish without an exception and still leave a process incomplete. For example, an application creates a synchronization request, the consumer calls an external API, and receives an inconclusive response due to a network interruption. If it retries without an idempotency key, it may create a duplicate. If it assumes success, it may leave the record unsynchronized.

Nor should a specific delivery semantic be taken for granted from the messaging infrastructure. The possibility of redeliveries and duplicate executions depends on the broker, its persistence configuration, acknowledgments, consumer behavior, and the failures that occur. The design must verify these properties in the chosen technology and, when duplicates or reordering can occur, explicitly tolerate them.

The operational question is not only “was the message consumed?”, but “can I prove that the expected effect exists, exactly once when applicable, and with the correct data?” That proof requires a source of evidence: a queryable response from the external system, a persisted remote identifier, a stored document, or a confirmed state change.

Retries, idempotency, and reconciliation: different responsibilities

Retries address transient errors: temporary unavailability, usage limits, brief locks, or network issues. It is advisable to define a retry limit, progressive delay, error classification, and a destination for messages that require attention. Retrying indefinitely can hide a data error or worsen an external incident.

Idempotency makes it safe to repeat an operation. It can be achieved with a stable operation identifier sent to an external provider, a unique database constraint, or a transactional check before the effect. It does not mean that the effect occurred: it means that repeating it should not multiply it.

Reconciliation looks for pending, incomplete, or contradictory operations and decides what to do with each one. It is especially necessary when there are external effects, batch processes, updates across several systems, or communications whose receipt cannot be proven from the sending application alone.

• Use retries to try again after failures classified as transient.
• Use idempotency to prevent retries or redeliveries from duplicating effects.
• Use reconciliation to verify the final state and repair detected differences.

Model the operation and retain verifiable evidence

A maintainable design separates three concepts. The requested job represents the intent, for example, “synchronize order 452.” The expected effect defines the observable outcome: “the external system contains the order with version 7.” The confirmation stores evidence that the outcome exists: remote identifier, version, timestamp, validated response, or the result of a subsequent query.

Before publishing a message, create an execution record in a durable database. If the application modifies its own data and publishes a message, consider the outbox pattern: store the business change and the pending event in the same transaction, and delegate publication to a later process. This reduces the risk of confirming the local change and losing the message, or publishing a message for a change that was rolled back.

The record must include, at a minimum:

• An immutable and unique operation_id, used to correlate messages, logs, and external calls.
• Operation type, affected entity, and version or fingerprint of the expected content.
• Current status, attempt count, next allowed attempt, and timestamps.
• Idempotency key and, if it exists, the remote resource identifier.
• Summarized evidence and secure references to responses or errors, without logging secrets or unnecessary personal data.
• Reason for closure, compensation, discard, or escalation to human review.

Define explicit transitions, for example: pending, processing, awaiting_confirmation, confirmed, retry_scheduled, manual_review, compensated, and not_applicable. Each transition must have an owner and a verifiable condition. A conditional update, such as moving to processing only if the previous status is pending, reduces race conditions between consumers.

Build the reconciliation process

Reconciliation can run through a scheduled PHP command, a dedicated worker, or an operational workflow. It must work with time windows: do not inspect operations created seconds ago if the external integration normally takes several minutes. Define the window using real latency data and review it when limits or providers change.

For each eligible operation, compare previously defined sources of truth. The local database may be authoritative for intent and the data version; the external system, for whether it received or created the resource. When no reliable query to the destination exists, evidence may be a signed acknowledgment, a provider identifier, or a deferred check through a result file.

1. Select unconfirmed operations that exceed their expected timeframe.
2. Check whether the effect exists using operation_id, the idempotency key, or an unambiguous business key.
3. Compare relevant fields and versions, not only the existence of the resource.
4. Classify the case as missing, correct, divergent, ambiguous, or not applicable.
5. Perform the authorized action and store the decision with its evidence.

An ambiguous outcome must not automatically become a requeue. If a call may have created a resource but there is no reliable way to query it, retrying could duplicate a charge, notification, or document. In those cases, block the automated action and send the case to an exception dashboard with enough context to decide.

Correct without introducing new harm

The action depends on the discrepancy and the cost of being wrong. Requeueing is appropriate when the effect is missing and the operation is idempotent. Compensating can reverse an incorrect effect through an explicit business operation, not through indiscriminate technical deletion. Marking for review is preferable in the face of ambiguity, version conflicts, or financial consequences. Closing as not applicable is useful when the entity was canceled or superseded according to documented rules.

Manual repairs must also leave an audit trail: who made the decision, what evidence they consulted, what action they applied, and what the outcome was. Limit permissions and avoid buttons that execute an operation without showing the entity, version, destination, and risk of duplication.

Observability and tests that validate the design

Logs correlated by operation_id make it easier to follow an operation across the web application, workers, and external services. Useful metrics are not limited to exceptions: measure the age of pending operations, the number in manual review, the divergence rate, retries by cause, and time to confirmation. Alerts should trigger on accumulation, age, or missed deadlines, not on every isolated error.

Test representative failures: a crash after the external effect and before persisting the confirmation; duplicate execution; an out-of-order message; worker restart; a timeout with an uncertain remote outcome; prolonged unavailability; and version changes while an operation remains pending. The test must verify both the final state and the absence of duplicates and the quality of the stored evidence.

Example: synchronizing a record with an external system

Suppose a PHP application synchronizes a customer record. When it is modified, it creates the sync_customer operation with a stable identifier and the expected local version. The worker sends those values to the destination as an idempotency key. If it receives valid confirmation, it persists the remote identifier and changes the status to confirmed.

If the timeout occurs after sending the request, the worker leaves the operation in awaiting_confirmation. The reconciler queries the destination by the idempotency key. If it finds the same version, it confirms. If it does not find it, it schedules another send. If it finds a different version, it marks the case for review instead of overwriting data that may have been legitimately modified in the other system.

Checklist for adding reconciliation without rewriting the process

• Inventory existing jobs and prioritize those that produce external effects, affect money, regulatory data, or processes that are difficult to repeat.
• For each type, document the expected effect, the source of truth, and the evidence that will allow it to be confirmed.
• Verify in the specific broker and consumers what happens in the event of crashes, late acknowledgments, persistence, redelivery, and message ordering.
• Add a stable operation_id and propagate it in the message, logs, external calls, and status records.
• Introduce an operations table with statuses, attempts, deadlines, idempotency key, and evidence; start in observation mode if necessary.
• Define conditional transitions and a written policy for retrying, confirming, compensating, escalating, or closing as not applicable.
• Implement a reconciler limited to a time window and a pilot operation type.
• Validate with duplicate, ambiguous timeout, crash between steps, reordering, and restart cases before automating corrections.
• Create an exception dashboard or query with age, entity, evidence, and recommended action.
• Periodically review metrics, stalled operations, and manual decisions to adjust deadlines, rules, and controls.

Gradual adoption makes it possible to improve reliability without replacing the entire architecture: first make uncertain operations visible, then confirm outcomes, and finally automate only the corrections whose safety you can prove.