Bulk imports in PHP often begin with a CSV uploaded by a client, an export from a provider, or an extract from a legacy system. The risk arises when they are treated as a simple file read followed by database inserts. A row may have a valid format and still create a duplicate, violate a business rule, overwrite current information, or trigger an external effect twice.
An import must be designed as an operational process with a defined lifecycle: intake, analysis, validation, preview, confirmation, execution, review, and recovery. This approach allows product to understand what will be incorporated, operations to intervene in the event of exceptions, and technology to limit the impact of defective data.
Treat the import as a business process
The file is not the source of truth by itself: it is a request to modify the application's state. Therefore, it is advisable to create an import entity with its own identifier, the user or system that initiated it, receipt date, data type, contract version, file or secure reference to it, overall status, and results summary.
Overall statuses must express an actionable situation, not merely a boolean. For example: received, analyzed, awaiting confirmation, in progress, completed, completed with issues, stopped, or canceled. An import with 9,800 accepted rows and 200 rejected rows is not necessarily a failure; it may be an execution completed with issues if the rejected rows are isolated and explained.
You must also decide which consequences belong to the load. Creating an order, updating a catalog, or adding contacts may require calculations, auditing, or notifications. Separating primary persistence from side effects reduces the risk that a retry sends repeated messages or runs integrations in an uncontrolled manner.
Define an input contract before accepting files
The import contract specifies what is expected to be received and how it will be interpreted. It must include the allowed format, encoding, delimiter, headers, data types, required fields, date format, normalization rules, size limits, and maximum number of rows. If spreadsheets are accepted, the relevant sheet and how empty cells, formulas, and automatically converted values will be handled must also be defined.
The fields that connect the source to the destination deserve special attention. A stable external identifier, such as the customer code in the source system, is preferable to using the line number or a name as a reference. It must be clarified whether that identifier creates a record, updates an existing one, or whether both operations are allowed.
Separate validation layers
Structural validation answers mechanical questions: can the file be read? Are the required columns present? Does the date have an accepted format? Is the amount numeric? Does the row comply with the length limit? This layer should detect early problems that prevent the data from being interpreted.
Domain validation applies business rules: a status may not be allowed, an end date cannot precede a start date, a percentage must remain within its range, or a combination of fields may be incompatible. Finally, checks against existing data verify references, permissions, uniqueness, and allowed transitions. For example, that a supplier code exists, that the user can operate on that organization, or that a record is not locked.
This separation improves messages and diagnostics. Reporting a missing column is not the same as reporting a nonexistent reference or an unauthorized change. In addition, domain rules must be reused by the application and the importer to prevent the load from becoming a shortcut that bypasses normal controls.
Preview changes and confirm explicit intent
The preview should not promise an exact execution if data can change between analysis and confirmation, but it should provide a verifiable estimate. Show the total number of rows read, valid rows, rows with warnings, rejected rows, expected creates, expected updates, and records that will not change.
Warnings are for cases that require attention but do not automatically invalidate a row: for example, a normalized phone number, a description truncated according to a known rule, or an empty optional field. Warnings must not hide rejections. Each result needs a stable code, an understandable message, and, where safe, the received value and the normalized value.
Confirmation must be associated with a specific version of the analysis. If the user replaces the file, corrects rows in an interface, or changes relevant parameters, the system must invalidate the previous preview and require a new analysis. This prevents confirming a summary that no longer represents the actual load.
Process in batches without ambiguous states
Processing all rows within a single transaction may seem safe, but it can hold locks for too long, exceed execution limits, or turn an isolated failure into a costly rollback. Processing one row per transaction, on the other hand, can generate too much overhead and make it difficult to coordinate related operations.
The unit of work must be chosen based on the dependency between records, volume, and cost of reverting. In many cases, a small, bounded batch allows changes to be committed progressively. Each batch must record its start, completion, number of rows handled, and result. The worker must be able to resume without depending on an open HTTP session: the load is confirmed from the interface, but it is executed as a background job.
Avoid keeping entire files in memory. Read sequentially, normalize each row, and store a working representation or a validation result when necessary for auditing and resuming. Enforce limits on size, rows, time, and concurrency. An unexpectedly large file must not block the resources that handle daily work.
for each pending batch:
mark batch as in_progress
for each row in the batch:
apply pending validations
persist or send to quarantine
record result per row
commit batch
mark batch as completedIf a worker is interrupted, it is not enough to blindly run the batch again. An expiring or recoverable locking mechanism is needed, and the status per row must make it possible to distinguish what is pending from what has already been committed.
Isolate errors in quarantine and preserve evidence
Quarantine allows invalid rows not to block valid ones without disappearing from the process. A quarantined row must retain the source number or identifier, the received data under access controls, normalized data if available, error codes, time of detection, and review status.
Not all errors receive the same treatment. A file without a required header is a file error and may stop the entire analysis. A reference to a nonexistent supplier may be a row rejection. A temporary failure of the database or an integration is a retryable technical error and must not be labeled as a data defect.
The operational interface must allow filtering by reason, exporting only authorized issues, and understanding what correction is expected. Correcting within the application may be useful for a few cases; for many records, it is usually more controllable to download the issues, correct them at the source, and submit a new load. In both cases, retain the history: modifying a quarantined row must not erase the original value or the initial reason.
Ensure idempotency and prevent duplicates
Idempotency means that repeating an operation with the same intent does not alter the result more than once. It is essential because retries occur: a timeout expires, a process restarts, or an operator confirms again after an uncertain response.
An idempotency key can be built from the import type, destination organization, and a stable external identifier. It must be backed by database uniqueness constraints when the model allows it; checking first and then inserting does not eliminate race conditions between concurrent workers.
For updates, define an explicit policy: replace fields, apply only non-empty values, reject conflicts, or require an expected version of the record. The latter helps detect that a user changed the data after the preview. Do not use a file fingerprint as the only mechanism: the same content may represent a different intent, and a corrected file may retain many rows that have already been processed.
Traceability, retries, and selective recovery
The result per row is the central piece for support and recovery. Record a status such as pending, processed, rejected, quarantined, retry pending, or skipped; the external identifier; the batch; reason codes; timestamps; and a reference to the created or updated record. Protect personal data and secrets: traceability must be sufficient to investigate, not an indiscriminate copy of sensitive information in logs.
Retries must be selective. Automatically retry transient technical errors with limits and progressive backoff; do not retry a violated domain rule indefinitely. When a missing reference or invalid data is corrected, reprocess only the corresponding quarantined rows. When a batch fails, continue from pending rows and use those already processed as evidence of progress.
Before putting the workflow into production, test empty files, altered headers, unexpected encodings, duplicates within the same file, duplicates against the database, interruptions between batches, resumptions, and insufficient permissions. Bulk imports in PHP are reliable when their behavior on failure is designed with the same precision as their successful path.
