A data export seems straightforward while the result fits in memory and can be generated in a few seconds. As the volume, data sensitivity, or number of simultaneous requests grows, sending the response directly can exhaust resources, exceed execution limits, and leave users unsure about what happened. Designing a data export in PHP means deciding how to produce it, protect it, and communicate its status—not just how to write a CSV.
When to stop generating the export in the request
Synchronous export can be suitable for small, bounded datasets that can be processed quickly. The application validates the request, queries the data, and returns the file in the same response. This is easy to understand and avoids managing background jobs and files afterward, but it ties response time to the cost of querying and serializing all records.
Consider switching to an asynchronous process when the duration is variable or long, the volume may grow, relevant time or memory limits apply, or simultaneous exports compete with interactive requests. It is also preferable when users need to start the job and come back later. There is no universal threshold: measure duration, peak memory, output volume, and the impact under concurrency in the real environment.
Synchronous processing is still reasonable if a clear limit can be enforced and response time is acceptable. Another option is to offer both: immediate downloads for small datasets and background generation for large requests. Limits should be explicit and communicated before the job starts, not appear as an unexpected error at the end.
Choose the format and delivery method for the use case
The format depends on the consumer. CSV is often practical for spreadsheets and simple integrations; JSON may suit consumers that need nested structures. If multiple files, data types, or metadata are required, packaging them may be appropriate. Consider size, compatibility, encoding, and representation rules, including delimiters, dates, time zones, and null values.
Define the export contract: columns and their order, applied filters, date formats, character handling, and the meaning of empty values. If the file will be opened in a spreadsheet, also assess the risk that user-controlled values could be interpreted as formulas. Mitigation depends on the format and consumer; do not silently alter data without documenting the behavior.
For a direct download, PHP can stream the content progressively if the query and format allow it. For large jobs, it is usually more controllable to generate a temporary file and offer it when complete. Separating generation from download makes it possible to show progress and avoids delivering a response that gets cut off midway, though it requires storage, expiration, and permission management.
Design an observable asynchronous workflow
A typical workflow includes these steps:
- Request: validate filters, format, and scope; create a job identifier and record who requested it.
- Authorization: check that the person can export the requested dataset, including its filters and sensitive fields.
- Generation: run the job in the background, record errors, and write to a non-public location.
- Availability: mark the file as ready only after writing is complete and has been verified.
- Download and expiration: check access again, serve the file, and delete it according to the defined policy.
Statuses should be understandable and queryable, for example: pending, in progress, ready, failed, and expired. Include actionable messages without revealing internal details. If useful, track progress through processed chunks rather than fictitiously precise percentages. The interface should distinguish a job that is still active from one that failed and allow users to request another generation according to the product policy.
The authorized identity and scope must accompany the job. Do not assume that a hard-to-guess identifier provides authorization. When checking status or downloading, verify job ownership and current permissions. Also define what happens if permissions change while the file is being generated: for sensitive data, it may be necessary to revalidate before download or cancel revoked jobs.
Process in chunks without exhausting memory
Avoid loading the entire result into an array before serializing it. Query records in ordered chunks and write each chunk to a stream, releasing references before continuing. In PHP, paginated queries or iterators can help, but their behavior depends on the database engine and driver: a seemingly iterative query may still buffer results on the client. Verify memory usage against the expected volume.
Offset pagination can become expensive with large datasets. Where appropriate, use keyset pagination, with a stable order and an unambiguous continuation column. Define what happens if the data changes during the export: a consistent snapshot may require a transaction or a specific strategy, with locking and duration costs that need to be evaluated. If a changing view is acceptable, document that semantics.
Write to a temporary file with a non-predictable name and restrictive permissions, outside the public root. Check for errors when opening, writing, and closing, as well as for available disk space. A write failure must not make a partial file downloadable. You can generate the file under a temporary name and mark it as final through a safe operation once writing is complete, within the guarantees provided by the chosen storage.
Protect downloads, expiration, and recovery
Downloads should go through an authenticated route that checks status, authorization, and expiration. Avoid building file paths from user-supplied parameters; resolve the job identifier through server-controlled metadata. If object storage is used, manage temporary access with appropriate restrictions and ensure that the URL does not replace the workflow's authorization checks.
Set a retention policy appropriate to the sensitivity, size, and user needs. A cleanup process should delete both expired files and orphaned temporary files, and update the associated status. Record who requested and downloaded an export when traceability is relevant, while avoiding storing exported data or secrets in logs.
When a failure occurs, record the operational cause and leave the job in a consistent state. Retries can duplicate costs or produce duplicate files; use job identifiers and idempotency rules to decide whether to resume safe generation or start again. Do not blindly append to a partial file: delete it or isolate it, and publish only complete output. Limit retries and define how abandoned jobs are recovered.
Resuming from a checkpoint requires more than simply retrying the job. Durably store the last confirmed chunk and a stable continuation key—for example, the last key processed in a deterministic order—along with the filters and job identity. On restart, validate that those parameters have not changed and continue from the next key. To avoid publishing inconsistent output, write confirmed chunks to temporary parts identified by the job and assemble the final file only when all parts are complete. If the format or storage cannot safely confirm and verify those parts, or if you cannot guarantee a consistent view of the data, discard the partial output and regenerate the file from the beginning. Controlled regeneration is usually simpler and safer than incorrect resumption.
Testing and production checklist
Test both the content and the lifecycle. Check that filters, permissions, and exported fields are correct; that a user cannot query or download other users' jobs; and that expiration prevents access. Include empty datasets, special characters, large values, and records containing sensitive data. Verify the format with the actual consumer where possible.
Simulate database errors, a full disk, interruption during writing, process loss, and repeated requests. Confirm that incomplete files are not published, retries do not unnecessarily duplicate work, and cleanup removes leftovers. If checkpoints are implemented, test restarts at every chunk boundary, detection of incompatible parameters, and final assembly. Measure memory, duration, and load under representative concurrency; also monitor the job queue, pending storage, and age of active exports.
- Define size, duration, and concurrency limits.
- Authorize filters, fields, status checks, and downloads.
- Process and write in chunks; measure actual memory usage.
- Publish only complete files and protect their location.
- Report statuses, recoverable errors, and expiration.
- Plan idempotent retries and automatic cleanup.
- Use checkpoints only if you can confirm chunks and resume with consistent parameters and data.
- Test permissions, partial failures, consistency, and load.
The main decision is not simply synchronous versus asynchronous: it is what guarantees the product can offer for waiting time, consistency, privacy, and recovery. Making those guarantees explicit makes it possible to choose a PHP implementation appropriate to the current volume, with limits and signals to evolve it before an export degrades the rest of the application.
