Reading large files with PHP.

This is a guide on how to read large files with PHP. To do this, we will be using the fopen and fgets functions.

file_get_contents throws a memory exhausted error.

The problem with file_get_contents is that it will attempt to read the entire file into a string. This is fine for most smaller files. However, on larger files you might encounter fatal errors about the allowed memory size being exhausted:

In my case, “large.log” was 4.5GB in size. As a result, PHP coughed up the following fatal error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 529084788 bytes) in file.php on line 3

This happened because file_get_contents attempted to read the entire file into memory. However, it failed because the default memory limit in PHP is 128MB, which is a lot smaller than 4.5GB!

Reading large files using fopen and fgets.

A better approach to larger files is to read them line by line. To do this, we can make use of PHP’s filesystem functions. Namely:  fopen, fgets and feof.

In the PHP above, we:

  • Specified the path to our large file. In this case, “large.log” is located in the same directory as my PHP script.
  • We opened the file in “reading only” mode and retrieved a file pointer resource. We did this by using the fopen function.
  • If our fopen operation fails, we throw an Exception.
  • We then started a while loop, which continues indefinitely until the end of the file has been detected. We use the PHP function feof to test for an end-of-file. If feof returns true, then the end-of-file has been reached and the while loop will stop executing.
  • Inside our while loop, we use the function fgets to retrieve the current line of the file. Our while loop will then iterate to the next line of the file.
  • At the end of the script, we used the function fclose to close the file pointer that we opened. This is important, as leaving a file open can lead to issues such as file corruption.

Fatal error: Maximum execution time of 120 seconds exceeded.

In some cases, you may run into issues with the maximum execution time being exceeded. By default, a PHP script will only run for 120 seconds. After that, a fatal error will occur and the script will be killed off.

On my local PC, it took nearly 400 seconds for PHP to read a 4.5GB file.

To get around this timeout issue, I had to disable PHP’s maximum execution time by providing the set_time_limit function with 0:

Note that you should exercise extreme caution with this approach, as you do not want to end up in a situation where your PHP scripts are all hanging because they’re stuck reading huge files. If your files are so large that they take longer than 120 seconds for PHP to read, then you might want to rethink your entire design. That, or you might need a faster server.

Basically: Don’t go disabling your time limits without putting some real thought into it.

Hopefully, you found this guide helpful!

Facebook Comments