How to read large files with PHP.

In this guide, we will show you how to read large files using PHP.

To do this, we will be using the fopen and fgets functions.

file_get_contents throws a memory exhausted error.

The problem with file_get_contents is that it will attempt to read the entire file into a string.

This is fine for most files.

However, with larger files you might encounter fatal errors about the allowed memory size being exhausted:

//Attempting to load a large log file
//using file_get_contents
$contents = file_get_contents('large.log');

In my case, “large.log” was 4.5GB in size.

As a result, PHP coughed up the following fatal error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 529084788 bytes) in file.php on line 3

This happened because file_get_contents attempted to read the entire file into memory. However, it failed because the default memory limit in PHP is 128MB, which is a lot smaller than 4.5GB!

Reading large files using fopen and fgets.

A better approach to larger files is to read them line by line.

To do this, we can use PHP’s filesystem functions. Namely: fopen, fgets, and feof.

//The path to the large file.
$fileName = 'large.log';

//Open the file in "reading only" mode.
$fileHandle = fopen($fileName, "r");

//If we failed to get a file handle, throw an Exception.
if($fileHandle === false){
    throw new Exception('Could not get file handle for: ' . $fileName);
}

//While we haven't reach the end of the file.
while(!feof($fileHandle)) {
    //Read the current line in.
    $line = fgets($fileHandle);
    //Do whatever you want to do with the line.
}

//Finally, close the file handle.
fclose($fileHandle);

In the PHP above:

  • We specified the path to our large file. In this case, “large.log” is located in the same directory as our PHP script.
  • Using the fopen function, we opened the file in “reading only” mode and retrieved a file pointer resource.
  • If the fopen operation fails, we throw an Exception.
  • We then started a while loop, which will continue until it detects the end of the file. We use the PHP function feof to test for an end-of-file. If feof returns true, then we know that we are at the end of the file.
  • Inside our while loop, we use the function fgets to retrieve the current line of the file. Our while loop will then iterate to the next line of the file.
  • At the end of the script, we used the function fclose to close the file pointer that we opened. This is important, as leaving a file open can lead to issues such as file corruption.

Fatal error: Maximum execution time of 120 seconds exceeded.

In some cases, you might run into issues with the maximum execution time being exceeded.

By default, a PHP script will only run for 120 seconds. After that, a fatal error will occur and the script will die.

On my local PC, it took nearly 400 seconds for PHP to read a 4.5GB file.

To get around this timeout issue, I had to disable PHP’s maximum execution time by providing the set_time_limit function with 0:

//Added to the top of my script, BEFORE
//any file reading is done.
set_time_limit(0);

Note that you should exercise extreme caution with this approach, as you do not want to end up in a situation where your PHP scripts are hanging because they’re stuck reading huge files.

If your files are so large that they take longer than 120 seconds for PHP to read, then you might want to rethink your entire design. That, or you might need a faster server.

Basically, don’t go disabling your time limits without putting some real thought into it.