PHP copy/rename from FTP folder corrupts files when upload is in progress
This took weeks to troubleshoot so maybe I can save you some time by relating the experience...
I have an application that reads files from a FTP drop folder and moves them to a queue folder. If the move is successful, a queue record is put into the database.
Now, you would have thought that is_file() would have been enough to detect if a file was ready (ie: not still being uploaded by a user). As it happens, the FTP server is 'clever' enough to give you read access during the upload, allowing you to see its progress. And so my queue script was copying and unlinking files that were incomplete.
So, I tried is_readable() - same thing.
Then I added is_writeable() - you'd think this would do it. Surely files were locked during an upload? Nope, same problem.
Then I tried doing a filesize() followed by a sleep then another filesize() and compared them. Didn't work. Then I added a filesize() > 0 check. No joy. I also made sure the stat data wasn't being cached by calling clearstatcache() each time. No difference.
Finally, the only thing that seemed to work (and yes I am still uncertain the problem has completely gone away!) is comparing a stat() on the file with a sleep() in between.
Anyway, here's the final moveFile function that appears to be working for now...
I have an application that reads files from a FTP drop folder and moves them to a queue folder. If the move is successful, a queue record is put into the database.
Now, you would have thought that is_file() would have been enough to detect if a file was ready (ie: not still being uploaded by a user). As it happens, the FTP server is 'clever' enough to give you read access during the upload, allowing you to see its progress. And so my queue script was copying and unlinking files that were incomplete.
So, I tried is_readable() - same thing.
Then I added is_writeable() - you'd think this would do it. Surely files were locked during an upload? Nope, same problem.
Then I tried doing a filesize() followed by a sleep then another filesize() and compared them. Didn't work. Then I added a filesize() > 0 check. No joy. I also made sure the stat data wasn't being cached by calling clearstatcache() each time. No difference.
Finally, the only thing that seemed to work (and yes I am still uncertain the problem has completely gone away!) is comparing a stat() on the file with a sleep() in between.
Anyway, here's the final moveFile function that appears to be working for now...
protected function moveFile($from, $to){
clearstatcache();
$this->write2log("Called moveFile ".$from. " > ".$to);
$this->write2log("File ".basename($from). " is " . intval(filesize($from)/(1024*1024/100))/100 ."MB");
if (is_file($from)){
if (is_readable($from)){
if (is_writable($from)){
//check for filesize change
$size = filesize($from);
$stat = stat($from);
sleep(5);
clearstatcache();
if ($size == filesize($from) && $size > 0 && $stat == stat($from)){
if (copy($from, $to)){
if (unlink($from)){
$this->write2log("Successfully moved ".$from. " > ".$to);
return true;
}
$this->write2log("FAILED to unlink ".$from);
}
$this->write2log("FAILED to copy ".$from. " > ".$to);
}
$this->write2log("UPLOAD IN PROGRESS ".$from);
}
$this->write2log("NOT WRITABLE ".$from);
}
$this->write2log("NOT READABLE ".$from);
}
$this->write2log("NOT A FILE ".$from);
return false;
}
Share your thoughts