Background:
So resizing large (many pixels) images takes much memory because the image must be expanded into memory (or swap or disk or whatever) as a bitmap for manipulation.
ZP's on the fly caching (and the precacher/Cachemanger, which just calls the on-the-fly cacher) does this resizing as parallel httpd processes via i.php and the number of parallel processes is only limited by server settings (apache for instance). This can result in very large memory usage and even on a quad core server, it's hard to imagine that handling more than about 8 (probably far less) in parallel is beneficial. At some point the process are competing for memory and cpu and no gain is achieved.
For me, I run out of memory entirely pretty soon and since I don't believe in swap, I get oom-killer (process killer), and it's a pain to make oom killer behave nicely. (several processes competing simultaneously for swapped memory would be nuts.. swap is for parking your web browser while you use your spreadsheet, not for parallel processing)
In contrast an antiquated single core server can handle serving the JPEGS at pretty high bandwidth (we're talking relative to SOHO here, not relative to google). Normal image serving only requires enough memory for the JPG and the process overhead. So once images are created, the resource requirement is much lower and images can be easily served in parallel. The same server can also get the images cached, just not all at a once.
Existing Solutions:
1) Use smaller images.
I don't want to, and it's not necessary
2) The Imagick memory limit doesn't work for me. I even tried modifying the code to apply limits to the all three imagick limits: "MEMORY" "MAP" "DISK"
I think it limits each process individually. I seriously doubt there's any inner-process accounting of resources involved in that option to imagick. Once you give enough freedom that any process can actually work, then they all work and you're back to square 1. Actually I gave it quite alot of room and never got any of them to work, but I didn't really see how far I could push that (because I don't like crashing my server and I already achieved high parallel memory usage without getting results). Anyway, it didn't work for me.
3) limit apache to a couple of processes.
As many have pointed out, web browsing is and should be very parallel. There's no need to limit the whole browsing experience, (in zp and my other pages) just because image caching is enormously resoruce intensive. It only happens once. After the image is cached, that's that. To me, this is not a reasonable option. The problem is parallel image caching. Serial web browsing is not a solution to parallel image caching problems. Serial image caching is a solution to parallel image caching problems.
My solution (so far):
So obviously then my solution is to serialize the image cacher.
The usual all purpose way to serialize something (if you can't just use exec instead of fork) is to use a blocking lock (mutex). So that's all I did.
So far I haven't done this beautifully. I just put this:
set_time_limit ( 1000 );
// make sure the file to lock is there
$fp = fopen("lockfile","w+");
fclose($fp);
// get a file lock
if ($fp = fopen("lockfile", "r+")){
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
// the rest of i.php goes here
....
....
} else {
//echo "i.php could not get a file lock";
} else {
echo "i.php could not get a file lock";
}
flock($fp, LOCK_UN);
fclose($fp);This isn't very nice or clever. At least I could invert the conditional to put all the code at the top and un-nest it a bit, but whatever. This was just a quick and dirty round one
...
and it works GREAT!
Now that my cache processes are not piling up they work fast and I can use ZP
I'm not sure the timeout change is needed. I threw it in before even testing. That's a long one too and it can also be handled globally in php.ini. As for browser timeouts, in the end it takes as long as it takes to do this, and it takes less time in series than it does when it's over-parallelized. For my purpose, I have not experienced any much timeout problems. I did need to hit refresh once when one last thumb didn't show up. That has not been reprodicuble though. Generally it has just worked and worked smoothly.
For the most part, I will use it with the pre-cacher anyway and that is working fine too.
Improvements
1) Improve the lock location
-- this only needs to go around the cacheImage function, not around the whole image processor. That can improve performance in some circumstances I think.
2) Change the lock mechanism (probably put it in a class or functions to generalize it, and then play with how to do it in those functions)
flock:
the bad: There seem to be some caveats about flock on various system working differently with different open modes. I think it's possible to avoid those problems, but some still feel it's not the most platform independent solution. It doesn't work on some old file systems, maybe not of NFS filesystems (I'm not sure).
the good: lock cleanup is automatic. No need to check process pid written to a file and clean stale locks (and pid's are not unique anyway, collisions are actually quite common)
While there may be issues, it's mostly portable.
mysql locks:
My better idea is to use mysql locks. Let mysql worry about portability of mutexes (and I guess they do). ZP uses mysql anyway, so use a mysql table lock on a dummy table. They also have automatic cleanup when a session is ended. The bad is that I don't have any experience with controlling mysql sessions in php but it looks not very bad.
3) Add a menu option. I think that's easy
4) Implement a configurable level of parallelization.
this requires looping through n non-blocking locks with a small wait time at the top of the loop.. and looping until one of those n locks suceeds. Unfortunately flocks non-blocking option has mixed reviews in windows. The good thing is the behavior would automatically revert to single process serialization, not break entirely, but maybe better not use flock then.
Even on my system I could benefit from 2 or three process running at once.
5) Add option to disable resizing when it's not being called by the precacher.
I'm on the fence on this option now since I was able to enforce serialization without it. If the DOS bug
http://www.zenphoto.org/trac/ticket/2262
has really been solved well, then I probably don't care.
However if it hasn't (I'll see soon). Then I have no real need for on-the-fly precaching once I've cached everything anyway, and it represents a real security threat. Once you're controlling use of the resizer anyway, it's easy to put in a conditional to ask if it's being requested by the CacheManager or not (by passing a related parameter, like the admin parameter). Once you've precached everything, there (for my use at least) should be no valid reason why anyone is requesting a resize anyway. I have also already tested that returning without resizing does not cause massive breakage. At worst, if the file really isn't precached, you just get a broken image icon exactly as you expect if you select such an option and don't precache properly.
6) yes there was a return statement in a code path in the middle of the file and I didn't put lock cleanup there. Anyway.. it's irrelevant for a couple of obvious reasons now.
That's all for this isntallment.