Photo Organizer

This is the bug tracker for Photo Organizer.
Tasklist

FS#167 - Image import/processing in parallel

Attached to Project: Photo Organizer
Opened by Solomon Peachy (pizza) - Saturday, 10 February 2007, 14:04 GMT
Last edited by Solomon Peachy (pizza) - Wednesday, 04 March 2009, 14:33 GMT
Task Type Feature Request
Category Backend / Core → Import
Status Closed
Assigned To No-one
Operating System All
Severity Low
Priority Normal
Reported Version 2.33
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

For those of us with multiprocessor/multicore systems it would be really nice if PO could process more than one image at the same time.

It would save a lot of time when importing large batches of images.

Perhaps the simplest way to do this would be to rewrite the import loop to "submit" images to the backend, instead of processing them in said loop.
This task depends upon

Closed by  Solomon Peachy (pizza)
Wednesday, 04 March 2009, 14:33 GMT
Reason for closing:  Implemented
Additional comments about closing:  I've imported several thousand images with this code, and it seems pretty damn solid now.
Comment by Solomon Peachy (pizza) - Saturday, 10 February 2007, 14:06 GMT
If we go to a "bulk upload tool" it would be able to submit as many images as it wants at once. So perhaps the real solution is to implemnt the necessary bits for a bulk upload tool?
Comment by Solomon Peachy (pizza) - Thursday, 18 September 2008, 01:07 GMT
Everything an image needs to be successfully imported is now contained within the $image_data[] structure. We could theoretically serialize this structure, stack up a bunch, and then process them in parallel until we're done.

Too bad PHP sucks for job control.
Comment by Solomon Peachy (pizza) - Tuesday, 14 October 2008, 12:26 GMT
I've been consolidating the import code -- now photo/version imports use the same function, and all per-file stuff (eg orientation/colorspace) is now stored on a per-file basis in the image_data structure. It's now possible to put all files for a photo into this structure, and import photo+allversions in a single batch.

All that's left in this phase is to modify the 'directory upload' code such that it assembles a single struct for all versions then kicks off the import for all versions at once.
Comment by Solomon Peachy (pizza) - Sunday, 19 October 2008, 01:32 GMT
Okay, the upload code is now fully serialized. All "import requests" are now written into the database, and a separate function pulls entries off the queue and imports them.

As soon as I figure out how to perform proper job/process control, we'll be able to have multiple workers in parallel.
Comment by Solomon Peachy (pizza) - Tuesday, 21 October 2008, 17:16 GMT
we're going to have to have the workers running out of band of the web server; ie as a "daemon" of sorts.
Comment by Solomon Peachy (pizza) - Wednesday, 22 October 2008, 03:56 GMT
There's still some ugliness dealing with cleaning up temporary directories.
Comment by Solomon Peachy (pizza) - Monday, 27 October 2008, 13:28 GMT
OOB daemon created. Still need a startup script, documentation, and other misc integration.

Also need the OOB "import results" channel created somehow.
Comment by Solomon Peachy (pizza) - Wednesday, 19 November 2008, 05:25 GMT
Import loop is now split into multiple transactions, allowing multiple workers to scale nearly linearly.

Once the OOB "import results" channel is created, this feature can be considered complete.
Comment by Solomon Peachy (pizza) - Thursday, 15 January 2009, 01:37 GMT
background "results" channel created. No way to query it yet.
Comment by Solomon Peachy (pizza) - Wednesday, 28 January 2009, 22:41 GMT
Form to display and clear the background channel data is complete. Everything works.

Loading...