Photo Organizer

Notice: Undefined index: tasklist_type in /var/www/flyspray/includes/class.tpl.php(128) : eval()'d code on line 85 Notice: Undefined index: tasklist_type in /var/www/flyspray/includes/class.tpl.php(128) : eval()'d code on line 90
  • Status Closed
  • Percent Complete
    100%
  • Task Type Feature Request
  • Category Backend / Core → Import
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Normal
  • Reported Version 2.33
  • Due in Version Undecided
  • Due Date Undecided
  • Votes 0
  • Private No
Attached to Project: Photo Organizer
Opened by Solomon Peachy (pizza) - 2007-02-10
Last edited by Solomon Peachy (pizza) - 2009-03-04

FS#167 - Image import/processing in parallel

For those of us with multiprocessor/multicore systems it would be really nice if PO could process more than one image at the same time.

It would save a lot of time when importing large batches of images.

Perhaps the simplest way to do this would be to rewrite the import loop to "submit" images to the backend, instead of processing them in said loop.

This task does not depend on any other tasks.

Closed by  Solomon Peachy (pizza)
Wednesday, 04 March 2009, 14:33 GMT
Reason for closing:  Implemented
Additional comments about closing:  I've imported several thousand images with this code, and it seems pretty damn solid now.
Solomon Peachy (pizza)
Saturday, 10 February 2007, 14:06 GMT
If we go to a "bulk upload tool" it would be able to submit as many images as it wants at once. So perhaps the real solution is to implemnt the necessary bits for a bulk upload tool?
Solomon Peachy (pizza)
Thursday, 18 September 2008, 01:07 GMT
Everything an image needs to be successfully imported is now contained within the $image_data[] structure. We could theoretically serialize this structure, stack up a bunch, and then process them in parallel until we're done.

Too bad PHP sucks for job control.
Solomon Peachy (pizza)
Tuesday, 14 October 2008, 12:26 GMT
I've been consolidating the import code -- now photo/version imports use the same function, and all per-file stuff (eg orientation/colorspace) is now stored on a per-file basis in the image_data structure. It's now possible to put all files for a photo into this structure, and import photo+allversions in a single batch.

All that's left in this phase is to modify the 'directory upload' code such that it assembles a single struct for all versions then kicks off the import for all versions at once.
Solomon Peachy (pizza)
Sunday, 19 October 2008, 01:32 GMT
Okay, the upload code is now fully serialized. All "import requests" are now written into the database, and a separate function pulls entries off the queue and imports them.

As soon as I figure out how to perform proper job/process control, we'll be able to have multiple workers in parallel.
Solomon Peachy (pizza)
Tuesday, 21 October 2008, 17:16 GMT
we're going to have to have the workers running out of band of the web server; ie as a "daemon" of sorts.
Solomon Peachy (pizza)
Wednesday, 22 October 2008, 03:56 GMT
There's still some ugliness dealing with cleaning up temporary directories.
Solomon Peachy (pizza)
Monday, 27 October 2008, 13:28 GMT
OOB daemon created. Still need a startup script, documentation, and other misc integration.

Also need the OOB "import results" channel created somehow.
Solomon Peachy (pizza)
Wednesday, 19 November 2008, 05:25 GMT
Import loop is now split into multiple transactions, allowing multiple workers to scale nearly linearly.

Once the OOB "import results" channel is created, this feature can be considered complete.
Solomon Peachy (pizza)
Thursday, 15 January 2009, 01:37 GMT
background "results" channel created. No way to query it yet.
Solomon Peachy (pizza)
Wednesday, 28 January 2009, 22:41 GMT
Form to display and clear the background channel data is complete. Everything works.

Loading...