Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.flyspray.php on line 113 Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.user.php on line 115 FS#78 : Failed imports with some images

Photo Organizer

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Backend / Core
  • Assigned To
    Solomon Peachy
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Devel
  • Due in Version Undecided
  • Due Date Undecided
  • Votes
  • Private
Attached to Project: Photo Organizer
Opened by Thomas Bleher - 2006-06-22

FS#78 - Failed imports with some images

While trying to upload a bunch of pictures, I noticed some small errors.
This is with the current development version from svn as of today (2006-06-22).

Patch with comments:
=== modified file 'include/exif.php'
--- include/exif.php
+++ include/exif.php
@@ -19,24 +19,7 @@
include_once "exifer/exif.php";

function validate_exif_string($exif_string) {
- $string_length = strlen($exif_string);
- $newstring = "";
- $j = 0;
-
- /* Remove non displayable ASCII characters */
- for ($i=0; $i < $string_length; $i++) {
- $ascii_num = ord($exif_string[$i]);
-
- if ($ascii_num == 0x27)
- continue;
-
- if ((0xff > $ascii_num) && ($ascii_num > 0x1F)) {
- $newstring = $newstring . $exif_string[$i];
- $j++;
- }
- }
-
- return $newstring;
+ return pg_escape_string(utf8_encode($exif_string));
}

function replace_resolution_unit($string, $resolution_unit) {

Comment: Postgres (at least v8.1) expects valid utf8 data, but some exif data is not valid.
This change encodes all ascii-data as utf8 and also makes it possible to have ' in the data.
Possible problem: If the data already is utf8, it gets doubly encoded. So a better way would
probably be to remove invalid characters, but I don't know how to do that in php.
=== modified file 'include/import.php'
--- include/import.php
+++ include/import.php
@@ -269,8 +269,8 @@
$result = pg_query($database, $query);
if (!$result) {
pg_query($database, "rollback");
- print "<li><font color=\"red\">Failed to insert rows into photo.</li>";
- print "<ul>$query</font></ul>";
+ print "<li><font color=\"red\">Failed to insert rows into photo.</font></li>";
+ print "<ul><li><font color=\"red\">$query</font></li></ul>";

error_reporting(0);
print "<li>Cleaning up the mess.</li>";

Comment: Just correct some invalidly nested html.
=== modified file 'include/rdf.php'
--- include/rdf.php
+++ include/rdf.php
@@ -264,7 +264,7 @@
$rdf_image_data['country']);
}
else {
- $image_data['location'] = "null";
+ $image_data['location'] = "0";
}

if (($rdf_image_data['camera_make'] != "null") ||

Comment: This change is needed due to the database schema change in the svn version.

Closed by  Solomon Peachy
2006-07-05 20:05
Reason for closing:  Fixed
Additional comments about closing:  I'm going to mark this one fixed, as all test images I have pass with the current exifer-based code.
Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.user.php on line 115 Admin
Solomon Peachy commented on 2006-06-24 01:08

It's good to see someone else mucking with the -devel tree; I have far more ideas than time, unfortunately. :)

rdf.php patch applied; r420 (devel) and r421 (stable)
import.php patch applied; r422 (devel) and r423 (stable)

As for the first patch.. I hadn't realized that we weren't properly escaping EXIF strings. I've committed a patch to do only that.

In the mean time, UTF-8 EXIF strings are supposed to be marked as such. Can you attach one of the problematic images so I can fiddle with things myself? Thanks!

Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.user.php on line 115 Admin
Solomon Peachy commented on 2006-06-24 01:27

It's worth mentioning that I don't have access to a postgres 8.1 installation.

This UTF-8 encapsulation is only necessary if the strings aren't pure ASCII, correct?

Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.user.php on line 115
Thomas Bleher commented on 2006-06-24 08:10

The problematic image is available from http://www.cip.ifi.lmu.de/~bleher/P1070461.JPG

The error message produced is:

> Importing the Joint Photographic Experts Group JFIF format file P1070461.JPG into folder upload
> Extracting EXIF data ...
> Warning: pg_query() [function.pg-query]: Query failed: FEHLER: ungültige Byte-Sequenz für Kodierung »UTF8«: 0x87 in /var/www/po/include/exif.php on line 198
>>>>>>>>>>> This means approximately: ERROR: illegal Byte sequence for encoding "UTF8"

> Warning: pg_query() [function.pg-query]: Query failed: FEHLER: ungültige Byte-Sequenz für Kodierung »UTF8«: 0x87 in /var/www/po/include/exif.php on line 219

> Warning: pg_query() [function.pg-query]: Query failed: FEHLER: aktuelle Transaktion wurde abgebrochen, Befehle werden bis zum Ende der Transaktion ignoriert in /var/www/po/include/exif.php on line 178
>>>>>>>>>>> This means approximately: ERROR: Transaction aborted, commands will be ignored until end of transaction

The UTF-8 encapsulation is needed whenever the string is not valid UTF-8 (for example ISO-8859-1 or just a binary stream). For ASCII, this is no problem, as valid ASCII is also valid UTF-8.
The problem is, that we cannot just call utf8_encode() because if the string is valid UTF-8 it will get doubly encoded.
So the proper way seems to be:
* If the string is valid UTF-8, just store it in the DB
* Else, if we know the encoding, convert from this encoding to UTF-8
* Else remove every illegal character

Alternatively we could change the DB so it doesn't expect UTF-8 but binary blobs.
However, I think having UTF-8 is nicer for displaying things.

BTW: I just found a project which might be useful for what we want:
http://phputf8.sourceforge.net/
utf8_bad_strip() might be what we want.

Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.user.php on line 115 Admin
Solomon Peachy commented on 2006-06-29 14:45

From appearances, the exifer library is horribly mangling your image to the point where it's pointless to talk about UTF-8 or any other type of encoding at all. I'm wading through the mess now.

Sigh. I wonder if it's time to migrate to PEL or something like that.

Deprecated: Function create_function() is deprecated in /var/www/flyspray/includes/class.user.php on line 115 Admin
Solomon Peachy commented on 2006-06-29 20:40

I think I've fixed the problem with exifer now; your image is parsing sanely, and images from all the other cameras I have access to also seem to still be parsing sanely.

the -devel and -stable code branches are updated with this fix.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing