Upgrade StorageSync after import from EndNote failing: 'Uploaded data not well-formed'
I moved my reference library from EndNote to Zotero (ca. 9000 entries with ca. 8000 PDFs). I set it up so that the data are synced to my account (and I upgraded to 2 GB), but not the attachment files. I have not been able to successfully sync my library either with the standalone or the firefox version. I keep getting the 'Uploaded data not well-formed' error.
My attempts:
- My 'zotero.sqlite' file was initially 760 MB. I cleared the index and the DB file is now 57 MB.
- The 'Check Database Integrity' passes with 'No errors were found in the database.'
- I used a sqlite editor to check 'mimeType' and 'path' of elements in the table 'itemAttachment' and to check 'value' of elements in table 'itemDataValues'. There were some weird entries which I cleaned up using Zotero.
- Several of my records had a corrupted date which I cleaned up using Zotero.
- I turned on logging of debug messages and started sync attempts (see extract below): At every attempt of syncing, I get the error code "INVALID_UPLOAD_DATA" after the select statement from table 'relations'. This table is empty -- is this ok? The debug log continues until a 'SELECT I.*, COALESCE(CASE (SELECT COUNT(*)...' statement. After which the error is reported and the stack traced. The log continues a bit more and then the sync stops.
I believe that I have cleaned the records as good as I could. I don't see any other obvious steps that I can take to figure out what entry has ill-formed data. The debug log nor the error message provide information to locate the problematic data in a way that are self-explanatory to me.
Thanks for any help!
Here the submitted Report ID: 116559452 / 704bdd77a5
[JavaScript Error: "Uploaded data not well-formed (Report ID: 704bdd77a5)" {file: "chrome://zotero/content/xpcom/sync.js" line: 2595}]
_error@chrome://zotero/content/xpcom/sync.js:2595:3
Zotero.Sync.Server 4.0.29.15, platform => MacIntel, oscpu => Intel Mac OS X 10.12, locale => en-US, appName => Zotero, appVersion => 4.0.29.15, extensions => Zotero Word for Mac Integration (3.5.14.SA.4.0.29.15, extension), Zotero LibreOffice Integration (3.5.12.SA.4.0.29.15, extension), ZotFile (4.2.8, extension), Java Applet Plug-in (Java 8 Update 121 build 13, plugin), Lync Meeting Join Plug-in (4.0.7577.5, plugin), Shockwave Flash (25.0.0.127, plugin)
Some of the debug log:
[about 500 000 lines look fine ...]
(5)(+0000000): SELECT itemID FROM itemAttachments WHERE linkMode IN (?,?)
(5)(+0000000): Binding parameter 1 of type int: 0
(5)(+0000000): Binding parameter 2 of type int: 1
(5)(+0001033): SELECT libraryID FROM relations WHERE libraryID LIKE 'local/%' LIMIT 1
(3)(+0000000): Uploaded data not well-formed (Report ID: 704bdd77a5)
(1)(+0000000): Uploaded data not well-formed (Report ID: 704bdd77a5)
(3)(+0000000): Reloading all tags
(5)(+0000000): SELECT ROWID FROM tags
[two dozen or so good lines]
(5)(+0007423): SELECT I.*, COALESCE(CASE (SELECT COUNT(*) FROM itemCreators IC LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1) WHEN 0 THEN NULL WHEN 1 THEN (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1) WHEN 2 THEN (SELECT (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1 ORDER BY orderIndex LIMIT 1) || ' and ' || (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1 ORDER BY orderIndex LIMIT 1,1)) ELSE (SELECT (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1 ORDER BY orderIndex LIMIT 1) || ' et al.' ) END, CASE (SELECT COUNT(*) FROM itemCreators WHERE itemID=I.itemID AND creatorTypeID IN (3)) WHEN 0 THEN NULL WHEN 1 THEN (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3)) WHEN 2 THEN (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3) ORDER BY orderIndex LIMIT 1) || ' and ' || (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3) ORDER BY orderIndex LIMIT 1,1) ) ELSE (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3) ORDER BY orderIndex LIMIT 1) || ' et al.' ) END, CASE (SELECT COUNT(*) FROM itemCreators WHERE itemID=I.itemID AND creatorTypeID IN (2)) WHEN 0 THEN NULL WHEN 1 THEN (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2)) WHEN 2 THEN (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2) ORDER BY orderIndex LIMIT 1) || ' and ' || (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2) ORDER BY orderIndex LIMIT 1,1) ) ELSE (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2) ORDER BY orderIndex LIMIT 1) || ' et al.' ) END) AS firstCreator, (SELECT COUNT(*) FROM itemNotes INo WHERE sourceItemID=I.itemID AND INo.itemID NOT IN (SELECT itemID FROM deletedItems)) AS numNotes, (SELECT COUNT(*) FROM itemAttachments IA WHERE sourceItemID=I.itemID AND IA.itemID NOT IN (SELECT itemID FROM deletedItems)) AS numAttachments, (CASE I.itemTypeID WHEN 14 THEN (SELECT sourceItemID FROM itemAttachments IA WHERE IA.itemID=I.itemID) WHEN 1 THEN (SELECT sourceItemID FROM itemNotes INo WHERE INo.itemID=I.itemID) END) AS sourceItemID FROM items I WHERE 1
(1)(+0002183): Error: Uploaded data not well-formed (Report ID: 704bdd77a5)
===== Stack Trace =====
Zotero.Sync.Runner</this.error@chrome://zotero/content/xpcom/sync.js:652:8
Zotero.Sync.Runner</this.sync/<.onError@chrome://zotero/content/xpcom/sync.js:622:5
_error@chrome://zotero/content/xpcom/sync.js:2604:3
Zotero.Sync.Server</sync/</result</uploadCallback@chrome://zotero/content/xpcom/sync.js:1763:9
Zotero.Sync.Server</sync/</result</bufferUploader/req.onreadystatechange@chrome://zotero/content/xpcom/sync.js:1832:11
=======================
[several dozen good lines before sync and debug log stop]
My attempts:
- My 'zotero.sqlite' file was initially 760 MB. I cleared the index and the DB file is now 57 MB.
- The 'Check Database Integrity' passes with 'No errors were found in the database.'
- I used a sqlite editor to check 'mimeType' and 'path' of elements in the table 'itemAttachment' and to check 'value' of elements in table 'itemDataValues'. There were some weird entries which I cleaned up using Zotero.
- Several of my records had a corrupted date which I cleaned up using Zotero.
- I turned on logging of debug messages and started sync attempts (see extract below): At every attempt of syncing, I get the error code "INVALID_UPLOAD_DATA" after the select statement from table 'relations'. This table is empty -- is this ok? The debug log continues until a 'SELECT I.*, COALESCE(CASE (SELECT COUNT(*)...' statement. After which the error is reported and the stack traced. The log continues a bit more and then the sync stops.
I believe that I have cleaned the records as good as I could. I don't see any other obvious steps that I can take to figure out what entry has ill-formed data. The debug log nor the error message provide information to locate the problematic data in a way that are self-explanatory to me.
Thanks for any help!
Here the submitted Report ID: 116559452 / 704bdd77a5
[JavaScript Error: "Uploaded data not well-formed (Report ID: 704bdd77a5)" {file: "chrome://zotero/content/xpcom/sync.js" line: 2595}]
_error@chrome://zotero/content/xpcom/sync.js:2595:3
Zotero.Sync.Server 4.0.29.15, platform => MacIntel, oscpu => Intel Mac OS X 10.12, locale => en-US, appName => Zotero, appVersion => 4.0.29.15, extensions => Zotero Word for Mac Integration (3.5.14.SA.4.0.29.15, extension), Zotero LibreOffice Integration (3.5.12.SA.4.0.29.15, extension), ZotFile (4.2.8, extension), Java Applet Plug-in (Java 8 Update 121 build 13, plugin), Lync Meeting Join Plug-in (4.0.7577.5, plugin), Shockwave Flash (25.0.0.127, plugin)
Some of the debug log:
[about 500 000 lines look fine ...]
(5)(+0000000): SELECT itemID FROM itemAttachments WHERE linkMode IN (?,?)
(5)(+0000000): Binding parameter 1 of type int: 0
(5)(+0000000): Binding parameter 2 of type int: 1
(5)(+0001033): SELECT libraryID FROM relations WHERE libraryID LIKE 'local/%' LIMIT 1
(3)(+0000000): Uploaded data not well-formed (Report ID: 704bdd77a5)
(1)(+0000000): Uploaded data not well-formed (Report ID: 704bdd77a5)
(3)(+0000000): Reloading all tags
(5)(+0000000): SELECT ROWID FROM tags
[two dozen or so good lines]
(5)(+0007423): SELECT I.*, COALESCE(CASE (SELECT COUNT(*) FROM itemCreators IC LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1) WHEN 0 THEN NULL WHEN 1 THEN (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1) WHEN 2 THEN (SELECT (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1 ORDER BY orderIndex LIMIT 1) || ' and ' || (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1 ORDER BY orderIndex LIMIT 1,1)) ELSE (SELECT (SELECT lastName FROM itemCreators IC NATURAL JOIN creators NATURAL JOIN creatorData LEFT JOIN itemTypeCreatorTypes ITCT ON (IC.creatorTypeID=ITCT.creatorTypeID AND ITCT.itemTypeID=I.itemTypeID) WHERE itemID=I.itemID AND primaryField=1 ORDER BY orderIndex LIMIT 1) || ' et al.' ) END, CASE (SELECT COUNT(*) FROM itemCreators WHERE itemID=I.itemID AND creatorTypeID IN (3)) WHEN 0 THEN NULL WHEN 1 THEN (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3)) WHEN 2 THEN (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3) ORDER BY orderIndex LIMIT 1) || ' and ' || (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3) ORDER BY orderIndex LIMIT 1,1) ) ELSE (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (3) ORDER BY orderIndex LIMIT 1) || ' et al.' ) END, CASE (SELECT COUNT(*) FROM itemCreators WHERE itemID=I.itemID AND creatorTypeID IN (2)) WHEN 0 THEN NULL WHEN 1 THEN (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2)) WHEN 2 THEN (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2) ORDER BY orderIndex LIMIT 1) || ' and ' || (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2) ORDER BY orderIndex LIMIT 1,1) ) ELSE (SELECT (SELECT lastName FROM itemCreators NATURAL JOIN creators NATURAL JOIN creatorData WHERE itemID=I.itemID AND creatorTypeID IN (2) ORDER BY orderIndex LIMIT 1) || ' et al.' ) END) AS firstCreator, (SELECT COUNT(*) FROM itemNotes INo WHERE sourceItemID=I.itemID AND INo.itemID NOT IN (SELECT itemID FROM deletedItems)) AS numNotes, (SELECT COUNT(*) FROM itemAttachments IA WHERE sourceItemID=I.itemID AND IA.itemID NOT IN (SELECT itemID FROM deletedItems)) AS numAttachments, (CASE I.itemTypeID WHEN 14 THEN (SELECT sourceItemID FROM itemAttachments IA WHERE IA.itemID=I.itemID) WHEN 1 THEN (SELECT sourceItemID FROM itemNotes INo WHERE INo.itemID=I.itemID) END) AS sourceItemID FROM items I WHERE 1
(1)(+0002183): Error: Uploaded data not well-formed (Report ID: 704bdd77a5)
===== Stack Trace =====
Zotero.Sync.Runner</this.error@chrome://zotero/content/xpcom/sync.js:652:8
Zotero.Sync.Runner</this.sync/<.onError@chrome://zotero/content/xpcom/sync.js:622:5
_error@chrome://zotero/content/xpcom/sync.js:2604:3
Zotero.Sync.Server</sync/</result</uploadCallback@chrome://zotero/content/xpcom/sync.js:1763:9
Zotero.Sync.Server</sync/</result</bufferUploader/req.onreadystatechange@chrome://zotero/content/xpcom/sync.js:1832:11
=======================
[several dozen good lines before sync and debug log stop]
1) I used a self-made AppleScript to add the group names of each of my 9000 records in EndNote to the 'KW' RIS field. And imported the RIS export and the attachments to Zotero.
2) Since the 'KW' field is imported as tags, I used some custom R code to manipulate the sqlite database directly and convert the tags to collections and to add my items to the collections. Obviously, I used an incorrect character set to generate the 8-len keys for the collections.
In an attempt to fix this, I used the character set ("23456789ABCDEFGHIJKMNPQRSTUVWXZ") defined by the function 'generateObjectKey' in the source code file utilities.js of Zotero v43.0.29.15 to re-generate the collection keys.
The sync still fails with the same error message 'Uploaded data not well-formed' and similarly looking debug output (Report ID: 2143605496).
In an attempt to undo what my custom code did the the sqlite database, I deleted all collections. Unfortunately, the sync still fails the same way (Report ID: 1011750234).
Did I completely mess things up -- or is there something else that could have gone wrong, e.g., Zotero is more strict with field content and/or character set of abstract field than EndNote?
https://www.zotero.org/support/dev/client_coding/javascript_api
or the server API
https://www.zotero.org/support/dev/web_api/v3/start
(the latter after your library is synced).
I'd wipe your library (by moving the zotero.sqlite to a different location on your HD and restarting Zotero) and restart.
I plan to look more closely at the JavaScript API and try to figure out how to create collections and add items.