Upgrade StorageSync issues correlating with Unicode emoticons in text content.
Issue
-----
My syncs are failing since some days constantly (details below).
Today I gave my recently starting Zotero content sync issues another bug fixing try. (Hey I really miss my working Zotero!). I figured out that Unicode Characters, that were synced fine in the past, are now expected to break my syncs.
I am currently testing workarounds by modifying the critical text (even it becomes inconsitent) because only trivia entries were affected (Funny characters? – not funny!)
I come back later on it.
-- Armin
Details
--------
Zotero Version is Standalone 4.0.29.15 on MacOSX 11.6 (no missing updates shown).
I get messages like this (in German, translation by accident):
Alert:
Ein/e oder mehrere lokal gelöschte/r/s Zotero collections wurde seit der letzten Synchronisation an einem nicht-lokalen Dateistandort geändert. Die aktuellsten Versionen wurden beibehalten.
In der Fehlerkonsole finden Sie eine vollständige List der entsprechenden Änderungen.
(one or more locally deleted Zotero coolections were changed in a non local file repository. The most recent versions were kept. See log for details...) (free translation not matching zotero wording)
Alert:
Ein Zotero Tag oder mehrere wurden seit der letzten Synchronisation zu Einträgen auf mehreren Computern hinzugefügt und/oder aus Einträgen auf mehreren Computern entfernt. Die verschiedenen Versionen wurden vereint.
In der Fehlerkonsole finden Sie eine vollständige List der entsprechenden Änderungen.
(one or more parts were added or removed from entries on multiplecomputers. The differnt versions were joint. See log for details...) (free translation not matching zotero wording)
more details...
-----------------
The errors do not disapper after some rerun/restart as before expected in most cases (I consider Zotero a very failproof application after 8 years of usage)
I ran the Debug console and finally found a common marker of the failing contet items.
The title or description/summary or a note attachment contained unicode emoticons like a guitar, some funny or boring elements the authors had places there proudly. Even some simple typographic quotes seem to mess this up now.
The actual Zotero items and attachments are quite old and there were around 15 entries from 2010 to 2016 that have been synced before sind more than a year or years now got me into trouble.
I archived my logs/errors in case but I am very restrictive with sharing error reports publicly.
I fact it seems that some Javascript code that was recently updated or the pipeline running it, has no tests to check for some kind of Unicode Chars and fails to deal with them not any longer.
Troubleshooting
---------------
I currently try to fix my local entries with obvious emoticons and / or deleted them or changed them to more common characters. I now resync the stuff and wait for the log of my quite big library of ~20GB on WebDAV and >2GB on Zotero.org.
When I come back later I can maybe share some relevant parts of my (very long) logs.
The Unicode issue is now filed and searchable so others are warned.
Maybe the second alert with the shared library will still be present. But no idea what this is related to.
-----
My syncs are failing since some days constantly (details below).
Today I gave my recently starting Zotero content sync issues another bug fixing try. (Hey I really miss my working Zotero!). I figured out that Unicode Characters, that were synced fine in the past, are now expected to break my syncs.
I am currently testing workarounds by modifying the critical text (even it becomes inconsitent) because only trivia entries were affected (Funny characters? – not funny!)
I come back later on it.
-- Armin
Details
--------
Zotero Version is Standalone 4.0.29.15 on MacOSX 11.6 (no missing updates shown).
I get messages like this (in German, translation by accident):
Alert:
Ein/e oder mehrere lokal gelöschte/r/s Zotero collections wurde seit der letzten Synchronisation an einem nicht-lokalen Dateistandort geändert. Die aktuellsten Versionen wurden beibehalten.
In der Fehlerkonsole finden Sie eine vollständige List der entsprechenden Änderungen.
(one or more locally deleted Zotero coolections were changed in a non local file repository. The most recent versions were kept. See log for details...) (free translation not matching zotero wording)
Alert:
Ein Zotero Tag oder mehrere wurden seit der letzten Synchronisation zu Einträgen auf mehreren Computern hinzugefügt und/oder aus Einträgen auf mehreren Computern entfernt. Die verschiedenen Versionen wurden vereint.
In der Fehlerkonsole finden Sie eine vollständige List der entsprechenden Änderungen.
(one or more parts were added or removed from entries on multiplecomputers. The differnt versions were joint. See log for details...) (free translation not matching zotero wording)
more details...
-----------------
The errors do not disapper after some rerun/restart as before expected in most cases (I consider Zotero a very failproof application after 8 years of usage)
I ran the Debug console and finally found a common marker of the failing contet items.
The title or description/summary or a note attachment contained unicode emoticons like a guitar, some funny or boring elements the authors had places there proudly. Even some simple typographic quotes seem to mess this up now.
The actual Zotero items and attachments are quite old and there were around 15 entries from 2010 to 2016 that have been synced before sind more than a year or years now got me into trouble.
I archived my logs/errors in case but I am very restrictive with sharing error reports publicly.
I fact it seems that some Javascript code that was recently updated or the pipeline running it, has no tests to check for some kind of Unicode Chars and fails to deal with them not any longer.
Troubleshooting
---------------
I currently try to fix my local entries with obvious emoticons and / or deleted them or changed them to more common characters. I now resync the stuff and wait for the log of my quite big library of ~20GB on WebDAV and >2GB on Zotero.org.
When I come back later I can maybe share some relevant parts of my (very long) logs.
The Unicode issue is now filed and searchable so others are warned.
Maybe the second alert with the shared library will still be present. But no idea what this is related to.
If you take a snapshot of that website yourself you find the crazy characters in the summary of your website entry. Hope this is reproducable. I could reproduce the characters here. The original snapshot formerly working during syncs was from February 2016.
If you need older logs I have the text here.
I fixed the above local issue myself using the same approach I posted here before 2015. It feels strange when you find your own former solution to a different issue in the forum and it reminds you how to find at least a workaround.
See detailed description of steps to get Zotero to work if you have at least one working copy or backup.
https://forums.zotero.org/discussion/comment/234037/#Comment_234037
What I did additionally:
# Before moving away the old zotero storage
IMPORTANT: Make sure autosync is off during the procedures!
## Export of entries newer than last working sync
1. Create a folder for your exports with an ISO datetime stamp in it (suggestion).
2. Create one subfolder for your main library and additional folders for each of every group in which you had added something.
3. Name the folders after the groups.
4. Export the main library and the groups one after each other with all notes and attachments included.
5. Take notes of your related items. As far as I can get it, the relations are lost during import in the backup, since they may cross exported and existing content.
# After restoring the backup storage
## Test the restored data
1. Test if Zotero is starting with the restored libraries and they are working.
2. Test if they are all syncing with a manually started sync from the regular sync icon in the UI.
## Import exported libraries
1. Import your main library using the GearWheel menu "Import" command.
2. Test if it syncs properly
3. Import group libraries one after another
test each sync first before proceeding to the next.
It came out that I could not finally determine the cause of the issue myself. The guessed causation related of the difference related to Unicode letters may have been just a correlation. I am looking forward to improved sync in Zotero 5. Please focus that! May the sync be with you!
The inability to sync emoji characters is a server-side issue that we're addressing (and which should already be fixed for some libraries). We'll have it fixed for all libraries soon.
I expect that noone should need to use a backup with more difference. Any larger amount of entries cannot be solved by hand properly ( and from brain memory), because of the possible issues @dstillman mentions above. It is just better than loosing these entries.
Any link to the best practise for this case is welcome. (Which one of the multiple officially documented procedures avoids export/import and still preserves latest entries?)
In my case restoring from the server took forever and failed. Any other options I tried last time, were too time consuming for me now. So this approach was a compromise for my case.
If I can suggest something, it is the ability to chunk large research libraries into years and add the ability to add an expiration date for automatic removal of content after time. Then you can simply skip whole areas that are unchanged during sync and in case restore.I need to cast that into another post.