Partial library sync between devices

JimGrisham

## Tl;dr

Summary: sync isn’t completely stopped, since ~some~ new items appear to sync in all directions, and individual missing items appear when moved (at least in my limited testing) to a sub-collection or edited.
- Certain items are just ‘skipped’, and I don’t see any error messages on either iOS device.

Items added from:
- iPad: show up on all devices
- iPhone: show up on all devices
- Windows: show up everywhere ~except~ iPhone

The issue might be intermittent / situational, sadly. :(


## Technical data


Configuration:
- Database sync enabled
- Files stored in WebDAV (this is ~not~ an attachment problem)


Zotero application versions:
- Windows: ‘Zotero 6.0.18’
- iPhone (iOS 14): ‘Zotero 1.0.8 Build 2’ (normally on 1.0.4 due to iOS attachment field editing bug, but I tried several builds to try and resolve this issue)
- iPad (iOS 15): ‘Zotero 1.0.7 Build 5’


{I realize these are beta releases, so I’m not looking for a personal fix - rather, I’m reporting this in case a more systemic issue exists.}


Debug ID from Windows: D359908096
Debug ID from iPad: D944385881
Debug ID from iPhone: D1137428150


Total items:
- Zotero web: ?? (no visible item count, and no ‘Unfiled Items’ section, but the items in question appear to be present)
- Windows: 1046 (414 in “Unfiled Items” list)
- iPad: 1046 (414 in “Unfiled Items” list)
- iPhone: 1025 (393 in “Unfiled Items” list)


## Discussion

I’m not sure it’s relevant, but the 21 missing items were all in the ‘Unfiled Items’ section. (The other 350+ items in that section appear everywhere, though.)

The most recent two items for which this problem exists are PDFs that I added to Zotero on Windows yesterday(?).
- One had a parent item auto-created with the correct metadata, and for the other I manually populated the parent item fields.
- Both parent items appear in the Zotero Web Library, and I know the attachment PDFs are on my WebDAV server since I can view them in Zotero on the iPad.
(Several other affected items were PDFs or webpage snapshots auto-added by the Windows ‘Zotero Connector’, running in a normal-release version of Chromium-based Edge, and not subsequently modified manually except for the two items discussed below.)


## Symptom elaboration

What I’ve tried:
  1. ‘Pull to refresh’ on “My Library”, “All Items”, and “Unfiled Items” screens on iPhone

  2. Moving one of the ‘unfiled’ items ~did appear~ on the iPhone, but only after adding it to a new subcollection using Zotero on the iPad. [The new count is 1046 items on Windows/iPad vs. 1026 on the iPhone]

  3. Similar to the previous item (#2), I added another of the ‘missing’ items to that same new collection using the Windows client. That item now appears on all devices. [The new count was 1046 items on Windows/iPad vs. 1027 on the iPhone]

  4. For another affected item, I edited the item metadata on Windows (by typing a value into the previously-blank “Short Title” field. That item now appears on all devices. [The new count was 1046 items on Windows/iPad vs. 1028 on the iPhone]

  5. I downgraded the iPhone version back to the oldest available 1.0.4 build, but got an error so re-installed ‘1.0.8 Build 2’. I was still required to log back in, and when I did the new count was unchanged at 1046 items on Windows/iPad vs. 1028 on the iPhone]



What I haven’t tried yet:
  1. Signing out/in to Zotero account in iPhone app settings, since it warns that “Any local data that was not synced will be deleted.”, and I don’ t know how to verify that all local data has been synced. (Tangent: Why wouldn’t un-synced local data be ~retained~ when logging out? It’s already on the device, right… is it being ~actively~ deleted here, or is this an artifact of some OS or 3ʳᵈ-party database synchronization framework?)

  2. Re-syncing the database in the Windows client (that didn’t seem promising, since the data there ~already~ matches the official Zotero Web library and the iPad client).


## Salutations

Thanks to the Zotero team members, official and unofficial, for all of the time you continue to devote to this extremely useful system!


## Related

{I realize these could be posted as separate issues - if there is interest in extended discussion of any of them, feel free to do so}

Going forward, should there be options on Zotero iOS to:
  1. Verify / resync database with official Zotero servers (would be reassuring prior to, e.g., device replacements, or even prior putting a device into ‘airplane mode’ or taking your phone white-water rafting, although I guess an iCloud or iTunes backup would be sufficient in either of those specific hypotheticals)

  2. Take account of the current sync status when composing the ‘logout warning’

  3. Show sync status (during or after a sync, or all the time in settings)

  4. Verify all ~local~ attachments are present on the server (whether ‘ZFS’ or WebDAV)

  5. Delete local copies of synced attachments (right now, the command on the ‘Local Storage’ settings page is to “Delete All Local Attachment Files”, which implies something that could have quite different results; some folks might plausibly wish to free up device space without risking data loss)

  • dstillman
    This just means there was some bug that prevented those items from being pulled down on the iPhone when they were first created elsewhere. The fact that they appear on multiple devices means that they exist correctly in the online library, as you can see for yourself.

    To debug a syncing problem, we need to see Debug IDs showing you 1) making the change and 2) syncing on the device where the data isn't being transferred (the iPhone here). We don't have any way of debugging a change you made previously that didn't sync — it's just something that happened in the past, and syncing has moved on. So unless you can reproduce it, there's nothing we can do at this point.

    To correct this, you can either make some change to all the affected items (e.g., adding them to a temporary collection) to cause them to be reuploaded or, if you can confirm that any data you added on the phone has been uploaded, simply log out and in on the phone to trigger a redownload of all data.

    While this of course shouldn't happen, we'll be adding an option to manually trigger a full sync — checking both sides to make sure all items have been transferred — to make it easier to recover if it does.
  • JimGrisham
    Thanks.
    - (Each of the above debug traces spanned both an app start and a sync / refresh request, but that's the best I could do.)

    Is there a practical way for a normal user to "confirm that any data (they) added on the phone has been uploaded", though?


    That full sync (or even a 'sync verification') feature will be very helpful, because right now there is no indication that items are absent unless the user explicitly goes looking for them (or compares item counts).

    - (e.g.: I had no idea multiple items were missing, or for how long. It was only discovered it by chance.)


    If the database on the server knows there are n items (database table rows) in a library, but the device app database only has n - x items, it seems a 'not fully synced' warning would be possible even before a full re-sync feature can be implemented.
    - The account page in the settings could even just say: "440 items on server, 440 items on device", and/or "Last synced: 5 minutes ago (2022-11-25 22:10 UTC)".
    - Even that small bit of sync status information would also help with identifying bugs, I imagine.
  • adamsmith
    Is there a practical way for a normal user to ""confirm that any data (they) added on the phone has been uploaded"", though?
    For an individual item, sure, you can just check the web library.
  • dstillman
    edited November 18, 2022
    If the database on the server knows there are n items (database table rows) in a library, but the device app database only has n - x items, it seems a 'not fully synced' warning would be possible even before a full re-sync feature can be implemented.
    Syncing is vastly more complicated than that. That wouldn't be a remotely reliable metric, and the absence of such a warning would give a false sense of security.

    The fix here is to fix the syncing problem.

    Can you provide the URLs (or just the 8-character item keys) of the items in the web library that haven't yet (or hadn't yet) synced? While not as helpful as a Debug ID for it happening, if this happened recently we might be able to figure something out from server logs.
  • JimGrisham
    edited November 21, 2022
    @adamsmith:
    For an individual item, sure, you can just check the web library.
    That would work for limited testing purposes.

    For a regular user (and especially for frequent users) of the app (with, e.g. hundreds or thousands of items) encountering that warning, I'm not sure what the solution might be.
    Hypothetical 'user story':

    Let's say Sally added a tag to an item last Tuesday, added a few annotations to another item on Monday afternoon, and added notes to a few items later in the week. If she encounters that error message on Friday afternoon, that's a lot of manual verification that her human memory may have challenges with. How far back does she have to verify? What if she installed the app 9 months ago and has been using it daily?

    @dstillman:
    Syncing is vastly more complicated than that.
    I totally agree, and I don't envy anyone working on it! My example was more in the spirit of a 'no go' test, not a 'go' test, that would hopefully be paired with other verifications. It doesn't provide much value if the test passes. If this sort of simple test fails, though, it alone provides prima facie evidence of a data sync problem.
    false sense of security
    That's a problem, indeed. That the current error message can cause a false sense of anxiety is also a problem.
    The fix here is to fix the syncing problem.
    Absolutely!

    Long-term, I hope a robust automatic verification system (even if verification is performed periodically, or upon user request) is also included, as many unexpected factors can affect data integrity.
    Can you provide the URLs (or just the 8-character item keys) of the items in the web library that haven't yet (or hadn't yet) synced?
    There are currently 1038 of my 1056 library items showing in the iPhone app. Ed: It appears these 18 remaining items (the ones I did not previously fix by manually editing) were added between 2022-11-09 233218 UTC and 2022-11-15 223413 UTC. Here are a few I was able to find. I'm happy to search forprovide data for more upon request:
    ...Key...	|	......Date added.......	|	Title
    --------	|	-----------------------	|	------
    QFMSWZDS	|	2022-11-09 23:32:18 UTC	|	Adj...
    8GW8ESAE	|	2022-11-10 06:50:37 UTC	|	Opt...
    84AHWNXD	|	2022-11-10 06:51:08 UTC	|	Sof...
    YZECDRGM	|	2022-11-10 06:53:59 UTC	|	For...
    	... 7 items omitted ...
    2B9DIS6K	|	2022-11-10 09:38:16 UTC	|	hea...
    IHKW6IHH	|	2022-11-15 22:24:15 UTC	|	Ama...
    	... 2 items omitted ...
    V3R4RAN8	|	2022-11-15 22:33:37 UTC	|	ISO...
    Q8WSKIIJ	|	2022-11-15 22:33:52 UTC	|	ISO...
    CDLZ7A72	|	2022-11-15 22:34:13 UTC	|	[MS...

    I couldn't figure out how to get an item URL or other identifier from the Windows app or the web interface, so I pulled the above item keys directly from the zotero.sqlite file. Ed: Wow... turns out getting them from the web was too obvious. Doh.

    Tl;dr: Even if the cause of this specific problem is discovered and corrected, we need to be able to detect (and hopefully automatically correct) future out-of-sync conditions without losing any user data.
  • dstillman
    As I say, we'll be adding a full-sync option to help recover from potential sync bugs. There's no need to dwell on this.

    We've identified a core bug here in the iOS app's sync process — in certain cases remote updates could be skipped. We'll fix this for the next version and try to trigger automatic reconciliation of existing items. (You can still fix these manually the way I say above.)

    Sorry about that, and thanks for helping us track this down.
  • JimGrisham
    You're welcome, @dstillman, and thank you for the update.


    It's reassuring that you found something, since this happened again last night.
    (Details: four new items were created in the Windows client via the browser extension; many hours later, simply changing the displayed PDF page, while viewing an attachment of just one of those items, was enough to trigger a sync that made all four of the missing new items appear on the iPhone within seconds.)

    If it's any consolation, it appears that .NET developers have encountered similar synchronization issues with Realm (PSA / realm-dotnet #2837) in the recent past.


    Tangentially: while debugging, I noticed that the maindb_nnnnnnn.realm file on the iPhone was 29.0 MB -- 50% larger than the 19.0 MB version on the iPad. I assume this is just due to delayed compaction / garbage collection, as both versions are 17.5 MB when exported as .json files.
    The zotero.sqlite file on Windows is only 11.0 MB, but that is to be expected as Realm trades off space efficiency for runtime performance.

    - Perhaps any future reconciliation can eventually be paired with a database compaction, at least when manually invoked via the settings screen.
  • dstillman
    This has nothing to do with Realm. It was just a bug in the syncing logic when an item was created locally before remote items were fetched.
  • dstillman
    @JimGrisham: If you want to try it, the latest Zotero for iOS beta should fix this and also force a full sync.
  • JimGrisham
    Ahh, that makes sense - the first thing I do most mornings is to add the daily newspaper to Zotero (via the iOS share sheet connector), and then I open the actual Zotero app for reading/markup; the app itself hasn't had a chance to sync any incoming changes at that point, since there's a good chance it hasn't been opened since before I left the office the previous afternoon.
Sign In or Register to comment.