Zotero Syncing - version history and restoring tags and attached files

I work with zotero on a large, collaborative project (about 8 people per library) where we store thousands of journal articles to be reviewed and work across 4 different taxa specific zotero libraries. A colleague opened an old computer yesterday (Oct 3, 2017) that she hadn't used since June 2017 and synced Zotero. She chose the "remote" version for all syncing but zotero updated the remote version with her local instance, thus deleting 6 months of organization, tagging, and file attachments to each of the references. Why did this happen given that she chose the "remote" version for all syncing? How can this be prevented? What is the best way to restore both the tags and information, as well as all of the pdf file attachments?
  • Are you on Zotero 4 or 5?
    If you have anyone with a recent library intact, make sure they disable auto-sync for now.

    This shouldn't be possible, in particular with respect to file attachments (with Zotero 4, syncing tags/collections can get you some wonky behavior, but that should always be to add tags to items or add items to collections, nothing else.)
  • Thanks for the response, we've all updated to Zotero 5. We all autosync so no one has a recent intact library except for the zotero.sqlite.bak files.

    Yes, my understanding was that this was not possible as well. I'm not sure if maybe the local instance on her old computer that hadn't been opened since June was just such a different version that Zotero got confused and overwrote the remote instance? Some tags appear to be current while other tags got deleted so Zotero must have been inconsistent when choosing to sync local --> remote and remote --> local even though it was told to sync remote --> local.
  • also we pay full price for a zotero subscription and unlimited space - not sure what that means for version history or for help restoring?
  • @dstillman will need to look at this

    Do make copies of the most recent .bak files. They'll get overwritten eventually, so you want to keep the last version.
  • Ok, we have several .bak file options. Waiting to hear from @dstillman thanks!
  • Yeah, this definitely shouldn't happen — and I don't think it should even be technically possible for a conflict with an older library to result in attachments being deleted, if that's really what you mean, because the file wouldn't have been deleted on either side, and Zotero only syncs explicit deletions.

    If you email support@zotero.org with the username this happened to and a link to this thread, we can look more closely. For what it's worth, it looks like there were uploads from Zotero 4 from two accounts in these groups in the last few days, so that could be part of it (though it shouldn't have been possible in 4.0 either).

    For recovering, the first thing (after everyone has made a copy of the .bak files) would be to temporarily disable auto-sync on a computer with a .bak file with a timestamp from before the changes were made or synced down, and then swap it in as zotero.sqlite. When you start Zotero, you should then see the library in the previous state.

    Let's start with that, just so you can be sure you have a copy of the library as you want it, and then we can walk you through how to apply those changes back to the online library. Before we do that, I'd want to know more about what kind of changes you saw.
  • Hi @dstillman, very appreciative of your help! We now have a working backup that we would like to use going forward. It is locally loaded to a fresh Zotero that is currently not synced. Can you please help us with the proper way to make sure this local version would replace select group folders on the Zotero server? We ideally would like to preserve URIs, as we are using those to track references within our project. We are ok with a straight replace, not merging or preserving anything on the server at this point for those select group folders. Obviously want to also know the exact steps to make sure everyone gets the restored copy only when the group eventually turns syncing back on. We would be happy to set up a phone meeting if easier. Thanks!
  • @conniekot: OK, so looking through the logs, it does look like you had upgraded to Zotero 5 before syncing, but then something happened during the upgrade sync that Zotero performs to switch from the old syncing system to the new one, and all the local items were uploaded.

    We'll look into this and see if we can figure out what triggered it. Generally speaking, there shouldn't be any problem syncing a library that hasn't been synced in months, even if other changes have been made elsewhere, unless many changes were made to the old library that were never synced with the server. But the upgrade sync is particularly complicated, so it's possible that the combination of that and all the remote changes resulted in unexpected bugs.

    There's not currently any sort of straight restore-to-server function in Zotero 5, but you should be able to force all local items to be uploaded by adding a temporary tag to all items, which you can do by creating a tag, pressing + in the middle pane to expand all items, doing a Select All, and dragging all the items to the tag. Once you've set up syncing (with auto-sync disabled for the moment), you can then right-click on the library and choose Sync to sync just that library. All items should then sync up. If you get conflicts, choose the local items for all changes. Once that goes through, you can delete the tag from the tag selector and sync again.

    I'm not sure what you mean above by "preserve URIs", or what you're saying happened with attachments. As I noted above, even with a problem with conflict resolution it wouldn't have been possible for attachments to have been deleted, and the library shows no recent deletions. It's possible that newly added attachments would've become separated from their parent items, even if they still appeared in the library. The steps above should restore those.

    Obviously, make sure you have a backup of the backup before trying this, in case you need to repeat it.

    Let us know how it goes and we can take it from there. And sorry for the trouble — we'll try to figure out happened here. I suspect, whatever the problem, it was specific to the upgrade sync, so once you're syncing with Zotero 5 things should work smoothly.
  • Thank you for your suggestions and time with our issues. We have followed the steps you recommended and unfortunately, it did not sync correctly for the group libraries. To be clear, we 1) imported a copy of the good contents of the Zotero folder to a new local computer, with the latest Zotero 5.0 version, with auto-sync off 2) Added a new tag "RESTORE" to the local copy, 3) right clicked the selected group library (IKI-BackgroundLit) and chose sync. The first time we did this, it only tagged a subset (~251 out of 351) references with "RESTORE." We ran the sync again with the tag "RESTORE_2" and that seemed to work. We replicated these steps (syncing multiple times before all of them were properly tagged with a "RESTORE" tag) with another larger group library and unfortunately saw incorrect syncing yet again. Namely, some tags that were in the original, local computer were lost when synced to the server. We then took it one more step in deleting all references in the IKI-BackgroundLit group library on the Zotero server (similar steps to delete posted here, but for a group library and we did not do any reset https://forums.zotero.org/discussion/27185/deleting-entire-library-for-a-re-import) and ran the sync again. As the sync ran for the 351 references, we then proceeded to see multiple references deleted from our local library (with no pop-up error messages) and when it came to the last 85 references, a pop-up dialog asked if we wanted to restore the local or remote copy. We chose the local copy for all 85 instances and after the sync finished, there was nothing in our library both on our local and Zotero server. Only a very small subset of references were found in the trash - why would it go to trash anyway????

    Please explain and suggest other steps to restore our local group copy (with attachments) to the online Zotero server so that we can share with multiple group members. Please keep in mind that we want to preserve Zotero keys/unique identifiers (which I had named as URIs). I understand these keys as unique within a group library so that creating an entirely new group library and moving references there, pushing it up to the Zotero server may be a work around, but these steps will create all new Zotero keys for those references and this is definitely not ideal for us.
  • The first time we did this, it only tagged a subset (~251 out of 351) references with "RESTORE." We ran the sync again with the tag "RESTORE_2" and that seemed to work.
    I'm not following this part. Adding a tag is something you do locally, before syncing. So I'm not sure what you mean by "syncing multiple times before all of them were properly tagged with a "RESTORE" tag".

    To clarify, the point of adding a tag is to create a local change that preserves the local state of the item, even if it triggers a conflict. If you assign a tag, sync, and then delete the remote item, that's just going to delete the local item too on the next sync.

    We're looking into the behavior of the conflict resolution checkbox, since it sounds like that's not working properly for you, and none of these steps will work if that's not working.
  • In step 2, we added the "RESTORE" tag locally, before syncing. We then synced the specific group library to the Zotero server (which at this point has references within it). After the sync completed, we saw that only a subset of references within the group library on the server received the RESTORE tag. We then tagged (locally) ALL references again with another tag "RESTORE_2", hit the sync button again and it looked like "RESTORE_2" were attached to all of the references. This first problem is disturbing in that syncing does not seem to be working with ALL references, and does not throw up any messages/conflict resolution dialogs. We only notice that it did not sync properly because it was easy to check whether or not ALL references got updated with a "RESTORE" tag.

    The second problem is that when we sync a local database to an emptied library on the Zotero server (same library, but all references manually deleted online), we could not successfully push the local copy to the server online. You mentioned: "If you assign a tag, sync, and then delete the remote item, that's just going to delete the local item too on the next sync." "On the next sync," we would expect a conflict resolution pop-up message where it would ask if we wanted to choose the local (present reference) over the remote (deleted reference) - is this correct? We would expect that "on the next sync," we would be able to choose "local" so that the present reference would be synced up to the Zotero server. Besides the fact that it is not working (all references lost on both Zotero server and local server after sync finished as a result), we also are worried as to why only a subset of references (85) were flagged as having a conflict when it should have been applicable to all of them (351).

    Please let us know how to resolve the conflict resolution checkbox issue, as it is not working as expected and what got us in trouble in the first place.
  • Assigning a tag to all items happens in the background and can take a few seconds for lots of items, so if you synced immediately it's possible it just hadn't been completely applied locally. You can confirm locally that the tag has been applied before syncing.
    You mentioned: "If you assign a tag, sync, and then delete the remote item, that's just going to delete the local item too on the next sync." "On the next sync," we would expect a conflict resolution pop-up message where it would ask if we wanted to choose the local (present reference) over the remote (deleted reference) - is this correct? We would expect that "on the next sync," we would be able to choose "local" so that the present reference would be synced up to the Zotero server.
    No, it doesn't work that way (and you wouldn't want it to) — it's comparing changes since the last sync, not just the current state of both sides. So if you assign a tag locally and sync, and either the item hasn't changed remotely or the changes on each side can be merged automatically, then the sync goes through. If you then delete the remote item and sync again without changing the local version, that's just a remote deletion and the local item disappears.

    But this automatic merging does mean that, depending on what the remote changes are, deleting the items remotely is probably a good idea to generate a cleaner conflict. You can do that on another computer that's already in sync, though, which should be easier than using the website. So here's an amended process:

    1) Make sure auto-sync is off.

    2) Make sure you have a backup of the whole data directory, including the 'storage' directory.

    3) Using a database (or another computer) that has that library and is fully in sync, delete all items, empty the trash, and sync again. All items online should be deleted without conflict. Confirm that on the website.

    4) Restore the backup database, create a temporary tag, expand all items in the middle pane by pressing +, Select All, and drag those to the tag. (Assigning a tag will update Date Modified for the parent items, so if you want to avoid that, you could alternatively create a saved search that matches Item Type 'note' or Item Type 'attachment', do a Select All while viewing the search, drag those items to the temporary tag, and then back in the library root drag all the top-level items to a temporary collection, which doesn't update Date Modified (but only applies to top-level items).) Click on the tag and/or collection to confirm that the changes have been applied to all items.

    5) Go to Help -> Debug Output Logging -> Enable.

    6) Right-click on the library and Sync. That will hopefully generate a conflict for all items in the library (since they're all modified locally and all deleted remotely), and you should be able to choose the local versions to keep them.

    If that doesn't work, submit the debug output and provide the Debug ID for the sync, and we can investigate further.
  • Ok, I have followed your instructions starting at #4. As stated earlier, we already have an emptied library on the Zotero server, thanks to the syncing error, so I confirmed again that it was empty on the Zotero server (IKI-BackgroundLit). I copied over the backup database/Zotero folders to a new computer and continued with steps 5 and 6. I have submitted the Debug ID D912360037. I also noticed that pdfs on restored databases were not linking correctly (got message "File not Found" on second computer, but file looks like it is CREATED when trying to open it through Zotero on the original computer - I will start a new thread on this issue shortly but mentioned it here on the off change that it is related.
  • No, let's keep everything in the same thread for now.
  • So you provided a Debug ID, but does that mean something isn't working? If not, what exactly happened?
  • Sorry - thought you would see in the debug id what exactly happened. Library is completely empty on local and Zotero server. As described before, most of the references started disappearing from the local server upon sync and and 3 conflict resolution dialog boxes popped up at the very end - one for 85 references (I choose to keep the local for all), another one afterwards for 11 references (chose local for all), and one last one for 11 references (also chose local for all). Sync continued and completed until all references were emptied from the group library.
  • OK, so all the items were actually uploaded, but they're in the trash. Do they not appear there in the local library?
  • I currently see 1061 items in the trash in my local library. I believe we started out with 46 items in the trash, and 1036 items in the library, so I am not sure how to tell which ones were moved to the trash from the library vs which ones should stay in the trash.
  • You'll have to figure that out, or you can start again and make sure the trash is empty 1) in a library that's currently in sync with the online library and 2) in the restored library before you first sync it.
  • I restored the backup again on another computer, turned off auto-sync, turned on debug error logging, then I emptied the trash on my local (I tried to avoid this in case I need those synced up - is this absolutely necessary?). I emptied the trash on the server for the group library (selecting 25 references at a time, is this the most efficient way?). Zotero online was definitely spinning for awhile for each page deleted until I refreshed the browser to show that the trash was emptied). I then added a tag to all 1036 items in my local group library and synced. 2 conflict error messages popped up for 1036 items, where I chose the local for all cases both times. It looks like there are 351 items on the server and 1036 items in the local. Almost there, but not quite - the tag I just put on the last trial (RESTORE-cyk2) is not on the items, but curiously all the other trial tags are there on the server (e.g., RESTORE, RESTORE_2, but they are not present on my local). I have submitted my debug id as D1968864437
  • Just to clarify, upon the last sync, I only added "REMOTE-cyk2" as an extra tag (not "RESTORE-cyk2) to the original database. After the last synced finished, I do not see the REMOTE-cyk2 tag on the server, though the references populated there (I'm not sure if it is all or a subset without the tag). Other tags that I have used in the past trials described in this thread (RESTORE, RESTORE_2, RESTORE_cyk) are listed as a tag on the server, but when I select it, there aren't any items that are found with those tags. I guess Zotero stores all tags ever used, even when orphaned???? On my local copy, I do not have those "orphan" tags listed.
  • It looks like there are 351 items on the server and 1036 items in the local.
    There are 1036 items online, with 350 top-level items.

    Are you saying you see all items separated without hierarchy in the local library?
  • I see them with a hierarchy locally, I just have 1036 items when expanded. I am not sure how to check that on the server
  • OK, so then it seems you're in sync.
  • I disagree, as my latest tag is not attached to any of the references, nor listed in the tags on the server. I just noticed that on my local, I have lost all tags and the only tag I have showing locally is "REMOTE-cyk2" and remote sensing," both of which are not listed on the server of the same library
  • sorry, I do have all of the tags locally (didn't notice that I had "remote" in my search box). REMOTE_cyk2 is a tag on all references locally, but I do want to know why I don't have REMOTE_cyk2 on the server references
  • It's not really worth thinking about. We're reviewing the conflict resolution code, and should have some fixes in the next releases, but for now just delete the temporary tags and you should be fine.
  • (In other words, while it seems like something went wrong with the initial sync of your outdated library, and we're looking into what might have happened there, the rest of this has just been trying to hack the syncing and conflict resolution system to get the desired result, in lieu of another existing method for doing a clean restore of a backup to the server. So we're well outside the bounds of normal syncing operation. Generally speaking, this should all just do what you want automatically.)
  • Thanks for your time and support. I'm going to contemplate the tags/check on this more, but another (maybe related?) quirk I noticed was that all of the pdfs are definitely not being restored. When I moved the good copy of the Zotero folder from one Mac computer (A) to another Mac computer (B), I was unable to open pdf attachments on computer B through the Zotero app (right clicking and "View pdf," though computer A could open and view when double clicking on the pdf attachment within Zotero (opening up a third party to read the pdf). When moving the same good copy of the Zotero folder from computer A to a different computer (C, a Windows machine), pdf attachments for the same references were able to be opened. Upon several trials (hours!), I saw that when right clicking on a pdf attachment to "Show File," I get a message "File Not Found" on all computers (A, B, C), but when double clicking on the pdf attachment within the Zotero app (in essence, "View pdf"), computer A and C is able to view/open the pdf using third party software. It looks like when "View pdf" is selected on computers A and C, the pdf suddenly appears (is it created?) in the proper storage folder so that the next time you select "Show File" it now points to the pdf that was "newly created" in the storage folder that was blank earlier.

    It seems like before copying the Zotero folders for a backup, the only way to port pdfs over correctly to another Mac computer is to open all pdfs in Zotero first???? I've done a trial where I opened 2 pdfs on computer A within the Zotero app (it created the pdf in the folder), copied the entire Zotero folder contents to computer B again to replace that computer's Zotero folder contents, and clicked on those references and was able to open/view them on computer B (a Mac), but no other pdfs that haven't been "created" before the copy were able to be found/linked/recreated on computer B.

    I'm not sure if it is even perfect on the Windows machine either, but any advice you can give for properly restoring pdfs from a Mac would be helpful.
  • In recent versions of Zotero 5, if file syncing is enabled, Zotero will automatically download a missing PDF that's available in the online library when you try to view it, regardless of the whether file syncing is set to download files at sync time or as needed. In earlier versions of Zotero 5 and Zotero 4, it had to be set to "as needed" for that to happen. So make sure that file syncing is enabled for groups on computer B and that you have the latest version of Zotero 5.

    But this would only matter if the files are missing from the 'storage' folder. That'd be the case if it's an older version of 'storage', a version from after the local items were deleted (regardless of whether you then swapped in a backup database), or you simply had "as needed" set before and the files never downloaded to that computer. As long as they're available online, there's no particular need to have them on every computer unless you'll be working offline.

    (It's possible that "Show File" should trigger the same automatic download action and then show the file, but that doesn't happen at the moment.)
Sign In or Register to comment.