Zotero Syncing - version history and restoring tags and attached files
I work with zotero on a large, collaborative project (about 8 people per library) where we store thousands of journal articles to be reviewed and work across 4 different taxa specific zotero libraries. A colleague opened an old computer yesterday (Oct 3, 2017) that she hadn't used since June 2017 and synced Zotero. She chose the "remote" version for all syncing but zotero updated the remote version with her local instance, thus deleting 6 months of organization, tagging, and file attachments to each of the references. Why did this happen given that she chose the "remote" version for all syncing? How can this be prevented? What is the best way to restore both the tags and information, as well as all of the pdf file attachments?
If you have anyone with a recent library intact, make sure they disable auto-sync for now.
This shouldn't be possible, in particular with respect to file attachments (with Zotero 4, syncing tags/collections can get you some wonky behavior, but that should always be to add tags to items or add items to collections, nothing else.)
Yes, my understanding was that this was not possible as well. I'm not sure if maybe the local instance on her old computer that hadn't been opened since June was just such a different version that Zotero got confused and overwrote the remote instance? Some tags appear to be current while other tags got deleted so Zotero must have been inconsistent when choosing to sync local --> remote and remote --> local even though it was told to sync remote --> local.
Do make copies of the most recent .bak files. They'll get overwritten eventually, so you want to keep the last version.
If you email support@zotero.org with the username this happened to and a link to this thread, we can look more closely. For what it's worth, it looks like there were uploads from Zotero 4 from two accounts in these groups in the last few days, so that could be part of it (though it shouldn't have been possible in 4.0 either).
For recovering, the first thing (after everyone has made a copy of the .bak files) would be to temporarily disable auto-sync on a computer with a .bak file with a timestamp from before the changes were made or synced down, and then swap it in as zotero.sqlite. When you start Zotero, you should then see the library in the previous state.
Let's start with that, just so you can be sure you have a copy of the library as you want it, and then we can walk you through how to apply those changes back to the online library. Before we do that, I'd want to know more about what kind of changes you saw.
We'll look into this and see if we can figure out what triggered it. Generally speaking, there shouldn't be any problem syncing a library that hasn't been synced in months, even if other changes have been made elsewhere, unless many changes were made to the old library that were never synced with the server. But the upgrade sync is particularly complicated, so it's possible that the combination of that and all the remote changes resulted in unexpected bugs.
There's not currently any sort of straight restore-to-server function in Zotero 5, but you should be able to force all local items to be uploaded by adding a temporary tag to all items, which you can do by creating a tag, pressing + in the middle pane to expand all items, doing a Select All, and dragging all the items to the tag. Once you've set up syncing (with auto-sync disabled for the moment), you can then right-click on the library and choose Sync to sync just that library. All items should then sync up. If you get conflicts, choose the local items for all changes. Once that goes through, you can delete the tag from the tag selector and sync again.
I'm not sure what you mean above by "preserve URIs", or what you're saying happened with attachments. As I noted above, even with a problem with conflict resolution it wouldn't have been possible for attachments to have been deleted, and the library shows no recent deletions. It's possible that newly added attachments would've become separated from their parent items, even if they still appeared in the library. The steps above should restore those.
Obviously, make sure you have a backup of the backup before trying this, in case you need to repeat it.
Let us know how it goes and we can take it from there. And sorry for the trouble — we'll try to figure out happened here. I suspect, whatever the problem, it was specific to the upgrade sync, so once you're syncing with Zotero 5 things should work smoothly.
Please explain and suggest other steps to restore our local group copy (with attachments) to the online Zotero server so that we can share with multiple group members. Please keep in mind that we want to preserve Zotero keys/unique identifiers (which I had named as URIs). I understand these keys as unique within a group library so that creating an entirely new group library and moving references there, pushing it up to the Zotero server may be a work around, but these steps will create all new Zotero keys for those references and this is definitely not ideal for us.
To clarify, the point of adding a tag is to create a local change that preserves the local state of the item, even if it triggers a conflict. If you assign a tag, sync, and then delete the remote item, that's just going to delete the local item too on the next sync.
We're looking into the behavior of the conflict resolution checkbox, since it sounds like that's not working properly for you, and none of these steps will work if that's not working.
The second problem is that when we sync a local database to an emptied library on the Zotero server (same library, but all references manually deleted online), we could not successfully push the local copy to the server online. You mentioned: "If you assign a tag, sync, and then delete the remote item, that's just going to delete the local item too on the next sync." "On the next sync," we would expect a conflict resolution pop-up message where it would ask if we wanted to choose the local (present reference) over the remote (deleted reference) - is this correct? We would expect that "on the next sync," we would be able to choose "local" so that the present reference would be synced up to the Zotero server. Besides the fact that it is not working (all references lost on both Zotero server and local server after sync finished as a result), we also are worried as to why only a subset of references (85) were flagged as having a conflict when it should have been applicable to all of them (351).
Please let us know how to resolve the conflict resolution checkbox issue, as it is not working as expected and what got us in trouble in the first place.
But this automatic merging does mean that, depending on what the remote changes are, deleting the items remotely is probably a good idea to generate a cleaner conflict. You can do that on another computer that's already in sync, though, which should be easier than using the website. So here's an amended process:
1) Make sure auto-sync is off.
2) Make sure you have a backup of the whole data directory, including the 'storage' directory.
3) Using a database (or another computer) that has that library and is fully in sync, delete all items, empty the trash, and sync again. All items online should be deleted without conflict. Confirm that on the website.
4) Restore the backup database, create a temporary tag, expand all items in the middle pane by pressing +, Select All, and drag those to the tag. (Assigning a tag will update Date Modified for the parent items, so if you want to avoid that, you could alternatively create a saved search that matches Item Type 'note' or Item Type 'attachment', do a Select All while viewing the search, drag those items to the temporary tag, and then back in the library root drag all the top-level items to a temporary collection, which doesn't update Date Modified (but only applies to top-level items).) Click on the tag and/or collection to confirm that the changes have been applied to all items.
5) Go to Help -> Debug Output Logging -> Enable.
6) Right-click on the library and Sync. That will hopefully generate a conflict for all items in the library (since they're all modified locally and all deleted remotely), and you should be able to choose the local versions to keep them.
If that doesn't work, submit the debug output and provide the Debug ID for the sync, and we can investigate further.
Are you saying you see all items separated without hierarchy in the local library?
It seems like before copying the Zotero folders for a backup, the only way to port pdfs over correctly to another Mac computer is to open all pdfs in Zotero first???? I've done a trial where I opened 2 pdfs on computer A within the Zotero app (it created the pdf in the folder), copied the entire Zotero folder contents to computer B again to replace that computer's Zotero folder contents, and clicked on those references and was able to open/view them on computer B (a Mac), but no other pdfs that haven't been "created" before the copy were able to be found/linked/recreated on computer B.
I'm not sure if it is even perfect on the Windows machine either, but any advice you can give for properly restoring pdfs from a Mac would be helpful.
But this would only matter if the files are missing from the 'storage' folder. That'd be the case if it's an older version of 'storage', a version from after the local items were deleted (regardless of whether you then swapped in a backup database), or you simply had "as needed" set before and the files never downloaded to that computer. As long as they're available online, there's no particular need to have them on every computer unless you'll be working offline.
(It's possible that "Show File" should trigger the same automatic download action and then show the file, but that doesn't happen at the moment.)