Library partially synced [solved]

tluge
I was using Zotero exclusively on a Linux computer and have been trying to sync its data to a Windows PC at work. My Zotero data is very large (over 15k items and ca. 100GB of files). It was thus not possible to download all to the new client in one go (due to office hours, ADSL disconnect, etc.). The data is on a WebDAV server.

My fist attempt failed miserably: I was granted permission to keep the PC on and keep syncing over night. However, the process had somehow stalled (Zotero was unresponsive and there was no data traffic) when I returned to the machine (possibly due to an ADSL disconnect?), so I had to quit Firefox. After restart, Zotero would travel the directories/files and then end the sync without downloading anything. I rest the file sync history and retried, but now Zotero would also upload data (about one third of all files downloaded) to the server. Obviously, this was plain wrong and I was thinking that the cause must either be corrupted files or encoding issues. I checked a few files that were being uploaded and they opened just fine (they were PDFs, so corruption would probably have broken them completely). I checked the encoding (although I am not completely sure I did this correctly) and that seemed fine as well. I thus reasoned something went completely wrong and I proceeded to end the process, reupload all data from my home (Linux) machine to the server, and delete the Zotero data on the new (Windows) machine.

My second attempt is now at a very similar position as the first one early on: I set up Zotero on the new machine and started the sync. This time, I manually ended it before leaving work. After returning from the holidays, I hit the sync button again and the sync finished without any uploads, although the vast majority of the data does not exist on this client.

I have not generated a report in order not to spam the Zotero devs. I guess I did something obviously wrong, but I am out of wits. Any pointers?

Thanks in advance!
  • adamsmith
    If you're struggling with a slow-ish connection and a very larger library, why not do the initial transfer via a USB stick and method A) described here:
    http://www.zotero.org/support/kb/transferring_a_library

    If you do want to try this with sync, it would help if you could distinguish between data and files - those are two separate, sequential sync processes: data syncing - i.e. transferring the items that appear in Zotero either works or it doesn't, it runs through the Zotero server and you can't get a partial sync. File syncing (i.e. attached snapshots and PDFs) runs through your webDAV and that can work incrementally by session.
  • tluge
    Thank you for your quick response. In fact, transferring the data and files directories via an external drive was the first method I tried, but Zotero failed to open the database on the Windows PC due to an access error I was not able to solve. The same file opened just fine on Linux. I was also unsuccessful in trying to copy only the file directory after an initial sync of the data. The files were ignored by Zotero.

    To make it more clear: the data sync works just fine. However, the file syncing via WebDAV is not completed.
  • adamsmith
    just copying over the storage folder in the data directory should almost certainly work: http://www.zotero.org/support/zotero_data
  • tluge
    Well, that page actually always talks of copying data and storage directories together, I am not sure where you read that this should "almost certainly work." Nonetheless, as I wrote just above, I did try this. when the Zotero database was not readable on Windows (although it works just fine on Linux), I started again with an empty Zotero directpry, synced only the data and then copied the storage directory in. However, as I mentioned above, Zotero acted as if there were no files there and proceeded to download the entirety of the WebDAV-stored files again (and that failed to complete).

    If you are _sure_ your suggestion works and is the only one to work, I can try it again. However, I would like to know whether I need to reset the sync history (for data and files or only for files) or not. There is not sufficient information about how Zotero actually behaves in these cases on the support pages.

    My big problem with this workaround is my fear that the first file sync operation will screw up the data on my WebDAV server yet again. I have already had to upload 100GB to it to return it to its old state and I would not like to have to do it again.

    Thus, my initial question from my fist post: why does my Win-based Zotero not download all the files from WebDAV although only ca. 10% of them are actually in the storage directory? If I want to get the files from the WebDAV server (which should obviously work, right?), what do I need to do? How can I go about troubleshooting this instead of repeating steps I have taken before that have only screwed up my WebDAV server state and caused much additional work? Is WebDAV known to be broken?
  • adamsmith
    I'm quite certain copying the storage directory should work, yes. It's usually not necessary, which is why it's not mentioned anywhere, but file sync does essentially that (in fact, there are people who sync the storage directory by placing a symlink into dropbox).
    I understand it looked like Zotero tried to download all files again (that doesn't necessarily mean it did - e.g. after resetting the history, it will _run_ through all files and check them, which will display the file sync progress bar, but it won't actually redownload or reupload anything that's already there) , but did you actually try accessing the files (i.e. opening one from Zotero) on the Windows machine?
    There are no known bugs with webDAV and Zotero - is this your own server, one supplied by an institution, or a commercial webDAV? In general, the troubleshooting steps here apply: http://www.zotero.org/support/kb/files_not_syncing
  • tluge
    I emptied the file directory and copied the folders from my Linux machine into it. The file sync did not only run through the folders, but proceeded to download the content already there locally anyway, just like before.

    So I proceeded to delete the Zotero folder (database & files) and started fresh yet again. As before, there were no problems with data sync, but the file sync does not resume, if I manually cancel the sync operation when it stalls indefinitely after network disconnect (well, I presume it's the latter, it may also be that the server becomes unresponsive from time to time). However, if I simply quit Firefox without cancelling the sync, new attempts at continuing the file sync work (thus far at least).

    Right now, Zotero is syncing just fine since yesterday morning (I restarted the browser twice without adverse effects). It will take a few more days for all files to be downloaded. I hope this time it will work and will report back. For now I can recommend anyone in my situation not to cancel the file sync when stalled, but simply to quit Firefox.

    Thank you again for your help, adamsmith! I hope I won't have to bother you again.
  • tluge
    I just wanted to add that I was finally able to sync all files to the Windows machine. I am still not able to explain the errors I encountered. Maybe it's something with the WebDAV server. But if someone runs into similar issues (i.e. an initial sync of a large set of files not continuing after the connection was reset or the server timed out), I would recommend trying to simply quit Firefox instead of canceling the file sync from within the Zotero add-on.

    Again, thanks for the help!
Sign In or Register to comment.