Unnecessary syncing with the Standalone

imuntean@gmail.com
I have the latest version of the Standalone Zotero and i use typically three computers to sync a library of about 2000 items. I have all my PDF files store on the box.com server. Every time I make a change one of the computers would sync the whole library (and it takes a while). Then if i make a change, add or edit an entry, the other computer would synchronize again the whole library.
I do not know what to suspect here: settings at box.com, a problem with the time-stamp of each file zotfile corrupt database or whatever.Any help would be greatly appreciated.
I do not have access to another webdav friendly server for the time being. If somebody using box.com as a webdav repository had the same problem, please let me know.
  • adamsmith
    how do you determine that a change of the whole library takes place? What exactly happens?
  • dstillman
    A couple things:

    1) If you have a large number of missing files and add/update a file, Zotero on other computers will currently recheck the server for all of the missing files on the next sync. It should be smarter about this. If no files have changed remotely, it knows enough not to bother checking, though.

    2) If you have a large number of files, you probably don't want to use manual syncing, which forces Zotero to check all local files for changes. With auto-sync, it only checks local files 1) on the first sync of a session, 2) once every three hours, and 3) if a file was previously modified in the last three hours (suggesting that you might be actively working on it). We're looking for other ways to improve this, but it's a tricky balance between keeping locally modified files synced and not negatively affecting performance, and we decided that having a click of the Sync button potentially not send up a locally modified file would be a bad thing. (We might still be erring on the side of too much checking, since the vast majority of files are likely never changed, but there are people who use PDF annotation, etc. It's also possible we could tie into OS-specific features for file notifications, but that'd be a lot of work and probably isn't feasible on all platforms.)
  • dstillman
    edited November 10, 2013
    To be clear, though, Zotero isn't actually syncing the whole library, as you claim. It never transfers files that haven't changed. But it does check local files for changes, and when you hover over the blue progress bar it shows the files being processed.
  • imuntean@gmail.com
    Adamsmith: thanks. The sync process shows the bar and in fact transfers something, as i monitor my network connection. In general even if i do not change anything on any computer, the process is happening randomly. It usually shows me downloads, but sometimes it does uploads, too. The process goes from 1800 files to 80 files, but again these are not new files.
  • dstillman
    The progress bar doesn't mean it's transferring files, though — it shows that for checking them locally for changes, as I describe above. And if you have missing files it would make requests to check the presence of remote files, which would create some network traffic, but it wouldn't be transferring files themselves.

    Anyway, if you provide a Debug ID for a sync that exhibits this I'll take a look.
  • imuntean@gmail.com
    Dan: I did not do a manual check, it just starts when I restart the program or after a hibernation. Another thing: after some long process of synchronization, I see the announcement: last sync: -300 sec or any other negative number. OTher than than, this process does sync the files and the PDF files I add or change. Again, I use a webdav server at box.com. I am considering moving my archive to a different webdav host, but I still want to see whether there is a solution.
    Dan: as you said, the process looks like it checks local files with no transfer, i agree. Interestingly enough, box.com does not record new added or modified files. But some traffic is present. I will provide a debug ID immediately. Thanks a lot!
  • dstillman
    Just to be clear, you have to post the Debug ID here for us to look at it.
  • imuntean@gmail.com
    Dan
    Yes, I finally figured out how to generate the Debug ID. Here it is: D42175052.
    To be clear, since my last posting i realized that when Zotero is checking the database, several things are happening,.
    - zotero process eats tons of resources.
    - sometimes the sync process takes one hour
    - the download button does show 0Kb,
    - zotero runs much slower than usual.

    the sync process occurs even if i do not close the zotero standalone
    Thanks again!
    Ioan
  • imuntean@gmail.com
    Just a last word: if i change an attached PDF file, the download process that in fact downloads nothing completely obliterates the upload process in which i am interested and takes again a lot of time to complete. At the end the file is uploaded.Thanks in advance for any suggestion.
  • dstillman
    OK, so a few things:
    Checked 1909 files in library 0 in 94829ms
    This shows a 95-second delay during auto-sync, during which it's checking 1900 files for changes. Is that the slowdown you're referring to? Is your data directory in its default location on a local hard drive, or have you moved it elsewhere?

    Again, this would occur in the three exact situations I describe above, and it's the best solution we have at the moment for the problem of keeping locally modified files in sync. It won't occur for most auto-syncs (as your debug output, which includes a bunch of syncs, shows). We're still looking for other ways to improve this, but your computer should still be usable during those 95 seconds every three hours — just a bit slow.
    Remote file not found for item 0_2ETEPKJT
    It looks like you have about 25 missing files, and Zotero will check the WebDAV server for them if the previous WebDAV sync uploads files. (It should really only do that on other computers, in case the previous sync uploaded a missing file, but I think right now that even happens on the same computer that just uploaded something, which is a bug.) Zotero will get smarter about dealing with missing files in the future, but you might as well delete those attachment items (and empty the trash).
    (3)(+0003680): HTTP GET https://dav.box.com/dav/zotero/55PIVIA3.prop succeeded with 404
    This shows that a simple request to Box is taking 3.6 seconds, and this request was not an outlier. This wouldn't cause any local system load, but it would definitely make file syncs take a while, including the checks for missing files described above. I do see some relatively slow requests to the Zotero servers as well, so I'm not sure if this is Box being slow or your Internet connection.
    - sometimes the sync process takes one hour
    I'm not seeing anything like this in your debug output. If you can capture one of those, I can certainly take a look.
  • imuntean@gmail.com
    DAn
    Thanks. I can record again a long sync process that was much longer that 95 seconds. But thanks for this valuable ideas. First off, how do I detect missing files? In other words, how do i find the entries with missing files? I suspect this is a major issue and I can take a look at it. I will come back to results after i will fix this
  • dstillman
    You should be able to find missing files easily by sorting by the blue dot column in your items list. (If it's not showing, click the column picker in the top-right and enable "Attachments".) If the blue dot is empty, there's an attachment with a missing file. (You may need to expand the parent item to see which attachment is missing.)
  • imuntean@gmail.com
    Dan
    That was easy. Now I eliminate dead links or updated links to all the items in my library. I follow the instructions on the forums about how to sync the other computers. I have the feeling that this was the problem. The computer with fixed links acts totally normal now. I'll come back later. Thanks a bunch!
  • imuntean@gmail.com
    Dear Dan and Adam
    I am pretty confident that the problem was solved and that the cause was a large number of missing files in my database. One reason was some PDF files which for some reasons have disappeared from my Webdav folder. The other reason was a number of links, not files that were related to some local azw files created by the kindle software. I forgot to install it on a computer so it reported them as missing files. I still have two entries with links to HTML files to NYT articles that are showing up as missing files, although the links and the articles are online. It looks like that you solved the problem for me.

    Congratulations for the amazing job you do with ZOTERO. All the kudos to the team!
  • imuntean@gmail.com
    Dan
    One more word about this issue: on different machines the paths to the azw files were different, as at work i have a different username than on my home computer. the solution came only recently with the inclusion of a relative path to folders. Without it i would not be able to find the files on all computers. I may unlink azw files,but i thought it was cool to be able to access azw files from my zotero library.
    All looks okay now. As kindle limits the number of PCs on which you can open their books I avoided installing those files on the PC at work.
    Thanks again. The ideas of a relative path helped a lot! Best wishes
Sign In or Register to comment.