Syncing large collections
This is an old discussion that has not been active in a long time. Instead of commenting here, you should start a new discussion. If you think the content of this discussion is still relevant, you can link to it from your new discussion.
Hold on-- it seems that the server side is getting a lot of attention these days, and the Zotero team is slowly adding support for larger and larger databases.
(Also, I bought a lot of extra storage from Zotero, thinking that would help with this problem, but I think Dan told me that the storage has nothing to do with that...so I'm not sure what the storage is for? Should I be saving my PDFs there instead of on my hard drive?)
Thanks,
MM
It hasn't sync'd since 22FEB2010.... same issue?
Should I go back to jungledisk webdav for the moment?
Debug output under D1190164621
[JavaScript Error: "[Exception... "'Error processing uploaded data (Report ID: e10b1d18)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]
(The reason the error message changed for you is that we worked around the previous third-party bug that had been preventing large uploads, but your DB is still too large to process. I've now restored a proper error message for such databases.)
Most people who posted to this thread should currently be able to sync, though your upload might remain queued on the server for a while. A few people will need to wait a bit longer while we work to support very large libraries (and reduce queuing for everybody). Thanks for your patience.
It's been several months since I was last able to completely sync my database (2.0 didn't magically work for me), so now that I've finally hacked through a solution, I'd like to share it. I'm posting here since the last error I got was "'Databases of this size cannot yet be synced. Please check back soon." I have 3446 articles in my personal library, and 3836 in my primary group library.
Whenever I tried to sync, I had a major problem: After reconciling the conflicts (usually 40 to 100), I got an unending stream of tag notices--not errors, but notices. I described the problem here: http://forums.zotero.org/discussion/10169/sync-error-report-id-2082003221-unending-tag-message-boxes/
I did not think that the two issues (endless tags and inability to successfully sync) were related until I read this thread, and noted Dan's comment from above: "The error message is a bit misleading—it can also happen if you have a lot of tags, authors, etc."
Aha! I knew that I had a lot of tags, which were almost all useless junk imported from gazillions of database keywords. So, I searched around and found this thread about removing tags: http://forums.zotero.org/discussion/4051/remove-all-tags/
Unfortunately, there's no easy way in Zotero to remove multiple tags; the functionality is not built in (yet). Thus, I went the hack way as described by lmullen in the thread: First I disabled the automatic addition of tags (described by Rintze in the thread). Then I backed up Zotero, I downloaded an SQLite browser (I obviously could not use the Firefox SQLite Manager extension), and deleted all tags in the Zotero database (for those who want to save user-generated tags, there are some ideas for this in the thread; I didn't bother--I zapped 'em all). I saved the database and exited, restarted Firefox, and synced. Bingo! First successful sync in several months.
In summary, the culprit to my problem was having too many tags for sync to work properly; this was caused by automatically adding tags from articles that I added to the database. The solution was to delete the tags directly from the SQLite database, and then disable automatic tag addition so that it doesn't happen again.
I hope this helps someone here who was as desperate as I was--I love Zotero, but it had reached the point of being unusable. Now I can love it again :-) If only the duplicate handling could be completed, then Zotero would be perfect for me :-)
My database is only about 6,000 citations (3.2GB). I have deleted all tags, but there is likely a few references with numerous and non-standard authors. If I thought it would help, I could manually delete the references with large numbers of authors, but it doesn't seem worth the effort - unless I knew for sure that was the problem...
Any suggestions on how I could troubleshoot this? I have already deleted all tags, and I am not keen on breaking my database across multiple firefox profiles, as this would be impractical to go back and forth between them.
I have tried generating an error log (preferences>advanced>debug output logging) when trying to sync, but the browser locks up briefly and the log file jumps to >350,000 lines written when it unfreezes (making it too large to open...).
Does it make any difference where my webdav is located (ie zotero webdav vs other)?
I have tested the integrity of the database (preferences>advanced>Database maintenance), and no errors are found.
-firefox3.6.3, zotero2.0.3
Any suggestions appreciated.
Thanks.
Thanks for any help.
And since this is a hard-coded limit that's keeping you from syncing, once you get it working, send a message to storage@digitalscholar.org and your storage subscription can be extended.
But happy to know you are working on it :-)
Report ID: 1542289418
[JavaScript Error: "[Exception... "'Databases of this size cannot yet be synced. Please check back soon. (Report ID: 2b919136)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "<unknown>" data: no]"]
That said, for people who are the single users of Dropbox and only need asynchronous access from multiple computers, I agree with gerhard221 that Dropbox is an excellent solution, probably better than the Zotero sync mechanism. Unfortunately, that's not my situation.
> Be patient a little bit longer—improvements really are forthcoming.
I posted earlier how deleting tags had solved my sync problems, but I have since added another large project, and now I can't sync again (I have at least three distinct libraries with over 5,000 citations each); maybe my problem now is too many authors.
Any update on when large library syncs will be fixed? This is a real show-stopper for me, since the nature of my research (systematic literature reviews) necessarily involves huge numbers of citations.