Syncing large collections

breckenr

I'm starting this thread to try to keep track of any progress towards syncing large collections, by which I mean those in excess of 10,000 records. My real database has some 12,000 items and at present I am not able to sync it. The error is "Error processing uploaded data (ReportID: acaf55bc)."

breckenr

Still not working with 2.0b7 -- error report no 832705364

[JavaScript Error: "[Exception... "'Empty response from server' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

adamsmith

hey - not sure that's the problem but at least for me file syncing was on by default and set to webdav - which might be the "" for the location.
Check in your sync preferences.

dstillman

Yes, 2.0b7 won't help with this, but we'll be making some server optimizations over the next few days to try to enable the syncing of larger libraries.

(adamsmith: No, this one's strictly a metadata problem.)

breckenr

Similar error with 2.0b7.1 report ID 639442110:

[JavaScript Error: "[Exception... "'Error processing uploaded data (Report ID: fde19b8c)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

Interestingly I do now see other people's postings to shared libraries, including attachments, which is very useful.

dstillman

Interestingly I do now see other people's postings to shared libraries, including attachments, which is very useful.

Yeah, Zotero pulls data from the server before it pushes modified data up, so you'll see changes in group libraries (or from other computers linked to your account with smaller libraries) even if the local data is too big to upload.

breckenr

Still no luck. Report ID 1458722680, very slightly different error:

[JavaScript Error: "[Exception... "'Databases of this size cannot yet be synced. Please check back soon. (Report ID: 30b4a4e3)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

breckenr

Report ID 804892608 [JavaScript Error: "Databases of this size cannot yet be synced. Please check back soon. (Report ID: a63dfbd8)" {file: "chrome://zotero/content/xpcom/sync.js" line: 796}]

breckenr

With 2.0b7.4:

[JavaScript Error: "[Exception... "'Databases of this size cannot yet be synced. Please check back soon. (Report ID: 357fed4d)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

Report ID 383038272

whshep

You're not alone. I just imported my 6600+ database and get the same kind of error message when I try to sync.

breckenr

No change. Report 1340037159

[JavaScript Error: "[Exception... "'Databases of this size cannot yet be synced. Please check back soon. (Report ID: 814f5e02)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

breckenr

Report ID 1722674086

[JavaScript Error: "[Exception... "'Databases of this size cannot yet be synced. Please check back soon. (Report ID: 298adc11)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

mkgilson

I have Report ID 185449670, same kind of error message: databases of this size cannot be synced. Frustrating.

breckenr

Something new with 2.0b7.5 Report ID 240575483:

[JavaScript Error: "[Exception... "'Empty response from server' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

dstillman

While these large syncs likely won't yet work in 2.0b7.5, we've made some major architectural changes server-side, among other reasons to allow us to support larger syncs. So progress is being made.

mkgilson

If there were a tool to remove duplicate entries, my database would become small enough to sync with the present software.

I'm probably not the only person in this situation, so maybe a duplicate remover could be an early "fix" for the large database problem for a lot of users.

Thanks, in any case, for Zotero. It is a terrific step forward.

mdobe

I am getting all kinds of errors, different ones each time I try to synch. I have more than 5000 records. I guess my problem is probably that I have too large a database.

Here's how I think I will solve this problem. I will install multiple (topical) copies of firefox portable in different directories, copy over the entire DB to these directories, delete the entries that don't apply to the topic of the firefox portable instance, and then synch the individual instances separately.

Despite all of the issues I have struggled with in the last week, I gotta tell you that Zotero is awesome. I mean, it hasn't been since Netscape released its first beta that I was interested enough in software to work on this stuff on Thanksgiving morning!

mi9cal

I have a medium sized (7700 entries) endnote library that also seems to be hitting the
filesize limit (error #156807521). Any sense of when this will be fixed? Otherwise
this is an amazing system. As much of an improvement over stand-alone desktop
apps as they were over papercards!

But I do need the sync function to work properly between home and office machines.

Mike

mdobe

The large size DB issue is killing me on this. My DB on my Mac laptop has over 14,000 entries. I have tried copying and breaking this DB out into smaller DBs, but the task is absolutely daunting. In my main DB, I've copied so many citations to different collections that I end up blowing away records when I try to clean up.

The DB on my Mac is working very well, so I guess I will go back to the sneaker net approach of copying to a portable external drive, bringing that copy in to work, and restoring to the network. Fortunately the DB is cross platform, so I assume this will work as it has with previous releases of Zotero.

breckenr

I agree that a duplicate remover would help, but Zotero will remain a victim of its own success. My database grows by at least 100 records a week -- which is the point, I suppose. Keeping manual track of the current database is a small price to pay for the functionality, and I actually worry more about overall performance as the database grows ...

With everyone else I must offer congratulations; Zotero is completely changing the practice of research.

mi9cal

Any news on the syncing issue with large collections? Again, mine isn't mega-large
(>8K entries) but I still get the 'database too large' error message whenever I try
to synch.

Happy holidays!

Mike

breckenr

Still the same (Report ID 187648062):

[JavaScript Error: "[Exception... "'Empty response from server' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

arodrigues

I have the same error (Report ID: 53509702) but I assumed it was because I was syncing the files as well. I have a database of ~6000 refs with a lot of attached PDFs, such that the total size of my Zotero folder is 1.5 GB. Being able to syncing the files between computers was actually the main reason why I wanted to upgrade to Zotero 2. So I decided it was worth the investment and bought 5 GB of storage.

So it seems that I now have one year's worth of purchased storage that I can't use. In my storage page it shows my nice 5 GB quota but 0% use. Can I at least cancel my subscription (or stop the clock on it) until this limitation is sorted?

I add my voice to everybody else in congratulating the development team - this is a wonderful tool and I can't remember how scientific life was ever possible before Zotero.

breckenr

After a promising delay in uploading, the same error with 2.0rc2:

[JavaScript Error: "[Exception... "'Empty response from server' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "" data: no]"]

Report ID: 1562481015

Unlike arodrigues I'm not trying to sync my PDFs.

e14

[JavaScript Error: "[Exception... "'Databases of this size cannot yet be synced. Please check back soon. (Report ID: 0f5d1aa2)' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "<unknown>" data: no]"]


Report ID: 1406865726

Version 2.02rc

Database size < 6000

syncing only references, not pdfs.

Had just bought 10GB of storage space for our research group and now cannot use it.
But think fundamentally Zotero is a great piece of work.

dstillman

We'll be working to address this as soon as 2.0 is out. (It's due to an underlying PHP bug, but we're going to try to find a workaround.)

If you haven't been able to use the space you bought, just send a note to support@zot....org once this is fixed and we'll extend your subscription.

e14

Dan,
thanks for your quick response. I think a lot of research groups are/will continue to jump onto this and the group sharing feature. Given that this problem exists, you might want to put a warning about this on the information about group sharing, and on the page selling storage space. I realise you probably don't want to slow down the rapid rise of Zotero, but until you get this sorted out you may end up with a lot of annoyed people.
And of course pull out all stops to get the problem sorted, as word goes around about the ability to share and synchronise within research groups, this will positively explode! tsobey a t mytum . de :-)

termatzor

Though it's really not new, I get the same error even with a tiny database of just ~400 entries.

"Databases of this size cannot yet be synced. Please check back soon."
Report ID: 1467579231.

Apart from some minor bugs this is really a wonderful tool. Congratulations.

dstillman

The error message is a bit misleading—it can also happen if you have a lot of tags, authors, etc. But we're working on this right now and should have a fix very soon.

erazlogo

Is it now possible to sync a collection of this size: 33416 items; zotero.sqlite=250.1 MB? My storage folder by the way is 10.71 GB, above your maximum storage offering. :)

I would love to experiment with getting this up to the server, not for backup but because it is difficult to set up groups for courses without syncing my own database.

Melani99

I'm hoping that the "soon" for this fix is now "very soon" because my difficulty in carrying around zotero on my thumb drive and saving it every time, etc. is really driving me crazy. Any idea when the "everything and the kitchen sync" will be up and running? Thanks.