Importing large collections

tshort

I'm trying to import a collection from endnote with over 30,000 entries. I've exported this from endnote to a "monster.ris" file. When I tried to import this into zotero, I notice the following:

* The progress bar in the import dialog froze.
* Firefox locked up.
* Firefox's CPU time went to 90+ percent.
* I eventually killed it.
* The references were coming in, but very slowly.
* In about 90 minutes, it had read in 1300 citations.

So, has anyone else tried this? Is there another format besides RIS that would be faster for import? Any other ideas?

tshort

I forgot to mention, this is in beta 4 (b4.r2.r1365).

dstillman

I'm not sure what the relative speeds of the various import formats are, but unfortunately 30,000 items is well beyond the library size that Zotero has been tested with. We did some massive performance optimizations for Beta 4 that made it possible (though occasionally sluggish) to work with libraries with 5K-10K items on a fast computer, but there are still a lot of parts that will need to be optimized for larger collections.

In the meantime I'd probably suggest importing just a subset of the EndNote library, if possible. You could also just let it run overnight on the fastest computer you have access to, but I suspect that trying to use the resulting Zotero library would be pretty frustrating.

It's definitely our goal to eventually support very large collections, but it'll take some more work until that's really feasible.

dcroxton

I've found it to be sluggish just moving items around between collections. I was trying to move 463 items from the library into a collection, and it just wasn't happening -- nothing was moving. Smaller amount (e.g., 10) would copy over fine. Maybe it would have worked had I let it sit for a long time, but patience is not my strong suit. The thing is, all it needed to do was add a record in the collectionItems table for each item, so I couldn't understand what was taking so long. Eventually I went into the database and moved them over using sql (naughty, I know). Is there something else going on besides just adding and removing records from the collectionItems table when adding items to, or removing items from, a given collection? The sql command I wrote executed almost instantaneously.

Sincerely,
Derek

dstillman

Derek: the actual process is a bit more involved than a single SQL statement, since, among other things, it involves determining and setting an orderIndex for the item, which will (probably) be used in future versions to allow user-defined orders in collections. (I imagine the items you added don't have orderIndexes set, unless you ran multiple SQL statements in a script.)

The delay, however, was unnecessary, and was caused by the UI notification methods being called after every insert. I grouped the adds and deletes into single transactions in r1369 on the development branch, and adds and deletes are now more or less instantaneous.

Thanks for bringing this to our attention.

twaring

I'd like to import an Endnote X (mac) library that contains linked pdfs (stored internally by Endnote). Is there a means of doing this automatically? It seems not from the lack of any mention of it in the documentation, but thought I'd ask.

Also, how many items do your Zotero development libraries usually have? This helps me reign in my commitment to Zotero until it's outta beta.

Best,

Tim

thomask

hi dan,

i'm having a similar problem to tshort, but with a twist of lime.

using ffox 2.0.0.4 on an 800MHz powerbook G4 mac with 512Mb RAM, zotero 1.0.0b4.r5

when i imported my endote v8 library (<3000 refs) i kept getting an "unresponsive script" error, which if i hit continue a couple of times eventually chugged away wit the rainbow wheel. i left the computer alone for a couple of hours (ffox was using 98% of CPU) and when i returned it seemed to have imported everything.

now however whenever i try to open zotero from the bottom bar of ffox the library won't open, instead i get the "unresponsive script" error.

also, if i try to import even a small (3) number of references from Pubmed i get the same error message. if i select the refs one by one it seems to work, albeit slowly.

is this a prob with my ancient G4, or is it a zotero thing? sorry about the non-technical jargon i'm only partially tech savvy :(

dstillman

thomask: We disable that message most places that it might occur, but if you have a particularly slow computer, it's possible you'd see it more often.

There's an easy fix, though: go to about:config in the URL bar, search for dom.max_chrome_script_run_time, double-click on that pref, and either increase the number of seconds or make it 0 to disable the warning completely. The default is 20.

Note that imports will be dramatically faster in the next release.