Importing Large Databases

jbonito
Hi,

I am new to Zotero but like what I see. The problem for me so far is that I have a very large database in refworks (over 7000 citations) and have been unable so far to import them into Zotero. I exported from refworks in BibTex format as instructed, but Zotero hangs each time I try to import. I've also tried to break the database into smaller files but it still hangs. Jabref imports the very same large file in roughly 30 seconds so I don't think it's a problem with the file. Has anyone encountered a limit on the number of references that might be imported and does anyone have a workaround?

By the way, I'm working on a Mac running leopard.

Thanks in advance.

Joe
  • jbonito
    I did some searching before and after posting, and found that it might be an import issue with BibTex, so I reset the "translators and styles" but still no joy. Am now exporting via RIS from Refworks--I'll see where that leads.
  • jbonito
    Well...

    I've had some success carving up the file in Smultron (a text editor) into 10,000 line files (100,000 lines total). It has imported the first three parts. I've thrown caution to the wind and put the last 70000 lines in one file to see if that works.

    And I upgraded to 1.5. Not great troubleshooting as I've changed two variables (or maybe three, as Smultron's method of saving might differ from Safari or Firefox), but I see a light at the end of the tunnel.

    Joe
  • jbonito
    No light. It wouldn't import subsequent files with 10,000 lines as it did with the first 3. I'm stuck...
  • Tjowens
    Generally the problem is not the length of the file but a few problem records which trip up the import and cause it to fail. If I follow you correctly, you broke the file into 10 chunks, the first three worked but the fourth one failed. Can you try the remaining six files? There is a good chance that they will slide right in, at which point you can break up the last file to isolate the problem record.
  • jjroper
    I am trying to import from Papyrus, via RIS format. The default settings for the import did not work, so I modified it a bit to make it look more compatible, and also did not work. When it "sort of" worked, not all the parts came across right, like the page numbering was messed up, or the identifier for the line was actually included in the data of the import, and other funny things. At any rate, I can't get my Papyrus imported. Has anybody actually had luck doing this, and if so, could you provide some pointers?

    Thanks!

    Jim
  • philgons
    I'm having the same difficulty importing a large database (12,000+) to a second and third machine. I have my entire Zotero database on my primary machine, but it took some work and patience to get it all imported properly. I did it in smaller chunks, and eventually it took. I was even able to sync it, because it synced bit by bit rather than all at once. The trouble was trying to get it onto my other two machines. I couldn't get them to sync, and importing wouldn't work either. I tried resetting the server, but all that did was made it so sync now won't work on any of my machines.

    I understand the difficulties with trying to sync a large database, but I'm really curious why I can't import (i.e., when uploading speeds and busy servers aren't issues). Is it the size, or is it a problem with one or more entries? I tend to doubt the latter, because I did get them imported on my primary machine.

    I'd love to find out if there's a way to import large databases.
  • octavabasso
    I have had trouble with this as well, even though my library is not nearly as large (around 3000). I would ask the developers to find a way to import all entries that are not error prone, and then Zotero could inform the user "There were X errors importing your library" and show which entries didn't make it. As of now, I have maybe 400 entries that just won't import. I've been breaking them up into seperate files, but this takes a lot of time and tedium - and considering Endnote is faster than Zotero and does not seem to deal with errors like this, this is a huge impediment for new users who might switch to Z.
  • birkland
    Same problem here. I have a ~4800 item Endnote file, exported to RIS, attempt to import and it hangs right at the get-go. Might have something to do with Firefox not being the stable browser that all the Microsoft haters think. Will try to do the same with BibTex, although I am not holding my breath. Stay tuned.
  • birkland
    And BibTex is no good either. I really want to transition to Zotero but I don't see the benefit given (1) its failure to work properly with track changes in Word, and (2) its inability to give me anything like a proper transition path from endnote to zotero. Not ready for prime time, it seems.
  • dstillman
    birkland: The progress bar will hang on large imports (due to technical reasons outside our control and that have absolutely nothing to do with the stability of Firefox), but that doesn't necessarily mean it's not working. But, as Trevor says above, you can break up the RIS or BibTeX file into multiple sections and Zotero will likely import each quickly.
  • birkland
    Fair enough. I will be trying this.
  • birkland
    OK, I found a solution--a bit kludgy, but not unmanageable since I only have 5500 records:

    1. Set up custom groups in Endnote titled 1-500, 501-1000, 1001-1500, etc. This keeps each file to a relatively small size.

    2 Search for records in Endnote in that range, and then add each set to the corresponding group.

    3. Export one group at a time.

    4. Import each group separately into Zotero.

    It took about 30 minutes to do this, and I didn't have to download and learn a new file splitter.
Sign In or Register to comment.