import very large bibtex

rdmediation
I've a 562000 bibtex file (250Mb) to import but the conversion of bibtex to sqlite zotero format is extremely long. Is there a script (python) to build the database directly?
  • adamsmith
    No, there's no alternative to import. Do you mean 562000 items or lines? I'm a little worried if this will work reasonably in Zotero at all. I'd also recommend splitting the file into chunks to prevent Zotero from running out of memory during import.
  • rdmediation
    items.I tried bib2ML, python libzotero but it doesn't work. I check the sqlite structure to see if it is possible to replace zotero client for import. 240Mb sqlite is not really big
  • adamsmith
    Importing 562000 items into Zotero won't go well. You might get them imported, but it will slow Zotero to a crawl -- if you want to work with a database of half a million items, Zotero is not going to be your tool of choice, sorry, I wouldn't waste my time on this (and the database would be much larger than the bibtex file, at least several gigabytes in size.)
  • rdmediation
    sure. I'll try another way sorry
  • galicarnax
    edited December 9, 2018
    I've tried to import a much smaller bibtex file, with only ~5000 items, and it took me more than an hour (about 1-2 items per second! ). It's clearly not a limit of SQLite, the bottleneck is on Zotero's side. What's impossible with fixing this in Zotero?
  • bwiernik
    That doesn’t seem normal—import should typically go much faster. What are the details of your setup (hardware, type of hard drive, operating system)? Do you have your Zotero folder somewhere unusual (like an external or network drive)?

    @dstillman Can we split this off?
  • galicarnax
    edited December 9, 2018
    ASUS G75VW notebook, Windows 10 Home, solid-state drive. I'm importing a local bibtex file. I also tested import from Mendeley sqlite-file, it works faster, but not that much (~5-6 items per second, yielding ~ 15 minutes total for ~5000 Mendeley items).
    UPD: If that helps, I tried also with and without Better Bibtex plugin enabled - this does not seem to make difference.
  • emilianoeheyns
    I wouldn't expect there to be a significant difference between BBT and stock zotero; most of the parsing work is up-front, after which the individual saves are going to be pretty much the same pace as any other importer. If you have a copy of those 5k items, I can have a look where most of the time is spent.

This is an old discussion that has not been active in a long time. Before commenting here, you should strongly consider starting a new discussion instead. If you think the content of this discussion is still relevant, you can link to it from your new discussion.

Sign In or Register to comment.