[solved] Record number limit in libraryRDF export?

mike2712
I am using Zotero 2.1.10 on Win 7.

I have an unusual problem with exporting my bibliographic library. It contains 13421 records, and export of records with notes and attachments fails (without error messages; the green progress bar in the "Exporting files..." box reaches the end, then will stay there for hours, until I close Firefox). The attachments are exported, but the .rdf file is zero length. Cutting the library into two halves then exporting each half works fine (attachments and the .rdf are OK). After a long process of trying different sets of records I find that exporting 8658 records works without problem, but exporting 8659 fails. The extra record in the 8659 set exports OK in other sets of records.

Has anyone else experienced this kind of problem? If so, please give me your advice! Is there a limit (i.e. 8658) on the number of records that can be exported in Zotero RDF format?
  • dstillman
    Check the Firefox Error Console (Tools->Web Developer->Error Console) for an error when it fails. Clear the console first, attempt the export, and see if anything shows up. If something does, click on it, press Ctrl-C, and paste it here.
  • mike2712
    OK, I get the following from the Error Console. I suppose it means that 4 Gb memory is not sufficient (I'm on 32-bit Windows):

    Error: uncaught exception: [Exception... "Component returned failure code: 0x8007000e (NS_ERROR_OUT_OF_MEMORY) [nsIConverterOutputStream.writeString]" nsresult: "0x8007000e (NS_ERROR_OUT_OF_MEMORY)" location: "JS frame :: chrome://zotero/content/xpcom/translation/browser_firefox.js :: <TOP_LEVEL> :: line 546" data: no]
  • adamsmith
    Yes, that's probably it. You can try again when Zotero 3.0 is out in a couple of days, it has all types of performance improvements. That said.

    1. I don't see any harm in exporting your library collection by collection.
    2. Why do you want to export your entire library to RDF? I can't think of a use case where there isn't a better way.
  • ajlyon
    The way we do RDF export, we build the entire datastore in memory, then write it out to an XML representation. There might be ways to improve performance for very large exports, but it won't be easy-- we could produce the XML record-by-record, but we currently need the full RDF graph to do things like item relations. I suppose we could make an option to ignore item relations when exporting, which would let us move much more quickly and write records one-by-one. Still, not sure if that's worth it-- first we'd need to know what the use case is.
  • mike2712
    Thank you both for your replies. I take the point that it can be done collection by collection, and will be happy with that.

    I was trying the entire library because the option is available, and became puzzled when it would no longer work (it used to work, when the library had fewer entries). The logic was that I think there are attachments remaining in the storage folder after deletion from the library itself, thus taking up disk space - not much at the present, but possibly increasing with time. Thus exporting would only save the database plus attachments that are still present, and would be the easy way to get rid of the unwanted ones by creating a new storage folder after import. Is this logic correct?

    Please let me know if the idea is wrong. Otherwise, I will go for the export in several parts that require less memory, then import them into a new Library to recreate the present one, with the correct number of attachments in storage.

    Finally, I am excited to hear that a new version of Zotero will be out soon. It is a superb program!
  • adamsmith
    Attachments shouldn't stick around when an item is deleted. Certainly not after you have deleted your Zotero trash.

    Exporting and re-importing your library to clean it up is not recommended. There may be marginal data-loss, but more importantly it breaks item IDs, which will break connections to Word documents and will cause Havoc if/when you use the sync feature. I would strongly recommend you stop doing that.
  • mike2712
    Your point is well taken. Thank you so much for your advice. Do I need to mark this thread as "Solved", if so, how do I do it?
  • adamsmith
    no need to mark threads solved, but if you want to you can edit your first post and put [solved] at the beginning of the thread title.
Sign In or Register to comment.