Zotero bibtex export - invalid records

Bionatsci

When I export my library to bibtex with notes (this is the first time I have done this) and then open the .bib in Jabref this fails with the message "Error in line 436: Expected = but received @". I tracked this down to some invalid "empty" records in the .bib file of the format


@misc{__????
},

@misc{__????-1
},


all the way up to


@misc{__????-13
},


The error refers to the line immediately after the invalid record. The first five invalid records are about 2000-3000 lines apart (didn't check the rest). When the invalid records are removed by hand this error goes away but is replaced by another one (probably due to my incompetent hand editing) "Error in line 24943: EOF in mid-string".

System details:

OSX 10.6.8
Firefox 15.0.1
Zotero 3.0.8
Jabref 2.8.1

I also have the word for mac integration, ZoteroQuickLook and ZotFile installed (all latest versions according to firefox addons manager). I did install autozotbib, but uninstalled it to troubleshoot (no change). I have already reset my zotero translators to no effect.

Thanks in advance for any help you can provide. Please let me know if you need me to carry out any further diagnostic/troubleshooting steps. While I would prefer not to make my database or .bib file entirely public I am perfectly willing to email it to developers (core or otherwise) if that would be helpful.

P.S. I tried exporting a couple of individual collections, these seemed to work fine.

adamsmith

could you track down which references come out as invalid?
Export these as Zotero RDF and either post the RDF to a public gist at gist.github.com or send it to me via e-mail - you can find my e-mail at the bottom of this blogpost: http://www.zotero.org/blog/zotero-workshops/

Bionatsci

Okay, will do - do you know how the bibtex translator decides on the order of records by any chance (I can't see any obvious pattern when I eyeball the output)?

If I can sort my zotero library the same way as the .bib I can probably do this today, if not it might be a few days as I have to pin down 13 items in an approx. 2000 item library (I'll try and find a field to sort on that clusters them together so I can at least halve my libary but as per Murphy there probably won't be one).


Thanks!

Bionatsci

Never mind - should have played around a little more, at least for partial exports it exports either by title, or whatever field is sorted on (not sure which, wanted to reply quickly so as not to put you to the trouble of answering my question above).

Bionatsci

Got it (at least partially). The invalid records seem to be produced for any standalone note. Exporting only my standalone notes (of which I have 14) gives only empty records of the format given above, excluding the standalone notes resolves the "expected...received" error, but leaves me with the "EOF in mid-string" error, exactly as when I manually removed the invalid records. Adding a new standalone note adds a ????-14 record to the end of the export.

Should I try and replicate this in a fresh firefox profile?

Would you also like me to see if I can isolate the remaining EOF error to a particular subset of items? (for troubleshooting purposes I will do this on a separate copy of my zotero library with the standalone notes deleted). I presume you don't need an RDF export of my standalone notes. They are probably the most sensitive bit of my library but if you need it I can give you a sanitised version assuming it still replicates the error.

adamsmith

great - I'll see what makes sense as a way of handling standalone notes. Probably just skipping them? Or do you have a suggestion of how they should be exported into bibtex?
I haven't tested, but I'd assume that's a general issue, so no need for the notes, yes.

For the EOF error, yes, please do try to isolate that.
Also, look at and around the line that the error points to, that maybe helpful? Unfortunately I don't know enough about jabref to tell you exactly what to look for - an illegal character or a missing bracket would be my best guesses.

noksagt

I would say to export stand-alone notes to @COMMENT{...} if "export notes" is selected & to ignore them if it is not.

Bionatsci

I'll see what makes sense as a way of handling standalone notes. Probably just skipping them?

I'm a newcomer to bibtex, so I can't offer any better suggestions I'm afraid - skipping them would be perfectly fine in my case, I don't know if they really fit into the bibtex data model anyway.
Missed Noksagt's post. @COMMENT sounds good if found not to cause any other issues.

For the EOF error, yes, please do try to isolate that.

Will do

Also, look at and around the line that the error points to, that maybe helpful?

It's the last line, probably should have mentioned that before.

an illegal character or a missing bracket would be my best guesses.

I guessed it might be a missing bracket, I hadn't thought of illegal characters though - I'll keep an eye out for strange characters as well.

Thanks.

Bionatsci

Got it! (for a repetitive task homing in on a problem item is surprisingly satisfying). Problem item:

https://gist.github.com/3857904

Without this item and the standalone notes Jabref import works fine.

It was an unmatched brace in an attachement filename. If you look at the .bib the highwire translator imported some markup (looks non-standard to me) leaving γδ as {gamma}{delta}. The auto-renamed file was then cut off just after the first brace in {delta}. Removing the unmatched braces by hand from the .bib resolves the problem. As a more permanent solution I'll obviously rename the attachment.

This is such a corner case (braces in a title and filename truncation happening to leave an unmatched opening brace) that it's probably not worth fixing, but I guess the most elegant solution would be to have the filename truncation code check for unmatched braces and either remove them or shift the truncation to the left of them.

As a workaround for the standalone notes issue (I'm too excited about using Docear to wait for a fix) I'm going to attach them to some dummy items for now.

Thanks for your help.

adamsmith

thanks for tracking that down - yes, the solution is exactly what you say, I thought we already did that, but probably only for other cases - obviously those braces shouldn't be in a Zotero field.

r-s-k

Had the same problem with a title containing braces. The truncated short title and filename were then missing closing braces, corrupting the Bibtex export.

Apparently, Lyx/Latex/Bibtex could still handle the unbalanced braces in the short title (old bib exports already had that error), but not in the "file = " entry introduced in Zotero 3.0.x.

So this should definitely be checked, preferably already while grabbing/importing.