BibTex Export

Hello,

I've been having trouble lately with the BibTex export. I've noticed it in particular since the use of html notes in 1.5. Basically, the export translator exports the notes and things as "comment" lines in the bibtex file. But I've found that if you use notes for much of anything, this causes a problem. I was having issues with documents that wouldn't compile due to unidentified control sequences... and in other cases, the bibliography produced in latex included things like the abstracts of articles.

I took a look at the bibtex.js file in zotero's translators folder. I noticed that the end of the file included the ability to export the notes as well as the "extra" field as a note. The extra field is not inherently problematic, but I've had trouble with it because the JSTOR translator puts the abstract in the extra field.

Anyway... so I just commented out that stuff... the end of my file now looks like this...

if(item.date) {
var date = Zotero.Utilities.strToDate(item.date);
// need to use non-localized abbreviation
if(date.month) {
writeField("month", months[date.month], true);
}
if(date.year) {
writeField("year", date.year);
}
}

//if(item.extra) {
// writeField("note", item.extra);
//}

if(item.tags && item.tags.length) {
var tagString = "";
for each(var tag in item.tags) {
tagString += ","+tag.tag;
}
writeField("keywords", tagString.substr(1));
}

if(item.pages) {
writeField("pages", item.pages.replace("-","--"));
}

if(item.itemType == "webpage") {
writeField("howpublished", item.url);
}
//if (item.notes) {
// for each (var note in item.notes) {
// writeField("comment", note["note"]);
// }
//}
Zotero.write("\n}");
}
}


Note the comments on the last "if" block - this removes the notes of course, and up above I commented out the if block for the extra field.

I just thought I'd post here... if you're having the kinds of issues I did, it's a simple and easy workaround.

As a more general note, I think it's pretty rare that I would ever want my notes (or even the abstracts for that matter) coming out in the bibtex file. Personally, I always use zotero as the manager, and just export the bibtex file purely for writing a paper. If I want to add a citation, I do it in zotero and re-export. I would guess I'm not alone in this. So for me then, I would much rather have (maybe a second) bibtex option in the output list that's just "BibTex (Simplified)" or something that basically does what I did above here... not export those big notes and extras... you could even drop the keywords... basically every field that won't ever be in a citation.

Mike
  • edited November 15, 2008
    I was having issues with documents that wouldn't compile due to unidentified control sequences...
    Did you determine the specific cause for this issue?
    and in other cases, the bibliography produced in latex included things like the abstracts of articles.
    But this is because some references had the abstract in the extra field, right?
    I noticed that the end of the file included the ability to export the notes as well as the "extra" field as a note.
    I would consider this correct behavior--BibTeX notes should be retained on import/export.
    The extra field is not inherently problematic, but I've had trouble with it because the JSTOR translator puts the abstract in the extra field.
    Then the JSTOR translator is what should be fixed!
    As a more general note, I think it's pretty rare that I would ever want my notes (or even the abstracts for that matter) coming out in the bibtex file.
    I disagree. Export should be as rich as possible. That being said, export should work. It sounds like the only thing we need to do to make it work might be to change the way that notes are exported as comments. There would seem to be very few downsides for including more information than was necessary to generate a bibliography, assuming that bibliography generation is successful.

    Further: there are plenty of ways to strip fields from a BibTeX file after you have one.
  • Hello,

    You're exactly right that one big issue was in the JSTOR translator... but it's beyond that.

    I was having general issues with latex compiling because of what I assume are the notes. The reason I figured this was the cause was that I would compile it, get an error, make the change and then not get errors. This seemed pretty definitive to me, as that was the only element changed.

    I also agree that it's good to have a rich export, but I would argue that most using latex for their writing have no interest generally in a rich export most of the time. The analogy I would make is that when you are using the Word plugin, you don't care about most of the information in zotero, you just want to find the citation, put it in your document and move on. If you want to actually work with the citations, you'll probably go to Zotero.

    LaTex is the same way... I am not working with the citations in the bibtex file... and I certainly have no interest in editing the file after I export it from Zotero, as you suggest.

    I posted this here because - at least for me, and I'd assume others - the zotero export contains far more than it needs... and this just tends to cause problems in LaTex. Why deal with it?

    This is why I would argue finally that it is best to either give options to the user about how rich that export should be, or provide a simplified export option.

    Mike
  • I was having general issues with latex compiling because of what I assume are the notes. The reason I figured this was the cause was that I would compile it, get an error, make the change and then not get errors. This seemed pretty definitive to me, as that was the only element changed.
    But why not fix these issues, rather than working around them? Can you provide a minimum failing example?
    I would argue that most using latex for their writing have no interest generally in a rich export most of the time.
    This is different from arguing that they want a simple export. If the rich export worked, what would the downsides be? I can think of no reference manager that limits BibTeX export, including BibTeX-specific managers (such as JabRef).
    and I certainly have no interest in editing the file after I export it from Zotero, as you suggest.
    If your .bib file worked, would you still edit it? If this is a corner-case, it would seem to be reasonable to have to use "BibTool" or something similar to remove fields.
    This is why I would argue finally that it is best to either give options to the user about how rich that export should be, or provide a simplified export option.
    If I'm missing some advantage of a more limited export when the full export works, please let me know. However, I would consider this to be adding needless complexity. I doubt that many people would use a limited export if a full export worked & I think there would be disagreement on which fields to include/exclude (e.g. you excluded Zotero's 'extra' (BibTeX's 'note'), but many BibTeX users would consider field essential).

    If you are dead-set on stripping fields & are against using bibtool or similar, I would probably suggest downloading & modifying the bibtex.csl style file & using that (rather than "export").
  • Hello,

    Sorry for the delay here... I think I tracked down the issue somewhat... Where I actually ran into trouble was on the girlfriend's computer. I am not sure how this was happening, but it looks like the exported file had some weird line breaks in it that I ultimately concluded couldn't be coming from the Zotero code. Those line breaks were I think what was causing the document to not compile in latex.

    I am not positive how those line breaks were getting in there, but I think it is likely that they are actually in the text of some of her fields. If that's right, the translator might want to be updated to remove or recode such breaks. I don't know much about how that would happen or what impact it would have... and I might be mistaken about the line breaks anyway.

    To get back to the main argument... you make a good point. If the thing works, why not have all the data in the exported file? As I thought about it, I guess there really isn't any great reason. The only real thing I would say is that imagine I store all my notes in Zotero... Which I think increasingly makes sense now that they have HTML support... well I could be looking at at least a page of html notes for hundreds of articles. All that data would then go into the exported bib file every time.

    This struck me as annoying. First, it's a (albeit small) waste of space on my computer - why make that file bigger than it needs to be. Second, and more importantly, when I am in WinEdt (or whatever editor) and I have to glance through the bib file for some reason - maybe to find a key -then I am scrolling through tons of lines of weirdly coded notes just to find references.

    My only point this whole time was that I think the utility of the export changes based on how you think of it... If you think of the export as a way of transferring data around, then absolutely it should have everything possible... but I doubt most latex users want, or think of it this way... I have never thus far wanted that ever... I would much rather have a small, simple, file that has what is needed for bibtex to produce the citations and nothing else to get in the way... What I want isn't so much an export, but rather just better, simpler support for latex.

    After giving this some thought, I decided that maybe it's really not a Zotero problem. Actually, I've been looking into how a WinEdt plugin could be put together to generate a simple bib file in the same directory as the document .tex file... Such a file would certainly not need the notes. That kind of plugin would work then in a convenient way not unlike the word or oo plugins. As I mentioned though... that's not so much a Zotero issue... it's a WinEdt feature. Still, I think it's analogous to the Word plugin. Right now, we don't have something like that for LaTex, so you end up with these unnecessarily detailed files that are ultimately more of a pain than what you need.

    Mike
  • Those line breaks were I think what was causing the document to not compile in latex.
    LaTeX/BibTeX should be tolerant of most whitespace variations. However, if you narrow down the problem, please let us know.
    I doubt most latex users want, or think of it this way...
    Keep in mind that you're talking with a LaTeX user & a TUG member! From those that I talk to, they want the best of both worlds--a full database of information & programs that make those files easy to work with. I don't know if I'd say their opinions are held by "most LaTeXnicians," but neither do I think that you are able to speak for most LaTeX users any better.

    Some editors have support for folding, which removes your concern about scrolling to much. Many also have excellent search facilities.
    Actually, I've been looking into how a WinEdt plugin could be put together to generate a simple bib file in the same directory as the document .tex file... Such a file would certainly not need the notes. That kind of plugin would work then in a convenient way not unlike the word or oo plugins.
    It isn't clear to me what the proposed plugin would do, but have you looked at the extension that provides lyxpipe integration? Unfortunately, WinEdt uses a different method to pipe references in. You can, of course, import your BibTeX into JabRef & use that.
  • Well, I can only emphasize that an export selector would help. I have some really long notes at some entries and by exporting them directly into the bib-file BibTeX encounters errors due to memory overrun. "Your field is more than 3000 characters" is what I am told.

    I could dig the error down to too long lines in the bib-file coming from too long notes.

    So if one could just disable the note/ annote export it would help a bunch.
  • edited August 6, 2009
    Which version of BibTeX are you using? I thought you had claimed in another thread to be using bibtex8, which should have a larger character limit.

    Note that you can also increase the size of this limit (at least at compile time) in bibtex as well.

    Other fields than 'note' can exceed this limit.

    I think 'extra' should always be exported, despite the fact that some people happen to have long 'extra' fields.

    But other translators do give the option as to whether or not to export attached notes & so it might be nice to make that optional for this too. If they are exported, should they be exported:
    • as-is
    • trimmed to 3000 characters
    • exported as additional LaTeX files that are called by the BibTeX file
    ? And should we start to export abstracts assuming we have a good solution for notes?
  • edited August 6, 2009
    Yes, I am the one that uses bibtex8 :)
    Version: bibtex8 --version
    This is 8-bit Big BibTeX version 0.99c
    Implementation: C for Unix
    Release version: 3.71 (31 May 2005)

    I call the bibtex run with bibtex8 --wolfgang, so it has already a very huge capacity. Still I get this error: "Your field is more than 3000 characters".

    If you would ask me I would say that the user should be able to decide: Either the notes (and similar fields) are not exported at all -- or they are exported as they are (not trimmed or excluded). The reason is simple: In the first case the user wants to use the bib file in his LaTeX document. He doesn't need the notes there. In the second case he wants the bib file to be a backup and so he needs all information from the database in there.

    I just discovered another issue today (don't know if I should open another thread for this -- feel free to split it into another post): The bibtex filter exports two fields: address and location. As it is exported the address relates to the publisher, the location relates to the actual repository the book can be found. Unfortunately, biblatex uses location with the publisher. It outputs: "location: publisher, year". So at the moment I rewrote the output in the biblatex style file into "address: publisher, year". But I guess the ordinary user won't do that.
    So is this just a mistake or was it done by reason?

    Cheers,
    Mathias
  • I call the bibtex run with bibtex8 --wolfgang, so it has already a very huge capacity. Still I get this error: "Your field is more than 3000 characters".
    It seems that --wolfgang doesn't set BUF_SIZE. I hadn't realized that. You must set this in bibtex.h at compile time, unfortunately.
    As it is exported the address relates to the publisher, the location relates to the actual repository the book can be found. Unfortunately, biblatex uses location with the publisher. It outputs: "location: publisher, year". So at the moment I rewrote the output in the biblatex style file into "address: publisher, year". But I guess the ordinary user won't do that.
    So is this just a mistake or was it done by reason?
    I don't think I'm responsible for deciding how to handle the export of those fields, but I will make this observation: 'address' is a standard BibTeX field that Oren Patashnik enumerated. Zotero is correctly putting the publisher's place into this field, I think. BibLaTeX is aware of this standard field & the docs claim that it will use 'address' if 'location' is absent. I am not sure why the BibLaTeX developers decided to use 'location' in preference to the established standard.

    'location' is not a standard field. It is used by other programs for quite a few different things (the most common being where a conference took place, BibLaTeX's 'venue'). I am quite surprised to see that BibLaTeX prefers it, when there are so many .bib files and other programs that do not agree on the meaning of the field. Arguably, Zotero's archiveLocation should not use this field due to the ambiguity. It isn't clear to me where archiveLocation should be exported instead (if anywhere).
  • Thanks for the insights. Under those circumstances it has to be kept as it is I guess. If biblatex runs abnormal Zotero need not to be changed. At least as long as biblatex is beta and not standard but bibtex is.
    I'll just keep my adjusted style files, this should do the trick in the next time.

    Regarding bibtex8 and buffer size: Well, it might be that it's possible to increase the buffer at compilation. But to be honest: Not more than 5% of the users will ever even consider to do this. But all need a working solution. So I think the way to go is not changing bibtex8 but the bib file.
  • Zotero is a great tool and I have been using it with BiBTeX export for some time. Several years ago, there were a few things that needed to be subsequently editted in the bib file, capitalization of letters and stuff. Having not written for a while, and after updating zotero, I now find that I have problems exporting many of my references.

    I think one of the problems comes from the 'doi' field. For example, export from Zotero gives:

    @article{aihong_indian_2005,
    title = {Indian Ocean variability and its association with {ENSO} in a global coupled model},
    volume = {18},
    issn = {08948755},
    url = {http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=18512094&site=ehost-live},
    doi = {Article},
    abstract = {The evolution of the Indian Ocean during El {Niño–Southern} Oscillation is investigated in a 100-yr integration of an Australian Bureau of Meteorology coupled seasonal forecast model. During El Niño, easterly anomalies are induced across the eastern equatorial Indian Ocean. These act to suppress the equatorial thermocline to the west and elevate it to the east and initially cool (warm) the sea surface temperature {(SST)} in the east (west). Subsequently, the entire Indian Ocean basin warms, mainly in response to the reduced latent heat flux and enhanced shortwave radiation that is associated with suppressed rainfall. This evolution can be partially explained by the excitation of an intrinsic coupled mode that involves a feedback between anomalous equatorial easterlies and zonal gradients in {SST} and rainfall. This positive feedback develops in the boreal summer and autumn seasons when the mean thermocline is shallow in the eastern equatorial Indian Ocean in response to trade southeasterlies. This positive feedback diminishes once the climatological surface winds become westerly at the onset of the Australian summer monsoon. {ENSO} is the leading mechanism that excites this coupled mode, but not all {ENSO} events are efficient at exciting it. During the typical El Niño {(La} Niña) event, easterly (westerly) anomalies are not induced until after boreal autumn, which is too late in the annual cycle to instigate strong dynamical coupling. Only those {ENSO} events that develop early (i.e., before boreal summer) instigate a strong coupled response in the Indian Ocean. The coupled mode can also be initiated in early boreal summer by an equatorward shift of the subtropical ridge in the southern Indian Ocean, which stems from uncoupled extratropical variability. {[ABSTRACT} {FROM} {AUTHOR]}},
    number = {17},
    journal = {Journal of Climate},
    author = {Z. Aihong and H. H. Hendon and O. Alves},
    year = {2005},
    keywords = {{EL} Nino Current, {INDIAN} Ocean, {METEOROLOGY,} {OCEAN-atmosphere} interaction, {SOUTHERN} oscillation},
    pages = {3634--3649},
    annote = {Accession Number: 18512094; Aihong Zhong 1 Hendon, Harry H. 1; Email Address: h.hendon@bom.gov.au Alves, Oscar 1; Affiliation: 1: Bureau of Meteorology Research Centre, Melbourne, Australia; Source Info: Sep2005, Vol. 18 Issue 17, p3634; Subject Term: {SOUTHERN} oscillation; Subject Term: {OCEAN-atmosphere} interaction; Subject Term: {METEOROLOGY;} Subject Term: {EL} Nino Current; Subject Term: {INDIAN} Ocean; Subject Term: {AUSTRALIA;} Number of Pages: 16p; Illustrations: 7 graphs, 8 maps; Document Type: Article},
    annote = {Copyright of Journal of Climate is the property of American Meteorological Society and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. {(Copyright} applies to all Abstracts)}
    },

    When compiled using MikTeX pdflatex, or read in BibTeXMng, none of the fields after the 'doi' field are used. If I manually delete the 'doi' field, the reference is read correctly.

    I too would like to have a "simple bib file" export, if possible where I could select the fields that I would like exported to my .bib file.

    Cheers, Fiona.
  • Hm, yeah, I used scaffold to add comments to bibtex.js and now get what I want.

    Thanks, mryckman.
  • I totally agree with fiona, we could have a simple list with some check boxes so we can simply choose to export (or not) some of the fields.

    I have some article classes that insert the "extra" fields in the end of each respective bib entry, and that's "garbage" that I can't have in the final document, but that I need to have in Zotero to have some extra feedback. I usually use the "extra" field to insert things like the document conclusions (maybe I should put it in a note, but for now it's what I do).

    I think this is a simple task to accomplish, as the checkboxes could be simply mapped in the Bibtex.js script to be some IF clauses.
  • The other reason for a more customizable bibtex export is that some BibTeX-y systems assume that any information which is optional in the bibliography style but included in your BibTeX file is information you want to use in your citation or bibliography. I use 'biblatex-chicago-notes' and it will include a URL if it's in your bibtex file, which I don't want for traditional ink-on-paper articles. Also, in Zotero I keep as much information as possible, but some of this (series information, for example) is optional in the actual bibliography, and I choose not to include it. For now I use bibtool after each Zotero export to prune fields. I'd be glad to do this inside Zotero.

    And I'd be glad if Zotero's "don't give the URL for articles whose data includes a page range" could be used in the BibTeX export, since my current method of using bibtool for pruning all URL fields is a bit of a blunt instrument.
  • I think I am having similar problems to # fiona.h.evans, although I also found some of the problems in compiling were due to percentages and other special characters in the 'abstract' field.

    Another, related problem is that there are some special characters which I want to display: authors with accents in their names for instance. Is there a way of getting zotero to translate the library properly into tex? For example, the name Hummelshøj exports into my .bib file as Hummelshøj, which once compiled is then displayed in the bibliography as Hummelshj - it would be useful if I could export it directly as Hummelsh\{o}j.

    I know I can change the .bib file using a text editor, but I have rather a lot of references, with a huge amount of accents and other weird characters in the author lists and titles, so I have a feeling this is going to take rather a long time.
  • Change your export character encoding to ISO-8859-1 & the bibtex will have the transliterated characters you want, rather than the UTF-8 entities that you now have.
  • +1 to scot! Please find a way to omit urls from 'printed paper' items. Some (most?) Journal (La)Tex styles add the URL if it is there, even though it should not be displayed. I am not sure whether the existence of a page range is good enough as the only criterion; books should be cited without an url. Urls should be exported for web sites, however.
Sign In or Register to comment.