Upgrade StorageBibTex Export
Hello,
I've been having trouble lately with the BibTex export. I've noticed it in particular since the use of html notes in 1.5. Basically, the export translator exports the notes and things as "comment" lines in the bibtex file. But I've found that if you use notes for much of anything, this causes a problem. I was having issues with documents that wouldn't compile due to unidentified control sequences... and in other cases, the bibliography produced in latex included things like the abstracts of articles.
I took a look at the bibtex.js file in zotero's translators folder. I noticed that the end of the file included the ability to export the notes as well as the "extra" field as a note. The extra field is not inherently problematic, but I've had trouble with it because the JSTOR translator puts the abstract in the extra field.
Anyway... so I just commented out that stuff... the end of my file now looks like this...
if(item.date) {
var date = Zotero.Utilities.strToDate(item.date);
// need to use non-localized abbreviation
if(date.month) {
writeField("month", months[date.month], true);
}
if(date.year) {
writeField("year", date.year);
}
}
//if(item.extra) {
// writeField("note", item.extra);
//}
if(item.tags && item.tags.length) {
var tagString = "";
for each(var tag in item.tags) {
tagString += ","+tag.tag;
}
writeField("keywords", tagString.substr(1));
}
if(item.pages) {
writeField("pages", item.pages.replace("-","--"));
}
if(item.itemType == "webpage") {
writeField("howpublished", item.url);
}
//if (item.notes) {
// for each (var note in item.notes) {
// writeField("comment", note["note"]);
// }
//}
Zotero.write("\n}");
}
}
Note the comments on the last "if" block - this removes the notes of course, and up above I commented out the if block for the extra field.
I just thought I'd post here... if you're having the kinds of issues I did, it's a simple and easy workaround.
As a more general note, I think it's pretty rare that I would ever want my notes (or even the abstracts for that matter) coming out in the bibtex file. Personally, I always use zotero as the manager, and just export the bibtex file purely for writing a paper. If I want to add a citation, I do it in zotero and re-export. I would guess I'm not alone in this. So for me then, I would much rather have (maybe a second) bibtex option in the output list that's just "BibTex (Simplified)" or something that basically does what I did above here... not export those big notes and extras... you could even drop the keywords... basically every field that won't ever be in a citation.
Mike
I've been having trouble lately with the BibTex export. I've noticed it in particular since the use of html notes in 1.5. Basically, the export translator exports the notes and things as "comment" lines in the bibtex file. But I've found that if you use notes for much of anything, this causes a problem. I was having issues with documents that wouldn't compile due to unidentified control sequences... and in other cases, the bibliography produced in latex included things like the abstracts of articles.
I took a look at the bibtex.js file in zotero's translators folder. I noticed that the end of the file included the ability to export the notes as well as the "extra" field as a note. The extra field is not inherently problematic, but I've had trouble with it because the JSTOR translator puts the abstract in the extra field.
Anyway... so I just commented out that stuff... the end of my file now looks like this...
if(item.date) {
var date = Zotero.Utilities.strToDate(item.date);
// need to use non-localized abbreviation
if(date.month) {
writeField("month", months[date.month], true);
}
if(date.year) {
writeField("year", date.year);
}
}
//if(item.extra) {
// writeField("note", item.extra);
//}
if(item.tags && item.tags.length) {
var tagString = "";
for each(var tag in item.tags) {
tagString += ","+tag.tag;
}
writeField("keywords", tagString.substr(1));
}
if(item.pages) {
writeField("pages", item.pages.replace("-","--"));
}
if(item.itemType == "webpage") {
writeField("howpublished", item.url);
}
//if (item.notes) {
// for each (var note in item.notes) {
// writeField("comment", note["note"]);
// }
//}
Zotero.write("\n}");
}
}
Note the comments on the last "if" block - this removes the notes of course, and up above I commented out the if block for the extra field.
I just thought I'd post here... if you're having the kinds of issues I did, it's a simple and easy workaround.
As a more general note, I think it's pretty rare that I would ever want my notes (or even the abstracts for that matter) coming out in the bibtex file. Personally, I always use zotero as the manager, and just export the bibtex file purely for writing a paper. If I want to add a citation, I do it in zotero and re-export. I would guess I'm not alone in this. So for me then, I would much rather have (maybe a second) bibtex option in the output list that's just "BibTex (Simplified)" or something that basically does what I did above here... not export those big notes and extras... you could even drop the keywords... basically every field that won't ever be in a citation.
Mike
Further: there are plenty of ways to strip fields from a BibTeX file after you have one.
You're exactly right that one big issue was in the JSTOR translator... but it's beyond that.
I was having general issues with latex compiling because of what I assume are the notes. The reason I figured this was the cause was that I would compile it, get an error, make the change and then not get errors. This seemed pretty definitive to me, as that was the only element changed.
I also agree that it's good to have a rich export, but I would argue that most using latex for their writing have no interest generally in a rich export most of the time. The analogy I would make is that when you are using the Word plugin, you don't care about most of the information in zotero, you just want to find the citation, put it in your document and move on. If you want to actually work with the citations, you'll probably go to Zotero.
LaTex is the same way... I am not working with the citations in the bibtex file... and I certainly have no interest in editing the file after I export it from Zotero, as you suggest.
I posted this here because - at least for me, and I'd assume others - the zotero export contains far more than it needs... and this just tends to cause problems in LaTex. Why deal with it?
This is why I would argue finally that it is best to either give options to the user about how rich that export should be, or provide a simplified export option.
Mike
If you are dead-set on stripping fields & are against using bibtool or similar, I would probably suggest downloading & modifying the bibtex.csl style file & using that (rather than "export").
Sorry for the delay here... I think I tracked down the issue somewhat... Where I actually ran into trouble was on the girlfriend's computer. I am not sure how this was happening, but it looks like the exported file had some weird line breaks in it that I ultimately concluded couldn't be coming from the Zotero code. Those line breaks were I think what was causing the document to not compile in latex.
I am not positive how those line breaks were getting in there, but I think it is likely that they are actually in the text of some of her fields. If that's right, the translator might want to be updated to remove or recode such breaks. I don't know much about how that would happen or what impact it would have... and I might be mistaken about the line breaks anyway.
To get back to the main argument... you make a good point. If the thing works, why not have all the data in the exported file? As I thought about it, I guess there really isn't any great reason. The only real thing I would say is that imagine I store all my notes in Zotero... Which I think increasingly makes sense now that they have HTML support... well I could be looking at at least a page of html notes for hundreds of articles. All that data would then go into the exported bib file every time.
This struck me as annoying. First, it's a (albeit small) waste of space on my computer - why make that file bigger than it needs to be. Second, and more importantly, when I am in WinEdt (or whatever editor) and I have to glance through the bib file for some reason - maybe to find a key -then I am scrolling through tons of lines of weirdly coded notes just to find references.
My only point this whole time was that I think the utility of the export changes based on how you think of it... If you think of the export as a way of transferring data around, then absolutely it should have everything possible... but I doubt most latex users want, or think of it this way... I have never thus far wanted that ever... I would much rather have a small, simple, file that has what is needed for bibtex to produce the citations and nothing else to get in the way... What I want isn't so much an export, but rather just better, simpler support for latex.
After giving this some thought, I decided that maybe it's really not a Zotero problem. Actually, I've been looking into how a WinEdt plugin could be put together to generate a simple bib file in the same directory as the document .tex file... Such a file would certainly not need the notes. That kind of plugin would work then in a convenient way not unlike the word or oo plugins. As I mentioned though... that's not so much a Zotero issue... it's a WinEdt feature. Still, I think it's analogous to the Word plugin. Right now, we don't have something like that for LaTex, so you end up with these unnecessarily detailed files that are ultimately more of a pain than what you need.
Mike
Some editors have support for folding, which removes your concern about scrolling to much. Many also have excellent search facilities.It isn't clear to me what the proposed plugin would do, but have you looked at the extension that provides lyxpipe integration? Unfortunately, WinEdt uses a different method to pipe references in. You can, of course, import your BibTeX into JabRef & use that.
I could dig the error down to too long lines in the bib-file coming from too long notes.
So if one could just disable the note/ annote export it would help a bunch.
Note that you can also increase the size of this limit (at least at compile time) in bibtex as well.
Other fields than 'note' can exceed this limit.
I think 'extra' should always be exported, despite the fact that some people happen to have long 'extra' fields.
But other translators do give the option as to whether or not to export attached notes & so it might be nice to make that optional for this too. If they are exported, should they be exported:
- as-is
- trimmed to 3000 characters
- exported as additional LaTeX files that are called by the BibTeX file
? And should we start to export abstracts assuming we have a good solution for notes?Version: bibtex8 --version
This is 8-bit Big BibTeX version 0.99c
Implementation: C for Unix
Release version: 3.71 (31 May 2005)
I call the bibtex run with bibtex8 --wolfgang, so it has already a very huge capacity. Still I get this error: "Your field is more than 3000 characters".
If you would ask me I would say that the user should be able to decide: Either the notes (and similar fields) are not exported at all -- or they are exported as they are (not trimmed or excluded). The reason is simple: In the first case the user wants to use the bib file in his LaTeX document. He doesn't need the notes there. In the second case he wants the bib file to be a backup and so he needs all information from the database in there.
I just discovered another issue today (don't know if I should open another thread for this -- feel free to split it into another post): The bibtex filter exports two fields: address and location. As it is exported the address relates to the publisher, the location relates to the actual repository the book can be found. Unfortunately, biblatex uses location with the publisher. It outputs: "location: publisher, year". So at the moment I rewrote the output in the biblatex style file into "address: publisher, year". But I guess the ordinary user won't do that.
So is this just a mistake or was it done by reason?
Cheers,
Mathias
'location' is not a standard field. It is used by other programs for quite a few different things (the most common being where a conference took place, BibLaTeX's 'venue'). I am quite surprised to see that BibLaTeX prefers it, when there are so many .bib files and other programs that do not agree on the meaning of the field. Arguably, Zotero's archiveLocation should not use this field due to the ambiguity. It isn't clear to me where archiveLocation should be exported instead (if anywhere).
I'll just keep my adjusted style files, this should do the trick in the next time.
Regarding bibtex8 and buffer size: Well, it might be that it's possible to increase the buffer at compilation. But to be honest: Not more than 5% of the users will ever even consider to do this. But all need a working solution. So I think the way to go is not changing bibtex8 but the bib file.
I think one of the problems comes from the 'doi' field. For example, export from Zotero gives:
@article{aihong_indian_2005,
title = {Indian Ocean variability and its association with {ENSO} in a global coupled model},
volume = {18},
issn = {08948755},
url = {http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=18512094&site=ehost-live},
doi = {Article},
abstract = {The evolution of the Indian Ocean during El {Niño–Southern} Oscillation is investigated in a 100-yr integration of an Australian Bureau of Meteorology coupled seasonal forecast model. During El Niño, easterly anomalies are induced across the eastern equatorial Indian Ocean. These act to suppress the equatorial thermocline to the west and elevate it to the east and initially cool (warm) the sea surface temperature {(SST)} in the east (west). Subsequently, the entire Indian Ocean basin warms, mainly in response to the reduced latent heat flux and enhanced shortwave radiation that is associated with suppressed rainfall. This evolution can be partially explained by the excitation of an intrinsic coupled mode that involves a feedback between anomalous equatorial easterlies and zonal gradients in {SST} and rainfall. This positive feedback develops in the boreal summer and autumn seasons when the mean thermocline is shallow in the eastern equatorial Indian Ocean in response to trade southeasterlies. This positive feedback diminishes once the climatological surface winds become westerly at the onset of the Australian summer monsoon. {ENSO} is the leading mechanism that excites this coupled mode, but not all {ENSO} events are efficient at exciting it. During the typical El Niño {(La} Niña) event, easterly (westerly) anomalies are not induced until after boreal autumn, which is too late in the annual cycle to instigate strong dynamical coupling. Only those {ENSO} events that develop early (i.e., before boreal summer) instigate a strong coupled response in the Indian Ocean. The coupled mode can also be initiated in early boreal summer by an equatorward shift of the subtropical ridge in the southern Indian Ocean, which stems from uncoupled extratropical variability. {[ABSTRACT} {FROM} {AUTHOR]}},
number = {17},
journal = {Journal of Climate},
author = {Z. Aihong and H. H. Hendon and O. Alves},
year = {2005},
keywords = {{EL} Nino Current, {INDIAN} Ocean, {METEOROLOGY,} {OCEAN-atmosphere} interaction, {SOUTHERN} oscillation},
pages = {3634--3649},
annote = {Accession Number: 18512094; Aihong Zhong 1 Hendon, Harry H. 1; Email Address: h.hendon@bom.gov.au Alves, Oscar 1; Affiliation: 1: Bureau of Meteorology Research Centre, Melbourne, Australia; Source Info: Sep2005, Vol. 18 Issue 17, p3634; Subject Term: {SOUTHERN} oscillation; Subject Term: {OCEAN-atmosphere} interaction; Subject Term: {METEOROLOGY;} Subject Term: {EL} Nino Current; Subject Term: {INDIAN} Ocean; Subject Term: {AUSTRALIA;} Number of Pages: 16p; Illustrations: 7 graphs, 8 maps; Document Type: Article},
annote = {Copyright of Journal of Climate is the property of American Meteorological Society and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. {(Copyright} applies to all Abstracts)}
},
When compiled using MikTeX pdflatex, or read in BibTeXMng, none of the fields after the 'doi' field are used. If I manually delete the 'doi' field, the reference is read correctly.
I too would like to have a "simple bib file" export, if possible where I could select the fields that I would like exported to my .bib file.
Cheers, Fiona.
Thanks, mryckman.
I have some article classes that insert the "extra" fields in the end of each respective bib entry, and that's "garbage" that I can't have in the final document, but that I need to have in Zotero to have some extra feedback. I usually use the "extra" field to insert things like the document conclusions (maybe I should put it in a note, but for now it's what I do).
I think this is a simple task to accomplish, as the checkboxes could be simply mapped in the Bibtex.js script to be some IF clauses.
And I'd be glad if Zotero's "don't give the URL for articles whose data includes a page range" could be used in the BibTeX export, since my current method of using bibtool for pruning all URL fields is a bit of a blunt instrument.
Another, related problem is that there are some special characters which I want to display: authors with accents in their names for instance. Is there a way of getting zotero to translate the library properly into tex? For example, the name Hummelshøj exports into my .bib file as Hummelshøj, which once compiled is then displayed in the bibliography as Hummelshj - it would be useful if I could export it directly as Hummelsh\{o}j.
I know I can change the .bib file using a text editor, but I have rather a lot of references, with a huge amount of accents and other weird characters in the author lists and titles, so I have a feeling this is going to take rather a long time.