Bibtex export

2
  • Just chiming in to indicate how my research group (and thus myself) generates bibtex keys. We use LastnameLastnameLastnameYY (last names of authors in order), and in the case of name collisions, we put 'a' or 'b' etc. at the end.

    Obviously one issue with key generation of this style is which paper gets the 'a', 'b', etc., because if you regenerate the list, you don't want your citations to change without you knowing. For example, if you have Smith05a and Smith05b, and add another reference, you want to make sure that neither Smith05a nor Smith05b become Smith05c, even after importing and exporting.

    However, when collaborating with people from outside of my group, I've had to deal with importing other peoples' bibliographies with different key styles, so better key management is something that definitely needs to be addressed for zotero to be my primary reference manager.

    Also, I haven't used it enough to know yet, but how well does zotero deal with mapping between unicode characters and Bibtex? For example, I'm citing someone who's last name is Jøsang, which should be represented as J\osang in Bibtex. I tried importing one of my bib files and zotero kept the \o. It doesn't seem to be a problem, but if I had gotten this reference from online, would it have changed it to the \o?

    Thanks for zotero btw. I currently use Jabref, but I very much look forward to primarily using zotero once the bibtex support is stronger!
  • Bibtex is one of the most commonly used citation format for research scientists and other latex lovers. There are 3 things that need to be improved for this format. First of all, the user should be able to design his own keys, like Dan Stillman suggested, something like {%c - }{%y - }{%t{50}} in the export tab of the preference. A good default would be like [last name]_[year]_[significant words of title] where the title words should exclude articles like a,the and also the markup words like [sup] [sub] etc.

    Second, the user can select what fields to be exported. Especially the fields like note and abstract.

    Third, I would suggest that abstract should be made a standalone tag just like note. it is too awkward to read it in the info tab.

    last, Thanks for your great work.
  • Is there a case here for importing a tad of the UNIX philosophy of having small tools operate together, rather than building everything in?

    I'm just thinking that Bibtool does such an efficient and fast job of editing/converting BibTeX, that replacing its functions with javascript might be developer time wasted.

    Would it be possible instead for Zotero to have a user-configurable setup for post-processing of exports (ie. spit zotero's export straight into Bibtool, or something else)?
    Users could then supply 'recipes' for wiring Zotero up to different means of transforming exports. Zotero's x-platform nature might of course make this difficult, but it's a thought.
  • So, what is the way to use custom key of bibtex entries? For me it is principle issue, otherwise I can't use zotero :(
  • I would agree with CB that it would be best to allow user to define their own key generation rules. Why force people to use one naming convention? Some might be more appropriate to some applications than to others...

    Whether that is done by external post processing or whether a small "bibtex" format field in zotero is secondary IMHO.
  • a small "bibtex" format field in zotero
    More generalized/accessible support for citation keys will be added eventually, but a format field already exists—it's just stored in the BibTeX translator itself.

    1) Make sure you're using at least Zotero 1.0.7, and then use Reset Translators and Styles in the Advanced pane of the Zotero prefs to ensure that you have the latest version of the BibTeX translator available at the time that version of Zotero was released. (The BibTeX translator isn't pushed to clients in Zotero 1.0.x.)

    2) Install Scaffold.

    3) Open Scaffold from the Tools menu, click the "Load from Database" button, and select BibTeX from the list (near the bottom). Click on the Code tab, and you'll see a format string at the top. It's not very sophisticated at the moment, but it's a start. If you want to add support for additional variables, you can edit buildCiteKey() further down in the code. Patches welcome on the dev list (though if you want to submit a patch make sure you've done Reset Translators and Styles in the latest 1.0 dev build, which has a more recent version, not 1.0.7).
  • Thanks Dan. That was a great step-by-step guide for a newbíe! Zotero has just become perfect. ;-)

    Out of curiosity: do you know if the code handles duplicate keys at the moment? After all it is not that unlikely that an author writes two articles in the same year that begin with the same word in the title: research topics don't change very quickly...
  • I firmly believe that there should be an additional field "citekey" that is editable in the Info tab of each entry. This could be set by default as done in the bibtex exporter.

    Instead of building the citekey within the bibtex exporter, this could be done in the main code (i.e. assigned after importing, and whenever saving the entry if the citekey field is blank) and sent directly to the bibtex exporter as a single variable "citekey".

    I'm willing to help develop and test if none of the devs have the time to do so.
  • See also my fervent justification of a "citekey" / "LocalID" / BibTeX key in Zotero in this thread, trying to take into account some of Bruce's larger concerns with this.

    http://forums.zotero.org/discussion/4331/bibtex-importexport-problems/#Item_17

    (Convinced, Bruce? :-)

    This could also eventually be useful for non-BibTeX plaintext citation engines, as requested here:
    http://forums.zotero.org/discussion/4007/onenote-support/#Item_6
  • I'd add another vote for exposing the key field as a csl variable. Is this feature planned?
  • @scot: I missed this earlier. No, I'm not convinced, since you haven't solved the (very real) problems I've outlined. Note, though, I posted some thoughts about this recently on zotero-dev.

    @andym_77: the issue isn't CSL, which already supports this in the form of the "label" variable. And Zotero can generate that variable, so it's not like it's not supported. It's some people want to maintain their own labels.
  • @bdarcus: You're right. I was trying to define what seemed to me a whole range of cases where robust universal identifiers weren't worth their cost (namely when I don't collaborate, or when I collaborate with a known group which agrees to certain constraints). I like working in plain text, and mainly I just don't want to *see* berzerk global IDs in my writing. But fair enough. I do want to be able to (1) collaborate with people I didn't start out planning to collaborate with (2) change the software I use to write and manage references with (3) merge datasets from various places. And I see that all of that is only possible if I use good identifiers everywhere. Also, the problem of unreadable, untypable identifiers can be solved in other places. If I want to use {Darcus_2025_Memoir} for my citation key while I'm writing in some plain-text workflow, (LaTeX, pandoc, etc) I just need to have some software support which lets me include the "real" universal identifier associated with that (document-local) key somewhere in or along with the document. It wouldn't be that hard to meet the needs both of the writers and the machines they use, and there are probably even other ways to do it, besides the {My_Key} = uuid model.

    Some editors (Emacs, presumably TextMate) could be taught to hide the uuid, but show some useful bibliographic data for the writer's sake. In any case you probably want some kind of automated citation insertion mechanism, which will save you from ever having to enter uuids by hand. Even BibTeX is more pleasant with such a system.

    But let's assume a basically BibTeX-like writing workflow, and see for a minute where that gets us with local keys vs. universal IDs.

    I come to a place where I want to make a citation. Assume the mechanism is \cite{Local_Key}. I'm thinking of the book to cite. I know the author, the title, and/'or the publication year. I can generate the citation two ways (1) without help: I have some easy personal key "system" which lets me get the right key for a resource 98% of the time. The rest I'm happy to debug later. or (2) software-supported. I pull up my citation-insertion application (or extension) and I type in some metadata which lands me on the right item in my database. I let the citation manager do the insertion. I do want to be able to see what I've done later, so I need some basic visible metadata which the UUID may not (or may, for all I know) provide. But this can be handled.

    In the second case, I don't even need a local ID. So long as I can either live with the UUID where the typesetter expects to see it, or I have a featurefull editor who can hide it from me.

    In the first case I do want a local ID of some sort, but even then, maybe it doesn't have to be in the database. (It's getting late and I can't think this through to the end, but I am convinced now that we should just quit fiddling around and move to global identifiers, and figure out human-friendly ways to do that as we go. I'm still partial to making it easy for existing BibTeXers in the meantime, on the basis (which you will surely have sympathy with, Bruce) that "Hey, doing software assisted academic citation is hard, and the competent options haven't been very numerous. The people who took the BibTeX route made a decent decsion given the circumstances. Maybe it won't compare to the suite of options we'll have made available in a few years, but just now, they're committed, and their data is committed, so lets put out a little something to give them an upgrade path. Those BibTeXers do have a lot in common with you (us) in the Zotero and CSL teams, anyway."

    Anyway, I don't know how far this gets us, or even what it means for Zotero. But it does make some sense to me, even this late, that we provide local key management at the same time as we encourage global key use. Pretty soon the advantages of global keys will come up and bite even the died in the wool BibTeXers. Until then, since LaTeX users have no present global-ID solution that I know of, we don't take away their (our) way of working for a better vision which hasn't quite arrived. And we do it in a way that those who don't need local keys don't pay too much for their inclusion.
  • edited June 1, 2009
    FWIW, I've been working through how the Zotero ought to be modeled (in RDF, but this is really more general than that). My argument is that you split the zotero item (the user data) from the source that it refers to. With with that approach, you can retain the global id(s) (which you attach to the latter), and the local ones (which you attach to the former). So upon, say, outputting to BIbTeX, you just flatten it a bit.

    I also think you're right that, no pun intended, the key thing is not the key (except in those styles that use it as the label!); it's an "easy-to-use way to indicate what you want to cite without having to switch context in order to find the damned ID!" It's certainly a major hassle, for example, to force users to have maintain their own keys if they have 3,000 items.
  • Zotero puts double braces in BibTeX epxort:

    ---------------------------------------------------------------------------------
    @article{meisel_electron_1976,
    title = {{ELECTRON} {EXCHANGE} {AND} {ELECTRON-TRANSFER} {OF} {SEMIQUINONES} {IN} {AQUEOUS-SOLUTIONS}},
    volume = {98},
    issn = {0002-7863},
    url = {http://apps.isiknowledge.com/full_record.do?product=WOS&search_mode=AdvancedSearch&qid=1&SID=1DHfjGH1lmBi918khCh&page=1&doc=1},
    number = {24},
    journal = {{JOURNAL} {OF} {THE} {AMERICAN} {CHEMICAL} {SOCIETY}},
    author = {D {MEISEL} and {RW} {FESSENDEN}},
    year = {1976},
    pages = {7505--7510}
    },
    --------------------------------------------------------------------

    this should be

    --------------------------------------------------------------------
    @article{meisel_electron_1976,
    title = {ELECTRON EXCHANGE AND ELECTRON-TRANSFER OF SEMIQUINONES IN AQUEOUS-SOLUTIONS},
    volume = {98},
    issn = {0002-7863},
    url = {http://apps.isiknowledge.com/full_record.do?product=WOS&search_mode=AdvancedSearch&qid=1&SID=1DHfjGH1lmBi918khCh&page=1&doc=1},
    number = {24},
    journal = {JOURNAL OF THE AMERICAN CHEMICAL SOCIETY},
    author = {D MEISEL and RW FESSENDEN},
    year = {1976},
    pages = {7505--7510}
    },
    -----------------------------------------------------------------------------------------------

    With the double braces, other programs (e.g. JabRef) have trouble reading what is exported from zotero. Of course, I could write a small Python script that does the job, but it'd suck to have to invoke it every time I export.

    Looking forward to using zotero soon!

    Cheers

    usenix
  • Zotero puts double braces in BibTeX epxort
    Yes, this is by design. It preserves the case of words with inter-word capital letters (e.g. {McLean}, {arXiv}, {AlMgSc}, {DNA}).

    In your example, you have several entries in allcaps that I don't think you want. How did you import that data? I suspect that this is where an improvement should be made. Note that you can also right-click on fields to transform text in zotero.
    With the double braces, other programs (e.g. JabRef) have trouble reading what is exported from zotero.
    JabRef handles most braces fine for me. If BibTeX can process the file & JabRef has problems importing it, I'd say the bug is with JabRef. If BibTeX can't process the file (which can happen with some Zotero output, but does not happen just because a string is enclosed in braces), the problem would be with Zotero's export (but might not be where you think it is).
  • Thank you for your reply, noksagt.
    You are right that BibTeX can process the files properly and JabRef can't handle the double braces-- I'll see how this can be fixed in JabRef.
    It is more the forced preservation of the allcaps entries that bugs me. I get these entries from isiknowledge. Entries for older articles _are_ in allcaps there (sometimes title&authors are in capitals and the abstract is "normal").
    So it's a question how zotero handles these. Obviously, when I use such a reference, I don't want it to be in forced allcaps but rather leave it in "unforced" capitals and then correct manually where necessary (less work than the other way round). For the authors, you'll get in right in 99% of the cases by setting all letter but the first ones of the names to lowercase.

    The most elegant way to achieve this would be that zotero recognizes when a title/authors are in capital letters and then wouldn't use the double braces. This would be a piece of cake in python but I don't know anything about JavaScript.
    The less elegant way would be an option to turn off double brace export for specific or all entries in the database.

    Keep up the good work!

    Best,

    usenix
  • All-caps fields would also provoke issues in the output of many CSL styles. Garbage in leads to garbage out.
    The most elegant way to achieve this would be that zotero recognizes when a title/authors are in capital letters and then wouldn't use the double braces.
    I disagree. It'd be more elegant to make such fixes when data was imported.

    Are you using the ISI translator? Do you have example links and/or a recipe that would lead to the references with all caps?
    This would be a piece of cake in python but I don't know anything about JavaScript.
    It is simple in javascript too.
  • I know I know, in the end it's the fault of isiknowledge (not using any translator), but I doubt they will change their database no matter how kindly we ask... of course it would be great to do some corrections already during import, so that not only BibTeX users benefit from that.

    To get some sample allcaps entries, go to isiknowledge.com, click "WebofScience", then click "Advanced Search" and search e.g. for "au=huie and py=1990"... you will get allcaps only. Note that these come without any abstract.
    For examples of title&authors in capitals, but a "normal" abstract, try e.g. "au=MATYUSHOV dv and py=1993" in the advanced search.

    best, usenix
  • Thanks.
    O/T for this thread (re. ISI translator): Other fields in their export (FN,LA,DT,JI,SC, etc.) are also in mixed-case. Perhaps the recipe would be to test if both TI and AU are all caps and, if they are, apply the title-case or senetence-case transformation, dependenting on the field.
  • I just tried to use scaffold to format BibTeX keys, following Dan's instructions above, however when I click the "Load from Database" button, there are no translators to choose from, just an empty window..

    Anyone know what might be causing it?
  • I don't think scaffold works w/ the beta version. The translators are kept as separate javascript files, rather than in the database now.
  • I also tried to use scaffold (with zotero 1.0.10) but I cannot install scaffold, Firefox says it is not compatible:
    "Scaffold 1.0.2 could not be installed because it is not compatible with Firefox 3.5.2."
    Am I doing something wrong here, or is scaffold outdated?
  • ijvm: I just bumped up the Firefox maxVersion for Scaffold on the update manifest to install on Firefox 3.5. Try again. (Firefox may cache the previous manifest, so restart Firefox and perhaps clear your cache if you have trouble.)
  • edited August 21, 2009
    Thanks Dan, now scaffold works like a charm.

    For some items zotero exports the DOI field to bibtex just fine:
    ... doi = {10.1038/nnano.2006.187}, ...
    Whereas for others it includes double brackets:
    ... doi = {{10.1103/PhysRevLett.97.137205}}, ...

    Double brackets in the DOI are not understood by common bibtex styles (e.g. made with makebst/merlin + urlbst). The result are the appearance of single brackets in the bibliography output and a non working hyperlink.
  • edited August 21, 2009
    OK, I fixed the issue with the exported DOI field to Bibtex using Scaffold.
    I just modified the function 'writeField' by changing line #1801 from
    "if(field != "url") {" to "if(!((field == "url") || (field == "doi"))) {"

    The fix is really needed, as there are DOI with illegal characters that were transformed into escaped characters, resulting in broken hyperlinks. For example consider the following DOI:
    doi = {10.1002/1521-3951(199712)204:2<817::AID-PSSB817>3.0.CO;2-D},

    Now what needs to be done to get this into the stable version of zotero?
  • I just came across Zotero and must admit I am ready to switch. However, the missing ability to customize the bibtex label is currently holding me back. In all of my papers (with latex, bibtex) I use my own labels and cannot imagine going back and changing them to a version mandated by Zotero.

    So what is the current plan for this problem? Why not let Zotero suggest a label after adding an item (according to the automated process currently used) and leave the possibility of a user changing it to whatever they want. I think trying to convince researchers to give up their labeling system will keep many bibtex users away from this awesome tool. So please add a bibtex label field...
  • I'm not quite sure what you need -
    if you want to change every key by hand, you can just modify the bibtex file in a text editor.
    If you have a key-structure that you prefer, you can modify the bibtex.js translator (either directly (on 2.0) or using Scaffold (on 1.0) as people describe at length above.
  • Changing all the labels in the .bib file defeats the purpose. I want to be able to export a current bib file from Zotero whenever I added a few references for a paper. In that case I would always overwrite the changes I made in the .bib file before.

    I am on 2.0beta, so Scaffold is at least currently not working. The bibtex translator again produces automated labels (correct?) and thus does not preserve my previously chosen fields.

    Why is it so problematic to allow a idiosyncratic \label field for those users who prefer it?
  • I am on 2.0beta, so Scaffold is at least currently not working.
    You don't need Scaffold. The BibTeX translator is just a JavaScript text file in the 'translators' directory in your Zotero data directory and is trivial to edit. Restart Firefox after editing.

    More flexible citation key management is planned for the future.
  • i just checked the bibtex.js code and i must say that e.g. hardcoding words that are to be excluded in the key is no the most elegant way to solve this. i agree with the critique above and that it would be good if in zotero was an easy way to
    i) set the bibtexId and
    ii) configure the way keys are generated automatically

    in my case i just struggle with the lyz addon which generates keys differently from zotero causing all kinds of problems...


    var citeKeyTitleBannedRe = /(\s+|\b)(a|an|from|does|how|it\'s|its|on|some|the|this|why)(\s+|\b)/g;
Sign In or Register to comment.