Improving BibTeX export

13
  • aurimas: "a bug report would be helpful."

    Of course, you're right. I should have kept track of my precise steps. The formulation of clarity will take some effort now.
    I didn't change the GS tranlator, the problem could have greater scope, I haven't tried other websites yet.

    adamsmith: "assume the problems are related to spartanroc's bibtex translator modifications"

    That, of course, is the concern.
    There have been certain indicators that seem to point to this. The most significant one being that Google Scholar's little icon doesn't seem to work since installation of spartanroc's modifications.

    Actually, it doesn't seem to be a general translator problem, the Wiki icon works, I just tested it.

    I reversed the changes, and Google Scholar worked, but slowly, sometimes saving the items in the 'Unfiled Items' section, with a lot of unresponsiveness, too.

    So, I uninstalled Zotero a number of times. Reinstalled Zotero, reinstalled spartanroc's modifications, Zotero wouldn't work, or the modifications wouldn't work. I continued this process, and eventually got it back to what it was, functional, except for Google Scholar's 'envelope' icon, and the 'Retrieve metadata' function.

    BibTeX import works, so that will have to be the 'workaround' for now.

    spartanroc's changes work on Firefox 11 Portable, at least on my system.
    So there is the possibilty of running two Zotero installations, one without spartanroc's changes, for seamless access to Google Scholar, and one with the changes, for filepath export.
    This is assuming that the Google Scholar problem is a predicate of those changes, and Im not sure that it is.
    Until full GS functionality can be achieved on a 'changeless' install, whilst compromised GS functionality is an unavoidable characteristic of the 'changed' install, I don't think the inference suggested by the earlier assumption ncessarily holds.

    There is a problem, but it has yet to achieve specificity as to its aetiology.
  • Until full GS functionality can be achieved on a 'changeless' install, whilst compromised GS functionality is an unavoidable characteristic of the 'changed' install, I don't think the inference suggested by the earlier assumption ncessarily holds.
    So you're saying that GS is not working with the default Zotero installation?

    If so, start a new thread and produce a bug report

    At the very least, google scholar should be showing the folder icon in the toolbar, since that does not depend on anything related to BibTeX.
  • The 'envelope' icon always appears. But it isn't fully functional. That is, with spartanroc's changes, clicking it produces a 'Can't Save, Known Translator Issues' response from Zotero.

    During the uninstalling/reinstalling of Zotero, the 'changeless' or 'default' install was working with GS, erratically at first, improving with each new install.
    But not quite back to where it was, before all the adjustments. So, yes, the 'default' install was working,
    and well enough, that I judged the reintroduction of spartanroc's changes would bring the same result as before, when GS was working like a racehorse with a 'default' install. It wasn't quite back to that yet. Perhaps because of all the adjustments?

    Reinstalling spartanroc's changes resulted in Zotero not even opening, a few times . After more uninstalls, I could add the 'changes' again, and things are back to where they were with the 'changes' install. GS doesn't work, it won't save items etc..

    If everything is working for you, then it should work here.
  • Back up your library if you haven't yet, then completely remove Zotero including the data directory and do a fresh install. See if everything is working fine then move on with spartanroc's modifications.

    Sounds like some of the modifications are being left behind after uninstall.
  • (if this is just the translator I would guess that "reset translators" from the advanced tab of the Zotero preferences might be enough to do the trick: since translators are in the data folder which isn't by default removed on de-install, I think changes to translators aren't affected by re-installing Zotero. So try this first - if I'm wrong, do what Aurimas says).
  • aurimas: That's what I thought. I backed up the data folder yesterday. I'm wary of deleting it, though.

    adamsmith: Just got your post whilst writing this.
    Resetting the translators is good for the default install, not good for spartanroc's install, obviously, as his files are replaced.

    As the problem is with Google Scholar, perhaps a more specific approach might be better? There is a Google Scholar translator. Might be worth looking at?
  • well - my suggestion would be to reset translators and then again replace the bibtex.js with spartanroc's version - if anything in the translator directory has gotten corrupted in the whole back and forth that would take care of it.

    I doubt whether looking at the GS translator will yield much - look at the error report created after triggering the error on google scholar - the content of that will tell us more.
  • This is from trying to save all the items on this page:

    http://scholar.google.com/scholar?q=zotero+translators


    "Could not save item
    An error occurred while saving this item.
    Check Known Tranlator Issues for more information"

    Report ID: 570393330

    Error report: [JavaScript Error: "this._sandboxManager.sandbox.ZOTERO_TRANSLATOR_INFO is undefined" {file: "chrome://zotero/content/xpcom/translation/translate.js" line: 1619}]


    The file referred to is one of spartanroc's. The line reads:

    "1619: var configOptions = this._sandboxManager.sandbox.ZOTERO_TRANSLATOR_INFO.configOptions;"

    aurimas, your name is in the GS file
  • edited April 21, 2012
    There has been considerable development activity in the translation layer in the Zotero 3.0 branch, and there is a mismatch between the translate.js file shipped with the bundle and the internal interfaces expected by Zotero.

    I may have missed something, but it looks like the code has been distributed with whole copies of the Zotero core files that have been modified (translate.js and translate_item.js). That isn't going to work very well; a diff against the current version of the files turns up many unrelated changes, and it's hard to see what is part of the feature patch, and what is just outdated code left over from the old file that was modified.

    To make this maintainable, patches should be derived from both files against the originals on which they are based. Someone should then fork the Zotero client code, and apply the patches there by hand, to work the changes into the current code. Once a set of patches is on file, developers can see what is intended, and make any necessary adjustments when conflicts arise due to changes in the underlying code.

    If someone undertakes maintenance of the fork, later changes in Zotero can be merged to it. That will turn up any conflicts, and allow the client to be quickly tried out as it would run in Zotero proper, to be sure everything is working correctly.

    (This is essentially what Simon requested here. If the code is already up in a fork and I've missed it, please forgive.)
  • edited April 21, 2012
    I may have missed something, but it looks like the code has been distributed with whole copies of the Zotero core files that have been modified (translate.js and translate_item.js).
    The patches are actually included, but they are in between files on the gist and are easy to miss. E.g. https://gist.github.com/956623/#file_bib_te_x_tan.patch
    The way to get this included in the main Zotero distribution would be to split up the separate features into separate patches and submit separate pull requests on GitHub.
    and they are not split up by feature. Though it's not a large undertaking to split them up, which might make everybody's life easier.
  • I didn't use the patches because I thought they were for an earlier version of Zotero.

    Are they necessary?

    I followed these instructions:

    "New Updates:

    updated: Mar 16, 2012

    1. updated for Zotero 3.0.3

    2. item_local.js is replaced by translate_item.js

    3. installation guide:
    (a) go to https://gist.github.com/956623/ and download the full tar file
    or the following 4 files: translate.js, translate_item.js, BibTeXTan.js,
    and BibTeXKeyOnly.js
    (b) go to your firefox profile directory, then use any zip tool to open
    extensions/zotero@chnm.gmu.edu/chrome/zotero.jar. Then navigate to the
    archive directory: content/zotero/xpcom/translation/ and use drag & drop
    to replace the two files of translate.js and translate_item.js in the jar
    file.
    (c) go to your zotero data directory (default location is called zotero
    right under your firefox profile directory). copy the following files
    to the translators directory: BibTeXTan.js and BibTeXKeyOnly.js
    (d) restart your firefox.
    "

    and:

    "List of files:

    reame -- this file
    bibtex_js_and_path.gz -- tarball of everything
    BibTeXTan.js -- to replace the translator BibTeX.js
    BibTeXTan.patch -- patch file relative to BibTeX.js of zotero v2.1.6
    BibTeXKeyOnly.js -- new translator for quickcopy / drag&drop export of cite keys
    BibTeXTanKeyOnly.patch -- patch file relative to BibTeXTan.js
    item_local.js -- to replace content/zotero/xpcom/translation/item_local.js
    item_local.patch -- patch file to that of zotero v2.1.6
    translate.js -- to replace content/zotero/xpcom/translation/translate.js
    translate.patch -- patch file to that of zotero v2.1.6
    "

    The patch files refer to Zotero 2.1.6.

    In the "updated: Mar 16, 2012" instructions , no patch files are mentioned.
    Although, this instruction - "BibTeXTan.js -- to replace the translator BibTeX.js" - is not in the updated instructions. It is important, because it implies removal of the BibTeX.js file, something I neglected to do, with BibTeX export not working until the file was removed.

    The other thing is I don't know what to do with a 'patch' file. Do I put it in the same folder as the file it 'patches'?
    If that is the case, then there is a possible contradiction in the file list.

    " BibTeXTan.js -- to replace the translator BibTeX.js
    BibTeXTan.patch -- patch file relative to BibTeX.js of zotero v2.1.6"

    If 'BibTeXTan.js' replaces 'BibTeX.js', then why is 'BibTeXTan.patch' relative to 'BibTeX.js' (which is supposed to be replaced by 'BibTeXTan.js)'?

    Also, why is it called 'BibTeXTan.patch', after another file?

    It must be an error.
  • edited April 21, 2012
    The patch files don't need to be installed anywhere; they just provide a record of the original changes made to the *.js files. A version control system like GitHub really does make handling these things easier.

    As a thought on the side, one thing that might be relevant to the failures is that on some systems (Linux, at least, maybe others), Firefox/Zotero will pick up a tilde backup file (such as "BibTeX.js~" if it is present, and ignore the original. It might be worth checking whether there are tilde backup files lurking in your translators directory.
  • fbennett: There are no tilde backup files in the translator's folder.

    A strange thing - I put BibTeX.js back in the translators folder. And spartanroc's changes worked this time, whereas they didn't before. I noticed the filepaths for two attachments were in reverse order. Google Scholar worked, too, but very slowly.

    These are the steps I used to implement spartanroc's changes:

    translate.js (in zotero.jar)
    translate_item.js (in zotero.jar)
    BibTeXKeyOnly.js(in "Translators" folder)
    BibTeXTan.js (in "Translators" folder)

    remove default item_local.js (from zotero.jar)
    remove default translate.js ( from zotero.jar)
    remove default BibTeX.js (from "Translators" folder)

    In Zotero. jar, these files are listed:
    tlds.js
    translate.js
    translate_Firefox.js
    translate_item.js
    translator.js

    Is all of this correct?
    Have I missed anything?
  • A quick update:

    Upon uninstallation of the Zotero/Firefox extension, AutoZotBib, Google Scholar is back to full functionality. Everything is fast, too.
  • contact the author of that extension, it's a recent addition and he sounded like he'd appreciate bug reports.
  • It's half a bug, really. Possibly a conflict between his customisation and spartanroc's. Both of them were coding from the presupposition of the official Zotero code base.
    I can send him an email, referring to this discussion thread.
  • Hi everyone,

    I'm Robin Wilson, the author of AutoZotBib. I'm sorry for any problems my extension may have caused - I'm fairly new to Zotero and this was my first go at an extension. I've got a few questions if that's ok (if this isn't the most appropriate place then let me know):

    1. Could you let me know what version of AutoZotBib you are using - it should be listed in the plugins/add-ons manager.

    2. When installing AutoZotBib - did you do the optional steps listed at http://www.rtwilson.com/academic/autozotbib under 'Advanced Usage'. That's the bit that starts using Tan's BibTeX translator. If so, could you tell me the content of all autozotbib.* items in about:config.

    3. I notice that the BibTeX translator in Spartanroc's package has an ID of 1234567, which I suspect may be causing some problems (is that a valid guess?). The slightly-altered version of Tan's translator that I link to on my site (http://www.rtwilson.com/academic/autozotbib) has a 'proper' ID put in there, which may help.

    Hope at least some of that is useful/makes sense.

    Cheers,

    Robin
  • Thank you for your reply, Robin.
    Your extension is very good, the problem is the interaction with sprtanroc's extension, which neither of you could be expected to account for.

    1. It was installed recently, this year. As it's uninstalled, and I can't find the setup .xpi file, version number can't be confirmed.

    2. No, just the basic steps. As I recall, the default citekey style differed from what I wanted.
    The "content of all autozotbib.* items in about:config", now, is one string, giving the location of the output file.

    That's all I can say for now.
    Will try a reinstall of AutoZotBib, when I get the chance.
  • Anything happening in this space now? I see Zotero's bibtex export is now better from another forum...is the auto-export/update bit implemented or should one still install a plugin/extension?
    Cheers :-)
  • "I see Zotero's bibtex export is now better from another forum"

    Is there a link for this?

    I'm not aware of any new developments regarding Zotero's bibtex export.
    Thank you.
  • The bibtex export constantly has small improvements that are made:
    https://github.com/zotero/translators/commits/master/BibTeX.js
    What still needs to be addressed on the translator level? I think BibTeX exporty would most benefit from the eventual addition of a 'local id' (citekey) field in the database, which is beyond the scope of BibTeX.js. From some suggestions in this thread, the translator performs file export, handles some additional UTF-8<->LaTeX entity transliteration, and basic HTML rich text markup<->LaTeX conversions. Less related to this thread, there have been improvements made to note export, corporate author support, case preservation of titles, access date support, and more.
  • Sorry theotechne - not my most useful comment. I can't find the forum post (it's on zotero) mentioning it now, but the improvement was in indexing the PDF file path when exporting to bibtex (I think).

    The other improvement that'd be great is if the auto-export was working without the AutoZotBib plugin (which I am now trailing - so fingers crossed it's all smooth!).

    Cheers!
  • What noksagt says on the bibtex translator improvements.
    There are some issue with preserving capitalization that are still unresolved:
    http://forums.zotero.org/discussion/25362/capitalize-title-in-bibtex-export/


    I'd also like to see better auto-export of bibtex, but unfortunately AutoZotBib is only a partial solution - due to the way it works, on reasonably large libraries it will lock your Zotero down for several minutes after each change to the database (i.e. adding or editing an item).

    Dan has mentioned elsewhere that solving this would likely require changes in the Zotero code itself, though he didn't specify what.
  • goodwin57
    "the improvement was in indexing the PDF file path when exporting to bibtex (I think)"

    adamsmith
    "There are some issue with preserving capitalization that are still unresolved"


    If this means that exporting item filepaths is to be incorporated into Zotero, that is good news.

    As to AutoZotBib, uninstalling it solved issues, as indicated in previous posts. Haven't had time to test AutoZotBib again. Spartanroc's modification works, that's good enough for now. My concern was the integration with SciPlore. That has been achieved.
  • Zotero exports filepath in bibtex. Has for a while now.
  • adamsmith

    "Zotero exports filepath in bibtex. Has for a while now"

    That is great news!
  • theotechne - my interest is in sciplore/docear too, but as I understand automating the bibtex export is quite an important part of that? I''m going to try and work with the AutoZotBib and see how far I get...
    (as it happens I seem to be having other issues with it too! but never mind!)
  • goodwin57: "my interest is in sciplore/docear too, but as I understand automating the bibtex export is quite an important part of that?"

    That was probably my intention, too, in installing AutoZotBib. Keeping SciPlore/Docear's BibTeX reference file automatically updated. However, it is not so crucial a function, for me. Large areas of my Zotero library are not properly referenced yet, many items lack bibliographical information. Not all academic websites are 'Zotero-aware' yet, manual work is necessary.
    It is not difficult to batch-export relevant items into one's SciPlore map, as, and when, necessary.
    After adamsmith's post, have uninstalled spartanroc's modification, and tested Zotero's native bibtex export. It seems that you can just 'drag and drop' the file from a Zotero item straight into SciPlore, and the resulting SciPlore link node can call up the document. I'm not sure this was possible before. With spartanroc's modification, one had to open a file explorer with Zotero's 'Show file', and drag the file from there.
    Am busy configuring other areas of 'workflow', sometimes it's difficult to keep up to date.
    Good luck with AutoZotBib!
  • @ adamsmith
    "Zotero exports filepath in bibtex. Has for a while now"
    is Zotero exporting only the path to the pdf? Where i can choose this possibility?
  • is Zotero exporting only the path to the pdf? Where i can choose this possibility?
    No action should be needed. If it doesn't work, you may need to reset your translators. If "export files" is not checked, you should get 'file=' keys that point to the attachments in your Zotero storage directory. If you select "export files", there will be a relative link to the copy of the file that is generated.
Sign In or Register to comment.