Conversion problem with Betterbibtex and Pandoc (Using Zotero with Ulysses)

Hi there,

TL;DR: Generating Zotero citekeys with Better Bibtex, using Ulysses to write in markdown. adding citekeys. Converting citekeys to citations and bibliography in a word-document with an automated script/program utilizing Pandoc. Final citations end up as Author (Year), instead of (Author, Year). No problem with bibliography.

Markdown/Ulysses and Zotero
Recently I have gotten fond of doing my writing in markdown, which has complicated my process slightly in regards of citations. As a work-around I have been drafting with Ulysses, and still done most of the writing in Word, entering all the citations using the Zotero add-on. It is a pain in the ass and I just want to do all my writing in Ulysses, so I have been looking for a better way of doing this. And then I found this guide (for anyone using Ulysses to write, this tutorial is a gem*): http://raphaelkabo.com/blog/posts/markdown-to-word

The first time I saw it, I was a bit put off as I'm not much of a coder, but today I decided to give it a go. Things have gone remarkably well. Now I can even add citations to Ulysses through the same kind of Zotero dialogue as for the Word add-on. Though I have problems with the final conversion from markdown to docx. It is not turning out like I would like it to.

The problem
Following the guide I have made a tiny little program that is supposed to automate the Pandoc conversion, and after some tweaking it actually works. Running through the script I get a docx document where the citekeys are citations and the bibliography is perfect, but there is a huge problem: The year shows up inside the parenthesis, whereas the author's names show up outside.
For example:

@holmquist1998 =
Holmquist, Schmidt-Gengenbach, & Yoshioka (1998), instead of
(Holmquist, Schmidt-Gengenbach, & Yoshioka, 1998)

First I thought it would have to be the style-sheet I was using that was the culprit, but now I have tried two different apa style sheets, and one Chicago author date style, and the results are the same. I'm starting to wonder whether it can have something to do with the citekeys themselves, or probably rather how I have set up betterbibtext.

I'm not really sure whether this forum is the right place to ask, but there seems to be a bunch of people around here that really knows what they are doing, so I thought I'd give it a try.

An overview of my settings:
Zotero settings/export: Better BibTeX Citation Key Quick Copy

Zotero settings/Better Bibtex/Citation keys:
Citation key format: [auth:lower][year]
Force citation key to plain text - checked
QuickCopy format: LaTex

Anything here that looks off? If things look fine here, any ideas why the author name shows up outside the parenthesis? Really appreciate it if anyone has the time to help me out with this. I'd be super happy if I can finally be able to do all my writing in markdown.



*Following the guide, I had no problems until the last bit, where you can make a little program that automates the conversion so that you can send your text directly from Ulysses to word and get everything converted. It seems like the code doesn't really work with the newer version of Pandoc, so I did some slight tweaks, and now it kind of works. This is the code I'm using at the moment: https://ibb.co/iJxMCw (which paths you need to change, you can see in the original code in the tutorial).



  • Pandoc requires "regular" citations in square brackets, i.e.

    ...is an important finding [@holmquist1998]. -->...is an important finding (Holmquist, Schmidt-Gengenbach, & Yoshioka, 1998).

    without brackets give you citations with authors in the text:

    as @holmquist1998 say--> as Holmquist, Schmidt-Gengenbach, & Yoshioka (1998) say.
  • Of course! I feel like such a noob now...
    With the square brackets everything's working great. Thanks for the help :)
  • Hi,

    @aastals: Are you still using the code that you posted for the last step of the guide? I get the following warning in Ulysses when I try with your code:

    „[WARNING] Deprecated: —latexmathml, —asciimathml, -m pandoc: Cannote decode byte ‚/xba‘:Data.Text.Internal.Encosing.decodeUtf8: Invalid UTF-8 stream

    Any suggestions would be very much appreciated!
  • @aastals,

    I'm attempting to set up the workflow found at http://raphaelkabo.com/blog/posts/markdown-to-word

    but I am encountering trouble with the zotpick-pandoc applescript. I've set up the service module in Automator and the key command in system preferences. However, when using the key command, I get the error: The action “Run AppleScript” encountered an error: “The operation couldn’t be completed. (com.apple.Automator error -212.)”

    In the automator workflow, I get the syntax error: Expected end of line, etc. but found “<”.

    Any thoughts about troubleshooting the applescript?
  • Hey - are people still using this workflow?

    I have set it up and have got it functioning well. I do have one more thing to fix though:

    When I enter a citation 'as is' from the search, it is entered as [#citekey][].

    The # must be then changed (manually) before running my export to [@citekey][]. Where do I go to change the symbol just before the citekey, or indeed where do I go to make the workflow look for the # instead!!

    Thanks
    Alistaur
  • That would be a question for Ulysses
  • @aswilson if the reference comes out as "[#citekey][]" you have zotpick-pandoc set up to request the `mmd` (MultiMarkdown) format. To get the `@`, have it reques the `pandoc` format instead.
  • Hi there @emilianoeheyns, I really appreciate your response - however on inspection of my application script I'm still not entirely sure what to change!!]

    Here it is in full, could you point me to the right bit please?

    title=$(basename "$@" .md)
    echo -e "\n\n# Bibliography" >> "$@"
    /usr/local/bin/pandoc -s -smart --filter=/usr/local/bin/pandoc-citeproc --bibliography "/Users/awuni/Dropbox/_UNI/04. Library/_REF.MGMT/aw1.bib" --csl "/Users/awuni/Dropbox/_UNI/04. Library/_REF.MGMT/Harvard-ISO-690-Lboro.csl" --reference-doc "/Users/awuni/Dropbox/_UNI/04. Library/_REF.MGMT/reference.docx" -f markdown-smart -t docx -o /Users/awuni/Desktop/Output/"$title".docx "$@"
  • I don't really know -- I only make the backend that powers it. What you posted looks like the script to compile with pandoc though, not the script that triggers the picker. The latter would have to be changed to get the "@" after picking a reference, but you'd have to ask the zotpick maintainer about that -- I'm not great at applescript.
  • Yes, you were right - I used the wrong Zotpick applescript. Amended line 23 to:

    set theReference to do shell script "/usr/bin/curl 'http://localhost:23119/better-bibtex/cayw?format=pandoc' 2>/dev/null; exit 0"

    and it worked fine. Thanks
  • Hi there,
    I'm experiencing a citation formating issue. I followed Raphael Kalbo workflow and it works perfectly. However, in the word document generated, my citation numbers in the text appear with quotes "(1)" instead of the traditional (1). How could I get rid of the quotes ? (also i m no expert in apple script)
    Thanks in advance,
    Julien
  • It shouldn't involve applescript, I don't think the quotes come from the picker. Can you post the snippet of your markdown that has the reference?
  • @emilianoeheyns,

    sure. Here it is :

    ...performed as previously described[@dimastromatteo2018a]. ==> ......performed as previously described"(1)".

    and here are my settings :

    Zotero settings/export: Better BibTeX Citation Key Quick Copy

    Zotero settings/Better Bibtex/Citation keys:
    Citation key format: [auth:lower][year]
    Force citation key to plain text - checked
    QuickCopy format: Pandoc
    Surround Pandoc citations with brackets - checked

  • BBT settings don't matter at this stage - the markdown snippet looks OK, and after the markdown has been produced, BBT is no longer in the chain. You'll have to talk to kalbo/pandoc about this, nothing about Zotero/BBT is involved in rendering existing markdown to anything.
  • @emilianoeheyns ,

    Ok. Will do. Thanks for your help.
  • I have been trying to implement that workflow too and had the same kind of problems. But I kind of work my way through the problem, not sure how. I think it has to do with how you modify the Markdown symbols in Ulysses.

    Now, my problem is that the headers and bibliographical notices in the final docx appears between brackets as :

    [Bibliography]

    [Arnstein Sherry R., 1969, « A Ladder Of Citizen Participation », Journal of the American Institute of Planners, vol. 35, n° 4 : 216‑224.]

    Weird !
  • Just to be more precise, I switched the symbols used in Ulysses Markdown to curly braces, while using brackets whenever I insert a citation key. That give me in line citations between parenthesis.
  • I'm in the process of trying to get this workflow set up, and I've encountered an issue getting the right citations to appear through zotpick.

    When I drag and drop my items into a text box, they show up as expected:
    [@marcuse1969]
    [@lowy2001]
    [@takiguchi1939]

    However, when I use zotpick, I get the following results:

    @MARCuse1969
    @LOWY2001
    @TAKIGUChi1939

    My settings are the same as Juleind73 above, ie:
    Citation key format: [auth:lower][year]
    Force citation key to plain text - checked
    QuickCopy format: Pandoc
    Surround Pandoc citations with brackets - checked

    Does anyone have any idea about why this might be happening?
  • 1. These two actions activate different things in Zotero/BBT; zotpick triggers CAYW, drag-and-drop uses export in the quick-format format. The translators could be configured differently for those paths.

    2. The various zotpick applescripts all request a specific format, we'd have to know which you are using.
  • Thank you for your reply.

    1. I understand, these are separate actions.

    2. I have copied the contents of zotpick-pandoc.applescript into the service that I am running through Automator. See below:

    else if zotRunning is "ready" then
    set theReference to do shell script "/usr/bin/curl 'http://localhost:23119/better-bibtex/cayw?format=pandoc' 2>/dev/null; exit 0"
  • Add &brackets=true to the URL
  • edited May 17, 2020
    Oh wait you mean the casing difference? If you execute

    curl 'http://localhost:23119/better-bibtex/cayw?format=pandoc'

    from the shell, what do you get?
  • I'm interested in fixing both, as it seems that the mixed capitalizing means it can't be recognized.

    I ran
    curl 'http://localhost:23119/better-bibtex/cayw?format=pandoc&brackets=true' from Terminal, using the example of a book by Lowy from 2001, page 12, and got the following:

    [@lowy2001, p. 12]

    This is correct.

    When I use a text editor, I can try citing the same book and page number repeatedly and I get different results one after another:

    [@lowy2001, P. 12]
    [@lOWY2001< P. 12]
    [@lowy2001< P. 12]
    [@lowy2001, p. 12}
    [@loWY2001, p. 12]


    This happens in a few different programs (DevonThink, Notes, TextMate). Here is the sequence when I used this window:

    [@lowy2001, P. 12}
    [@lowy2001, p. 12]
    [@lowy2001, p. 12]
  • edited May 17, 2020
    If curl consistently gets the correct results, it's guaranteed a problem of either the text editor or the input handler (I've seen reports of problems with Japanese input methods being active). BBTs scope of influence ends at the point where the results get delivered to the HTTP consumer - in this case, curl. Anything beyond that I can't even see, much less influence.
  • Ah, yes, I do have a Japanese input active, good to know that is a factor. I understand this is out of your control, thank you very much for your time.
  • I don't know for sure it is, but I do recall something of the kind.
  • edited May 18, 2020
    What do you use for processing these documents? Pandoc?Pandoc. I may have something for you later today.
  • All but the 4th one (with the }) is correctible. The 4th would be complex.

    Does it make any difference if you paste the information rather than perform it as keystrokes?
  • You know, I happened to update my version of Mac OS (from Sierra to Mojave) yesterday and that seems to have fixed this issue entirely.

    Now I'm going to hop over to the thread about generating live citations in Word, because if that is possible it would open up a whole new world!
Sign In or Register to comment.