Help: From Markdown to Zotero live citations

aleksmaster
Hi I am new on the forum but have been using Zotero for a few years now with the word plug-in. I want to change my workflow to write everything in Obsidian with pandoc references. Then convert the md file to a word document with live Zotero citations for editing. So @emilianoeheyns solution seems like the best one: https://retorque.re/zotero-better-bibtex/exporting/pandoc/#from-markdown-to-zotero-live-citations

But I am a complete noob to pandoc and working with md files so I could use some help.
This is the command I tried to run:
pandoc /Users/aleks/Library/Mobile\ Documents/iCloud~md~obsidian/Documents/Notes/Zotero/TestFile.md -s -o --lua-filter=/Users/aleks/Google Drive/Study/Archive/Better\ BibLaTex/zotero.lua --metadata=zotero_scannable_cite:true --metadata=zotero_client:jurism /Users/aleks/Library/Mobile\ Documents/iCloud~md~obsidian/Documents/Notes/Zotero/TestFile.docx

where I get:
[WARNING] Could not deduce format from file extension
Defaulting to html
pandoc: Drive/Study/Archive/Better BibLaTex/zotero.lua: openBinaryFile: does not exist (No such file or directory)

Could anyone please help?

Here are the directories:
Biblatex file:
/Users/aleks/Google\ Drive/Study/Archive/Better\ BibLaTex/My\ Library.bib

Pandoc filter:
/Users/aleks/Google\ Drive/Study/Archive/Better\ BibLaTex/zotero.lua

Input:
/Users/aleks/Library/Mobile\ Documents/iCloud~md~obsidian/Documents/Notes/Zotero/TestFile.md

Output (where I want a new docx file to go):
/Users/aleks/Library/Mobile\ Documents/iCloud~md~obsidian/Documents/Notes/Zotero/TestFile.docx

Content of the test md file:
This was an amazing idea [@levyContraPolitanism2020, pp.45]
In the beginnging was the deed [@hallBernardWilliamsBasic2015, pp. 24-25]
What do you usually do on your day off? [@rossiBeingRealisticDemanding2019, 45]

Thanks in advance!
  • emilianoeheyns
    You at least have not escaped all spaces in the lua filter path (I'd recommend using quotes, much more readable), and you use "-o" but do not follow it with the file to output.
  • aleksmaster
    Hey! Thanks for the reply. Sorry for the basic mistakes. I rewrote the line:
    pandoc --lua-filter="/Users/aleks/Google Drive/Study/Archive/Better BibLaTex/zotero.lua" --metadata=zotero_scannable_cite:true --metadata=zotero_client:jurism "/Users/aleks/Library/Mobile Documents/iCloud~md~obsidian/Documents/Notes/Zotero/TestFile.md" -s -o "/Users/aleks/Desktop/TestFile.docx" --to docx --standalone

    And now get a different error:
    zotero-live-citations a266bf0e6
    Error running filter /Users/aleks/Google Drive/Study/Archive/Better BibLaTex/zotero.lua:
    PandocHttpError "http://127.0.0.1:24119/better-bibtex/export/item?pandocFilterData=true&translator=json&citationKeys=rossiBeingRealisticDemanding2019,hallBernardWilliamsBasic2015,levyContraPolitanism2020" (HttpExceptionRequest Request {
    host = "127.0.0.1"
    port = 24119
    secure = False
    requestHeaders = []
    path = "/better-bibtex/export/item"
    queryString = "?pandocFilterData=true&translator=json&citationKeys=rossiBeingRealisticDemanding2019%2ChallBernardWilliamsBasic2015%2ClevyContraPolitanism2020"
    method = "GET"
    proxy = Nothing
    rawBody = False
    redirectCount = 10
    responseTimeout = ResponseTimeoutDefault
    requestVersion = HTTP/1.1
    }
    (ConnectionFailure Network.Socket.connect: : does not exist (Connection refused)))
    stack traceback:
    [C]: in function 'pandoc.mediabag.fetch'
    ...ks/Google Drive/Study/Archive/Better BibLaTex/zotero.lua:1732: in upvalue 'load_items'
    ...ks/Google Drive/Study/Archive/Better BibLaTex/zotero.lua:1742: in function 'zotero.get'
    ...ks/Google Drive/Study/Archive/Better BibLaTex/zotero.lua:1895: in function <...ks/Google Drive/Study/Archive/Better BibLaTex/zotero.lua:1878>
    (...tail calls...)

    Any more tips would be greatly appreciated :)
  • emilianoeheyns
    Is zotero running when you run this?
  • aleksmaster
    Yes, it is
  • emilianoeheyns
    Sorry, that should have been: is Juris-M running while you run this?
  • aleksmaster
    It worked! Thank you :)

    Separately, is there a way to get rid of the grey square field brackets that appear in word after conversion?
  • emilianoeheyns
    Someone who uses Word will have to help with that.
  • bwiernik
    Can you show a screenshot of what you mean? (Upload somewhere and link here)
  • parfitt.christine
    edited May 11, 2022
    Hi,

    @aleksmaster thanks for your comments and @emilianoeheyns thanks to you too. This page was the final piece of the puzzle and I'm super excited that I got md to word with scannable citations to work!! woop :)

    https://retorque.re/zotero-better-bibtex/exporting/pandoc/#from-markdown-to-zotero-live-citations.

    TIPS / PROCESS FOR THE COMPLETE BEGINNER:

    As another "noob", I have got this working but had to piece together lots of different pieces of information. I think this took me 2 full days to figure out?, and I would have appreciated a lot more of the basic information. I've added a lot of this basic information for the next person who gets stuck!

    This code worked fine for me, more details below:

    pandoc --lua-filter=zotero.lua test.md -s -o success.docx


    To get there I needed to:

    1. Download and install pandoc
    1a. read their getting started guide: https://pandoc.org/getting-started.html

    2. For a simpler code, all files need to be in the same folder. Mine was documents/pandoc-test

    3. Download the "Pandoc Filter" mentioned here: https://retorque.re/zotero-better-bibtex/exporting/pandoc/#from-markdown-to-zotero-live-citations .
    3a. Note to do this, right click download linked file. Save this file in the folder you created in step 2. It will be called zotero.lua

    4. Sort out Zotero, maybe someone can comment on which bits are necessary in a,b,c. You need the betterbibtex add on. Downloadable from here: https://retorque.re/zotero-better-bibtex/installation/
    4a. Zotero Preferences: export -- quick copy -- item format -- Better BibTeX Quick Copy: [@citekeys]
    4b. Zotero Preferences: Better BibTex -- Export -- Quick-Copy -- Format -- Pandoc citation and checked the box surround Pandoc citations with brackets
    4c. Zotero Preferences: Better BibTeX -- Citation Keys -- format [authors1]_[year] (I think this step is unnecessary)
    4d. Export Zotero Library. (Exit out of preferences) Right click My Library (or whatever your main library is called) -- Export Library -- Format Better CSL YAML.
    4e. Save this to the same folder mentioned in step 2. Mine is called MyLibrary.yaml

    5. Create a markdown file with citations (save in folder created in step 2). Mine was called test.md
    5a. The citations can be copied from zotero using quick copy (cmd shift c) and pasted in your markdown file. When pasted they will look something like this: [@SandhuEtAl_2021]
    5b. I had no need for the meta data mentioned here: https://retorque.re/zotero-better-bibtex/exporting/pandoc/#from-markdown-to-zotero-live-citations. Though I'm sure that when you understand more, this is helpful.

    6. Your folder created in step two now has three files. MyLibrary.yaml | test.md | zotero.lua

    7. choose a name for the document you want to create - I called mine success.docx

    8. Open Terminal to use Pandoc. Type each command below and push enter.
    8a. To set the location of your files - type in terminal: cw documents/pandoc-test
    8b. To check you've set the right location - type in terminal: pwd
    8c. To run the pandoc file conversion - type in terminal: pandoc --lua-filter=zotero.lua test.md -s -o success.docx

    From what I understand, the terminal command means:

    pandoc = use the pandoc program

    --lua-filter=zotero.lua = check out this code. Also known as a filter. (None of which I understand).

    test.md = take this markdown file

    -s = create a “standalone” file

    -o success.docx : output a new file called success.docx

    The below also work, and I assume are useful if you understand more.

    pandoc --lua-filter=zotero.lua --metadata=zotero_client:zotero test.md -s -o success.docx

    pandoc --lua-filter=zotero.lua --metadata=zotero_scannable_cite:true --metadata=zotero_client:zotero test.md -s -o success.docx

    Outputted Word Doc:

    The ugly grey square field brackets that appear in word after conversion seem to be bookmarks and can be removed. You need to use Visual Basic to do so.

    Things I didn't need to do:
    - to use homebrew
    - to understand what PATH meant
    - to understand what a filter was
    - to have some sort of LaTeX installed
    - to have metadata at the top of my .md file
    - to use or understand Juris M
    - to read pandoc advanced user guide for citations

    Note: I'm doing this in the most basic way and I'm sure all these other things are helpful when you know what you're doing. But they led me down confusing reading paths.
  • danifirman
    Guys, this is awesome. Thanks to @aleksmaster for bringing up this topic in the first place. Thanks also to @emilianoeheyns, especially for mentioning Jurism which I am about to have a look at further later. However, I must admit that the latest explanation and very detailed steps by @parfitt.christine really helped me a lot to understand the necessary steps. Yes, I did it as suggested, it works!

    ps: The "things I didn't need to do" by @parfitt.christine is my favourite part.
  • sergiotsar
    Thanks @parfitt.christine , your instruction is so helpful. I followed it but pandoc tells me something like "@nameofcitation is not in Zotero" and Word Plugin doesn't see citations too in final document. Can't figure what am I doing wrong.
  • emilianoeheyns
    It means that the markdown has a reference @nameofcitation (actual examples are always better than made-up examples), and that there is no item in Zotero which has that citekey.
  • sergiotsar
    @emilianoeheyns thanks! This looks like:

    zotero-live-citations f34bfc770
    could not fetch Zotero items:
    @_Newfundsallocated2022 not in Zotero
    @2020b not in Zotero
    @326FZ not in Zotero
    @pilottoContextualDeterminantsUse not in Zotero

  • emilianoeheyns
    That means that no items in Zotero could be found that have citation keys _Newfundsallocated2022, 2020b, 326FZ or pilottoContextualDeterminantsUse.
  • trickard
    Hi. Thanks for the work in summarising what can be a right headache. I am a noob too, but seem to have got far. I am trying to get from Logseq MD filed to a Word document with live citations.
    Your instructions produced a document, but it says "Do Zotero Refresh" next to the citations. I have tried exporting library as updated or not, and tried refreshing Zotero, of course. Any tips would be most welcome to get me over the last hurdle.
  • trickard
    Hi. I have figured it out - refresh the Zotero Word plugin! So, I wanted to add a little workflow I now have from Logseq to Word so that you can keep working in Logseq with active links to citation pages all the way up to final submission. Please let me know if there is an easier way!

    **ASSUMING**: Zotero and BetterBibTex, Pandoc and zotero.lua, and Word Zotero Plugin are installed and configured - see https://forums.zotero.org/discussion/93944/help-from-markdown-to-zotero-live-citations. And that you have Typora or some MD editor with find and replace.

    Best to place everything you need in one folder, the library, zotero.lua and MD file for conversion. I have a special conversion folder. Remove any spaces in file names.

    Create journal article in Logseq using double brackets around [[@citekeys]] (no parentheses) where your want your inline references to go. This is for items imported from Zotero (I use the native Logseq Zotero plugin that creates a page with metadata).

    Add page links with double brackets if this is useful for writing and the [[language]] appears natural - brackets will be sorted by the end of the process.

    Use Markdown headers, italics, bold, embedded links, etc., as you please to format the article. EXCEPT - using single or double brackets will complicate this, as they will be removed. Sorry about that!

    Article ready?

    Ensure brackets are displayed in Logseq (ctrl+c ctrl+b). Go to Document View (a plugin). This removes bullet points from view while retaining brackets and rich text.

    Select all and copy. Paste into Typora (or another MD editor with find and replace).

    Replace double brackets [[@citekey]] with single brackets [@citekey]. This is what Pandoc looks for to convert all citekeys that correspond to a Zotero item. Save.

    Open PowerShell, go the folder with your files, and input: pandoc --lua-filter=zotero.lua "MD_to_convert.md" -s -o "Preffered_output_title".docx

    Check for any errors you may need to deal with in PowerShell

    If successful, open Word document, which should be in the same folder as the files, and click "refresh" on Zotero Word plugin, selecting your citation style. If your citation style uses brackets, DO NOT create the bibliography yet.

    Find and Replace remaining [single] brackets - left over from any non citekey page links - with nothing. This is why you should only use page links that sound natural in the text.

    Create bibliography with Zotero

    Finish formatting to taste/necessity

    Save as PDF in Word if you like

    TADA! You can work in Logseq with operational @citekey links to their pages and all their connections, notes, the PDF in Zotero, etc., all the way up to a document for submission or collaboration outside Logseq.

    Jesus, why isn`t this easier!

  • aleksmaster
    edited June 1, 2023
    Hi everyone! Thank you for all the input. Has anyone tried to make this work with the Obsidian Enhancing Export plugin. There are instructions here (https://betterhumans.pub/obsidian-tutorial-for-academic-writing-87b038060522) how to use it to convert md into docx with formatted citations. I assume there is a way to do the same to export with live citations. I tried doing it myself but again I am a bit of tech dunce. This would be cool because obsidian then provides its own GUI for exports which would be a massive help for peeps like me.
  • FeralFlora
    @aleksmaster Hi! Yes, I described the setup steps for Enhancing export in the Obsidian forum: https://youtu.be/6HZ3Cc10HS8?feature=shared&t=2230
  • fulaoshi
    edited May 10, 2024
    Thanks everyone, I've written my own guide here. I think it is a simpler workflow, as it requires less steps each time you export (no need to use the terminal). If you have any suggested edits or corrections, please let me know!

    https://garden.oxus.net/technology/obsidian-to-word/

    UPDATE: With some feedback from the Better BibTeX developer and people in the Obsidian Discord, I've re-written the guide, making it even simpler (many steps were unnecessary) and also adding instructions where new users might get stuck.
Sign In or Register to comment.