Convert plaintext citekey to "live" MS Word citation

JasperGo

Hello everyone,

is there a way to convert the citekeys (or other plaintext references) of Better BibTeX into live citations fields in MS Word? I'm running into some collaboration issues now and am sorely missing a feature like this.
The existing discussions around the RTF scan functionality made me a little pessimistic already, but perhaps there was some development or add-on for this in the meantime.
Thanks!

emilianoeheyns

Can you describe the workflow you had in mind, even if just roughly? If you're using BBT, you're probably using LaTeX, and I don't know of a robust way to make Word files out of a LaTeX source, which would be the first hurdle to clear.

JasperGo

Sure. I'm actually not using LaTeX, I just installed BBT to get unique citekeys for each entry.

I'm drafting in a MD processor and want to include a citekey or plaintext ref which I want to have converted into a live MS Word citation when I copy the draft into word. Similarly, when collaborators use a different reference manager, we would agree on one citekey format and use that between files so I can also copy those drafts into MS Word to get the live citation to format with the Zotero MS Word plugin. I was hoping BBT came with some sort of such functionality as a unique citekey for each entry would be the first requirement.

emilianoeheyns

If you use a MD processor, it's not at all unlikely that it's using pandoc under the hood. If that's the case, a filter might be able to accomplish this. I could take a stab at this as a proof of concept. Do you have a sample?

JasperGo

Unfortunately not a standalone MD processor, but the RoamResearch service, sorry. I suppose that means that there is no such functionality via some addon?

emilianoeheyns

How would that addon interact with RoamResearch? With pandoc and a filter, it can be done I think, as the AST seems to have enough information to do it, but if all your notes live inside RoamResearch, at what point would it become an MS Word file with live citations?

bwiernik

Yeah, the best approach here would be to export your notes from RoamResearch as markdown and then use pandoc to convert them to Word, rather than exporting directly to Word.

Are you by chance an R user? If so, RStudio comes with pandoc built in, so you can convert Md files with no additional setup. Otherwise, I can walk you through the steps of using pandoc.

JasperGo

@emilianoeheyns: I didn't think of an interaction with RoamResearch, I thought of something that takes plaintext (in or from whatever source) and converts citekeys in this plaintext into MS Word citation fields.

@bwiernik: Ah, that would be an option. Yes, I am an R user and going through the Md -> pandoc -> docx route would work, I suppose. I assume that I have to set up pandoc in a way that it recognises the BBT citekeys and turns them into Word citation fields? Anyways, thank you for the suggestion. I'll look into that!

emilianoeheyns

That's sort of what I was suggesting. If you can get markdown out with pandoc-compatible references, you can use that chain to make a live Word document. I think. I'll run some experiments.

bwiernik

Look at this package: https://github.com/paleolimbot/rbbt

It connects to BBT in Zotero. Soon it will have a function to scan your document for citekeys, then generate a bibliography (CSL JSON) file for the cited items

Otherwise, you can export your whole Zotero library to “Better CSL JSON”, then put this file into your YAML header at the top of your document.

bwiernik

See the citations section of this page for a walkthrough: https://wiernik-datasci-guide.netlify.app/citations-with-pandoc-and-zotero.html

djross3

(I'm hesitant to add this because you're already looking into a number of options I haven't used, but I had a similar question recently and at least for my uses I was able to convert plain text into usable MS Word citations indirectly via ODF as an intermediate step. It's not particularly convenient, but it can work, so I'm not sure if that would really apply in your case. Here's the info on what I did: https://forums.zotero.org/discussion/81890/automatically-enter-multiple-citations-e-g-whole-collection-in-word)

emilianoeheyns

Hey, if it works, it works. The docx format is pretty funky, so if I don't need to touch it that's just fine by me.

djross3

Again, I was doing this somewhat differently than suggested/discussed here, and it's a roundabout, awkward process. But, yes, it works. I'm not sure if the format would easily connect to the OP's needs (or if then it would just add one more step to the indirect workflow), but it might work out.

JasperGo

Thanks a bunch for all the input. I'll try it tomorrow!

emilianoeheyns

I have a pandoc lua filter at https://retorque.re/zotero-better-bibtex/exporting/pandoc/ that successfully (as far as I can tell) converts to RTF/ODF scan compatible odt, but I have not yet been able to create a word file with live markers. It *should* work as far as I can tell, but word deems the file corrupt, and xmllint says it has errors. No idea why yet.

The RTF/ODF scan part of the filter should work with the current BBT release, the Word part of the filter needs a patched BBT which I have on a branch.

emilianoeheyns

?!?! if I open the docx file in LibreOffice, save it as-is, and open it in Word, the live markers are present.

yay I guess?

emilianoeheyns

Filter updated on https://retorque.re/zotero-better-bibtex/exporting/pandoc/

emilianoeheyns

Found the error.

emilianoeheyns

And fixed. Live references from markdown are go; I'll include the patch and the docs for it in the next release.

bwiernik

Woah!

emilianoeheyns

Yeah :)

It should be fairly easy to make this work with RMarkdown. RMarkdown uses pandoc, and as it's a Lua filter, it should work as-is with any pandoc version > 5.2 (I think). It will filter out citations for which it can find all keys; citations where one or more keys cannot be found are passed through, as there are filters that (ab)use non-existing references to create cross-references within the doc. The filter will complain on stdout about mising keys -- it would be easy to spit those out as eg json so that an automated toolchain could do something with them.

emilianoeheyns

I'm still tinkering with the interaction between the filter and BBT a bit, but it works as-is, it's just a bit of cleanup now.

emilianoeheyns

Instructions + new filter on https://retorque.re/zotero-better-bibtex/citing/pandoc/

mcv2

Asking a very basic question here in the hope that it might be of use not just to me, but someone else in the future with limited technical skills who is trying to set this up.

I am getting the following error:

/Users/MYNAME/.pandoc/filters/zotero.lua:168:
module 'lunajson.sax' not found:
no file 'lunajson.sax.lua' in pandocs' datadir
no field package.preload['lunajson.sax']
no file '/usr/local/share/lua/5.3/lunajson/sax.lua'
no file '/usr/local/share/lua/5.3/lunajson/sax/init.lua'

(...and so on, various other missing files)

I was looking at this github page to try to figure out how to use (or 'install') a lua filter, and thought that putting it in the .pandoc/filters folder would do the trick, but it seems no. (FWIW, I created the .pandoc and the /filters folder myself, but that seems to be par for the course.)

For reference, the code I am running is:

/usr/local/bin/pandoc -s --lua-filter=zotero.lua
--filter=/usr/local/bin/pandoc-citeproc
--bibliography /zotero/my_library.bib
--csl /Users/MYNAME/.pandoc/chicago-fullnote-bibliography-with-ibid.csl
--reference-doc /Users/MYNAME/.pandoc/reference.docx -f markdown+smart
-t docx -o /Users/MYNAME/Desktop/"$title".docx "$@"

emilianoeheyns

That shouldn't be necessary, it should have been bundled. I'll take a look.

emilianoeheyns

pandoc-citeproc wouldn't do anything in this pipeline as zotero.lua will have replaced all citations it could act on; likewise, the --bibliography and --csl won't do anything.

mcv2

I got the same error when I ran:

/usr/local/bin/pandoc -s
--lua-filter=zotero.lua
--reference-doc /Users/MYNAME/.pandoc/reference.docx
-f markdown+smart -t docx -o /Users/MYNAME/Desktop/"$title".docx "$@"

emilianoeheyns

Does https://gist.github.com/76d81cb264339a69ab88d39ecb75fabb work for you?

mcv2

Yes, it does!! Thank you :)

emilianoeheyns

Super, thanks. This version is up on the regular site now.