Export Bibliography but Preserve Hierarchy of Sub-folders?

kmlawson

Thanks as always for superb support and a wonderful application. Zotero is such a life saver.

My question:

Is there a way to export a bibliography that preserves the sub-folders in the result, perhaps marked by a subheading or saved as separate files, or whatever? Or a plugin that does something similar?

Let us say I have a folder with 20 sub-folders. I often like to share a list of sources broken up into sub-sections represented by folders, and would love for this to be preserved, rather than having them all end up in a single unbroken alphabetical list on export. I can currently accomplish the same result by manually by exporting each of the sub-folders separately and then merging the result with some headers to indicate each sub-topic.

My thanks in advance,
Konrad

adamsmith

There's nothing out of the box, no. Citation styles don't have access to collection information and translators can't access citeproc to create citations.

I'm wondering if you could somehow rig something together with better bibtex and a pandoc-based script? Maybe @emilianoeheyns has an idea at least conceptually

emilianoeheyns

I have done something similar, but I did so using biblatex. I used a BBT postscript to turn the collections into tags on the items, and then made a sectioned bibliography using defbibfilter/filter= and keyword= on printbibliography.

It is technically possible to run citeproc in translators by webpack'ing citeproc into a translator.

You can export all your collections into separate files by creating a BBT auto-export and ticking the recursive option after creation. After that it shouldn't be hard to create a script to create a pandoc source that creates a source for each; you can create standalone word documents for each and then use Word's multi-document feature to assemble them. You can also export your items to a BBT CSL file, and use a BBT postscript to add the collection info to the CSL items, and then use a script to break the one CSL file to multiple based on the section info you put in there, create a pandoc source document for each, etc etc.

kmlawson

Dear @adamsmith and @emilianoeheyns,

Thank you kindly both for your replies. I haven't used pandoc with bibtext before, but see it in the docs. When you say "BBT auto-export" do you mean an "auto-export" using the "zotero-better-bibtex" add-on?

If so, I will give this a shot (installing the add-on now). If indeed, in one go, all sub-folders can be exported into separate bibtext files (I'll have to see what it names those files so hopefully there is soom hook to the collection names), and these are understood by pandoc, then, as you say, the matter would simply be a script that handles the post-export loop through the files to convert and merge them together.

My thanks again,

Konrad

emilianoeheyns

Personally I'd use pandoc with CSL rather than bibtex (no t at the end) unless you're producing PDF through latex, and my guess would be you don't have a latex toolstack installed. If you want anything else but that (eg you want Word), I'd use CSL. Fortunately BBT outputs that too.

Indeed, when I say BBT I mean zotero-better-bibtex. The latter is a bit of a mouthful, and the fact that it covers more scenarios than Zotero's stock bibtex import/export translators (at the cost of speed and complexity) is now overshadowed by the improved workflow it provides for text-based authoring with Zotero. Also, as mentioned, it no longer just exports bibtex, and the name zotero-better-bibtex-and-csl-and-oh-yeah-citekeys-and-doc-migration-oh-wait-autoexport-and-reference-picker-and-and seemed a bit excessive.

To get all collections in one go, you need to make an export using one of the translators whose name starts with "Better " of your library or a collection, make sure to check the "keep updated" checkbox, then go into the preferences, and check "recursive" on the auto-export config it created. BBT will export each collection as a file named after the full path to the collection from the point of export (library or collection). If you have duplicate collection names under one parent (which Zotero allows, for good reasons), it is indeterminate which will overwrite which, so just don't do that.

If your library is large, you may want to set export to "on idle" rather than "on change" in the preferences.

kmlawson

Thanks so much for your reply @emilianoeheyns. I'm sure I can get a wee script together to convert an exported collection of formated citations from, say a collection of markdown files into a one large markdown file with subsections. I have lost an afternoon, however, trying to figure out how to go from better-bibtex or better-csl json exports to a formatted collection of files. Ultimately, I was able to get to markdown by convering from bibtex to -t markdown_strict (I kept getting stuck with the fact that 'markdown' output produces some kind of halfway format, and going to latex first and then to markdown added a pre[] section with citation id)

You have already helped a lot, but if you have any thoughts on what I'm doing wrong with the CSL (though better bibtex export seems to work) I would appreciate it. The problem seems to be my use of Pandoc (2.12, OS X).

Pandoc conversion of the CSL json export (to any format) doesn't seem to work for me:

pandoc test.json --citeproc -t [insert any format here] -o test.[any format]

produces:

JSON parse error: Error in $: mempty
(however, the exported CSL seems to be valid, according to https://jsonlint.com/)

I noticed the better bibtex website (I'm guessing that is you judging by github commits to repo?) also suggested using CSL (https://retorque.re/zotero-better-bibtex/exporting/pandoc/), shame I can't get it to work.

Pandoc conversion of better-bibtex to markdown_strict seems to work fine and I'll work on a script from that to assemble the converted files into one large markdown file:

pandoc test.bib -s --citeproc -t markdown_strict -o test.md

This works, so I need merely create a shell or python script to concatenate the output of this after converting all sub-folder exports.

emilianoeheyns

Can you put up the files somewhere so I can have a go (and correct, I'm retorquere in most places)

kmlawson

Sure, here is a "Better CSL JSON" export of a small sample collection of 22 books and articles:

https://pastebin.com/KRWthHFL

Here is a test command on that file which produces the JSON parse error:

pandoc export-test.json --citeproc -t plain -o export.txt

(2.12, OS X, Better BibTex 5.2.125)

I have a clunky shell script that does what I wanted here, just in case it is of use to anyone else:

https://gist.github.com/kmlawson/3bdfd917ef55969faec7768c140dcdaa

There must be many better ways to do this, or at least better scripts. Gladly accept any improvements or easier ways to go about this.

emilianoeheyns

What is the source document?

kmlawson

of the pastebin JSON? It is is a single collection in my (standalone) zotero app, which I have right-clicked and chosen "export collection" then chosen Better CSL Json export.

emilianoeheyns

I'm not sure what your command line means -- here is what I would use

pandoc -s --bibliography csl-json-file.json --citeproc --csl csl-style.csl source-document -o output-document

emilianoeheyns

of the pastebin JSON?

No, I understand that that is your bibliography. You want a rendered bibliography, right? You need to have a source document that cites these sources.

emilianoeheyns

This will take all the csl-json files on the command line, generate a doc on the fly for it, and render to html

#!/usr/bin/python3

import json
import sys
import os
import shlex

for src in sys.argv[1:]:
if os.path.splitext(src)[1] == '.json':
with open(src) as f:
bib = json.load(f)
md = src.replace('.json', '.md')
html = src.replace('.json', '.html')
with open(md, 'w') as f:
for entry in bib:
f.write('@' + entry['id'] + '\n')
print('pandoc -s --bibliography ' + shlex.quote(src) + ' --citeproc --csl apa.csl ' + shlex.quote(md) + ' -o ' + shlex.quote(html))


invoke as ./render.py export-test.json export-test-2.json or ./render.py export-test.json export-test-2.json | bash if you trust the output.

emilianoeheyns

But if you just want a rendered bibliography for all each export, you can do that directly in python. No need to involve pandoc when you don't actually have a source doc to render.

emilianoeheyns

(WARNING: the above script would blithely overwrite export-test.md, export-test.html, export-test-2.md and export-test-2.html. It does no checking to see if these exists)

kmlawson

Thanks so much for all these suggestions and the python script. It looks like that scripts works nicely, producing a list of citations, which I assume can be used to create a bibliography (but this adds a step - my script above doesn't require it).

I probably just didn't make things clear in the beginning. The end goal is just to easily share a list of sub-divided readings with someone.

I have a list of books in a collection, that collection has many sub collections, each with many books. I want another person (who doesn't have zotero) to get a document which contains any kind of list of info of the books in that collection divided into sub-sections. Zotero is amazing but doesn't make it easy to simply share a list of suggested readings to students and friends in a way that divides a collection up by its sub-collections. If I have a student interested in urban history and I have 1000 books and articles in a folder with lots of sub-folders according to city/theme/approach etc., there is no an easy way to share those reading suggestions with students without manually exporting each sub-collection, one at a time, or giving them an undivided dump of all 1000 entries with no division by sub-topic. While doing this once is not difficult, my collections are evolving all the time, and I'm updating them often, and sharing them often with students and colleagues. The citation format doesn't really matter, nor even the final outputted document type, all that is key is that some list of titles of works are listed in a single document, but divided by sub-topics. Ideally, one could right click on a folder "export" it, and produce a single list of readings that include sub-headings.

It is of course a very niche request so I knew there would have to be some fiddling. Thanks to your wonderful "better bibtex" plugin, I now have an easy (if multi-step) process which perfectly solves this desired task, if anyone else happens to have the same goal as me of sharing reading suggestions with people that are divided by sub-topics.

Here is what ultimately worked:

1) Install Better bibTeX. Export the folder (that has sub-folders) with Better bibTeX, checking "keep update" - export to its own empty folder to be on safe side.
2) Then go to Zotero preferences, switch to Better bibTeX "automatic export" settings and turn on "Export all child collections" - clicking that will re-export the folder again but this time it will also export every sub-folder as a separate bibTeX file.
3) navigate to the folder on the command line, which should now contain a list of .bib files there ready to be merged. After giving it execute permissions, run the gist (https://gist.github.com/kmlawson/3bdfd917ef55969faec7768c140dcdaa) from within the same folder. It will convert all bib files in the folder to markdown bibliographies in the default Chicago note formate (without need for any other "citation" source document or specifying any csl). It will then merge all those markdown files together. You can optionally uncomment two lines in there if you also want it to clean up by deleting all the exported .bib files and other .md files leaving only the final merged result.

At the end of the proces you will have a list of all sources in a collection and all its sub-collections, but these sub-divisions having headers dividing them, based on the collection names. Though clunky, that works great for now!

emilianoeheyns

If it works, it works.

kmlawson

Thanks again!

floriand

Hello, for those stumbling on this thread and struggling a bit to make it work on windows: i made a slightly adapted version that works inside a conda-environment on windows here: https://gist.github.com/floriandierickx/9c93f73c0cc7bca5b56c9a77f39e1916

Thanks to both script-makers!