CSL Json Export

joaoxxx

Hi there,

Is there anyway to edit the CSL Json export file so as to get my Zotero tags in there?

I'm using the Json file from my Zotero database (+2000 entries) as a feed for an online dataTables render. I would like to just export the file to an FTP and have the database update.

adamsmith

@Rintze @fbennett CSL does have a keyword variable, so we could do this legally. It's not used in any style, afaik, so it wouldn't break anything. Any objections/thoughts? Frank, citeproc-js just expects a string there, I assume?

joaoxxx

FYI: at the moment I'm using the XML export, which I edited (the "Endnotes xml" file), and then I'm converting to JSON. I would like to skip this middle step.

adamsmith

I think to customize this, you should be able to hook into the
doExport function: https://github.com/zotero/translators/blob/master/CSL JSON.js#L68

You already have the item (which contains item.tags) and you have the CSL JSON created by ZU.itemToCSLJSON(item), so you'd just need to add the tags into the JSON in the format you'd like (likely an array?) before pushing the item to the data array.

We can see if we'd want to modify the itemToCSLJSON function more generally -- that's my question above -- but a) that's going to take longer and b) it'd almost certainly include the tags as a (comma or semicolon delimited) string, which isn't ideal.

joaoxxx

I think I understood the gist of it, but the truth is I have no idea how to code js to get that done.

adamsmith

I'd have to test this myself, but my starting point would be something like
function doExport() {
var item, data = [];
while(item = Z.nextItem()) {
var itemJSON = ZU.itemToCSLJSON(item);
itemJSON = itemJSON.push(item.tags);
data.push(itemJSON);
}
Z.write(JSON.stringify(data, null, "\t"));
}

joaoxxx

This doesn't work. There's a javascript error during the CSL Json output.

adamsmith

yeah, as I said, it was untested code, more designed as an outline than working code. I'm not going to have time to debug this, sorry. We can see what people say about putting the functionality in the upstream function.

joaoxxx

Yeah, I understand. But let me tell you, javascript/jquery datatables fed by Json arrays are very useful at the moment, and I think it would really benefit Zotero to work in tandem with them with a simple (but thorough) export. My implementation already works rather well but tags would be useful in order to have simple category filters (or other kinds of metadata tricks).

adamsmith

Completely agree. We're working on a full JSON export in JSON-LD format based (mostly or entirely) on schema.org. No ETA, but hopefully going to happen first half of this year.

zuphilip

Try this code instead (not tested):
function doExport() {
var item, data = [];
while(item = Z.nextItem()) {
var itemJSON = ZU.itemToCSLJSON(item);
itemJSON.keyword = item.tags.join(", ");
data.push(itemJSON);
}
Z.write(JSON.stringify(data, null, "\t"));
}

joaoxxx

It partly works! But it returns [object Object].

zuphilip

itemJSON.keyword = item.tags.map(o => o.tag).join(", ");

joaoxxx

That is perfect, my friend. Thank you very much!

Rintze

CSL does have a keyword variable, ..., citeproc-js just expects a string there, I assume?

Yes. See https://github.com/citation-style-language/schema/blob/v1.0.1/csl-data.json#L350

david_lindemann

Here is a version of the CSL JSON translator that exports attachments, notes, tags, and "extra": https://github.com/dlindem/zotero_lexbib_rdf/blob/af0379536a0fec316df62ceb2ab745e650b95f90/zotero/LexBib JSON.js#L109

emilianoeheyns

These things can also be done with BBT CSL JSON, with a postscript. Adding attachments by default was discussed here, where it was suggested that someone pitch the idea on xbiblio-devel. If it would get agreement there, I see no issue in adding it to BBT CSL JSON exports.

kmlawson

Hello everyone. Back in 2021, this forum posting was extremely useful to me as I was looking for a way to get JSON export with tags. I was able to do this with a modification of the CSL JSON.js translator, using some combination of the suggestions above. I created a new "JSON with Tags.js" translator for this purpose which served me well.

The use for this ability is as follows: A friend of mine put together a nice little TypeScript script which loads a JSON list of exported Zotero items in a webpage and turns its tags into buttons (https://github.com/henriksen/bookie). When clicked, the buttons display a list of sources with URLs linking to the titles.

This was created for the purpose of sharing large lists of open access historical books/documents that are often available on archive.org. See an example here: https://froginawell.net/frog/primary-sources-the-malay-world/#books

Yesterday, I discover that at some point the modified JSON export with Tags translator, using one of the methods from above was broken. I believe this may have been due to some kind of change in the way translators are done. I was also not able to get things up and running with BBT as an alternative. Nor was david_lindemann's link above working for me anymore.

I ended up updating the modified script to bring in changed code in the latest CSL JSON.js I could find, and it seems to be functioning again after fiddling with the doExport() function again. Sharing the modified translator below in case a) this is useful to anyone else or b) this is not a good way to go about it, in which case, happy to accept suggestions for alternatives. For now, happy it is up and working again.

Any reason why JSON export can't include tags by default? Something to do with CSL standards?

CSL JSON with tags:

https://gist.github.com/kmlawson/0f3eb75f462bca340897a4d1daf2c708

adamsmith

Any reason why JSON export can't include tags by default? Something to do with CSL standards?

Basically, yes. CSL does have a keyword category, but CSL JSON has a defined schema where that's just a single string, which isn't a great choice for tags, and Zotero has never supported it. Most problematically, adding tags as an array might break citeprocs.

BBT does support various JSON export formats, though, and @emilianoeheyns might have suggestions of which to use?

kmlawson

Thanks! I think I'll stick with this hack for the time being. BBT export uses different names for fields and handles dates in the JSON different, so using that would mean I would have to go and tweak the bookie script as well. Fortunately, this is now working again!

emilianoeheyns

BBT export uses different names for fields and handles dates in the JSON different

Different in what way? Most of the BBT CSL generation is done by Zotero itself, I only augment here and there, and CSL-JSON dates are in a standard format, so I don't know what "handles dates in the JSON different" means? Perhaps that I parse them differently, but export them in the same format? If you find an instance where Zotero does it different from BBT and you think the Zotero variant is better, I'd love to hear about it. Can't fix what I don't know.

bwiernik

In general, I would say that any differences between Zotero and BBT CSL JSON export, the BBT version is better

kmlawson

No critique of the wonderful BBT at all! I have a script which was created by someone else: it worked with a hack of CSL JSON in 2021, but doesn’t work with BBT export. I updated the hack to work with newer 2023 Zotero. I posted update here to help others who may see this post and see that the old solution here (not BBT, which was never explained in original, just mentioned as possible alternative) no longer works.

I was just here to report a *solution* for benefit of others, it was not to report a problem! :-)

CSL JSON saves dates differently than BBT, has “author” instead of “creators” etc., all of this is fine and perhaps better, but it just means it won’t work out of box for me with Bookie code written back in 2021 by someone else and it is far easier for an amateur like me to mess with one thing than several!

Thanks again to everyone: repeat - nothing to see here unless you are interested in my very one-off special solution! I was just trying to be helpful.

If you want JSON and you want tags with your JSON you can add a single translator like the one above, or you can install the large, powerful, and useful collection of the tools of bbt, which adds a bit of time to every launch of the Zotero app (but worth it for what it offers).

Whatever works for you!

emilianoeheyns

I'd still very much like to see a sample of such differences.

"CSL JSON saves dates differently than BBT", if that means ""CSL JSON saves dates differently than BBT CSL JSON", means that one of them is off-standard, as the way dates are represented in CSL JSON is fixed. They can't be structurally different and both be valid, and if BBT CSL JSON is out of spec, I'd love to fix that.

emilianoeheyns

Wait -- you are saving in BetterBibTeX JSON, not Better CSL JSON. These two are entirely unrelated formats.

kmlawson

Hi @emilianoeheyns,

Thanks, glad you understood what was going on. I wish you luck in your debugging!

Here is CSL JSON example:
https://pastebin.com/UMVL9SYs

Here is BetterBitTeX JSON example:
https://pastebin.com/V1dG3nku

That includes tags (the topic of this forum thread) but, as I mentioned above, does not work with the Bookie code unmodified because of differences with this format without further work.

Here is an example with your Better CSL JSON:
https://pastebin.com/ugXGQuQh

Which is not helpful here since it doesn't include tags (defeating the purpose of using BBT). This is why this is not the export format I'm referring to as it offers no advantage over the original. You have a post above with something to do with postscripts, which looks like an interesting alternative solution but not one that was explained here and I couldn't make much sense of the github thread or the documentation pages for this.

The absolute simplest solution for my specific need was to duplicate and modify the translator again and add

if (item.tags) {
cslItem.tags = item.tags;
}

That saves me messing with Bookie. That is pretty much the modification to that file I linked above.

Thanks so much for your hard work on BetterBibTex - a wonderful resource.

emilianoeheyns

I wish you luck in your debugging! ... Here is BetterBitTeX JSON example

There's nothing to debug. BetterBitTeX JSON is not relevant to the discussion.

The absolute simplest solution for my specific need was to duplicate and modify the translator again and add

You can do that with https://retorque.re/zotero-better-bibtex/exporting/scripting/, which won't be overwritten on updates.

kmlawson

Ok, I think I’m a bit tired now. You failed to understand the problem, you never provided an actual solution inside this thread (which I have, hack though it is), and you forward people to massive pages that don’t actually include the concrete solution for them (instead you offer more work for people who may not be coders as they try to decipher things). You fail to see the “relevance” of a format that is the only one you have that includes tags on export, when that is, in fact, the whole point of this thread (exporting tags in JSON, CSL is not the topic). In short, you have not been helpful here, at all. I think I’m done. If you have an actual solution to the problem you are welcome to post it here. I, however, have no more time and will move on to other tasks.

emilianoeheyns

Suit yourself. I failed initially to understand the problem because you brought in irrelevant information; you said you had a script that only understands CSL JSON, and if that's so, then BBT JSON is out. Then you failed to understand my questions and that is somehow a knock against me. I pointed at a structural solution, which you reject, which is fine. It was meant to probe whether you were open to this solution. If yes, then I usually write these postscripts for users.i don't expect users to know this level of detail m

And BBT json is absolutely not the only format to export tags (RDF does too for example).