Storing Additional Structured Data (JSON?) in Notes or Extra Field

davidamichelson

Apologies if there is a thread about this, a quick search did not turn it up. We have a need to expand our zotero records with a few extra fields specific to our bibliographic project and are wondering about safe ways to add structured data and still use the Zotero API to harvest it.

If were to put JSON encoded data into either a Note or the Extra field would the Zotero database escape everything properly or would we risk injecting unclosed JSON objects into our data if we forget a closing bracket or similar. An initial test seems to show the Zotero API escaping everything properly.

Is there a preferred way other than this? Is this something that might be different once support for JSON-LD is fully implemented.

Thanks!

adamsmith

Could you say what sort of data you're looking to embed? There are a couple of other applications (Better BibTeX, the QGIS add-on, juris-m) that use the Extra field for additional data. None of them (not quite true: see below) use JSON -- typically they just use newline separated key-value pairs.
I think in part this just because it's easier to write, but also, JSON would of course give you JSON within JSON in the API which seems like a mess to me.

fbennett

Jurism does encode data in JSON within the Extra field, but @adamsmith is right that it's messy (and fragile to boot), and I have a plan to move it to a key:val format.

davidamichelson

Thanks for the replies! We mainly would just have simple key:value type data, but it would sometimes be useful to have nested objects and other hierarchical data structures that JSON would allow.

Here are some examples of basic key/value data items which fall outside of Zotero's current data model, some are familiar as things that perhaps will be in a future Zotero release:

1. Entering DOIs for books
2. Entering page number ranges for some types which do not allow them
3. Entering series name and number when a book belongs to more than one series
4. Entering URIs for authority control
5. Entering multiple URIs or identifier numbers for citations to link them to other online bibliographies

This is a partner project of ours who is doing similar stuff: https://github.com/biblia-arabica/arabic-bible-bibliography/blob/master/documentation/BiAr_Manual.pdf

@fbennett I had not had a chance to catch up with Juris-m until now but perhaps we might follow your example. Is there a best way to contact you outside the forums?

adamsmith

Especially for the first two, see https://www.zotero.org/support/kb/item_types_and_fields#citing_fields_from_extra also for generating correct citations.

davidamichelson

@adamsmith Thanks this is exactly what we were looking for.

emilianoeheyns

BBT also supports its own JSON in the extra field.

sarcanon

Using Extra for fields not currently in Zotero looks like a promising solution for a problem I've encountered, but I am unclear on how to add multiple fields as a user of the Web API. The reference given above to https://www.zotero.org/support/kb/item_types_and_fields#citing_fields_from_extra shows an example using colon-delimited name-value pairs, each delimited with an EOL, presumably. But, in my case, these values will need to be fed to the API within a JSON object. So, instead of CSL style, can I just send them as a nested JSON object? Or is there some other method for doing this? Thank you.

emilianoeheyns

The most stable way to add data to the extra field is to add data items one per line.
I've since moved away from JSON in the extra field; my own "extra-fields" are marked with a leading "tex."

Which API do you mean? The Zotero API? That just treats the extra field as an opaque blob of UTF-8 text. It doesn't attach any meaning to "lines", that's something that the CSL processor inside Zotero does.

sarcanon

Thank you, @emilianoeheyns. By API, I do mean the Zotero Web API. It seems I don't understand well enough what the relationship is between CSL and the Zotero Web API. Perhaps if I just fiddle by trial and error I will puzzle out exactly how the Web API processes data coming in via 'extra'. Thanks again!

emilianoeheyns

By which you mean https://www.zotero.org/support/dev/web_api/v3/start ? That's doesn't do anything with citeproc. In this API, extra is just plain text. I'd recommend against putting JSON in the extra field these days. Zotero beta seems to pick out more items from the extra field and assumes one-per-line and that didn't play well with the JSON I put there (the code to parse out JSON was also pretty icky tbh)

sarcanon

Your tips are much appreciated, @emilianoeheyns. Thank you.