BibTeX export/Import and Chinese text in UTF-8

moncur_d

Here at the Australian National University, we have a facility under development to allow researchers to poll a number of data sources and generate a unified list of their publications, published data sets and conference papers.

The aim behind this is to increase portability and allow people to take their work with them.

We have chosen BibTeX as an export format as Zotero, Endnote and Mendeley can all accept BibTeX as an import format.

We have however come unstuck as to what to do about Chinese UTF-8 text as BibTex, by default does not have a standard method for handling UTF-8.

Pragmatically, as we're using BibTeX as a lowest common denominator data migration format all we need do is what our migration targets do.

Is there a document describing how Zotero reads utf-8 encodings from BibTeX formatted files ?

dstillman

I'm not sure what you mean by "utf-8 encodings". UTF-8 is the encoding, and Zotero supports UTF-8 for BibTeX import/export (and for all other file formats). My understanding is that not all BibTeX tools support Unicode (or at least didn't a few years ago) but as long as you generate a valid UTF-8 file it should work as you would expect.

What's the problem, exactly?

moncur_d

The problem is that bibtex is pre unicode and does not have a standard way of representing unicode characters. In fact it's only really happy with single byte characters using the escape conventions for characters with the top bit set.

For data interchange purposes this means that a file generated from application A may not load correctly into application B, or at least give odd results.

In the absence of a standard method what we need to do is generate files in a way that is compatible with our candidate target applications, ie if I give you Japanese text in a BibTeX document how do you expect the non ASCII characters to be represented ?

dstillman

I'm not sure what you answer you're looking for from us. The Zotero BibTeX translator has a mapping table of escape sequences for many extended (but of course not Chinese) characters that's used on import and if you set the export character set to ASCII or similar, but Zotero happily imports and exports UTF-8 in all supported formats, BibTeX included. And, as I say above, my (uninformed) understanding is that there are in fact some BibTeX tools that can handle UTF-8. If you're using tools that can't, I'm not sure what to tell you. Use more modern tools. This works fine on Zotero's end.

noksagt

Zotero will import UTF-8 encoded files. That's the only way to get CJK in.

While the 'bibtex' program is not fully compatible with UTF-8, there are plenty of other clients that can use a UTF-8 BibTeX file (including biber/biblatex for the toolchain of actually using this in a LaTeX document). I think (but have not checked) that Mendeley is one. I had thought that Endnote couldn't import BibTeX (certainly, in earlier versions, it was necessary to convert the file to a format it could import).

moncur_d

That's the bit of the jigsaw we need - that you don't do anything funky with UTF-8 input ...