ISBN import with Umlaut

zuphilip · December 10, 2013

There seems to be a problem with Umlaut ö, ä, ü (maybe other characters?) in the ISBN import. The following should be reproducible:
1. Import by ISBN = 9783531149950
2. Quick search for the author "Dörre" -> no result
3. If one retype the author again by hand, then the search will work again.
There might be a problem with the character encoding? Can someone check?

adamsmith · December 10, 2013

yeah - World Cat is notorious for weird character encoding, unfortunately. I'm not quite sure what to do there & I don't understand why they can't just do proper and regular utf-8

zuphilip · December 11, 2013

I guess you are using the API of WorldCat, i.e. something like

http://xisbn.worldcat.org/webservices/xid/isbn/9783531149950?method=getMetadata&format=xml&fl=*

The output then is

<?xml version="1.0" encoding="UTF-8"?> 
<rsp xmlns="http://worldcat.org/xid/isbn/" stat="ok">
      <isbn   oclcnum="180899327 470849617 77077455" lccn="2007385274" form="BC" year="2006" lang="ger" ed="1. Aufl." title="Im Schatten der Globalisierung : Strukturpolitik, Netzwerke und Gewerkschaften in altindustriellen Regionen"  author="Klaus D&#246;rre ; Bernd R&#246;ttger. Unter Mitarb. von Birgit Beese."  publisher="VS, Verl. f&#252;r Sozialwiss."  city="Wiesbaden"    url="http://www.worldcat.org/oclc/180899327?referer=xid">9783531149950</isbn>
 
</rsp>

This means that WorldCat uses the HTML encoding for Umlaut, e.g. & # 2 4 6 (without the spaces);. Could we replace them by the utf-8 encoding?

noksagt · December 11, 2013

I believe Zotero uses the RIS that worldcat provides:
http://www.worldcat.org/search?q=isbn:9783531149950&=Search&qt=results_page&client=worldcat.org-detailed_record&page=endnotealt

In any case: isn't this just an issue of precomposed vs. decomposed characters:
https://en.wikipedia.org/wiki/Precomposed_character#Comparing_precomposed_and_decomposed_characters

adamsmith · December 11, 2013

yes to both of noksagt's points.
I find the pre/decomposed characters a bit of a mystery, but Zotero does indeed not handle the decomposed (I believe) ones elegantly, not the first time this comes up.
We could look at the API, that might work better, but would need some testing. I thought that was restricted, but that's clearly not the case.

zuphilip · December 11, 2013

You are right, the API is restricted to "1000 requests per day" http://xisbn.worldcat.org/xisbnadmin/index.htm

adamsmith · December 11, 2013

well, but 1000 requests per day - since it doesn't seem to use a key, so this would be for each individual user - wouldn't really be a problem. I don't think anyone has ever imported 1000 ISBNs into Zotero in a day, and even if, we go to the LoC first and only fall back to Worldcat.

zuphilip · December 11, 2013

Hm... there might be other restrictions hidden in their "WorldCat Affiliate Terms and Conditions" http://www.worldcat.org/ldap/terms.jsp , i.e. " ...implicitly denying faculty researchers the option of building their own tools for data extraction and analysis" cf. http://journal.code4lib.org/articles/8715 .

adamsmith · December 11, 2013

I'm not concerned about that. You can't really use Zotero to "mine and harvest a material amount" of Worldcat data. Also, I'm sure that's also forbidden using the Worldcat webpage, which we currently use for ISBNs.
I think switching this is the right thing to do as long as data quality doesn't suffer. Among other things should also make it faster and use up less bandwidth.

adamsmith · March 25, 2014

I've looked at this in some detail and the API's xml data isn't usable. The authors are, unfortunately, formatted to irregularly, so we'll keep this as is. That also means we're stuck with the Worldcat decomposed characted issue. sigh.

aurimas · April 8, 2014

That's a shame about the authors, but we could probably use the API to supplement some metadata (e.g. publisher info). The API appears to be quite fast, so there should be minimal overhead.

adamsmith · April 8, 2014

are you thinking for Worldcat in general or just for the search translator?
Another thing that I found odd with the xisbn is that the results are at times different - as in a different copy of the book - from the ISBN search on WorldCat

aurimas · April 8, 2014

Hmmm, I'll have to take a look at some ISBNs and see what we can do. I can't say this is surprising, since an ISBN search on WorldCat sometimes returns multiple results and the xisbn API appears to only return one. We can use OCLC ID when using the web translator, which should return the data for that exact item. For search, we can go with ISBN.

Edit: or, since we're only supplementing, we can just go with OCLC. Or vice versa... ugh. I'll play around with this. If you happen to stumble on some ISBNs with inconsistencies between API and their web page, or where the authors are messed up, post them here.

zuphilip · April 9, 2014

The example in the documentation of xISBN is 0596002815:
http://xisbn.worldcat.org/webservices/xid/isbn/0596002815?method=getMetadata&format=xml&fl=*
It will list 19 oclcnum and the metadata shown are from the smallest (?) oclcnum, cf. url.

A search for the ISBN 0596002815 in OpenWorldCat returns 3 entries, where each consists of 2, 104, respectively 1 editions/formats; thus there are a total of 107 editions/formats in OpenWorldCat:
http://www.worldcat.org/search?q=bn:0596002815&qt=advanced&dblist=638

adamsmith · April 9, 2014

(that example also shows the problem with authors. E.g. here it returns "by Mark Lutz and David Ascher." - but that's in no way consistent and may vary by language etc.)

zuphilip · April 9, 2014

Yes, I can confirm the language dependency of the author-statement, see e.g.
http://xisbn.worldcat.org/webservices/xid/isbn/978-3-939352-01-3?method=getMetadata&format=xml&fl=*
The author-statement is here "von Birgit Grüb. [Universität Mannheim]". Is this actually proper UTF-8? I wouldn't expect something like &# 252; ...

adamsmith · April 9, 2014

The charset is correct - XML can handle regular UTF-8 and doesn't require any escaping except for the handful of protected characters (< >\"&) that's actually the nice thing about this & why aurimas would have liked to switch to it more generally.

zuphilip · April 9, 2014

But it is escaped (added manually a space to show it):
author="von Birgit Gr&# 252;b. [Universit&# 228;t Mannheim]"
Thus, the xml will look good in a browser but not in a text editor. But maybe it is not a big problem...

zuphilip · April 9, 2014

WorldCat has also some (experimental?) RDF data, e.g.

http://experiment.worldcat.org/oclc/177669176.rdf
http://experiment.worldcat.org/oclc/162462118.rdf
http://worldcat.org/oclc/866798552.rdf

We need the oclc number, but this might be useful... How about a new search translator given a oclc number (maybe as a first step)?

adamsmith · April 9, 2014

sorry, I misread you as saying you _would_ expect escaping - you're right, of course, that shouldn't be there in utf-8 text.

aurimas · April 9, 2014

UTF-8 doesn't disallow character references, so it's not wrong, but it's also unnecessary. Shouldn't be an issue if you use DOMParser. I'll experiment with the RDF (I was actually looking for the RDF API, but got distracted when I found the xISBN API instead)

aurimas · April 10, 2014

So for the RDF data, looks like we'll have to implement schema.org vocabulary in our RDF translator first (if not for OWC, it will probably come up at some point in the future anyway)

For the xID API, the API seems to not work so well with OCLC numbers. http://xisbn.worldcat.org/webservices/xid/oclcnum/154684429?method=getMetadata&format=xml&fl=* returns essentially no metadata and http://xisbn.worldcat.org/webservices/xid/oclcnum/864085353?method=getMetadata&format=xml&fl=* (the article we discussed on GitHub) returns a server error.

I'm giving up on this for now. Will notify WorldCat about the errors though.