Name and year import issues with new NASA ADS

pbhemphill

I've been experiencing three issues when importing from the new NASA ADS (ui.adsabs.harvard.edu) using the Firefox connector.

First, two-part last names are inconsistent. For example, https://ui.adsabs.harvard.edu/#abs/1995A&A...303..497V imports with a first author named "Kirkwijk, Van" and a second author named "H., M.," where in reality the first author is Maarten H. van Kerkwijk. Weirdly, this does not happen for the second author, Jan van Paradijs - he's imported with his first and last names in the right place.

I haven't checked extensively, but this does not happen all the time: papers by van Paradijs, Di Stefano and de Jager all import correctly, but "van der meer" imports similarly to the above, with a first author named "Meer, Van Der."

Second, dates are imported incorrectly. The van Kerkwijk et al. (1995) paper above imports with "1995-11-00" in the "date" field. This might be related to the new ADS only providing month and year, but if this is the case, why is a zero being assigned to the day? This is a minor annoyance - it means my exported bibtex files end up with "1995-11-00" in the "date" field, which in turn means my documents cite "van Kerkwijk et al. (1995-11-00)" (although that could also be blamed on a LaTeX or BibTeX stylefile...).

Third, importing from the old ADS would assign tags to an entry based on the keywords in the paper. This is no longer the case when importing from the new ADS.

I logged my attempt at importing van Kerkwijk et al. (1995); it's debug ID D752623063. I'm running Firefox 62.0, Zotero 5.0.55, and Zotero Connector 5.0.41 on Debian 9.5.

(edited to add third issue - keywords not importing)

pbhemphill

I should also mention, importing from the "old" ADS (http://adsabs.harvard.edu/abs/1995A&A...303..497V) works perfectly well - authors have their first and last names in the right place, and the date is mostly correct (it's stored as "November 1, 1995," which BibTeX seems to be able to handle reasonably well), and tags are assigned based on the keywords in the paper.

adamsmith

Yeah, I'm seeing all of these -- all look like Zotero problems to me; we'll take a look, but not sure how quickly.

pbhemphill

After reading the troubleshooting guidelines a bit more carefully, I see now that when importing from ADS 2.0, Zotero is using the "embedded metadata" generic translator, while with the older ADS it's using a dedicated NASA ADS translator. So the issue (in part, at least) is that there is no specific translator at the moment for the new ADS (although it's also due to the generic translator not handling these cases properly).

I'm fooling around with modifying the existing ADS translator to work with the new system, but I'm not much of a programmer and definitely don't know Javascript well enough to do much on my own. For now the old ADS works fine, but NASA is definitely looking to phase it out.

adamsmith

while yes, this is using the generic translator (and ADS has worked with us to make sure this is properly recognized by Zotero), the ADS metadata is in very good shape and there is no reason Zotero should import it as poorly as it does

emilianoeheyns

FWIW, BBT imports names, dates and tags correctly, but does not import the journal name, because it's exported by ui.adsabs.harvard.edu as "\aap", which is not generally defined and treated as empty by BBTs parser.

adamsmith

we're not using the bibtex at all, currently; this is automated import from html metatags (which I think we'll keep using even with a translator).

emilianoeheyns

Whoops, my bad, I thought this concerned bibtex import.

adamsmith

@pbhemphill -- on the date issue, have you actually tested this? I'm getting correct BibTex (year = 1995, month = Nov) and BibLaTex (date = 1995-11) from Zotero for this, even though it shows up with the 00.

@dstillman Zotero seems to handle the date in the (illegal) form 1995-11-00 correctly as YM, but shows ymd as parsed; that looks like a (small) bug?

adamsmith

OK, I have a fix for the author issue: https://github.com/zotero/translators/pull/1749
I don't think the date should actually cause any problems as per above.
Keywords don't import to generic translators (they're too often crappy), so this won't happen before we get a dedicated translator for the new ADS.

pbhemphill

Ah, on the date issue that's my bad, it looks like it might be more of a Better BibTeX issue. I checked more carefully, and it's only a problem for references with no month or day (e.g., https://ui.adsabs.harvard.edu/#abs/1993AIPC..280..291W/abstract) and it only shows up when I export using Better BibTeX or Better BibLaTeX, which put "yyyy-00-00" into the "year" field. The builtin BibTeX and BibLaTeX exporters don't have the same issue.

adamsmith

@emilianoeheyns I think I would consider that a bug in BBT?

emilianoeheyns

I'm not entirely sure. BBT's dateparser tries to make sense of a date and if it can't treats it as verbatim (which is why it ends up in the "year" field in the BibLaTeX export). BBT has no problem with dates that have no month or day ("1993" or "1993-01" would have worked just fine), but 1993-00-00 isn't actually a date.

I can add -00 and -00-00 as exceptions, but why would ADS export the date this way?

emilianoeheyns

5.1.1 has the workaround

pbhemphill

Works like a charm, the "year" field is populated correctly now for those cases. Thanks!

adamsmith

@pbhemphill Authors will now also import correctly from ADS. Keywords, unfortunately, won't happen as I explain above. I looked at writing a dedicated translator, but the main attraction of that would be importing multiple items and I could not make that happen given the page structure, so this is likely going to be as good as it gets for the foreseeable future.