as for fields:
Reference --> Loc. in Archive
Description --> Title
Date --> Date
Held by --> Archive
Legal status -->Rights
Access conditions --> ??? (maybe add to rights?)
Publication note: --> note (or abstract?)

Won't happen immediately but seems well worthwhile doing.

working with XML would be much better in general, yes - unfortunately their API is (currently?) restricted http://discovery.nationalarchives.gov.uk/SearchUI/api.htm
so I can't see that XML, nor can we use it for Zotero, which really is too bad.
If you think the XML provides better guidance for field mapping, you can post it to a public gist at gist.github.com (where it'll be easier to read than here) and link to it.
Any comments on the mapping above?

It seems there might be a problem with the character set (certainly the fact that it is genuinely utf-16 seems a little implausible to me!).

Let me explain how it works. There are 7 levels to Discovery. Level 1 is a department like INF - the Ministry of Information. Level 6 is where an piece of art like INF 3/108 is sitting and level 7 would be an item level description - like COPY 1/400/23 (which is a photo in a box where the box is COPY 1/400).

In Discovery references must be formed by working up the <ParentIAID> chain and gluing together the <Reference>s.

The tricky thing is that we don't want all of them. We are only interested in levels 1 ("INF"), 3 ("3") and 6 ("108"). So although we need to follow the chain of <ParentIAID>s upwards, we need to throw away levels 2, 4 and 5. These tell us archivally useful things about the object but they are not part of forming the reference and if we include their <Reference>s, the value we generate will be a nonsense and generate an error when we come to look it up.

All clear as mud so far?

If we follow this rule there may be some issues since this is not completely uniform but there is probably a list of the rulebreakers which could be provided.

Does that make sense?

Jo

Yes, as an archivist that kind of abstraction is familiar from elsewhere. To clarify, are you saying the Discovery Reference (in the Search UI, not the XML <Reference> tag, hmm this is confusing!) is nearly always formed from a combination of Levels 1 + (optionally, depending on what hierarchical level you wish to refer to) 3 + 6 + 7?

But in any case the rulebreakers must be somewhere coded into the Discovery API already else how does the Search UI itself know how to form a proper Reference?

Zotero's author field. I've had a look at how this is handled from NARA's ARC catalog - appears to pick up the department ie in Discovery's terms <Title> where <SourceLevelID> = 1, or the top of the 'hierarchy' box in the SearchUI.

More recent descriptions I notice have a separate <Creator> tag though...

Quoting amme2
"are you saying the Discovery Reference [...] is nearly always formed from a combination of Levels 1 + (optionally, depending on what hierarchical level you wish to refer to) 3 + 6 + 7? "

Yes that is correct. I suggested to mentionthewar that there might be exceptions, but now that I think about it, I don't think there are. There are (legacy?) divisions called "subclasses" where at either level 3 or 6 ("series" and "piece") there are Reference fields which contain a forward slash character, but if you treat them as strings and concatenate them, they should still work. The problem really occurs in the other direction where you want to convert a Citable Reference into an IAID. The fact that the slashes are not necessarily delimiting references at different catalogue hierarchy levels makes this task virtually impossible to do reliably.

For example, "PRO 30/89/40" looks like department "PRO", series "PRO 30", piece "PRO 30/89", item "PRO 30/89/40". It is actually series "PRO 30/89", and when you look up that IA in Discovery you will see that the Reference field says "30/89". Painful when the "/" is usually used to delimit the different hierarchy levels.

I don't think there is a simple rule that allows you to work out which bits of the Citable Reference are the series, piece and item references. The number of slashes can vary even within a piece :(

Fair enough!

Rights - I notice this is left blank in just about everything I have in Zotero unless it's explicitly CC. So let's leave (that can of worms and just leave) it blank here too.

Title - When things don't have a title, the reference should be the title, I reckon.

An example of what citation output is supposed to look like can be found at www.nationalarchives.gov.uk/records/citing-documents.htm

There you can see that citations are either supposed to look like:

The National Archives (TNA): Public Record Office (PRO) INF 3/140

OR:

The National Archives (TNA): Public Record Office (PRO) C 139 Chancery: Inquisitions Post Mortem, Series I, Henry VI

(But the latter is a citation of a scoundrel, since it refers to 181 different files.)

But let me have a go with the Titanpad

It seems that the catalogue for this archive has changed recently, so the translator isn't working at the moment. I thought it would just be a case of updating the code to match the new URL structure, but it now appears that we can't access the XML via their API, even though it seems that the API hasn't changed much (the URLs for the XML data remain the same so far as I can see, but apparently you have to apply for access).

Can anyone confirm that they've tightened access to the API, or am I just missing something?

Thanks - good to know I wasn't just missing something. I've emailed the Discovery@ address provided with the API documentation. I'm also going to be at the archives this week and next so if we don't hear back I might see if it's possible to corner someone about it.

I heard back - they're looking into it and are aware of this thread, so hopefully something will get sorted soon. However, there's no telling how quickly they'll be able to act. So, in the meantime, I've cobbled together a very rough-and-ready workaround that scrapes from the page. It's imperfect and limited it to single items at the moment, as I really just put it together to serve my own purposes, but I thought I'd share just in case we're waiting a while for a real solution.

I've uploaded the code to: https://gist.github.com/rt-bell/95723931b04144db3633

If anyone uses it and has any problems, let me know.

Thanks, rtbell and Adam. I've given it a try and it's working correctly for me on a few different types of record. Thanks so much.

Hi Adam (and Zotero users),

Sorry about the problems caused by the temporary loss of the API. It is back online now. Would you be able to test the API-based Parser and switch back to it if everything works okay?

We have identified an issue with the API being unable to return data for non-National Archives records (those records that have recently been imported into Discovery from Access2Archives, the National Register of Archives and Archon), but those records have a different format of Discovery URL anyway and aren't offered for citation by the Zotero browser plugin. We've noted this API issue and will add it to the backlog.

Incidentally, I notice that there was talk of using the Discovery "Title" field (where present) for the Zotero "title" mapping, but that the current parser always uses a standard text of "The National Archive of the UK, COPY 1/2". Is this intentional?

Steven

OK, this uses the API again. We're again using the "Title" field, too, though I'm finding that it doesn't exist for most entries, so if there are better ideas I'm happy to hear them.

Thanks again everyone for reporting to us and the Nat'l Archive and for the folks at the archive for fixing it.

In Discovery, I believe we use the first n characters (followed by ellipsis) of the ScopeContent->Description field where title is absent.

yes--and that makes total sense for a catalog display. Having elipses in seemingly random places on import in Zotero, however, isn't satisfactory, so we'd need a different solution. It'd be possible to try to parse the description up to a period or so.

correct, we currently don't import items not from TNA, so that's not an issue at this point, but if we do (though it sounds like that might not currently be possible in the API?) we'll take that into account.