Import from Endnote RIS: translator, editor, edited book, extra titles as notes

JonEP · June 11, 2009

Hi all,

I'm trying to import a library from endnote, and am using just a few sample entries as a trial run. So, I've exported to RIS format. Then imported with zotero.

In zotero, I am getting multiple notes for such items as the translator or the editor, and these do not show up in the "info" section for the entry. I notice that these are listed as N1 in the .txt file that I imported.

Also, an item listed as an "edited book" in endnote imports as a web page in zotero.

For a book section entry, the editors of the book are listed as "Contributor" instead of editor.

Have I possibly done something wrong in importing, or is it possible to modify the RIS file so that the fields are recognized more accurately by zotero? Hand-grooming the entire library is not an option any time soon (would be days of work).

Thanks very much for your help. What I've used so far in zotero, I like!

Best,
J

JonEP · June 11, 2009

Here is a sample of a few of the entries, as they appear in the .txt file:

TY - JOUR
AU - Condominas, Georges
PY - 1974
TI - L'entraide agricole chez les Mnong Gar (Proto-Indochinois du Vietnam central)
SP - 407-420
JF - Études Rurales
VL - 53-54-55-56
N1 - L'entraide agricole chez les Mnong Gar (Proto-Indochinois du Vietnam central)
KW - highlanders
Vietnam
religion
ID - 2007
ER -
TY - BOOK
AU - Condominas, Georges
PY - 1977[1957]
BT - We Have Eaten the Forest: The Story of a Montagnard Village in the Central Highlands of Vietnam
CY - New York
PB - Hill and Wang
N1 - Foulke, Adrienne
N1 - We Have Eaten the Forest: The Story of a Montagnard Village in the Central Highlands of Vietnam
ID - 6679
ER -
TY - CHAP
AU - Condominas, Georges
PY - 1998
BT - Formes Extrêmes de Dépendance: Contributions à l'étude de l'esclavage en Asie du Sud-Est
ED - Condominas, Georges
CT - Présentation
CY - Paris
PB - Ecole des Hautes Études en Sciences Sociales
SP - 11-21
N1 - Présentation
SN - 2713212715
N1 - HT1315.A785 F67 1998
N1 - sous la direction de Georges Condominas.
KW - Southeast Asia
slavery
ID - 8088
ER -
TY - EDBOOK
AU - Condominas, Georges
PY - 1998
TI - Formes Extrêmes de Dépendance: Contributions à l'étude de l'esclavage en Asie du Sud-Est
CY - Paris
PB - Ecole des Hautes Études en Sciences Sociales
N1 - Formes Extrêmes de Dépendance: Contributions à l'étude de l'esclavage en Asie du Sud-Est
KW - slavery
ID - 8299
ER -

noksagt · June 11, 2009

It may be useful to refer to the RIS specification. Note that RIS is a flat, limited format & it all-but-impossible for it to represent the data in either EndNote or Zotero with complete fidelity.

In zotero, I am getting multiple notes for such items as the translator or the editor, and these do not show up in the "info" section for the entry. I notice that these are listed as N1 in the .txt file that I imported.

N1 is for notes. Zotero is importing them to specification. From the records you gave, it seems there is no way for Zotero to differentiate the translator/editor from any other note.

What field(s) is this info in in Endnote? Perhaps you can modify their RIS export.

Also, an item listed as an "edited book" in endnote imports as a web page in zotero.

These use 'EDBOOK', which is note an enumerated reference type in te RIS specification. You can modify the EndNote exporter to use a valid type, but perhaps Zotero can improve the RIS import by supporting that type too. Is the AU in this type an editor? Are there any other features that differentiate this from the standard "BOOK" type?

For a book section entry, the editors of the book are listed as "Contributor" instead of editor.

The specification does not differentiate ED/A2 tags. However, it is probably usually safe to assume that "ED" means "editor." I'd think the Zotero translator could be improved here.

JonEP · June 11, 2009

Thanks noksagt,

That helps a bit. So, if N1's are undifferentiated, it would seem that most users with big endnote libraries find themselves in this situation? How are people addressing the problem?

Are there any steps that can be taken to get around it? Is the RIS format the only way to go for an export from Endnote? What about using the "Endnote Export" or other export style in endnote? I realize that this conversation has probably been going on for some time, but I wonder if there is a chance that there is renewed effort to translate endnote libraries now that the lawsuit has been dismissed...

Thanks for any help or advice you can give,
J

JonEP · June 11, 2009

As an addendum, I note that the "translator" field for books seems to be the %? field, or 'subsidiary author' field in endnote (see http://bibsonomy.blogspot.com/2006/03/endnote-export.html for fields). This would be in the "generic" entry, for instance. It seems that endnote reads a different value for fields based on the entry type (ie., book, edited book, etc.). So it would, in theory, be possible to make a script or some other sort of translation device that would assign new values to the exported library file for different entries, based on their characteristics? Which would allow for a much more accurate import into zotero?

I'm not a programmer, but does this sound at all right? And is it a big enough factor for users seeking to switch from endnote to zotero that it would be worth doing? I imagine the hours spent writing the tool would pale in comparison to the cumulative editing that would be done by thousands of users around the world as they comb through their libraries making edits?

Thanks again,
J

dstillman · June 11, 2009

The specification does not differentiate ED/A2 tags. However, it is probably usually safe to assume that "ED" means "editor." I'd think the Zotero translator could be improved here.

http://forums.zotero.org/discussion/5200/ris-export-author-handling/#Item_2

JonEP · June 11, 2009

Perhaps it is more promising to attempt to use 'trunk translators' (ie: http://forums.zotero.org/discussion/7269/importing-endnote-using--tags-issn-doesnt-import/#Item_3)?

noksagt · June 11, 2009

Are there any steps that can be taken to get around it?

As I said, above: one step is to see where EndNote stores this data & to look into whether it would be possible to modify the RIS export in EndNote to improve it.

Is the RIS format the only way to go for an export from Endnote?

No, but is probably the best supported way. You're welcome to try the refer/bibix/endnote format or the bibtex format. I've not found any of EndNote's export formats to be that impressive (the newer/richer XML formats included).

but I wonder if there is a chance that there is renewed effort to translate endnote libraries now that the lawsuit has been dismissed...

The suit had nothing to do with bibliographic metadata exchange.

As an addendum, I note that the "translator" field for books seems to be the %? field

That is for the refer/bibix/endnote export format. As I said, you're welcome to try that. But RIS seems to work better.

or 'subsidiary author' field in endnote

Presumably this is what I was after: the name of the field that EndNote uses in its database/interface (rather than in any export format). However, I don't think we can assume that all "subsidiary authors" are "translators," so there may not be a good way out here.

Perhaps it is more promising to attempt to use 'trunk translators' (ie: http://forums.zotero.org/discussion/7269/importing-endnote-using--tags-issn-doesnt-import/#Item_3)?

This thread also talks about the refer/bibix/endnote file format (and, again, I don't think that's what you want to use).

JonEP · June 11, 2009

Thanks for your help noksagt and Dan,

If RIS is the best option, and if RIS is providing (quite) imperfect imports into Zotero, I wonder if it is possible to address the problem in a way that doesn't require massive editing of already existing libraries...

Just looking at the text file exported by an Endnote RIS export, it seems clear that even Endnote could not re-read it and make sense of it (and indeed, this is the case--attempting to import the file into a new library in Endnote leaves the translator, etc, in the note fields, interprets editors as authors, etc).

So, if that is the case, is there a better way to transfer these libraries? Or, rather, would it be possible to devise a better way, and is there a desire for that among users and/or prospective users?

Would it be possible, for instance, to create a new endnote export style, so that Endnote would import all the relevant information, and so that zotero would be able to read, and correctly assign, all the information?

Has anyone found another way to do this? Or is transferring to zotero really going to require manually comparing the zotero library with the endnote one and then making refinements?

Thanks very much,
J

Thanks,
J

noksagt · June 11, 2009

Let's review. You have enumerated three gripes about using RIS as an intermediate format:

translators get added as notes
edited books get typed as web pages
editors get cast as contributors

I've said that 2&3 can be improved by modifying the Zotero translator.

But 1 seems like a fundamental issue if EndNote does not actually have a 'translator' field, but only a 'subsidiary author' field. It is probably possible for you to modify the Endnote exporter so that these subsidiary authors would get imported into Zotero as contributors.

So, if that is the case, is there a better way to transfer these libraries? Or, rather, would it be possible to devise a better way, and is there a desire for that among users and/or prospective users?

While incremental improvements can be made, there are also severe limitations with EndNote's ability to export data & also some deeper differences in the database design of the two programs.

Would it be possible, for instance, to create a new endnote export style, so that Endnote would import all the relevant information, and so that zotero would be able to read, and correctly assign, all the information?

As above, this is probably impossible due to fundamental limitations. That being said, see discussion at:
http://forums.zotero.org/discussion/70/importing-endnote-files-into-zotero/
and in other threads here.

JonEP · June 12, 2009

Thanks for the link to the other discussion.

Sorry if it seemed like 'griping'. In fact, my hope was to raise a serious issue for discussion by the zotero community. I also realize that this is a discussion that has been going on for a while and the zotero developer community is probably fed up with it. However, I have to agree with jcw and jborche at this forum discussion:
http://forums.zotero.org/discussion/4808/import-problems/

Without an adequate way to get out of Endnote, zotero is simply not an option for those researchers who have invested heavily in commercial reference management packages. They simply will not have the time to manually re-enter vast amounts of data in order to be able to use zotero.

My hope is that the zotero community take the initiative on this; blaming Endnote for being a bad piece of software is not going to solve any problems. There was some talk about using endnote's xml export? What if endnote users were to sort their libraries by reference type, and then export into smaller .txt files, one for each reference type. There might be a possibility of using find/replace to change the field codes for things like 'tertiary author' into translator, etc. I only use that as one example--there are a lot of different glitches involved in an endnote export / zotero import using RIS, and there needs to be a way to address them.

Again, sorry if this seems like griping. I am not complaining about zotero at all-- I want to be able to use zotero. I'm also not interested in blaming the problem on endnote. But I am sure this is a big issue, and a real roadblock for people, like me, who have not started doing research in the last three or four years.

Thanks,
J

bdarcus · June 12, 2009

I've not really followed this thread, but ...

My hope is that the zotero community take the initiative on this; blaming Endnote for being a bad piece of software is not going to solve any problems.

No, but it can identify where the problems are, so that you don't blame the wrong parties, and hence expect them to fix a problem that is not of their making.

And there are real issues on the Endnote end that make things really difficult for users like you who are interested in migrating their data, and for developers like the Zotero team that are trying to accommodate them.

I'll give you a simple example: for the longest time there was a known, and unfixed, bug in Endnote's RIS output style that output multiple keywords like this:

KW  - one
two
three

This is wrong; it should be this:

KW  - one
KW  - two
KW  - three

What this meant is that any implementation that supported RIS correctly imported Endnote RIS files in ways that a user would consider a "bug." So then applications like Zotero need to add code to work around that bug, which takes time, and has to be done in the context of a long list of other priorities.

There are other issues like that. For those issues, I think it's quite good practice to assign blame. It allows people to see the issues clearly. I mean, this RIS bug has been here forever, and it's still not fixed!

There was some talk about using endnote's xml export? What if endnote users were to sort their libraries by reference type, and then export into smaller .txt files, one for each reference type. There might be a possibility of using find/replace to change the field codes for things like 'tertiary author' into translator, etc. I only use that as one example--there are a lot of different glitches involved in an endnote export / zotero import using RIS, and there needs to be a way to address them.

Given what's been said in this thread, I have a feeling this wouldn't work. XML is no magic bullet here.

You need a stable way to map from Endnote field to RIS tag (or potentially a "translator" value somewhere in the XML). If there is no stable way to do that, I don't see how to program it on the Zotero end.

Finally, I looked through the bibutils source code, which is the gold standard in bib conversion. There is an import mapping for translator from "Endnote" (Refer). But there is not for RIS.

Could you maybe post a single record that identifies the problems, both in RIS and Refer?

Note: please wrap them in "code" tags so that the spacing is properly preserved. I waste 10-15 minutes cleaning up the example you have up top.

bdarcus · June 12, 2009

Actually, I may have read the source wrong:

lib/endxmlin.c:		{ "translated-authors", "%?" },

This seems to suggest there's some "%?" value in the Endnote XML output that bibutils is using for the translator. Can you check that?

If that's the case, then Zotero could add this (assuming they have an importer for Endnote XML).

noksagt · June 12, 2009

I didn't intend to have any negative connotation when I said 'gripe' (which I just used to describe an understandable frustration that things weren't working perfectly).

and the zotero developer community is probably fed up with it.

I don't think this is the case either. I think everyone would love better migration & it is being worked on.

Without an adequate way to get out of Endnote, zotero is simply not an option for those researchers who have invested heavily in commercial reference management packages.

Keep in mind that many people already have migrated away from EndNote. While this is not perfect (and probably can't be), it has certainly been "adequate" for quite a few people, some of whom have very large libraries.

My hope is that the zotero community take the initiative on this; blaming Endnote for being a bad piece of software is not going to solve any problems.

I didn't claim it would. We try to work-around EndNote limitations where possible. But sometimes it will not possible, and it is important to point that out. Lock-in sucks ;-) !

There was some talk about using endnote's xml export?

Yes. There is an open ticket for this. This is not a panacea, as their schema varies by version, EndNote can produce invalid XML, EndNote cannot always import XML that it exports, etc. But it would be better than RIS.

What if endnote users were to sort their libraries by reference type, and then export into smaller .txt files, one for each reference type.

This is a bit of a hackish work-around & should not be needed, as EndNote does give the type for all exported items. We also wouldn't want "EndNote fixes" to break import of files actually made to the standard.

There might be a possibility of using find/replace to change the field codes for things like 'tertiary author' into translator, etc.

Does EndNote have an explicit translator field or not?

noksagt · June 12, 2009

Could you maybe post a single record that identifies the problems, both in RIS and Refer?

This would be useful. Please give as many of the following that you can too:

EndNote version
Date of RIS/ISI translators
the name of the relevant fields within the EndNote interface

Actually, I may have read the source wrong:

In refer/bibix/endnote, it seems that bibutils will export the translator to "%H". Perhaps the REFER format will be better for users with a lot of translated content for the short-term (e.g. before an EndNote XML importer)?

lib/endxmlin.c: { "translated-authors", "%?" }This seems to suggest there's some "%?" value in the Endnote XML output that bibutils is using for the translator.

Their current DTD enumerates both 'subsidiary-authors' and 'translated-authors elements'. From the bibutils source and from the DTD, I'd expect those to be the name of the nodes. I don't know how EndNote/bibutils handle these in practice, since bibutils uses '%?' for the field of both.

JonEP · June 15, 2009

Here I am providing three of the endnote exported RIS entries, hope this is correct and helpful:


TY  - EDBOOK
AU  - Aldenderfer, Mark
AU  - Maschner, Herbert D.G.
PY  - 1996
TI  - Anthropology, Space, and Geographic Information Systems
CY  - Oxford
PB  - Oxford University Press
N1  - Anthropology, Space, and Geographic Information Systems
KW  - remote sensing
ID  - 7514
ER  -

This became a webpage, with the editors listed as authors. I believe EndNote writes AU for the editors of an EDBOOK entry...


TY  - EDBOOK
AU  - Arens, Richard
PY  - 1976
TI  - Genocide in Paraguay
CY  - Philadelphia
PB  - Temple University Press
N1  - Genocide in Paraguay
KW  - Ache
genocide
N1  - ache genocide
UR  - C:\Documents and Settings\Jp\My Documents\ISSUES FILES\Anthropology\SlaverySession-example of use of Ache genocide in teachin.htm
ID  - 1390
ER  -

Again, in Zotero became a Web Page, with Arens, Richard as an author. The note "ache genocide" attached successfully as a note.


TY  - BOOK
AU  - Condominas, Georges
PY  - 1977[1957]
BT  - We Have Eaten the Forest: The Story of a Montagnard Village in the Central Highlands of Vietnam
CY  - New York
PB  - Hill and Wang
N1  - Foulke, Adrienne
N1  - We Have Eaten the Forest: The Story of a Montagnard Village in the Central Highlands of Vietnam
ID  - 6679
ER  -

This became a book, with an attached note reading: Foulke, Adrienne.

A few other noted glitches: Endnote fields for ISBN, Call Number, and Label are all written by zotero as individual notes attached to entries.
Newspaper articles with a separate "year" and "date" entry will export the year to Zotero's "date" field, and zotero will attach a note containing the "date" (e.g., "September 24,") field.

NB, wondering why Endnote exports the title for each record twice (ie, as BT for the title of the book, and as TI for the title of the edited book).

So, all info is transferring, with a few glitches.

Trying to figure out the issue of 'translator' for Zotero, but will raise that question in a distinct thread.
JP

bdarcus · June 15, 2009

On your last example, this is a bug in Endnote and/or RIS. They're dumping the translator to a note, so Zotero is correctly reading that.

What about the Refer version?

JonEP · June 15, 2009

Sorry about that-- Endnote X2, and prior to exporting, I downloaded the most recent version of the RIS / Refman style, created in May of 2009.

bdarcus · June 15, 2009

No, I mean Refer/Endnote format output. It looks similar to RIS, but is a little different. I'm just speculating that it might properly export the translators, but I don't know.

JonEP · June 15, 2009

Hi, one more note on all of this, in the process, "book sections" exported by Endnote wind up with the book editors listed as contributors.

JonEP · June 15, 2009

@bdarcus

Sorry about that. Here it is, but I think, looking at it, that it does not correctly output a translator:


%0 Book
%A Condominas, Georges
%D 1977[1957]
%T We Have Eaten the Forest: The Story of a Montagnard Village in the Central Highlands of Vietnam
%I Hill and Wang
%C New York
%O Foulke, Adrienne
%O We Have Eaten the Forest: The Story of a Montagnard Village in the Central Highlands of Vietnam