Upgrade StorageImport from EndNote all the fields (even custom) via ReferBiblX - my experience
Dear all,
Maybe my experience with importing EndNote records into Zotero will be useful for someone or maybe I was doing all the conversion totally wrong and a better solution exists. But officially proposed conversion via RIS style didn't suit me because lack of language and other fields import, as stated here: http://forums.zotero.org/discussion/22844/ris-import-problem/#Item_4 and it didn't export custom fields which I needs. I utilized all 7 custom fields in Endnote so there is a big problem to transfer such records to Zotero. I searched to solution, but didn't find any that satisfy me. So I created my own.
First step - exporting from EndNote.
I choose ReferBiblX conversion. But it didn't show all the fields in exported txt file at first. So I modified "EndNote Export" style: replaced data in all reference types with data of "General" type. So, all types of references were exported with all fields after that.
Second step - modifying ReferBiblX translator in Zotero.
That was hard task for me as I am not a programmer. But I managed to some success. I added several rules of conversion in "var fieldMap = {":
6:"numberOfVolumes",
G:"language",
"!":"shortTitle"
and some other.
Such configuration suited for me, everyone can do own. Maybe language variable can be included to official Zotero conversion file? Is it not common to all reference types?
"Acces date" was the big problem. Because it is marked as %\ in Endnotes file but \:"accessDate" or "\":"accessDate" not worked in translator. My workaround was to change all the fields in Endnotes file from %\ to symbol that isn't used in conversion. I choose number 5, so I changed all %\ to %5 and added to translator rule 5:"accessDate". And "Accession date" appeared in Zotero.
All custom fields I arbitrary linked with free field variables in Zotero. Names of variables (but not all) I found there: https://www.zotero.org/trac/browser/extension/branches/1.0/chrome/locale/en-US/zotero/zotero.properties?rev=3446
There was problem with "pages" and "number of pages". EndNote links "Number of pages" in "Book" reference type and "Pages" in various types of articles and "Book section" to the same item %P. So either pages or page numbers didn't appear after importing.
I created such a code (sorry, I don’t know what correct terms are) in “function processTag(item, tag, value)”:
else if(tag == "P") {
if(!item.pages) {
item.pages = value;
if (item.itemType == "book") item.numPages = value;
Similar problem was with ISBN and ISSN. My solution – add such lines:
else if(tag == "@") {
if(!item.ISSN) {
item.ISSN = value;
if (item.itemType == "book" || item.itemType == "bookSection" || item.itemType == "encyclopediaArticle") item.ISBN = value;
}
Adding to Extra field several items didn’t worked, so I changed code from
else if(tag == "H" || tag == "O") {
item.extra += "\n"+value;
to
}else if(tag == "H" || tag == "O") {
if (!item.extra) {
item.extra = value;
} else {
item.extra += "\n"+value;
}
There were some other tweaks that suited for my particular tasks. It is only partial solution. I need only few types of references: book, book section, electronic article, encyclopedia article, journal article, magazine article, newspaper article, unpublished work and personal communication. But maybe my experience helps to somebody.
The only problem is reference type "Edited book" in EndNote. It converts to book in Zotero so editor becomes author. Maybe best solution is convert "edited book" to "Classical", manually change author to editor and change to book after that if there is no better solution. But I don't know what is field item name of item type "Classic".
Maybe my experience with importing EndNote records into Zotero will be useful for someone or maybe I was doing all the conversion totally wrong and a better solution exists. But officially proposed conversion via RIS style didn't suit me because lack of language and other fields import, as stated here: http://forums.zotero.org/discussion/22844/ris-import-problem/#Item_4 and it didn't export custom fields which I needs. I utilized all 7 custom fields in Endnote so there is a big problem to transfer such records to Zotero. I searched to solution, but didn't find any that satisfy me. So I created my own.
First step - exporting from EndNote.
I choose ReferBiblX conversion. But it didn't show all the fields in exported txt file at first. So I modified "EndNote Export" style: replaced data in all reference types with data of "General" type. So, all types of references were exported with all fields after that.
Second step - modifying ReferBiblX translator in Zotero.
That was hard task for me as I am not a programmer. But I managed to some success. I added several rules of conversion in "var fieldMap = {":
6:"numberOfVolumes",
G:"language",
"!":"shortTitle"
and some other.
Such configuration suited for me, everyone can do own. Maybe language variable can be included to official Zotero conversion file? Is it not common to all reference types?
"Acces date" was the big problem. Because it is marked as %\ in Endnotes file but \:"accessDate" or "\":"accessDate" not worked in translator. My workaround was to change all the fields in Endnotes file from %\ to symbol that isn't used in conversion. I choose number 5, so I changed all %\ to %5 and added to translator rule 5:"accessDate". And "Accession date" appeared in Zotero.
All custom fields I arbitrary linked with free field variables in Zotero. Names of variables (but not all) I found there: https://www.zotero.org/trac/browser/extension/branches/1.0/chrome/locale/en-US/zotero/zotero.properties?rev=3446
There was problem with "pages" and "number of pages". EndNote links "Number of pages" in "Book" reference type and "Pages" in various types of articles and "Book section" to the same item %P. So either pages or page numbers didn't appear after importing.
I created such a code (sorry, I don’t know what correct terms are) in “function processTag(item, tag, value)”:
else if(tag == "P") {
if(!item.pages) {
item.pages = value;
if (item.itemType == "book") item.numPages = value;
Similar problem was with ISBN and ISSN. My solution – add such lines:
else if(tag == "@") {
if(!item.ISSN) {
item.ISSN = value;
if (item.itemType == "book" || item.itemType == "bookSection" || item.itemType == "encyclopediaArticle") item.ISBN = value;
}
Adding to Extra field several items didn’t worked, so I changed code from
else if(tag == "H" || tag == "O") {
item.extra += "\n"+value;
to
}else if(tag == "H" || tag == "O") {
if (!item.extra) {
item.extra = value;
} else {
item.extra += "\n"+value;
}
There were some other tweaks that suited for my particular tasks. It is only partial solution. I need only few types of references: book, book section, electronic article, encyclopedia article, journal article, magazine article, newspaper article, unpublished work and personal communication. But maybe my experience helps to somebody.
The only problem is reference type "Edited book" in EndNote. It converts to book in Zotero so editor becomes author. Maybe best solution is convert "edited book" to "Classical", manually change author to editor and change to book after that if there is no better solution. But I don't know what is field item name of item type "Classic".
} else if(tag == "E") {
if (item.itemType == "document" || item.itemType == "book") {
var type = "seriesEditor";
} else {
var type = "editor"; }
if (item.itemType == "interview") {
var type = "interviewer";
Tag E should be converted as series editor in book and document; all other must be converted as editor except interview. This is not working only with edited book type )edited book was mapped as document). What's wrong?
I just read your post and am dealing with almost the same problem. I have an Endnote X5 database of approximately 6600 references and use not only all the custom fields but some of the "unused" fields in my database as well (I don't deal with books so those fields are fair game).
I am having trouble using boolean searches in Endnote and was told that Zotero might be the answer. So far I have NOT seen this. Even without the custom fields being imported I believe the search capabilities of Zotero are no better than Endnote and in some cases worse. I am unable to combine searches in any meaningful way, or use true "If/Then/Else" logic.
Specifically, in the advanced search of Zotero you can search for the absence of presence multiple criteria. You can either search for "any" of those (the equivalent of the boolean OR) or "all" (the equivalent of "AND", so you can do quite a lot.
However, if you want to combine AND and OR in a search - as in
(A AND B) or C
you need to string saved searches.
In the example above you create a saved search by searching "match all" A, B and call it D, then do a "match any" search for C and D.
This seems somewhat convoluted. This is especially true because of what I am trying to do. I manage a DB of 150+ investigators' publications for a cancer center. We have different research programs which the each of the investigators are aligned with/in. In the past I have used 1st reference manager (up to version 12), then EndNote (now version X5). Although I love many of the capabilities of EndNote, the limited search has made my life a constant search for something better. I need to be able to easily see how investigators' publications work with each other (if there is more than one of our PIs in a pub are they in the same program, different programs, or some in different and some the same). To do this I have had to use all the custom fields and then some to hand code each publication for which program each author is in. When we realign an author, this can cause major problems as everything has to be recoded.
I want to be able to just search on their names as I did in refman and then get the results from that. Since we have 5 programs, that would mean innumerable chained searches from your scheme mentioned above.
Zotero needs better search capabilities, not to mention better import capabilities (my son, who went to George Mason, said the import problem had something to do with a suit that Zotero lost to EndNote?)
If I understand you correctly, you could just use a tag for each of the five programs, and then tag each publication with the program(s) of the author(s) - you can do that quickly retrospectively by doing a simple search for an author and then mass tagging all results.
I don't see at all why you'd need to combine AND and OR searches for what you're doing, but maybe I'm misunderstanding you - what would be a sample Boolean search query that you have in mind? well, it's got better search and import than the software you're shelling out several hundred dollars for...
Seriously, though - because Zotero also has better organizational features than Endnote - e.g. the combination of tags and collections - this may just be a matter of getting used to different software and how to use it effectively.
As for import - the problem is that you're using software that's notorious for it's poor quality and non-standard-compliant exports (something that's also true for ProCite and RefMan, both of which are also owned by Thomson Reuter IIRC). Since endnote's exports aren't even consistent between version numbers, there's really not that much Zotero can do about that. that's incorrect. There was a lawsuit by Thomson Reuters against Zotero/GMU/The Commonwealth of Virginia, but it was thrown out (on a technicality) - so if anything Zotero won - and it only affected citation styles, not import formats.
Let me try to explain this better though. I use the custom fields in EndNote to designate the programs for each of our authors in a certain publication. For example, if authors A, B, C, and D are on a publication and author A, B, and C are in program 1 and author D is in program 2, I put a "1" in custom fields 1, 3, and 4 (since 2 is reserved) and I put a "2" in custom 5, then in custom 6 I put a "+-" designating it as an "inter/intra" programmatic publication ("+" is inter and "-" is intra). Then I create a smart group that searches the custom fields for the program number and the collaborative designation (search for : custom 1 contains 1 OR Custom 3 contains 1 OR custom 4 contains 1 or custom 5 contains 1 AND custom 6 is +-)
This search returns the number of inter/intra programmatic publications which involve an author from program 1.
I want to be able to just search on the authors without having to hand code each publication and then change that coding every time we realign an investigator (which is daily when we are about to renew a grant). So far this hasn't worked in EndNote, mostly because of the 10 item search limit and the inability to combine saved searches (unless I'm missing something, which is entirely possible).
Zotero seems to have unlimited items, but I didn't see how to combine saved searches - I'll try your suggestion.
I'll start another thread about exporting ALL the fields from EndNote. Although I like @maras' solution, I am far from a programmer (that's part of he reason I'm using EndNote instead of a SQL DB), and it seems like you need a minimum of understanding to alter these files.
I think I understand what you're trying to do. Here is how I would solve that in Zotero:
1. Create five saved searches, one for every group, by searching for all researchers in that group combined with "ANY" (i.e. an "OR" search in boolean terms).
2. If you want to be very systematic, you can then create 26 saved searchesfor AND combination of those 5 (i.e. 1*2*3*4*5, 1*2*3*4) etc.)- that gives you all possible combination of the 5 research groups. That's a bit tedious, but you only have to do this once.
3. If you want to, you can add one more saved search - an "ANY" search over all those 24 saved searches - that will give you all items that are in a cross-group program.
Now, when a researcher switches groups, you just have to edit the relevant saved searches under 1. (right-click on the saved search --> edit saved search -->e.g. remove researcher from saved search 1, add her to saved search 2).
All your inter-group saved searches will then re-populate using the updated group memberships.
Obviously you want to test this before applying it on a large scale, but I don't see why this wouldn't work.
Depending on how else you manage this, you may want to also use tags - they're really convenient for filtering - but you'd have to delete and reassign them globally when you switch researchers between groups, so YMMV
Again, though - I just know how this should work and haven't tested this out in practice, so I'd strongly advise to play around with this in a smaller test set-up to make sure this works as you think it should before putting too much work into implementing it whole scale.
I'd suggest you move all further questions on this over to a new thread with an apporpriate title, we've already derailed this one quite a bit - I'll see any new posts by you.
So two tasks were for me: 1. exporting and 2. importing.
1. So, questions are: a. is there possibility to export all the fields from EN; b. Which format I should choose for that?
1.a. Yes, it is possible. As I've said in my first commentary, I
replaced data in all reference types you needed with data of "General" type. So, all types of references were exported with all fields after that. And one amendment - Only fields that have titles are exported. So all your custom fields must have names (they can be set in Preferences -> Reference types -> Modify reference types).
1.b. I chose "EndNote export" format because lack of knowledge about other export formats. I know only, that RIS format didn't understand crucial for me language field and some other. Maybe BibTex export option can be good alternative, but I can see very small portion of fields that is covered by it. So only EndNote export format suits for our goal.
2. And this is harder part of all the conversion because it needs some coding. Maybe it is possible to create universal converter with ReferBiblX translator. And I think, that this is possible, at least to create better import translator than with RIS style. But I don't have enough knowledge of all Zotero (and CSL) fields - it is essential for creating mapping of EN - Zotero fields.
So my way was different as described above if my first post. For noobs like me I suggest to create several copies of ReferBiblX translators - one for each type of reference (book, edited book, chapter etc.). And then it should be quite easy to (re)map EN fields to Zotero field variables (where is listed full list of them for all reference types? I didn' find). You can choose mapping whatever you want. Some fields (custom) you may map to tags, I think.
After such preparation (and testing!) conversion could be done: export one type of references at once and import them via appropriate translator. It is better way than recode several thousand references, I think :).
I am guessing that the PMID should be in the call number field. But were is the import filter for Zotero located, so I can try to edit it, when you install the standalone version?
Also where should the PMCID be stored in Zotero? In EndNote it is in Custom 2, but now that the NIH is requiring it for most grants we need it stored in the database properly.
http://www.zotero.org/support/zotero_data
that's true for standalone as well as Zotero for Firefox. It's much easier to work with translators in Zotero for firefox because you have the Scaffold IDE
http://www.zotero.org/support/dev/translators/scaffold
We'll almost certainly eventually have proper fields for PMID and PMCID in Zotero, currently I'd put them in "Extra" (which can be cited).
C:\Documents and Settings\<username>\Application Data\Zotero\Zotero\Profiles\3kag4bog.default\zotero\translators
var fieldMap = {ID:"itemID",
T1:"title",
T3:"series",
JF:"publicationTitle",
CY:"place",
JA:"journalAbbreviation",
M3:"DOI"
};
And add line AN:"whatever_field_you_want". That is PMID stored in RIS (reference type - book). But I don't see Custom 2 field in RIS file.