export from REFMAN to Zotero without losing special characters

dhercot
Exporting from REFMAN did mess up all the special characters when I
tried the export function in REFMAN. I finally found a way around
after reading through the discussions on zotero forum and a lot of
trial and error.

My way around to export from REFMAN to Zotero without losing special
characters:
  1. inside REFMAN, create a bibliographic style that mimics
    the RIS format.
    • you can import the REFMANRECORD style template from REFMAN
      website
    • adapt the bibliographic output to your needs for instance check that the publication name is included in the output
      because in my file it was missing.
    • To do this:
      • go to Tools/Bibliography/open output style and

      • select the file [RISrefman.os] which you just saved from
        refman website.
      • select the "bibliography" tab, and

      • add or adapt a line for T2 like :
        "T2··-·<[11]·Periodical>|¬"

      • repeat this for each publication type with a * at the end
        you are using in your export.e.g. "Journal*"

  2. Exporting from REFMAN

    3. As the export function doesn't keep special characters in REFMAN
    12, I used the Tools/Bibliography/Generate from reference list and
    the adapted REFMANRECORD output style I did this based on the
    documentation page import from endnote 
    This generates an *.rtf file 
  3. Fine tuning the RIS codes
    4. A number of RIS codes are ill formatted in the rtf generated by REFMAN.
    I used notepad++ "find and replace" command to clean these codes :
    • open the document in rtf format
    • copy all text and paste in notepad++- This will remove the hidden characters in the rtf file.-
    • then start cleaning:
      • basics:
    * erase all ¬
    * replace all %K by KW - REFMAN only imports the first keyword
    with a KW and subsequent ones with %K
    * move the REFMAN ID into a "note": if not, you have a conflict
    with issue when importing replace "ID - " by "N1 -"
    See discussion below. ID should not generate a conflict for most exports
    and can be set as a note in zotero preferences - Thanks to aurimas
    *doi : replace U5 - by DO -
    *maintain link to pubmed: change "UR - PM:" to "UR -
    http://www.ncbi.nlm.nih.gov/pubmed/"
    See discussion below. PUBMED ID should become an attachement so "L2 -
    PM:" - Thanks to aurimas
      • Additional recodes based on your needs:
    *localisation in archive: replace AV - by AN -
    *archive replace RP - by DB -
    *maintain editor field : change "A2 - " to "ED -
    * to maintain the adress into the field "extra": note that the
    extra field does not get exported from ZOTERO in the end if you
    want to export from ZOTERO at a later stage, maybe you should
    prefer to use a note then (N1). change "AD -" to "M2 -"

      • Additional comments:
    *Other fields who gave me troubles include JA - J2 - TA -
    *L2 is a second URL in refman and will become an attached item
    *Zotero cannot handle a link to a folder on our server like
    X:/digitaldocs/DREE and this generates a failure of the import
    process. I manually moved all those links to notes doing a search
    for "L2 -"
  4. Importing in Zotero
    the standard way     should happen without problems.

I did this importing around 2000 items at a time. Any comment or
suggestion for improving this entry is welcome.

Edited on April 24
  • aurimas
    * move the REFMAN ID into a "note":
    if not, you have a conflict with issue when importing
    replace "ID - " by "N1 -"
    Not sure what you mean by conflict here. ID should be properly ignored (or imported into a note based on your hidden preferences)
    *maintain link to pubmed:
    change "UR - PM:" to "UR - http://www.ncbi.nlm.nih.gov/pubmed/"
    PubMed links should not go into URL field. They should go into attachments, so L2.

    Could you post a sample RIS file you generated _before_ manual edits? Some of the fields you attempt to fix should work out of the box, so I'd like to see what the problem is (including file attachments). Post on gist.github.com and link to it here.
  • aurimas
    Oh, also, do these fields match if you just use REFMAN's RIS export (make sure you update to the latest filter)?
  • dhercot
    edited April 22, 2014
    @aurimas thanks for your comments.

    Not sure what you mean by conflict here. ID should be properly ignored (or imported into a note based on your hidden preferences)

    The first import I managed to do in zotero with the ris file as is generated by the REFMAN RIS export function conflicts between ID field and Issue field: "Issue" ended up in a note saying ZOTERO could not find matching fields for this code while the ID took the place of the IS - like this once exported out of zotero again see example :

    Expot from REFMAN using the export function:
    TY - JFULL
    ID - 18463
    T1 - alert
    A1 - BRAL
    Y1 - 2013///
    KW - Brussels
    KW - European Union
    KW - small areas
    KW - urban
    RP - NOT IN FILE
    JF - Alert
    VL - 23
    IS - 378
    AV - J Alert
    ER -

    export from zotero back once imported :

    TY - JOUR
    TI - alert
    AU - BRAL
    DA - 2013///
    PY - 2013
    VL - 23
    IS - 18463
    AN - J Alert
    DB - Not in File
    N1 -

    The following values have no corresponding Zotero field:
    issue: 378

    KW - Brussels
    KW - European Union
    KW - small areas
    KW - urban
    ER -

    PubMed links should not go into URL field. They should go into attachments, so L2.
    Thanks for the suggestion. I took the suggestion from maba on this discussion thread https://forums.zotero.org/discussion/520/importexport-reference-manager/
    same goes for the A2 - ED - change suggested.

    As per your request, I posted a RIS file of my 2010 collection here : https://gist.github.com/dhercot/ac06e7720f078f44447d
    This is a standard export using RIS specification in REFMAN 12.

    Look at line 6603 for an item that bugged ZOTERO completely. I suspect it's the L2 field.
    I manually removed all those kind of L2 links.

    I posted another RIS file using the initial RIS style Bibliography which I catched from the refman website here https://gist.github.com/dhercot/3913abfa3fbd4ed49361
    where you can find the keyword problem and the absence of the publication name.

    Finally I could not find any updated RIS export filter for REFMAN and the one I have installed bugs the special characters completely which makes it a no go for me. My library has many french articles.
    We have the REFMAN 12.0.3 (Bld3262) version of REFMAN installed and we believe that is (and will be )the last one Reuters has made.

    Hope this helps.

    Best.
  • adamsmith
    odd - I can't replicate the ID/IS issue and I see no reason why that could happen. If you select the RIS above (from TY to ER -) and use "Import from Clipboard" in Zotero, does that work correctly?
  • dhercot
    Hi @adamsmith,
    I'm out of office so I can't replicate the issue above on my office computer. Here at home it seems to work. Strange. I don't know why it happened back in January when I started exploring this issue.

    anyhow my zotero standalone is again having troubles syncing I guess I'm doing too many operations at a time and should give more time to zotero to digest it. I mean waiting till one operation is fully synced before doing the following one (like copying 2000 items from my library into the group, deleting it from my library and emptying the trash.) Anyhow, I will see if thinks get solved by Tuesday when I get back to the office. I'm now off for the Week End. Thanks for jumping in and getting back in touch next week for exploring this further.
  • aurimas
    I posted another RIS file using the initial RIS style Bibliography which I catched from the refman website here https://gist.github.com/dhercot/3913abfa3fbd4ed49361
    where you can find the keyword problem and the absence of the publication name.
    Are you sure this has not been edited before uploading? If it hasn't, then I'm a bit confused about a couple things.

    Take a look at the keywords for "ID - 18540"
    KW - adolescent
    KW - Adult
    KW - Belgium
    KW - benefit
    KW - Brussels
    KW - census
    KW - demography
    KW - education
    KW - health
    KW - Health Status
    KW - history
    KW - Maghreb
    KW - migration
    KW - mortality
    KW - nationality
    KW - public health
    KW - region
    KW - research
    KW - Risk
    KW - sociology
    KW - status
    KW - Young Adult
    KW - young adults
    As you can see, they are unlike most of the other items and have "KW -" for subsequent keywords instead of "%K ". Also, note the "Health Status" keyword, which has three spaces instead of two after "KW". Is REFMAN really this inconsistent in their export?

    Also, I cannot find any U5 entries that you're referring to in your first post
    doi : replace U5 - by DO -
    All of them appear to be DO
    *maintain link to pubmed: change "UR - PM:" to "UR -
    http://www.ncbi.nlm.nih.gov/pubmed/"
    I made it store PMID in extra like we do for all other translators. In the future (hopefully not too distant) Zotero will get a dedicated (hyperlinked) field for PMIDs, at which point, the PMID from extra will automatically transition there. But currently, this is not going to be hyperlinked (obviously you can alter the translator or the file to suit your purposes though)
    *archive replace RP - by DB -
    From your sample file (and actually from EndNote exports as well), the RP field seems quite useless. It almost always contains "In File", which I don't actually know what it means. Either way, I'm leaving this one out of the database field.
    *maintain editor field : change "A2 - " to "ED -
    All of the instances of A2 in your sample file are for reports. In Zotero, reports don't have an "editor", they only have "series editor", which is what A2 is mapped to for reports by default. REFMAN does say that this should be "editor" in their user guide. @adamsmith, do you know why we don't have "editor" for reports? Does it not make sense for that item type?
    Zotero cannot handle a link to a folder on our server like
    X:/digitaldocs/DREE and this generates a failure of the import
    process. I manually moved all those links to notes doing a search
    for "L2 -
    I'm not sure why you're having issues with this. I cannot reproduce it on my end. Can you be more specific about what you mean by "generates a failure on the import process"? If it's literally breaking import, can you generate a Debug ID for an attempt to import a single REFMAN record with a link to the file?

    Updated RIS translator will be live in a few minutes.
  • adamsmith
    @adamsmith, do you know why we don't have "editor" for reports? Does it not make sense for that item type?
    no reason except legacy. Definitely planned:
    https://github.com/ajlyon/zotero-bits/issues/31
  • aurimas
    For reference, I'm going to leave a link to REFMAN 12 user guide here

    http://www.refman.com/support/docs/ReferenceManager12.pdf
Sign In or Register to comment.