Mendeley -> Zotero via BibTeX: links to attached files are lost

steffenrust
Hi everyone,
I'm trying to migrate from Mendeley to Zotero. On importing the bibtex file exported by Mendeley, many file links are lost. My guess at an explanation: Mendeley renames files using non-ASCII characters. When it exports to bibtex, these characters are escaped. Links containing escaped characters (mostly German umlauts) are not imported by Zotero.

My library contains > 17.000 items, so manual editing is not an option. Jabrefs automatic renaming feature fails.

Any help is appreciated,
Steffen
«1
  • dstillman
    We can look into this, but have you tried the official import steps?
  • steffenrust
    Hi,
    yes, I did, but maybe I got it wrong. As I understood the info on that page, I would have to upload all my pdfs (>35 GB) to Elseviers servers, in order to make any information about attached files available by syncing to an old Mendeley? I'd like to avoid that.

    I wouldn't mind loosing any information not in the bibtex export if this is necessary to change to Zotero. So the problem might be solved on the Zotero side rather than on Mendeleys.

    Regards,
    Steffen
  • adamsmith
    could you try Endnote XML instead of bibtex as a Mendeley export format?
  • steffenrust
    With Endnote XML I loose the file links as well, but RIS preserves them! RIS drops the tags I assigned to papers in Mendeley, however.
  • dstillman
    Can you share an example RIS entry that loses tags?
  • adamsmith
    I'd also like to see a sample Endnote XML entry if you don't mind?
    I think we may actually be doing something wrong with the Endnote XML.
    IIRC, RIS export in Mendeley doesn't include keywords/tags
  • steffenrust
    This is one, where file link now works, but a tag is lost:

    TY - JOUR
    T1 - General q-exponential model for tree height, volume and stem profile
    A1 - Petrauskas, Edmundas
    A1 - Rupšys, P
    Y1 - 2010///
    JF - Latest Trends on System
    VL - II
    IS - Volume II
    SP - 555
    EP - 560
    SN - 9789604742141
    UR - http://www.wseas.us/e-library/conferences/2010/Corfu/SYSTEMS/SYSTEMS2-34.pdf
    L1 - file:///D:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Petrauskas, Rupšys - 2010 - Latest Trends on System.pdf
    ER -
  • dstillman
    Right, so no tags in there.
  • steffenrust
    edited August 15, 2019
    The same entry in Endnote xml. Looses file link and tag, and gives an error message indicating that my Mendeley tag ended up in "label":

    <?xml version="1.0" encoding="UTF-8"?><xml><records><record><database name="Petrauskas.enl" path="Petrauskas.enl">Petrauskas.enl</database><ref-type name="Journal Article">0</ref-type><contributors><authors><author>Petrauskas, Edmundas</author><author>Rupšys, P</author></authors></contributors><titles><title>General q-exponential model for tree height, volume and stem profile</title><secondary-title>Latest Trends on System</secondary-title></titles><periodical><full-title>Latest Trends on System</full-title></periodical><pages>555-560</pages><volume>II</volume><issue>Volume II</issue><keywords/><dates><year>2010</year></dates><isbn>9789604742141</isbn><urls><pdf-urls><url>internal-pdf://Petrauskas, Rupšys - 2010 - Latest Trends on System.pdf</url></pdf-urls><web-urls><url>http://www.wseas.us/e-library/conferences/2010/Corfu/SYSTEMS/SYSTEMS2-34.pdf</url></web-urls></urls><label>Allometrie</label></record></records></xml>
  • adamsmith
    Are all your PDFs in /pdf/Mendeley/Artikel/ ?
    If so, if you place the XML file into that folder and import it from there, does the file import (it should)*

    We can then likely tell you how to get label to import as tag (although that is, of course, a Mendeley bug)

    *Technically: you'll note the "internal-pdf" in the filepath -- Zotero guesses the filepath for internal-pdf based on the location of the XML file it is importing.
  • steffenrust
    Yes, all my PDF are in this folder. Importing the XML from there did not import the file.
  • adamsmith
    ah yes, we add a /PDF/ in there because Endnote does. Too bad.

    @emilianoeheyns does BBT do better here with importing filenames with non-ascii? Worthwhile for steffen to install that and retry import?
  • dstillman
    (If so, we'd be happy for a patch to the stock BibTeX translator. Escaped characters should obviously be decoded before filename resolution.)
  • dstillman
    (I assume there's no way to export BibTeX from Mendeley as UTF-8, as you can in Zotero, but that would obviously work too.)
  • emilianoeheyns
    BBT might because I'm passing most things through my latex translator. I'd give it a shot.
  • steffenrust
    BBT was already installed when I did the tests.
  • dstillman
    Did you try without BBT? BBT overrides the stock BibTeX translator, so it's important to say you have BBT installed when reporting something BibTeX-related.
  • steffenrust
    Sorry to not be specific enough:
    I started importing without BBT, realised the missing file links, found BBT and installed it, made further tests, and then started this discussion.

    So my impression is that BBT does not make a difference.
  • adamsmith
    I don't think we cover escaped characters. We have some dedicated code to deal with all sorts of Mendeley-introduced weirdness (https://github.com/zotero/translators/blob/master/BibTeX.js#L457 ) but I'm not seeing where we'd actually re-encode escaped characters.
  • emilianoeheyns
    If you open an issue on BBTs github I can have a look. I'm currently working on the parser I can perhaps work this in.
  • dstillman
    @adamsmith: It looks like the problem is just that we use 'rawValue' instead of 'value', which is already unescaped. I'm not sure if there's a good reason for that, but changing that block to use 'value' fixes this for me.
  • emilianoeheyns
    In which case I'd love to have a sample if BBT doesn't import it properly.
  • dstillman
    Just export an item as non–UTF-8 BibTeX that includes an attachment with an umlaut in the filename.
  • steffenrust
    The item with file
    "Oertel, Ruck - 2012 - Bioströmungsmechanik(5).pdf"
    is exported by Mendeley as

    @article{Oertel2012j,
    address = {Wiesbaden},
    author = {Oertel, Herbert and Ruck, Sebastian},
    doi = {10.1007/978-3-8348-8631-6},
    file = {:D$\backslash$:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Biostr{\"{o}}mungsmechanik(5).pdf:pdf},
    isbn = {978-3-8348-1765-5},
    keywords = {DOPPELT},
    mendeley-tags = {DOPPELT},
    publisher = {Vieweg+Teubner Verlag},
    title = {{Biostr{\"{o}}mungsmechanik}},
    url = {http://link.springer.com/10.1007/978-3-8348-8631-6},
    year = {2012}
    }

    On import to Zotero, the pdf is lost. Manually replacing "{\"{o}}" with "ö" does not help.
  • noksagt
    Most BibTeX reference manager do not escape the file field in this way.
    On import to Zotero, the pdf is lost. Manually replacing "{\"{o}}" with "ö" does not help.
    Did you also replace $\backslash$?
  • adamsmith
    (the code I link to above should handle the weird $\backslash$ that Mendeley puts in the filepath already)
  • noksagt
    edited August 17, 2019
    I also don't remember if there importer will gracefully handle the colon after the drive letter. It could be that Mendeley is trying to make the backlash serve as the escape, "\:" to make that a literal colon ("D:Users......"), while the JabRef style way of representing files uses the colon to separate elements (name,location,filetype).
  • steffenrust
    I added the missing attachment to the item manually in Zotero, exported the item using BBT biblatex:

    file = {D:\\Users\\Steffen\\Documents\\Literatur\\pdf\\Mendeley\\Artikel\\Oertel, Ruck - 2012 - Bioströmungsmechanik(5).pdf},

    and imported back into Zotero -> attachment is gone.

    I then read the exported item into Jabref, cleaned it, saved it:

    file = {D:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Bioströmungsmechanik(5).pdf:PDF},

    imported into Zotero -> attachment is gone.

    The same happens with previously successful imports from the Mendeley export.

    Meanwhile I've tried the import via the RIS format on a different machine, but Zotero hasn't finished importing 1/10 of my library in 48 hours (i7 with 16GB RAM). So I cannot report results there.
  • noksagt
    If you're relying on BBT, you should open a ticket on their github. Roundtripping (attach in Zotero, export from Zotero, import back into Zotero) should work with the stock translators, but we can't help you with how BBT is configured.

    If you've used JabRef as an intermediary, you should try opening the file from within JabRef. My suspicion is that it will not work. Unlike Zotero, JabRef makes no attend to see if there's a file there until you actually try to open the link.

    If you make the attachment in JabRef and then import the created .bib file into Zotero (again, using the stock translator), it should probably work.

    Again, my suspicion is that Mendeley really wants to point to the file in some way like this:
    file = {:D\:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Bioströmungsmechanik(5).pdf:pdf}
    Note the initial colon to separate the file name (blank) and location. Note that the second colon (after drive letter) is escaped with a backslash.

    Finally: note there are various ways to encode 'ö'. You should make sure that your browser and file explorer are able to find the file using the encoding you are using. If suggest copying and pasting from your file explorer, rather than topping it or pasting from another source.
  • steffenrust
    In JabRef the link worked.

    Exporting from Zotero w/out BBT has the same result: JabRef finds the file, Zotero does not when I reimport.
Sign In or Register to comment.