Mendeley -> Zotero via BibTeX: links to attached files are lost

2»
  • Which link worked in JabRef? You've given a lot of different links.

    When I use the stock Zotero to export a BibTeX containing both a stored file and a linked file, I get the following when I leave "Export Files" unselected:
    @book{osmith_stored_2019,
    title = {Stored copy of file},
    author = {öSmith, John},
    year = {2019},
    file = {öSmith - 2019 - Stored copy of file.pdf:C\:\\Users\\noksagt\\Zotero\\storage\\7H24VG95\\öSmith - 2019 - Stored copy of file.pdf:application/pdf}
    }

    @book{smith_linked_2019,
    title = {Linked copy of file},
    author = {Smith, John},
    year = {2019},
    file = {ö.pdf:C\:\\temp\\ö.pdf:application/pdf}
    }

    I get the following when I select "Export Files":

    @book{osmith_stored_2019,
    title = {Stored copy of file},
    author = {öSmith, John},
    year = {2019},
    file = {öSmith - 2019 - Stored copy of file.pdf:files/15/öSmith - 2019 - Stored copy of file.pdf:application/pdf}
    }

    @book{smith_linked_2019,
    title = {Linked copy of file},
    author = {Smith, John},
    year = {2019},
    file = {ö.pdf:files/4/ö.pdf:application/pdf}
    }
    All of these import successfully into Zotero. All of these are also links that JabRef is able to open.

  • edited August 17, 2019
    file = {C:\\temp\\ö.pdf}
    file = {C:/temp/ö.pdf:PDF}

    Either style of link (the first is exported from BBT on Windows when you have "Export Files" unselected and the second is the style you said you edited manually), will not import into either stock Zotero. If you try to open either style of link in JabRef, it won't find the file. That is because the unescaped drive letter colon is understood by both programs to be a field separator. I created a BBT issue. Perhaps you can submit a Report ID and/or provide any other information in that issue.

    BetterBibTex actually does the correct thing if "Export Files" is selected.

    Either of these work in both Zotero and JabRef, though:
    file = {C\:\\temp\\ö.pdf}
    file = {:C\:/temp/ö.pdf:PDF}
  • @dstillman with
    Just export an item as non–UTF-8 BibTeX that includes an attachment with an umlaut in the filename.
    you mean something like

    @article{Oertel2012j,
    address = {Wiesbaden},
    author = {Oertel, Herbert and Ruck, Sebastian},
    doi = {10.1007/978-3-8348-8631-6},
    file = {:D$\backslash$:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Biostr{\"{o}}mungsmechanik(5).pdf:pdf},
    isbn = {978-3-8348-1765-5},
    keywords = {DOPPELT},
    mendeley-tags = {DOPPELT},
    publisher = {Vieweg+Teubner Verlag},
    title = {{Biostr{\"{o}}mungsmechanik}},
    url = {http://link.springer.com/10.1007/978-3-8348-8631-6},
    year = {2012}
    }

    ?
  • edited August 21, 2019
    That D$\backslash$: is simply wrong BTW. file is a verbatim field so that literally just means D$\backslash$:, not D\:.
  • I guess that also answers the question on why BBT doesn't import files with things like \"o in it. This is unfortunately not easy to change in BBTs bibtex parser. I'll give it some thought.
  • @steffenrust what OS are you on? Would it be feasible for you to change the file field name into files?
  • you mean something like
    No — that's from Mendeley, right? I'm just saying, to reproduce the problem with extended characters in filenames, you can just export from Zotero as non–UTF-8 BibTeX and reimport. As I say above, the fix for the stock translator appears to be to use value instead of rawValue, but I don't know the other implications of that.

    It'd be helpful if any discussion about BBT could move to the BBT GitHub tracker and we could keep this thread about the stock translator — this is getting too confusing.
  • edited August 23, 2019
    Sorry for any contribution to the confusion.

    In different messages, the OP pointed out what I would consider two different bugs in the way Mendeley and Better BibTeX were sometimes exporting links for files on Windows. emiliano has patched the latter.

    The stock translator and BBT both export correctly formatted links on Windows now & either translator is able to import from itself or from the other.

    Mendeley's exported links are still broken:
    file = {:D$\backslash$:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Biostr{\"{o}}mungsmechanik(5).pdf:pdf},

    Dan is correct that the current use of raw value mean that field is uncorrected. The transform leads to no changes:
    :D$\backslash$:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Biostr{\"{o}}mungsmechanik(5).pdf:pdf
    For me, replacing rawValue with value fixes the umlaut and keeps the drive colon escaped, but does not seem to make a resolvable path:
    :D\:/Users/Steffen/Documents/Literatur/pdf/Mendeley/Artikel/Oertel, Ruck - 2012 - Bioströmungsmechanik(5).pdf:pdf
    Is this because of the forward slashes used?
  • If you open an issue on github, I can get this resolved for BBT. I have an idea on how to approach it.
  • edited August 28, 2019
    @steffenrust I have a fix in BBT, I'd really appreciate it if you can drop by on github and help me test this. I don't have access to a Windows machine.
  • The stock translator and BBT both export correctly formatted links on Windows now & either translator is able to import from itself or from the other.
    That's not the case, though, at least for the stock translator, until the value/rawValue thing is fixed. If you export from Zotero as Western, this is what you get:

    file = {F{\"u}bar.jpg:files/22/F{\"u}bar.jpg:image/jpeg}

    I don't know if that's correct or not, but if you import that back, the file isn't included with the translator unless rawValue is changed to to value.
  • For BBT I have a fixed lined up, but I'd really appreciate some help testing for windows.
  • edited October 26, 2019
    I don't know if that's correct or not
    It's probably not (seeing as how the file field is verbatim) but it's also probably an acceptable compromise when targeting something that doesn't grok utf-8. For bibtex exports, this is going to be a compromise one way or the other.

    BBT can now be told whether file should be treated as verbatim on import. I'm still pondering whether to just fix this behavior to "no", or what to set the default to if not fixed.
  • edited October 27, 2019
    I also came across the problem with the tex-coding of special characters (umlauts and accents) in the file names when exporting a large Mendeley database to BibTeX and then trying to import these to Zotero. I tried both the stock importer and Better-BibTeX (Zotero 5.0.76, BBTex 5.1.145). As I use linux, I do not have the $backslash$ problem.

    Is there a fix in the meantime?

    By the way, the problem does not only affect the file field, but also the url field, where Mendeley for example encodes & as {\&}.
  • Importing shouldn't pose any problem for BBT, but in order to trigger the Mendeley-compatible behavior (which, to reiterate, is against spec, but I can envision a reason for that compromise) in BBT you either have to rename the file field to files or remove file from the hidden preference verbatimFields.

    As to the way Mendeley exports that URL, that's just simply and inexcusably wrong. You can make BBT import it by likewise removing url from verbatimFields<rant>, but that one really shows the Mendeley team hasn't bothered to ever open the bib(la)tex manuals. Aside the fact that the url field is not supported by most bibtex styles so should be seen as out-of-spec for bibtex, and that in biblatex it is a verbatim field so should not be escaped, it is at the same time the dumbest way I've seen & exported, because even if it were a supported field, and not a verbatim field, the braces are simply unnecessary, and if they'd omitted that, BBT would already have imported it as you'd wanted to, because I skip backslashes in URLs (because too many bibtex files have them to escape stuff like this).</rant>

  • Thanks for the quick reply @emilianoeheyns. I have also contacted Mendeley support of course, but I fear that they will not have a fix soon. Obviously, they simply encode all fields in TeX-Syntax, which is complete bogus, without even thinking about it. Guess, why I am trying to convince a whole institute to switch to zotero and thus need to convert several large databases.

    I removed file and url from verbatimFields, which seems to properly handle umlauts and accents.

    However, he still cannot handle all filenames. For example, file = {:.../Barreau et al. - 2009 - Impact of {\{}Cu{\}}-rich growth on the {\{}CuIn{\_}{\{}1-x{\}}Ga{\_}xSe{\_}2{\}} surface morphology and related solar cells behaviour.pdf:pdf} fails. The actual filename is Barreau et al. - 2009 - Impact of {Cu}-rich growth on the {CuIn_{1-x}Ga_xSe_2} surface morphology and related solar cells behaviour.pdf so the extra curly braces are the problem here as well. Do you see a possibility to catch that?

    In the url field, he now imports {\&} as <span class="nocase">&</span>.
  • For the URL field, turn on https://retorque.re/zotero-better-bibtex/installation/preferences/hidden-preferences/#suppressnocase . That's going to be on by default in the next release anyhow.

    For the problem with the extra braces I'll need an actual sample for diagnosis. You can open an issue on the BBT issue tracker for that.
  • Thanks, suppresnocase = True actually seems to have done the job also for the braces, which I realized when doing the debug for an BBT issue. Now, using also zotero's `Find available PDF` option, I have more PDFs in the database than previously with mendeley.
  • I'll see whether it's possible to do a "mendeley mode" without polluting my parser too much.
  • If you need some "dirty" BibTeX files for testing, let me know ;-)

    The hidden options work for me in the meantime.

    It's really incredible what Mendeley does both intentionally and through sheer ignorance to make interoperability with other programs a pain in the a*.
  • That would actually be helpful, specifically if the mess could be constrained to a one- or two-entry bibtex file, plus the attachment file. Preferably zipped and attached to an issue on github.
  • Is there anything in the bibtex file that would identify it as being generated by Mendeley?
  • Doesn't look like it. Shame.
Sign In or Register to comment.