Import PDFs linked in BibDesk file into Zotero

mpavsic

Hello,

Is it possible to import PDFs alongside references from BibDesk file to Zotero? In the BibDesk file the PDFs are linked in the form of Bdsk-File (there is also Bdsk-Url field, however it is not needed). I guess this needs some kind of conversion to a regular file link, however haven't find any solution to this yet.

Thank you in advance.
Miha

adamsmith

Could you post a bibdesk file (that's be bibtex?) here -- we can certainly take a look.

mpavsic

Sure! Here is a sample file with two entries. I guess in the previous version BibDesk used some URL scheme, but now this Bdsk link is different and uses macOS file alias. In the example below I've just shortened the abstract text to "ABSTRACT".
Thanks!


*****

%% This BibTeX bibliography file was created using BibDesk.
%% https://bibdesk.sourceforge.io/
%% Saved with string encoding Unicode (UTF-8)

@article{Senior:2020aa,
Abstract = {ABSTRACT},
Author = {Senior, Andrew W and Evans, Richard and Jumper, John and Kirkpatrick, James and Sifre, Laurent and Green, Tim and Qin, Chongli and {\v Z}{\'\i}dek, Augustin and Nelson, Alexander W R and Bridgland, Alex and Penedones, Hugo and Petersen, Stig and Simonyan, Karen and Crossan, Steve and Kohli, Pushmeet and Jones, David T and Silver, David and Kavukcuoglu, Koray and Hassabis, Demis},
Date-Added = {2020-01-30 16:19:59 +0100},
Date-Modified = {2020-01-30 16:19:59 +0100},
Doi = {10.1038/s41586-019-1923-7},
Journal = {Nature},
Journal-Full = {Nature},
Month = {Jan},
Pmid = {31942072},
Pst = {aheadofprint},
Title = {Improved protein structure prediction using potentials from deep learning},
Year = {2020},
Bdsk-Url-1 = {https://doi.org/10.1038/s41586-019-1923-7},
Bdsk-File-1 = {YnBsaXN0MDDSAQIDBFxyZWxhdGl2ZVBhdGhZYWxpYXNEYXRhXxATU2VuaW9yXzIwMjBfYWFhLnBkZk8RAXoAAAAAAXoAAgAABGJpY2kAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABCRAAB/////xNTZW5pb3JfMjAyMF9hYWEucGRmAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/////AAAAAAAAAAAAAAAAAAEAAgAACiBjdQAAAAAAAAAAAAAAAAAMQmliRGVza190ZXN0AAIAQS86VXNlcnM6bWloYTpEcm9wYm94Ol9OYWJpcmFsbmlrOkJpYkRlc2tfdGVzdDpTZW5pb3JfMjAyMF9hYWEucGRmAAAOACgAEwBTAGUAbgBpAG8AcgBfADIAMAAyADAAXwBhAGEAYQAuAHAAZABmAA8ACgAEAGIAaQBjAGkAEgA/VXNlcnMvbWloYS9Ecm9wYm94L19OYWJpcmFsbmlrL0JpYkRlc2tfdGVzdC9TZW5pb3JfMjAyMF9hYWEucGRmAAATAAEvAAAVAAIAC///AAAACAANABoAJAA6AAAAAAAAAgEAAAAAAAAABQAAAAAAAAAAAAAAAAAAAbg=}}

@article{Pavsic:2014aa,
Abstract = {ABSTRACT},
Author = {Pav{\v s}i{\v c}, Miha and Gun{\v c}ar, Gregor and Djinovi{\'c}-Carugo, Kristina and Lenar{\v c}i{\v c}, Brigita},
Date-Added = {2020-01-30 16:19:18 +0100},
Date-Modified = {2020-01-30 16:19:18 +0100},
Doi = {10.1038/ncomms5764},
Journal = {Nat Commun},
Journal-Full = {Nature communications},
Mesh = {Antigens, Neoplasm; Cathepsin L; Cell Adhesion Molecules; Cell Membrane; Crystallography, X-Ray; Epithelial Cell Adhesion Molecule; Epitopes; Glycosylation; Humans; Models, Molecular; Mutation; Protein Conformation; Protein Multimerization; Protein Stability; Protein Structure, Tertiary},
Month = {Aug},
Pages = {4764},
Pmid = {25163760},
Pst = {epublish},
Title = {Crystal structure and its bearing towards an understanding of key biological functions of EpCAM},
Volume = {5},
Year = {2014},
Bdsk-Url-1 = {https://doi.org/10.1038/ncomms5764},
Bdsk-File-1 = {YnBsaXN0MDDSAQIDBFxyZWxhdGl2ZVBhdGhZYWxpYXNEYXRhbxAVAFAAYQB2AHMDDABpAGMDDABfADIAMAAxADQAXwBhAGEAYQAuAHAAZABmTxEBhgAAAAABhgACAAAEYmljaQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEJEAAH/////EFBhdiNGRkZGRkZGRi5wZGYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAQACAAAKIGN1AAAAAAAAAAAAAAAAAAxCaWJEZXNrX3Rlc3QAAgBFLzpVc2VyczptaWhhOkRyb3Bib3g6X05hYmlyYWxuaWs6QmliRGVza190ZXN0OlBhdnPMjGljzIxfMjAxNF9hYWEucGRmAAAOACwAFQBQAGEAdgBzAwwAaQBjAwwAXwAyADAAMQA0AF8AYQBhAGEALgBwAGQAZgAPAAoABABiAGkAYwBpABIAQ1VzZXJzL21paGEvRHJvcGJveC9fTmFiaXJhbG5pay9CaWJEZXNrX3Rlc3QvUGF2c8yMaWPMjF8yMDE0X2FhYS5wZGYAABMAAS8AABUAAgAL//8AAAAIAA0AGgAkAFEAAAAAAAACAQAAAAAAAAAFAAAAAAAAAAAAAAAAAAAB2w==}}

adamsmith

@dstillman @bwiernik do you happen to recognize the type of string in the Bdsk-File-1 field?

Bibdesk refers to them as MacOS aliases: https://sourceforge.net/p/bibdesk/wiki/FAQ/#what-are-all-these-bdsk-file-and-bdsk-url-fields-that-appear-in-my-bibtex-file
Is there any way we can convert them to something useable?

emilianoeheyns

Doesn't look promising: https://sourceforge.net/p/bibdesk/mailman/message/20948649/

edit: maybe https://gist.github.com/sprig/851295c454b0b6d1d545a4068a9d3428 , but that uses CoreServices, so MacOS only.

mpavsic

Dear all, thanks for the suggestions! During the weekend I've been experimenting a little bit and found a solution. Briefly, to import the bib library along with the files to Zotero:

1. Set BibDesk to rename (autofile) the PDFs (and other attached files - careful when importing in Zotero/Zotfile to that also non-PDFs are handled) to something simple without any non-English characters, i.e. YEAR+AAAA+few unique characters.

2. Install the LinkedFilesToFields.applescript from https://github.com/andriusvelykis/bibdesk-scripts into BibDesk (i.e., copy it to the Library/Application Support/BibDesk/Scripts).
Before that I suggest to edit it (instructions are in the comments within the script file) so that the field for file path will contain absolute path (the relative starting with ./ is not recognized by Zotero, at least it did not function for me) and that the path field name is "File". For multiple files linked to the same reference the fields will then be File, File-2, File-3 etc.

3. Run the abovementioned script from within BibDesk, save and close the bib file.

4. Open the bib file in, for example, TextEdit and rename any File-2, File-3 etc. fields to File. Only this way the Zotero will recognize all the attached filenames. Save the edited file.

5. Import the file into Zotero (enable import of linked files). If Zotfile is installed and configured the PDFs under the File field will be imported and renamed according to the set rules. Only the files that were previously linked under File-2, File-3 etc. fields won't be omported into Zotfile folder - but that is why there is AAAA in the filename of them. One can do a search of AAAA within the library, select all the files that have not been renamed yet, and trigger renaming from the Zotfile menu. Voila!

emilianoeheyns

can you get me a sample of such a file pre-field-rename? So as you'd have it after the LinkedFilesToFields.applescript?

mpavsic

Sure - below. Abstracts are shortened to ABSTRACT.

***
%% This BibTeX bibliography file was created using BibDesk.
%% https://bibdesk.sourceforge.io/

%% Saved with string encoding Unicode (UTF-8)

@article{Senior:2020aa,
Abstract = {ABSTRACT.},
Author = {Senior, Andrew W and Evans, Richard and Jumper, John and Kirkpatrick, James and Sifre, Laurent and Green, Tim and Qin, Chongli and {\v Z}{\'\i}dek, Augustin and Nelson, Alexander W R and Bridgland, Alex and Penedones, Hugo and Petersen, Stig and Simonyan, Karen and Crossan, Steve and Kohli, Pushmeet and Jones, David T and Silver, David and Kavukcuoglu, Koray and Hassabis, Demis},
Date-Added = {2020-01-30 16:19:59 +0100},
Date-Modified = {2020-02-03 22:42:27 +0100},
Doi = {10.1038/s41586-019-1923-7},
File = {/Users/miha/Dropbox/_Nabiralnik/BibDesk_test/2020_aaaaaaaa.pdf},
Journal = {Nature},
Journal-Full = {Nature},
Month = {Jan},
Pmid = {31942072},
Pst = {aheadofprint},
Remote-Url = {https://doi.org/10.1038/s41586-019-1923-7},
Title = {Improved protein structure prediction using potentials from deep learning},
Year = {2020}}

@article{Pavsic:2014aa,
Abstract = {ABSTRACT},
Author = {Pav{\v s}i{\v c}, Miha and Gun{\v c}ar, Gregor and Djinovi{\'c}-Carugo, Kristina and Lenar{\v c}i{\v c}, Brigita},
Date-Added = {2020-01-30 16:19:18 +0100},
Date-Modified = {2020-02-03 22:42:27 +0100},
Doi = {10.1038/ncomms5764},
File = {/Users/miha/Dropbox/_Nabiralnik/BibDesk_test/2014_aaaaaaaa.pdf},
File-2 = {/Users/miha/Dropbox/_Nabiralnik/BibDesk_test/2014_aaaaaaab.pdf},
Journal = {Nat Commun},
Journal-Full = {Nature communications},
Mesh = {Antigens, Neoplasm; Cathepsin L; Cell Adhesion Molecules; Cell Membrane; Crystallography, X-Ray; Epithelial Cell Adhesion Molecule; Epitopes; Glycosylation; Humans; Models, Molecular; Mutation; Protein Conformation; Protein Multimerization; Protein Stability; Protein Structure, Tertiary},
Month = {Aug},
Pages = {4764},
Pmid = {25163760},
Pst = {epublish},
Remote-Url = {https://doi.org/10.1038/ncomms5764},
Title = {Crystal structure and its bearing towards an understanding of key biological functions of EpCAM},
Volume = {5},
Year = {2014}}

emilianoeheyns

What's "mesh" and "pst"?

mpavsic

Mesh is Medical Subject Headings of Pubmed (https://www.ncbi.nlm.nih.gov/mesh); it is automatically added to BibDesk entries from Pubmed searches.
For Pst - I don't know, but I certainly didn't put it there manually.

emilianoeheyns

Thanks. Next version of BBT will import this as posted (but if you rename the file fields that's OK too)

emilianoeheyns

those values of "epublish" and "aheadofprint" don't correspond to anything in the bibdesk UI?

mpavsic

@emilianoeheyns Thanks, fantastic!

Btw, here I was using Zotero 5.0.82, Better BibTeX 5.2.14 and Zotfile 5.0.15.

BibDesk UI shows those as additional fields.

emilianoeheyns

There's no label in front of them?

mpavsic

The label is the same as in the bib file. All this info is imported into bib when downloading entries from Pubmed directly from within BibDesk or via some reference export (RIS) from websites which are then imported into BibDesk.

emilianoeheyns

Alright. I'll cut a new release when I have cleared #1442 and #1391, and the change that the above will import attachments will be included.

edsu

I ran into this problem too and after reading this thread and a bit of research wrote a little Python utility that would rewrite the Bdesk-File-n entries as File entries after base64 decoding them and parsing the plist data:

https://pypi.org/project/bibdesk2zotero/

emilianoeheyns

Can you get me a sample of a file with such an Bdesk-File-n? I'll add support to BBT.

aaaaaaaaaaaaaaa

How would one achieve the reverse of this? That is, import PDFs linked in Zotero into BibDesk? I use ZotFile and BetterBibTeX for Zotero, and I'm trying to point BibDesk to the Zotero library because BibDesk provides some macOS native scripting features that are lacking in Zotero.

Better BibTeX generates a .bib file from my Zotero library, which can then be opened in BibDesk. This imports most of the reference info and metadata but BibDesk doesn't 'see' the links to the PDF files. The exported .bib has information about the linked files, formatted as an absolute path associated with the `file` field for each reference.

I haven't tried Zot2Bib yet because it isn't updated for Zotero 6. Looks like it could be promising if updated by @gmac or another community member.

tothzit

I was looking for this too. Will report back how it goes.

To @aaaaaaaaaaaaaaa 's question: I was experimenting (want to preserve the 2-way compatibility in case I change my mind). What worked for me is to open the exported .bib file in a text editor and replace all occurrences of 'file' with 'bdsk-file-1'. When more than 1 files, that will be a problem with this approach, but I didn't replace those and will just handle them as they come.

aaaaaaaaaaaaaaa

Thanks for the update, @tothzit. I didn't realize the `bdsk-file-1` field works with regular (non-base64 encoded alias) file paths as well!

I've started a discussion about this on GitHub and @emilianoeheyns has provided a code snippet that almost automates the process you described. But at the moment it still doesn't work as expected. I'll report back here if we're able to work out the kinks, and in the meantime you can join in or follow along here: https://github.com/retorquere/zotero-better-bibtex/issues/2181#issuecomment-1156051891

EDIT: Here is the working code from Emiliano. In the Better BibTeX pane of Zotero's settings, navigate to the Export section and then the postscript panel. Then paste this:

if (Translator.BetterTeX && item.attachments) {
item.attachments.forEach((att, i) => {
entry.add({ name: `bdsk-file-${i + 1}`, value: (att.saveFile ? att.defaultPath : '') || att.localPath, enc: 'verbatim' })
})
}

tothzit

Great, thanks! Will try it, but the end of the day, I may just try to maintain the 2 parallel; not ideal and will see how it goes.
(I'm really just starting with Zotero. Used it a long time ago, then abandoned it and now returned mostly because of the iOS support, which Bibdesk & co. sorely lacks.)

EDIT: this is GREAT, thanks @aaaaaaaaaaaaaaa (and emiliano). This works for me perfectly. If I file something in Zotero, the relevant entry in the .bib file also ends up pointing to the same attachment.

zbiener

Just FYI, importing a bib file into Zotero, following @mpavsic tips above, seems to ignore file paths that have ";" in them. This is on a mac, where the character is allowed in file paths.

I don't know how deep this goes and whether it is a BBT or Zotero issue, but if it is an easy fix, it might be worthwhile.

emilianoeheyns

If someone could get me a sample in a BBT github issue I could have a look.

zbiener

Will do. Thx.