The changelog for 5.0 indicates that 'Future versions of Zotero will simplify the structure of the data directory for easier browsing from the filesystem'. This interests me: would it be possible to know more about how the planned new structure of the data directory?
@dstillman Many thanks for moving my query here, and apologies for asking it in the wrong thread. Here is why this matter interests me. I illustrate it with a few scenarios (I am currently running a Writing for Conservation workshop for early career conservation scientists from Madagascar, Comoros and Mauritius, as part of which I have introduced the use of Zotero - and the event raised many questions on this topic):
1) A Malagasy postrgraduate student would like to share her first published article via My Publications. However she can't do so because she uses a file structure outside Zotero (with Zotfile) to manage her PDFs. She does not have the financial resources to purchase extra Zotero storage (even though it is not expensive).
2) A journal editor using the ScholarOne journal management system often attaches PDFs of articles from the journal to an e-mail being sent to an author or reviewer from within the system. There is no facility to drag and drop a PDF and therefore she has to browse to the file location. This is nigh on impossible with Zotero storage, and therefore she uses Zotfile, managing PDFs with an A to Z folder structure (representing the first letter of an author's surname), within which there are subfolders of the type AbdullahM, with articles renamed AbdullahM_2017_This_is_a_title. With such a structure it is relatively easy to browse to an article from within ScholarOne, even with 1000s of PDFs.
3) A Mauritian academic has 1000s of PDFs of articles that are spread over many, many folders. He would like to use Zotero to organize them in a coherent and sensible way. However, he is deeply uncomfortable about storing them within a structure that he cannot browse to separately within the filesystem (there is also overlap here with scenario (1)).
I apologize for this long-winded explanation for my interest in potential future changes to the Zotero file structure!
FWIW, for 1, you can turn on file sync in Zotero and then temporarily disable auto-moving of attachments and add files to My Publications and sync those (which surely will be under 300MB and thus free) with Zotero. I don't know exactly what Zotero is planning, but I'd be surprised if they'd move the storage folder away from the database, so the human-readable structure wouldn't help with that.
Any reasonable solution would definitely help with 2 and possibly, depending on the exact nature of the reluctance, with 3.
Thank you @adamsmith for the suggestion regarding scenario 1. Whilst this would work, I have found during this writing workshop that people are uncomfortable with complicated solutions to what feels like a relatively simple need (i.e. to be able to browse to the folder structure within which the PDFs are stored). Zotero 5.0 is now using a folder location in an more accessible location. This is a start. A browsable folder structure (i.e. something similar to the A to Z method mentioned in my post above) within that location would, I think, work very well, and address this problem.
This is not the first time that I have come across this reluctance to store PDFs within Zotero's default structure. Many students and academics from a geographically wide range of countries with whom I have worked (and provided instruction for Zotero) over the last few years are deeply uncomfortable with the default folder structure.
@yusuf%20martin there is a solution to the key store file structure.
If you right click on an entry you get a menu with the options to 1. "Open" Of course, which usually allows dragging from the titlebar. 2. "Show File" --> This opens the keyed folder which contains your PDF
It allows you to rename files by rules FROM the meta data. So instead of a DOI, seemingly random string many journals use, etc. You end up with something like:
Goodyear - 2016 - Modeling the time-varying density distribution of .pdf
for:
Goodyear, C. P. 2016. Modeling the time-varying density distribution of highly migratory species: Atlantic blue marlin as an example. Fisheries Research 183:469–481.
And
Worm et al. - 2005 - Global Patterns of Predator Diversity in the Open .pdf
for:
Worm, B., M. Sandow, A. Oschlies, H. K. Lotze, and R. A. Myers. 2005. Global Patterns of Predator Diversity in the Open Oceans. Science 309:1365–1369.
Thanks you @BenCarr. I already use Zotfile to move and rename files in more or less the way that you recommend, and many of my students do the same.
With respect to your suggestion for scenario 1, this is not a solution because within ScholarOne the only way to attach a PDF (or any other file) is to click a browse button within ScholarOne and browse to the location. To do this using Zotero's default storage one has to do the following:
1) Open Zotero and search for the required reference. 2) Right click it and use Show File. 3) With this open in one part of the screen to show the path and the 8-letter storage location (e.g. VX87TREV) (or one writes the folder name on a piece of paper) and ScholarOne open in a browser in another part of the screen, one can then browse to the required folder. However, if one has 1000s of such folders this is a non-trivial task.
Try doing this to send a few files to somebody: it is extremely time consuming.
If one uses Zotfile and the A to Z structure and file renaming (as I described in a post above) this process is very easy: one simply needs to know the author's surname.
Ah. I'm not sure I have had the "pleasure" of interacting with ScholarOne yet; others yes, I have yet to find one that "plays nice".
I do have a simple "hack"/"workaround" for 2/ScholarOne: Use the export function: Select the articles you want in one place (you don't have to do it all at once but both shift and ctrl-click work) and export them.
The menu option is "Export Items...." Again I fully admit this is an imperfect solution, but with Zotero being 10 years old the 8 character limit was sadly still an issue for users at the time on many OSes; though some tried to abstract that for the user.
When you export your items, the "Zotero RDF" has no "cruft", select the "Export Files" option and dump them all to your a folder on your desktop (or somewhere else if you have space/backup issues)
This creates the structure (And I fully admit this is a hack):
tyr:~/tmp benc$ find /Users/benc/tmp/Exported\ Items /Users/benc/tmp/Exported Items /Users/benc/tmp/Exported Items/Exported Items.rdf /Users/benc/tmp/Exported Items/files /Users/benc/tmp/Exported Items/files/3037 /Users/benc/tmp/Exported Items/files/3037/Worm et al. - 2005 - Global Patterns of Predator Diversity in the Open .pdf /Users/benc/tmp/Exported Items/files/7355 /Users/benc/tmp/Exported Items/files/7355/Wernet - 2004 - The Structure of the First Coordination Shell in L.pdf /Users/benc/tmp/Exported Items/files/7388 /Users/benc/tmp/Exported Items/files/7388/Worm - 2006 - FICTION Armageddon in the Oceans.pdf
You can of course rename the Exported Files folder in the export dialogue. Discard the RDF file, and then as we all agree ZotFile is a good thing, your PDFs are just one folder deep and in one place. Again, a hack.
To be honest I am happy that Zotero stores the files in the filesystem at all unlike other managers that shove them in a SQLite or proprietary database.
My zotero.sqlite alone is 454MB.
If that were to die I would at least still have the flat directory structure to recover from with the files unlike others.
One problem I see with A-Z is that it is ANSI ASCII, and doesn't cover the UTF-8/16 area; simply making people less afraid of opening PDFs from me required the addition of Emiliano's Zotero Ascii Rename: https://github.com/ZotPlus/zotero-ascii-rename
As people were scare off by ASCII approximation of non-latin or even accented filenames that would end up L~&78nd-2011-Fish.pdf or some such.
I might be missing some complication here. But it seems all three requirements are easily met by using the "attach link to file" option, and storing the PDF files as per your desired folder structure in any location on your hard drive, including Dropbox type sync folders.
1) isn't (as My Publication requires attached files) and 2 and 3 require either manual attachment or ZotFile set-up as well as setting up relative links etc. Doable (and that's what the users appear to be doing now), but obviously nicer if Zotero could do this out of the box.
1) Put the My Publications paper/document PDF in a public folder inside (Dropbox, G.Drive etc) and add the URL to the citation, maybe in Extra or as an attached note?
Thank you for the various responses: which offer various possibilities for my scenarios 1 and 3 (but not 2), and all of which I am aware of and already use to some extent myself. They are not simple, however, and in providing students (e.g. scenario 1) and others (e.g. scenario 3) guidance in using Zotero it is far from optimal - and in some instances challenging - to introduce relatively complicated solutions to what could potentially be more simple.
Returning to my original point: 'Future versions of Zotero will simplify the structure of the data directory for easier browsing from the filesystem'. This interests me: would it be possible to know more about this planned new structure?
would it be possible to know more about this planned new structure?
Nothing to share at the moment, other than that the goal is to make it easier to browse the folder directly.
[ZotFile] allows you to rename files by rules FROM the meta data. So instead of a DOI, seemingly random string many journals use, etc.
Note that this exists in Zotero proper too, albeit with much less flexibility of naming options (though I'd be happy to extend that) — it happens by default when you save files via translators, and you can use Rename File from Parent Metadata for files you add directly.
The usual solution we've always recommended is to create a virtual folder in the OS that shows all PDFs within the 'storage' directory. You can access that from the file picker like any other folder and select files that way.
On macOS, you can also drag a file into the default file picker to select it, and that works not only in combination with Show File but even for dragging from Zotero directly. So for ScholarOne, you'd click "Choose File…" (or whatever), search for the file in Zotero, and just drag it into the file picker. This works for both linked and stored files and is, I would argue, far and away the best solution. I don't know if there's equivalent functionality on other OSes.
I don't know if there's equivalent functionality on other OSes.
It works on Ubuntu too (though might not have worked from Zotero before 5.0). Doesn't seem to work on Windows (which extremely unhelpfully copies the dragged file to the folder you're selecting from).
@dstillman many thanks for these further comments. I look forward to developments.
FYI, in the workshop that I have been teaching this week one student was already using Mendeley. As the students wanted to know where their PDFs are being stored in Zotero, we looked at home Mendeley manages this: one can choose where PDF files are stored (a la Zotfile) and there are some simple facilities for renaming (but not as fine grained as one can obtain with Zotfile). We discussed this matter during the workshop. There was a general consensus that the Mendeley way is far more intuitive and simpler, although they understood the advantages of using Zotero. I suspect that some of them, post-workshop, will switch to Mendeley solely for the simpler file structure, as it works out-of-the-box in a way that they intuitively expect .
Upgrade Storage
With thanks for the wonderful Zotero 5!
1) A Malagasy postrgraduate student would like to share her first published article via My Publications. However she can't do so because she uses a file structure outside Zotero (with Zotfile) to manage her PDFs. She does not have the financial resources to purchase extra Zotero storage (even though it is not expensive).
2) A journal editor using the ScholarOne journal management system often attaches PDFs of articles from the journal to an e-mail being sent to an author or reviewer from within the system. There is no facility to drag and drop a PDF and therefore she has to browse to the file location. This is nigh on impossible with Zotero storage, and therefore she uses Zotfile, managing PDFs with an A to Z folder structure (representing the first letter of an author's surname), within which there are subfolders of the type AbdullahM, with articles renamed AbdullahM_2017_This_is_a_title. With such a structure it is relatively easy to browse to an article from within ScholarOne, even with 1000s of PDFs.
3) A Mauritian academic has 1000s of PDFs of articles that are spread over many, many folders. He would like to use Zotero to organize them in a coherent and sensible way. However, he is deeply uncomfortable about storing them within a structure that he cannot browse to separately within the filesystem (there is also overlap here with scenario (1)).
I apologize for this long-winded explanation for my interest in potential future changes to the Zotero file structure!
Many thanks
Any reasonable solution would definitely help with 2 and possibly, depending on the exact nature of the reluctance, with 3.
This is not the first time that I have come across this reluctance to store PDFs within Zotero's default structure. Many students and academics from a geographically wide range of countries with whom I have worked (and provided instruction for Zotero) over the last few years are deeply uncomfortable with the default folder structure.
Again, with thanks.
If you right click on an entry you get a menu with the options to
1. "Open" Of course, which usually allows dragging from the titlebar.
2. "Show File"
--> This opens the keyed folder which contains your PDF
I also highly recommend the ZotFile extension:
https://github.com/jlegewie/zotfile
It allows you to rename files by rules FROM the meta data. So instead of a DOI, seemingly random string many journals use, etc. You end up with something like:
Goodyear - 2016 - Modeling the time-varying density distribution of .pdf
for:
Goodyear, C. P. 2016. Modeling the time-varying density distribution of highly migratory species: Atlantic blue marlin as an example. Fisheries Research 183:469–481.
And
Worm et al. - 2005 - Global Patterns of Predator Diversity in the Open .pdf
for:
Worm, B., M. Sandow, A. Oschlies, H. K. Lotze, and R. A. Myers. 2005. Global Patterns of Predator Diversity in the Open Oceans. Science 309:1365–1369.
With respect to your suggestion for scenario 1, this is not a solution because within ScholarOne the only way to attach a PDF (or any other file) is to click a browse button within ScholarOne and browse to the location. To do this using Zotero's default storage one has to do the following:
1) Open Zotero and search for the required reference.
2) Right click it and use Show File.
3) With this open in one part of the screen to show the path and the 8-letter storage location (e.g. VX87TREV) (or one writes the folder name on a piece of paper) and ScholarOne open in a browser in another part of the screen, one can then browse to the required folder. However, if one has 1000s of such folders this is a non-trivial task.
Try doing this to send a few files to somebody: it is extremely time consuming.
If one uses Zotfile and the A to Z structure and file renaming (as I described in a post above) this process is very easy: one simply needs to know the author's surname.
With thanks!
I do have a simple "hack"/"workaround" for 2/ScholarOne:
Use the export function:
Select the articles you want in one place (you don't have to do it all at once but both shift and ctrl-click work) and export them.
The menu option is "Export Items...."
Again I fully admit this is an imperfect solution, but with Zotero being 10 years old the 8 character limit was sadly still an issue for users at the time on many OSes; though some tried to abstract that for the user.
When you export your items, the "Zotero RDF" has no "cruft", select the "Export Files" option and dump them all to your a folder on your desktop (or somewhere else if you have space/backup issues)
This creates the structure (And I fully admit this is a hack):
tyr:~/tmp benc$ find /Users/benc/tmp/Exported\ Items
/Users/benc/tmp/Exported Items
/Users/benc/tmp/Exported Items/Exported Items.rdf
/Users/benc/tmp/Exported Items/files
/Users/benc/tmp/Exported Items/files/3037
/Users/benc/tmp/Exported Items/files/3037/Worm et al. - 2005 - Global Patterns of Predator Diversity in the Open .pdf
/Users/benc/tmp/Exported Items/files/7355
/Users/benc/tmp/Exported Items/files/7355/Wernet - 2004 - The Structure of the First Coordination Shell in L.pdf
/Users/benc/tmp/Exported Items/files/7388
/Users/benc/tmp/Exported Items/files/7388/Worm - 2006 - FICTION Armageddon in the Oceans.pdf
You can of course rename the Exported Files folder in the export dialogue. Discard the RDF file, and then as we all agree ZotFile is a good thing, your PDFs are just one folder deep and in one place. Again, a hack.
To be honest I am happy that Zotero stores the files in the filesystem at all unlike other managers that shove them in a SQLite or proprietary database.
My zotero.sqlite alone is 454MB.
If that were to die I would at least still have the flat directory structure to recover from with the files unlike others.
One problem I see with A-Z is that it is ANSI ASCII, and doesn't cover the UTF-8/16 area; simply making people less afraid of opening PDFs from me required the addition of Emiliano's Zotero Ascii Rename: https://github.com/ZotPlus/zotero-ascii-rename
As people were scare off by ASCII approximation of non-latin or even accented filenames that would end up L~&78nd-2011-Fish.pdf or some such.
2 and 3 require either manual attachment or ZotFile set-up as well as setting up relative links etc. Doable (and that's what the users appear to be doing now), but obviously nicer if Zotero could do this out of the box.
Returning to my original point: 'Future versions of Zotero will simplify the structure of the data directory for easier browsing from the filesystem'. This interests me: would it be possible to know more about this planned new structure?
With thanks.
The usual solution we've always recommended is to create a virtual folder in the OS that shows all PDFs within the 'storage' directory. You can access that from the file picker like any other folder and select files that way.
On macOS, you can also drag a file into the default file picker to select it, and that works not only in combination with Show File but even for dragging from Zotero directly. So for ScholarOne, you'd click "Choose File…" (or whatever), search for the file in Zotero, and just drag it into the file picker. This works for both linked and stored files and is, I would argue, far and away the best solution. I don't know if there's equivalent functionality on other OSes.
FYI, in the workshop that I have been teaching this week one student was already using Mendeley. As the students wanted to know where their PDFs are being stored in Zotero, we looked at home Mendeley manages this: one can choose where PDF files are stored (a la Zotfile) and there are some simple facilities for renaming (but not as fine grained as one can obtain with Zotfile). We discussed this matter during the workshop. There was a general consensus that the Mendeley way is far more intuitive and simpler, although they understood the advantages of using Zotero. I suspect that some of them, post-workshop, will switch to Mendeley solely for the simpler file structure, as it works out-of-the-box in a way that they intuitively expect .