Creating My Own File Structure for Zotero

Hello, I'm a 3-days newbie to Zotero and a Mendeley convert ;)
Zotero has given me 3 days of headache trying to figure out how to work with it, but I surely hope that this will benefit me in the long run...

So my problem is, I don't like how Zotero stored PDF attachments and everything in unreadable subfolders. I want to be able to neatly see my PDF collections in one place, just like Mendeley, and be able to access them from anywhere.

I figured out how to create Symbolic Links and move the Zotero storage to my Dropbox, but I'm still in the dark of how to bypass the unreadable subfolders problem.

I've tried to search, found out about Zotfile and how it's supposed to be able to "move new & existing attachments to different locations", but I can't make it work, and I can't find the foolproof step-by-step instruction anywhere.

And Zotero itself said something about "setting the base directory" so that the attachment links can work on different computers; but I can't seem to find any explanation on how this work with Dropbox and symbolic links. I can't even find a clear explanation about what's the difference between Data Directory and Base Directory.

My working environment consists of a PC, two Macs, and an iPad; I'm hoping that I can read & annotate PDF in my iPad and have them available back on my computers. I installed ZotPad on iPad, but I don't know whether I should use Zotfile's "Send to Tablet" or is it enough to set up Goodreader to sync with my Dropbox folder containing the PDFs. Which takes me back to the first problem: how to neatly store my PDFs in one folder in Dropbox while make them available in Zotero's Library.

Please kindly provide your help... I really want to be able to use this software, but my head really hurts right now >,<
(Sorry for being kinda emotional ^^;)
«1
  • I can offer you a few tips:

    First, a question. Why do you want to have all of your pdfs in one folder? What is the use case of this? I find that the value of using a reference manager like Zotero is that I can search for pdfs, notes, annotations, etc. without having to ever interact with a native file structure. Especially with what you are describing, I don't think I see any value of directly interacting with the pdf storage folder.

    Quick Clarification: The "Data Directory" is where the database file that has all of the reference information (not the attachments) for your library is stored. Unless you have a compelling reason to store this somewhere other than the default, leave it there--you never have to interact with this outside of Zotero.

    The "Attachment Base Directory" feature makes it easier to sync across multiple computers if you link to attachments, rather than storing them directly in Zotero. If you don't do this, you won't use this feature. I recommend storing them directly in Zotero (where you will see the little pdf symbol next to the attachment, rather than the chain link symbol). I will explain these a bit more below.

    You also seem confused by the fact that there are multiple different tools available for using a tablet with Zotero. Zotfile "Send to Tablet" and ZotPad are separate solutions; you aren't intended to use both.

    Here is what I recommend for your desired uses:
    1. Store your pdfs as Attached Files in Zotero, not linked files. Sync them using Zotero storage or a WebDAV-compatible cloud storage. I recommend Zotero storage--buying extra space is very cheap and supports further development. If you need a free solution, I suggest CloudMe or Box.

    2. Use the ZotPad or Papership apps on your iPad. Papership is newer and has a nicer interface and more features. However, it doesn't presently support group libraries. If you need that, use ZotPad. With either of these apps, you can search your library using the Zotero interface and find your pdfs. You can then annotate them and save the annotated files in the app, all without ever having to deal with the file storage structure. You can annotate pdfs directly in Papership. With ZotPad, you send them to Goodreader, annotate them, then select the "Send Back" command in the Share menu in Goodreader.


    I don't recommend using Dropbox for storing your pdfs. While Dropbox is excellent for its intended use of syncing your working files across computers, it does not play well with other apps or services, like Zotero. It doesn't support WebDAV, which is the most stable way to deal with file syncing in Zotero besides using Zotero storage.

    If you absolutely have to use Dropbox, here are the steps you follow:
    1. Do not make a symbolic link to move your Zotero storage folders to Dropbox. This method bypasses the Zotero storage system.

    2. In Zotero preferences, under "Advanced", "Files and Folders", revert to Absolute Paths. You will be using Zotfile to manage your storage path, not this feature.

    3. In the Zotfile preference, in the general settings pane, under "Location of Files", select Custom Location, and select the folder where you will store your pdfs in Dropbox. You can tell it to create subfolders organized by year, author, journal, etc. if desired (read the instructions by hovering over the (i) button). This setup will automatically change attachments to links that direct the pdfs to the folder you select.

    4. Use Zotpad on your iPad. You need to go into the Zotpad settings in the Settings app. Under "File Service", select "Dropbox". Click "Advanced Dropbox Settings". Change the "Location of files in Dropbox" to whatever you set it in Zotfile on your computer. Set the "Custom Filename Settings" to match exactly the settings in the "Renaming Rules" pane of the Zotfile preferences on your computer. If you set up any journal/author/year subfolders in Zotfile on your computer, you need to set these settings exactly the same in the "Subfolder pattern" field.

    5. To annotate a pdf, search for it in Zotpad, send the pdf to Goodreader, annotate it, and use the "Send Back" command in the Share menu in Goodreader. Do not set up your Goodreader app to access the pdf storage folder directly.

    If the previous section was confusing, I understand. It is rather complicated to set up. That is why I recommend using the sync systems that Zotero has built in (Zotero storage or WebDAV).

    Hope this cleared some things up a bit for you.
  • Agree with everything that bwiernik says except step 2.) of the instructions to use Zotero with Zotfile and Dropbox. If you do want this working of different computers you should, indeed, use relative paths and set the base directory to point to the dropbox folder on each of your computers.
  • I am also a new user of Zotero and first of all I want to say it is a wonderful tool! It has really changed my work

    The only problem I have, is the issue described in this post: To find an efficient way to combine Zotero with tablets.

    This is a problem which seems to be in the forums for quite a few years. However, myself, I think that it solution lies elsewhere, I would really appreciate it if bwiernik and adamsmith could help us:

    ---We need a plugin to save articles, automatically, in anything but the "random string"---

    Why do I think this is the solution for our issues? These are my arguments:

    1) In my case, I use an android tablet. I have my zotero library/data folder in dropbox. To sync my dropbox with android you can use dropsync (which provides two-way sync in tables), you can also use foldersync app for all the other cloud services. However, because all the article folders in zotero have a random name... you cannot consult them efficiently...

    2) In ios devices this is even harder... because it does not have a shared folder structure... However, some apps, such as goodreader or iannotate can automatically like your dropbox folder everytime you launch... However, again, because of the random folder name, it is difficult to consult your articles...

    3) In windows, linux or macos computers, this is a smaller issue because you can directly use zotero, qnotero and also the file explorers have an inbuilt search option... However, it is still far from efficient....

    Plugins such as zotfile can rename the articles pdfs to anything you want but not the folders which contain them....

    I could easily write a code which can rename the folders random name to the articles name or keyname, for example, but this would mess the zotero library...

    So I ask again, is there anyway to change the <randomstring> in the zotero file structure? Also, I know I can like the files to the location I want but this breaks the purpose of zotero great autodownload features.

    Please any advice would be most welcome.
  • Plugins such as zotfile can rename the articles pdfs to anything you want but not the folders which contain them....
    that's incorrect. Zotfile can move your files wherever you want it to as bwiernik explains in step 3.

    That said - if you leave the files in Zotero's structure and use Zotero's storage or a webDAV sync service, you can access the files on your iOS device using free apps like ZotPad and PaperShip, so tablet use is not a reason to try to avoid Zotero's file structure. On Android that's less well supported, but I believe you should be able to do this with the zotero reader sample webpage https://secure.zoteroreader.com/ (which the developer has announced he is developing into an app)
  • Hi adamsmith: Thank you very much for the reply! However, I still have these concerns:

    1) With zotfile you can move the files... But you have to do it manually, no? Each time you download a new you need to select it and click on "send to tablet" to move them to knew folder... I need to go back to the zotfile manual... But so far, it has been creating a copy of each pdf and it does not move the other attachments from the article...

    2) I am very happy with zotero it is an awesome app: That is why I would like to combine it with other great apps: There are awesome pdf readers in android to view your articles and modify them. I will take a look at this zotero web reader...

    I hope you agree with me it is "weird" to use a randomstring to generate folder names... We would not even need zotfile if this could be changed...
  • 1) no. Zotfile really does move files automatically - reread what bwiernik writes above and try it out. The send to tablet functionality is entirely separate from that and if you're moving all files to dropbox automatically you probably don't want to use it at all.

    I understand why people don't like the randomstrings, yes, I don't think they're weird and they do have advantages on the Zotero side of things, including e.g. the ability to elegantly handle webpage snapshots, something most other reference managers can't.
  • Thanks again for your reply: Sorry I do not want to annoy you but I really want to learn how to do this (I make video tutorials (http://www.youtube.com/watch?v=ZsQPx5ow1JY) for science software and zotero is high on my list)

    In the first reply you told me:

    "that's incorrect. Zotfile can move your files wherever you want it to as bwiernik explains in step 3."

    That's why I thought it was it's only functionality... I do not have zotero close by but I will check this as soon as possible...

    Please just give me this confirmation:

    This means that with zotfile everytime you download a reference and its article (by the normal means) It will save the files to the location you want (with the author, year structure options described by bwiernik) and then if you add notes from zotero, for example, they will also be saved there?

    Thanks again
  • This means that with zotfile everytime you download a reference and its article (by the normal means) It will save the files to the location you want (with the author, year structure options described by bwiernik)
    correct.
    if you add notes from zotero, for example, they will also be saved there?
    if you actually mean notes added in Zotero, then no: Those are not saved as files anywhere, but part of the Zotero database and as such easily accessible and editable e.g. on the zotero.org webpage, which has a nice mobile version.
  • Thank you very much you made me very happy! :D

    I will check this immediatly
  • Dear bwiernik, delosari, adamsmith: thank you for great explanations and discussion! I will try to apply the steps soon and then Report back here.
    Delosari, if you have done checking the Zotfile ability to move files automatically, would you please tell us too? Thank you!
  • Just to tie things up I have been trying this set-up and it works nicely. Yesterday, I formatted my pc and below I include the steps I employed to install Zotero and Zotfile in case someone needs them.

    However, there are two issues I must mention:

    1) Now, thanks to zotfile, zotero downloads the articles, automatically, to folders with any structure we want. However, the files already in our collection remain in the old folders... This is an issue if you have a big library... There must be an problem with my installation if anyone has and suggestion on how to solve it, I would be most welcome.

    2) To solve the previous problem I found a solution which could be also the procedure to restore your library to a new computer. However, I am not sure this is right procedure to back up your library (specially if you have many plugins) so I need to investigate more (proceed carefully):

    -Use the export function in the "File" menu to back up your library
    -Delete your library in zotero (At this point you may want to disconnect your computer from the internet to avoid messing your internet library)
    -Configure your zotfile to save your library pdf attachments to the location you want
    -Use the import function in the "File" menu to bring back you library. Now, zotfile will impose its configuration in the old files...

    Any advice/warning will be most welcome. Anyway this is my installation for zotero/zotfile (using ubuntu 64):


    ----Zotero installation: Following guide at http://anterotesis.com/wordpress/2011/11/installing-zotero-standalone-on-ubuntu-11-10/

    1) Get zotero http://www.zotero.org/download/

    2) Download file Zotero-4.0.16_linux-x86_64.tar.bz2

    3) Extract folder Zotero_linux-x86_64

    4) Move folder sudo mv Zotero_linux-x86_64/ /opt/

    5) Create file-link in sudo gedit /usr/share/applications/zotero.desktop

    6) Paste these lines:

    #!/usr/bin/env xdg-open
    [Desktop Entry]
    Type=Application
    Name=Zotero
    GenericName=Bibliography Manager
    Icon=/opt/Zotero_linux-x86_64/chrome/icons/default/default48.png
    Exec= /opt/Zotero_linux-x86_64/zotero %f
    Categories=Office
    Terminal=false

    **Note** The addresses used here (in "icon" and "Exec" should be adjusted to your operative system and its architecture (32bits, 64bits)



    ----Zotfile installation: http://www.jlegewie.com/zotfile.html

    7) Download zotfile https://addons.mozilla.org/en-US/firefox/addon/zotfile/

    8) Install xpi addon In zotero go to "Tools->Add-on" and in the corner icon go to "Install-Addon from file"

    9) Now in the zotero main window click on the gear icon and go to "Zotfile Preferences"

    10) On the "General Settings" in the middle lower region you have the "Location of Files" section. In here, "choose..." a "custom location". In my case, I selected a folder within dropbox. Incidentaly, you can activate the subfolder option to create subfolders for authors, years, publication... Now zotero will save new files to this location.

    11) In the "Zotfile Preferences" you can also go to "Renaming Rules" to adjust file naming to your needs (the rules can be found by clicking on the link close to the subfolder option we saw at point nine)

    **NOTE** If you had zotero managing your attachements location you need to go to "Zotero preferences" and in the "Advanced" menu go to the "Files and Folders" subtab and click on "Revert to Absolute Paths" as described above.
  • For issue 2) above, export/import is strongly discouraged. It will mess, e.g. with your old LibreOffice documents and also re-sync everything.

    You can simply select items and click on Manage attachments --> Rename attachments. This doesn't just rename them, but also - if that's part of your Zotfile settings - move their pdf attachments.

    For installation on Linux I'd recommend this ppa https://github.com/smathot/zotero_installer#using-the-ubuntu-ppa (or the script on the same page) rather than the manual instructions.
  • Thanks to bwiernik and adamsmith for your detailed answers. I'd like to point out that when adamsmith says "select items and click on Manage attachments → Rename attachments", it's important to realize that's "items" in the plural. I selected all the items with attachments in my library and moved/renamed all the files at once. It doesn't get any easier.

    suaracahaya, I've written up instructions--which partially have to do with switching my PDF library from calibre to Zotero--on my blog at http://freecity.commons.gc.cuny.edu/2013/12/14/setting-up-a-portable-library-with-zotfile/. Thanks for starting this thread. I've been using Zotero for a couple of years, but just today I set out to do the same thing as you, so your question couldn't have come at a better time.
  • Thanks leehach, these are great (recommended for everyone else).

    A stylistic quibble (if I may) - your 4.) where you specify the Zotfile directory, which arguably is the most important part of the whole operation, is so small that it's easy to miss. Maybe add a screenshot for that, too?
  • Thanks adamsmith, I've added the suggested screenshot.
  • leehach, that's a very beautifully written instructions... Thank you for summing up everything!
    Delosari, thanks for sharing your set up! Now I only need to find enough time out of my baby's distraction to get my own Zotero working - or probably wait patiently until Monday comes to work from the university ;-)
    Really, thanks everyone for the guidance... m(_ _)m
  • edited December 14, 2013
    looks great, leehach, thanks.
  • As a newcomer to Zotero, I find this exchange to be extremely informative, since treatment of PDFs is the biggest question I've had about it. bwiernik makes a good case for using Zotero itself to store PDFs, while leehach's post on his blog made me think that it might make more sense for me to continue to use Dropbox and deploy zotfile to assist with that. I admit I'm having a hard time choosing between retaining my pre-Zotero use of Dropbox to store PDFs, which I'm very used to, and using Zotero itself, which seems to promise stability and efficiency but which may be a less intuitive way for me to handle files. I'm a Windows user, and the familiar "folder" way of thinking about files dies hard.
  • Now I've got Zotero & Zotfile works with Dropbox! Thanks for all the discussion :)
    (Although, Zotpad on my iPad still won't download the pdfs from dropbox eventhough I already complete the setting...)

    I have a new situation: I have some lecture notes in pdf already compiled in Zotero directory, with naming format: "Lecturer_LectureName_#lecture" e.g. "Romi_DM_01.pdf" until "Romi_DM_05.pdf". I put all the 5 pdf under one parent item, "Data Mining Lecture Notes".
    I want to move them from Zotero inner directory to the new Dropbox Base Directory and have them as linked attachment; but if I use Zotfile's "Rename attachment", all of them will have the same name.

    Question: Is there any way to make a linked attachment without using Zotfile's "Rename attachment"?
  • Zotfile will not give them all the same name. If there is a naming conflict, it will append "2", "3", "4", etc. (but no number for the first file with the same name). I think if you rename the PDFs one at a time in the right order, rather than rename them all at once by clicking on the parent item, you will get lecture 1 named <base name>.pdf, lecture 2 will be named <base name>2.pdf, lecture 3 will be named <base name>3.pdf, etc.

    Also, I don't know if you already know this, but you can set the name explicitly to whatever you want. Select the PDF item, and in the right-hand pane you will see the PDF name, along with various file characteristics and a rich text notes field. Click the PDF name at the top (the one in bold type, not the one next to the "Filename:" label) and you will get a dialog for a "New title:" along with an option to "Rename associated file". So you could rename all the files manually, or you could do the above procedure for automatic renaming, and then edit lecture 1 so that it actually has "1" in the filename.
  • @suaracahaya

    The methods that leehach describe work well. I like to have more automated control over the file names. To do that, I create a separate Zotero item for each lecture where I add a lecture number using the Short Title field. I then create an separate Zotero item for the class as a whole that I connect to the individual lectures as Related Items. This is also how I handle entries for a book and its individual chapters. This method allows me to automate renaming of attachments, as well as to add individual tags and notes to each lecture.
  • This is a great thread. Thanks to everyone.

    I recently realized I have to take my pdf management up a notch. Up until now, I've kept copies of pdfs in all sorts of places: any one of the computers I use in 3 different offices, a laptop, an iPad, an Android phone, a departmental server, any one of multiple work sites in my university's LMS (Sakai), or one of several cloud services, backup drives, and even SDHC cards. You see the first part of the problem: I need to keep everything better organized and secure.

    But it's more complicated than just this. I have over 12,000 entries in my Zotero database. Of these, I have (or will have shortly) perhaps a few thousand pdfs organized chaotically as described above. But at any one time I need only perhaps one or two hundred pdfs on my iPad to read or to reference.

    What I've been doing up until now is using Zotero just as a reference manager and manually copying pdfs to pdf-notes on the iPad, which I then use for reading and annotating. But I'm also outgrowing pdf-notes, largely because of its limited single-level folder structure, inability to search across multiple pdfs, inability to search for tags across folders, and inability to create virtual folders based on searches.

    I've started playing with ZotPad and PaperShip, but neither works well. PaperShip repeatedly tries to sync with my Zotero library, a process taking considerable time, and then crashes. ZotPad is quicker and more stable, but its interface is very awkward on an iPad and is limited (e.g. I don't see any way to limit a view only to items with actual attachments). So now I'm thinking of just using Zotfile to organize my pdfs and send them to a pdf reader (I'm looking into iAnnotate or GoodReader) on my iPad, where I will organize the few hundred pdfs I work with at any one time.

    I'm thinking of having Zotfile store the pdfs on my university's LMS by using WebDav. I realize this has some shortcoming, but as leehach says on his page, this will give access to the files outside of Zotero. For example, I could give my students access to a subset of pdfs without requiring the students to learn Zotero.

    My main question is will this work?

    Also, given how knowledgeable everyone on this thread is, any suggestions for improving this workflow will be much appreciated.
  • I'm not sure what you mean by this:
    I'm thinking of having Zotfile store the pdfs on my university's LMS by using WebDav.
    ZotFile doesn't really interact with WebDAV.
    It sounds like you just want to use ZotFile's Send to Tablet functionality, which works with any service that syncs folders across devices (box.com, Dropbox, spideroak etc.)
  • I'm not sure what you mean by this:
    The Zotfile page says it "stores the file as a Zotero attachment to this item (or alternatively, moves it to a custom location)." I thought I could could use WebDAV on my Mac to mount a directory on the LMS server and define this directory as a custom location for Zotfile.

    And yes, I would use Zotfile's Send to Tablet functionality. I just wish it were better integrated with the iPad.

    For example, imagine a lightweight app that let you search your Zotero library for items with attachments, written by "John Doe," and having tags "foo" and "bar." This might yield a relatively small number of items from which one could check selected ones and have them sent to the tablet, either by using services like box, etc. or by letting the tablet use the same WebDAV link or by accessing the items in Zotero File Storage. I think ZotPad can almost do this, but it first loads a list of the entire library. I guess I'm imagining something more streamlined.
  • For example, imagine a lightweight app that let you search your Zotero library for items with attachments, written by "John Doe," and having tags "foo" and "bar." This might yield a relatively small number of items from which one could check selected ones and have them sent to the tablet, either by using services like box,
    I'm confused now - that's exactly what ZotFile's Send to Tablet does. You use Zotero to search for the files (why do you need a lightweight app for that if you have Zotero?) Send them to your box folder, let box do the syncing and have the files available on your tablet via box. What am I missing?
    Unless you want to have that app on the tablet? I think you're imagining that part a lot easier than it is, given that it'd be very hard to do these types of search operations, especially wrt tags, without a significant amount of syncing beforehand, at which point you're quite close to ZotPad again.
  • Yes, I was thinking of having the app on the tablet.

    W/ r.t. ZotPad, there are two aspects. One is how it actually accesses Zotero's data. The other is what the user sees. Given that tablets have limited screen real estate, I'd like the app to start out showing the user something much more sparse than the complete library.

    I don't know how Zotero's internal indexing is organized. If possible, rather than have the app load the entire index (and data?), I'd rather have the user specify a filter, have the app generate a (sql?) query for the database, and return only the information the user requested.

    Right now Zotpad seems to start out loading everything and culling down from there.

    The basic idea is to start with an almost blank screen, tell the app "Hey, give me a list of all items by "John Doe" with tags "foo" and "bar" and having attachments." Once the list appears, check the ones you want and either tell the app "Send attachments to the checked items to my Box folder" or "Send the attachments to my designated pdf viewer."
  • OK, so all I can tell you it that it doesn't currently exist, I don't think anyone is working on it--it's likely technically possible, though not trivial given that you'd have to structure this as API requests. You can ask the app developers, but I wouldn't get my hopes up.
    I'm thinking of having Zotfile store the pdfs on my university's LMS by using WebDav.
    I was mainly trying to answer your original question, i.e. whether that'd work.
    I don't think WebDAV is a particularly easy way to set up a sync folder across computers and you obviously don't have an iPad app, which is really where my confusion with your original idea comes from.
    If you're going to use ZotFile, you'll almost certainly want to do that with drobox, box, or something of the kind, which is what the developer had in mind.
  • Thanks. I didn't understand this. It's very helpful to know.
  • Thanks for your feedback Adam.

    After some investigation, I've decided to use Zotfile and a iPad pdf-annotator (TBD) for now. But since so many of my pdf files are for "deep storage" (i.e., not being used right now), I want to rearrange them.

    Ideally, I'd keep the ones I'm not using right now on my university's LMS and access them via WebDAV. I'll experiment with accessing them via Zotero, but even having to move them manually would not be a big problem because they're not being used. OTOH, I would move everything I'm currently using to my Zotero File Storage (ZFS) account and sync the easy way. If the WebDAV experiment works, I think I may be able to put the ZFS stuff into a group, so my Zotero clients would be able to see both sets of files simultaneously.

    If this approach will work, then the first step will be retrieving all my files currently on ZFS, going through them, deciding where to keep them, and moving some of them to the LMS. Next, I'll clear out the files that were moved to the LMS. Finally, I'll add the files I'm using for my current projects to ZFS.

    This raises at least these questions:

    1. To experiment with WebDAV on the LMS, can one temporarily change the settings in a Zotero client from the current ZFS setting to the WebDAV server without causing major harm?

    2. I run both the Firefox & Standalone versions of Zotero. If I change one, will it affect the other?

    3. How does one isolate only those items in the library that have attachments on ZFS? (E.g., as distinct from linked attachments.)

    4. Is there an easy way to batch download all the attachments in #3, or does one have to do it item-by-item?

    5. Is there an easier way to move attachments between two different services other than opening individual attachment from Zotero and saving them one-by-one in an attached folder on the other service? (ZFS & the LMS WebDAV in this case, but in general people might be interested in Box, Dropbox, etc.)

    Thanks ahead of time to Adam or anyone else who has answers to these questions.
  • 1. no likely issues that I can think of
    2. no (preferences are not shared between the two)
    3. I don't think that's possible
    4. Not sure what you mean by batch downloading? Without Zotero? No. With Zotero it's just a sync operation.
    5. This suggest your fundamentally misunderstanding how sync works, so in light of this, maybe you _can_ do damage re. 1)

    File sync is (basically) all or nothing. WebDAV sync works exactly the same way as ZFS sync. You just enter the sync information and sync. That's the only thing needed to switch between two services. Really, you should never manually move any file that's attached or linked to a Zotero item. You'll almost invariably break the link to Zotero. If you want to convert attached files to linked files or vice versa, use ZotFile.
Sign In or Register to comment.