Linked or stored library?

edited March 1, 2023
I've been using Zotero as a linked library - all entries point to a file in a large folder inside my iCloud, where everything is organised into folders (by topic) and sub folders (often book chapter files are put into a folder). I find it useful because if I want to access a file outside of Zotero I can find it through these folders (on my phone for example or when using a different pdf reader). If book chapters would not be put together into a folder I doubt I would ever find them, for example.

But I want to work more on my iPad. I now convert the linked file into a stored file and then do some complicated juggling to put it back the way it was when I finished reading and annotating. I am tempted to convert everything into a stored library so I can access it easily from laptop and iPad. But then I loose my organised file structure.... I could only ever find files via Zotero or via search, and not through browsing files. Also, if ever Zotero dies (hope not) I would have a mountain of unorganised files. I need this to be very distant-future proof....

What are other people's thoughts - what are the pros and cons of the two systems? Stored or linked?
Thanks!
  • I love the convenience of stored files, the fact that I never have to think about set up on any device, and I have no need to search for academic PDFs anywhere else (being able to search/sort/order by metadata is way better than anything even a much-more-disciplined me would be able to do in a file system), so don't care about the structure on disk.
    In terms of future-proof: the collection information is available in three different places -- the Zotero API, the Zotero sqlite, and Zotero export to RDF. Each of these has already been used by other projects to access & import Zotero data. In the unlikely event that Zotero just dies, even if it does so with no adequate warning, there'll be scripts to export the library either to a different reference manager, including collection information, and/or to your file system.
  • @OldSlippers: I'm sure you've seen it, but just for others, Stored Files and Linked Files explains the differences and the reasons we recommend stored files.
  • @OldSlippers, I have been in the same situation, except that I didn't bother to sort my pdfs into folders).

    When I switched to Zotero (from Citavi), I stuck to my storage location (i.e. linked files) since I wanted to sync those across different devices. I used dropbox because I didn't realize/understand that I can use my own webdav storage for file storage together with syncing the database via zotero.org and I have way more pdfs than the 200 MB of free file storage on zotero.org.

    TBH, I was also somehow sceptical toward the whole idea of Stored Files because it meant that I render control of my files to Zotero and I generally don't like apps encouraging me to let them manage stuff for me. And in the case of Zotero 5, I saw very few benefits of handing things over to Zotero (i.e. switching from linked files to stored files). I seem to remember that one downside was that Zotero-links are not supported for linked files. But apart from that, using stored files would have made accessing and annotating those pdfs on different devices a pain.

    But then - BOOOM - Zotero 6 came and it was life-changing! (You really can't overstate the importance of that update - thanks everyone who contributed to it!). All of a sudden, I didn't need any third-party pdf reader/annotating tool. Zotero Reader now does it all. But what about syncing? It was only then that I discovered that I can use stored files in combination with any webdav storage.

    But would it work properly? Surely, there would be some limitations and hickups when the database is stored on one server, and the files on another, and then I access these files from different devices? My mind was blown away by the fact that it "just worked", as they say. I open articles in the Zotero app on my iPad, annotate them and (at some point - I'm not sure what the delay depends on) those annotations will be available on my desktop computer for further processing in Obsidian. And the fact that annotations are stored in Zotero, not in the pdfs means that I don't have to worry about pdfs syncing or not syncing ever again.

    So far, I have not converted all my linked files to stored files yet, I just convert those files that I currently work with and that way, when I come across a linked files, it tells me that I haven't touched that article for a long time, which triggers me to make sure that any existing pdf annotations are imported into Zotero. But I so no reason why you shouldn't just convert all of them at once.

    tl;dr Just go ahead and switch to stored files.

    I you're unsure, you can testdrive by converting only the files you are currently working with and _not_ deleting the original file (it asks you every time you convert) so you can go back to your original setup any time. (Note however that even when the original files are not deleted, they will no longer be linked to your zotero item. So, if you ever go back, you'll have to link them back manually.
  • edited March 2, 2023
    Thank you for these helpful answers! Much appreciated.
    Adam, you write "In terms of future-proof: the collection information is available in three different places -- the Zotero API, the Zotero sqlite, and Zotero export to RDF." But I'm afraid I don't really know what all that means. I do manual backups of the whole Zotero folder now and then, so that would include the sqlite file, which, I understand, is the important one. And, like you say, not having to think about access would be great (worked fine before I had an iPad and just worked on one computer). I suppose the most important thing is that the PDF's are on my harddrive and get backed up when I back up the Zotero folder.

    Haug, thank you. I don't know what webdav is, sorry. But for me too, Zotero 6 made all the difference. The pdf reader, annotations and notes make sure almost all of my workflow now happens inside Zotero. It's brilliant. And you are right, I could go slow, just converting as and when I need it so if I decide to go back, it won't be all of my files. I suppose I could start with the lowest paid storage tier.




  • If I do convert all, what do I do with the Zotfile settings? Do I still need zotfile or should I even uninstall it (somehow)?
  • You can likely just uninstall ZotFile, unless you want the more advanced file renaming options. (Zotero will be gaining more customizable renaming in a future version, but for now it will name files "Creator - Year - Title".)

    If you do keep it installed, you would want to disable the feature that converts stored files to linked files.
  • Adam, you write "In terms of future-proof: the collection information is available in three different places -- the Zotero API, the Zotero sqlite, and Zotero export to RDF." But I'm afraid I don't really know what all that means.
    You don't need to understand the technical details. In practical terms, it means that if Zotero dies, deteriorates, or a significantly better tool comes along, there are many ways in which either competitors or frustrated users who know how to code could provide a tool to get all data out of Zotero. Zotero, e.g., did that for Mendeley when things started getting worse and worse over there (even though Mendeley made this significantly harder).
  • Thank you to you both. Very helpful. I'll take Haug's advice, taking into account everything you all said, and start converting file by file. Loving the iPad app (and the recent improvements) btw!
  • Oldslippers, thank you for your question because I am in a pretty same situation : I used a workflow with zotfile and linked files too. But then I stopped using Zotero for a while (pregnancy and parent leave..) and Zotero 6 appeared and I am not updated yet ... So I have to ! it is time. I will leave back my usual workflow and change all. So I have to read again to understand what I have to do. (to see how to convert to this new way).
    I want to thank the Zotero team for this wonderful update. thanks !
  • I just got an iPad and found this topic very interesting. I saw all these records of people suggesting heavy settings with zotfile and linked files and missed the fact that zotero 6 changed something.
    If I understand correctly, it appears to me that you cannot seamlessly open a pdf from an entry in zotero's iOS app if you use linked library, right?
    Also, @haug, could you please specify what you meant by "I can use stored files in combination with any webdav storage"?
  • Linked files do not work with the Zotero iPad app for technical reasons (iPadOS does not have a general system for linked files). To use the iPad app, you need to use Zotero file storage or WebDAV
  • @wassimbz, you can use a configuration with webdav. I changed my "linked files+zotfile" configuration for this one and it is nice and more secure.
  • Linked files do not work with the Zotero iPad app for technical reasons (iPadOS does not have a general system for linked files). To use the iPad app, you need to use Zotero file storage or WebDAV
    Hello, thanks for the info. I am wondering what the technical limitations are?

    Opening files and folder contents outside of an app has been possible on iOS and iPad since iOS 12/13. Apps can also bookmark folders to maintain permission to access the folder. An app just has to ask for permission from the user and implement file/folder access as decscribed at https://developer.apple.com/documentation/uikit/view_controllers/providing_access_to_directories

    In Zotero on my Mac, I've used linked attachments that are stored in my iCloud Drive. On my iPad, I have several apps that I have given specific permission to access/bookmark the folder where these attachments are stored, and I am able to easily access the attachments in those apps after giving the app permission.

    For example, the References app on iOS/iPad (https://apps.apple.com/us/app/references/id1481843213) is another reference manager that uses .bib files. I keep a BibTex file up-to-date via Better BibTeX auto exports, and then in the References app on my iPad, I add the iCloud folder where my attachments are synced as a Linked Folder in the app, and can open the attachments in the app in place. It's basically the same thing as setting the attachment base directory in Zotero on desktop.

    As another example iOS/iPad app, I can give PDF Search (https://apps.apple.com/us/app/pdf-search/id1303227688) permission to access, index, and even OCR files in the iCloud folder where my linked Zotero attachments are synced.

    There are several other apps that implement file/folder access features like this.

    It would be great if the Zotero iOS app could implement linking/bookmarking folders in this way for linked attachments as well.

    Are there other technical limitations that couldn't be overcome by implementing the API features described in the Apple Developer article linked above?

    Thanks again!
  • Hello, interesting discussion on linked vs. stored libraries. What are thoughts on using Zotero to mange your active/working files?

    I have 2 different use cases for Zotero.
    [CASE 1] as a local desktop only document manager. In my work environment, we cannot use sync features. I started using Zotero to manage research items. It has been much easier to find past research and pull together a collection of items that I want to use for a particular project.

    [CASE 2] wanting to use Zotero to manage work products (e.g. MS Office files) for the ability to add notes, tags, relationships, and collections. But I have not wanted to move files from the pre-established directory structure. However it requires a lot of discipline to effectively maintain linked files. I think my desired functionality would be an option to store files in a user-defined hierarchy and then use collections for project work. This would allow me to browse for files when not using zotero but retain the benefits of zotero when in the app. The biggest limiting factor is not having the ability to define a few customer Item Types.

    Zotero seems to be the most flexible solution for both research and content management that I have found. With the growing need to associate meta-data and supplemental notes/comments to work product for a more integrated work environment… It could almost function as a replacement for the “Documents” directory of Windows Explorer (or similar on other platforms).
  • Thanks for the outstanding application. I'm about to become an unlimited storage user but I'd like to find a bulk solution to one need I have. I'd like to copy my linked files to stored files such that there would be two child items to an entry; the original linked file and the new stored file.
    The current Tools > Manage Attachments > Convert... is great, but it converts (i.e., moves) the item. Is there a way Tools > Manage Attachments > Copy Linked Files to Stored Files... such that after the operation two child items remain (a linked and a stored file)?
  • @ford17 Why would you want to duplicate your attached files? To me, that strategy is a recipe for disaster (or at least confusion).

    Anyway, from what I can see, the only way to achieve that would be to manually create a new attachment

    > @haug, could you please specify what you meant by "I can use stored files in combination with any webdav storage"?

    @wassimbz Sorry for the late reply (I didn't receive a notification for your post, see https://forums.zotero.org/discussion/comment/450284/#Comment_450284). I assume that you figured this out by now, but just to clarify: by that I mean that when I use stored files, In the File sync settings, I can choose whether those files should be stored on (or synced via) WebDAV or Zotero storage, and I chose WebDAV, which has worked flawlessly for me.
  • Thanks, @haug, the per-item conversion is all I could see, too. And FYI, the use-case is if one is restricted in accessing the internet for some time or location or machine, it would be helpful to have a local attachment.
  • edited December 9, 2023
    @ford17: But stored files are…stored on your computer. Zotero saves everything locally on your computer by default. Just leave it set to download files "at sync time" and you'll have all stored files in your library on that computer.
  • my reason(s) for persisting with linked files are the much more intuitive and customizable folder structure outside of zotero.

    what this means is that if I want to share a bunch of PDFs with someone outside of zotero, I can do so without a whole lot of mucking around. As some of my reading is for teaching, this is essential for my situation, in a way that might not be generally true. I'm not sure why zotero structures its storage folder the way it does, but assume there's a good reason for it, and until/unless that changes I'm in team linked. means I can't use the apps though, which is not good.

    I came here with a different question though. is there any way to search for linked vs stored attachments? just for general housekeeping?
  • @vijay.krishnan:
    what this means is that if I want to share a bunch of PDFs with someone outside of zotero, I can do so without a whole lot of mucking around
    You can still access stored files easily in various ways. From the documentation:
    To use stored files outside of Zotero, you can use Zotero's search and organization abilities to quickly find the relevant items and then either drag the attachments straight from Zotero (e.g., into an email) or right-click and choose Show File to view the files in your file manager. If you prefer to find files without going through Zotero, you can use your operating system's search features (e.g., Spotlight on macOS) or create a smart folder in your OS to show a list of all PDFs within your Zotero data directory and interact with the files directly. Zotero automatically renames files based on the parent item's bibliographic data, so you can easily find files by title, author, or year even from outside Zotero.
    Other questions:
    I'm not sure why zotero structures its storage folder the way it does, but assume there's a good reason for it, and until/unless that changes I'm in team linked
    Yes, the structure ensures the files are always accessible from the associated items in Zotero and can be reliably synced between devices. People frequently report here that their linked files are no longer working from within Zotero, because the path changed, they renamed something outside of Zotero, they switched computers, etc. With stored files, as long as you're syncing and have storage space, your files will be available from Zotero on all your devices.
    is there any way to search for linked vs stored attachments? just for general housekeeping?
    Not currently, but you can press + to expand all item rows and just look for the link icon. (And if you wanted to convert all items from linked files to stored files, you could do that using Tools → Manage Attachments → Convert Linked Files to Stored Files.)
  • @vijay.krishnan you can get a list of all linked attachments (not just PDFs) with the following code run under Tools\Developers\Run Javascript:

    var filepathnames = await Zotero.DB.columnQueryAsync('SELECT path AS filepathnames FROM itemAttachments WHERE path IS NOT NULL AND path NOT LIKE ? ORDER BY path','storage:%');
    return filepathnames.join('\n');

    If you have a Linked Attachment Base Directory setting, all listed file paths will be prefixed with "attachments:". If you need the full path, insert your LABD there in a text editor. If you have no LABD setting, you will get the attachment's full explicit path.

    For a list of stored PDF (only) file attachments under Zotero\storage, use:

    var filepathnames = await Zotero.DB.columnQueryAsync("SELECT TRIM(key || '\\' || REPLACE(path,'storage:','')) AS filepathnames FROM itemAttachments JOIN items USING (itemID) WHERE libraryID=1 AND path IS NOT NULL AND path LIKE ? ORDER BY key",'storage:%.pdf');
    return filepathnames.join('\n');

    Of course you could also do a search in your operating system for *.PDF under Zotero\storage (and you could save that as a saved search). It is however possible that might identify PDFs that are not actually known to Zotero (eg PDFs you put there manually, or 'orphaned' PDFs).

    The process of checking if your linked-files folder contains files that are NOT linked in Zotero (either deliberately or inadvertently*), or if your Zotero\storage hierarchy has PDF files NOT known to Zotero, is a little more complicated. But it can be done with some external coding (eg comparing the linked-files folder's directory listing to the linked-files list above).

    *the fact that Zotero does not delete linked attachments when an item containing them is deleted can lead to orphaned PDFs in the linked-files folder. That's why the delitem plugin should be used routinely to delete linked attachments within Zotero:
    https://github.com/redleafnew/delitemwithatt

    There is currently no way to save the above code snippets in Zotero for easy re-use, but apparently that is planned.
Sign In or Register to comment.