Upgrade StorageLinked or stored library?
I've been using Zotero as a linked library - all entries point to a file in a large folder inside my iCloud, where everything is organised into folders (by topic) and sub folders (often book chapter files are put into a folder). I find it useful because if I want to access a file outside of Zotero I can find it through these folders (on my phone for example or when using a different pdf reader). If book chapters would not be put together into a folder I doubt I would ever find them, for example.
But I want to work more on my iPad. I now convert the linked file into a stored file and then do some complicated juggling to put it back the way it was when I finished reading and annotating. I am tempted to convert everything into a stored library so I can access it easily from laptop and iPad. But then I loose my organised file structure.... I could only ever find files via Zotero or via search, and not through browsing files. Also, if ever Zotero dies (hope not) I would have a mountain of unorganised files. I need this to be very distant-future proof....
What are other people's thoughts - what are the pros and cons of the two systems? Stored or linked?
Thanks!
But I want to work more on my iPad. I now convert the linked file into a stored file and then do some complicated juggling to put it back the way it was when I finished reading and annotating. I am tempted to convert everything into a stored library so I can access it easily from laptop and iPad. But then I loose my organised file structure.... I could only ever find files via Zotero or via search, and not through browsing files. Also, if ever Zotero dies (hope not) I would have a mountain of unorganised files. I need this to be very distant-future proof....
What are other people's thoughts - what are the pros and cons of the two systems? Stored or linked?
Thanks!
In terms of future-proof: the collection information is available in three different places -- the Zotero API, the Zotero sqlite, and Zotero export to RDF. Each of these has already been used by other projects to access & import Zotero data. In the unlikely event that Zotero just dies, even if it does so with no adequate warning, there'll be scripts to export the library either to a different reference manager, including collection information, and/or to your file system.
When I switched to Zotero (from Citavi), I stuck to my storage location (i.e. linked files) since I wanted to sync those across different devices. I used dropbox because I didn't realize/understand that I can use my own webdav storage for file storage together with syncing the database via zotero.org and I have way more pdfs than the 200 MB of free file storage on zotero.org.
TBH, I was also somehow sceptical toward the whole idea of Stored Files because it meant that I render control of my files to Zotero and I generally don't like apps encouraging me to let them manage stuff for me. And in the case of Zotero 5, I saw very few benefits of handing things over to Zotero (i.e. switching from linked files to stored files). I seem to remember that one downside was that Zotero-links are not supported for linked files. But apart from that, using stored files would have made accessing and annotating those pdfs on different devices a pain.
But then - BOOOM - Zotero 6 came and it was life-changing! (You really can't overstate the importance of that update - thanks everyone who contributed to it!). All of a sudden, I didn't need any third-party pdf reader/annotating tool. Zotero Reader now does it all. But what about syncing? It was only then that I discovered that I can use stored files in combination with any webdav storage.
But would it work properly? Surely, there would be some limitations and hickups when the database is stored on one server, and the files on another, and then I access these files from different devices? My mind was blown away by the fact that it "just worked", as they say. I open articles in the Zotero app on my iPad, annotate them and (at some point - I'm not sure what the delay depends on) those annotations will be available on my desktop computer for further processing in Obsidian. And the fact that annotations are stored in Zotero, not in the pdfs means that I don't have to worry about pdfs syncing or not syncing ever again.
So far, I have not converted all my linked files to stored files yet, I just convert those files that I currently work with and that way, when I come across a linked files, it tells me that I haven't touched that article for a long time, which triggers me to make sure that any existing pdf annotations are imported into Zotero. But I so no reason why you shouldn't just convert all of them at once.
tl;dr Just go ahead and switch to stored files.
I you're unsure, you can testdrive by converting only the files you are currently working with and _not_ deleting the original file (it asks you every time you convert) so you can go back to your original setup any time. (Note however that even when the original files are not deleted, they will no longer be linked to your zotero item. So, if you ever go back, you'll have to link them back manually.
Adam, you write "In terms of future-proof: the collection information is available in three different places -- the Zotero API, the Zotero sqlite, and Zotero export to RDF." But I'm afraid I don't really know what all that means. I do manual backups of the whole Zotero folder now and then, so that would include the sqlite file, which, I understand, is the important one. And, like you say, not having to think about access would be great (worked fine before I had an iPad and just worked on one computer). I suppose the most important thing is that the PDF's are on my harddrive and get backed up when I back up the Zotero folder.
Haug, thank you. I don't know what webdav is, sorry. But for me too, Zotero 6 made all the difference. The pdf reader, annotations and notes make sure almost all of my workflow now happens inside Zotero. It's brilliant. And you are right, I could go slow, just converting as and when I need it so if I decide to go back, it won't be all of my files. I suppose I could start with the lowest paid storage tier.
If you do keep it installed, you would want to disable the feature that converts stored files to linked files.
I want to thank the Zotero team for this wonderful update. thanks !
If I understand correctly, it appears to me that you cannot seamlessly open a pdf from an entry in zotero's iOS app if you use linked library, right?
Also, @haug, could you please specify what you meant by "I can use stored files in combination with any webdav storage"?
Opening files and folder contents outside of an app has been possible on iOS and iPad since iOS 12/13. Apps can also bookmark folders to maintain permission to access the folder. An app just has to ask for permission from the user and implement file/folder access as decscribed at https://developer.apple.com/documentation/uikit/view_controllers/providing_access_to_directories
In Zotero on my Mac, I've used linked attachments that are stored in my iCloud Drive. On my iPad, I have several apps that I have given specific permission to access/bookmark the folder where these attachments are stored, and I am able to easily access the attachments in those apps after giving the app permission.
For example, the References app on iOS/iPad (https://apps.apple.com/us/app/references/id1481843213) is another reference manager that uses .bib files. I keep a BibTex file up-to-date via Better BibTeX auto exports, and then in the References app on my iPad, I add the iCloud folder where my attachments are synced as a Linked Folder in the app, and can open the attachments in the app in place. It's basically the same thing as setting the attachment base directory in Zotero on desktop.
As another example iOS/iPad app, I can give PDF Search (https://apps.apple.com/us/app/pdf-search/id1303227688) permission to access, index, and even OCR files in the iCloud folder where my linked Zotero attachments are synced.
There are several other apps that implement file/folder access features like this.
It would be great if the Zotero iOS app could implement linking/bookmarking folders in this way for linked attachments as well.
Are there other technical limitations that couldn't be overcome by implementing the API features described in the Apple Developer article linked above?
Thanks again!
I have 2 different use cases for Zotero.
[CASE 1] as a local desktop only document manager. In my work environment, we cannot use sync features. I started using Zotero to manage research items. It has been much easier to find past research and pull together a collection of items that I want to use for a particular project.
[CASE 2] wanting to use Zotero to manage work products (e.g. MS Office files) for the ability to add notes, tags, relationships, and collections. But I have not wanted to move files from the pre-established directory structure. However it requires a lot of discipline to effectively maintain linked files. I think my desired functionality would be an option to store files in a user-defined hierarchy and then use collections for project work. This would allow me to browse for files when not using zotero but retain the benefits of zotero when in the app. The biggest limiting factor is not having the ability to define a few customer Item Types.
Zotero seems to be the most flexible solution for both research and content management that I have found. With the growing need to associate meta-data and supplemental notes/comments to work product for a more integrated work environment… It could almost function as a replacement for the “Documents” directory of Windows Explorer (or similar on other platforms).
The current Tools > Manage Attachments > Convert... is great, but it converts (i.e., moves) the item. Is there a way Tools > Manage Attachments > Copy Linked Files to Stored Files... such that after the operation two child items remain (a linked and a stored file)?
Anyway, from what I can see, the only way to achieve that would be to manually create a new attachment
> @haug, could you please specify what you meant by "I can use stored files in combination with any webdav storage"?
@wassimbz Sorry for the late reply (I didn't receive a notification for your post, see https://forums.zotero.org/discussion/comment/450284/#Comment_450284). I assume that you figured this out by now, but just to clarify: by that I mean that when I use stored files, In the File sync settings, I can choose whether those files should be stored on (or synced via) WebDAV or Zotero storage, and I chose WebDAV, which has worked flawlessly for me.
what this means is that if I want to share a bunch of PDFs with someone outside of zotero, I can do so without a whole lot of mucking around. As some of my reading is for teaching, this is essential for my situation, in a way that might not be generally true. I'm not sure why zotero structures its storage folder the way it does, but assume there's a good reason for it, and until/unless that changes I'm in team linked. means I can't use the apps though, which is not good.
I came here with a different question though. is there any way to search for linked vs stored attachments? just for general housekeeping?
var filepathnames = await Zotero.DB.columnQueryAsync('SELECT path AS filepathnames FROM itemAttachments WHERE path IS NOT NULL AND path NOT LIKE ? ORDER BY path','storage:%');
return filepathnames.join('\n');
If you have a Linked Attachment Base Directory setting, all listed file paths will be prefixed with "attachments:". If you need the full path, insert your LABD there in a text editor. If you have no LABD setting, you will get the attachment's full explicit path.
For a list of stored PDF (only) file attachments under Zotero\storage, use:
var filepathnames = await Zotero.DB.columnQueryAsync("SELECT TRIM(key || '\\' || REPLACE(path,'storage:','')) AS filepathnames FROM itemAttachments JOIN items USING (itemID) WHERE libraryID=1 AND path IS NOT NULL AND path LIKE ? ORDER BY key",'storage:%.pdf');
return filepathnames.join('\n');
Of course you could also do a search in your operating system for *.PDF under Zotero\storage (and you could save that as a saved search). It is however possible that might identify PDFs that are not actually known to Zotero (eg PDFs you put there manually, or 'orphaned' PDFs).
The process of checking if your linked-files folder contains files that are NOT linked in Zotero (either deliberately or inadvertently*), or if your Zotero\storage hierarchy has PDF files NOT known to Zotero, is a little more complicated. But it can be done with some external coding (eg comparing the linked-files folder's directory listing to the linked-files list above).
*the fact that Zotero does not delete linked attachments when an item containing them is deleted can lead to orphaned PDFs in the linked-files folder. That's why the delitem plugin should be used routinely to delete linked attachments within Zotero:
https://github.com/redleafnew/delitemwithatt
There is currently no way to save the above code snippets in Zotero for easy re-use, but apparently that is planned.