Group sharing through WebDAV and linked base directories

mateolan

Several discussions have arisen on this site about the lack of functionality for groups to leverage shared files on linked servers and/or WebDAV in the same way that individuals can utilize these resources. Core Zotero developers have stated that this process would have to come from the community, because of lack of resources. Therefore, I am starting this discussion thread to enable focussed comments about the best way to proceed. Not having familiarity with the Zotero code base, and being an only an occasional programmer, my first hope is that knowledgeable core developers will contribute wisdom about the best places in the source code to enable this feature. My second hope from this thread is for other members of the Zotero community to contribute meaningful (and concise!) use cases for guiding development efforts. Finally, it would be great if this discussion could motivate people who are currently users but non-developers make the leap into Zotero on GitHub, and contribute to community code development.

dstillman

If someone is interested in this, I'd recommend starting with extending base directory support to groups rather than group-based WebDAV. The latter is considerably more complicated and can't really happen until after some major upcoming architectural changes in Zotero.

mateolan

OK great. Could you please point out the relevant section in the code where 'base directory' is located. Also, are the upcoming architectural changes documented somewhere? It would be a shame to invest time in working on something, only to see it obsoleted by a new architecture.

adamsmith

the base directory changes aren't going to be affected by the sync architecture update. You shouldn't spend time on anything that directly involves syncing until Zotero 4.1 has been rolled out as a beta.

The commit that added relative linked attachment file paths is here:
https://github.com/zotero/zotero/pull/51/files
but this is not a small feat, it will require some time familiarizing oneself with zotero code more generally.

jsfzot

It seems that current Zotero business model is at odds with the users' desire to link Zotero to locally networked files instead of the cloud. Is that correct? If so, perhaps the creation of a paid Zotero Pro product, that allows linking to networked files rather than the Zotero server, would help provide the necessary resources to create and support the features that Zotero Group users have been requesting.

adamsmith

It's not as much the business model as you think it is. Dan has said repeatedly that they would accept a patch to this extent (as they did for relative file paths for personal libraries, which similarly affects the business model; this did include, as all complex pull requests do, a pretty significant amount of core dev time for code review, as you can see when you follow my link above). I'd also guess that the total effect that this would have on storage subscriptions is pretty marginal.

It's just that someone has to do it and core devs are busy with plenty of things with higher priority, including the sync architecture change that Dan mentions. Many other features, entirely unrelated to Zotero's revenue stream, have also long awaited implementation.

Charging for open-source software (rather than services related to open source software) would be hard and creates incentives to fork Zotero that I think the Zotero team has every reason to want to avoid.

mateolan

Thanks for the link @adamsmith. You are correct when you say "no small feat".
I don't know why, but in my mind, it seems like it would be simple for each person who used a base path architecture to have it configured something akin to:

storage/MyUsernameLibrary/FolderCodeN/pdfname.pdf
storage/GroupNameALibrary/FolderCodeX/AttachmentName.pdf
storage/GroupNameBLibrary/FolderCodeY/AttachmentName.html
etc.
In this fashion, Group Admins could choose to share or not the attachment folders with others through a file-sharing service. Zotero could remain agnostic as to whether the files were shared, and those with out the shares simply wouldn't get the attachments.

I know I am oversimplifying to be sure--but what essentially am I missing?

dstillman

The implementation.

mateolan

Indeed...While I appreciate your humor (really), I was wondering whether my conceptual architecture fell in line with your logic.
Assuming that is the case, then my next question is whether there is a use case for NOT making this architecture default.
That is, by having:
storage/MyUsernameLibrary/FolderCodeN/pdfname.pdf
for private libraries, and
storage/MyUsernameLibrary/GroupName/FolderCodeN/pdfname.pdf
for group libraries
(of course substituting BasePath for storage if users chose that option)
Is there any functionality that would be lost by having these as default settings?

adamsmith

Zotero doesn't have anything to do with the directory structure of linked files. Zotfile can do some of that if you want to (though it doesn't currently work for groups, so someone would have to expand that if that functionality is wanted), but Zotero itself is entirely agnostic about this and I think that's how it should be, so that people can set this up any way they want. So, no, you shouldn't require any default folder structure for relative paths to work and I also don't see why that should be necessary. In the above example, if the base directory is just set to "storage" everything will work just fine.

One thing that I'm wondering about is whether it'll be necessary to have a separate base directory for each library. That's a giant mess GUI wise, but if you're, e.g., hosting your own files on Dropbox, but also want to share a group via a network drive that'd be necessary.

If you're talking about the structure of attached files in the Zotero storage directory, those are a separate topic and not related to the issue of relative file links at all.

mateolan

"One thing that I'm wondering about is whether it'll be necessary to have a separate base directory for each library. That's a giant mess GUI wise, but if you're, e.g., hosting your own files on Dropbox, but also want to share a group via a network drive that'd be necessary."

YES!
However, to be frank, I don't gain from linking files using ZotFile because the directory paths are based on file metadata, rather than user/group information.
I currently have my storage folder synched through my campus's cloud storage. It works well for me on multiple computers. If there were user/group name folders between the base storage folder and the article folders, I could share them appropriately.
In terms of coding, it seems to me that if username/group name were inserted as part of the variable for standard file path within Zotero storage, then this would result in relatively little code change--just altering the places where the SQL statement(s) fill the path variable...I am not sure why any of this would be a "GUI nightmare".

I also can't see a downside to moving to this architecture in terms of loss of function...or what am I missing?

adamsmith

sorry, but you need to familiarize yourself better with Zotero before this makes sense as a technical conversation. If you're talking about the Zotero storage folder, you're not talking about relative links and you're not talking about a base directory in Zotero terms. Those are the things that Dan and I are talking about and those would be the most promising avenues to look at for getting a version of file storage for groups at this point.

Playing around with the storage folder _may_ be possible (Dan would have to say which other purposes the flat folder structure serves - it certainly would affect sync), but it's not currently on the menu and it wouldn't help users, e.g., to put files onto a network drive unless you're playing around with symlinks and the like—which is possible, but probably not the general user experience Zotero is going for.

mateolan

Perhaps I switched tacks too quickly...I certainly don't think it is a familiarization problem (although, again, correct me if I am wrong).
I realize that you pointed me down the Base directory path, and since you mentioned ZotFile, I added it to my reply.

Thinking it through, if this change is made in the native Zotero path structure--it could easily be extended to the Base Directory module mentioned above, as well as to the WebDav storage...this way, group-library level sharing would be enabled for all of these user experiences.

I'll restate that as far as I can tell (awaiting Dan's feedback), there is no user-experience downside toward adding a library-level grouping layer in the Zotero storage path. There is however, a tremendous upside in terms of improved functionality for users who take advantage of any kind of networked storage.

dstillman

OK, "base directory" (which you called this in the thread title) means something specific, and as adamsmith says that's totally separate from the storage directory — that's why we were confused. All previous conversations about linked files in groups relate to the former, since linked files aren't stored within the storage directory at all. And also as adamsmith says, base directories are really the user-facing mechanism for this. A handful of users or their admins may play around with symlinks in the storage directory, but that wouldn't be something we'd recommend as a general solution.

Anyhow, library-specific subfolders for the storage folder are planned — it's just the way it is now because it predates group library functionality — but migration will be extremely tricky. We probably need to make 4.0 fall back to the new structure if it can't find a file in the old structure, and then make the next major version gradually migrate the files in the background and write new files to the new structure. But I don't think we can actually move directories currently, so we'll need to do this file by file even for HTML snapshots. And thanks to security software, disk access can fail at any time, so there will need to be all sorts of error handling.

But none of that has any bearing on group-specific base directories, which would be a totally separate mechanism and require the fairly complicated UI that adamsmith mentions (because yes, it would need to be configurable per library).

mateolan

Right. ..thinking about it a bit differently... It seems the complicated UI stuff might be solved if the logic for the groups were somehow built into the key...what if as an intermediate step, instead of having an additional layer of folders, the library and key ids were concatenated
so for libraryID=123456, and itemKey='SUJ9RMU9' the folder name could be '123456-SUJ9RMU9'..
this isn't a perfect solution but would allow folks to share all of the networked folders belonging to specific library, without introducing some of the headaches you mentioned above.

dstillman

Sorry, but this discussion is a waste of everyone's time. You're not understanding the issues here and are still conflating two completely unrelated things. (The UI is for base directories and doesn't involve key-based filenames in any way, and a directory name prefix isn't useful for any sort of network-based sharing that I'm aware of and doesn't help with migration anyway and certainly wouldn't be an intermediate step.) If someone wants to work into either of these things, they can familiarize themselves with the existing code and functionality and submit a patch, but I can't discuss this further.