Group syncing to own, private storage

For private group uses, a private file storage solution would be essential, for legal and privacy reasons. There is an earlier closed thread about this at http://forums.zotero.org/discussion/9485. As sean suggested, I opened a new thread for this, as I did not find an open one.

Would it be possible to get an update on the status of this feature request? In the earlier thread the developers argued that technically it can not be achieved with WebDAV, but the discussion seemed to indicate that one direction to consider for getting such a feature would be to base it on private installations of Zotero File Storage. Would that be a feasible strategy?

Please do not argue that the private storage option for groups is not necessary (that argument consumed most posts in the earlier thread) - some people and institutions think that it is and really need it to make Zotero a viable standard for the groups in their organizations, so it would be nice to focus on how it could be done. If Zotero developers are not interested in doing it, maybe they could give an idea of how someone else could do it.

[Please accept my apologies if this matter has been taken up again somewhere else and I just missed it.]
  • There is no update on WebDAV and I'm not aware of any work on this.

    The Zotero server code is now available in full (it has been for a while as I note in that thread):
    http://www.zotero.org/support/dev/dataserver_setup
    so it's theoretically possible to run your local instance of the Zotero server, but Zotero does provide no support for getting local server instances to work (though there are some non-Zotero people on the zotero-dev listserv who frequently answer questions on that). This is a relatively demanding solution technically, though.

    Since you're only interested in storing files locally and storing metadata with Zotero doesn't seem to be a concern, you can also use links to attachments on a local server. Depending on your set-up it may even be possible to automate that using the Zotfile extension. I haven't explored that in depth, but my guess would be that this is only feasible if all users in the group are using the same family of operating systems (i.e. *nix or Windows).

    Finally, note that it would be possible to have everyone in the group share a single user account and use a local WebDAV version. That should be relatively easy to do.

    (And no one in the other thread argued that there are no legitimate reasons for storing files locally - there obviously are privacy, security, and even legal concerns in many cases. We just argued that there are also legitimate reasons why this isn't easily possible for groups and that there aren't any sinister reasons for that).
  • It's a real shame this can't be done. I've used Zotero personally for a couple of years to manage libraries of journal articles. Other people in the company wanted to have access to the same functionality and we wanted to store company documents in the same groups that were relevant to projects and articles. Unfortunately the company is heavily against any form of storage outside it's own servers, even for the copyrighted pdf's of articles so it looks like I'll have to look for a commercial solution similar to Zotero and Zotero will lose my subscription and I'll lose Zotero. Not a happy solution but without server based group storage the functionality isn't there that we need. The other workaround solutions take away the ease of use. A single user solution's no good because the corporations too big and the username and password will soon be too widely circulated. It's really disappointing because I think the tool is great but unfortunately won't be suitable for a security conscious corporation.
  • mjsim: To be clear, you're asking for WebDAV support for group library files, yes? Storing the group item data externally isn't a problem? (If it is, then you're asking for something much different and much more difficult, as explained in adamsmith's comments about the dataserver above.)

    WebDAV support for group libraries isn't really on the near-term roadmap, though we are planning to support linked files in groups with per-library relative paths, at which point, if you were willing to use linked files, you could use some other central store that everyone in your company had access to (which could probably even be WebDAV, with some help from the OS).
    I'll have to look for a commercial solution similar to Zotero
    A "solution similar to Zotero" that supports WebDAV file storage for group libraries is a pretty specific request, so I'm not sure what you'd find, but if you're willing to pay for a commercial solution, you could also possibly just pay someone (perhaps us) to build this functionality into Zotero.
  • Thanks Dan,

    We'd need any storage solution that allows our files to be kept on our secure servers. WebDAV is just the solution that zotero partly supports now so that's why I mentioned it. Any solution that allows us to sync the stored files on the company secured servers would work. We can live with storing the group item data externally as that won't contain anything company confidential or copyrighted.

    The only thing I worry about with linked files is that creating the items and linking the files will take extra time. One of the things I like most about Zotero is the ability to save files to the library really quickly, rename files from the item header and move things around simply. I'd worry that some of those steps would be lost or take longer once the file was only linked to the item rather than part of it.

    Paying for it is a possible option but it'd take a while to get the budget for it (as would buying commercial software, which as you say don't quite do the same things, it's the simplicity and functionality of Zotero that makes it so attractive. Not the low cost).

    Our IT team is global and very competent so if there are technically demanding solutions to the problem (that don't compromise functionality to the end user, who won't have the skills to change setups, fix bugs etc) then I can probably get someone on it, but if those solutions require consistent IT support to maintain and manage whatever the solution is then there'd be less support as no one likes to take on extra work.
  • @mjsim -- There appears to be a growing group of institutions that are implementing the Zotero dataserver (for item metadata) and storage; you may want to post to zotero-dev (http://groups.google.com/group/zotero-dev) with your situation. I think we'll reach a tipping point relatively soon where we will have such users supporting one another and providing a viable server-in-a-box experience for Zotero. If you have a bit of institutional support on your end, that might be what it takes to push this over the edge.
  • Someone has (quite a long time ago) requested the creation of a Debian package to allow for easy installation of zotero-dataserver on Debian and its derivatives. That request is at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=709925
    but nobody has taken it up as of this writing.

    I've created an issue proposal on freedom sponsors in case anyone wants to contribute a bounty to try to move this along. The page is
    https://freedomsponsors.org/issue/661/rfp-zotero-dataserver-dataserver-for-the-zotero-client
  • edited October 9, 2015
    Just to get this right and i am no programmer whatsoever-
    Do i understand correct, that it is possible to create a debian/ubuntu package (or another format that is easy to work with for most people) from the source code of zotero to get your own server (that you can install whereever you like)?

    So basically if we get enough people donating for that someone could build and maintain that package for amount x?
  • Essentially yes--not sure that a deb package is the right way to go, but it would very likely be possible to package the server in a more digestible way.
    You'd also have to use patched Zotero versions, but those are pretty minimal changes, so if you have someone to maintain the server, it'd be easy enough to maintain those.

    Note, though, that the problem isn't just the money but also to find someone to do it--recruiting good developers is harder than you may think it'd be.
  • thanks for your clarification adam.
    If you don't think that deb would be the right way to go - what would in your opinion be the best solution?
  • docker would probably be the most common approach
  • I'm curious to see the sync server source (and the standalone client source).

    This link (posted earlier) does not help ..

    http://www.zotero.org/support/dev/dataserver_setup

    Can you point me to the source repo?
  • edited October 10, 2015
    All Zotero code is at
    https://github.com/zotero/
    There are various discussions, instructions, and links to more info&repos on zotero-dev, too.

    edit: Dan deleted the page linked to above because it was outdated to the extent of being misleading and since server installation is probably better hosted by the people who actually are going through the trouble of documenting/maintaining it.
  • There isn't one "right" way to go. If you have a Debian/Ubuntu server, a good deb package lets you check a box/run a command to install zotero-dataserver and have it up and running pretty quickly, with streamlined updates through the system package manager. If you don't have such a server, yes, a deb package doesn't do much for you.
  • correct, in technical terms there is no right way. I mean "right way" in the sense that--if you're going to spend serious money on it and find someone to develop&maintain it, it probably makes sense to go with a more flexible solution than .deb, which is why I suggested docker might work. But as I say, the main issue is going to be to find someone. The implementation details only matter if there's any chance of this happening at all. I once asked on the dev list if anyone running a successful server was interested in doing this for pay for a client and no one expressed any interest at all.
  • Fair enough. There is a non-zero chance of something happening, though. I've done some work for Debian and was considering making this package myself. It just hasn't been a priority for me yet.
  • A Debian package would be fantastic!
  • Has anyone succeeded in syncing Zotero data using a private server? I was trying to find some details on this in Zotero's github repos, but no results so far.

    It seems like there is no way to change the sync server URL via Zoteo GUI now, but if you open about:config, there will be a few parameters that contain 'sync' in their names and URLs in values.

    Having a few accounts with some shared group libraries, with all of the data synced entirely via a private network sounds an extremely useful idea for lots of organizations. Would be interesting to know about someone's experience in doing so.
  • edited June 2, 2016
    No; Group syncing is via the zotero server only. The Zotero server relies on amazon s3.
  • There are a handful of groups running private Zotero servers for groups. You can find several discussion on that involving several players on the dev list
    https://groups.google.com/forum/#!forum/zotero-dev
    You're right, though -- at least at this time, this also means using a (lightly) patched client (that's on purpose, so that the alternate version is logged clearly for any error reports).
  • Thank you for the link adamsmith! I looked through the topics, but could not any solutions there unfortunately. These are the most relevant discussions that happened within the last 3 yrs or so:

    https://groups.google.com/forum/#!topicsearchin/zotero-dev/subject$3Aserver/zotero-dev/MEwLaptJIzI

    https://groups.google.com/forum/#!topicsearchin/zotero-dev/subject$3Aserver/zotero-dev/TFA7FKGxZHE

    https://groups.google.com/forum/#!topicsearchin/zotero-dev/subject$3Aserver/zotero-dev/tF1M3FZ-GLk

    If anyone knows anything else, it would be just great if you could share! Running a custom private Zotero sync server in a local network would be just a paradise for my team!
  • I don't understand -- http://git.27o.de/dataserver/about/ on that link has the full code&setup you need for running the local server.
Sign In or Register to comment.