The way sync works it would be sufficient if the tags get restored on the server from one user and then sync to everyone else - I'm not sure how you think you can track that person down? Or did that student receive a message about tags during syncing?
I don't know about the structure of your group, but my understanding is that all students in an environmental studies (or so?) program are in it. Having a large group in which every users has editing privileges may just mean that you have to deal with issues like this (especially, frankly, when we're talking undergraduate students). If you need to keep a group library highly organized and structured, you should probably think about restricting editing privileges.
Right, so my interpretation of what you say above is that this could possibly happen merely by syncing an old version, not necessarily by doing editing (e.g., of tags) in the group library, then syncing. Anyway, once we experienced the problem (again, with limited users) it became hard to track back to the first instance, but the issue that got me worrying today was one student's use of a shared computer that apparently didn't sync successfully, then a number (not all) of changes I had made to collection names were undone. So in this latter case, I see nothing specifically about tags on the user's end nor mine, and the problem that resulted was with collection names, not tags (possibly more, but the collection names were the obvious thing).
And re. group use challenges, such is social learning, which clearly Zotero folks appreciate! This is an innovative feature of our program (see sge.lclark.edu), and we are slowly working it out. This is where some slightly more granular privileges could be helpful, possibly allowing members to add records but not change tags or collections or whatever...I don't have the granular structure figured out for sure, but a fuller delineation of functionalities could help us work through what is properly within the control of each student, and what admins only should be able to do.
This certainly _shouldn't_ be happening just with an old version.
I'll hope Dan can chime in with some more details at some point, but to illustrate, let's take two users, A&B and 4 time periods from 1-4.
1 - User A syncs
2 - User A modifies tags or collections
3 - User B modifies the same tags or collections
4 - User A syncs again
That's when you get conflicts - for items you get a merge dialogue, but for tags and collections Zotero automates that process - Dan will have to say according to which rules exactly.
On the other hand
1 - user A syncs
2 or 3 - user B modifies tags or collections
4 - user A syncs again
Will not lead to any conflicts, but sync the changes made by B to A's computer.
An unsuccessful sync shouldn't affect any of this. I'd guess this was a coincidence.
Okay, I'm understanding this a bit better. Basically, what is apparently required are two users modifying the same tag or collection, though this doesn't always trigger an error. This is interesting in that, for the most part, our students are working on their own projects: they may be assigning a given tag to one of their items, for instance, but not usually modifying tags, nor modifying tags on the same item. And I'm pretty sure the issue I faced earlier today was that I modified collection names, but no one else did; then I synced and all was fine. But later some (not all, not random...a certain sector in the library) of the old collection names reverted. That's why I suspected that an unsuccessful sync, for whatever reason, messes with the group library a bit, maintaing some uploaded changes but uploading other, older changes (hence reverting).
Looking forward to getting this figured out enough so that we can be confident it won't haunt us in future! I really appreciate all your time and support.
I went back and looked at one of the uploaded databases to see what specific issues you would have run into, at least with that database. Sync logic with conflicting data requires some tradeoffs, and unfortunately you probably ended up on the wrong side of most of the current design decisions.
Some particulars:
1) Conflicts happen when data changes on both sides between syncs. Since you have users who haven't synced in months, any changes they've made during that time could trigger conflicts.
2) As adamsmith noted, tag and collection conflicts, unlike item conflicts, don't show a conflict resolution window, since asking the user what to do about 50 individual tags isn't really going to work. Instead, Zotero handles them automatically, erring on the side of preserving data. To oversimplify, the general rules are 1) if there are metadata changes (tag/collection name or parent collection) on both sides, keep the most recent version, 2) if one side has been deleted, keep the version that still exists, 3) if collection-item or tag-item associations have changed on both sides, restore all associations. The downside of these decisions is that, when edits involve removing lots of tags or removing lots of items from collections, conflicts can cause them to be restored.
Turning off library editing for non-admins is a good step. What that means, just to be clear, is that if any non-admins have local changes to the library and try to sync, Zotero will prompt to reset the library with the server version, overwriting their local changes. But that may be better than the alternative.
Finer-grained permissions might help in the future, though they'd be fairly complicated to implement (and, to some extent, to understand as a user). For now, the best way to prevent these going forward is to make sure everyone's using auto-sync and that people don't continue to make edits if their Zotero syncing isn't working for some reason.
Well, after getting more feedback from students and thinking about the above for awhile, I have a few more thoughts:
(1) Not all students who had problems syncing were fixed by the latest dev XPI. I was thinking we could manually update their group libraries, but since Zotero stores all personal and group library data together we'd be wiping out their personal libraries. I'm now thinking that we could remove them from the group, then have them sync (i.e., only their personal libraries, removing all group library data from their computers), then add them back to the group library so they could sync from scratch as another possible way to fix the problem with these particular students.
(2) The above is not the biggest problem; we still worry about how to control sync overrides from older versions of the group library. As I understand the above, the challenge comes when a student with an old group library (i.e., who hasn't synced in awhile) does something to the group library (or even their personal library?) *prior to* syncing. This could be due to the student not enabling auto-syncing, but I think it's much more probable that (a) the student hasn't used Zotero in awhile, hence syncing is still proceeding while the student is making changes (since our group library is rather large...I know it took about ten minutes to sync from scratch in a trial I did), or (b) the student's internet connection is not active and syncing is not possible. These are possibilities we simply can't control.
(3) Assuming the above will continue to happen on occasion, our group library may revert to older versions. We can backup the group library regularly, and if this happens we can lock it down (allow admin library editing only) and restore. Some recently added student data would be lost, of course, but assuming all data are dragged from their personal libraries to the group library (this is how we teach them to do it), it's not a huge loss. We'll institute this policy immediately.
(4) One reason I'm a bit worried is that the general rules you presented above, Dan, don't seem to be followed in all cases: yesterday, for instance, changes I made to collection names were synced successfully, then mostly (not entirely) undone with some else's sync attempt. It's highly doubtful that this other person edited any collection names, but if the person edited even one would that timestamp undo my changes? And again, all my changes were successfully synced before they were undone, which doesn't follow the general scenario of two sets of unsynced changes.
(5) I'm also worried in that your understandably conservative rules (e.g., if one user has deleted something, keep the version the other user still has) preempt any administrative cleanup we need to do, unless we always lock the library down (allow admin editing only) first, as there's always a possibility of a member working on the group at the same time. Could you add a test for user roles, and always prefer admin syncs over member syncs? This could, in our estimate, solve the problem (I'm assuming that if members e.g. add items while admins are doing cleanup these items would be preserved upon sync). Clearly we're still wrapping our heads around the complexities of sync, but some admin-preference rule seems like it could be helpful.
Okay, thanks again for all your support, and we'll find our way through this yet!
I'm now thinking that we could remove them from the group, then have them sync (i.e., only their personal libraries, removing all group library data from their computers), then add them back to the group library so they could sync from scratch as another possible way to fix the problem with these particular students.
Yes, that's the easiest thing to do (though we're happy to look at their individual issues if they can provide Report IDs).
As I understand the above, the challenge comes when a student with an old group library (i.e., who hasn't synced in awhile) does something to the group library (or even their personal library?) *prior to* syncing.
No, changes to their personal library have no effect. And I'm fairly doubtful that it's an issue of changes within a few minutes. Remember, unless someone hasn't synced in a while or there have been a lot of recent changes (locally or remotely), a sync should be pretty much instantaneous. The issue here was almost certainly just that people hadn't synced in months. That would slow down syncing for everyone as all their changes were dealt with in all synced libraries.
One reason I'm a bit worried is that the general rules you presented above, Dan, don't seem to be followed in all cases: yesterday, for instance, changes I made to collection names were synced successfully, then mostly (not entirely) undone with some else's sync attempt. It's highly doubtful that this other person edited any collection names, but if the person edited even one would that timestamp undo my changes?
No, but like adamsmith, I'm still not clear on how you think you know whose sync it was that reverted the changes if this was a large group.
And again, all my changes were successfully synced before they were undone, which doesn't follow the general scenario of two sets of unsynced changes.
They synced months ago. At some point, they edited collections (before or after you, but before they synced). You recently edited those same collections and synced. They finally synced. The collections are in conflict.
as there's always a possibility of a member working on the group at the same time.
The important thing to keep in mind here is that, if auto-sync is enabled and working, all edits should be synced within 15 seconds, and anyone making local edits should get new changes within 15 seconds.
Ultimately, though, you may just be pushing the limits of what can be done reliably in a group—though keep in mind that this probably happened largely because of the bug that was preventing some of your members from syncing. A future version of Zotero might add support for finer-grained permissions (e.g., read-only for select members), which might allow you to use multiple groups that everyone could still access within Zotero but that not everyone could write to. (You can do this now only with admin/non-admin.)
Okay, thank you again for your careful response; I sure appreciate it. We'll process this a bit more and figure out a workable procedure on our end. I may report another sync issue soon (on another thread), but will investigate it a bit first.
Upgrade Storage
I don't know about the structure of your group, but my understanding is that all students in an environmental studies (or so?) program are in it. Having a large group in which every users has editing privileges may just mean that you have to deal with issues like this (especially, frankly, when we're talking undergraduate students). If you need to keep a group library highly organized and structured, you should probably think about restricting editing privileges.
And re. group use challenges, such is social learning, which clearly Zotero folks appreciate! This is an innovative feature of our program (see sge.lclark.edu), and we are slowly working it out. This is where some slightly more granular privileges could be helpful, possibly allowing members to add records but not change tags or collections or whatever...I don't have the granular structure figured out for sure, but a fuller delineation of functionalities could help us work through what is properly within the control of each student, and what admins only should be able to do.
Thanks again,
Jim
I'll hope Dan can chime in with some more details at some point, but to illustrate, let's take two users, A&B and 4 time periods from 1-4.
1 - User A syncs
2 - User A modifies tags or collections
3 - User B modifies the same tags or collections
4 - User A syncs again
That's when you get conflicts - for items you get a merge dialogue, but for tags and collections Zotero automates that process - Dan will have to say according to which rules exactly.
On the other hand
1 - user A syncs
2 or 3 - user B modifies tags or collections
4 - user A syncs again
Will not lead to any conflicts, but sync the changes made by B to A's computer.
An unsuccessful sync shouldn't affect any of this. I'd guess this was a coincidence.
Looking forward to getting this figured out enough so that we can be confident it won't haunt us in future! I really appreciate all your time and support.
Thanks,
Jim
Some particulars:
1) Conflicts happen when data changes on both sides between syncs. Since you have users who haven't synced in months, any changes they've made during that time could trigger conflicts.
2) As adamsmith noted, tag and collection conflicts, unlike item conflicts, don't show a conflict resolution window, since asking the user what to do about 50 individual tags isn't really going to work. Instead, Zotero handles them automatically, erring on the side of preserving data. To oversimplify, the general rules are 1) if there are metadata changes (tag/collection name or parent collection) on both sides, keep the most recent version, 2) if one side has been deleted, keep the version that still exists, 3) if collection-item or tag-item associations have changed on both sides, restore all associations. The downside of these decisions is that, when edits involve removing lots of tags or removing lots of items from collections, conflicts can cause them to be restored.
Turning off library editing for non-admins is a good step. What that means, just to be clear, is that if any non-admins have local changes to the library and try to sync, Zotero will prompt to reset the library with the server version, overwriting their local changes. But that may be better than the alternative.
Finer-grained permissions might help in the future, though they'd be fairly complicated to implement (and, to some extent, to understand as a user). For now, the best way to prevent these going forward is to make sure everyone's using auto-sync and that people don't continue to make edits if their Zotero syncing isn't working for some reason.
Regards,
Jim
(1) Not all students who had problems syncing were fixed by the latest dev XPI. I was thinking we could manually update their group libraries, but since Zotero stores all personal and group library data together we'd be wiping out their personal libraries. I'm now thinking that we could remove them from the group, then have them sync (i.e., only their personal libraries, removing all group library data from their computers), then add them back to the group library so they could sync from scratch as another possible way to fix the problem with these particular students.
(2) The above is not the biggest problem; we still worry about how to control sync overrides from older versions of the group library. As I understand the above, the challenge comes when a student with an old group library (i.e., who hasn't synced in awhile) does something to the group library (or even their personal library?) *prior to* syncing. This could be due to the student not enabling auto-syncing, but I think it's much more probable that (a) the student hasn't used Zotero in awhile, hence syncing is still proceeding while the student is making changes (since our group library is rather large...I know it took about ten minutes to sync from scratch in a trial I did), or (b) the student's internet connection is not active and syncing is not possible. These are possibilities we simply can't control.
(3) Assuming the above will continue to happen on occasion, our group library may revert to older versions. We can backup the group library regularly, and if this happens we can lock it down (allow admin library editing only) and restore. Some recently added student data would be lost, of course, but assuming all data are dragged from their personal libraries to the group library (this is how we teach them to do it), it's not a huge loss. We'll institute this policy immediately.
(4) One reason I'm a bit worried is that the general rules you presented above, Dan, don't seem to be followed in all cases: yesterday, for instance, changes I made to collection names were synced successfully, then mostly (not entirely) undone with some else's sync attempt. It's highly doubtful that this other person edited any collection names, but if the person edited even one would that timestamp undo my changes? And again, all my changes were successfully synced before they were undone, which doesn't follow the general scenario of two sets of unsynced changes.
(5) I'm also worried in that your understandably conservative rules (e.g., if one user has deleted something, keep the version the other user still has) preempt any administrative cleanup we need to do, unless we always lock the library down (allow admin editing only) first, as there's always a possibility of a member working on the group at the same time. Could you add a test for user roles, and always prefer admin syncs over member syncs? This could, in our estimate, solve the problem (I'm assuming that if members e.g. add items while admins are doing cleanup these items would be preserved upon sync). Clearly we're still wrapping our heads around the complexities of sync, but some admin-preference rule seems like it could be helpful.
Okay, thanks again for all your support, and we'll find our way through this yet!
Jim P.
Ultimately, though, you may just be pushing the limits of what can be done reliably in a group—though keep in mind that this probably happened largely because of the bug that was preventing some of your members from syncing. A future version of Zotero might add support for finer-grained permissions (e.g., read-only for select members), which might allow you to use multiple groups that everyone could still access within Zotero but that not everyone could write to. (You can do this now only with admin/non-admin.)
Jim