So syncing does not apparently work well for you. This is frustrating, but there's little reason to make accusations (especially when it does work for some of the rest of us).

Zotero syncing does not work in the way you described. Metadata syncing is atomic: it is all-or-nothing. Auto-syncing is of great benefit to those users where syncing consistently works. The Zotero server code is released & others have set it up to run. It is just not supported. I personally think it is fair that the Zotero developers get to choose how they prioritize their time and resources (among other things, to fulfill promises to funding agencies).

No that isn't right. It *does* work for me - perfectly.

But only when the server isn't busy (which I'm measuring by the server queue re-try responses).

I *haven't* made any accusations; I've made feature requests. In fact I said used the word "perfect" w.r.t the software. The primary request I made is that it advertise the server load, and advertise what is happening during a sync more clearly (not necessarily techno-speak).

I think the software should be clearer about what it is doing, so that people feel in control of it (without opening the source code)

Auto-syncing is not a feature I suggested being removed; I suggested that it be off by *default* to avoid congestion issues.

That's not a silly suggestion.

Very many small queries to a server looks to me like a DOS attack. Like I said.

ajilon - I'm not actually syncing. I deleted from the server, and I'm up-loading de-novo from my local repository.

Absolutely we need an indicator, and more information about what's happening?

why? because I need to access this data from another part of the country. And I'm going to miss my train because I expected it to work the way it did in the past!

It's not a complaint, still less an "accusation" as some would have it (???)

It's a *problem*. If it's network congestion, I'll pay for bandwidth happily (rather than storage).

If it's an issue of design, I'd respectfully suggest it be thought about (I agree certainly that it should be assembled locally and sent atomically however).

But not worrying about it isn't always an option for busy poeple. And *no* indicator and *no* choice about whether and when to sync to a repository would leave me, and I think many of my colleagues, feeling helpless as lambs, and maybe looking for other offcampus solutions.

Zotero remains the best I've tracked down so far.

I *haven't* made any accusations

By "accusations", I merely meant that you seemed to be mis-diagnosing the issue & then describing design decisions as "nuts" and "unfair".

Auto-syncing is not a feature I suggested being removed; I suggested that it be off by *default* to avoid congestion issues.

But not very many users would want it off. Those who did can easily turn it off.

I'm not actually syncing. I deleted from the server, and I'm up-loading de-novo from my local repository.

Why did you do this? Unless you are debugging something in particular, this is a waste of time. You have a much larger sync operation than would have to be performed compared to if you just had a few records to add/update on the server. Since syncs are all-or-nothing, your sync should be of no better quality this way (and, factoring in the time, leaves you in a worse position overall).

Would it help if sync status information were available through Zotero.org? I know that Dan can often tell people what has been recently synced and whether they have syncs queued-- if such information were available to users directly, it might help.

Another question is when it is safe to disconnect from the network, which reveley raised above. Can one of the Zotero developers explain? Specifically, if Zotero tells me that it is waiting for updated data or is otherwise queued, can I disconnect without the sync failing?

reveley: So, for what it's worth, and as has been more or less pointed out above, basically all of your assumptions for how syncing works (SQL statements, increasing retry times, auto-sync, "network congestion") are mistaken. There's of course no reason or need for you to know how it works, but there's also not much reason to suggest remedies based on quite erroneous assumptions.

SQL statements in debug output are local operations and have nothing to do with the server side of syncing.

The increasing retry times you see in the debug output are by design. They follow a server-dictated back-off schedule and don't indicate that the sync servers are slowing down—they're what prevent DOSing of the web servers when the sync servers are busy.

Auto-sync is in fact extremely beneficial to sync performance. If syncing consisted solely of small auto-synced operations, it would be more or less instantaneous for everyone. Large operations are what slow it down, since, as noted above, sync transactions are atomic, and transactional lock contention is currently by far the biggest detriment to sync performance.

Sync performance has nothing to do with network congestion. We have plenty of bandwidth.

And as noksagt noted, you would've made things much worse for yourself by clearing server data and resycing, if that's in fact what you did.

We've made more or less constant improvements to the sync architecture to handle increasing server loads. As I've said before, on busy days queue times peak at about one hour for the longest (and largest) sync operations in the middle of the day U.S. time. Most syncs take much less time, and for most of the day the vast majority of operations should take only seconds.

Overall queue time aside, I would say the main practical problem at the moment is that, while a sync is queued, any local changes since that sync was sent up aren't sent to the server, which is a real problem for people who want to, say, catch a train. So while it would be somewhat detrimental to server performance and cause an overall delay in a person's data showing up on the server, it may be preferable to have new local changes revoke queued uploads that haven't yet entered the processing queue. That way a user could know that all changes they've made so far have been sent up and shut down their computer and head to the train station. (It's not 100% certain that queued uploads will go through successfully, but nearly all potential errors are caught within a few seconds by an error preprocessor and returned to the user.)

To be entirely clear, we would still like to (and intend to) improve sync performance dramatically, particularly for the high-visibility cases that ajlyon describes, and I'm hoping to do so in the next few weeks (particularly since the school year is starting).

In the meantime, we might be able to provide some additional sync status information, but it's not entirely clear to me what would be useful. Users can already see what has synced by checking the website. Individual queuing state might be helpful, but only for the fairly rare cases where the sync icon is spinning without a queued request on the server—which is a bug if it happens. Current size of the queue and maximum wait time would give some indication as to overall server load, but they could also be misleading, as the queue is processed from a number of directions, which is why Zotero doesn't display any sort of ETC for syncs. (We could try to, but I'm not sure how accurate we could get it.) And there's not a whole lot a user can do with that information.

But some architecture improvements to reduce lock contention and potentially the queuing change I describe above are the things that would make the biggest difference for users.

I cleared the system because it's buggy, as well as slow btw. it repeats items on sync.

Syncing should not create duplicates. As far as I know, nearly all reports of duplicates being generated on sync have been because there really were two separate records that have two different keys corresponding to one reference. This may happen if you have duplicates in a single zotero library or it could happen if you sync from two or more machines that have different keys for the same items. The latter can happen if you were merging two or more somewhat different libraries that were gathered manually or if you had been using some manual method to sync (particularly import/export) or if you had copied the zotero 1 data directory to multiple machines, upgraded to zotero 2, and then synced. If none of these describes your situation, it is very unusual & should be looked into.