Reset to Server - how do I know when it's done?

james.k.russell

I recognize that Reset to Server is for exceptional situations, and have had it recommended that I use it. I've converted a large EndNote collection (30,000+ items) to Zotero on one computer (my laptop), culled it hard (now ~11,000 items), and now need to coordinate usage between two computers (laptop & office PC). On laptop, I've "reset to server". On the office PC, I've turned syncing off, and have copied the whole Zotero directory from the laptop. So far, so good. But when I go on Zotero website and look at My Library, it is evidently partly done (I see only some old collections, no new collections, and I see items I know I've culled). Web access is very cumbersome with a collection this size. How can I know Reset To Server has been accomplished so I can turn on AutoSync from both computers and let things get back to normal?

dstillman

Syncing is all-or-nothing, so there's no such thing as a partial sync.

You have a large queued upload from a month ago on the server, and Restore to Server won't do anything as long as that's there. (In theory you should see an error to that effect.)

Assuming you're trying to overwrite what's on the server, would you like me to clear your queued upload? You can then do the reset again.

Make sure auto-sync is disabled everywhere for now.

james.k.russell

Dan, yes, please kill the queued upload. Will keep auto-sync off until I know all is clear.

Anything I can do to help? Would it help to transfer the zotero.sqlite file?

Thanks for your help.

dstillman

OK, you're cleared. The Restore to Server should go through now.

james.k.russell

thanks again

james.k.russell

The computer that was doing the reset crashed (due to an unrelated problem), apparently before the process was completed. The web site shows no change. I can't see whether the process is queued or aborted. Should I start it again (from the other computer), or would this possibly mess things up more?

dstillman

The reset went through—you can see for yourself that you don't have any items on the server now. So if your local library is how you want it, you can just do a normal sync.

james.k.russell

Actually, I now see 8889 items in the Library on the web site, which is most (but perhaps not all) of the items on my laptop (it shows 11103 items when I select all - but perhaps the discrepancy stems from counting sub-items in one environment by one method (the web site reports a total number of items) and not the other (the standalone does not, which is why I used select all)? In any case, the web site now clearly reflects the reset (with the large amount of trash removed).

However, I notice on the web site that the old categories remain, and none of the new categories are present. I recall from reading the documentation about categories and tags that categories will not transfer between libraries, though tags will. I suspect this is the basis of this problem. I like the hierarchical functionality of categories.

While I can see that it may not be possible to sync the categories to the server (and I am okay with that, as I don't intend to access the library via the web site much), I would like the categories to be synced between the desktop and the laptop. An approach to this that might work would be for me to use auto-sync (to the server) only with one of the computers, and to keep the two computers essentially identical by copying the zotero.sqlite file between them with external file synchronization software (which I have to run anyway for other purposes). Please let me know if you see a problem with this approach (other than the time it will take due to rather large file size).

Thanks.

adamsmith

by "categories" you mean collections?
They absolutely sync, any contrary information is flat wrong. They should by syncing to - and be visible on - the online display of the library, too.

james.k.russell

Yes, sorry, collections. They have not synced yet, in any case. On the online library, there are:
JKR EndNote Lib__A
JKR EndNote LibB
JKR EndNote LibC
JKR EndNote LibD
JKR EndNote LibE
JKR EndNote LibF
Trash

On my laptop, there are no references to EndNote, and there is a hierarchical structure with:
My Library
_temp
intriguing
working
Author
Author_favorite
(38 different author collections)
Author_first
_error
_none
(26 collections, one for each letter of the alphabet)

Topic
(6 different subject matter topics so far)

Is this perhaps because I have only Reset to Server so far? I haven't turned on auto-syncing yet, as I'm not sure I have everything quite figured out in this respect yet.

Thanks again.

dstillman

You now have 18,000 items on the server, but you have a massive 60MB upload that's stalled on the server. I don't know how you ended up with an upload so big, but that will never finish.

You're going to have to either reduce the amount of data that's syncing or wait (a number of months) for Zotero to be able to sync libraries of the size you're currently trying to sync. I see many deleted items here. How many items are in your trash?

Regarding collections, syncs or all or nothing, so either the online library reflects the state of your library at some point or you synced from another machine as well.

I haven't turned on auto-syncing yet

Auto-syncing doesn't do anything special—it's just a normal sync that happens automatically. Restore to Server clears your online library and performs a normal sync.

james.k.russell

I emptied the trash, and now it has 25153 items in it again. In any case, zotero.sqlite is currently 178 MB.

Am I pushing this beyond supported limits? Perhaps I should go back to EndNote. I really prefer some functionality in Zotero, but it doesn't seem to be up to a library of this size (after culling, ~11,000 items currently).

dstillman

I emptied the trash, and now it has 25153 items in it again.

What do you mean by that? What's the "it" there?

Am I pushing this beyond supported limits? Perhaps I should go back to EndNote.

Are you having problems locally, or just with syncing? Zotero itself should work OK with a library of this size, and many people also build up databases of that size and use syncing—the potential issue is trying to sync the entire database at once. But assuming your trash is clear, you can try another sync.

As I noted above, we're working on some syncing changes that will better allow very large libraries to sync. In the meantime, it can take some work to get them to go through. Many people do the import in parts, syncing as they go. But you can try again first.

james.k.russell

The "it" was the trash. I've since emptied it. zotero.sqlite is now 109 MB after emptying the trash (again). Still very big - if you consider a 60 MB upload prohibitive, as your remarks above suggest. If 60 MB will require literally months to finish, I don't see how I can use syncing. It seems like I should go to syncing zotero.sqlite between the two computers I use simply as a file, and avoid the online site. This is how I used EndNote.

dstillman

The upload size and the database size aren't the same.

james.k.russell

One being much larger? Which?

dstillman

They're not particularly related. Just try the sync again.

james.k.russell

= reset to server for present purposes?

dstillman

That depends on what the 18,000 items on the server currently are. If those are items that you've since deleted, it'd be better to start again with a Restore to Server. If those are all items you want and you just added more, a normal sync is better.

james.k.russell

Sorry this is so much trouble. I'm willing to keep trying, and I appreciate your help. But, I want to avoid making another misstep. You reference 18,000 items on the server. When I look through a web browser, it reports 8889 items (and still has only old, not new, collections organization). So, it is hard for me to tell which to do next, as your information and mine do not seem to agree.

dstillman

My numbers all include child items. It's 8,889 top-level items.

The old collections organization had to have existed at some point after the reset, either on this computer or another synced computer.

dstillman

Your new upload was still 60MB.

dstillman

There are over 8,600 trashed items in that upload, and over 32,000 non-deleted items. It seems like you may have synced before you actually finished the cleanup?

There are also 32,000 deletions in the upload—that is, items you've purged from the trash locally that the client is telling the server to remove—so a Restore to Server, which will clear the delete log, is actually probably better. But you should only do that after making sure that everything you don't need has been removed from the library root and that the trash is empty.

(As a side note, one part of the issue here is that the RIS translator recently started creating a separate note for all the non-standard fields that can't be mapped to Zotero fields. That's a clean place to put the data, but all the extra items—3 or 4 for each top-level item—do create an overhead for syncing. It might be better for the RIS translator to add the extra fields to either a single note or the Extra field. This is an issue for the translator maintainers to discuss elsewhere, but for you, if there are specific non-standard fields that you don't need, you can search for them by their tag name, two spaces, a hyphen, and another space, all in quotes—e.g., "AD  - "—with the quick search bar set to All Fields & Tags. Then do a Select All with Ctrl-A/Cmd-A to select just the matching notes, delete the notes, and empty the trash. You'll want to do this cleanup before doing another Restore to Server.)

james.k.russell

As you know, I've been having difficulty getting the my online library synced with my standalone installation. It occurred to me that my corporate firewall could be part of the problem. I just Reset to Server from outside the firewall. Please kill anything older that may be queued. If you like, you can delete the whole library and let me know, and I'll reset from outside the firewall again.

Thanks

dstillman

It's unlikely that the firewall is the problem.

I've cleared your upload, but remember that Restore to Server doesn't do anything if you have a queued upload—and it should be giving you an error to that effect. (Is it not?)

But you can try the Restore to Server now. Just make sure you've deleted all the items you don't need (including any individual notes created by the RIS importer that you don't need, using the method I described above) and emptied the trash first.

You should check how many total items—including child items—you have in your library before resetting. Check the library root (not a collection) and check the trash.

james.k.russell

I have never received any error message about a queued upload.

I tried looking for individual notes created by the RIS importer as you instructed, but could not discover any useful pattern - they were quite miscellaneous, and it would require many, many hours to delete them case by case.

In my local installation, my trash is empty. Counting all sub-items by the means you give yields 24368. Counting top level only yields 11171.

In my online library - it is the same as it has been for many days - 8889 items, no new collections, just quite old ones.

As I noted above, I did a Reset to Server from home just before commenting about it here. Please let me know whether that was aborted by you clearing the queue, i.e. whether I should do it again. I don't care either way, I just don't want to create more trouble. Again - you can blow away the whole library online if that helps. As I understand it, that is what Reset to Server should do anyway. It claims it is doing that when I issue the command, but I've seen no such effect.

dstillman

You can do the Restore to Server.

mark

Am I pushing this beyond supported limits? Perhaps I should go back to EndNote. I really prefer some functionality in Zotero, but it doesn't seem to be up to a library of this size (after culling, ~11,000 items currently).

Just to note that there is no need to go back to EndNote. I have a Zotero library of 15782 total items (7678 top-level ones) and everything works totally fine. What you're running into is some technical bottlenecks in case of a wholesale reset, which should be necessary only once.

james.k.russell

Thanks Mark - it is nice to hear there's a light at the end of the tunnel. If I didn't believe this was true I wouldn't be trying so persistently. I really like the Zotero system.

Thanks also to Dan and Adam for their patience and responsiveness.

Here's hoping it is all over soon.

james.k.russell

10 days later, online library still empty. I can live without it - syncing standalone's on two computers works for me - but it is a bit curious. Am I looking at 10 more days, 10 more weeks, 10 more months?

dstillman

Did you think you were waiting for us for something? I said to do the Restore to Server, and it looks like the clear, at least, went through, though apparently not the sync. You don't have anything queued on the server right now. Since your online library is empty you should just be able to perform a normal sync, and we can see if that works. What's the issue on your end now?