Zotero starts uploading all (5GB) attachment files after a restore of zotero.sqlite from server

oldtimer

Hi,

Zotero was very slow in starting up on my T61 laptop (Windows 7), with file-synching
that sometimes had to be restarted several times in order to complete.
I started from a state where the database and attachments were in sync
across several different machines.

I found that zotero.sqlite on T61 machine was 168MB large, whereas
on a different Windows 7 laptop it is 13MB and on two Linux machines,
16 and 22MB. How can the sizes of synchronized data bases be so different ?

Since 168MB was clearly too much, I started "Restore from File Server" for the
database, in Zotero settings/Sync/Reset, resulting in a synched zotero.sqlite file
of only 10.5 MB (!)

Hitting the sync-button in Zotero then started file-synching of attachments.
Problem: Zotero is apparently uploading (almost?) the complete set of attachments,
more than 5GB, even though they were already synchronized beforehand.
(I did not click on "Reset File Sync History".)
The actual data upload is about half of the progress in KB to be uploaded.
At present, the status is roughly
Uploads: 5,200,000 KB remaining (1100/4900 files).
The Zotero storage directory has a size of 5.4GB (6.2GB on disk), 285,000 files, 5500 folders.

What can I do to synchronize without 5GB of uploads ?

Thanks !

mark

Sometimes it looks like it's uploading files but in fact it is just comparning local and remote versions to decide whether or not to upload. Could that be the case?

oldtimer

Unfortunately, not. Network activity showed that it did upload about 100MB for a claimed progress of about 200MB. (Is the upload compressed ?)
(By the way, doesn't the Zotero sync use something like the rsync protocol ? Then there should be little network traffic to find out that 5GB of data are already identical.)

oldtimer

Update: One of several restarts of Zotero made the remaining byte count
jump by several GB (the other restarts did not do that).
It is now at 1.3GB / 2000 files.

dstillman

I found that zotero.sqlite on T61 machine was 168MB large, whereas
on a different Windows 7 laptop it is 13MB and on two Linux machines,
16 and 22MB. How can the sizes of synchronized data bases be so different ?

You had PDF indexing enabled on one but not the others.

Problem: Zotero is apparently uploading (almost?) the complete set of attachments,
more than 5GB, even though they were already synchronized beforehand.
(I did not click on "Reset File Sync History".)

Zotero can't upload any files that already exist on the server (and that have the same filename)—the server doesn't let it. You simply had some files that weren't uploaded for some reason. You saw the jump in progress not after a restart but simply because Zotero finished uploading files that weren't on the server.

And the Reset options have the same effect as Reset File Sync History.

oldtimer

"You had PDF indexing enabled on one but not the others."
Thanks, that explains it. (How can I quote with a box ?)

"And the Reset options have the same effect as Reset File Sync History."
Ah ! Not obvious. The Reset page in Zotero preferences clearly distinguishes a reset of the database from a reset of the files, just like the Sync Settings page.

By the way, the description of "Reset File Sync History", namely "Force checking of the storage server for all local attachment files" leaves unclear which side is reset.
I expected that the files on the local machine would be set invalid and data from the server be treated as valid. Apparently it is the other way round ? Maybe the description could be modified ?

"Zotero can't upload any files that already exist on the server (and that have the same filename)—the server doesn't let it."
But if a local file should be modified, it will be updated on the server, right ?

"You simply had some files that weren't uploaded for some reason. You saw the jump in progress not after a restart but simply because Zotero finished uploading files that weren't on the server."
That does not seem to figure. The local data base was in sync with the server before the reset. It had been successfully synchronized many times, including file sync, over the last several weeks, with little change to the data base and files during that period.
Thus: Is it to be expected that the synchronization of two identical sets of files, 5GB total (but many files, 280,000), takes beyond 300MB of upload traffic ? Could this have been the synchronization information ? That would also explain the giant jumps (several GB!) in supposed data to be uploaded.

Thanks !

dstillman

(How can I quote with a box ?)

<blockquote> in HTML mode

Ah ! Not obvious. The Reset page in Zotero preferences clearly distinguishes a reset of the database from a reset of the files, just like the Sync Settings page.

It's a side effect. The modification times and hashes of the local files are stored in the database (so that it can tell when they're changed), and the database is wiped out on Restore from Zotero Server. It doesn't really matter, though, since Reset File Sync History should generally have no effect in a properly synced database.

But if a local file should be modified, it will be updated on the server, right ?

Yes.

That does not seem to figure. The local data base was in sync with the server before the reset. It had been successfully synchronized many times, including file sync, over the last several weeks, with little change to the data base and files during that period.

I'm not saying that the files in question shouldn't have been in sync—just that, for some reason, they might not have been, in which case they would be sent up with Reset File Sync History.

Thus: Is it to be expected that the synchronization of two identical sets of files, 5GB total (but many files, 280,000), takes beyond 300MB of upload traffic ? Could this have been the synchronization information ?

No. That should only takes a handful of bytes per file. (And it doesn't sync 280,000 files, since it compressed multi-file attachments into a single file before sending.)

In any case, I wouldn't worry about it.

oldtimer

Ok, thanks for the information, on a Saturday :)
The files have finished synchronizing, so it's settled for now.