Automated cloud backup and the zotero directory/database

zoyoni

Hi all (hope all you east coasters are safe in this hurricane!)

I have a question that may not be resolvable, but maybe we can think about it a bit.

I use Spideroak, a cloud backup system with a very high security priority, and the client sits in the backround of my system (win7) automatically scanning the HD for file changes in specified directories.

This is all fine and good for 99% of applications, but the problem begins when I fire up Firefox and Zotero's sqlite files (and some other translators) update themselves ALL THE TIME. (incidentally I don't know why, because I am not adding/changing anything - is this because of Zotero's auto-sync?)

This has the effect of completely clogging up the backup queue in the client and, at times, needing for me to clear its queue through some complicated manuevers. Now, I realize this isn't a Zotero problem, per se, but there are really only two options here:

1. Figure out with SO how to change their automated backup algorithm to only backup files with changes outside of a certain timestamp range. This seems to me to be intractable because when it comes to other things - for example, saving Word docs - I want SO to recognize instantly the changes.

2. Exclude the Zotero directory from automated backup and do it manually. Also, this seems undesirable because of the philosophy of automated backup is all about the failsafe factor; you can't "forget" to backup.

3. Work with Zotero to figure out how to go around this - maybe store changed only when the db is actually changed... Or have the working files outside the main Zotero directory, and so when I backup the main dir I am not backing up all the time...

4. ???

Thanks for your help all.

adamsmith

I don't know why you'd see constant updating - certainly not for anything other than the sqlite: Translators should only update once every 24hs, e.g.
In the case of the sqlite, yes, I do think it logs last syncs in the DB, so that would explain that (but not sure).

The ideal solution would be to exclude zotero.sqlite from permanent updating via spideroak and see if you can find a way to have it back-up periodically (say every 1h) rather than automatically on every change.

In case it's helpful: The only things in that folder that need to be backed-up are the zotero.sqlite, ideally various sqlite.bak files, and the /storage folder with all its content. Everything else will be restored with a standard Zotero re-install.

dstillman

I do think it logs last syncs in the DB, so that would explain that (but not sure).

It does. But we can possibly change it to update the various last-sync times (for syncing, translator updates, etc.) only when it changes something, since, unless I'm overlooking something, reusing the previous last-check time should be equivalent to using a later timestamp at which nothing has changed. I'm not sure we actually have a good reason for always updating the last-sync times.

he only things in that folder that need to be backed-up are the zotero.sqlite, ideally various sqlite.bak files

If your backups are working, you can probably exclude zotero.sqlite.bak, or at least zotero.sqlite.1.bak. (Alternatively, change extensions.zotero.backup.numBackups to 1 in about:config and it won't create a zotero.sqlite.1.bak.)

zoyoni

Hi guys, thanks for your help. First, I unchecked all the translators and changed the option to just one backup.

I don't think I can change the software to backup just a file in a separate profile than everything else. A limitation of SO, perhaps, but something I need to live with for now.

But looking at Dan's comments, maybe this is something that can be changed in Zotero?

zoyoni

Dan, adam,
It's still happening despite my modifications - here for example is a screencap of the SO software from last night when the computer was on. Firefox was running in background but there was definitely nothing else going on...

http://i.imgur.com/ywKFN.png

dstillman

As I said, currently Zotero updates the database whenever it auto-syncs. In addition to after changes you make, that happens every hour the computer (or Firefox/Standalone, maybe—I forget) is idle.

If you don't want this to happen, you can disable auto-sync in the Sync pane of the Zotero prefs, but we don't recommend that.

Faolan pointed out that we do need to record the last-sync time even when there are no changes in order to display it in the Sync button tooltip, but we could possibly record the display time separately and leave the database unchanged—the worst thing that would happen would be that, if someone transferred their database but not the separate file containing the display time, the tooltip would be incorrect.

zoyoni

Hmm, I see. Well, can I push for that [tooltip inconsistency], even if I am an outlier case?

estubbinsbates

I have the same problem as zoyoni, and also use Spideroak and Zotero, though in my case I use Zotero Standalone. I have also disabled the auto-sync function: I sync Zotero manually when new references are added.

However, in my experience, the entire Zotero folder backs up to Spideroak every time the program is opened (regardless of whether I sync it or make any changes). This can be problematic when I am drafting and editing, as my documents are stuck in Spideroak backup queues, behind the unchanged Zotero data.

Do you have any suggestions?

dstillman

What do you mean by "the entire Zotero folder"? Zotero modifies the database periodically as described above, but it doesn't regularly change other files.