Upgrade StorageBacking up Zotero and deduplication: can we ignore the *bak files with daily backups?
I do daily or almost daily backups of my laptop (to external disks and remote storage), including the Zotero directory. For backups I use Restic (https://restic.net/) which employs deduplication.
I've reread carefully https://www.zotero.org/support/zotero_data#backing_up_your_zotero_data and https://www.zotero.org/support/zotero_data#restoring_your_zotero_data_from_a_backup1 but I cannot find answers to these two questions:
1. Is it safe to run the backup while Zotero is opened? Or should I close Zotero before running the backup? In case it could matter, I am using Linux.
2. With daily backups, do I need to keep backups of zotero.sqlite.bak and zotero.sqlite.1.bak?
Why? I've been noticing that my daily backups can sometimes grow by as many as 600 MB just from the sum of the stored (after dedup and compression) zotero.sqlite + zotero.sqlite.bak + zotero.sqlite.1.bak. My zotero.sqlite file is of about 960 MB. I am not adding hundreds of megabytes worth of new data, but it is my understanding that even adding just a few entries, especially with full text indexing, can lead to the SQLite files changing a lot at the byte level, which would render the deduplication of Restic almost powerless.
I think that, if I close Zotero before launching Restic's backup (or if a backup of zotero.sqlite made while Zotero is running is always recoverable), I might just backup zotero.sqlite (and forego keeping backups of zotero.sqlite.bak and zotero.sqlite.1.bak). Otherwise, I think I should keep both zotero.sqlite and zotero.sqlite.bak so I never lose much more than 24 hours worth of Zotero state. In both cases, with daily backups zotero.sqlite.1.bak would not be a file I need to backup. But I am concerned I might be missing something.
(P.S. Heavily edited after I realized a previous idea that depended on calling sqlite3 to make a backup would not work, as I think Zotero keeps a lock on the zotero.sqlite file).
I've reread carefully https://www.zotero.org/support/zotero_data#backing_up_your_zotero_data and https://www.zotero.org/support/zotero_data#restoring_your_zotero_data_from_a_backup1 but I cannot find answers to these two questions:
1. Is it safe to run the backup while Zotero is opened? Or should I close Zotero before running the backup? In case it could matter, I am using Linux.
2. With daily backups, do I need to keep backups of zotero.sqlite.bak and zotero.sqlite.1.bak?
Why? I've been noticing that my daily backups can sometimes grow by as many as 600 MB just from the sum of the stored (after dedup and compression) zotero.sqlite + zotero.sqlite.bak + zotero.sqlite.1.bak. My zotero.sqlite file is of about 960 MB. I am not adding hundreds of megabytes worth of new data, but it is my understanding that even adding just a few entries, especially with full text indexing, can lead to the SQLite files changing a lot at the byte level, which would render the deduplication of Restic almost powerless.
I think that, if I close Zotero before launching Restic's backup (or if a backup of zotero.sqlite made while Zotero is running is always recoverable), I might just backup zotero.sqlite (and forego keeping backups of zotero.sqlite.bak and zotero.sqlite.1.bak). Otherwise, I think I should keep both zotero.sqlite and zotero.sqlite.bak so I never lose much more than 24 hours worth of Zotero state. In both cases, with daily backups zotero.sqlite.1.bak would not be a file I need to backup. But I am concerned I might be missing something.
(P.S. Heavily edited after I realized a previous idea that depended on calling sqlite3 to make a backup would not work, as I think Zotero keeps a lock on the zotero.sqlite file).
Closing Zotero before backups is awfully inconvenient for me. I run daily external backups that are launched automatically and opportunistically. I could close Zotero from the script that launches the backup, but that would rarely be convenient (and I'd want to have a notification that Zotero is closing in so many minutes, then dutifully stop Zotero work, etc ---a hassle). So I am now backing up zotero.sqlite and zotero.sqlite.bak (but not zotero.sqlite.1.bak). I think I pay in storage what I gain in convenience.
If I have to recover from my Restic backups, it is possible I'd be able to use zotero.sqlite, unless I had the misfortune of that Restic snapshot being taken during a transaction. If that fails, I'll have the zotero.sqlite.bak, which is made by Zotero itself, at a moment it considers safe. And, from https://www.zotero.org/support/zotero_data#data_directory_contents "(...) zotero.sqlite.bak (an automatic backup of zotero.sqlite, which is updated periodically if the existing zotero.sqlite.bak file hasn't been updated in the last 12 hours) ". So the the extent of the disaster shouldn't go back more than about 36 hours in the worst case scenario.