No literature added < 2014 anymore

Horschig

Hey guys,

I got a somewhat strange problem, namely that all my literature that I added before January 2014 is gone. It is somewhat annoying, but luckily not a huge problem for me as I just finished and printed my PhD thesis and now just have to finish one more article... anyway, it would have been a real pain if I wasn't in that lucky situation. Now, I just want to know if there is a way to trace back why this has happened (it might e.g. be that I unintentionally deleted or overwrote the database during an incomplete syncing), and if there is a way to get back all the stuff that got deleted. Really strangely, I am extremely sure that I had all my literature just a few months back when I last worked on that article and changed citation style for another journal *after* I had already changed the working station and synced the db succesfully...

dstillman

Zotero only syncs explicit deletions — that is, deletions you did manually — so there's no way to overwrite a database during syncing. (The only exception would be if you used the reset options in the Sync->Reset pane of the Zotero preferences.) There's also no such thing as an incomplete sync in Zotero 4.0 or earlier.

It looks like you've deleted ~350 items in your library: a large bunch in May 2014, another large bunch in January 2015, and then a handful in February and April. I can't tell you what those were, but if you still have a document created from some of those items, you can right-click on a citation, choose Toggle Field Codes, and provide the zotero.org URL that you see in the code, and I can tell you if a given item is among the deleted ones.

dstillman

But unless you were somehow syncing with a different account, the only way to get back the items would be if you have a backup copy of the zotero.sqlite file from your Zotero data directory from the time when you still had the items.

Horschig

That is indeed strange, because I am sure that I *never* deleted any items from my library other than snapshots, apart from the double entries I deleted beginning of this year... anyway, nothing I can do. First of all, where did you see that I deleted 350 files? I am not sure it helps much if you can tell me if a cited item was deleted, because if it's not in my db, it must have been deleted. Second, assuming I still got this sqlite file, can I merge that one and the new one? (I found a small one from October 2014, but I am afraid that is from before I was syncing)

Horschig

btw, could be that I using Zotero just from 2014, so no wonder there are no entries before that. Anyway, fact is that I am missing items, and now I am looking for a way to get them back.

dstillman

First of all, where did you see that I deleted 350 files?

From the delete log on the server.

I am not sure it helps much if you can tell me if a cited item was deleted, because if it's not in my db, it must have been deleted.

The point would be to confirm that those items were indeed in your current account and previously synced to the server.

To be clear, though, there's simply no way that Zotero somehow lost your items on its own — it's just not possible, given the way that Zotero works.

Horschig

I understand. Well, maybe I'm getting old and forgetful. But if it's technically not possible otherwise, then I must have deleted them manually unintentionally... so, some URLs for you to check:
http://zotero.org/users/1874745/items/TE47IJXI
http://zotero.org/users/1874745/items/65RCUSE4
http://zotero.org/users/1874745/items/XBE2QXXR
http://zotero.org/users/1874745/items/XR4SF6Z4


Anyway, nothing we can do, so back to merging. How can I do that? Btw, I am 100% sure that the files were in the db of my account.

dstillman

One of those — XBE2QXXR — still exists in your library. There's no sign of the others, deleted or otherwise, which means they were never synced to the server. It would be possible for items from a period of time later than other items to disappear if you were temporarily not syncing (say, because auto-sync was off, or because there was an upload error that you didn't notice for a while) and then switched to another computer or otherwise ended up with an empty database that pulled data down from the server. It looks like there were full syncs for your account on two days, one in July 2014 and another in November 2014, for what that's worth.

I can't think of any way this could have happened for older items, though, since the first time you synced anything it would have synced everything in your library. And there's essentially no way that you could have synced these items to another computer given that there's no record of them online.

In any case, you should temporarily disable auto-sync in the Sync pane of the preferences and make a full backup of your Zotero data directory, and then you can try closing Zotero (or Firefox), swapping in any backup copies of zotero.sqlite you can find, and restarting Zotero/Firefox and seeing if they're there. If they are, you can actually just sync and it will pull down all newer items from the server, but let's pause before that to assess the situation and see if we can figure out what happened (again, assuming you actually find these items).

Horschig

Hi Dan,

thanks all lot for your awesome support. The only reason I could think of why not everything was synced is that maybe there was not enough storage to sync everything? Could that explain why some of my earlier items were synced while others were not? Around end of last year/beginning of this year I found out that I am out of space, so I deleted some snapshots etc. and tried to free up some space. Since I switched jobs in the meantime, and thus working stations, I do not have access to the old, local Zotero db.
Unfortunately it turns out that the 'backup' I thought I found is the initial, empty sqlite. So, I guess I have to fill my library manually again :/ Anyway, as I said, not a dramatic issue for me in my current situation, luckily. If it's so that my items were never synced because of a lack of space, maybe a warning every once in a while about a lack of cloud space would be nice. I am out of space again I just realized after logging in onto the webpage, but I wasn't notified whatsoever when just using the FF or standalone version.

dstillman

That wouldn't be it — there's no limit on how many items you can sync to Zotero. Storage space affects only file syncing, which happens separately.

(Zotero also does display a warning in the client when you hit your file storage quota. If you click on the Sync icon manually, it's a popup. Otherwise there's a warning icon next to the Sync icon. But in either case all your item data is synced in full first.)

Horschig

hm that's weird then. Is there a way to see the client IP of the November sync? I'm curious to know if it was from my old or new working place.

edit: And when in July 2014 was the sync? I got the first draft of that article I am talking about ready on 23 July, so any sync around that time should have included said items. An earlier sync might not have included the items (I thought I had autosync on all the time, but maybe I'm mistaken here).

dstillman

Send an email to support@zotero.org with a link to this thread and we can provide the IP addresses, but the November 10 full sync was from an IP address that hadn't been used before. July full sync was on July 7th.

dstillman

And if you started using Zotero on a new computer via syncing (as opposed to copying the data directory manually), that'd be the full sync.

dstillman

It looks like there were some gaps in uploading, notably between Jul. 7 and Nov. 10 (and also between Nov. 10 and Jan. 7). So if you created items shortly after 7/7, you were probably getting an upload sync error on that computer, and then when you switched to a new computer and did a full sync on Nov. 10, you lost all the items between Jul. 7 and Nov. 10, because it only pulled down what was on the server.

Horschig

Hey Dan,

hm, yea, that sounds like what could explain the missing items. I switched working station in September and was then working in parallel for some months on both machines. So the July sync was from the old working station, and the November from the new one if I recall correctly (which explains the formerly unknown IP). Probably I added some items after July 7, just when I started writing that one paper, and unfortunately they didn't get synced probably. Just out of curiosity, why could such a sync error happen? Is it something that I messed up (just fyi, I was also using Zotero on my home machine)?

dstillman

It was most likely a conflict, actually, rather than a sync error. If you modified the same item differently on two computers between syncs, then Zotero would need you to resolve the conflict, which requires clicking the sync button manually. For auto-syncs it would display an error icon that you could click on to the left of the sync button. (It does this because otherwise you'd get an alert popping up every 15 seconds while making changes.)

The next major version of Zotero should be better at resolving some conflicts itself, but it's possible that we should have it at least display an alert on conflict for the first auto-sync of a session, since you're not the first person to not notice that auto-syncs had been paused, and clearly the consequences can be somewhat dire.