database missing files, but folders still exist

pvanimpe

Dear all,

I noticed a discrepancy between the number of papers in Zotero (about 1000) and the actual number of folders in my library directory (about 2000). Going through some of them manually, I noticed that indeed several pdfs in these folders do not appear in Zotero. The folders contain the pdf and its correct info and cache file but they are not in the database. Most likely this happened when two library directories were merged. (I'm still trying to find out how and when that happened) Is there a possibility to rebuild the database ? Manually finding all missing papers and importing them again seems very painfull (considering there about a 1000 of them).

best regards,Peter

adamsmith

There are (almost) always more folders than items, as each attachment gets its own folder - e.g. an item with a pdf-file and a web-site snapshot attached, like many items downloaded from databases - will have two folders. So this problem is likely a lot smaller than you think.

If there is no corresponding entry in the database, any item in the Zotero folder is just like any random file on the computer. You cannot use those to rebuild any part of the database in any way.

I'm not sure how you "merged" library directories - but that seems like a very likely cause - if you still have both libraries there could be a way to get your data back - be very specific in what you did so we can get a sense.

pvanimpe

Hi Adam,

I suspect something went wrong with my local sync. I work at different offices and I have a server backup at each location. I think my sync at one location combined the (wrong) storage folders, leading to an accumulation of files. I'm afraid at one location somebody messed up the storage folder, it's a shared disk.
I'm a big fan of zotero, but the data structure makes it very sensitive to data loss. Losing the references in the database isn't that much of a problem. Re-importing all the pdf files however will be a huge pain.

Thanks anyway. Have a nice weekend. P.

adamsmith

I don't understand how any of this would affect the database, though? In order for items to be in the storage folder and not the database, one of two things would have had to happen:

1. The database (i.e. zotero.sqlite) was corrupted or replaced by an incomplete version
2. Items were added to zotero/storage that were never in the database

It sounds like #2 happened, though your description is very vague. But that means the items were never in that particular database. This actually isn't a type of data loss Zotero is very sensitive to. It practically never happens and I'm still not sure if you actually lost data or just duplicated data. (Though using non-Zotero syncing is the easiest way to corrupt/lose your data).

Gregor02

Dear all,

maybe I have a similar problem and I can give a closer description of waht happens. I'm using Zotero within Firefox both on my desktop and on my notebook. Like all other data I synchronyze the zotero folder between both of them using Microsoft SyncToy 2.1.

Only today I realised the following effect: I'm working with the current version of my data on the desktop and I add references into Zotero. Now it happens that someone uses my notebook to quickly look up an information on the internet, starting a Firefox session on the notebook, of course without adding oder altering anything concerning Zotero. Nevertheless, when I do a synchronization afterwards, the new entries I made on my desktop disappear because the "zotero.sqlite" from the desktop obviously is overwritten by that one from the notebook - although I didn't make any alterations on my notebook.

Unfortunately I've already lost several hundred entries before I realized this effect. Is There really no possibility to "scan" oder "reload" the entries from the folders in my Zotero-directory?

Thanks in advance for any advice.

adamsmith

Like all other data I synchronyze the zotero folder between both of them using Microsoft SyncToy 2.1.

I'm sorry. We do everything we can to warn people against doing this for exactly the reasons you are now experiencing. You're actually lucky that this hasn't damaged your database beyond repair. That's why Zotero sync exists.

If you don't have an external back-up of your database there is nothing you can do.

Gregor02

ad: adamsmith

Thank you for your response to my inquiry. - In a case like mine, which seems to be a not too rare one, it would be helpful to have an assisting function in zotero. At the moment there is the "Show file"-command, which shows for an existing entry in which folder the related files can be found. On this basis there could be a function "Show all 'orphaned' folders" - i.e. those sub-folders in the zotero-folder no actual entry is related to. This would assist a manual rebuild.

Gregor02

After I had felt a bit desperate for some time I decided to choose a hand-knitted solution. So for all those who have a similar problem, here is at least a way to solve it - even if it is a bit uncomfortable. Unfortunately it only works if for those lost entries which contained a "snapshot" of a website (which is the case in my case).

The solution is based on the fact that Zotero stores information on each of such snapshots in an own subfolder within the "storage"-folder within the "zotero"-folder on your hard disk.

I describe the way I solved my problem. But of course I can't make any guarantees that it works in any other cases or circumstances. If you try it's on your own risk.

(1) Close your internet browser within which you're using zotero.

(2) Use Windows-Explorer (or another file manager) to make of copy of your whole Zotero-folder including all subfolders (e.g. copy, not move, the whole "zotero"-folder to a different folder on your harddisk). I will refer to this copy as "backup copy".
If any goes wrong, you always can fall back on this copy.

(2) Once you have made this backup copy, start your browser+zotero and use zotero to delete (!) all entries (references, collections etc.). After havong done so, close your browser+zotero again.

(3) Now go back to the Windows-Explorer: Go to your actual zotero-folder again (not to the backup copy) and change to the "storage" subfolder. All those subfolders which are still within the "storage"-folder contain the "orphaned" references which had disappeared from your Zotero database.

(4) Optional step: Print the content of the "storage"-folder, so that you have a list of the names of all subfolders within it (how to do it: http://support.microsoft.com/kb/321379/en-us)

(5) Now rename your actual zotero folder (e.g. to "zotero_lost_entries") and MOVE it to a different folder in your harddisk.

(6) COPY your original zotero folder (i.e. the backup copy) back to its original place (leaving the backup copy untouched).

(7) Optional Step: Now you can use the printed list produced in step 4 to manually delete all orphaned folders from your actual "zotero/storage"-folder. Alternatively you can simply leave them in place as orphaned folders and jump over this step.

(8) Now the crucial step: Go to the "zotero_lost_entries" folder created in step 5. There you go to the "storage" subfolder and manually open the html-file in each of its subfolders. If you do so, the snapshop stored in zotero appears in your browser. Unfortunately without the original URL. So from each opened html file you have to reconstruct the original address the snapshot was derived from, using some detective skills. After you've found the original source, you simply can use it to add a new reference to your zotero-database.


It took me a whole weekend to do so for several hundred lost entries. But in the end I managed to restore my lost information.

If anythink goes wrong, you alway can use the backup copy (created in step 2) to copy it back to its original place. Nevertheless if you make an attempt it's on your risk.

If there are any better or easier solutions I would be glad to know.