File Storage Question

jsderouchie · October 27, 2019

I recently broke away from a group, copying the entire library of 45000+ entries over to my own library. Only in the final months of my participation in the group did we seem to go beyond the "free" version of Zotero. Yesterday, I was syncing my library, and it kept saying I didn't have enough storage capacity. I ended up having to go to "unlimited" storage space. Now this morning my 1 TB hard drive on my computer says "Your disk is almost full," and when optimizing storage it says I have four Zotero copies with different addresses for every document. I don't want this. What do I need to do? What did I do wrong? ––Jason

dstillman · October 27, 2019

Only in the final months of my participation in the group did we seem to go beyond the "free" version of Zotero.

That's pretty unlikely for a library of that size, unless you simply weren't syncing files, or added way more items at the end. A free account only has 300 MB.

when optimizing storage it says I have four Zotero copies with different addresses for every document

Could you give an example? Are you talking about folders within the 'storage' directory of your Zotero data directory, or do you have multiple data directories on your system? Did you leave the group yet? Note that items in separate libraries would have separate attachment files. It's also possible you ended up with some orphaned files in your 'storage' directory through some failed imports, but we'd need examples (with the 8-character folder names) to tell you more.

jsderouchie · November 13, 2019

Thanks for the reply. It has taken me some days to get our IT dept's help, but today we were able to tackle this. I just used "Duplicate File Finder Pro" (cost $10.00), and it notified me that I have 65012 duplicate files totaling 382.3 GB; most if not all are deriving out of Zotero. For example, in folder IJJTQTN3 there are 56 copies of a 35.2 MB file, totaling 1.97 GB in the folder! The file itself is a JBL1263––a whole journal volume. I have hundreds of examples of duplicate files just like this, and it is taking up massive space on my hard drive. Yet only one of the files is visible in my Zotero program, and only one of the 56 file addresses is linked to by the attached doc. As such, I can't just use Duplicate File Finder Pro to delete all but one of the folders, for the one it keeps may not be the one linked to in my Zotero platform.

Two notes: (1) I used to be partnered with another Zotero Power User (Group Ezra710) and only a month ago copied all files over to my personal library and separated from the group. (2) I believe all of the reduplication happened when I recently upgraded Zotero to the highest package ($120/year), for right after the purchase and the syncing, I was notified that my 1 TB hard drive was almost full. I couldn't believe it.

Do you have any suggestions for me? If you want to pass on an email address, I could send you screen shots of Duplicate File Finder Pro's results.

dstillman · November 14, 2019

Oh, so that's something totally different — and duplication of files within a single storage subfolder is mostly outside the bounds of something Zotero could even do itself.

When you say there are 56 copies of a file, are you talking about a PDF or something else? What are the files named? There should never be multiple files within a 'storage' subfolder for PDFs.

Are you using any Zotero plugins?

(You can either upload screenshots somewhere (e.g., Dropbox) and link to them from here or, if you'd rather not share them publicly, email them to support@zotero.org with a link to this thread.)

dstillman · November 15, 2019

For example, in folder IJJTQTN3 there are 56 copies of a 35.2 MB file, totaling 1.97 GB in the folder!

OK, so just to clarify, this isn't what your screenshots show. They show the same file appearing in 56 different folders, not the same file appearing 56 times in one folder. While still odd, this is much more explicable with normal Zotero operations, particularly given what you describe about another group member creating 9 copies of the same file.

Now, as for what happened, that's harder to say. If you look closely at the results for that one file (and I would assume others), you'll see that they're grouped into sets of 9 (or sometimes 10 or 11) for each timestamp. If you can recall what you did at those times, that would tell us what caused this. Generally speaking, orphaned files happen when some operation in Zotero that's creating new items doesn't complete successfully, so the database changes roll back and the files are left behind. (That shouldn't happen, but it might due to some bug, or if Zotero is force-quit during an operation.) The most likely scenario is that you tried to copy the library items to another library and the operation failed or stalled partway through. Perhaps you remember something like that?

At least in this case, the files corresponding with entries in your library all have the earliest timestamp, so you can delete the unused ones simply by selecting all the ones with other timestamps. I'd guess the same is true for others.

A feature that looks for orphaned files in the storage directory is a pretty common request, and I suspect we'll create one in the not-too-distant future.