Zotfile: should bulk moving of stored files not reduce the size of the storage folder ?

fschuman

Im trying to convert all my attachments to a linked attachments folder synced via OneDrive. My library has about 9500 parent items with attachments, mostly PDFs. The Zotero/storage folder contains 8.7GB of data in about 70.000 files.

To my understanding, Zotfile can bulk his process by selecting all items and 'rename and move' them. If I do this, Zotero runs at 100%+ for a long time, suggesting that it's doing something.

However, should this not (very much) remove the size and number of items in the storage folder ?

Zotfile does appear to work correctly if I add new items with attachments, they are moved to the right folder. But it doesn't appear to work for such bulk moving.

tim820

Whenever I have done bulk moves with Zotfile, I have done it in smaller chunks. You have a lot of items and attachments. So for example sort your library alphabetically by title or creator, and then select and do Rename and Move first on all the A's, then all the B's, etc. If you do right click Show File before and after the Move you can verify that individual PDFs have indeed been moved from the zotero\storage folder hierarchy to your Zotfile Custom Location folder (so the total file size under the zotero\storage folder hierarchy should then be less).

fschuman

I've tried moving them in chunks as you suggested. And then sampled the file location for a larger number of randomly selected items, which are all placed correctly in the linked folder. However, the storage folder still has ~70.000 files with ~8.7 GB. (The Zotero database is ~ 1GB).

Could this be the attachments in shared collections ?
It appears those can only be renamed, but not moved.

tim820

You can get a list of ALL PDFs (still) in local zotero\storage using your OS's file search for *.PDF under that folder hierarchy. You can also use the code here to get a list of PDFs in local zotero\storage from My Library (ie local storage PDFs that Zotero actually has database entries for):
https://forums.zotero.org/discussion/comment/417599/#Comment_417599

Relative to your numbers, I have ~5000 items in My Library. Under my zotero\storage folder hierarchy there is ~3.5 GB (very few PDFs left there), with ~11,500 files and ~5,000 folders.

My Zotfile Custom Location folder of linked PDFs is ~10 GB.

So maybe your 8.7 GB zotero\storage folder hierarchy's size is just due to (non-PDF) storage required for your 9500 items (about twice the number that I have in my ~3.5 GB) ?? My zotero.sqlite is ~340 MB. Your 70,000 actual files under zotero\storage does still sound somewhat high though.

But the above PDF search under zotero\storage should tell you if Zotfile has done its job in moving all PDFs you asked it to. BTW empty your bin/trash first, to get accurate counts of current files only.

adamsmith

I'm pretty sure there have been reports of ZotFile not removing the linked files from the storage folder under some conditions. If that's what's happening (and it'd certainly explain the lack of drop in storage folder size) it shouldn't be too hard to ascertain.

tim820

FWIW I don't recall Zotfile ever failing to move a PDF for me. It's worked fine for ~3500 PDFs.

OTOH are there OTHER scenarios where local PDFs could remain under zotero\storage while no longer in the Zotero database (so they would not appear in the Zotero library window, so would never be selectable for Zotfile to move) ? There are some threads referring to such Zotero 'orphaned' PDFs (different to Zotfile orphaned PDFs in the LINKED folder location, which arise when deleting items with linked attachments, unless the Delitem add-on is used).

fschuman

Thanks ! Looking up the titles of some of the PDFs that are in the stored folder in Zotero, they are not orphaned but do belong to shared collections. Those apparently have grown much bigger than I had noticed. I was confused as a lot of these items are also in my personal collection, so those PDFs are saved both in the linked and in the stored attachment folder.