Previosly synced files deleted

y.sapolovych

I have quite a large corpus (30k items+snapshots) in one group, and I need to transfer it to another group. Copying those records seemingly goes fine at first - I get a few thousand items in the new group, but then reopen it again and see that it is completely blank - all items just misteriously disappeared (they're not in trash or any other group/collection).

And this happened three times in a row. The last time it actually got 15k articles copied, but they all disappeared. Some specifications:
1.Zotero storage is unlimited
2.There is enough space on hard drive to sync new files

I can't record debug output because items 'just disapper', i.e. I can't predict when exactly it's happening.

I know that the number of items is indeed very big, but I really need to handle this smh, so any help would be greatly appreciated.

sdspieg

Update on this - we have now tried transferring these files from one (sub-sub)-group library collection to a top-level group library) 5(!) times on three different computers. Always with the same outcome: it happily starts working, we see the items appear one by one in the target folder, but then at some point the process stops and then mysteriously all items that were previously there are gone.The furthest we got was that it copied over 15'500 of them. But then poof!. Contrary to what Yevhen suggested - none of these ever synced. What he meant was that they were visibly fully transferred at that moment, but that's all. Let me also specify that from our second attempt onwards, we did all of this offline, with all plugins disabled. But that didn't help either.

We then tried to export the whole collection to an rdf-file and re-importing it. That went extremely slowly also, but after a few hours got stuck after 7149 items. So that doesn't work either.

 So a bit more info on this corpus:

• we're talking about 32,892 items;


• almost all of them are in Russian;


• what we are trying to do is textmine all of these files to detect how topics, sentiments, entities, causal hypotheses, n-gram co-occurrences etc. change over time in this corpus.


• almost all of them have attachments - typically (html-based) snapshots, also a few pdfs


• the rdf export (including files and notes) folder has 1,193,329 files in 97,283 directories representing 23.876GB. So yes - its big [btw - when we look in those storage folders, we see that MOST of them are things like js and irrelevant for our purposes - we really only need the doc.html].

Our questions:

• could the sheer number of files explain why this doesn't work?


• is there a size/time limit on what can be transferred? (if we knew, we could do it in batches)


• are there hidden preferences that we can change that would improve our chances to transfering this successfully?


• is there a way to exclude things that are being transferred from the command line or so? E.g. transfer all from here to there but only include html-files


• any other ideas?

We really would be grateful for any answers or ideas... And we'd also be happy to give any of you who would want to take a look themselves access to this collection - just ask and we will grant access. Thanks

dstillman

Copying in batches should work, but try again in the latest Zotero beta.

(The failed attempts probably did create some orphaned folders in the 'storage' directory, which we should fix. There are some third-party scripts that can clean those up, but that's probably something we'll do automatically in a future version of Zotero.)

sdspieg

I tried in the latest beta. It added a few items, which then disappeared again. As we're waiting for the built-in ability to clean up orphaned folders, could somebody please point us to these third-party scripts? Thanks!

dstillman

It added a few items, which then disappeared again.

Can you provide a Debug ID for this?

sdspieg

Dan - I'm running it again now (with debugging on). It's still running smoothly. But so I'll submit a debug ID if (/when?) it happens again.

sdspieg

Debug ID D910331489. And I also got a sync error (1806555414) - but am not sure whether the two are related