Upgrade StorageCollections went missing again!
I'm at the end of my rope with Zotero. For well over a year I have been experiencing missing subcollections. How do I describe the problem? Simply, subcollections just disappeared. I've reported this to Zotero support several times. At first, support indicated that I may have deleted the collection. I can report that is not the case. Not even accidentally. The other suggestion is that I may have accidentally moved a collection into another collection (since this appeared to be the case on at least one previous occasion). However, in the past 3-months subcollections have just disappeared and they could not be found in any other collection. Yes, the citations could be found in the unfiled folder, but that is no solution for me. As the sub-collections that disappeared usually contain anywhere from 30-60 sub-collections based on topics etc... all deep level filing. Recreating that is not an option.
It happens so often then restoring from a backup is just not an option. Mostly, because there is no way to know when a sub-collection has gone missing. Today I opened up the client and went to a sub-sub-collection containing citations I made in Oct 2017. The collection was gone. There is no way I deleted it. No way I moved it. And a restore back to that period is obviously not an option.
Today I stopped all my work and lost 6-hours trying to rebuild sub-collections, deleting hundreds (maybe thousands) of duplicates that have infested my system each time I've had to import backups etc... to rebuild after another sub-collection disappearance.
I love the platform. However, it's job is to be a secured storage location for extremely important research citations. Those citations need to be there a day, week, month or years later. There is no way one should trust Zotero anymore to not "disappear" collections.
I simply can not trust Zotero anymore. What to do?
Sigh!
B...
It happens so often then restoring from a backup is just not an option. Mostly, because there is no way to know when a sub-collection has gone missing. Today I opened up the client and went to a sub-sub-collection containing citations I made in Oct 2017. The collection was gone. There is no way I deleted it. No way I moved it. And a restore back to that period is obviously not an option.
Today I stopped all my work and lost 6-hours trying to rebuild sub-collections, deleting hundreds (maybe thousands) of duplicates that have infested my system each time I've had to import backups etc... to rebuild after another sub-collection disappearance.
I love the platform. However, it's job is to be a secured storage location for extremely important research citations. Those citations need to be there a day, week, month or years later. There is no way one should trust Zotero anymore to not "disappear" collections.
I simply can not trust Zotero anymore. What to do?
Sigh!
B...
Regarding one of your previous posts, it seems you weren't able to figure out how to expand all collections in order to search by name. That definitely works — in the client, not the website — so I'd encourage you to try that again. Click a collection and press "+". You can then search for collections by name to see if one somehow ended up below another collection. (If you share the name here or in an email to support@zotero.org with a link to this thread, we can also take a look.)
In your account logs, I see a bunch of collection deletions from December 27, when you last posted, and then two more on two separate dates this month (not counting one a couple hours ago). Do you recall making those deletions? Of the 76 collection deletions in your online library since October 2017, there were 10 collections deleted over the course of an hour in March and 16 collections deleted in a single sync when you last posted in December. All other deletions were of 1–5 collections at a time. So either you're misremembering the numbers or those 30–60 collections are still somewhere in your library (or they were created and lost without ever making it to the online library, which is unlikely).
In any case, for now, the best thing you can do to avoid losing data is simply to keep longer backups of your Zotero data directory, or at least the database. You can do that with an external backup system (e.g., Time Machine), by manually making backups of zotero.sqlite, or by going to the Config Editor in the Advanced preferences and changing extensions.zotero.backup.numBackups to a higher number. (If you do the last one, make sure you have enough disk space for numBackups times the size of your database. I'd recommend just using Time Machine and making sure your backup disk is big enough that you have a long history of weekly backups.)
I'm not sure what you mean re: duplicates, but you won't get duplicates restoring from proper backups. If you're restoring from exported files (RIS, BibTeX, RDF), you'll end up with duplicates, but that's not a recommended approach. See our instructions for restoring a deleted collection from a backup.
1. yes, I am now absolutely certain there has been no accidental deletion or accidental moving of one collection into another. Yes, thanks to your help over the past year I have worked through that and I'm afraid this bug cannot be attributed to user error (I'd prefer that it was my fault as I could then do something about it.
2. The duplicates I painfully erased yesterday were indeed from an import not a backup up restore. As I have noted in my first reporting of this problem, I was able to catch an unsynched version of the missing collections during the first couple of instances and either through exporting and importing or another method was renaming the missing collection to "-new" then resynch I was able to save the folder structure. Yes this created duplicates.
3. Since day 1 with the bug, I have never been able to rely on a bag up using zotero.sqlite. The most recent backup made by the system was never truly up-to-date and therefore the choice was to try the export method or lose all the new citations from whatever day that backup was
4. Collections: I rarely delete collections on my own. The 30-60 collections I was referring too would have been around August 20, 2018. This was my Production collection. Under it are approximately 100 collections and sub-collections. The top-level Production had just disappeared as I reported.
5. About a week later my Elections top-level collection disappeared (Aug 28th I reported it)
6. Between then and this past December it happened with a small number of collections that I managed to save.
7. This December it became to much of a problem and I reported it again
8. January 2019, yesterday I lost my Writing top-level collection. And even though I managed to save it, there was a critical sub-collection in it that was just gone. And I could not figure out when it disappeared. This was a subcollection of notes I made and quotes taken from books I have been reading.
9. I have a complicated mix of drives connected to all my computers as I boot externally from SSD not internally. I will get a carbon copy cloner set up on the one boxs where I can get a weekly backup done. On my list for Q1.
10. Could you please send me the list of the collections you show deleted and the dates.... this would be huge. Since: (1) I rarely delete collections and if you see deletions over 1-5 in a day then those events are MOST Certainly the bug. I could easily look at the list and match that with my daily work calendar and I can report back if they were my deletions or part of the bug. Please send to my email: brent.stafford@shakyegg.com
Let's see if we can shed some more light on what's going on.
What you're seeing isn't happening to other Zotero users (of which there are >>100,000) and as Dan says it's highly unlikely to happen under normal usage given the code base, so if it isn't something that you actively do as a user, your specific set-up is the next most likely cause.
Complicated is mostly for me. As I run regular carbon copy cloners on my external data drives not my system drives. If I ever have a problem with system drives I would never restore from back up. I always do the reformat and fresh install. Why backup a crappy or corrupted system when you can install fresh.
So let's see, here are specifics.
HOME NETWORK
#1 - MacBook Pro Laptop - System Drive (INTERNAL SSD) Zotero installed on System and Zotero library is stored on System
#2 - iMac 2012 - System Drive (EXTERNAL SSD) Zotero installed on System and Zotero library is stored on System
Computer #1 and #2 are shared on the same network. However, I rarely if ever mount the system drive from one computer onto the other. I don't think I ever have as there would be no reason. I just call up the external drives mounted on the computers in order to transfer files as need. Which is not often as there are NO local files on either of these computers. All are synced via DropBox. Rarely would #1 connect to #2 or the other way around. Note DropBox on #1 is located on an external and Dropbox is located on an internal for #2.
However, #1 and #2 do see each other in the shared computers on my finder bars.
OFFICE NETWORK
#3 - iMac 2015 - System Drive (INTERNAL SSD) Zotero installed on System and Zotero library is on stored on (EXTERNAL 4TB HDD). This is my main video edit box and the internal System Drive is not large enough to store applications and Zotero library. EXTERNAL is Western Digital USB3 (fast enough to edit 2K video) and the computer is set to never put hard drives to sleep.
#4 - iMac 2013 - System Drive (INTERNAL SSD) Zotero installed on System and Zotero library is stored on System
Computer #3 and #4 are shared on the same network AND the above notes on usage for #1 and #2 apply.
Is there anything here that stands out as being the issue?
Thanks for the help!!
Do you recall the names of any subcollections that you can't find? There's a "PRODUCTION" collection in your library, created in December 2017. It doesn't have any subcollections and has one item. Perhaps that's a different collection? As I noted previously, there's still an "ELECTIONS" collection in your library, created in February 2017 and modified in August 2018. It doesn't have any subcollections or items. There's a "WRITING" collection in your library, created in February 2017 and modified yesterday. It doesn't have any subcollections or items.
If you deleted duplicate items rather merging them, it's possible you deleted the items that were in these collections originally. I can't say why the collections wouldn't have any subcollections if you just dragged them somewhere by mistake, though. Regardless, I'd strongly encourage you to use "+" and find-as-you-type to locate these collections in your local Zotero to see if they're where you expect them to be. Generally not, though you can certainly lose data if the drive is unmounted while Zotero is open.
In your last message, you say there is still a "PRODUCTION" collection in my library with only one item. That collection was the first to go missing. That collection is no longer in my library. As I reported earlier I was able to save the collection and it is now "PRODUCTION-NEW". I don't see a "PRODUCTION" collection. If you do, you are seeing the original. My new collections have nearly 100 sub-collections. Do you not see "PRODUCTION-NEW"?
This is the same for "ELECTIONS". This collection does not show in my library, but the recovered collection is there "ELECTIONS-NEW".
This is also the same for "WRITING". I no longer see that collection. When it went missing yesterday, the second computer in my office Zotero was not running. So I disconnected the computer from the internet. Turned Zotero on then created a new collection and moved the sub-collections into it. Then turned the internet on and synched. This saved the sub-collections. And now there is a new collection "WRITING-NEW" with a ton of sub-collections.
If you don't see any of the NEW collections and only the old...I don't see the old and only the new.
Somewhere in this has to a be a pointer to what's going on.
As I mentioned before, I'm a damn good trouble-shooter on computer issues. That is what's so frustrating about this.
If this is me, I have no ego in admitting that. Frankly, I'm stumped.
b...
Have you tried "+" and find-as-you-type?
PRODUCTION is under OFFICE → INSURANCE.
There are two ELECTIONS (not counting NEW). One is under ISSUES → SCIENCE → BIOLOGY. The other is under ISSUES → MEDIA → INDUSTRY → FACEBOOK.
WRITING is under OFFICE → WEBSITES.
You should see the collections at the following URLs:
PRODUCTION: https://www.zotero.org/shakyegg/items/collectionKey/828N5XTG
ELECTIONS (1): https://www.zotero.org/shakyegg/items/collectionKey/T3H3H5AM
ELECTIONS (2): https://www.zotero.org/shakyegg/items/collectionKey/JYJZ3VBJ
WRITING: https://www.zotero.org/shakyegg/items/collectionKey/QZRJ8VT5
The "ELECTIONS (1) and (2)" seem to be the missing "ELECTIONS" collection. But neither have the sub-collections. So these two would have somehow, NOT me were moved.
The WRITING collection you pointed to, also appears to be the missing collection, without the subcollections. So we can confirm the collection and subcollections are somehow migrating to some other place in the library on their own. So the have gone missing as I reported but not deleted. Notes that I did "MERGE" all the duplicates yesterday over 6 hours. So it's probable that those collections were cleared in the merges.
@dstillman is there any way it could be explained how collections are being moved and tucked into other collections like this. I swear there is no way that I could be accidentally doing this. And considering that it's happening over and over and I am so careful because of these problems it's impossible I would not have notice that I was doing it??
Lastly, your "+" and type quickly is really a very non-intuitive system. Perhaps if there was a search field that one could see when doing the search it would be easier. 4 times out of 5 nothing happens. The library expands but the app does not take me to what I am searching for. Half the time it feels like ghost typing.
The next time a collection goes missing, which is certain to happen I will attempt to search it out within my library.
None of this was a problem in 2014, 2015 or 2016 or 2017. It was only last year this problem started.
Thanks
Let us know if you see it again, and we can see if we can glean anything from the surrounding edits. Find-as-you-type is just the standard behavior of most tree views — e.g., Mac Finder. In our case it comes from Mozilla, where you'll see the same thing in the Firefox bookmarks manager. We're hoping to at some point offer a better collection search that provides better visual feedback and does non-left-bound searches, but the current behavior really does work. You just have to make sure you pause long enough before starting to type so it doesn't confuse it with whatever you typed previously. It also won't match more than one word.
Also, thanks for acknowledging the find-as-you-type is not ideal. I don't do much interaction with other tree views so it would make sense that the behavior is a bit foreign. I will put the + technique to work next time for sure.
On that note: we are good for now. At least I now have things cleaned up in terms of no dupes and nothing is unfiled. When it happens again the first thing i will do is the + technique and I will try to capture any other information that I can and get back to you guys.
Your support is excellent. Zotero does rock.
Will stay in touch.
Best
b...
I had a new collection highlighted (as you know one is always highlighted so I don't think it matters that it was new). I was scrolling through the tree to find a citation, quite some ways down from the said highlighted collection (it was no longer visible on the screen). And I clicked on a sub-collection folder to see its contents and snap, the folder that was highlighted above moved below into the collection I clicked on. The tree kind of "bounced" is the best way to describe it. It was jarring. The collection is "FAKE NEWS - GOVERNMENT CONTROL" in Elections/Federal/2019. I moved it back so fast that I forgot to note what collection I had clicked on that it moved to. But I actually saw it happen and it fits to my experience with what has been going on. since I have so many collections and if say it had been an hour or two since the last time I used the tool, that would explain why it's always a collection I felt i had just been working on. But there being no explanation where it went. If you are not looking directly at the (target) collection you might not see it bounce into it. To restate, I clicked on a collection and the previous collection that had been highlighted moved into the collection I clicked on. Obviously, this does not happen all the time otherwise that would be noticeable each time. Damn. It's like seeing a ghost!
* keep debug logging on
* save the debug log on a schedule (default 10 minutes)
* save the debug log when the "collection" notifier fires, including the collection(s) for which it was triggered.
It doesn't monkey-patch anything, it just uses a stock notifier and setTimeout.
Maybe something in the log will help get to the bottom of this. Note that having debug logging on will affect performance.
First, the + search is great. I had no idea. Now I can track down lost subcollections much more easily than my previous kluge of searching for a word in an article title. At least if I remember the missing subcollection's name, which is much more likely than my remembering a uniqueish word in an article title.
Second, I'm seeing similar behavior to what shakyegg sees: I'll be cruising through subcollections, and one that I've clicked on shows up in another that I've visited (visited later, I think). I can't say if this is the only way in which my subcollections get moved unintentionally, but it seems to certainly be one way.
Third, @emilianoeheyns I'd be happy to take you up on the pop up idea. I don't move or delete subcollections so much that it would be an annoyance. And it would prevent the very real problem of my mislaying subcollections.
Fourth, also @emilianoeheyns I'd love to get that debug log plugin. I really, really want to fix this thing before it causes me serious heartburn.
collectiontrigger fires; when thecollectiontrigger fires, it will also show a progress window for 8 seconds so that you can more easily find the relevant log(s).Having this plugin installed means it's active, and when it's active, it *will* slow down Zotero because it turns on debug logging when it starts. To deactivate it you need to uninstall it.
I also had @emilianoeheyns plugin installed and configured with a path to save. However, I didn't see any logs being created. Is the log only saved when a specific event is triggered?
Debug ID: D1304321284
Edit 1: I reconfigured zotero-collection-tracker plugin, and now I think it is saving the logs. I'll try to create the issue again and report back.
@dstillman is there a method we can submit the log from zotero-collection-tracker plugin privately? Maybe you can have a Google form to submit the log?
Edit 2: Debug ID: D462777790
Note: I have the log saved by zotero-collection-tracker plugin that corresponds to this debug ID. Let me know if you need it.
The issue happened to me similar to one reported by AucMGris and shakyegg. I noticed a collection called "Download" appeared under the collection "Electric double layer (1)". If I remember correctly, this collection was NOT at the same location previously. I am unsure how the replicate the issue.
Edit 3: Debug ID: D1788472761
Note: I have the log saved by zotero-collection-tracker plugin that corresponds to this debug ID. Let me know if you need it.
Now I am confident something weird is happening. I opened Zotero and pressed + to expand all collections and subcollection. I then clicked on the first collection which had subcollection. I click the arrow on this collection few times (~3-4 time) to collapse and uncollapse the visibility of the subcollections. I noticed suddenly that all the subcollections were duplicated under the collection. When I closed and reopened Zotero, everything appeared normal and I did NOT see duplicated Zotero. I confirmed this issue occured multiple times. I did further investigation and noticed that I do NOT have to close Zotero and reopen for things to go back to normal. IF I see duplicated subcollections, I can just collapse the visibility and uncollapse again for the duplicated subcollections to disappear and things to appear normal again. I do notice that when these duplicated collections appear, there is a lag in the GUI in collapsing/uncollapsing the visibility. The lag occurs even when zotero-collection-tracker plugin is disabled.
Note: This issue is NOT the same as reported by AucMGris and shakyegg where the subcollection appears to be moved to a different location. I am still in the process of trying to replicate this.
Note that disabling the tracker plugin doesn't do anything until Zotero is restarted.