Collections went missing again!

edited 20 days ago
I'm at the end of my rope with Zotero. For well over a year I have been experiencing missing subcollections. How do I describe the problem? Simply, subcollections just disappeared. I've reported this to Zotero support several times. At first, support indicated that I may have deleted the collection. I can report that is not the case. Not even accidentally. The other suggestion is that I may have accidentally moved a collection into another collection (since this appeared to be the case on at least one previous occasion). However, in the past 3-months subcollections have just disappeared and they could not be found in any other collection. Yes, the citations could be found in the unfiled folder, but that is no solution for me. As the sub-collections that disappeared usually contain anywhere from 30-60 sub-collections based on topics etc... all deep level filing. Recreating that is not an option.

It happens so often then restoring from a backup is just not an option. Mostly, because there is no way to know when a sub-collection has gone missing. Today I opened up the client and went to a sub-sub-collection containing citations I made in Oct 2017. The collection was gone. There is no way I deleted it. No way I moved it. And a restore back to that period is obviously not an option.

Today I stopped all my work and lost 6-hours trying to rebuild sub-collections, deleting hundreds (maybe thousands) of duplicates that have infested my system each time I've had to import backups etc... to rebuild after another sub-collection disappearance.

I love the platform. However, it's job is to be a secured storage location for extremely important research citations. Those citations need to be there a day, week, month or years later. There is no way one should trust Zotero anymore to not "disappear" collections.

I simply can not trust Zotero anymore. What to do?

Sigh!

B...
«1
  • I certainly understand the frustration of losing data, but I'm afraid there's not much we can do without steps to reproduce the problem. Because of how syncing works, there's no way for a collection to disappear locally and remotely except via an explicit delete operation, so something has to be triggering that. As you say, in your previous threads we discussed the possibility that either you're accidentally moving collections elsewhere in your library or something UI-related is causing you to accidentally trigger deletions.

    Regarding one of your previous posts, it seems you weren't able to figure out how to expand all collections in order to search by name. That definitely works — in the client, not the website — so I'd encourage you to try that again. Click a collection and press "+". You can then search for collections by name to see if one somehow ended up below another collection. (If you share the name here or in an email to support@zotero.org with a link to this thread, we can also take a look.)

    In your account logs, I see a bunch of collection deletions from December 27, when you last posted, and then two more on two separate dates this month (not counting one a couple hours ago). Do you recall making those deletions?
    the sub-collections that disappeared usually contain anywhere from 30-60 sub-collections based on topics
    Of the 76 collection deletions in your online library since October 2017, there were 10 collections deleted over the course of an hour in March and 16 collections deleted in a single sync when you last posted in December. All other deletions were of 1–5 collections at a time. So either you're misremembering the numbers or those 30–60 collections are still somewhere in your library (or they were created and lost without ever making it to the online library, which is unlikely).

    In any case, for now, the best thing you can do to avoid losing data is simply to keep longer backups of your Zotero data directory, or at least the database. You can do that with an external backup system (e.g., Time Machine), by manually making backups of zotero.sqlite, or by going to the Config Editor in the Advanced preferences and changing extensions.zotero.backup.numBackups to a higher number. (If you do the last one, make sure you have enough disk space for numBackups times the size of your database. I'd recommend just using Time Machine and making sure your backup disk is big enough that you have a long history of weekly backups.)

    I'm not sure what you mean re: duplicates, but you won't get duplicates restoring from proper backups. If you're restoring from exported files (RIS, BibTeX, RDF), you'll end up with duplicates, but that's not a recommended approach. See our instructions for restoring a deleted collection from a backup.
  • Hi @dstillman thanks for the quick response.

    1. yes, I am now absolutely certain there has been no accidental deletion or accidental moving of one collection into another. Yes, thanks to your help over the past year I have worked through that and I'm afraid this bug cannot be attributed to user error (I'd prefer that it was my fault as I could then do something about it.

    2. The duplicates I painfully erased yesterday were indeed from an import not a backup up restore. As I have noted in my first reporting of this problem, I was able to catch an unsynched version of the missing collections during the first couple of instances and either through exporting and importing or another method was renaming the missing collection to "-new" then resynch I was able to save the folder structure. Yes this created duplicates.

    3. Since day 1 with the bug, I have never been able to rely on a bag up using zotero.sqlite. The most recent backup made by the system was never truly up-to-date and therefore the choice was to try the export method or lose all the new citations from whatever day that backup was

    4. Collections: I rarely delete collections on my own. The 30-60 collections I was referring too would have been around August 20, 2018. This was my Production collection. Under it are approximately 100 collections and sub-collections. The top-level Production had just disappeared as I reported.

    5. About a week later my Elections top-level collection disappeared (Aug 28th I reported it)

    6. Between then and this past December it happened with a small number of collections that I managed to save.

    7. This December it became to much of a problem and I reported it again

    8. January 2019, yesterday I lost my Writing top-level collection. And even though I managed to save it, there was a critical sub-collection in it that was just gone. And I could not figure out when it disappeared. This was a subcollection of notes I made and quotes taken from books I have been reading.

    9. I have a complicated mix of drives connected to all my computers as I boot externally from SSD not internally. I will get a carbon copy cloner set up on the one boxs where I can get a weekly backup done. On my list for Q1.

    10. Could you please send me the list of the collections you show deleted and the dates.... this would be huge. Since: (1) I rarely delete collections and if you see deletions over 1-5 in a day then those events are MOST Certainly the bug. I could easily look at the list and match that with my daily work calendar and I can report back if they were my deletions or part of the bug. Please send to my email: brent.stafford@shakyegg.com

    Let's see if we can shed some more light on what's going on.
  • 9. I have a complicated mix of drives connected to all my computers as I boot externally from SSD not internally. I will get a carbon copy cloner set up on the one boxs where I can get a weekly backup done. On my list for Q1.
    Could you say more about that? Is the same HD potentially connected to multiple computers at times? Depending on your set up, I can absolutely see this as part of what's going on.

    What you're seeing isn't happening to other Zotero users (of which there are >>100,000) and as Dan says it's highly unlikely to happen under normal usage given the code base, so if it isn't something that you actively do as a user, your specific set-up is the next most likely cause.
  • That's a good point @adamsmith. Note though when I google missing collections Zotero there are most certainly other people reporting missing collections. Some of the same advice re: backups etc... Dan has given me over the past year are also in responses to other users.

    Complicated is mostly for me. As I run regular carbon copy cloners on my external data drives not my system drives. If I ever have a problem with system drives I would never restore from back up. I always do the reformat and fresh install. Why backup a crappy or corrupted system when you can install fresh.

    So let's see, here are specifics.

    HOME NETWORK
    #1 - MacBook Pro Laptop - System Drive (INTERNAL SSD) Zotero installed on System and Zotero library is stored on System

    #2 - iMac 2012 - System Drive (EXTERNAL SSD) Zotero installed on System and Zotero library is stored on System

    Computer #1 and #2 are shared on the same network. However, I rarely if ever mount the system drive from one computer onto the other. I don't think I ever have as there would be no reason. I just call up the external drives mounted on the computers in order to transfer files as need. Which is not often as there are NO local files on either of these computers. All are synced via DropBox. Rarely would #1 connect to #2 or the other way around. Note DropBox on #1 is located on an external and Dropbox is located on an internal for #2.

    However, #1 and #2 do see each other in the shared computers on my finder bars.

    OFFICE NETWORK

    #3 - iMac 2015 - System Drive (INTERNAL SSD) Zotero installed on System and Zotero library is on stored on (EXTERNAL 4TB HDD). This is my main video edit box and the internal System Drive is not large enough to store applications and Zotero library. EXTERNAL is Western Digital USB3 (fast enough to edit 2K video) and the computer is set to never put hard drives to sleep.

    #4 - iMac 2013 - System Drive (INTERNAL SSD) Zotero installed on System and Zotero library is stored on System

    Computer #3 and #4 are shared on the same network AND the above notes on usage for #1 and #2 apply.

    Is there anything here that stands out as being the issue?

    Thanks for the help!!
  • So does Dropbox sync include the part that contains the Zotero data folder?
  • Nope. The DropBox app, like the Zotero app is installed on the System Drive for each computer. But DropBox data never resides on the same drive as Zotero Library in any of my setups. Let me ask, is there a problem with having the Zotero library on an external? Keep in mind that for the most part 99% of the time all my computers are running, and they are always set to never put hard drives to sleep...
  • edited January 30, 2019
    Since day 1 with the bug, I have never been able to rely on a bag up using zotero.sqlite. The most recent backup made by the system was never truly up-to-date and therefore the choice was to try the export method or lose all the new citations from whatever day that backup was
    No, that's a misunderstanding. As long as you're syncing, you wouldn't lose data by restoring from a backup and following the restore steps we provide for restoring a collection. (The only possible data you would lose would be organizational changes you made to that specific deleted collection since the backup.)
    Collections: I rarely delete collections on my own. The 30-60 collections I was referring too would have been around August 20, 2018.
    Only 5 collections were deleted from your online library in August 2018. I've sent the log via email, though note that we don't keep the names of deleted collections, only timestamps.

    Do you recall the names of any subcollections that you can't find?
    This was my Production collection. Under it are approximately 100 collections and sub-collections.
    There's a "PRODUCTION" collection in your library, created in December 2017. It doesn't have any subcollections and has one item. Perhaps that's a different collection?
    About a week later my Elections top-level collection disappeared (Aug 28th I reported it)
    As I noted previously, there's still an "ELECTIONS" collection in your library, created in February 2017 and modified in August 2018. It doesn't have any subcollections or items.
    January 2019, yesterday I lost my Writing top-level collection
    There's a "WRITING" collection in your library, created in February 2017 and modified yesterday. It doesn't have any subcollections or items.

    If you deleted duplicate items rather merging them, it's possible you deleted the items that were in these collections originally. I can't say why the collections wouldn't have any subcollections if you just dragged them somewhere by mistake, though. Regardless, I'd strongly encourage you to use "+" and find-as-you-type to locate these collections in your local Zotero to see if they're where you expect them to be.
    Let me ask, is there a problem with having the Zotero library on an external?
    Generally not, though you can certainly lose data if the drive is unmounted while Zotero is open.
  • Hi @dstillman thanks the additional details on what you see on your end. I think we may be zeroing in on something here. Please follow this link for screen caps of my Zotero library: https://www.dropbox.com/s/9vddg49wu2fynkc/BRENT STAFFORD ZOTERO CAPS.zip?dl=0

    In your last message, you say there is still a "PRODUCTION" collection in my library with only one item. That collection was the first to go missing. That collection is no longer in my library. As I reported earlier I was able to save the collection and it is now "PRODUCTION-NEW". I don't see a "PRODUCTION" collection. If you do, you are seeing the original. My new collections have nearly 100 sub-collections. Do you not see "PRODUCTION-NEW"?

    This is the same for "ELECTIONS". This collection does not show in my library, but the recovered collection is there "ELECTIONS-NEW".

    This is also the same for "WRITING". I no longer see that collection. When it went missing yesterday, the second computer in my office Zotero was not running. So I disconnected the computer from the internet. Turned Zotero on then created a new collection and moved the sub-collections into it. Then turned the internet on and synched. This saved the sub-collections. And now there is a new collection "WRITING-NEW" with a ton of sub-collections.

    If you don't see any of the NEW collections and only the old...I don't see the old and only the new.

    Somewhere in this has to a be a pointer to what's going on.

    As I mentioned before, I'm a damn good trouble-shooter on computer issues. That is what's so frustrating about this.

    If this is me, I have no ego in admitting that. Frankly, I'm stumped.

    b...
  • No, I see all the NEW ones as well.

    Have you tried "+" and find-as-you-type?

    PRODUCTION is under OFFICE → INSURANCE.

    There are two ELECTIONS (not counting NEW). One is under ISSUES → SCIENCE → BIOLOGY. The other is under ISSUES → MEDIA → INDUSTRY → FACEBOOK.

    WRITING is under OFFICE → WEBSITES.

    You should see the collections at the following URLs:

    PRODUCTION: https://www.zotero.org/shakyegg/items/collectionKey/828N5XTG
    ELECTIONS (1): https://www.zotero.org/shakyegg/items/collectionKey/T3H3H5AM
    ELECTIONS (2): https://www.zotero.org/shakyegg/items/collectionKey/JYJZ3VBJ
    WRITING: https://www.zotero.org/shakyegg/items/collectionKey/QZRJ8VT5
  • Got it. So this "Production" collection is under Insurance as it's a subcollection for production insurance. It is not the missing "PRODUCTION" collection.

    The "ELECTIONS (1) and (2)" seem to be the missing "ELECTIONS" collection. But neither have the sub-collections. So these two would have somehow, NOT me were moved.

    The WRITING collection you pointed to, also appears to be the missing collection, without the subcollections. So we can confirm the collection and subcollections are somehow migrating to some other place in the library on their own. So the have gone missing as I reported but not deleted. Notes that I did "MERGE" all the duplicates yesterday over 6 hours. So it's probable that those collections were cleared in the merges.

    @dstillman is there any way it could be explained how collections are being moved and tucked into other collections like this. I swear there is no way that I could be accidentally doing this. And considering that it's happening over and over and I am so careful because of these problems it's impossible I would not have notice that I was doing it??

    Lastly, your "+" and type quickly is really a very non-intuitive system. Perhaps if there was a search field that one could see when doing the search it would be easier. 4 times out of 5 nothing happens. The library expands but the app does not take me to what I am searching for. Half the time it feels like ghost typing.

    The next time a collection goes missing, which is certain to happen I will attempt to search it out within my library.

    None of this was a problem in 2014, 2015 or 2016 or 2017. It was only last year this problem started.

    Thanks
  • Notes that I did "MERGE" all the duplicates yesterday over 6 hours. So it's probable that those collections were cleared in the merges.
    Merging items doesn't affect collections, so that wouldn't explain that.
    is there any way it could be explained how collections are being moved and tucked into other collections like this
    We're not aware of anything, but you have an order of magnitude more collections than most people, so you very well might be hitting some bug that others aren't.

    Let us know if you see it again, and we can see if we can glean anything from the surrounding edits.
    Lastly, your "+" and type quickly is really a very non-intuitive system.
    Find-as-you-type is just the standard behavior of most tree views — e.g., Mac Finder. In our case it comes from Mozilla, where you'll see the same thing in the Firefox bookmarks manager. We're hoping to at some point offer a better collection search that provides better visual feedback and does non-left-bound searches, but the current behavior really does work. You just have to make sure you pause long enough before starting to type so it doesn't confuse it with whatever you typed previously. It also won't match more than one word.
  • @dstillman Ok got it! Good to know that I am a crazy person with the number of collections I have ;) (you should see my DropBox!)

    Also, thanks for acknowledging the find-as-you-type is not ideal. I don't do much interaction with other tree views so it would make sense that the behavior is a bit foreign. I will put the + technique to work next time for sure.

    On that note: we are good for now. At least I now have things cleaned up in terms of no dupes and nothing is unfiled. When it happens again the first thing i will do is the + technique and I will try to capture any other information that I can and get back to you guys.

    Your support is excellent. Zotero does rock.

    Will stay in touch.

    Best

    b...
  • Perhaps for this particular case the debug log could be enabled by default by changing the launch shortcut to pass "-ZoteroDebugText"? Then the debug log submitted will have the SQL queries -- perhaps if it can be spotted right after it happens, the SQL queries performed against the database could provide a clue to what triggered this.
  • (alternately I'd be happy to put together a plugin that pops up an alert whenever a collection is deleted or moved -- which would be hugely annoying of course, but at least an unexpected collection change would be detected instantly and the actions leading up to it could be described by the user. Unless it happens before the notifier is active of course)
  • @emilianoeheyns a plugin would be an excellent idea! @dstillman for the first time I actually witness the move happening. It just happened. It's quick.

    I had a new collection highlighted (as you know one is always highlighted so I don't think it matters that it was new). I was scrolling through the tree to find a citation, quite some ways down from the said highlighted collection (it was no longer visible on the screen). And I clicked on a sub-collection folder to see its contents and snap, the folder that was highlighted above moved below into the collection I clicked on. The tree kind of "bounced" is the best way to describe it. It was jarring. The collection is "FAKE NEWS - GOVERNMENT CONTROL" in Elections/Federal/2019. I moved it back so fast that I forgot to note what collection I had clicked on that it moved to. But I actually saw it happen and it fits to my experience with what has been going on. since I have so many collections and if say it had been an hour or two since the last time I used the tool, that would explain why it's always a collection I felt i had just been working on. But there being no explanation where it went. If you are not looking directly at the (target) collection you might not see it bounce into it. To restate, I clicked on a collection and the previous collection that had been highlighted moved into the collection I clicked on. Obviously, this does not happen all the time otherwise that would be noticeable each time. Damn. It's like seeing a ghost!
  • Wild. Are you able to reproduce this if you just click around collections a lot? If so, a Debug ID for it would be quite helpful. (You can also leave that running and periodically clear the output so it doesn't get too big before you reproduce this.)
  • @dstillman now that I know what to look for on my next round of adding citations I'll turn on debugging and try to replicate. Will be back in touch.
  • @dstillman if you think it's helpful, I have a plugin which will:

    * keep debug logging on
    * save the debug log on a schedule (default 10 minutes)
    * save the debug log when the "collection" notifier fires, including the collection(s) for which it was triggered.

    It doesn't monkey-patch anything, it just uses a stock notifier and setTimeout.

    Maybe something in the log will help get to the bottom of this. Note that having debug logging on will affect performance.
  • edited 28 days ago
    Couple things.

    First, the + search is great. I had no idea. Now I can track down lost subcollections much more easily than my previous kluge of searching for a word in an article title. At least if I remember the missing subcollection's name, which is much more likely than my remembering a uniqueish word in an article title.

    Second, I'm seeing similar behavior to what shakyegg sees: I'll be cruising through subcollections, and one that I've clicked on shows up in another that I've visited (visited later, I think). I can't say if this is the only way in which my subcollections get moved unintentionally, but it seems to certainly be one way.

    Third, @emilianoeheyns I'd be happy to take you up on the pop up idea. I don't move or delete subcollections so much that it would be an annoyance. And it would prevent the very real problem of my mislaying subcollections.

    Fourth, also @emilianoeheyns I'd love to get that debug log plugin. I really, really want to fix this thing before it causes me serious heartburn.
  • Okay. I just saw it happen. I used + to show all my subcollections. Then I clicked on all the top-level subcollections to close them. After 6 or 7 clicks and closes, one of the top-level subcollections I had previously clicked just showed up as a subsubcollection in the subcollection that I was preparing to click next. The subcollection that moved had been clicked about 3 or 4 closes before it jumped. I hope this is at least a bit useful.
  • As I say, if you can reproduce this easily, we'd want a Debug ID for it happening.
  • edited 28 days ago
    @AucMGris I'd prefer to know if @dstillman thinks it's useful -- every plugin installed increases the complexity of what the Zotero crew has to analyse. The plugin will turn on debug logging, save the log every 10 minutes (interval is configurable) and every time the collection trigger fires; when the collection trigger fires, it will also show a progress window for 8 seconds so that you can more easily find the relevant log(s).
  • (the potential benefit of the plugin would be that @AucMGris wouldn't have to reproduce this, as the log would just be available when it happened)
  • Given that @dstillman hasn't answered, could we have a go with the plug-in? It's happened again, btw. Same as last time. Working in one subcollection. Close it. Go to another subcollection and open it. The subcollection I was working in then appears in the newly opened subcollection.
  • edited 25 days ago
    How you generate the Debug ID is up to you. From your description, it sounded like you could generate one in a few minutes of trying. But if the plugin is easier, that's fine. Since we can't reproduce this ourselves, one way or another we'll need a Debug ID to troubleshoot this further.
  • https://github.com/retorquere/zotero-collection-tracker/releases doesn't automatically generate a debug report (although it probably could). I don't know what the debug ID does more then send off the current debug log -- if that's what it does, then with this plugin you will have the debug log around the event, but you'd have to get it to the Zotero crew in some way. Read the README on the repo to see how to set it up.

    Having this plugin installed means it's active, and when it's active, it *will* slow down Zotero because it turns on debug logging when it starts. To deactivate it you need to uninstall it.
  • Thanks to both of you. I'll install it and see how things work out.
  • edited 25 days ago
    I tried to replicate issues reported by @AucMGris and @shakyegg. My subcollections didn't disappear BUT I did notice that few of my subcollections were duplicated. If I close Zotero and reopen it, then the duplicated subcollections disappeared and returned back to normal.

    I also had @emilianoeheyns plugin installed and configured with a path to save. However, I didn't see any logs being created. Is the log only saved when a specific event is triggered?

    Debug ID: D1304321284

    Edit 1: I reconfigured zotero-collection-tracker plugin, and now I think it is saving the logs. I'll try to create the issue again and report back.

    @dstillman is there a method we can submit the log from zotero-collection-tracker plugin privately? Maybe you can have a Google form to submit the log?


    Edit 2: Debug ID: D462777790
    Note: I have the log saved by zotero-collection-tracker plugin that corresponds to this debug ID. Let me know if you need it.

    The issue happened to me similar to one reported by AucMGris and shakyegg. I noticed a collection called "Download" appeared under the collection "Electric double layer (1)". If I remember correctly, this collection was NOT at the same location previously. I am unsure how the replicate the issue.

    Edit 3: Debug ID: D1788472761
    Note: I have the log saved by zotero-collection-tracker plugin that corresponds to this debug ID. Let me know if you need it.

    Now I am confident something weird is happening. I opened Zotero and pressed + to expand all collections and subcollection. I then clicked on the first collection which had subcollection. I click the arrow on this collection few times (~3-4 time) to collapse and uncollapse the visibility of the subcollections. I noticed suddenly that all the subcollections were duplicated under the collection. When I closed and reopened Zotero, everything appeared normal and I did NOT see duplicated Zotero. I confirmed this issue occured multiple times. I did further investigation and noticed that I do NOT have to close Zotero and reopen for things to go back to normal. IF I see duplicated subcollections, I can just collapse the visibility and uncollapse again for the duplicated subcollections to disappear and things to appear normal again. I do notice that when these duplicated collections appear, there is a lag in the GUI in collapsing/uncollapsing the visibility. The lag occurs even when zotero-collection-tracker plugin is disabled.
    Note: This issue is NOT the same as reported by AucMGris and shakyegg where the subcollection appears to be moved to a different location. I am still in the process of trying to replicate this.




  • edited 24 days ago
    Zotero needs to be restarted after the plugin settings have changed as it will only read the settings on startup; I've added this to the README.

    Note that disabling the tracker plugin doesn't do anything until Zotero is restarted.
Sign In or Register to comment.