Stuck on the beta...

I'm running 5.0.97-beta.18+1eb888d02, and it is basically useless. When I open it, it works for maybe a minute, but as soon as Zotero starts using more than 2 GB of RAM or so, there's nothing that can be done anymore. I can also not sync from other computers anymore - presumably because the sqlite my main computer is in the new file format.
I did not make a copy of my previous sqlite. I also don't remember being advised to. And Zotero itself seemingly also didn't make an automatic backup (as I would have expected it to). Mo own last zotero.sqlite.bak dates from about a year ago, and misses a lot of items we have added since then.
So if anybody has any idea about how I can get back 'into business', I'd be very grateful! Will the release of 5.1 still last days/weeks/months? Is there anything I can do already now? Can I do something to help debug?
Thanks!
  • 1) Zotero makes an automatic backup in the data directory before the upgrade. It will be the highest-numbered file (e.g., zotero.sqlite.109.bak or similar).

    2) If you were using syncing, you can restore from any backup or no backup and simply sync to pull down all remaining data.

    3) If you're having trouble with the beta, we'd like your help investigating that, since no one else has reported anything like this, and unless you're experiencing something specific to the PDF reader or new note editor it shouldn't really be acting any differently from the current release version of Zotero. If you disable the PDF reader from the General pane of the preferences, it should be indistinguishable from 5.0.96.2.

    Can you say more about what the actual problem is? Are you saying Zotero uses more than 2 GB after a minute of being open, without your doing anything? Does this happen with all extensions disabled? Can we see a Debug ID for Zotero startup through this happening?

    Also, it's worth reminding us (and bearing in mind) how extreme your situation is when posting here about a performance issues. You're likely in more groups than any other Zotero user and have over 600,000 items. It's possible there are some undiscovered performance regressions in the beta, but you might have also just added even more items since when you were last on the release version.

    We have an open ticket to allow removing groups from a given local install, and it seems like that would be useful in your case to keep the database size manageable on a given computer at a given time.
  • edited April 29, 2021
    1) Zotero makes an automatic backup in the data directory before the upgrade. It will be the highest-numbered file (e.g., zotero.sqlite.109.bak or similar).
    FWIW, this almost definitely did not happen for me on a much more standard-sized library (<20k items in <20 libraries, ~600MB sqlite).

    I updated on my main machine shortly after the beta came out (I have an upgrade log from March 13 in the data folder, which sounds about right) and I currently only have .bak and .1.bak, both from this week.
    I don't run any cleaning tools and only Window's built-in security, and obviously I never muck around manually in the data folder, so I'm fairly certain this never existed. Happy to send the upgrade log if that's helpful.

    (To be clear, I'm not having any of the above problems, and no need to get off the beta branch, I was just curious about the automatic .bak)
  • edited April 29, 2021
    Thanks Dan! On your points

    • Extreme situation - yes, I realize I am an unusual user :). But we run a bunch of projects per year, with multiple team members, many projects involve text-mining of larger full-text corpora, and we always use Zotero, which really IS my online library besides being my bibliographic management program. But so I guess I'll set up separate profiles with just the groups we're working on at any give time. It's duplication and has an impact on storage, but if that's the only way to do this...[Just FYI - it worked quite well until this beta, EVEN with my outsized library] But yes, I will make that clear next time :)

    • Automatic backup - as I indicated, my data directory only has one bak file (zotero.sqlite.1.bak), which dates from 5/11/2020. So (just as Sebastian suggested) it really didn't create a backup. At least not there. (And btw - you can see my upgrade log here)

    • Sync - I was just trying to think why it doesn't fully sync. So given Zoteros' recent behavior, I immediately go to the folder I want to sync, right-click on it and do sync. The icon starts spinning, but it never fully completes (as I see in the Web library)

    • Debugging - ok, I'll try a few times with different options  (starting with my current extensions, and then reducing them) and see what it gives. 

    But thanks again for looking into this!

    UPD - I submitted a Debug ID early on after rebooting (while I still could) and it's D2105253048. Will keep trying with different settings.
    UPD2 - I disabled all add-ons, and now it seems to keep working. Here are the Debug IDs (all from the same runs, with the first ones first - D1376789218; D1923403473; D97104933, D878328403, D182165189. So the last, most recent one, is the one to look at I guess. I'll let it run now, and will still submit one after it has completed loading all libraries
    UPD3 - it still got stuck.

    Not sure whether this helps you, but there is also this. Note the 2.7 GB of RAM and the 'very high' power usage...

  • FWIW, this almost definitely did not happen for me
    @adamsmith: It could also be that later updates removed the 'zotero.sqlite.109.bak' file. Did you check your backup? I found such a file in my backup from right after switching to the beta, but I don't see it in the active Zotero data folder. Since I don't remember removing it, the file might have been deleted automatically in subsequent updates.
    my data directory only has one bak file (zotero.sqlite.1.bak), which dates from 5/11/2020
    @sdspieg: If your last automatic backup dates from 5/11/2020, this seems to indicate some problem that needs to be debugged. Maybe this is related to the size of your 'zotero.sqlite' file. Have you checked whether there's enough space for multiple copies of that file on your disk?
  • I don't have a backup between 3/13 and 3/17, but it's not in the 3/17 back-up, no. I remember from previous updates that the automatic update back-ups staid around for quite some time, so if it got deleted after some days of use, that'd be undesirable imo (and presumably a bug?)
  • @sdspieg: If your last automatic backup dates from 5/11/2020, this seems to indicate some problem that needs to be debugged. Maybe this is related to the size of your 'zotero.sqlite' file. Have you checked whether there's enough space for multiple copies of that file on your disk?

    My sqlite file is 3.5 GB and is on a drive that has 121 GB free now. I also very much doubt it would have had that little space when I switched to the beta to try out the pdf annotation too.  If there is anything you'd like me to do to help in troubleshooting this, please let me know.

  • It could also be that later updates removed the 'zotero.sqlite.109.bak' file.
    Yes. There are minor DB updates that don't make automatic backups and major ones that do, and it looks like currently if there's another minor DB update afterward, it will still remove the last automatic backup file from the previous major update. It's possible it shouldn't do that, but the alternative is just always having a fourth copy of the file in the data directory, even years after a major DB upgrade. (On the one hand, if you don't have disk space for all of them, it won't be able to make a backup when it needs to. On the other hand, you might benefit from the extra space temporarily when one doesn't exist.)

    But regardless, @sdspieg, if you're missing a zotero.sqlite.bak and have a .1.bak from 2020, then automatic backups aren't working at all for you. We could debug that, but I'm not sure automatic backups should even be taken for databases greater than 500 MB or so…

    I'll look at the Debug IDs for the performance issues. Again, there shouldn't be any difference in this regard between the release and beta versions, so if there is we'll want to fix that.
    So given Zoteros' recent behavior, I immediately go to the folder I want to sync, right-click on it and do sync. The icon starts spinning, but it never fully completes (as I see in the Web library)
    See Changes Not Syncing for that.
  • Just an update - since I turned off automatic syncing, things work as well as before. Zotero now uses some 800MB, and works mostly fine (I still have a problem with the connector - see 1655429151, but I'm looking into that,
  • edited May 7, 2021
    If there's a specific action (e.g., syncing) that causes memory or performance problems, we'd want to see a Debug ID for just that action. There haven't been any major changes to syncing in the beta. After a recent change, the latest betas also should use far less memory when open for a while than the release version.
Sign In or Register to comment.