False error report after refresh - 163149263

I am using Zotero in a 300-page WORD document with around 540 citations, so I run it with automatic refresh off. When I run a manual refresh it takes about 20 minutes and as far as I can see correctly updates the footnotes and builds the bibliography. It then says that an error occurred. It does not say what kind of error and I can see nothing wrong, but it is puzzling and has been happening consistently for a few weeks now, so I sent you a report id. 163149263. I'm happy to send the document too if you wish.
«1
  • As a first step, try to delete the bibliography from the document, then refresh and reinsert it.
  • @AndySymons: If it's still happening after that, we'd want a Debug ID (different from a Report ID) for this occurring.
  • This (false) error just happened again - ReportID 2090408165.

    Following your instructions, I enabled debugging in Zotero, deleted the Bibliography in WORD and started ‘refresh’ without a Bibliography.

    The next step involves guesswork because Zotero gives no indication of progress with refresh, or even indicates that it is running (a shortcoming I have previously mentioned and have now submitted in a separate thread as a feature request). It does not even give a message when it is finished. I usually tell that is it finished by seeing that the Bibliography has been completed.

    There being no Bibliography, I waited an arbitrary 20 minutes; thinking that was about how long is usually takes. I then attempted to insert a bibliography. I was evidently too early; no bibliography appeared. Shortly after, as it happens, a pop-up dialog asking me if I want to keep changes to a footnote appeared, so apparently refresh was still running.

    I answered “no”, as always, because in my case any changes to the Zotero fields can only be accidental. I waited a further 10 min and tried again with the same result. I was evidently being too optimistic about how long it takes. Maybe it is slower with debugging turned on?

    I took another period of 10 minutes. The changes to the dialog popup appeared several more times (I’m not sure why there were so many today, but this does not seem to be a problem).

    Then another 10 minutes – taking me up to 50 altogether.
    Then another 10 minutes – taking me up to the full hour.
    Then another 10 minutes – taking me up to 1 hour and ten minutes.

    After about an hour and five minutes the error message appeared "An error has occurred. Do you want to see troubleshooting instructions?"

    I tried (for about the third time) to insert a Bibliography but I got the message “You must insert a citation before performing this operation” (!).

    This generated a whole heap of debug report (6559 lines). I hope it helps

    The DebugID is: D863527690

    Happy reading!


  • edited December 1, 2020
    A little update ... I starting afresh today. I found the document with updated citations but no Bibliography. I inserted one and got that message "An error has occurred. Do you want to see troubleshooting instructions?". I said no. Nothing happened and I went away to do other things, but leaving the document open. About an hour later, the Bibliography suddenly appeared. I guess it must have done the whole refresh again?

    As I originally said, I am not actually seeing anything wrong with any citation or the Bibliography. I just want to know why there is an error message if there is no error!
  • Can you repeat this operation with the debug logging enabled (if you get the same error) and submit a log? Make sure that no Word operations are running already (e.g. restart Zotero before doing this).

    Looking at your previously submitted log the updates are indeed slow. You might benefit from splitting your document into multiple documents for each chapter, working on them separately and then recombining at the end.
  • 1. Did the previous log tell you why there was a false error report?

    2. I think refresh ran slower than usual when error logging was on. It took over an hour and it usually takes about 20 minutes without the debug log. Anyway, I would rather leave it running for an hour than go through the palaver of splitting the document, which would be a lot more effort!
  • 1. Did the previous log tell you why there was a false error report?
    There seem to be multiple types of errors occurring, but it is unclear whether they have the same source or not. We can only target one specific reproducible issue at a time so that we know what we're dealing with and figure out how to fix that.
    Anyway, I would rather leave it running for an hour than go through the palaver of splitting the document, which would be a lot more effort!
    This isn't really all that involved. If you can split your document into 4-6 different chapters (by simply copy-pasting the relevant parts into new documents), you will have a much faster time updating with Zotero. Depending on your Mac and how well Word runs it might help with the performance there too. At the end of writing you just have to recombine all parts and refresh. This is standard advice for "heavy" documents.

    On the other hand it is likely that you will be able to reproduce the bug and be able to do faster troubleshooting on a reduced version of the doc.
  • "There seem to be multiple types of errors occurring, but it is unclear whether they have the same source or not. We can only target one specific reproducible issue at a time ...
    I only get one error message: at the end of refresh. Can you see what that one didn't tell me?
  • The previous log contained multiple operations. A refresh that completed successfully followed by an attempt to insert a bibliography which failed and seems to have displayed an error message properly, followed by another attempt to insert a bibliography that failed when Zotero couldn't retrieve any fields from the document and displayed an error saying you should insert a citation first.

    Unless we can isolate this to a specific action that is repeatable and always produces the same result we cannot debug it properly. How old and what kind of Mac this is that you're working with? Retrieving references takes a really long amount of time and in the last year Mac Word speed was quite decent, although I don't know if Big Sur could have made a negative difference.
  • I gave a detailed account of what I did and saw in my posting above (yesterday at 12:22 pm Zotero Forums time). I can't make it any more repeatable than that! I did not do anything else in Word or Zotero while this test was running.

    I am running on a late 2012 27-inch iMac with 3.2GHz quad core i5 processor and 24Gb of RAM. The disk is a 1 Tb fusion drive. I can see how the computer might affect performance (I am happy with it) but not how it would create the error message?
  • I can't make it any more repeatable than that
    If you cannot isolate this to a single repeatable action, hopefully in a more compact document (by e.g. trimming it down with step 10 from debugging broken documents) then we will not be able to fix it, since we eventually need to be able to reproduce it ourselves. You claim that the bibliography inserted fine when you were logging the actions in the previous logs, but based on what I see in the logs that can't have happened.
    I am running on a late 2012 27-inch iMac with 3.2GHz quad core i5 processor and 24Gb of RAM. The disk is a 1 Tb fusion drive. I can see how the computer might affect performance (I am happy with it) but not how it would create the error message?
    I was just generally wondering whether the performance you have is on a relatively new system or something older. The age of it explains some of the slowness.
  • edited December 4, 2020
    I carried out a series of tests as you requested followed the procedure:
    1. Opened the last version of the thesis (dated 201204, set to read-only) with macros enabled (but I did not run any macros)
    2. Duplicated it and saved the file as “Zotero test n.docm” in a desktop folder.
    3. Deleted from the start of the document to the location indicated in the table below. Saved again. Of course the chapter numbers and pages are all different now, but I ignored that.
    4. Went to the Bibliography. This was always in the state “Automatic citation updates are disabled. To see the bibliography, click Refresh in the Zotero tab”.
    5. Went to Zotero and enabled debug logging.
    6. Went back to word and pressed refresh in the Zotero menu.
    7. Noted the start time in the table below.
    8. Waited until either the Bibliography was filled-in or an error message appeared.
    9. Noted the end time in the table below.
    10. In Zotero, submitted the debug log, copied the number to the clipboard and pasted it in the table below.
    11. Saved the test file in its final state.

    In these tests, I did not have any occurrence of the dialogue box asking if I wanted to keep changes (i.e. there were no changes; must have cleared them all yesterday).

    When it appeared, the error message was always “Zotero experienced an error updating your document. Do you want to see troubleshooting instructions?”. I answered no.

    These are the results.
    Test Scope Start time (GMT) End time (approx.) Duration (approx.) Result Debug log

    Test 1 From ch 8 to end 21:10 21:27 0:17 PASS. No error message. Bibliography inserted. D1323491916

    Test 2 From ch 6 to end 22:04 22:11 0:07 PASS. No error message. Bibliography inserted. D1981162491

    Test 3 From ch 2 to end 22:17 22:30 0:13 FAIL. Error message and Biography blank. D831396530

    Test 4 From ch 4 to end 22:35 22:42 0:07 FAIL. Error message and Biography blank. D1784676899

    Test 5 From ch 5 to end 22:46 22:50 0:04 FAIL. Error message. Bibliography still says “Automatic citation updates are disabled. To see the bibliography, click Refresh in the Zotero tab.” D750226190

    Test 6 From about ½ chapter 5 22:56 22:03 0:07 FAIL. Error message and Biography blank. D249557457

    Test 7 From about ¾ chapter 5 23:10 23:13 0:03 PASS. No error message. Bibliography inserted. D1822112187

    Test 8 From about 5/8 of chapter 5 23:21 23:31 0:10 FAIL. Error message and Biography blank. D272329732


    I have mailed you a zip file with a better formatted version of this report (!) and all 8 test files referencing the original report id 163149263
  • If the error is related to a specific reference (?) then this narrows it down to three pages.
  • Today I conducted two more tests.

    In test 9 I copied and pasted all of chapter 5 into a new document. I did the refresh and this PASSED. Debug ID D1679714153.

    In test 10, I selected the same text, but by taking a copy of the whole original document then deleting the parts before chapter 5 and chapter 6 to the conclusion, leaving in the list of figures and the index as I had in test 1 to 8. This test also passed! Debug ID D399389155.

    Sadly this confirms my previous experience that simple binary division of the document does not lead one to the error. It would not appear to be any particular citation that is 'causing' the problem. Neither would it appear to be related to the presence of macros, the list of figures and the Index that was generated by WORD.

    I am at a loss where to look now, but I have kept a copy of the 'test' version of the thesis and the test documents (after refresh) in case you want me to run any further tests.

  • I tried refreshing your test files with an early 2015 MBP running Big Sur with Word version 16.43 and ran into no errors. It seems that Zotero/Word fails to set the bibliography text field when you get the error. I can't think of anything else to attribute the errors you are seeing to than computer/Word/macOS slowness, especially since they do not happen every time.

    The updates on my Mac took quite a substantial amount of time too. But after monitoring how it went for a bit I can offer you these two pieces of advice:
    1. Switch focus to Word and then Zotero when performing Word processor operations, but especially refreshing. It seems that macOS is giving very little CPU time to Zotero if it is in the background, making the citation processing Zotero needs to do extremely slow. We'll investigate it and see how we can improve the situation in this regard, but for now you should make sure Zotero is in the foreground.
    2. If the bibliography update fails, remove the bibliography field from the document and use the Add/Edit Bibliography option. This will prevent the Zotero from updating every field in the document, which should make the bibliography insertion/update operation faster.

    The other advice still stands:
    1. Split your document into multiple chapters/sections and work on them individually and recombine at the end.
    2. If you can, use a Windows machine (or even a Windows virtual machine on macOS) to refresh the document, which will make it much faster and error-proof.
    3. If you can, switch to an in-text citing style (like APA or Chicago author-date) while working on the document. Note styles/citations in notes are unfortunately yet another thing that makes citing slower. You can switch back to the required citation style at the end of writing. Note that if you need to add some extra text in a footnote next to a citation, you can still insert a footnote manually, add the comment and then the citation. Vice-versa, when converting to an in-text style, if there are comments in a footnote next to a citation, Zotero will not convert the citation to an in-text one.

    Thanks for the detailed debugging and I'm sorry we cannot offer anything more substantial for these errors at the time.
  • Thanks for the tips. Do I take it that you found no 'hard' errors in the logs, and that you think the errors are occurring randomly due to speed issues (noting that speed was not my main issue, though faster is always nicer)?

    It does seem erratic. Yesterday's version of the document took about 35 minutes to refresh (which is fine by me) and completed without error. I was streaming a movie while it ran.

    1 - Focus to word: I notice that Zotero comes to the foreground if I press the refresh button twice. I'll open Activity Monitor next time I run it.

    2 - Reinsert Biography. That is already my standard response if it goes blank after a failure, but it doesn't seem to make any difference. It seems to do another refresh if you insert a biography (?). Anyway I need the citations updated.

    1 - Split document. No can do. I have done it for testing purposes but it's a nightmare 'cos it breaks chapter numbering, page numbering, cross references, and more. I do already often prepare text in smaller documents before pasting into the main one. In particular, I usually sort out a citation with its translation and (often multiple) references in a small document, then paste it to the big one when it is right.

    2 - I don't have a Windows machine. I could perhaps try a virtual machine, but would expect that to be slower. Speed as such is not a priority for me. I could also try running it on a different MacOS machine, one that I am not using for anything else at the time.

    3 - Change citing style. Not my choice! Either MHRA or Chicago Full Note is required. I am in the final stages of the document already; and many citations are part of a footnote anyway, for example when I give the original language text of a citation I have translated into English.
  • Do I take it that you found no 'hard' errors in the logs, and that you think the errors are occurring randomly due to speed issues (noting that speed was not my main issue, though faster is always nicer)?
    Well I couldn't get any errors so they are definitely sporadic. It's hard to say what's the cause when we cannot reproduce them, but they could be performance related, e.g. due to Zotero issuing operations to Word too fast and them running out of order, although that shouldn't happen and we have only recently started seeing errors like these (you're not the only person reporting similar issues).

    I've done some further testing regarding performance and it seems that the macOS "App Nap" feature that's supposed to conserve battery life and speed might be detrimentally affecting the speed of updates. To ensure apps do not go "napping" while refreshing and slowing the process down make sure at least part of both Word and Zotero windows are visible on screen and it's best to keep Zotero on the top, since Word appears to be less affected by this.
    1 - Focus to word: I notice that Zotero comes to the foreground if I press the refresh button twice. I'll open Activity Monitor next time I run it.
    Zotero will focus if there's a Word integration process currently running and you press any of the plugin buttons in Word.
    2 - Reinsert Biography. That is already my standard response if it goes blank after a failure, but it doesn't seem to make any difference. It seems to do another refresh if you insert a biography (?). Anyway I need the citations updated.
    If you use Refresh Zotero will refresh the text (and underlying field codes) of every citation in the document. If you insert a bibliography Zotero will only touch the text of citations that need updating.
    2 - I don't have a Windows machine. I could perhaps try a virtual machine, but would expect that to be slower. Speed as such is not a priority for me. I could also try running it on a different MacOS machine, one that I am not using for anything else at the time.
    I know this sounds improbable, but for big documents Windows Word Zotero updates are 500x-1000x faster. So even if the VM is 10x slower than native Mac Word you will still have a performance increase of 50x or more. Moreover Windows Word integration is super robust because there are better APIs for Zotero to use and such sporadic errors will be a non-issue.
  • It's hard to say what's the cause when we cannot reproduce them
    Were you unable to reproduce the error in the files I sent you - those that correspond to failures in my test results? I sent them all by email in a ZIP file with the test results; and referred the email to this ReportID.
    it seems that the macOS "App Nap" feature that's supposed to conserve battery life and speed might be detrimentally affecting the speed of updates
    I am running on an iMac, so there is no battery and presumaby no app-nap. I also do not have the option to put the disk to sleep enabled. There is only a screen-saver when it is not busy.
    If you insert a bibliography Zotero will only touch the text of citations that need updating.
    That is good to know! So deleting the biography and re-inserting it instead of using refresh would be quicker? That would be enough for me most days, because I will only have changed a handful out of about 500 citations.
    for big documents Windows Word Zotero updates are 500x-1000x faster.
    Interesting! Why? I'll look into emulation possibilities...
  • I am running on an iMac, so there is no battery and presumaby no app-nap.
    App Nap exists on desktops too. You can see it in the Energy tab of Activity Monitor.
  • Were you unable to reproduce the error in the files I sent you - those that correspond to failures in my test results? I sent them all by email in a ZIP file with the test results; and referred the email to this ReportID.
    Yes, I've tested on all the documents that you have sent in and none produced an error.
    for big documents Windows Word Zotero updates are 500x-1000x faster.

    Interesting! Why? I'll look into emulation possibilities...
    If you have a spare Mac you can also try installing Windows alongside macOS with Boot Camp.
  • Thanks for the clarification.

    I am aware of boot camp, but I believe it requires reformatting the disk, and I would have to pay for a Windows licence. Might try it sometime though!
  • Microsoft these days allows installing and using Windows without a license. It simply restricts certain features, such as customizing appearance, but nothing that will prevent you from using Zotero or Word.
  • edited December 8, 2020
    You are right (of course) :-)
    1. Boot camp makes it easy to install Windows and it creates a partition without erasing the disk. It makes space by clearing iCloud files that are on the cloud anyway.
    2. While Microsoft does not admit that Windows is free, I was able to download it without a charge (.iso). During installation, it asked for a license key, but I just said I didn't have one and it carried on anyway. I chose the educational version in the hope that if they do ask for a key later, I will be able to get one from my uni.
    3. There was a hitch that Windows did not initially recognise the WiFi hardware, so could not connect to the internet. Apparently this is common. In my case it was resolved by completing the installation of Boot Camp in Windows (which fires up automatically first time in) and then restarting.

    Having got Windows going, I installed WORD and Zotero (and Zotfile). I was not able to sync with iCloud because iCloud for Windows does not apparently support the 'optimise storage' option and is unable to download everything because the drive is too small (even before I installed Windows. Nevertheless I could download my theses by hand from the iCloud web site. I did my trick of adding a dummy citation to make Zotero delete the biography, then pressed refresh. And, hey presto, you are right again: it refreshed the entire document in under 2 minutes.

    That doesn't mean I'm a convert to Windows though. (a) because I don't like it (I used it for many years in my professional life and have always favoured Mac on my private computers); (b) the inability to synchronise with iCloud is a deal-breaker. In MacOS, iCloud just keeps the most recently used documents on the local computer and downloads others as required from the cloud (diplacing older ones if necessary). That was one of the reasons I got iCloud in the first place; I have 2Tb of cloud storage, but my largest real computer has 1Tb. The laptop that now has Windows on it has 500Gb, of which I gave 128Gb (25%) to Windows.

    Thanks for the tips. Can we MacOS users lobby the Zotero developers to work on the MacOS version to make it as efficient as the Windows version??
  • You would have to lobby at Microsoft for this. I have spent a lot of time (much much more time than for any other Word processors) on the Word for Mac plugin troubleshooting bugs and optimizing it and it is still very slow (perhaps not as slow as the Google Docs one, but Google Docs is a completely online word processor) and frequently unreliable. Microsoft has a universal JS based API to integrate with all versions of Word (Windows, Mac, Online and iOS), but it is missing certain features that prevent us from reimplementing the plugin with it. Microsoft has promised to add them on their uservoice, but they have not provided a timeline for when that might arrive. Either way we don't know if the new JS based API would be any faster, but this is the most likely avenue for an improvement to the Zotero integration for Word for Mac.

    Sorry you had to take a detour via a different OS, but I'm glad you got it working.
  • OK. Seems there's nothing more you can do then. thanks for all the effort anyway.

    For anyone else reading along: Windows and iCloud are not really matters for this forum but FYI ...

    1. I should mention that Windows 10, although downloaded and installed free is not 'activated' for free. That supposedly limits some functionality (it is not clear what). There are various hacks on the net for activating for free, but I do not know if they are legal, ethical and secure (!).

    2. The failure of iCloud for Windows may be a bug or setup error. I spent an hour on the phone with Apple Support yesterday and they took it seriously but have not yet fixed it. I can syncnhronise photos but not documents. We will talk again today...


    Let's close this Zotero issue now :-)

  • Could you install the Zotero Beta, trigger the error again and submit another Report ID?
  • OK, not closed then :-)

    Test 14 in my log: I downloaded the Beta 5.0.94-beta.4+31c928a3f, then ran refresh on a copy of the same document used for all the previous tests (4 Dec version) with no text removed. It took 00:31 minutes to complete but PASSED :-) That is to say, it updated the Biography and produced no error message.

    The only hitch was that Zotero hung up immediately after submitting the debug report. I think it was sent and I managed to note the number by hand: D1920001109. Hope you got it.

    That seems to be an improvement then, thanks, though I should mention that when I refreshed yesterday's version before the Beta update that also completed OK.

    Apropos performance, I ran it with the Activity Monitor turned on, both before and after the Beta update. In both cases over 98% of CPU time was going to WORD and less than 2% to Zotero (plus 15% to Activity Monitor; don't ask me how the sums work!). Having the focus on one or the other app seems to make no difference.

    I already reported that it goes 30x quicker on Windows 10 (on the same hardware, under Boot Camp) but this is not yet very convenient. I can download/upload the document manually from iCloud.com but iCloud for Windows is proving problematic. Apple and I are still working on it ;-)
  • So the Beta version doesn't fix anything, simply adds better logging that might help us determine the cause of the error.

    As for CPU usage, it alternates over the transaction. When Zotero asks Word to fetch Zotero fields Word CPU usage will go to 100%, when Zotero then processes them CPU usage will go to Zotero, and then in the final state where Zotero updates the document it will primarily be Word again.
  • Ah. So, any news?
  • If you're able to reproduce the error, we'd like to see at the very least a Report ID, which might help us identify the cause of the problem.
Sign In or Register to comment.