Upgrade StorageFalse error report after refresh - 163149263
I am using Zotero in a 300-page WORD document with around 540 citations, so I run it with automatic refresh off. When I run a manual refresh it takes about 20 minutes and as far as I can see correctly updates the footnotes and builds the bibliography. It then says that an error occurred. It does not say what kind of error and I can see nothing wrong, but it is puzzling and has been happening consistently for a few weeks now, so I sent you a report id. 163149263. I'm happy to send the document too if you wish.
Following your instructions, I enabled debugging in Zotero, deleted the Bibliography in WORD and started ‘refresh’ without a Bibliography.
The next step involves guesswork because Zotero gives no indication of progress with refresh, or even indicates that it is running (a shortcoming I have previously mentioned and have now submitted in a separate thread as a feature request). It does not even give a message when it is finished. I usually tell that is it finished by seeing that the Bibliography has been completed.
There being no Bibliography, I waited an arbitrary 20 minutes; thinking that was about how long is usually takes. I then attempted to insert a bibliography. I was evidently too early; no bibliography appeared. Shortly after, as it happens, a pop-up dialog asking me if I want to keep changes to a footnote appeared, so apparently refresh was still running.
I answered “no”, as always, because in my case any changes to the Zotero fields can only be accidental. I waited a further 10 min and tried again with the same result. I was evidently being too optimistic about how long it takes. Maybe it is slower with debugging turned on?
I took another period of 10 minutes. The changes to the dialog popup appeared several more times (I’m not sure why there were so many today, but this does not seem to be a problem).
Then another 10 minutes – taking me up to 50 altogether.
Then another 10 minutes – taking me up to the full hour.
Then another 10 minutes – taking me up to 1 hour and ten minutes.
After about an hour and five minutes the error message appeared "An error has occurred. Do you want to see troubleshooting instructions?"
I tried (for about the third time) to insert a Bibliography but I got the message “You must insert a citation before performing this operation” (!).
This generated a whole heap of debug report (6559 lines). I hope it helps
The DebugID is: D863527690
Happy reading!
As I originally said, I am not actually seeing anything wrong with any citation or the Bibliography. I just want to know why there is an error message if there is no error!
Looking at your previously submitted log the updates are indeed slow. You might benefit from splitting your document into multiple documents for each chapter, working on them separately and then recombining at the end.
2. I think refresh ran slower than usual when error logging was on. It took over an hour and it usually takes about 20 minutes without the debug log. Anyway, I would rather leave it running for an hour than go through the palaver of splitting the document, which would be a lot more effort!
On the other hand it is likely that you will be able to reproduce the bug and be able to do faster troubleshooting on a reduced version of the doc.
Unless we can isolate this to a specific action that is repeatable and always produces the same result we cannot debug it properly. How old and what kind of Mac this is that you're working with? Retrieving references takes a really long amount of time and in the last year Mac Word speed was quite decent, although I don't know if Big Sur could have made a negative difference.
I am running on a late 2012 27-inch iMac with 3.2GHz quad core i5 processor and 24Gb of RAM. The disk is a 1 Tb fusion drive. I can see how the computer might affect performance (I am happy with it) but not how it would create the error message?
1. Opened the last version of the thesis (dated 201204, set to read-only) with macros enabled (but I did not run any macros)
2. Duplicated it and saved the file as “Zotero test n.docm” in a desktop folder.
3. Deleted from the start of the document to the location indicated in the table below. Saved again. Of course the chapter numbers and pages are all different now, but I ignored that.
4. Went to the Bibliography. This was always in the state “Automatic citation updates are disabled. To see the bibliography, click Refresh in the Zotero tab”.
5. Went to Zotero and enabled debug logging.
6. Went back to word and pressed refresh in the Zotero menu.
7. Noted the start time in the table below.
8. Waited until either the Bibliography was filled-in or an error message appeared.
9. Noted the end time in the table below.
10. In Zotero, submitted the debug log, copied the number to the clipboard and pasted it in the table below.
11. Saved the test file in its final state.
In these tests, I did not have any occurrence of the dialogue box asking if I wanted to keep changes (i.e. there were no changes; must have cleared them all yesterday).
When it appeared, the error message was always “Zotero experienced an error updating your document. Do you want to see troubleshooting instructions?”. I answered no.
These are the results.
Test Scope Start time (GMT) End time (approx.) Duration (approx.) Result Debug log
Test 1 From ch 8 to end 21:10 21:27 0:17 PASS. No error message. Bibliography inserted. D1323491916
Test 2 From ch 6 to end 22:04 22:11 0:07 PASS. No error message. Bibliography inserted. D1981162491
Test 3 From ch 2 to end 22:17 22:30 0:13 FAIL. Error message and Biography blank. D831396530
Test 4 From ch 4 to end 22:35 22:42 0:07 FAIL. Error message and Biography blank. D1784676899
Test 5 From ch 5 to end 22:46 22:50 0:04 FAIL. Error message. Bibliography still says “Automatic citation updates are disabled. To see the bibliography, click Refresh in the Zotero tab.” D750226190
Test 6 From about ½ chapter 5 22:56 22:03 0:07 FAIL. Error message and Biography blank. D249557457
Test 7 From about ¾ chapter 5 23:10 23:13 0:03 PASS. No error message. Bibliography inserted. D1822112187
Test 8 From about 5/8 of chapter 5 23:21 23:31 0:10 FAIL. Error message and Biography blank. D272329732
I have mailed you a zip file with a better formatted version of this report (!) and all 8 test files referencing the original report id 163149263
In test 9 I copied and pasted all of chapter 5 into a new document. I did the refresh and this PASSED. Debug ID D1679714153.
In test 10, I selected the same text, but by taking a copy of the whole original document then deleting the parts before chapter 5 and chapter 6 to the conclusion, leaving in the list of figures and the index as I had in test 1 to 8. This test also passed! Debug ID D399389155.
Sadly this confirms my previous experience that simple binary division of the document does not lead one to the error. It would not appear to be any particular citation that is 'causing' the problem. Neither would it appear to be related to the presence of macros, the list of figures and the Index that was generated by WORD.
I am at a loss where to look now, but I have kept a copy of the 'test' version of the thesis and the test documents (after refresh) in case you want me to run any further tests.
The updates on my Mac took quite a substantial amount of time too. But after monitoring how it went for a bit I can offer you these two pieces of advice:
1. Switch focus to Word and then Zotero when performing Word processor operations, but especially refreshing. It seems that macOS is giving very little CPU time to Zotero if it is in the background, making the citation processing Zotero needs to do extremely slow. We'll investigate it and see how we can improve the situation in this regard, but for now you should make sure Zotero is in the foreground.
2. If the bibliography update fails, remove the bibliography field from the document and use the Add/Edit Bibliography option. This will prevent the Zotero from updating every field in the document, which should make the bibliography insertion/update operation faster.
The other advice still stands:
1. Split your document into multiple chapters/sections and work on them individually and recombine at the end.
2. If you can, use a Windows machine (or even a Windows virtual machine on macOS) to refresh the document, which will make it much faster and error-proof.
3. If you can, switch to an in-text citing style (like APA or Chicago author-date) while working on the document. Note styles/citations in notes are unfortunately yet another thing that makes citing slower. You can switch back to the required citation style at the end of writing. Note that if you need to add some extra text in a footnote next to a citation, you can still insert a footnote manually, add the comment and then the citation. Vice-versa, when converting to an in-text style, if there are comments in a footnote next to a citation, Zotero will not convert the citation to an in-text one.
Thanks for the detailed debugging and I'm sorry we cannot offer anything more substantial for these errors at the time.
It does seem erratic. Yesterday's version of the document took about 35 minutes to refresh (which is fine by me) and completed without error. I was streaming a movie while it ran.
1 - Focus to word: I notice that Zotero comes to the foreground if I press the refresh button twice. I'll open Activity Monitor next time I run it.
2 - Reinsert Biography. That is already my standard response if it goes blank after a failure, but it doesn't seem to make any difference. It seems to do another refresh if you insert a biography (?). Anyway I need the citations updated.
1 - Split document. No can do. I have done it for testing purposes but it's a nightmare 'cos it breaks chapter numbering, page numbering, cross references, and more. I do already often prepare text in smaller documents before pasting into the main one. In particular, I usually sort out a citation with its translation and (often multiple) references in a small document, then paste it to the big one when it is right.
2 - I don't have a Windows machine. I could perhaps try a virtual machine, but would expect that to be slower. Speed as such is not a priority for me. I could also try running it on a different MacOS machine, one that I am not using for anything else at the time.
3 - Change citing style. Not my choice! Either MHRA or Chicago Full Note is required. I am in the final stages of the document already; and many citations are part of a footnote anyway, for example when I give the original language text of a citation I have translated into English.
I've done some further testing regarding performance and it seems that the macOS "App Nap" feature that's supposed to conserve battery life and speed might be detrimentally affecting the speed of updates. To ensure apps do not go "napping" while refreshing and slowing the process down make sure at least part of both Word and Zotero windows are visible on screen and it's best to keep Zotero on the top, since Word appears to be less affected by this. Zotero will focus if there's a Word integration process currently running and you press any of the plugin buttons in Word. If you use Refresh Zotero will refresh the text (and underlying field codes) of every citation in the document. If you insert a bibliography Zotero will only touch the text of citations that need updating. I know this sounds improbable, but for big documents Windows Word Zotero updates are 500x-1000x faster. So even if the VM is 10x slower than native Mac Word you will still have a performance increase of 50x or more. Moreover Windows Word integration is super robust because there are better APIs for Zotero to use and such sporadic errors will be a non-issue.
I am aware of boot camp, but I believe it requires reformatting the disk, and I would have to pay for a Windows licence. Might try it sometime though!
1. Boot camp makes it easy to install Windows and it creates a partition without erasing the disk. It makes space by clearing iCloud files that are on the cloud anyway.
2. While Microsoft does not admit that Windows is free, I was able to download it without a charge (.iso). During installation, it asked for a license key, but I just said I didn't have one and it carried on anyway. I chose the educational version in the hope that if they do ask for a key later, I will be able to get one from my uni.
3. There was a hitch that Windows did not initially recognise the WiFi hardware, so could not connect to the internet. Apparently this is common. In my case it was resolved by completing the installation of Boot Camp in Windows (which fires up automatically first time in) and then restarting.
Having got Windows going, I installed WORD and Zotero (and Zotfile). I was not able to sync with iCloud because iCloud for Windows does not apparently support the 'optimise storage' option and is unable to download everything because the drive is too small (even before I installed Windows. Nevertheless I could download my theses by hand from the iCloud web site. I did my trick of adding a dummy citation to make Zotero delete the biography, then pressed refresh. And, hey presto, you are right again: it refreshed the entire document in under 2 minutes.
That doesn't mean I'm a convert to Windows though. (a) because I don't like it (I used it for many years in my professional life and have always favoured Mac on my private computers); (b) the inability to synchronise with iCloud is a deal-breaker. In MacOS, iCloud just keeps the most recently used documents on the local computer and downloads others as required from the cloud (diplacing older ones if necessary). That was one of the reasons I got iCloud in the first place; I have 2Tb of cloud storage, but my largest real computer has 1Tb. The laptop that now has Windows on it has 500Gb, of which I gave 128Gb (25%) to Windows.
Thanks for the tips. Can we MacOS users lobby the Zotero developers to work on the MacOS version to make it as efficient as the Windows version??
Sorry you had to take a detour via a different OS, but I'm glad you got it working.
For anyone else reading along: Windows and iCloud are not really matters for this forum but FYI ...
1. I should mention that Windows 10, although downloaded and installed free is not 'activated' for free. That supposedly limits some functionality (it is not clear what). There are various hacks on the net for activating for free, but I do not know if they are legal, ethical and secure (!).
2. The failure of iCloud for Windows may be a bug or setup error. I spent an hour on the phone with Apple Support yesterday and they took it seriously but have not yet fixed it. I can syncnhronise photos but not documents. We will talk again today...
Let's close this Zotero issue now :-)
Test 14 in my log: I downloaded the Beta 5.0.94-beta.4+31c928a3f, then ran refresh on a copy of the same document used for all the previous tests (4 Dec version) with no text removed. It took 00:31 minutes to complete but PASSED :-) That is to say, it updated the Biography and produced no error message.
The only hitch was that Zotero hung up immediately after submitting the debug report. I think it was sent and I managed to note the number by hand: D1920001109. Hope you got it.
That seems to be an improvement then, thanks, though I should mention that when I refreshed yesterday's version before the Beta update that also completed OK.
Apropos performance, I ran it with the Activity Monitor turned on, both before and after the Beta update. In both cases over 98% of CPU time was going to WORD and less than 2% to Zotero (plus 15% to Activity Monitor; don't ask me how the sums work!). Having the focus on one or the other app seems to make no difference.
I already reported that it goes 30x quicker on Windows 10 (on the same hardware, under Boot Camp) but this is not yet very convenient. I can download/upload the document manually from iCloud.com but iCloud for Windows is proving problematic. Apple and I are still working on it ;-)
As for CPU usage, it alternates over the transaction. When Zotero asks Word to fetch Zotero fields Word CPU usage will go to 100%, when Zotero then processes them CPU usage will go to Zotero, and then in the final state where Zotero updates the document it will primarily be Word again.