Error while refreshing a large document

AndySymons

Report no: 542758186

I get an error when refreshing a document. I have tried steps 1-9 of the Troubleshooting instructions (as far as applicable). Step 10 is impractical as I have several hundred citations. (When I tried this in the past I got no conclusive result). I would therefore appreciate an indication of which citation is at fault. As it is a large document (a PhD thesis) I have automatic update turned off, but refresh the citations and bibliogrpahy at the end of each day. It takes up to an hour.

Thanks.

dstillman

Step 10 is impractical as I have several hundred citations.

This is a misunderstanding. The whole point of that step is that you can very quickly cut down a document, even with thousands of citations, in no more than 10 tries.

I have automatic update turned off, but refresh the citations and bibliogrpahy at the end of each day.

Note that there's no particular reason to refresh every day. You don't even need to keep a bibliography in your document — unless you're adding uncited items, there's no difference whatsoever between a bibliography you insert at the beginning of writing and one you insert at the end at the end. You should periodically try refreshing just to make sure it completes successfully, so that if there's a problem (as there seems to be) you're aware of it long enough before your deadline to troubleshoot, but there's no advantage to having it run for an hour every day.

AndySymons

'Periodically refreshing' is precisely what I meant by doing it every day - I do it to check it still works and to correctly complete the references. There is nothing special about the bibliography, it is just the one that is automatically generated on a refresh.

I am disappointed that you do not have a better solution than step 10. This is extremely time consuming as each refresh takes about an hour ... to that is up to 10 hours to find the problem!

dstillman

that is up to 10 hours to find the problem!

No, it's really not. On the very first step you're cutting the document in half. Most of the steps will be with a tiny fraction of the document. It's an extremely efficient way to find a problematic citation even in a huge document.

AndySymons

I reluctantly followed your advice, but did not solve the problem. For the other readers, I want to make clear how tedious step 10 is …

Since I do daily refreshes I was able to narrow my search by selecting the chapter I had last been working on, copying that to a new document and running the refresh. I confirmed the error was there.

I then separated out the four sections in the chapter. Three sections were OK; the problem was in the third. This consisted of 10 pages.

It is already getting complicated now because of course the page, section and footnote numbers are all different in the copies, so I have to note the page range from the original document section in the file name so I know where I am.

Of the 10 pages, I found the problem was in pages 6-10 (15 footnotes). I tried splitting pages 6-8 from 9-10 but taken individually, neither of these had any error!

I then tried taking each of the pages 6 to 10 individually from the original extract. Again none of them displayed any error.

I confirmed that the document section with all pages 6-10 still had the error. Next, I tried deleting the footnotes one at a time starting at the end. When I got to 13 the error went away. I then checked by deleting only footnote 13 from the page 6-10 extract (leaving 14 and 15 in). I confirmed then the error had gone away.

It seemed I had solved the problem, so I then went back to the original document and deleted the same footnote (by searching, it is #318 in that document). I refreshed and … the problem is still there.

See report 1915071296.

All this took an hour.

What do you suggest I do next?

dstillman

The goal of that step is to isolate the smallest possible section of the document where the problem occurs. It sounds like you did that fairly quickly (though your process sounds a bit more circuitous than necessary — e.g., you tried four sections individually instead of just halving).

If you can reproduce it reliably in a small section, then as the page says, you should send it to us at support@zotero.org with a link to this thread so that we can investigate and hopefully fix the underlying problem. You can remove any text you don't want to include as long as you can still reproduce the problem.

'Periodically refreshing' is precisely what I meant by doing it every day - I do it to check it still works and to correctly complete the references.

It it makes you more comfortable, that's fine, but my point was just that doing it every day isn't particularly necessary — it just means your computer has to churn for a long time every day for no real reason. Even if you encounter a problem, there's no appreciable difference in the amount of time it takes to debug a 50 page document or a 100 page document. That's the benefit of the halving process.

dstillman

@AndySymons: And just to explain this further, while we won't know what the exact problem is until you share a document excerpt, for many errors, a citation in an unexpected state — e.g., because of a Word bug that corrupts a field, or Track Changes gone awry — will cause a problem during bibliography generation, or at some other stage of the refresh process, which inherently involves the whole document. So Zotero itself often can't identify the actual citation that caused a later problem in order to show it to you. That's why Step 10 is often necessary.

AndySymons

It would still be very helpful if Zotero identified the citation at which the fault is found.

bwiernik

Like Dan said, “So Zotero itself often can't identify the actual citation that caused a later problem in order to show it to you.”

AndySymons

I sent you the portion of the text that seems to reliably display the fault. Further halving produced ambiguous results.

adomasven

Thanks for sending in the snippet. Could you also send in the citation style you are using, since it doesn't seem to be one of the default styles from Zotero style repositories?

In general, Zotero is certainly not happy with the index entries that are placed throughout the document (which you can see by enabling display of formatting marks under Home -> ¶), especially inside citations. It also seems that you have modified quite a few citations in the document manually. You should generally avoid that and fix the cited item metadata in Zotero instead (and then refresh the document), or modify the citation style.

Something is also definitely broken with the Zotero Word for Mac plugin, so that one's on us, although I haven't identified what's causing the error yet, and it might take a while to figure out and fix. In the meantime if you have access to a Windows computer the document (or at least the snippet you sent us) will refresh fine there, so that's the most immediate workaround. We'll post here once we have a fix for the Word for Mac plugin bugs. Sorry for the trouble!

adomasven

@AndySymons Download the Zotero Beta and see whether you can update your document (or at least that broken snippet to start) with it.

AndySymons

1. I am using *MHRA 3* style, which is a standard in Zotero.
2. The index entries are added automatically, using a concordance file, so I do not have control of where they are. I will however try removing them all by search-and-replace.
3. There are *accidental* edits to a few (I counted three) citations, but I always select 'no' to remove these when they are found.
4. Sorry I do not have access to a Windows computer at present. I look forward to a fix for the Mac plug-in.
5. I'll try Zotero Beta. Does that have an updatd Mac plug-in?

AndySymons

Good news! The index entries seem to be the main problem. I did not know any were there, but it appears I inadvertently pasted in some old entries from another file.

FYI, all index entries in a document can be removed as follows (after backing up the document):
1. press the ¶ button to reveal all field codes
2. type "^d XE" (not including the quotation marks) in the search box
3. leave the replace box blank
4. click 'replace all'
5. press the ¶ button to hide field codes

After doing that, my whole document refreshed OK without needing to go to Zotero Beta.

It would be good if you could fix Zotero to ignore all WORD fields!
In the meantime I guess the workaround is to run the indexing routine only AFTER refreshing Zotero, then remove the entries again (as above) before doing any further work on the document.

adomasven

Ideally you should only be running indexing after finalizing the document, at least in terms of Zotero citations, as whenever Zotero will need to update your citations for whatever reason (whether it's disambiguation, changed metadata in your Zotero database or other) all index marks in them will get lost.

On the other hand Zotero should not fail to update the document at all just because there are index entries, even if they are inside Zotero citations. It would be very useful if you could download the Zotero Beta and see whether you can refresh the document with index entries present, since it will give us useful feedback that other users won't run into such issues again.

AndySymons

Yes, there are several reasons to run indexing as the last step, and it is anyway wise to delete all entries before running automark. I have set up my document to use automark only.

But yes, having the index fields present should not cause an error. When I have a little time, I will try running Zotero Beta to see if the problem is solved.

AndySymons

I tried the beta version 89 on my whole document (241 pages, 350 citations, 717 index entries) and found that I am now able to refresh AFTER creating an index – in other words while all the index entries are in place. The problem seems to be fixed ... mostly. I did notice that I got an error after the bibliography had refreshed, which is a bit odd. It is possible that I pressed 'refresh' more than once, since it is hard to tell when it has been pressed. You should grey it out and have a progress bar while refreshing. The only way I know to see if refreshing has finished is to look at the bibliography and see that it has been filled. Seems to be an great improvement anyway. Thanks for your help.

adomasven

Excellent! If you're able to cut down the document significantly (without removing the bibliography) and reproduce the final refresh error, we would also like to see that snippet (support@zotero.org). Otherwise thanks for helping with debugging this, we'll be including the fix in the next version of Zotero.