[Bug Report] Handle HTTP reponse code 429 (Google Docs & Zotero)

popovic

Tested on Gentoo Linux, Zotero version 6 and 7 beta, with both stable and beta version of Zotero connector for Firefox
Debug ID: D624117995

In large documents, Zotero easily riches the Google Docs quota limitation. This is correctly reported from the Google side by http response code 429. And Google suggests how to handle this error here:
https://developers.google.com/drive/api/guides/handle-errors
https://developers.google.com/drive/api/guides/limits#exponential

Searching the forum, I see this as quite common error affecting a lot of users, but with easy fix. Zotero should simply retry the POST request to Google Docs after 1,2,3... + random milliseconds. Is there a chance to fix this upstream? I am willing to test it.

dstillman

We've added proper 429 handling in the latest Zotero Connector beta (5.0.124beta1) for Firefox. Let us know if that works better for you.

popovic

@dstillman thank you for a quick solution!!! It seems to work now, but I don’t get the 429 response in the debug log. Maybe to include a message to the user about the quota to know that the update will be slower after receiving code 429?

PS. I still get some errors, but at least it is working for the most parts:

(5)(+0000018): "{\"error\":\"Google Docs ScriptError\",\"message\":\"Exceeded maximum execution time\",\"stack\":\"Zotero.Messaging</this.init/Zotero[ns][meth]</</<@moz-extension://3cce25bb-c58d-44aa-9760-de2a8ee9a1db/messaging_inject.js:79:17\\n\"}"

(3)(+0000032): Error: HTTP Integration Error init@chrome://zotero/content/xpcom/connector/server_connectorIntegration.js:67:19 Zotero.Server.DataListener.prototype._processEndpoint<@chrome://zotero/content/xpcom/server.js:548:16 tryCatcher@resource://zotero/bluebird/util.js:16:23 module.exports/PromiseSpawn.prototype._promiseFulfilled@resource://zotero/bluebird/generators.js:97:49 module.exports/Promise.coroutine/<@resource://zotero/bluebird/generators.js:201:15 Zotero.Server.DataListener.prototype._bodyData@chrome://zotero/content/xpcom/server.js:338:10 Zotero.Server.DataListener.prototype._headerFinished@chrome://zotero/content/xpcom/server.js:313:10 Zotero.Server.DataListener.prototype.onDataAvailable@chrome://zotero/content/xpcom/server.js:207:12 From previous event: captureStackTrace@resource://zotero/bluebird/debuggability.js:915:23 CapturedTrace@resource://zotero/bluebird/debuggability.js:807:22 longStackTracesCaptureStackTrace@resource://zotero/bluebird/debuggability.js:482:19 module.exports/Promise.prototype._then@resource://zotero/bluebird/promise.js:261:17 module.exports/Promise.prototype.then@resource://zotero/bluebird/promise.js:154:17 sendCommand@chrome://zotero/content/xpcom/connector/httpIntegrationClient.js:48:55 Zotero.HTTPIntegrationClient.Document.prototype[method]@chrome://zotero/content/xpcom/connector/httpIntegrationClient.js:91:41 Zotero.Integration</this.execCommand@chrome://zotero/content/xpcom/integration.js:289:22

adomasven

Out of interest, how big is the document you are working with, and how many citations do you have? It seems like perhaps Google has recently changed something about its quotas, since this was rarely an issue over the past few years, but we've had a few reports recently. Also you're getting a maximum execution time error, which is different, and would only occur on a huge document unless Google changed something.

As for logging, if you go into Zotero Connector preferences -> Advanced and tick Enable Logging, the 429 responses will be logged. You should not keep that checkbox ticked for long though as it may cause instability in the extension and browser.

popovic

@adomasven The document is huge book with 450 pages and a lot of citations, thus I did expect some problems. However, I have tested it before switching from LibreOffice to collaboration in Google Docs and it used to work fine a few months ago (slow but running). So, I guess Google did change something in the mean time.

Thanks for the hint on debugging.

dstillman

Yeah, for what it's worth, we wouldn't recommend working on a file that big in Google Docs (maybe in general, but certainly with Zotero) — there's just too much that's outside of our (and your) control.

We've built a much faster version of Google Docs integration that might handle very large documents better, but we haven't been able to release it for a couple years due to a bug in Google Docs that Google hasn't fixed.

popovic

Google is extremely slow fixing stuff that are not of their prime concern, I had similar experience with localization bugs. Fingers crossed to resolve this issue, it would be great to see better Docs integration.

And thank you both for the help with Zotero, I managed to fix the document for now and the next time we should definitely split it by chapters.