Available for beta testing: Much faster Google Docs integration

edited June 22, 2022
Update: This is now available for testing in all browsers.

In the latest Zotero Connector beta, available for Firefox, we've switched to a new method of integrating with Google Docs that should dramatically speed up Zotero operations in large Google Docs documents. This brings Google Docs performance in line with Zotero integration in Word for Windows.

If you're using Firefox, you can try it simply by installing the Zotero Connector beta from the above page and reloading Google Docs.

If you run into any problems, let us know in a new thread.

You can temporarily disable the new mode by disabling "Enable Google Docs Integration v2 (BETA)" from the Advanced pane of the Zotero Connector preferences or by switching back to the release version of the Zotero Connector.
  • Can you describe a bit the change in technology? (To help with support questions)
  • edited May 25, 2022
    When we initially developed the Google Docs integration the only way to edit and modify Google documents was via Google's Apps Script which is also the preferred Google way for developing Docs addons by adding a sidebar interface. The native Apps Script extensions however are limited to that sidebar UI and wouldn't allow us to interface with the native Zotero client running on the user's system, which is why we created a hybrid Google Docs extension, part of which lives in the Zotero Connector and part of which is deployed to the Google Docs Apps Script environment as a "standalone script".

    The current extension (approximately) works as follows:
    1. The Zotero Google Docs extension UI is managed by the Zotero Connector.
    2. Citation processing and the citing UI is managed by the Zotero client.
    3. Reading the document for citation processing and then editing its contents is done via our standalone script running on Apps Script.

    This update changes the part 3 of the above. In 2019 Google introduced an alternative way to interface with Google Docs - the Google Docs API. Apps Script is JS code that you deploy to Google's Servers and that's where that code runs whenever you need to interface with any document. The Google Docs API allows you to retrieve the full contents of the document via a HTTP GET request as JSON object and modify its content via HTTP POST requests.

    The main reason Apps Script is slow is that each time it runs, Google needs to initialize a server with our Apps Script code, then fetch the document and preprocess it so that it can be read with the Apps Script programming API even if we only need to do something very simple. Even when we did our best to limit the number of times we need to call the Apps Script code (which is about 3-5 times for a citation insert), for a large document (think 100+ pages) just spinning up the Apps Script instance on Google's servers takes 5s+. If we needed to update all citations in a large document (say for changing the citation style), we couldn't do it in a single Apps Script call as the script would timeout after 60s of running, so we had to batch those updates, increasing the Apps Script environment overhead even more.

    With the Google Docs API this is much simplified. Document reading and processing code is folded directly into the Connector. We only need to retrieve the document content JSON object once, we then batch as many updates as we need and submit it as a single call to the Google Docs API (i'm simplifying a bit, sometimes we need to refetch the document or submit multiple update calls, but this is the gist). The API itself is also much more performant than Apps Script, adding to the speed benefit.
  • Okay, cool, thanks!
  • Very quick first impression:
    - It is indeed way faster and much more fun to use
    - IIRC the old version prevented me from inserting citations while in suggest mode. I tried to insert a citation in suggest mode and it let me do this but then threw an "Error updating Document" (Report ID: 2132666965). I assume this still isn't possible to do, so should still be disabled (though it'd be great if it were possible -- as you probably know, many collaborative projects work in suggest/track changes a lot)
  • edited June 1, 2022
    I tried to insert a citation in suggest mode and it let me do this
    @adamsmith: Thanks — should be fixed in the latest connector beta.
  • Indeed it is much faster than before but still a bit slower than native LibreOffice integration IMHO.

    Just when we switched away from Google Docs due to it being too slow :(
  • We've released this in Zotero Connector 5.0.95 for Firefox. Barring any problems, we'll release it for other browsers next week.
  • This is now available in Zotero Connector 5.0.95 for Edge. Chrome and Safari still to come.
  • We're leaving this disabled by default in release builds for now, but the new method can now be tested in Chrome, Firefox, and Edge. Make sure you're running Zotero Connector 5.0.96, enable "Enable Google Docs Integration v2 (NEW)" from the Advanced pane of the connector preferences, and then reload any open Google Docs tabs.

    We'd particularly appreciate testing with large documents containing many citations, where you should see much faster performance. If you encounter any errors, please start a new thread and include a Debug ID from the Zotero Connector for reproducing the problem.
  • Hi and thank you for this, it sounds great!
    Is it OK to try the beta connector while other collaborators are still on the release version, or should everyone use the same version of the connector? Just to be sure. - Thanks
  • @eduvelle: It's fine to use different versions. The changes to the document are the same.
  • @dstillman thank you!
    For information, it's working very well so far - definitely faster than the previous version - and I have about 150 references. If it was faster, it would be perfect, but it looks like that's not possible.
    Small thing, sometimes I can't see the Zotero menu anymore (even if I'm in Edit mode) and in these cases I have to close Firefox & Zotero and restart both (maybe in a specific order? not sure) and it always ends up working. This may not be specific to this latest update though.
    In any case: Thank you!
  • Very impressive work there Sir Zotero! I'm about to test all this amazing software on a docx that started out as a pure Google Doc, but then got too big to handle quickly (168 pages, 38,000 words, and 54 "footnote" citations in the text). Sounds like a regular size for when people switch over to heavy duty db software. I also took it offline - back into originally a .odt and then I settled on .docx on my local machine - due to the need to add a lot of imagery. In fact the file is now 54 MB! So I think I will try the LibreOffice export stylesheet method, and if that fails, re-upload it to google docs, and run this web based method as backup plan.
  • Just to give an update on this, there's unfortunately a Google Docs bug that results in errors in some documents when using the new integration method. We reported it to Google a few months ago but are still waiting for a fix from them. Until then, we're leaving the v2 integration disabled by default.

    You can still enable the new method from the preferences if you want the faster performance — just be aware that you might hit an error that will require switching back to the old method.
  • At the top of the thread, is says available on all browsers but I do not see where the download is for chrome. It just sends you to the bottom of the document
  • There's no separate download. It's part of the standard Chrome connector, you just have to enable it in the connector or preferences as described
  • @anpotter74 The steps to enable the new Google doc integration are:
    - right click the Zotero connector on your browser and click "preference" (I am using Chinese version so not sure if I translate that correctly);
    - You'll see "Enable Google Docs Integration v2 (NEW)" under "Advanced" > "Google Docs Integration".
  • We don't currently recommend using Google Docs Integration v2 due to a bug in Google Docs that causes integration to break with some documents. We reported the bug to Google last year but are still waiting for a fix. We'll be disabling v2 in an upcoming update.
Sign In or Register to comment.