Web snapshots no longer wait for pages to load subresources

jryans

One of the page types I am often capturing with web snapshots is Twitter threads. The initial release of the SingleFile-based snapshots was a huge improvement here, as it actually resulted in a readable page.

Recent connector builds seem to have regressed this functionality, producing snapshots of Twitter threads with various issues, e.g. all images and avatars missing, or even worse, no content at all and just a loading spinner.

As a specific example, I tried capturing https://twitter.com/girba/status/1465822051378384906 in several browsers, each failing in various ways: Firefox Nightly 96 (images missing, CSS seems partial) and Safari 15.0 (no content, loading spinner).

macOS 11.6.1
Zotero 5.0.97-beta.57+07df7d0de
Firefox Connector 5.0.92beta10
Safari Connector 5.0.92

adomasven

The Safari Connector never included SingleFile saving due to technical reasons and won't for the foreseeable future.

The current connector Beta is testing updates SingleFile code due to issues in saving on ScienceDirect and possibly other websites, and 5.0.92beta9 was definitely very broken on Firefox, however the current version should be fine.

I am not seeing any issues with the latest Firefox Connector beta on the page you've posted. Did you wait for the page to fully load before saving with the connector? If you install the non-beta Connector for Firefox does it save well?

jryans

Thanks for taking a look @adomasven!

The Safari Connector never included SingleFile saving due to technical reasons and won't for the foreseeable future.

Hmm, I see... but when I save a web snapshot in Safari, the file stored in Zotero is a single HTML file and says "Page saved with SingleFile" at the top of the file's source, which matches the source comment in snapshots taken with Firefox as well, so that led me to assume both browsers had access to SingleFile...

I am not seeing any issues with the latest Firefox Connector beta on the page you've posted. Did you wait for the page to fully load before saving with the connector? If you install the non-beta Connector for Firefox does it save well?

Yes, in all cases I waited for page to fully load in settle before capturing a snapshot.

Reverting to the release Firefox Connector (5.0.91) does indeed fix the issue, yes.

With the current beta (5.0.92beta10), I get a ~300 KB snapshot which is missing most avatars, images, etc. With release (5.0.91), I get a 3.8 MB snapshot that includes all images, etc.

Here's a screenshot of the bad version: https://user-images.githubusercontent.com/279572/144415070-d9ffa86a-95ab-46fb-92ba-95c72ccfad4f.png

I just submitted debug output when capturing with the beta connector, debug ID is D1256370357. There were no errors shown in the connector, so I did not submit a regular error report.

Please let me know if any other details would be helpful.

jryans

Aha, I've learned a few more things...

The missing images behaviour with the beta connector only happens in Firefox Nightly 96. If I use Firefox Release 94, then the beta connector works well just like the release connector does.

Also, if create a clean profile in Firefox Nightly 96, the beta connector works well there also.

So, something about my default Firefox profile in Nightly 96 and the beta connector don't play well together... I'll try to see what it might be.

dstillman

Hmm, I see... but when I save a web snapshot in Safari, the file stored in Zotero is a single HTML file and says "Page saved with SingleFile" at the top of the file's source, which matches the source comment in snapshots taken with Firefox as well, so that led me to assume both browsers had access to SingleFile...

They're both SingleFile, but Safari is SingleFile in the Zotero app, which wouldn't necessarily reflect the current state of the page in the browser. (The old system always involved doing the snapshot in the app. We swapped out the old Mozilla method for SingleFile so Safari can still get single-file snapshots, but it's worse than doing it in the browser.)