Upgrade Storage[10.0.1] Runaway memory in document-worker during full-text indexing of a Chinese-language PDF
Zotero 10.0.1 (x64), Windows 11 26200, locale zh-CN. All plugins disabled. Library has roughly 7,400 indexable items.
Since upgrading to Zotero 10, memory climbs to 18–27 GB within a few minutes of every startup and one CPU core stays pinned at 100%. Disk and network activity are both at zero while this happens, and "Minimize memory usage" in about:memory has no effect at all.
Report ID: 765642344
Debug ID: D937242720
--- about:memory ---
A Measure run attributes 14,988 MB of 15,871 MB explicit (94%) to a single worker:
worker(resource://zotero/document-worker/worker.js)
10,857 MB class(Array)/objects/gc-buffers/elements/normal
2,140 MB string(length=1, copies=7235, "/")/gc-heap/two-byte
1,335 MB string(length=1, copies=7234, "/")/gc-heap/two-byte
So one array holds nearly 11 GB, and roughly 14,500 copies of the single character "/" account for another 3.4 GB.
Reader tabs are not involved: window-objects totals only 209 MB, and 7 of my 9 tabs are in the reader-unloaded state.
--- Debug output ---
I narrowed it to full-text indexing of one specific attachment. The log ends like this:
Extracting full-text content for 4 attachments
Indexing item 1/34EKB4TZ
Getting fulltext content from item 1/34EKB4TZ
HTTP GET resource://zotero/reader/pdf/web/cmaps/GBK-EUC-H.bcmap
HTTP GET resource://zotero/reader/pdf/web/cmaps/Adobe-GB1-UCS2.bcmap
HTTP GET resource://zotero/reader/pdf/web/standard_fonts/LiberationSans-Regular.ttf
(nothing further — memory then grows without bound)
The item is a Chinese-language book PDF using the Adobe-GB1 character collection. Nothing at all is logged after the CMaps load.
--- Workaround ---
Setting extensions.zotero.fulltext.pdfMaxPages to 0 and restarting stops the problem completely, which is consistent with the runaway being in the text-extraction path rather than anywhere else.
Given that 10.0 reworked full-text indexing and added CJK phrase matching, I suspect a regression in CJK text extraction in document-worker.
--- Files ---
I'm emailing the saved about:memory report (.json.gz) to support@zotero.org with a link to this thread, since it can't be attached here.
I'd rather not post the PDF publicly since it's a copyrighted book, but I'm glad to send it privately, or to cut it down to a minimal reproducing file if that's more useful. Just let me know which you'd prefer.
Since upgrading to Zotero 10, memory climbs to 18–27 GB within a few minutes of every startup and one CPU core stays pinned at 100%. Disk and network activity are both at zero while this happens, and "Minimize memory usage" in about:memory has no effect at all.
Report ID: 765642344
Debug ID: D937242720
--- about:memory ---
A Measure run attributes 14,988 MB of 15,871 MB explicit (94%) to a single worker:
worker(resource://zotero/document-worker/worker.js)
10,857 MB class(Array)/objects/gc-buffers/elements/normal
2,140 MB string(length=1, copies=7235, "/")/gc-heap/two-byte
1,335 MB string(length=1, copies=7234, "/")/gc-heap/two-byte
So one array holds nearly 11 GB, and roughly 14,500 copies of the single character "/" account for another 3.4 GB.
Reader tabs are not involved: window-objects totals only 209 MB, and 7 of my 9 tabs are in the reader-unloaded state.
--- Debug output ---
I narrowed it to full-text indexing of one specific attachment. The log ends like this:
Extracting full-text content for 4 attachments
Indexing item 1/34EKB4TZ
Getting fulltext content from item 1/34EKB4TZ
HTTP GET resource://zotero/reader/pdf/web/cmaps/GBK-EUC-H.bcmap
HTTP GET resource://zotero/reader/pdf/web/cmaps/Adobe-GB1-UCS2.bcmap
HTTP GET resource://zotero/reader/pdf/web/standard_fonts/LiberationSans-Regular.ttf
(nothing further — memory then grows without bound)
The item is a Chinese-language book PDF using the Adobe-GB1 character collection. Nothing at all is logged after the CMaps load.
--- Workaround ---
Setting extensions.zotero.fulltext.pdfMaxPages to 0 and restarting stops the problem completely, which is consistent with the runaway being in the text-extraction path rather than anywhere else.
Given that 10.0 reworked full-text indexing and added CJK phrase matching, I suspect a regression in CJK text extraction in document-worker.
--- Files ---
I'm emailing the saved about:memory report (.json.gz) to support@zotero.org with a link to this thread, since it can't be attached here.
I'd rather not post the PDF publicly since it's a copyrighted book, but I'm glad to send it privately, or to cut it down to a minimal reproducing file if that's more useful. Just let me know which you'd prefer.