Clear Index

wenhsuNYC
Hi fellow zoteros: I've got about 50,000 entries in Zotero, mostly with PDF attachments. About 10,000 got indexed. Then indexing stopped (no matter how long I wait). Clearing Index does not work. Error message: "Error: Transaction timeout, most likely caused by unresolved pending work." This has been going on for quite some time (years!) now, meaning the issue did not get fixed over many release cycles. Settings: Maximum characters …: 50,000,000; Maximum pages …: 100,000; — see attached screenshots. Ideas? Any help very much appreciated. Thanx, Wen.
https://s3.amazonaws.com/zotero.org/images/forums/u15782064/ixx5239b5rjt3rgzkkms.png
https://s3.amazonaws.com/zotero.org/images/forums/u15782064/6rqt58ytsctxc6lo58ej.png
https://s3.amazonaws.com/zotero.org/images/forums/u15782064/ll9ce81jx3w6pyklqebl.png
  • dstillman
    edited November 20, 2024
    You may have made the index unworkably big by increasing those numbers from the defaults.

    Why exactly are you trying to clear the index?

    Have you tried running it and then not performing any other tasks in Zotero (including by temporarily disabling auto-sync)?

    If so, can you provide a Debug ID for reproducing this?
  • wenhsuNYC
    Hi dstillman, thanks for looking into this.

    ad 1): 100 pages is not enough for an index of the entire book, given that I have books in Zotero with more than 1,000 pages. I've no idea about the number of characters in a book. —Should I consider the Default values to be 'upper limits'?

    ad 2) I try to clear / delete the index in order to be able to create a new index that will actually index ALL documents in the library. Indexing stopped at roughly 10,000 documents and so far I failed to 'restart' indexing … in order to get the remaining files indexed. The error message indicates that there is some unfinished business, but that's been the case for years now (I was hoping that some update will eventually fix the issue but now got impatient).

    ad 3) Auto-sync has been permanently disabled for many years — it can't be the source of the problem. I think I have tried everything many times, including looking at the Activity Monitor on MacOS in order to see if Zotero is actually doing something. A change has appeared over time: In the 'early days' of the problem, I would not get an error message; Zotero would just stop indexing eventually. I think the error message started to pop up sometime last year. —I'm on beta release and usually update in a timely fashion. There should not be any limitations on the side of the hardware with MacBook Pro M3Max 128GB RAM 4TB SSD … I hope.

    ad 4) Yes, I can create a Debug ID. What exactly do you want me to do … after "Restart with Logging Enabled …"?
  • dstillman
    OK, forget about clearing the index for the moment. Let's focus on the actual problem you're facing. You're saying you add a new file and it doesn't get indexed? If you click on the attachment item, it shows Indexed: No?

    Can you provide a Debug ID for clicking the reindex button next to an unindexed attachment and (presumably?) having it not work?
  • wenhsuNYC
    ad 1) No. If I add a new PDF it gets indexed with no issue (I don't really check this, but having added a bunch of files today, I did not see any issues with this individual indexing of new PDFs). However, there are tons (i.e. 30,000+) of documents in the library that for whatever reason have not (automatically) been indexed in the past — see screenshot of one example that most likely got downloaded via Zotero Extension in some web browser. I reckon I was one of the early users of Zotero, so the library (50,000+ documents) goes back quite a few years.

    https://s3.amazonaws.com/zotero.org/images/forums/u15782064/1jevjilqz8ilwjm8sen5.png

    ad 2) I played with a few documents: The "reindex button next to an unindexed attachment" works fine — the file gets indexed. The counter for indexed files in the Index Statistics gets updated accordingly. No issue here.

    I think my problem is: 'bulk-(re)indexing' of un-indexed documents (PDFs) in the library. —Would a Debug ID for Rebuild Index… or Clear Index… be of help? (D1032347041 for Clear Index…). Clear Index does not clear the index: see error message posted above.
  • wenhsuNYC
    Addendum 1: Regarding the 30,000+ unindexed documents. I did not check this. I get this number from the Index Statistics. There is at least the possibility that some/many of those "unindexed documents" are (still) actually indexed — it's probably just that Rebuild Index managed to go only through roughly 10,000 documents … and does not 'know' the indexing status of the remaining documents.

    Addendum 2: The above posted error message points to "unresolved pending work". So Clearing Index encounters this "unresolved pending work", but there appears to be no way forward. How would/could Zotero get past this "unresolved pending work" situation?
Sign In or Register to comment.