One possibility would be that you're somehow starting afresh with a new database and pulling down items via sync, though you'd presumably notice that.

Another possibility is that you used ZotFile to move the attachment items, which I believe actually deletes and recreates the items, which would clear the index entries.

There's not really any code in Zotero to clear full-text index entries for existing items other than the code that runs when you clear the index manually.

Well, #1 was specifically somehow ending up with a new, empty database (say, from changing your Zotero data directory setting without moving your data) and then syncing to pull down data from the online library, not just ongoing syncing.

But you're saying that you did something on another computer and then just synced your existing library on this computer and the index shrunk? What exactly did you do on the other computer? If you can reproduce this, can you provide a Debug ID for a sync where the index shrinks in size?

I finally managed to catch part of a sync where the index is shrinking,
it is the debug ID: D1498775293
And another more extensive one, which also catches the end of that sync: D1317263503

No idea, what triggered this extensive full-text sync, as only a handful of files should have been changed in the meantime.

It is quite annoying, I basically rebuild the index every day and sometime later it is largely depopulated again. A colleague of mine came to me with the same problem.

Yes, exactly that. If you reindex items on one computer, as far as Zotero is concerned that's new full-text content that hasn't yet been synced to other computers, so it will mark the items on other computers as needing to be indexed ("Queued", in the right-hand pane) until the new content is processed in the background, and in the meantime the items won't show up in full-text searches. I guess technically it'd be better if they were simply marked as needing to be reindexed without throwing out the old content, but this mostly doesn't come up.

Why exactly are you reindexing things at all? Generally speaking, you shouldn't need to use any of the manual indexing/reindexing functionality — this all should just work automatically, indexing things immediately on the computer where you added them and on idle after syncing on other computers. The main reason to use manual indexing would be if you OCRed items yourself after adding them to Zotero, but in that case there wouldn't be content on other computers to begin with, so the current behavior wouldn't be a problem.

"Background indexing" (I'll use your terms) is simply processing the full-text content that was synced from another computer — i.e., adding that content to the local index. Before those items are processed, they'll show as "Queued" in the right-hand pane. This should happen pretty quickly, but only when the computer is idle and Zotero is open. In normal usage where you're using multiple computers, you mostly shouldn't notice it. If you add a few attachments on one computer and then go to another, they'll be processed the first time the computer is idle for more than 30 seconds after the new items are synced down.

"Forced indexing" would be re-processing the attachment file itself — i.e., extracting its text and adding it to the index as new data, which will then sync elsewhere. It doesn't use the queued content from the sync — it replaces it by reprocessing the local file.

I strongly suspect all the problems you're having are just from trying to manually intervene here (which is one reason I've wanted to de-emphasize/hide/remove the index stats and manual index functions). If you just let it do its thing, it should pretty much just work.

Well, I guess it all started because we migrated from Mendeley with several large group databases. So that put several thousand items in the queue. Then a colleague came to me and was very unhappy that the full text search was not functioning as desired and we realized the index was not complete and tried to speed up the process.

If you would hide/remove the index statistics, we would have filed an issue that the full text search is not functioning. So maybe, one could change the wording in the statistics to explicitly state the number of items 'in queue for background indexing' or something similar.

After waiting a few days, I realized that the indexing stalls after a while. On my laptop, I have about half of my library indexed with the process not proceeding any more. New additions are indexed, but he is not indexing the remainder of the existing library. If I choose re-index, he will build up the rest, but as discussed above, the index on my office PC will be lost.

For a colleague it is even worse, he has less than 10% of the library indexed and Zotero won't index the rest.

Well, the weird thing is. Once I start the output logging, he does index! Here are 2 debug IDs where he proceeded in indexing:
D1106067759
D108860634

But between them, I left the computer idle for the whole night and nothing happened.

And here are some folder names from `queued` attachments:
NN8C4PLP, 9496UYPW
And here from some attachments with status `unknown`:
QVEVMHIB, T672LXH4

Could it be that the idle criteria are too strict, e.g. that any possible other background process keeps it from running?

As I say, though, we might be able to do more of the processing faster or to make searches work before everything is indexed. I've created a ticket to track that.

The stats aren't correct — you should ignore them. I'm not sure what causes the Unknown, and I'll look into that, but it doesn't mean the files aren't indexed.

If you search for words or phrases from those PDFs, as long as they contain extractable text, they should be found.

The latest Zotero beta should fix a number of issues here. The state will no longer be reset to "Unknown" if you reindex the same PDF elsewhere, and both manual reindexing from the info pane and "Index Unindexed Items" will now use the synced full-text content if available rather than reindexing the file to avoid triggering additional full-text content syncing and indexing back and forth. (If you rebuild the index completely, it will go back to the original files.)

I still recommend mostly not worrying about this, but "Index Unindexed Items" should now be a way to force immediate indexing of any full-text content synced from other computers.