I'm having the same problem, though my library is about a fourth of the size of @danielborek's. I am using Better BibTex with an automatically updating bib file. Any change (adding a source, moving a source, deleting a source, adding a subdirectory) causes lagging on the order of 10-15 seconds, though sometimes much longer, with the problem getting worse the longer Zotero is open. Problems seen within the main Zotero app, during which Zotero consumes a ton of CPU. I'm on a M1 mac running Sonoma 14.4.1.
I've reproduced this with the following Debugging ID performed when attaching a file: D764863472
Having a day or so to experiment led me to discover that BBT syncing a bibliography (BibTeX) file was the source of the problem. I've yet to determine a workaround, though I suspect that having the sync occur less often or perhaps not syncing to a OneDrive folder might do the trick.
I've reproduced this with the following Debugging ID performed when attaching a file: D764863472
I'll also try with BBT disabled.