That's somewhat similar to the issues discussed in this thread. Basically, Attachment File Type matches attachment items, not parent items, and "is not" "PDF" is causing it to match all items — of any type — that aren't PDF attachment items. Some of the changes we're discussing there should help with this, but because of how the search conditions work now, I don't think there's currently a way to search for parent items that don't have PDFs.

In the meantime, you can of course create a saved search for books and then use the attachments column (the blue dot, though that will match all file attachments) and/or + (to expand all items) to find items without PDFs.

That will still match some PDFs in most libraries, since PDF attachments don't need to include "pdf" in the title.

It's not related to the OS or showing file extensions. Depending on how you save the files and your settings, some might be named the same as the filename, but that's not at all guaranteed and won't be true in various cases.