Upgrade StorageExtracted PDF highlights include soft hyphen (U+00AD)
I appreciate that when I make highlight annotations in PDFs in Zotero, the extracted text that appears in the left sidebar automatically expunges line breaks and line-ending hyphens. However I've noticed that when making highlights in some PDF files, an invisible soft hyphen character (U+00AD) is sometimes copied over into the extracted annotation. Usually this character appears in the middle of a word, which can prevent the annotation from being returned when running a search for the term in the left-sidebar annotation window (for example, searching the term "economic"—without the soft hyphen—would fail to return annotations containing "economic"—with the soft hyphen). This can also lead to abnormal behavior when copying the extracted quotes into other programs like LibreOffice. Would the devs consider automatically removing this invisible character from extracted highlights, in the same way that line breaks and line-ending hyphens are removed?
I'm using 7.0.0-beta.54+6b996d4f9 on macOS and the testflight beta on my iPad/iPhone. It seems that highlight annotations created on iOS include the soft-hyphen character, and annotations made on my computer remove the soft-hyphen and substitute a regular space character in its place.
I'm using 7.0.0-beta.54+6b996d4f9 on macOS and the testflight beta on my iPad/iPhone. It seems that highlight annotations created on iOS include the soft-hyphen character, and annotations made on my computer remove the soft-hyphen and substitute a regular space character in its place.
We would like to investigate in which cases PDFs use those characters. Could you send an example PDF (or just single page) to support@zotero.org with a link to this thread?
Do you have examples of other PDF viewers that work better with these PDF files?
\u00AD, which is a soft-hyphen.The app PDF Expert behaves almost flawlessly, but not 100%. PDF Expert properly removes the soft-hyphen from the selected text when I copy/paste the excerpt in question onto my clipboard. For example, if I select and copy/paste the word "preserved" (printed on the page with "pre-" at the end of the line followed by "served" at the start of the next line), the text on my clipboard automatically merges the two fragments together, forming the full word
preserved(as opposed to the broken wordspre servedorpre served, which are the results from iOS Zotero and macOS Zotero, respectively—note that they might look the same onscreen depending on your browser, but in fact each of the three versions include different characters). However, Zotero and PDF Expert both behave in a similarly-flawed way when I make a highlight/annotation that includes a word split by a soft-hyphen at the end of a line: the text contents of the highlight annotation—which are automatically extracted from the page and displayed in the left-sidebar alongside other highlights/annotations—includes a space in the middle of the word. To reiterate, PDF Expert properly mends together the split fragments of the word when copy/pasting from the page, but it behaves sub-par (like Zotero) when making highlights on the page, to the extent that it fails to merge the split fragments of the word separated by a soft hyphen at the end of line.This is unexpected behavior because Zotero automatically merges together the split fragments of a word if the hyphen at the end of the line is a typical "Hyphen-Minus" (0x2d) character. I would expect that same behavhior from a soft-hyphen, especially in light of the fact there might not be any visual difference between these distinct unicode hyphen characters. To the eye, they appear identical, yet they behave differently. Oddly (pleasantly), Zotero is doing something behind the scenes that makes the "Hyphen-Minus" character behave like a soft-hyphen (as it should!), yet Zotero fails to perform the same behind-the-scenes treatment on actual soft hyphens, thus rendering the soft hyphen improperly as a "space" character.