Upgrade StoragePDF reader: space lost after an Arabic word ending in a diacritic or followed by punctuation
In right-to-left text, a highlight or copied selection in the PDF reader loses
the space after:
- a word whose last character is a combining mark:
"مقسوماً على" is stored as "مقسوماًعلى"
- punctuation:
"الجذر. يُقاس" is stored as "الجذر.يُقاس"
Spaces between plain, unvocalised words are kept. I see it with PDFs from
different producers (XeTeX and Chromium), so it does not depend on the file.
Zotero 10.0.8, Linux (Flatpak).
Cause, as far as I can tell: in split() of the structure module in the PDF
worker, the gap between two neighbouring characters is measured right-to-left
only when both are RTL letters:
let rtl = isRTL(char.c) && isRTL(char2.c);
sp = rtl ? char.rect[0] - char2.rect[2] : getSpaceBetweenChars(char, char2);
A combining mark (NSM) or a neutral such as "." is not RTL by that test, so
the pair falls back to getSpaceBetweenChars, which assumes left-to-right
order. In a right-to-left line that gap is negative, so no space is recorded.
A change that fixed every case in my test file, without affecting
left-to-right runs inside the line: let a mark or neutral take the direction
of the strong character beside it, and of the line when both are neutral.
let strong = c => isRTL(c) ? 'R'
: (/[\p{L}\p{Nd}]/u.test(c) && !/\p{M}/u.test(c) ? 'L' : null);
let s1 = strong(char.c), s2 = strong(char2.c);
let rtl = (s1 || s2 || (hasRTL ? 'R' : 'L')) === 'R'
&& (s2 || s1 || 'R') === 'R';
I can share a one-page test PDF.
the space after:
- a word whose last character is a combining mark:
"مقسوماً على" is stored as "مقسوماًعلى"
- punctuation:
"الجذر. يُقاس" is stored as "الجذر.يُقاس"
Spaces between plain, unvocalised words are kept. I see it with PDFs from
different producers (XeTeX and Chromium), so it does not depend on the file.
Zotero 10.0.8, Linux (Flatpak).
Cause, as far as I can tell: in split() of the structure module in the PDF
worker, the gap between two neighbouring characters is measured right-to-left
only when both are RTL letters:
let rtl = isRTL(char.c) && isRTL(char2.c);
sp = rtl ? char.rect[0] - char2.rect[2] : getSpaceBetweenChars(char, char2);
A combining mark (NSM) or a neutral such as "." is not RTL by that test, so
the pair falls back to getSpaceBetweenChars, which assumes left-to-right
order. In a right-to-left line that gap is negative, so no space is recorded.
A change that fixed every case in my test file, without affecting
left-to-right runs inside the line: let a mark or neutral take the direction
of the strong character beside it, and of the line when both are neutral.
let strong = c => isRTL(c) ? 'R'
: (/[\p{L}\p{Nd}]/u.test(c) && !/\p{M}/u.test(c) ? 'L' : null);
let s1 = strong(char.c), s2 = strong(char2.c);
let rtl = (s1 || s2 || (hasRTL ? 'R' : 'L')) === 'R'
&& (s2 || s1 || 'R') === 'R';
I can share a one-page test PDF.