Strange characters have appeared in footnotes!!

amarcussells

Hi, so I was working on a document when zotero told me that it had installed an update and would like to restart. I restarted zotero and when I next inserted a footnote strange boxes appeared in about half of my footnotes. I have tried editing the footnote - they do not appear in the zotero toolbar - and have tried deleting the footnote and then redoing it - the same boxes appear.

The version of zotero now installed is 4.0.11. I am using it on a MacBook Air OS X 10.8.5 with Word for Mac 2011 version 14.2.5.

I have screen shots of some of the footnotes, but cant seem to attach them to or include them with this post.

adamsmith

post screenshots to a free image hosting site like imgur.com and provide the link here

amarcussells

Thanks!

http://imgur.com/jhfTBWo,XuTRTXB

amarcussells

Did those help?

adamsmith

kind of. I can see them. I don't know what they are.

amarcussells

is there any way to delete all zotero citations? it would be a pain, but then I could copy and past the text into another document, redo the citations and see if that solves the problem

amarcussells

Also, I just noticed, but they seem to only be in citations where the title of work cited has special characters in it (diacritics, because the titles are transliterated from arabic), even when the title is hidden ("ibid.")

adamsmith

That's very helpful to know. Maybe fbennett has an idea what to look for. Have you tried just copying the whole doc over to a new document?

amarcussells

I just tried that, but all the citations - with their strange characters - remained.

If the new version of Zotero can no longer handle diacritics, that would be a huge problem for me - is there any way to return to the previous version?

adamsmith

Of course Zotero will support diacritics - and if this were a general problem we'd have a lot more people complaining. You can try reverting to an older version, but I'm not terribly optimistic it will help.
All old versions follow the same url pattern, e.g. 4.0.9 would be
http://download.zotero.org/standalone/4.0.11/Zotero-4.0.9.dmg
but I'd be surprised if that helped.

amarcussells

okay. let me know if you think of something else. If not, I will try reverting to the old version tomorrow.

aurimas

Could you check the encoding of the document? File -> Options -> Advanced -> General -> Web Options -> Encoding. What does "Save the document as" say?

Can you copy and paste the contents of the footnote on http://gist.github.com and link to it here?

What font are you using in the footnote? When you place the cursor next to the box, what font does Word indicate?

amarcussells

Hi, I'm sorry, but in my version of word there is no "options" under the file menu. There is "preferences" but that doesn't "advanced" or "web options," ect. There is a File -> Save as Web Page -> Encoding, which says the document is saved as Western European (Macintosh). Is that what you needed?

When I got to "save document as" it tells me that the format is Word Document (.docx)

The font of the footnote text is times new roman, which is also the font on either side of the box.

I copied and pasted one of the footnotes into gist, below. The box doesn't appear. Gist did change the font and removed the formatting (italics, ect.).

https://gist.github.com/anonymous/7e597367534bd4b311ef#file-gistfile1-txt-L1

amarcussells

So I tried a few things and just wanted to tell you how they went:

When I changed the citation style to author-date the boxes remained, and the parentheses of the parenthetical citations also got messed up: http://imgur.com/4wgWRHT,QMMNvQP#0

I also tried starting a new document and creating some footnotes from the same sources. The boxes appeared again, but this time they had a question mark inside them.

I tried some other sources and sources with diacritics don't seem to be the problem - its the same three sources that have the little boxes.

aurimas

There appears to be a "Right-to-left embedding" character at the beginning of that string, probably coming from the last name field (?): "U+202Bal-KuntU+012B, "

In Zotero, place the cursor to the right of "a" in "al" for that name and press Delete a couple times (twice should be enough). Then re-insert the "a". See if that fixes it.

amarcussells

all of the sources that get a little box seem to have the same author.

in this example footnotes 1 and 2 are different works by the same author - while 3 is a work by a different author

http://imgur.com/5nAN2cL

aurimas

So did you try editing the name in Zotero?

Edit: Just to clarify, the boxes are shown when a particular character cannot be displayed with a selected font. Usually this will happen either when the document encoding is selected incorrectly (which I don't think it is) or if you have some Unicode characters in your strings that were not covered when the font was designed.

In the last screenshot, it seems that the unrecognized characters are at the beginning of Zotero fields starting with "Ma'had" and "Maroc". The character may not be visible in Zotero (it may be zero-width, i.e. take up no visible space), so you will want to delete the character immediately following it and keep pressing delete a few more times to make sure it is gone, then re-insert the deleted character.

We would also like to know how those names were entered into Zotero, so that we can prevent this in the future.

DWL-SDCA

edit Aurimus posted his second message while I was keying this post from my phone.

This seems very similar to a problem I have had with some items I've downloaded from WorldCat.

Please try a few more things:
Verify that the titles appear correctly in the Zotero records.

Go to the site where you obtained the items, copy one of the titles and paste it to a new word document. Do the boxes appear?

If they do, try pasting the title into a document within one of the good free text editors ( such as textwrangler ). Be sure that the character encoding for the editor is set to UTF-8. Copy the title from the editor to the Word document. If the boxes still appear in Word, you will need to wait a little while. I'll post again with details of how to do character encoding check that aurimus suggested.

amarcussells

I'm sorry, it didn't work. I tried deleting and retyping the "al" that begins both the last name of the author and the title in zotero and then refreshing the document. The boxes are still there.

Just after the restart there was a pop-up. I can't remember exactly what it said, but it was approximately: "you have made changes to this citation, would you like Zotero to save these changes and prevent you from making any further changes," and I believe that I clicked "yes."

DWL-SDCA

How did you get the item metadata into Zotero? Did you directly import it using the URL icon? Did you download an RIS file? Did you hand enter?

As I said above, some websites (WorldCat) do not provide characters encoded the in the way they say they do. This is one cause for the mess.

For some sources I know that I will need to paste titles or names into a text editor and then copy it from the text editor to Zotero or a word processor document for the characters to be correct.

amarcussells

Hi,

The original mess was probably caused that way - I imported the item metadata from worldcat, which resulted in a title in Arabic characters, then I deleted those and retyped the data. It makes sense that a right-to-left embedding character got stuck in there during the transition.

However, I just went through and deleted every citation from those sources, then I deleted those sources form zotero (by moving them to the trash). Then I manually created new sources in zotero and typed in all the information myself. Unfortunately, when I got back to word and put in a citation using these newly created sources, the boxes are still there!! I have no idea why that would be!

DWL-SDCA

Also, try highlighting everything and changing the font face to something like Times New Roman. It could be that the special characters are not available in the font that you have chosen.

amarcussells

Everything is already in Times New Roman in Word - do you mean changing the font in Zotero?

aurimas

Just after the restart there was a pop-up. I can't remember exactly what it said, but it was approximately: "you have made changes to this citation, would you like Zotero to save these changes and prevent you from making any further changes," and I believe that I clicked "yes."

This would have prevented any changes you made in Zotero (like deleting the offending characters) from carrying over to Word.

but....

However, I just went through and deleted every citation from those sources, then I deleted those sources form zotero (by moving them to the trash). Then I manually created new sources in zotero and typed in all the information myself. Unfortunately, when I got back to word and put in a citation using these newly created sources, the boxes are still there!! I have no idea why that would be!

This probably would have fixed the problem. Can you select the record that produces boxes and (right-click) export it in Zotero RDF format? Open it in any text editor, paste it on http://gist.github.com , and link to it here.

Is this Chicago Manual of Style (full note)?

aurimas

Unfortunately, when I got back to word and put in a citation using these newly created sources, the boxes are still there!!

There's a chance that you re-inserted an old reference that was stored in Word.

Could you try this in a fresh Word document?

amarcussells

https://gist.github.com/anonymous/bf42c7aa1b883371f6c2

and yes, Chicago Manual of Style (full note)

aurimas

OK, that record looks clean. I cannot reproduce the box in front of "al-Kuntī" with CMoS in Word, but I am on Windows, so this could be Mac-specific.

If you can reproduce this in a new Word document and be certain that you're inserting that same record that you linked to (i.e. maybe change the title from the one in Trash, though it should _not_ show up in Word anyway), the characters might be inserted by Zotero. Frank would then have to take a look at this.

amarcussells

okay! I can get rid of the boxes.

I changed the title slightly and put them into a new word document. No boxes!!

BUT the formatting was wrong because I had not specified the language of the source as Arabic. When I went back to zotero and put in ar under language and refreshed the word doc the boxes came back! When I took ar out of the language box and refreshed the doc the boxes went away.

notes: playing with the language field has no effect on the boxes in the original document, only in the new blank doc

aurimas

BUT the formatting was wrong because I had not specified the language of the source as Arabic.

What formatting is missing? In my tests, the only difference I can see is an odd character before "2 vols." when using "ar" in the language field. (I'm also not able to get the boxes to show up, but I'm convinced it's due to the difference in Times New Roman versions we have)

You may want to check if you have a conflicting set of Times New Roman fonts, where an older version is taking precedence. http://word.mvps.org/Mac/fontweeding.html (not tested) If your fonts check out, then sticking to English (or nothing) in the language field may be your best bet for now.

fbennett will have to take a look at this. He was working on adding support for right-to-left citations, but I'm not sure if he ever finished and this is the reason you're getting the boxes with Arabic language.

adamsmith

nevermind. Misses the last two posts before writing.

amarcussells

The difference in format has to do with the capitalization of the titles. You should notice that if the language field is blank the first letter of each word is capitalized. This is incorrect when the title is a work in Arabic (or French and many other languages for that matter). Putting 'ar' in the language field fixes this problem.

I have never installed a Times New Roman font - I am using the version that came with word.

I am, however, typing with a keyboard called "American diacrits" that I installed to make typing all the diacritics easier. It is unicode keyboard and has never given me any trouble in Times New Roman before.

Thank you for helping me with this. Since this problem only appeared after Zotero updated I would like to try reverting to the old version of Zotero, but the link that adamsmith posted above is broken.

Finally, I will be traveling for the next few days. I will check in when I can, but will be much slower on responses. I do still very much care about this problem, however.