Wrong representation of date of access, sorting issue in bibliography

verano

After the upgrade to 2.1.1 I encountered two problems in my bibliography (in Word) that weren't there before. This is quite urgent because I have to hand in my thesis in a few weeks.

1. Dates of access are misrepresented in the bibliography, as it seems only if it contains the month of January.
For example:
- "2011-02-10" in the "Accessed" field shows as "February 10, 2011" in the bibliography (correct)
- "2011-01-10" shows as "2011" (month and day missing)
- "2011-02-10 13:12:45" shows as "February 10, 2011" (correct)
- "2011-01-10 13:12:45" shows as "12:12:45" (time less one hour instead of date)

2. Zotero now sorts bibliography entries by capital letters first, i.e. "ABC 2011" comes before "Aaa 2011", and "Xyz 2011" comes before "abc 2011".
Not sure if this change is intended, but I would need to switch it back to sorting that ignores the case. How?

I am using Firefox 3.3.16, Zotero 2.1.1 and WinWord Integration 3.1b1 with a customized Harvard style. I am not aware that my customizations have caused this, at least it worked fine before the upgrade. The first problem also occurs for the original "Harvard Reference format 1" style in the Zotero test pane (the second issue only appears in my word doc).

If there is no quick fix for this I need to downgrade, so please let me know!
Thanks!

fbennett

Both issues will be fixed soon; thanks for reporting them.

I've checked a fix for the issue with January being dropped into the multilingual branch, which will at least provide a starting point for a fix on the trunk. I'll check out the sorting issue tomorrow, and take care of it in the processor if necessary. It might take a few days for these changes to work their way into a release, but certainly not weeks.

dstillman

Simon already fixed January on the trunk and branch.

fbennett

Yep, crossed in the post. Now working on the case-sensitive issue.

verano

Thanks for the prompt answer!

Meanwhile I noticed another issue regarding dates:
If the "Date" field in the database contains something other than the year there are problems as well.
For instance, the citation becomes "(Zipkin Spring)" instead of "(Zipkin 2001)" if the Date field contains "Spring 2001".

I have one more questions: the update has undone all my manual changes to citations, and it has added an additional space after/before pre- and suffixes. Can this be circumvented? (I don't want to adapt 500 references again...)

fbennett

"(Zipkin Spring)" instead of "(Zipkin 2001)"

This has been fixed in the trunk, and in the upcoming 2.1.2 release.

fbennett

it has added an additional space after/before pre- and suffixes

This too.

fbennett

undone all my manual changes to citations

Not sure what you mean, but it's worth waiting to see how you fare with the next release.

fbennett

"ABC 2011" comes before "Aaa 2011"

Actually, a test shows that the processor is correctly performing case-insensitive sorts. Is it possible that your style combines the author/title and the date elements in a single macro, instead of setting them as separate sort keys, as described here?

verano

"undone all my manual changes to citations"
...by this I mean that all manual edits that I made via "Show Editor..." are lost now, is that correct? Would be a nightmare...

With respect to "ABC 2011" comes before "Aaa 2011": Are you refering to this piece of the csl file?
<sort>
<key macro="author"/>
<key variable="title"/>
</sort>

macro="author" author does this:
<macro name="author">
<names variable="author">
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="."
delimiter-precedes-last="never" delimiter=", "/>
<label form="short" prefix=" (" suffix=".)" text-case="capitalize-first"/>
<substitute>
<names variable="editor"/>
<text macro="anon"/>
</substitute>
</names>
</macro>

fbennett

Again, in the test framework (running the CSL processor outside of Zotero), I get correct sorting behavior with this code. It is very unlikely that the behavior would change when the processor is run inside Zotero (although I have been proven wrong in the past). Are you sure that there is no anomaly in the input -- are both ABC and Aaa set in the Author field of their respective entries (or both in the Title field) in your test? If so, then let's look a bit closer, beginning with the style. If you post it to http://gist.git.com/, and post the link back here, I can take a look.

(Edit: Despite my qualified confidence expressed above, sorting does rely on localeCompare(), which can be flakey in Firefox. If it does turn out to be churning out case-sensitive sorts, there are ways of fixing it -- all we need to do is apply the appropriate hammer to it. Testing with your style will be the first step, though, to be sure that this really is a processor issue.)

verano

ok. I haven't changed the style for quite some time, though, and the sorting issue does not show in my word file from Sunday (before I upgraded).

Actually, I am more worried about the undone manual edits of the in-text citations - they'd take a lot of time to replicate - time that I don't have...

I think the safest for me would be to try to completely undo the upgrade. I happen to have a full backup of the Zotero data folder just before the upgrade - if I replace the entire folder, would that do the trick? Or do I also need to reinstall the older Zotero & plug-in version? (which ones - how can I see what I had before the upgrade?)

adamsmith

the manual edit issue is fixed, too:
http://forums.zotero.org/discussion/17142/deletion-of-edited-part-of-citations-after-upgrade/#Item_6

verano

@ adamsmith: great! then I'll delay my nervous breakdown and hope for 2.1.2...

fbennett

@verano: We should probably carry forward with checking out your style, then. When you have a moment, if you post the code as a public gist at http://gist.github.com/ and post the URL back here, I'll take a look.

verano

@fbennett: I really think the problem lies elsewhere ... I've just created a new Word doc with the original Harvard Reference format 1 style with 5 test entries. This was the result:

Citations:
(aaa 2011) (Aaa 2011) (zzz 2011) (ZZZ 2011) (AAZ 2011)

Bibliography:
AAZ, 2011. Test.
Aaa, 2011. Test.
ZZZ, 2011. Test.
aaa, 2011. Test.
zzz, 2011. Test.

Pls let me know if you can't replicate that result, and I'll upload my style!
Cheers

fbennett

I can't replicate that result.

verano

@fbennett: Sorry, didn't see it until today.
Here you go: https://gist.github.com/892422

The problem persists with other styles, though (maybe some issue with my local machine?!). This is the Bibliography output of "Chicago Manual of Style (Full Note with Bibliography)":
AAZ. “Test”, 2011.
Aaa. “Test”, 2011.
ZZZ. “Test”, 2011.
aaa. “Test”, 2011.
zzz. “Test”, 2011.

fbennett

I'll take a look tomorrow.

fbennett

I've arranged in the processor to avoid the localeCompare() method for sorting in Firefox, and rely on a collation instead (the method used in Zotero itself, which has proven reliable in that context). The revised processor will be released soon, and may be in time for 2.1.2, if the core devs feel comfortable with the changes.

torad

@fbennett:

Again, in the test framework (running the CSL processor outside of Zotero), I get correct sorting behavior with this code. It is very unlikely that the behavior would change when the processor is run inside Zotero (although I have been proven wrong in the past).

I have the same problem using the native APA 6 style. It sorts correctly in the CSL editor (chrome://zotero/content/tools/csledit.xul), but not in the Word and OpenOffice plugins.

There is an additional problem with the sort order of Norwegian characters. They are sorted before other characters, but it should be the other way around.

Example:
Correct sorting: 1) Henriksen, 2) Høgskolen i Oslo
Wrong sorting (from the Word/OpenOffice plugin): 1) Høgskolen i Oslo, 2) Henriksen

fbennett

From your descriptions, it sounds like the localeCompare() function is performing a binary sort (comparing two strings on the basis of the literal Unicode value of each of its characters, rather than their "alphabetical" sort value). We've seen this before in other contexts, and the solution that worked in that case was to use a different comparison function that is less convenient to code, but more stable.

The new sort comparison operator will be in the processor for 2.1.2. Let's see how it goes.

torad

I have installed the 2.1.2 version of the Zotero. Two of the problems mentioned in this thread are fixed (date issues and case-insensitive sort), but the sorting of Norwegian characters is still wrong. The CSL editor sorting works, but not the word processor plug-ins.

fbennett

Very good to hear that sorting has improved.

Failure in the plugins is odd. Sorting in both contexts is done by the same processor. A different instance, but they run in the same environment. Hmm.

verano

It seems that 2.1.2 has corrected all the issues I've described before, so that's awesome!!

However, I found one additional issue just now which wasn't corrected. A couple of citations are not minimzed anymore, and I have no idea why:
was: (Berns et al. 2009b)
is now: (Berns, Townend, Zayna, Balagopal, Reeves, Hopkins & Kruschwitz 2009b)

Don't know why this happens - another item with eleven authors is correctly minimized to (Tilman et al. 2009), and other items with 3 or 4 authors are still minimized, too...

This should be the relevant code, I guess - do I need to change it?!
<citation>
<option name="et-al-min" value="3"/>
<option name="et-al-use-first" value="1"/>
<option name="et-al-subsequent-min" value="3"/>
<option name="et-al-subsequent-use-first" value="1"/>

fbennett

Is there another article in the set by Berns, Townend, Zayna, Balagopal, Reeves, Hopkins & Kruschwitz? If so, that would explain it.

There should be a line in the citation part of the style that reads:<option name="disambiguate-add-names" value="true"/>If you delete that line, CSL will not attempt to distinguish between citations by adding extra names, but will instead rely exclusively on the year-suffix for disambiguation.

adamsmith

Is that desired behavior though? If given name disambiguation doesn't help, should Zotero still print all the authors?

verano

@fbennett: You're right, that line is still there. If that solves it I'm happy - I just hope there are no other effects...
@adamsmith: I agree - this behaviour doesn't disambiguate anything, and the "b"-suffix does the job.

fbennett

Yes, this is a tricky one. It could be more sophisticated, but it will stay with the current logic at least until 2.1 is running happily. The problem is that it may help. You could have authors like this:Smith, Jones, Brown, Hart, Rodgers 1999
Smith, Jones, Brown, Hart, Rodgers 1999
Smith, Jones, Brown, Wilson, Taylor 2001
Smith, Jones, Brown, Wilson, Taylor 2001When formatted with et-al-min="3", et-al-use-first="1" and disambiguate-add-names="true", this would yield something like:Smith, Jones, Brown, Hart et al. 1999a
Smith, Jones, Brown, Hart et al. 1999b
Smith, Jones, Brown, Wilson et al. 2001a
Smith, Jones, Brown, Wilson et al. 2001bWhen there is only one such group, the names could be shrunk back, but identifying when that can take place and what the constraints are will introduce some complexity, and will need to be thoroughly vetted (for the rules) and tested (for the code). The disambiguation code is very orderly now (two years on), so this level of care should be possible. But I'd like to wait awhile (and gauge how loud the objections to the current setup are) before digging into it.

adamsmith

ah yes, I see, that makes sense. There's a reason they call you Frank "disambiguate-this!" Bennett ;-)

fbennett

It's made me a very popular dinner guest, I can tell you.