"disambiguate-add-year-suffix" changing at each refresh

brazuca

I have a weird problem with my style that I don't know how to identify or to solve.

I have multiple references where the author is actually an institution ("CEPAL"). I have entered this in the last name field, leaving the first name blank.

In a word document, I enter these in text references, and generate a bibliography.

Problem is that each time I refresh the document, the citations change the disambiguation letter (e.g.: CEPAL, 2010b changes to 2010c and back each time I refresh).

The same doesn't happen when I use Chicago Author-Date, so I'm sure I'm doing something wrong, probably in my Author macro.

Could someone take a look and tell me how, and why the problem is occurring?

Another problem that I think is related is that often the *first* entry is replaced by "---", when it shouldn't.

My code must be making a mess of the order.

Here is the .csl file: http://dl.dropbox.com/u/258390/UN_ECLAC_CEP

adamsmith

I'll have a look at this later, but generally for institutional authors, use the one-field option (click on the little white strip behind author). That might already fix things.

brazuca

Thanks adam. I changed all the references to one-field option, but the problem remains (the disambiguation letter switches back and forth).

I think it has to be my style because the same problem doesn't occur when I use Chicago Author-Date. I think it must be related to how I have my Author macro and the <choose> blocks in the Bibliography layout section.

But I don't know enough about how the csl processor works to identify the issue.

fbennett

Sounds like it might be caused by an invalid style confusing the processor. Have you validated it against the 0.8.1 schema?

brazuca

@fbennet:

Just ran the style through the validator.nu page (as per the instructions) using the csl given here: http://www.zotero.org/support/dev/creating_citation_styles

It validates fine. Don't know if its the 0.8.1 schema or not.

adamsmith

I can't download the style - could you check the URL? I always recommend using gist.github.com for such purposes - it's fast, no registration required and I can look at the style even before downloading.

brazuca

Weird. Its a dropbox public link, but looks like Win XP messed up the copy/paste:
http://dl.dropbox.com/u/258390/UN_ECLAC_CEPAL-v4.1-EN.csl

In any case, here it is in githup: http://gist.github.com/543014
I had tried uploading to gist.github.com as well as pastie et al, but something was odd when I send the data from work (kept getting a "network reset error".

It works fine from home. Lots of odd errors in the last few days...moon must be in phase.

adamsmith

OK got it now - I'll have a look but that style is quite a labyrinth - is there really no way the author section could be simpler?
You know you can use macros in the substitute section, right? So instead of the whole
<else-if variable="publisher" match="any">
you could just use
<text macro="publisher-author"/> (or so) as the third substitute option and equally
<text macro="container-substitute"/> (or so) as the fourth one.

Anyway - maybe someone else can take a look, too, this might be a bit hard to track down.

brazuca

Part of me wants to rework the entire thing based on Chicago MLA (it looks very clean).

The style I'm working on is complex. The UN has some weird rules. We have most of it working, but this error is a deal breaker (I just found it while we started using Zotero on our first large publication, so its a problem).

I went through the code last night and couldn't come up with anything that was odd. I'm sure the overall structure is making a mess of things, but I still think it should output something that is wrong, or flip back and forth the way its doing now without some way to track down the error.

I'm going to start modifying the code to see if I can track where things are going bad. I would appreciate any help.

adamsmith

OK - I've tracked this down and it looks like a bug and not a problem in your style:

This happens when the bibliography is sorted in the descending order of publication.
If it's ascending (the default) things are fine, but descending throws Zotero off. It doesn't matter if you use the issued variable or a macro that's limited to the year.

This sorting generally does something odd - even if both years are the same and you add a third sort variable (e.g. title) Zotero keeps doing this wrong.

Maybe someone can check if this can be easily fixed?

ajlyon

Since 2.1 is not far away, I doubt that anyone is going to work on fixing CSL bugs from <=2.0. If this is just for a single publication, you may be better off moving to the development versions of Zotero 2.1 and the integration plugins to do this. I'm confident that the new processor won't have this bug, and if it does, then Frank is bound to fix it ASAP.

Since you don't want to switch to Zotero 2.1 pre-pre-alphas for all your work, I would recommend making a copy of your data directory and installing the development XPI in a new Firefox profile, setting it to use the copy of the data directory, opening the document, and fixing the citations. If you ignore the funny switching for now, you can do this as the final step in manuscript preparation.

Simon

What happens if you add

<key variable="citation-number"/>
to the end of your bibliography <sort> section?

adamsmith

@Simon - nope, still the same problem - I thought along the same lines - but adding another search category doesn't fix this.

@ajlyon - I know of course - the problem is that this is not for brazuca alone - it's for a large chunk of CEPAL/ECLAC, the UN Economic Commission for Latin America - as he has mentioned before, they're considering using Zotero exclusively - I thought if this is just a three line bug maybe someone would take a heart given the circumstances.

ajlyon

I'm not very familiar with the guts of the CSL parser, new or old. I just now took a look at the current implementation (in 2.0), and I don't see any obvious reason why this should be happening. But the disambiguation code is pretty hairy.

fbennett

brazuca,

The fault may lie in the Javascript engine itself. Here's a (probably outdated) expression of mild annoyance from the citeproc-js source.
//
// note for posterity: Rhino and Spidermonkey produce different
// sort results for items with matching keys. That discrepancy
// turned up a subtle bug in the parallel detection code, trapped
// at line 266, above, and in line 94 of util_parallel.js.
//
I suspect that it will be okay in 2.1, but if you'd like to work up a test case, I can run it through the processor and see what happens.

brazuca

I'd be glad to do what needs to be done, but I would have to learn how to do a test case. It doesn't look quick.

Is there a way to just rework an existing test? From what adamsmith wrote it is just a problem of reversing the bibliography order. Any chance of just making a 1-line change to an existing test?

brazuca

@fbennet,

Looking at the test cases, looks like there is one done by you that looks at this bug:
http://bitbucket.org/bdarcus/citeproc-test/src/tip/processor-tests/humans/bugreports_OldMhraDisambiguationFailure.txt

edit: another: http://bitbucket.org/bdarcus/citeproc-test/src/tip/processor-tests/humans/disambiguate_YearSuffixAndSort.txt

The entire thing is a bit over my head, but let me know if you need me to make any changes to it or add my specific csl code. There seem to be actually quite a few bugs labeled "disambiguate", and I'm not sure which one would best fit this case.

We are all in a bit of a crisis with this bug since it is currently in use on an official document and the references are coming out wrong.

adamsmith

I've tried to think about a work-around, but I haven't come up with anything useful, sorry.

Edit: and promptly it is supplied by ajlyon below - yes, that sounds like the best solution I can (not) think of.

ajlyon

Frank's question about the test case is to make sure that this isn't still a problem in the new, upcoming Zotero 2.1. Unfortunately, it won't fix your problem.

If you are in a real bind, you can follow my instructions above to do this using the development version of Zotero. That should definitely work well enough to complete this document and publish it.

If you find yourself having this problem frequently, I recommend creating a separate Firefox profile and Zotero.org account that you use to finish documents, with the 2.1 development version installed in that profile. You can then author documents using Zotero 2.0 and items in a Zotero group library from all of your computers. When you are done, just open the document from the Zotero 2.1 profile, refresh the citations, and everything should fix itself.

fbennett

It's certainly worth running the document through 2.1, as ajlyon suggests. In case what I wrote above wasn't clear, I think it's very likely that the sorting issues will clear up for you in 2.1

One note on the processor: the current Zotero 2.1 is using processor version 1.0.52. User feedback (from one of the project co-founders) identfied a bug in disambiguation recently, which has been addressed in the next (1.0.53) version. It should appear in the "trunk xpi" installable version in the next few days. It will hurt nothing to take the processor for a spin in the meantime, but you'd want to use the later version for finalizing a document.

brazuca

Thank you for looking into the issue. Unfortunately, we have zotero deployed across the institution. We trained staff on its use and set up the editorial division's workflow to adjust to inputs that use zotero. While the proposed solution might work for myself, there is no chance that we would be able to do this with the 3 reference editors, who have to interact with editors and translators, some of them contractors, in parallel. If the system isn't simple, we simply can't use it. This also comes while the technology committee is looking at all possible (commercial) alternatives.

The good news is that editorial is willing to change the style to reverse the sort order, avoiding the problem, but it is a bit disappointing that such a glaring and serious bug will remain until the next revision.

By the way, is there an ETA on 2.1? It was originally planned for March, and if there is no further bug fixes on the current version, we would like to plan accordingly.

ajlyon

By the way, is there an ETA on 2.1? It was originally planned for March, and if there is no further bug fixes on the current version, we would like to plan accordingly.

I (not a Zotero team member) don't know of any new features that are necessarily holding up the start of the 2.1 beta cycle. I still hope we can get some improvements to item fields to land before 2.1, as well as support for multilingual fields, but I'd rather see a quick 2.1 release and an aggressive schedule for 2.2 than make people wait any longer for CSL 1.0 and the new citation processor to be deployed.

ajlyon

But note http://www.zotero.org/support/development_roadmap -- perhaps we are waiting for the recommendation engine? That could be interesting, but pushing it to 2.2 wouldn't be the end of the world.

fbennett

Sorry for not looking into this earlier, and for any alarm caused by my casual reference to variations in JS sort ordering et cetera.

The problem in the current processor turns out to be very simple to fix. I've filed a ticket and a patch (which should be merged and available in 2.0 clients fairly soon, I think).

Hope this helps!

brazuca

Great news!! Thanks!