Any test cases available?

Is there anyone out there willing to share a set of test cases that they use for testing their CSL?

It would be great if we could have an exported "Zotero RDF" or zotero.sqlite with test documents (various combinations of empty/non-empty fields across all document types).
  • edited September 8, 2009
    I made a group for this recently that we could use for this purpose: CSL styles development. I think that would be the best way to share test cases.
  • I've started working on this in the group - artifical items where the title describes what is going on.
  • The CSL test suite is here. Would be good if people could contribute to that.
  • sorry, I'm confused at what this does.
    I just want a couple of references that let me see quickly if I've done everything right in a style I'm working on. This appears to be something else, but maybe I'm also just not getting it?
  • Maybe I just got thrown by the "test cases" language. What I was pointing to is to ensure that processor do as expected.
  • The tests that Bruce points to were written to check that the processor behaves correctly, but building a bundle of tests for each major style (using the same sort of test framework used for the processor tests) would also be a very good thing. It would save work in the long run, and it would be a very good thing for Zotero and for CSL.

    To test a change made to a style using the test cites in mark's group, you would (I think) load the modified style to Zotero, then select a suitable test entry in Zotero, then call up the csledit.xul page, and then select the style you're testing, then inspect the output and judge whether it does what you expected. That tests one instance of behavior. To test other behavior, you would select other entries, run the style, and again inspect the results visually, and compare them with your expectations. This method of testing leaves behind no lasting record, and can't be repeated without going through the same point-and-click steps all over again.

    Using something like the processor tests (i.e. a program that you run from the command line that processes the test data outside of Zotero), the task of checking the output for correctness can be automated, and the tests can be batched together into a single "style test". Each time you make a change to the style, you would also write one or more tests to make sure that everything is working, and add them to the style's test bundle. Then you would run the entire bundle of tests, to be sure that they all return "ok". That lets you lock in the current behavior of the style, and protects against breakage that might be caused by later revisions to the style CSL or the processor.

    Having a good, full set of tests for each of the major styles is an essential step toward the widespread adoption of CSL as a standard tool that publishers positively recommend to their editors and authors. I would be very happy to help build a test framework and bundling architecture that is tailored to the needs of style testing.
  • Frank - my idea is to have a set of bibliographic items that cover important instances and most typical issues. I want to use the same ones all the time and I want to use them to quickly check my created and/or altered styles.
    What I do is to go into the test panel, select all of the items, and see if they do what I want. As long as I'm working in the chrome panel - which I always use for small edits - the updating is automatic and I have all of the result in the lower part of my screen - so it's actually a lot less complicated than you indicate.

    For style creation purposes I'm not sure how that can be more automated, because obviously the program doesn't know (yet) what I want from the style.

    What you seem to talk about is automatic checking for existing styles (i.e. all major styles as you say) - that's out of my league.
  • Mark, thanks for putting the group together. I think it is a good start. Two nice-to haves though:
    1/ All test citation documents packaged/bundled in a single file for an easy import into zotero.
    2/ A holistic approach to building the set of test cases toward covering all possible combinations of item types and field contents.

    Bruce, sorry for the ambiguity. I am not a CSL-processor developer. I am interested in modifying CSL styles so I am looking for test cases to test the CSL file content.

    Adam, I think we are on the same page.

    Frank, thanks for the elaboration. I you have any capacity to build such a framework, it would definitely be appreciated. For my purposes (regression test of a specific CSL file) at the moment I hoped to use the chrome://zotero/content/tools/csledit.xul page with all the items in the test citation library selected.
  • 1) is very easy to do once we have all we want - but if you join the group you have them anyway, right in your Zotero.
    2) OK, that would require a more systematic approach - suggestions on how to proceed? All combinations doesn't make sense - that's too much stuff to look at.
  • Adam,
    1) excuse my ignorance, I have not used the group feature yet.
    2) I agree, all combinations would be an overkill. On the other hand it is also difficult to determine which ones are possible/make sense. I hope someone can come up with good suggestions. Maybe we could review available style guides and compile a superset of examples and then try to extend them.
  • I think it makes sense to have the most common item types at least once:

    Journal Article
    Book Section
    Conference Paper and

    I think for most of these there are common features to be left out - e.g. a Journal Article will always have a Journal title, and a title, the most crucial things that can miss are issue, volume, and page numbers.

    There is significant overlap for some parts - we only need to test what happens without author and without year for one item - regardless of type.
    These are the four that I've already put in the group (in APSA style). The respective title tells you what you're supposed to see.

    Author1, One. Book With Series Title, 3rd Edition, no date. Testville: Test Publisher.

    Author1, One, Two Author2, and Three Author3. “Article, 3 Authors, abbreviation given..” Test Journal 12(2): 12-17.

    Author1, One, Two Author2, and Three Author3. 2008. “Article, no issue, no pages, URL, full date..” Test Journal 12. Available at: [Accessed December 16, 2008].

    Author1, Otherone, and Two Author2. 2001. “Book Section, 2 Authors, 2 Editors, 1st. Volume.” In Edited Volume Book, eds. One Editor1 and Two Editor2. Testplace: Test Publisher, p. 1-17.
Sign In or Register to comment.