I'd like to corral a few different threads of discussion here:

http://forums.zotero.org/discussion/385/
http://forums.zotero.org/discussion/78/
...and Bruce's useful prodding

Right now, Zotero has what's called a "flat" data model - there are a set number of item types, and each one has a fixed number of fields. This makes for easy coding, and is the model that Endnote and virtually every other research management tool use. As much as anything, this model grows out of the bibliographic citation; according to MLA or Chicago or whatever, there are a certain number of citation types, and if you're writing software that is ultimately about facilitating bibliography-creation, then the smart thing to do is to just implement those item types in a flat model.

The problem, though, comes when you want to do more interesting things with your data than simply create references; there are *relationships* between items that aren't expressed in a flat model. If you have several chapters from an edited volume in your library, for example, you'd want the data model to reflect the fact that, while each chapter is its own discrete object, they also share a parent (the book itself). That way, you wouldn't have to repeatedly enter the same information for the book each time you added a new chapter to your library; you'd just link each new "Edited Volume Chapter" item to its parent collection. By adding parent-child relationships, you move from a flat to a "hierarchical" data model, which is more elegant, cleaner, and just all around more appealing.

It also raises a few really big issues, which is why Zotero's still using a flat data model.

*Interface*

Right now, we're pretty happy with the iTunes-like Zotero interface; there's a library, and items in a list within it. That list can be sorted in a few different ways, but it's fundamentally a flat list. There is a hierarchical relationship between items and snapshots/links/notes, but even that's been problematic. (Should notes only be listed underneath the items to which they're attached? What about freestanding notes? What if you want to browse your notes - what's the intuitive interface for toggling between modes?)

A hierarchical data model raises huge interface questions which I haven't really seen solved anywhere; that may just mean that Zotero's going to be the app to do so, but it's going to be a long slog (we were lucky that Apple had spent huge amounts of time refining the iTunes interface to the point where we could just kludge many of its conventions), especially if we're committed to creating software that's intuitive and understandable not just for us power users, but for the broader population of researchers who are less willing to approach a more outside-the-box interface.

*Sharing*

Bound up with many of the conversations around data models has been a second, important-to-distinguish issue: user-created fields/types. Right now, the Zotero data model is immutably fixed; there are a set number of item types, each with set fields. We're still cleaning these up and adding new ones (the "Blog Post" entry for example, has a "Website Type" field which is about as useful as the human appendix, and which shares a vestigial explanation as the "Blog Post" item type grew out of the "Webpage" item type).

Flat data models lend themselves more easily to data exchange - if the ontology of types/fields is the same across all Zotero users, I can share my library with you seamlessly. If I wanted to customize things a bit, I could do what Endnote does and add in a few "Custom Fields" which are essentially containers for any extra information I might want to tack on at the end of an item. There's a reason why all the popular social software sites (del.icio.us, cite-u-like, flickr, youTube, etc.) operate with flat, immutable data models - to make different users' collections interoperable, you've got to have the same ontology.

If it's not clear, interoperability is extremely important to us; we're planning a server that'll allow users to share, exchange and collaborate on Zotero collections, and we're incredibly excited by the possibility of visualization, recommendation and other newfangled bells and whistles that will transform the way that we as scholars relate to both our research and our colleagues. However, this only works if the data is interoperable.

Now, I'm not trying to conflate user-defined fields with flat vs. hierarchical data models - they're two separate issues, and should be treated as such. However, if you want to enable users to tailor the software to their particular needs (which we do) because you think they know better than you what they'd need (which we do), a flat data model is *much* easier to deal with, because you're talking about a few extra, easily-excludable fields. If you start letting users create their own item types, interoperability becomes problematic, and if you enable the fitting together of existing and user-created item types together in parent-child hierarchies, the likelihood of normalization decreases exponentially. It's an awkward analogy, but you go from fitting square pegs of different lengths into square holes to dealing with pegs and holes of any size and shape imaginable.

*Where we're at*

When we started building Zotero, we had a few pragmatic goals: to build a tool that was more open and robust than Endnote and other bibliographic reference managers, to make its interface accessible to as many users as possible, and to build it with an eye toward enabling social-software-like functionality. When we ran into the question of what sort of data model to use, we made a tough decision - though it might not seem like it, we think *a lot* about the data model, and we spent quite a bit of time debating whether to go with a hierarchical model from the start.

Ultimately, we decided to start off with a flat, fixed model, and focus our energies on the infrastructure that would allow users to get data into the system, as well as mechanisms for getting data back out (either in citations or via utilities, about which I'll post more elsewhere). We knew from the start that this was a compromise, and one that we'd have to address later on; right now, our plans are to get version 1.0 out the door, then tackle this problem headlong.

There are two distinct pieces here: the hierarchical data model and user-defined types/fields. The current plan is to do hierarchical types first, with a defined ontology of types and fields. That's going to occupy a good amount of time, less for the implementation in the software itself (that'll be a pretty straightforward process) than for the interface design; figuring out exactly how to represent and work with hierarchical items isn't easy, and we've got a lot of work to do.

On the second piece (user-created types/fields), we want these as badly as everyone else, but the moment that Zotero opens up the data ontology to users we irrevocably lose the ability to cleanly aggregate and share item data. One of the basic rules of software development is that you can always add features but never take them away, so we're going to tread cautiously and carefully to make sure we enable users to customize their personal data without threatening the interoperability of Zotero collections.

With all of that said, this would be a good place to start a discussion on the strategy as I've laid it out, as well as the more focused questions raised elsewhere on the forums and dev list. I've also created a page on the Zotero Trac wiki (https://www.zotero.org/trac/wiki/HierarchicalOntology) on which to flesh out a new, hierarchical model - if we can come to a consensus of what this should look like, we'll be able to roll it into the post-1.0 Zotero much more quickly.

Josh, thanks for your detailed post, I fully agree with you that a hierarchical data model poses many interface & compatibility issues which are far from being trivial. Your post very much describes the reasoning why we haven't adopted a hierarchical model for refbase yet (although this was discussed with Bruce and others and had been planned since a long time, see e.g. http://sourceforge.net/forum/forum.php?thread_id=1135397&forum_id=218757).

While a hierarchical model has many advantages, a flat model has definitively some advantages that go beyond easy implementation & sharing. As an example, a flat model allows different users to use different variants for a particular container, e.g. while some of my colleages want (or need) to cite a non-english book or journal with its original (say, german) title, others always cite the title translated into English. Yet another (french) researcher may need to cite the very same resource with a french translation of the title (e.g. for a french grant application). Surely, these things can be all modelled with a hierarchical model but they require *a lot* of thought. In addition, each use case has to be accounted for in the interface. Contrastingly, a flat model lets users use custom container values for their own resources.

That said (and not thinking of any implementation details), I'd personally favour a solution that would let me choose on item-level whether I'd like to link a particular resource (like a book chapter) to its container item OR whether I'd like to enter custom container values for that particular resource. I.e., by default hierarchical relations would be encouraged by the software but I'd also be able to "unlink" individual items (of course, a link to a container item could still be stored by the software for unlinked items). With this in mind, we've thought about using a flat model along side to a hierarchical model for the refbase project. This would mean partially redundant data in the database but would allow us to use the best model for a given use case.

Just some thoughts... I very much appreciate this discussion. Matthias

Josh,

A few comments first:

"As much as anything, this model grows out of the bibliographic citation; according to MLA or Chicago or whatever, there are a certain number of citation types, and if you're writing software that is ultimately about facilitating bibliography-creation, then the smart thing to do is to just implement those item types in a flat model."

I thank you for laying this out and starting the discussion, in part because it highlights some differences.

This probably stems from us handling different ends of the development puzzle (you more about GUI, me about formatting configuration), and from the fact that as a user, I have quite demanding needs (in fact, not unlike an historian!).

First, the style guides are not at all as focused on types as you assume. A whole lot of them are about classes of documents and relationships. They use types as illustrations.

Second, while it might be easier for you to code, flat models also actually don't match a whole lot of real world citation needs. It is only a convenience for programmers (and because relational databases and RDF didn't exist when most current bibliographic applications were designed). The thread about primary sources illustrates this really well, so I won't repeat those arguments.

Third, if you assume a flat list of types that can be extended, how do you handle a) data exchange, and b) formatting? In the latter case, do you require a style author to create templates for all 65 or so of the types that one can identify in Chicago?

That's been my focus. CSL is designed around a mix of a hierarchical and flat model. Part of the reason for this is that the styles become both more compact and easier to debug, as well more reliable. If there is no template for a given style, the formatting system is supposed to use one of the three generic fallbacks. These are determined based on the structural characteristics of the data.

"The problem, though, comes when you want to do more interesting things with your data than simply create references; there are *relationships* between items that aren't expressed in a flat model."

This is only one small part of the problem. It is about also the need to have both flexible and reliable data exchange and formatting.

"Now, I'm not trying to conflate user-defined fields with flat vs. hierarchical data models - they're two separate issues, and should be treated as such. However, if you want to enable users to tailor the software to their particular needs (which we do) because you think they know better than you what they'd need (which we do), a flat data model is *much* easier to deal with, because you're talking about a few extra, easily-excludable fields."

It might be easier for your programmers. But that's not to say that it's easier for formatting, or data exchange.

Here's the practical issue:

1. Zotero has its internal data model, and then another one it presents to its users
2. CSL has its own data model
3. the RDF model I designed -- and which you are partly using -- has still another data model, though designed in conjunction with CSL

It is critical to future interoperability of all these pieces for the three models to be in sympathy. A CSL formatter has to know what to do with a citation source. It has to know which template to use. Otherwise, the citations won't get formatted correctly. And to do that, the data models MUST be consistent.

Likewise with data exchange.

Now, I'd like to complicate this a little by pointing out (as Matthias does) that one need not necessarily chose flat vs. relational. There can be ways to have both. For example, maybe you use a little configuration language to setup a "type" for a GUI, but to keep a hierarchical structure for formatting and exchange. Just emphasizing this is no black-or-white choice.

First, thanks to Josh for initiating this discussion, and also for linking to that thread from last October, which helped answer some of my questions. This should be an interesting discussion.

One of the reasons that the inadequacies of biblio programs became my hobby horse is that before I entered academia, I worked a bit with MS Access and learned some of the rudiments of relational database design. At one time, I wanted to use Access to design a relational database that would handle my notes and citations - but that's another story. The point is that when I started shopping for biblio/notetaking software, I was shocked to find that it was unheard of for such programs to have an authors table, titles table, publishers table, etc. Instead, as you've all noted, the standard is one big table with lots of repetitive data and data entry.

These days, I do most of my academic writing with Nota Bene, which I like because of the sophistication of its word processor and the simplicity and user-friendliness of its text retrieval module, which is great for organizing research notes. And its biblio database module is adequate on its own terms, and very well integrated with the rest of the program. But I've also played around some with Library Master, which is the most user-customizable biblio program I've been able to find. Like everything else, it works on a flat, one big table model, but I found a way to make it create the kind of compound citations I needed in a way that approximated a relational database solution.

I'm pasting in below an old email I sent to the Nota Bene users list, and forwarded to the LM users list, describing this method. I think that the underlying concept (inserting record markers for "containers" within the "container" fields of other records) could be a fairly straightforward way to address many of my own complaints about flat biblio databases, and I suspect that one could create a user-friendly interface for it without too much trouble (along the lines that Matthias mentions above). It would also help stop the proliferation of record types that bdarcus points to: Instead of 100 different record types, we could have 10 document types, and 10 container types, and achieve the same (or better) result. Consider that the only difference between citations of a journal article and an anthology chapter is the "container." Have a single "titled short work" type for the actual items, and then a journal type and an anthology type as choices for the container. Lots of neat possibilities.

For the sake of making this thread more readable, I've replaced the full text of the email with the URL for the original:

http://www.hamline.edu/~wnk/notabene/2003/msg05806.html
(scroll down to the part labelled "Multi-level shortened citations")

Here's another URL for a more recent commentary of mine (more specifically directed towards improving Nota Bene, but relevant to the more general problem of achieving these complex citations without a completely relational database):

http://wnk.hamline.edu/mhonarc/notabene/msg00373.html

I hope you find these comments interesting.

It's hard to describe how delighted I was to see this discussion rubric pop up the very day I subscribed to the forum. The issues of hierarchy and type are at the core of a bibliographic data quest that I hope Zotero can help us solve. Please forgive me if what follows seems idiosyncratic; rather, I hope it will serve as an interesting use case. Note: I've had to divide my comments into multiple posts, since there's evidently a character limit on comments that's irritatingly not advertized in advance.

Context: Collaborative Ancient Geography

I direct the Ancient World Mapping Center's Pleiades Project, which aims to create an online workspace for ancient geography. Our springboard is a legacy data collection assembled by the American Philological Association's Classical Atlas Project over the twelve-year period that culminated in the publication of the Barrington Atlas of the Greek and Roman World (R. Talbert, ed., Princeton, 2000). With startup funding from the National Endowment for the Humanities, our 2-person development team is crafting the supporting web application on top of the Plone content management system. Plone provides us with a customizable framework for the creation of structured content types, scriptable editorial workflow and an extensible user portal. Our goal is a compelling, intuitive environment in which community approaches will combine with academic-style editorial review, to enable anyone — from university professors to casual students of antiquity — to suggest updates to geographic names, descriptive essays, bibliographic references and geographic coordinates. Once vetted for accuracy and pertinence, these suggestions become a permanent, author-attributed part of future publications and data services.

Issue: Bibliography in Pleiades

The Ancient World Mapping Center has developed a bibliographic database that contains over 4,000 records deriving from the bibliographic citations that supported each mapped feature in the Barrington Atlas. These records have been augmented with data drawn from MARC records and added by hand, so that they now include alternate and translated titles (each coded for language), holdings information, standard numbers and more. This database must be migrated from its current, stand-alone configuration to operate seamlessly within the Pleiades web application as live content. Not only do we wish to harness the community's effort to capture new bibliographic information, we also require that each suggestion for an addition, deletion or change to a name, place or location be supported by either an explanatory essay or a bibliographic citation. Obviously, the accuracy of these citations, and the ease with which they can be created, are key concerns.

To be continued ...

"For hierarchy, has there been any consideration of FRBR? Should there be?"

I've spent a lot of time looking into and prototyping FRBR-related data stuff. It's a killer data model that is really sound. But you pay for that in complexity. My conclusion is that it's too complicated for these use cases.

That said, I helped a bit with the FRBR RDF work that Ian Davis (from Talis) and Rich Newman did, and more recently I've been referencing it from my RDF model. I blogged about this here:

http://netapps.muohio.edu/blogs/darcusb/darcusb/archives/2006/09/11/plugging-into-frbr-killing-marc

In my data model, BTW, I have journal as a subclass of periodical, which is a subclass of collection. MARC is good to learn from, but less-than-ideal otherwise. Library people think about bibliographic metadata differently, reflecting different priorities.

Glossary:

XML (eXtensible Markup Language) - a way to tag plain-text with structured information

RDF (Resource Description Framework) - a standard (basically relational) data model for representing just about anything; builds on XML and the web (resources are named using URIs). Think of something like a relational database for the web. Well-suited to needs of Zotero.

CSL (Citation Style Language) - an XML language to describe citation styles; written by me, with some help from one of the Zotero guys, an used in Zotero. There's some interest in standardizing it and using it witin OpenDocument.

I mentioned this in one of the earlier threads, but to address CloudOfDust's question (and to clarify the discussion for others), the internal database structure used by Zotero is pretty much irrelevant to the issue at hand. How we store data internally is mostly an issue of disk space efficiency and performance. I'll just reproduce what I said on that other thread, as it's germane to this discussion:

There's nothing inherent in the current database design that prevents there from being dependencies/relations and automatically incorporating parent fields in dependent items without duplicating data. Normalizing publications out to a separate table might make a few things easier (and other things more convoluted), but it doesn't avoid any of the issues that make this difficult. You still need a way to select a parent/publication from within another item, still need to prompt the user for what to do on deletes, still need to determine whether the change to publication data in one item changes it in the parent, etc. And you need a mechanism for relating items in complex ways regardless of whether publication data exists independently.


The UI suggestions from Matthias above are a great start to answering these sorts of questions. More ideas along those lines would be most welcome.

Aside from UI issues, the other major requirement is a hierarchical map of reference types of the sort Bruce has been working on, which would allow not only for proper CSL fallback but also for certain internal logic within Zotero—for example, I would imagine that certain communication-based types might share the same logic for auto-generating a proper title field from the values of the creator types designated in the database as equivalent to "author" and "recipient" (whatever the type-specific names might be). A planning document that arises from this discussion would ideally include descriptions of such logic.

Bruce, does your RDF model account for actual fields or just types? (I don't see any mention of fields on the RDF schema page, so perhaps you haven't integrated them yet.) In particular, I'm wondering whether child reference types inherently contain all the fields of their parent types (though perhaps with different names).

(CloudOfDust: We already do exactly what you suggest for creators and creator types, since, among other things, they require many-to-many relationships with items. Separating out the other fields into separate tables wouldn't provide any benefit and would simply make the code much more convoluted, as it would require adding custom logic for every single field (and there are quite a lot of fields). One thing that would be appropriate, from a data normalization perspective, would be to add an itemDataValues table with all the unique values and simply reference them in itemData—it would make working with the data somewhat more tedious and possibly make certain operations slower, but the overall increase in storage efficiency (at least for large libraries) would probably make it worth it. But, again, a separate matter from the issues at hand.)

Re: this question:

"I'm picturing related tables for "people," "titles," "journals," "publishers," "places," etc."

As Dan says, it's all really a question of trade-offs, but when I've worked on such models, I've tended to have tables for "agents" (includes publishers), "collections" (includes journals, archival collections, etc.), and "resources." I've always had "title" as a column for resources and collections, rather than a separate table (seems like overkill).

OK, Dan.

Validation in RDF is different than in XML. See:

http://danbri.org/words/2005/07/30/114

In short, RDF has this "open world assumption" which expects merging of data, and so to say something is "invalid" per se makes no sense.

But I'm pretty sure that subclasses inherit the characteristics of their parents, including restrictions about the occurrence of properties (if you define them of course). In other words, in answer to your question "yes, the former."

To me an ideal model to follow for what we need here is DOAP: it's good, clean RDF, and there's a nice RELAX NG schema that validates nice, clean, XML (that is RDF, but most people wouldn't know it).

CloudOfDust — a couple quick things:

Re: "Cambridge, MA": Just to be clear, my point was that having a separate "Place" table in the database is wholly unrelated to correcting "Cambridge, Mass." to "Cambridge, MA" universally. That can be done at the moment in one line of SQL. It could be done for items with specific item types, for items created in the last two weeks, for items from a particular publisher...in other words, for items matching any arbitrary set of criteria. It's just a question of writing the SQL. But the current database design is quite capable of all of that. It just requires implementing the batch editing and search/replace features to let you do it in the interface. How it's stored is more or less beside the point.

Re: "auto-generating a proper title field": I meant the latter. By title field for the communication-based types, I was merely referring to what is displayed, say, in the items list.

Yes, your first paragraph is correct.

For the second, you don't need separate tables to have parent/child relationships, and having separate tables doesn't make the edit/delete issues any easier, since the difficulties are in the UI. The parent data wouldn't be stored with the child—most simply, you'd just have a parent column in the items table, and on retrieval, the data layer would grab the data for both items.

The other, more fundamental, issue is that, in our case, parents and children aren't necessarily different types of entities as far as an editing interface is concerned. You want to create a record for an edited book the same way you want to create a chapter—making them fundamentally different entities would just add unnecessary complexity to the code.

In a relational database, data doesn't need to be in separate tables for you to relate it.

Yes, but again, data duplication is still a completely separate issue from what we're (theoretically) discussing. As I said above, a separate table containing all unique values, linked to itemIDs in the itemData table, would remove all redundant data. It'd save a few bytes per associated item for longer field values (the difference between the integer and the string). It'd save no space whatsoever (or take up more space) for fields that were already integers or short strings. But you certainly don't need separate tables for each type of field or item type. And even for those rows in itemData linking the unique values to the individual items, you certainly don't need to duplicate them in the child items, as seems to be an assumption.

It would be beneficial to get back to the issues of UI design and data mapping.

OK, my two cents on Dan's UI questions (probably duplicating some of what CloudofDust says):

"You still need a way to select a parent/publication from within another item"

Auto-complete?

I'm imagining the more complicated case of entering book chapters. Let's say I start not with having an already-entered edited book, but just decide to enter the chapters.

So I choose a "chapter" type and get the fields. I get to the "book title" field and get auto-complete options. Since I've not yet entered the book, I won't see what I'm looking for. So I then enter the additional fields.

Alternately, perhaps the first field is the ISBN, and the data can then be auto-populated by pinging a server (I suppose a title field could do that too).

"still need to prompt the user for what to do on deletes"

I'd say deleting a child should never delete a parent. But deletion of a parent should bring a serious warning message.

"still need to determine whether the change to publication data in one item changes it in the parent"

Maybe if a user selects already-entered data (an existing parent record) the UI for those fields somehow indicates that, and one needs to explicitly choose to edit it (maybe by switching to the parent entry)?

Just to brainstorm on various UI issues, to follow from what bdarcus and CloudofDust are proposing--in theory would it be possible to set up the following interface? (this doesn't address hierarchy models, only UI).

(the following is relevant to this issue: "You still need a way to select a parent/publication from within another item")

Let's say I'm entering a new item--a letter published in a book:

Theodor W. Adorno to Walter Benjamin, 7 March 1938, in The Complete Correspondence, 1928-1940, ed. Henri Lonitz (Cambridge, Mass.: Harvard University Press, 1999), 240-242.

I will start by entering the fields available in the current "Letter" Zotero item type: Author, Recipient, Date of Letter (fields necessary in other cases would include type of letter, place sent, and place received).

Then instead of a repository field for archive I would get a line where i could choose possible parent item type (let's call it a "container" type): Repository (as in archive), Book, Journal, Magazine, Newspaper, etc. The container type field would be a pulldown field like the menu currently reserved for author type. I choose "Book".

Then I would start typing a title, which would autocomplete from the book title field. If the book already exists, typing "The Complete Correspondence, 192..." would bring up the full title. If the book doesn't exist, I enter the entire title.

I then hit the return key. The interface expands to reveal all the fields for the "Book" item type. If the book already exists in the database, Zotero shows data from existing fields. (If two identical titles exist, I would get a prompt to choose one) If not, I go ahead and enter the new data. (A new item then should be created in Zotero just for that book.)

When I come back to this item later, all I see in "Info" is the letter fields plus the title for the book in the "container" line. Clicking on the title reveals all the fields for the book, which I can then edit.

This would be great interface for entering letters, interviews, and artworks, because all of these have few item-specific fields (i.e. type for letter, medium for interview, medium and size for artwork) but all of these can be found in archives or published or reproduced in books, journals, magazines. This method could also be applied to book sections & articles in periodicals.

This does not resolve the question of what I'd do if I want to also record archival information for this same letter--do I enter it as a separate item or (better yet) is there a second or third "container" option available within the same record (just as I can now add more authors by clicking the "plus" on the right of the author's name)?

This interface also calls for a separate "Archival Collection" item type, which I think Zotero needs anyway because researchers may need to record information about a collection or archive in advance to actually doing there and entering specific sources.

On these issues:

"still need to prompt the user for what to do on deletes" and "still need to determine whether the change to publication data in one item changes it in the parent"

I think users should be able to add and edit parent items directly from the child record, with some warning ("editing this field will alter the container item and all linked items")--it seems easier than opening a separate window. They also should be warned on deleting the item that has "child" items in the same way they are now prompted about deleting notes and files--just as currently when we choose not to delete a note we just end up with a standalone note, we'd end up with a letter/interview without a "container" item linked to it.

By the way, sometimes letters do have titles--newspapers and magazines often title their letters to the editor, and business memos often include a "subject" which would be useful to enter into a "letter title" field.

Ugh, I just spent 30 minutes on a reply, but it got eaten! I don't have the energy to repeat it all, but will boil it down to:

1. we need to decide the policies for when something becomes a formalized "type."

I suggest we focus on a smart core with some hierarchy, and leave room for customization with a "genre" or "type" column/property. Some of your types are too specific in my view.

Maybe another wiki page for this?

Also, it's important to me for the RDF that we have a hierarchy of types (per my blog post).

2. I think ditching the collection and using the "Ancillary item types" is a bad idea. Why are you doing this again?

Thanks Josh and Sean for working on this! Like Bruce I have problems with the distinction between ancillary item types and actual item types. There are use cases where it would be handy to enter the ancillary item types as actual items (i.e. stand-alone database entries). So I would simply take away this distinction and treat all item types as equal, but then allow to relate these item types freely with each other.

Also, I must admit that, as a third-party developer, I'd like to agree upon a hierarchical model that would not only work for Zotero but also for other bibliographic applications.

That said, and not thinking of any Zotero implementation details, please allow me to go back to a more conceptual model for brainstorming purposes. Personally, it helps me to think of relationships like this:

First of all, all items are on the same level and can be *freely related* with each other (this is very important if the model wants to address all different kind of needs).

Speaking of relationships, I think of "classes", "subclasses", "items" and "item properties" (more about properties below).

In case of "classes", I'm thinking of the main basic elements that occur in every bibliographic citation/reference. "Subclasses" are contained within these basic classes and would usually work as fallback elements when generating citations. "Items" would be contained within subclasses and would (in case of "resources" & "events") represent the actual database entries (think Zotero item types).

So, following this markup scheme:

class:
- subclass: item1, item2, item3, ...

I think of these classes, subclasses and items:

Agents:
- Person: author, editor, translator, inventor, contributor, recipient, ...
- Organization: publisher, authority, ...
- ...

Ressources:
- Collection: periodical, series, archive, internet site, proceeding, ...
- Document: article, book, section, chapter, thesis, statute, map, image, blog, ...
- Communication: letter, email, instant message, interview, ...
- Broadcast: radio, television, podcast, ...
- ...

Events:
- Conference
- Legal Case: brief, decision
- Hearing
- Expedition
- ...

Places:
- Country
- City
- ...

Dates:
- Year
- Month
- Day
- ...

Note that there is no dedicated "Collections" class, since collections are IMHO simply a resource that can contain other resources, so basically collections are resources as well. Therefore, I'd consider them as a subclass of the "Resources" class. Take a book as an example - would this be a collection or a (document) resource? The answer is, of course: it depends. A book can be considered as a stand-alone document as well as a container of book chapters -- or, as a container of book sections which in turn are containers of book chapters. I.e., depending on the situation, a book can be considered as a collection or as a document, but it's in any case a resource.

Speaking of "items", all items could have multiple "properties", such as "title", "name", "language", "locator", "identifier" or "descriptor". A qualifier (i.e. the elements after the colon below) could specify in more detail the nature of that property, e.g.:

- name: given, family, suffix, display, sort, ...
- title: long, short, abbreviated, translated, alternate, descriptive, ...
- language: fulltext, summary, ...
- locator: volume, issue, pages, edition, code, patent number, ...
- identifier: url, doi, issn, isbn, pmid, lccn, archive id, local call number, ...
- descriptor: keyword/tag, category/group, ...
- ...

In addition, a universal "type" property could always specify in more detail the actual nature of the item. For example, a periodical could have one of these type properties:

- journal
- court reporter
- magazine
- newspaper
- ...

Or a thesis item could have one of following type properties:

- dissertation
- master thesis
- bachelor thesis
- ...

And all place and date items could have a type property such as:

- published
- presented
- sent
- received
- ...

In theory, all items from all classes could be related freely with each other to form a citation that suits the user's needs.

However more practically, the software could provide "relationship templates" for typical use cases (say, a letter within an archive, or a book within a series, etc). This would be the equivalent to Zotero's current item types. The user would select one of these templates from a dropdown menu. Zotero would prefill the edit mask with all necessary fields but would visually distinguish between item-specific and container-specific fields. As discussed by others ealier in this thread, the contents of container-specific fields would make a separate database record.

Note that it should be also possible to nest a container within a higher-level container. This would be required to properly cite e.g. a book chapter within a book within a book series.

I hope that my thoughts make sense to you.

CoD- "Reproduced in" is not an item type. It's just a UI element pointing at a back-end association between one item and an another. Functionally, this means that before entering a letter, you'll first want to enter the book where the letter is reproduced. Then letter item details, and click "Reproduced in" to point back to the parent item.

"Reproduced in" should never point to an ancillary type, since a journal article is not likely to be "reproduced in" an issue. That's its original form.

On the question of "ancillary" vs. other item types, I made a mistake in not being clear - this is a UI distinction, rather than an ontological one in the data model. From the data perspective, there's no difference between a journal issue and an article; the difference is that in the Zotero interface, you'd never be able to create a journal issue on its own (we'd be hacking in a bunch of UI decisions like this in order to make the software more accessible to people who don't think about this stuff as much as we all do).

On the "collection" type, Matthias hit it on the head when he said above: "collections are IMHO simply a resource that can contain other resources, so basically collections are resources as well." That's how we see it, and the "collection" type seems an unnecessary elaboration.

"On the "collection" type, Matthias hit it on the head when he said above: "collections are IMHO simply a resource that can contain other resources, so basically collections are resources as well." That's how we see it, and the "collection" type seems an unnecessary elaboration."

But the notion of collection is significant both WRT to the GUI (as you both mention), the citation formatting, and data exchange. If you were to change "ancillary types" to collection and be more precise about it, would you really be deviating from your intentions?

FWIW, I created the main collection class becuase that notion is commonly used in bilbiographic data, and because it groups together a number of similar structures: series, archival collections, tv and radio shows, web sites, etc.

Matthias --

"As per my previous post, I'd prefer if item types would merely be a pre-made set of relationships."

Yes. Well, look at how I do it in CSL. I have a convention that says you concatenate the primary level type with its container type. So you can do "article-periodical" or "article-magazine" and so forth. I could imagine the same for these assembled GUI types.

Please note that I didn't ditch the "collection" hierarchy entirely (I consider it useful), I was just emphasizing that a "collection" can also be a valid resource (similar to an article, letter, etc). Therefore, all of its members should be treated equal compared to members of the "document" tree (i.e. all these elements being "items" in my view).

Josh, I understand (and like) the idea of a communication being different from a document. Also, I think that a broadcoast (one to many) is different from a communication (one to one).

Bruce said:
"For example, how would you deal with an interview published in a book, or broadcast on the internet?"

This is a valid point and it highlights my point of a book being a "collection" OR a "document" depending on the situation. With this in mind, I'd argue that the relationships between items and its higher-level categories (subclasses) should not be dogmatically fixed. Why shouldn't it be possible to relate an interview to a book in one instance and to relate it to an internet broadcast in another instance?

How about if we'd merely regard subclasses as some means to provide meaning between items and classes, i.e. for example, a book could belong to subclass "collection" OR subclass "document" depending on the situation? (just thinking out loud here) Would this be feasable in an RDF model?

On the "reproduced in" idea, it seems to me that the relationship between a journal article and its parent "Journal" item is different from that of a journal article and a book in which it's reproduced; this might just come down to a Benjamin-like notion of the "aura" of originality, but in every case we could come up with, there is an original and authentic item (of course, we're thinking as historians, so that might also play a role here). It seems useful to avoid conflating an authentic original with its reproductions - we cite them differently, and treat them differently in the context of research.

As for "collections", if a book can be either a collection or a document depending solely on perspective, then it fundamentally can't be described in an objective way (which is crucial to data portability and interoperability). Since one could *always* make a gestalt shift between collection/document, I don't see the usefulness of maintaining them as somehow ontologically distinct. Why not just level the distinction between them, and simply talk about items that can also be pieces of other items (and in turn be composed of smaller items as well)?

I've just had a long post erased by a connection blip. A pox on fuzzy wireless connections.

I had two points, I think: first, re: Josh's comment: "It seems useful to avoid conflating an authentic original with its reproductions - we cite them differently, and treat them differently in the context of research."

We cite them differently? The citation gives the information for the item and tells the reader where to find it. I don't see how it makes any difference to the bibliographic software whether the "container" is the original or a reproduction.

Second, at some points in this discussion we've misunderstood each other because distinctions between the data structure, the GUI, and hierarchies of entry types aren't clear. It seems to me that these are three quite different (though obviously interrelated) things. My dearly departed post spelled out how I saw the differences among them, but I won't try to recreate that. The key thing is that there seems to be widespread support for the idea of a hierarchy-free data structure - records are records, and books can be parents, children, or stand-alone items, depending on how they're related to other records in a citation. The framework for organizing citation styles and entry types is more rigidly hierarchical - the category "communication" contains sub-categories "letter," "email message," etc. These hierarchies of types will help expedite the creation of new citation styles and entry types, and eventually make it possible to open these up to user customization without devolving into utter chaos. The GUI will also be hierarchical, but in a different sense - it will display the hierarchical relationships between items and their parents (and potentially grandparents).

I hope that's clear - a more carefully composed version is gone forever.

CoD: "We cite them differently? The citation gives the information for the item and tells the reader where to find it. I don't see how it makes any difference to the bibliographic software whether the "container" is the original or a reproduction."

When I'm citing an artwork, I do need to indicate in the citation whether I'm working off of the original or a reproduction; this is part of how I lay out an evidentiary chain for my argument. The question of reproduction *does* matter, because if I'm not working off of the original, errors or noise might have been introduced in the process of reproduction, and a later reader needs to be able to track down the particular reproduction on which I based my claims (hence the need for a particular kind of "Reproduced in" relationship)

CoD, on your second point, I think I agree: in principle, the more flexible the model the better, but the implementation of these concepts in a particular piece of software tailored to a particular set of practices (i.e. Zotero) will hard-code much more rigid and explicit hierarchical relationships into both Zotero's UI and internal data model (neither of which preclude exporting said data in the more abstract and generic RDF form as needed). Because my first concern is Zotero (with the broader utility of Zotero data in other contexts a close second), I tend to slip back into the latter two modes, rather than instinctively staying at the more generalized level.