Drat, OK, will do.

Still a 409.

Might be something like number of files in the directory?

Markus: Thanks for following up. No, there's no hard limit currently—it's just the number of files the user is syncing * 2.

Splitting up into smaller folders is a possibility, but it's unlikely we'd be able to get to it in the near future, I'm afraid.

Ah that is a pity...Markus is there no way to circumvent the 1000 file limit specifically for zotero users? I think there are a lot of zotero users that are using your service and many of them will hit the 1000 file limit (I did too, although I am just a graduate student).

Well, I am another of the happy MyDrive users who has hit the "1,000 files" limit. If I understand correctly the issue with MyDrive, the only way to solve this issue is to have the possibility of several independent folders, each with less than 1,000 files each. For example, it would be great if Zotero just requested that the user creates a zotero folder (as it is now), but then once a limit of files per folder is hit, a new "zotero1" folder is created, then "zotero2", etc. It would be really a pity to stop using MyDrive, it is a really great service!

Dan, thanks for your answer, I can also understand your position.

Dan, you're probably aware of this but limiting the folder size would be a good idea in general, with thousands of files the PROPFIND response grows to several megabytes, just for listing a folder.

Ah, I just found this thread. I'm sorry for cross-posting (http://forums.zotero.org/discussion/14149/zotero-webdav-saves-all-files-in-one-folder/#Item_1).

Read the code.google.com link below to see why the files are being split across multiple folders: it makes a lot of sense from the server perspective (loading several thousand files into memory within PHP is very server intensive and impractical for shared hosting services). Since zotero already generates the filenames, why not automatically generate subfolders to limit the number of files per folder?

http://code.google.com/p/sabredav/issues/detail?id=76&q=zotero&colspec=ID%20Type%20Status%20Priority%20Component%20Summary

Dan, just another point to consider: filesystems have arbitrary limits too, and even though they're usually ridiculously high there are still always performance issues involved with folders containing thousands of files. So Zotero could benefit from split folders in general, not just with our own WebDAV server.

Well, you really didn't have to look that far

http://forums.zotero.org/discussion/69510/

Like I said there, so far, so good.

hello
we from softronics ave opened a thread:
http://forums.zotero.org/discussion/14156/split-the-data-in-diferent-folders/#Item_1

There are apparently good technical reasons why the amount of files in folders should be limited, but Dan et al. I am sure are aware of this, and I trust they will get round to it if/when it is necessary.

The "best" is probably the Zotero storage, it is made to work with it!

I do not need to access my papers from any browser, so I do not need the Zotero storage facility, but my library is still growing. I have therefore gone the "sync" route. I copied my zotero folder to "My Documents" (with FF closed), then pointed Zotero to it (Preferences->Advanced->Data directory location). You can then sync that to the cloud using any number of providers: Dropbox is very popular but there are plenty others (box.net, sugarsync, jungledisk...). I am experimenting with Spideroak, which seems very interesting. Uploading right now, at about 1 Mps. I'll report here on performance later.

Warning: shameless self promotion: if you click here I get extra space. https://spideroak.com/download/referral/76edd4430c757219830ea3d16cf23420

Hi Dan (Stillman),

I realize that you "pretty much said all [you] can say on this in [your] previous post." I just wanted to point out that this is an issue that other people are talking about, and thought the discussion on the SabreDAV development site might be relevant for you.

I appreciate that you guys can't drop everything and attack this issue immediately, but you are going to hit a point in the near future where the number of files in a single directory becomes a major performance-limiting issue. There are really simple workarounds for this, such as creating subdirectory names based on a short hash (perhaps to just 2 alphanumeric digits) of the already generated filename, and stuffing files into said subdirectories. I have only entered 222 items into my library, and already have 431 items in my /zotero/ WebDAV folder. This will grow dramatically as my group members add their references. It's already causing a tremendous slowdown in the HTML WebCT interface at my university, since it needs to generate an internal representation for all of the files in the folder whenever a single file is queried. Likewise, this will eventually become problematic in the local filesystem.

I look forward to when you guys have time to modify the zotero behaviour. Perhaps I'll make the required changes for my own use, and send you an updated copy. However, I'm a bit worried about messing up my own library while playing with the zotero code.

Thanks again, this is an awesome piece of software.

Dan, I assume Zotero depends on PROPFIND requests? If not a work-around for Zotero might be possible.

toupeira: I actually spoke too soon here.

Zotero does not use PROPFIND requests for the actual syncing, specifically because that would make for very slow sync performance. (Not because of any inherent slowness of PROPFIND, but just because pulling a file list on every sync would be a bad design.)

It uses PROPFIND only when verifying the server and when purging orphaned files. The former uses "Depth: 0", so it's just a query on the folder metadata. The latter uses "Depth: 1" to get the full file list. But it looks like the latter isn't even hooked up in the code right now, so I believe Zotero clients shouldn't be making any non-"Depth: 0" PROPFIND requests currently.

Does that get us anywhere? Rather than preemptively splitting up directories, could you just, say, block "Depth: 1" and "Depth: Infinity" PROPFIND requests if there are more than a given number of files in the folder?

So if the "Depth: 1" code isn't hooked up does this mean Zotero doesn't purge orphaned files?

It purges deleted files, because it knows what those are. It doesn't currently purge orphaned files—files it has no record of that happen to exist on the server. Those shouldn't exist under normal usage.

We're currently looking into several performance improvements and will hopefully be able to raise the limit in a few weeks, and maybe also implement this work-around for Zotero.

Sounds good. I'd note that this wouldn't really be a workaround for Zotero—it'd simply be blocking the actual calls that were affecting performance (which Zotero doesn't make) rather than splitting up directories in anticipation of such calls.