Unexpected file upload status 409

[JavaScript Error: "[Exception... "'Unexpected file upload status 409 in Zotero.Sync.Storage.WebDAV._onUploadComplete()' when calling method: [nsIStreamListener::onStopRequest]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "<unknown>" data: no]"]
report id 1167518090

hmm
about to generate a debug id.

Jon.

OK, debug id D1916169234

Ubuntu 10.04 firefox 3.6.8
Got plenty of room on my webdav server still
«1
  • 409 Conflict isn't something Zotero returns—it's from your WebDAV server—so you'd have to check with your WebDAV provider.
  • Drat, OK, will do.
  • edited August 31, 2010
    Hi Dan, don't know if this is relevant or not - I tried verifying the server and got this:

    "An unknown error occurred. Please check your file sync settings or contact your server administrator."

    The Debug ID is D393813943.

    My webdav provider is mydrive and I can upload files through their interface, no problem. I've contacted them, so will wait to see what they say
  • Still a 409.

    Might be something like number of files in the directory?
  • edited August 31, 2010
    Hello there,

    I'm a developer working for Jon's WebDAV provider :) He contacted us about this issue, and the cause is indeed that we started limiting the number of objects per folder today to max. 1000 (to avoid certain performance issues).

    Is there a hard limit on the number of files Zotero creates, or does this depend on the number of objects the user syncs with Zotero?

    Would it be possible to split it up into smaller folders?


    Cheers,
    Markus from mydrive.ch
  • Markus: Thanks for following up. No, there's no hard limit currently—it's just the number of files the user is syncing * 2.

    Splitting up into smaller folders is a possibility, but it's unlikely we'd be able to get to it in the near future, I'm afraid.
  • Hi Markus and Dan, well that explains it then. Drat, drat and double drat. Well, I'll have to look for a different webdav supplier then, which is a pity as till now mydrive had been rock solid.
  • Ah that is a pity...Markus is there no way to circumvent the 1000 file limit specifically for zotero users? I think there are a lot of zotero users that are using your service and many of them will hit the 1000 file limit (I did too, although I am just a graduate student).
  • Dan, I assume Zotero depends on PROPFIND requests? If not a work-around for Zotero might be possible.

    vogeltje, if enough MyDrive users complain here maybe they'll put more priority on reworking Zotero to use multiple folders :)
  • Well, I am another of the happy MyDrive users who has hit the "1,000 files" limit. If I understand correctly the issue with MyDrive, the only way to solve this issue is to have the possibility of several independent folders, each with less than 1,000 files each. For example, it would be great if Zotero just requested that the user creates a zotero folder (as it is now), but then once a limit of files per folder is hit, a new "zotero1" folder is created, then "zotero2", etc. It would be really a pity to stop using MyDrive, it is a really great service!
  • Dan, I assume Zotero depends on PROPFIND requests?
    Yes, it does.
    vogeltje, if enough MyDrive users complain here maybe they'll put more priority on reworking Zotero to use multiple folders :)
    Not in the immediate future. We provide Zotero File Storage, which is fully supported, and we support properly configured WebDAV servers without arbitrary limitations. (I understand this was done for performance reasons, but it's still arbitrary from the perspective of a WebDAV client.) We don't have the resources to do a major and sudden reworking of functionality for the sake of one third-party provider. So MyDrive users are out of luck for now.
  • Dan, thanks for your answer, I can also understand your position.
  • I also am a mydrive.ch user who no longer can sync his archive.

    It would be terrific if Zotero would change its storage to limit the number of files in a single folder as I previously ran into this limit with another WebDav provider.
  • Dan, you're probably aware of this but limiting the folder size would be a good idea in general, with thousands of files the PROPFIND response grows to several megabytes, just for listing a folder.
  • One more here :(

    I've looked around a bit but can't find an alternate WebDAV provider. Syncing was super-fast with MyDrive!
  • Ah, I just found this thread. I'm sorry for cross-posting (http://forums.zotero.org/discussion/14149/zotero-webdav-saves-all-files-in-one-folder/#Item_1).

    Read the code.google.com link below to see why the files are being split across multiple folders: it makes a lot of sense from the server perspective (loading several thousand files into memory within PHP is very server intensive and impractical for shared hosting services). Since zotero already generates the filenames, why not automatically generate subfolders to limit the number of files per folder?

    http://code.google.com/p/sabredav/issues/detail?id=76&q=zotero&colspec=ID%20Type%20Status%20Priority%20Component%20Summary
  • I pretty much said all we can say on this in my previous post.
  • Dan, just another point to consider: filesystems have arbitrary limits too, and even though they're usually ridiculously high there are still always performance issues involved with folders containing thousands of files. So Zotero could benefit from split folders in general, not just with our own WebDAV server.
  • All right then, for us 'USERS' who still want to sync NOW, we either need a grace period from MYDRIVE until zotero will split files into folder, or we just need to leave NOW.

    Nico_sub -- or any other 'filed-out' ZOTERO lover -- please tell us, if you found a trustable alternative, even if we need to pay.
  • edited September 2, 2010
    Well, you really didn't have to look that far

    http://forums.zotero.org/discussion/69510/

    Like I said there, so far, so good.
  • another one who is no longer able to sync... :(

    Guess I will have to look for another server
  • hello
    we from softronics ave opened a thread:
    http://forums.zotero.org/discussion/14156/split-the-data-in-diferent-folders/#Item_1
  • edited September 2, 2010
    toupeira, gamerakel, and others: There's no point repeatedly trying to convince us on the merits. We can probably make this change eventually, but there's simply no way we can work on it right now. That's all there is to it.
  • There are apparently good technical reasons why the amount of files in folders should be limited, but Dan et al. I am sure are aware of this, and I trust they will get round to it if/when it is necessary.

    The "best" is probably the Zotero storage, it is made to work with it!

    I do not need to access my papers from any browser, so I do not need the Zotero storage facility, but my library is still growing. I have therefore gone the "sync" route. I copied my zotero folder to "My Documents" (with FF closed), then pointed Zotero to it (Preferences->Advanced->Data directory location). You can then sync that to the cloud using any number of providers: Dropbox is very popular but there are plenty others (box.net, sugarsync, jungledisk...). I am experimenting with Spideroak, which seems very interesting. Uploading right now, at about 1 Mps. I'll report here on performance later.

    Warning: shameless self promotion: if you click here I get extra space. https://spideroak.com/download/referral/76edd4430c757219830ea3d16cf23420
  • I copied my zotero folder to "My Documents" (with FF closed), then pointed Zotero to it (Preferences->Advanced->Data directory location). You can then sync that to the cloud using any number of providers
    This is not recommended. Third-party syncs do not have a way to properly mirror your zotero.sqlite database that is in your zotero folder. You can run into unresolvable sync conflicts (leading to data loss) and even database corruption.

    Zotero data sync is free, so there's little reason not to use this for your database.

    You can more safely sync the storage subdirectory, which contains the static PDF/web snapshots using a third party sync. But the supported Zotero file storage and WebDAV mechanisms work quite well.
  • Hi Dan (Stillman),

    I realize that you "pretty much said all [you] can say on this in [your] previous post." I just wanted to point out that this is an issue that other people are talking about, and thought the discussion on the SabreDAV development site might be relevant for you.

    I appreciate that you guys can't drop everything and attack this issue immediately, but you are going to hit a point in the near future where the number of files in a single directory becomes a major performance-limiting issue. There are really simple workarounds for this, such as creating subdirectory names based on a short hash (perhaps to just 2 alphanumeric digits) of the already generated filename, and stuffing files into said subdirectories. I have only entered 222 items into my library, and already have 431 items in my /zotero/ WebDAV folder. This will grow dramatically as my group members add their references. It's already causing a tremendous slowdown in the HTML WebCT interface at my university, since it needs to generate an internal representation for all of the files in the folder whenever a single file is queried. Likewise, this will eventually become problematic in the local filesystem.

    I look forward to when you guys have time to modify the zotero behaviour. Perhaps I'll make the required changes for my own use, and send you an updated copy. However, I'm a bit worried about messing up my own library while playing with the zotero code.

    Thanks again, this is an awesome piece of software.
  • you are going to hit a point in the near future where the number of files in a single directory becomes a major performance-limiting issue
    The only reason people are discussing this now is that MyDrive made a change on their end. People have been using WebDAV syncing in Zotero for a couple years now, and Zotero has been storing files in a single directory locally for four years, so there's not some ominous point in the "near future" where this is going to suddenly be a problem for everyone.

    And the change is a bit more complicated than you make out, because it would have to either 1) migrate existing users' WebDAV data (which is anything but simple) or 2) apply only to new setups and/or new uploads (which would require keeping track of which structure is used and look in the appropriate place).

    But we'd be happy to look at any patches that you provided.
  • edited September 2, 2010
    Dan, I assume Zotero depends on PROPFIND requests? If not a work-around for Zotero might be possible.
    toupeira: I actually spoke too soon here.

    Zotero does not use PROPFIND requests for the actual syncing, specifically because that would make for very slow sync performance. (Not because of any inherent slowness of PROPFIND, but just because pulling a file list on every sync would be a bad design.)

    It uses PROPFIND only when verifying the server and when purging orphaned files. The former uses "Depth: 0", so it's just a query on the folder metadata. The latter uses "Depth: 1" to get the full file list. But it looks like the latter isn't even hooked up in the code right now, so I believe Zotero clients shouldn't be making any non-"Depth: 0" PROPFIND requests currently.

    Does that get us anywhere? Rather than preemptively splitting up directories, could you just, say, block "Depth: 1" and "Depth: Infinity" PROPFIND requests if there are more than a given number of files in the folder?
  • Hmm... that might be a possibility. So if the "Depth: 1" code isn't hooked up does this mean Zotero doesn't purge orphaned files?

    We're currently looking into several performance improvements and will hopefully be able to raise the limit in a few weeks, and maybe also implement this work-around for Zotero.
  • So if the "Depth: 1" code isn't hooked up does this mean Zotero doesn't purge orphaned files?
    It purges deleted files, because it knows what those are. It doesn't currently purge orphaned files—files it has no record of that happen to exist on the server. Those shouldn't exist under normal usage.
    We're currently looking into several performance improvements and will hopefully be able to raise the limit in a few weeks, and maybe also implement this work-around for Zotero.
    Sounds good. I'd note that this wouldn't really be a workaround for Zotero—it'd simply be blocking the actual calls that were affecting performance (which Zotero doesn't make) rather than splitting up directories in anticipation of such calls.
Sign In or Register to comment.