ZotFile - Advanced PDF management for Zotero

1246763
  • Can someone try the new version (1.0.6) and tell me whether the reported problems are solved? It works at my computer...

    Zotfile 1.0.6 Changes:

    - The filename for items which already have an attachment was not correctly determined (author_2000_titel23.pdf instead of author_2000_titel3.pdf when the file author_2000_titel2.pdf already exists)
    - zotfile stopped working after using it a couple of times after starting firefox
  • No difference for me. I don't get any renaming, even the first time after restarting firefox.
  • can you check out whether 1.0.4 still works?

    http://www.columbia.edu/~jpl2136/zotfile_104.xpi
  • Yes, 1.0.4 works for me (I only tested one item).
  • I finally tested zotfile 1.0.5 under Zotero 2.0b7.2, Snow Leopard, and ff3.5.3 and it works great--I zotted 3 separate item/paper combinations with no troubles. Thanks Joscha! Despite the great rename from metadata feature of Zotero proper, I still find it more convenient to have Zotfile pull the file out of the downloads folder, attach it to the parent item, and do the renaming all in one step.
  • gregsharp, can you check out this version?
    http://www.columbia.edu/~jpl2136/zotfile_106b.xpi

    The problem is that I can't reproduce the error so I am trying to figure which change between 1.0.4 and the newer versions produced the problem...
  • ...106b did not work for me, but 1.0.4 works fine. Great tool!
  • I just installed zotfile - using Firefox 3.5.5 on Windows Vista. I changed the directories in about:config as outlined, changing the source directory to "C:\AddToLibrary".

    When I click zotfile icon it gives the error message "Unable to find file in "C:\AddToLibrary". There is a new pdf file in this directory. I tried "C:/AddToLibrary"; "C:/AddToLibrary/"; and "C:\AddToLibrary\" all with the same results.

    I've also tried uninstalling and reinstalling--no change. BTW - the file says it is version 1.0.6 but the addons dialog says its 1.0.5

    gary
  • I uninstalled 1.0.6 and installed version 1.0.4 --everything seems to be working fine.

    Thanks
    gary
  • Version 1.0.4 is back on the website because some people have reported problems and I can't really get into it right now. You can still download 1.0.6 if it works for you.

    http://www.columbia.edu/~jpl2136/zotfile.html


    1.0.6 includes two improvements (which are lost when you downgrade):

    1) Instead of overwriting existing files with the same name, Zotfile adds an index starting at 2 to the filename if the file already exists in the destination folder. This allows you to add more than one attachment to a zotero item with zotfile and solves the problem described by migugg.

    2) Thanks to RoelofOomen, the bug with the disappearing icon was fixed.
  • Hi Joscha,

    Sorry for the delay. No, 1.06 doesn't work for me either. This is firefox 3.5.3, zotero 2.0b7.4, and windows 2000SP4.

    It is a small thing, but I notice that 1.06 shows up in the add-ons list as version 1.05. Also, 1.04 shows up as 1.03.

    I will test again on linux. Maybe it is different there.

    -Greg
  • Well, on linux it seems to rename the file OK. But it doesn't attach the pdf to my entry. Did I forget something?
  • I figured out the problem with attachments not working on linux. Here is the fix:

    In moveFile, change this:

    return(file.path);

    to this:

    dir.append (filename);
    return dir;

    Then, if you do the above, you can delete the stuff that creates "newfile" from "newfile_path". Instead you can just do this:

    var newfile=this.moveFile(lastfile, this.prefs.getCharPref("dest_dir"), filename);
  • Hi Joscha,

    thanks for the great zotero plugin. It is proving to be a real help in keeping the mass of pdfs under some semblance of control

    I was wondering how easy it would be to add the ability of saving the page number as well? I have quite a few articles in my library that are written by the same authors, in the same volume of the same journal and have the same title (except for Part 1, Part 2 and Part 3 etc at the end of the title). This means that except by having very long file names (including Part 1 etc) the file names are no-longer distinct. By having a page number in the filename then the filename becomes unique.

    Thanks again

    jd
  • This is esp. to GregSharp -- I'm also using linux, and I'm also not having a problem where the file gets moved and renamed, but not attached. The item doesn't have other attachments, so that's not the problem. I tried following Gregsharp's instructions about modifying the extension, but I'm not sure I did it right--I edited file:///home/peter/.mozilla/firefox/lqs6604f.default/extensions/zotfile@columbia.edu/chrome/content/zotfile/zotfile.js
    so it looks like this from line 260-337 ... any suggestions? did I not edit it correctly, or edit the wrong thing?

    moveFile: function(file, destination, filename){
    // create a nslFile Object of the destination folder
    var dir = Components.classes["@mozilla.org/file/local;1"].
    createInstance(Components.interfaces.nsILocalFile);
    dir.initWithPath(destination);

    // move file to new location
    file.moveTo(dir, filename);
    // changing this in hopes it fixes for linux: http://forums.zotero.org/discussion/5301/3/zotero-plugin-to-rename-move-and-attach-your-pdfs-to-zotero-items/
    // return(file.path);
    dir.append (filename);
    return dir;

    },

    getFilename: function(item){
    // create the new filename from the selected item
    var item_type = item.getType();
    var rename_rule=this.prefs.getCharPref("renameFormat");
    if(item_type==19) var rename_rule=this.prefs.getCharPref("renameFormat_patent");
    if (!this.prefs.getBoolPref("useZoteroToRename")) filename=this.createFilename(item, rename_rule);
    if (this.prefs.getBoolPref("useZoteroToRename")) filename=Zotero.Attachments.getFileBaseNameFromItem(item.itemID);
    return(filename);
    },

    zotfileAction: function(){
    var items = ZoteroPane.getSelectedItems();
    var item = items[0];

    //check whether it really is an bibliographic item (no Attachment, note or collection)
    if (!item.isAttachment() & !item.isCollection() & !item.isNote()) {

    // create the new filename from the selected item
    filename=this.getFilename(item);

    // get the last modified file from a directory
    lastfile=this.lastFileInDir(this.prefs.getCharPref("source_dir"), this.getpara("filetypes"));
    if(lastfile!=-1 & lastfile!=-2 ) {
    var lastfile_oldpath=lastfile.leafName;

    // complete filename with extension and then move the file to destination
    var filetype=this.getFiletype(lastfile.leafName);
    var filename = filename + "." + filetype;

    var confirmed=1;
    if (this.prefs.getBoolPref("confirmation")) var confirmed=confirm("Do you want to rename and link the file \'" + lastfile_oldpath + "\' to the currently selected Zotero item?");
    if(confirmed){
    var newfile_path=this.moveFile(lastfile, this.prefs.getCharPref("dest_dir"), filename);

    // recreate the lastfile nslFile Object
    // (for some reason the Attachment is linked to the wrong location without recreation)
    // changing based on suggestion here: http://forums.zotero.org/discussion/5301/3/zotero-plugin-to-rename-move-and-attach-your-pdfs-to-zotero-items/

    // var newfile = Components.classes["@mozilla.org/file/local;1"].
    // createInstance(Components.interfaces.nsILocalFile);
    // newfile.initWithPath(newfile_path);

    var newfile=this.moveFile(lastfile, this.prefs.getCharPref("dest_dir"), filename);

    // Attach last file to selected Zotero item
    if(this.prefs.getBoolPref("import")) {
    // Attach last file to selected Zotero item
    Zotero.Attachments.importFromFile(newfile, item.itemID);

    //Delete the old file that is not longer needed
    newfile.remove(false);
    }
    if(!this.prefs.getBoolPref("import")) Zotero.Attachments.linkFromFile(newfile, item.itemID);

    // Show message
    this.infoWindow("Zotfile","File \'" + lastfile_oldpath + "\' changed to \'" + newfile.leafName + "\' and added as an attachment.",8000);
    }
    }
    else this.infoWindow("Zotfile Error","Unable to find file in " + this.prefs.getCharPref("source_dir"),8000);
    }
    else this.infoWindow("Zotfile Error","Selected item is either an Attachment, a note, or a collection.",8000);
    }
    };
  • I, too, tried the change outlined above to address the import error on linux and now the file doesn't move and it is not imported (I'm not trying to link.) I also receive no errors, it just doesn't work.

    Does anyone have any other suggestions for getting this great plugin to work on linux?

    Thanks,
    P
  • So, it turns out that the moveTo nsIFile method might be broken on linux because, although it moves the file it fails to update its path information in the object. Then when you make the call to Zotero.Attachments.importFromFile the filename information is incorrect and it can't attach it.

    I changed the moveFile code to look like this:

    moveFile: function(file, destination, filename){
    // create a nslFile Object of the destination folder
    var dir = Components.classes["@mozilla.org/file/local;1"].
    createInstance(Components.interfaces.nsILocalFile);
    dir.initWithPath(destination);

    // move file to new location
    file.moveTo(dir, filename);
    var newfile = Components.classes["@mozilla.org/file/local;1"].
    createInstance(Components.interfaces.nsILocalFile);

    // HACK (is there a better way to set the path?)
    newfile.initWithPath(destination+'/'+filename);

    return(newfile);
    },

    and now instead of all the stuph with the newfile_path in zotfileAction, I just make this call:

    var newfile=this.moveFile(lastfile, this.prefs.getCharPref("dest_dir"), filename);

    With that hack in place, where we create a new file handle to the newly renamed file, it seems to work. The old workaround failed for the same reason because moveFile was returning the file.path, which hadn't been updated properly.

    I wonder if this is a full on Mozilla bug...

    Also, does anyone know the proper way to join a directory and file in javascript that will be cross-platform? In python I would use os.path.join, but I don't know what to do here, so I just concatenated the strings, but it will clearly not work on windows.

    Best,
    Per
  • Never mind! I looked into it and gregsharp's solution is correct and works. Could it be an extra space in dir.append (filename);? was messing it up? I don't know js, but it works for me now without the space between append and (filename).

    P
  • Hi. I've seen in the website
    and Zotfile works on linux. (ubuntu 10.04 beta2+updates) firefox 3.6.3 zotero 2.0.2.
    Tnx
  • Sorry for the newbie question, but can this plugin work with files that already exist on computer? Not only the last downloaded file?
  • Just a thank you. Zotfile has really improved my satisfaction with Zotero! Perhaps it should be worked into the main Zotero package. Anyway, keep up the good work!
  • I do agree with ike9898 and many others above.

    Thank you very much Joscha !

    I was just wondering, however, if it were possible with not to much effort to allow a hierarchical organisation into folders, using metadata of the source ?

    For example :
    ~/Documents/Literatur/...
    ... - Journal1/Author1/ title & year
    ... - Journal1/Author2/ title & year
    ... - Journal2/Author3/ title & year

    etc.
    I think Mendeley allows such an automated construction, but with other lacks...

    Thanks again,
    Thierry
  • It would not be difficult to make Zotfile do this, but it would apply to only files imported by Zotfile.

    There is some discussion about this in the forums. Probably the best way to do this would be either a standalone script that reads Zotero database and copies files (or links in operating systems that support this) to a folder structure or a plugin that does this. Not particularly difficult to code, but this far no-one has done it.
  • edited November 19, 2010
    @mronkko:

    Indeed, it shouldn't be a difficult implementation, even natively in Zotero. Without knowing Javascript (but C++, unpracticed for years), I managed to modify Zotfile and it works. With basic possibilities for now.

    Nevertheless, I think that it could be made in a much more elegant, generic and easily customizable manner by someone who is already familiar with Zotero's & Zotfile's codes. And Javascript, by the way...

    Finally, I have the feeling that the "scope" of Zotero could be greatly increased by coupling features of Zotero, Zotfile, and Zotrename. Let me explain my thought:
    1) Take tags, collections & notes and other metadata of items;
    2) Consider several directories to be watched for sources;
    3) Let the root destination directory be plural, if needed;
    4) Distinguish (if desired) between files a) attached from watched folders (Zotfile), b) attached manually (Zotero), and c) imported, copied locally (Zotero);
    5) Let the parameters be flexible between couples of source-destination directories;
    6) Automate renaming, also when e.g. tags or name of collections are modified;
    >>> You get a very powerful and flexible managing tool, for your files and your notes, too. You have the opportunity to centralize or to group by clusters.

    I make the distinction between methods a), b), and c) because it allows more flexibility. For my personal use, I would probably "merge" a) and c) such that c) behaves in fact as if the file is re-routed to a watched folder, i.e. renamed, categorized and linked.
    As for the method b), it allows for instance to create links to some files that you don't want to be moved from a given folder. If necessary, a fourth method could copy such a file to a centralized location...

    I will have a look to see what I can do (attachment.js), but as I said, there are probably people in there that know a lot more about Zotero and Javascript.

    Thierry
  • edited November 20, 2010
    Here is my take on the problem. This perl script creates a folder structure that contains links to the Zotero data directory to approriate files in the form of JournalName/Year/filename. If you take this script and schedule it to run daily, it currently does or can be easily modified to most of the things in the post above. Currently only works for items stored inside Zotero because it was not immediately apparent from the Zotero database how paths to linked items are encoded.

    Tested on OS X 10.6 and should work without modifications with other unixes too.



    #!/usr/bin/perl

    use DBI;

    my $zoterostorage="/Users/mronkko/Documents/Research/Zotero Data";
    my $target="/Users/mronkko/Documents/Research/Articles";

    my $dbh = DBI->connect("dbi:SQLite:dbname=$zoterostorage/zotero.sqlite","","");

    # Query all PDF attachments

    my $sth = $dbh->prepare('SELECT idv1.value, idv2.value, i2.key, ia.path FROM itemDataValues idv1, itemDataValues idv2, itemData id1, itemData id2, items i, items i2, itemAttachments ia WHERE i.itemID=id1.itemID AND i.itemID=id2.itemID AND idv1.valueID=id1.valueID AND id1.fieldID=14 AND idv2.valueID=id2.valueID AND id2.fieldID=12 AND ia.sourceItemID=i.itemID AND ia.linkMode=1 AND ia.mimeType="application/pdf" AND i2.itemID=ia.itemID' ) or die "Couldn't prepare statement: " . $dbh->errstr;

    $sth->execute() or die "Couldn't execute statement: " . $sth->errstr;

    print "Building links\n";

    while (@data = $sth->fetchrow_array()) {
    # date, journal, key, path
    my $year=substr($data[0],0,4);
    my $file=substr($data[3],8);
    my $itempath="$zoterostorage/storage/$data[2]/$file";
    my $linkpath="$target/$data[1]/$year/$file";
    print "\nLinking item\n$itempath\n--> $linkpath\n\n";

    #Create the link, but only if the file exists
    if(-e $itempath){

    #And if the same file is not already linked
    if(-e $linkpath){
    print "Skipping because link already exists";
    }
    else{
    #Ensure that the path exists

    unless(-d "$target/$data[1]"){
    mkdir "$target/$data[1]" or die "Unable to create path $target/$data[1]";
    }
    unless(-d "$target/$data[1]/$year"){
    mkdir "$target/$data[1]/$year" or die "Unable to create path $target/$data[1]/$year";
    }

    symlink($itempath,$linkpath) or die "Unable to create link";
    }
    }
    else{
    print "Skipping non-existingfile";
    }
    }
  • It's not a big deal, but you may want to use the <code> tags for code in the future.
  • Hi,

    http://www.columbia.edu/~jpl2136/zotfile_150b.xpi

    here is a new version of zotfile for testing. The new version can also (batch) rename and move existing attachments which I think a lot of people would like to do. Please be careful with the batch renaming at this point! You can basically select all the items in your library, start the zotfile function 'ZotFile - Rename Existing Attachments' and all attachments are moved and renamed (+ changed back and forth between imported to linked) according to the logic defined in the zotfile options.

    It would be good to hear whether the new version works for you or not and what system you are using (Is there anyone using FF4 on Lunix?).

    Here is a list of changes:
    - FF4 compatible
    - Rename Existing Attachments. It's now also possible to (batch) rename and move existing attachments. Specifically, there are now two functions: 1) add new attachments items ('ZotFile - Attach New File') and 2) rename and move existing attachments. You can now select one or multiple items, right click and select 'ZotFile - Rename Existing Attachments'. The attachments from these items will be renamed and moved according to the zotfile logic. So in principle, you can seamlessly move between linked or imported attachments or different naming conventions. Currently, items with multiple attachments are skipped, though, but I plan to change that.
    - I have removed the zotfile button. Currently you can only start the two function by right clicking on items and selecting them from the context menu. What do you think about that? Should there be a button? I would like to add zotfile to the drop-down menu from zotero's paper-clip button but I don't know how to do that right now. Any ideas?

    And here is what I am still planing for 1.5:
    - make the whole thing work for items with multiple attachments
    - hopefully solve the bugs (not working in second window, linux compatible)

    Maybe I will also look into the folder thing but I am not sure yet. I agree with those who wrote that it's not difficult at all but I just have to make sure that I am not spending too much time with this. Also, I am pretty much only doing statistical programming so that I didn't had any javascript background when I started this but was able to learn things quickly. So I can only encourage people to try things out as long as you have some programming background. I am also happy to help if you are looking at the zotfile code. I think that the biggest hurdle right now is the bad documentation of the Zotero API.


    Joscha

    ps: Does any one know how to a) add the zotfile functions to zotero's paperclip drop-down menu and b) how I can make the zotfile options in the right click menu only appear for certain item types?

    ps2: For those who have already looked at the zotfile code: zotfileAction is not called AttachNewFile.
  • Here is yet another version which supports subfolders - i.e. the hierarchical organisation of files into folders, using metadata of the source:

    http://www.columbia.edu/~jpl2136/zotfile_150b2.xpi

    There are three new options under about:config

    .confirmation_batch - The number of items selected for batch processing which require a user confirmation (default is 2).

    .subfolder - organize attachment files into subfolders using the rule defined under 'subfolderFormat'. This only applies to linked attachments (default is false).

    .subfolderFormat - rule used to organize files into sub-folders. You can use the same wildcats as for the naming of the file. Make sure to insert the appropriate separators for folders. The default, for example, "/%j/%y" organizes the pdfs into folders [dest_dir]/[journal name]/[year of publication]. zotfile just replaces the wildcats %X using metadata and then appends the dest_dir with the subfolder rule. Always start the subfolderFormat with an / and I think \ for windows.


    Again, some testing would be appreciated! I have only tested the folder stuff on Mac OS... So any reports from Windows (and other OS) users would be good.

    Joscha
  • @ Joscha

    Thank you very much for both updates. I tested b2 briefly and it seems it perfectly works with Windows (with \ for folders).

    I just noticed that the functions appears twice in the context-menu. Maybe it is a problem from the modifications I previously did. However, it shouldn't, since I desinstalled the previous, modified version of Zotfile before installing the new one.

    Is there a special reason for sometimes repeating an if statement with the opposite expression rather than using something like "else" ?

    For what concerns the removed button: to my opinion, this is not mandatory, but might be useful to "remind" the user that Zotfile is installed :D.
    However, what may be more useful, but certainly more difficult to program, is the possibility to have keyboard shortcut(s).

    I had a quick look at your code and, indeed, it's far more elegant than my tentative. I will look at it in further detail to see what and how it might be improved (regarding e.g. your questions).

    Thanks again!

    Thierry

    P.-S.: Is the preference ".import" a new one, too?
  • Do others also see the double reference for the functions in the drop-down menu?

    @ Thierry: Thanks for the report.
    No, there is not really a special reason for repeating an if statement with the opposite expression instead of using 'else'. Probably, 'else' is even faster because the statement does not needs to be evaluated twice (the difference should be completely neglect able, though). I guess, I just don't really like else statements and prefer having the condition in front of each code block. Sometimes, I am also not be sure whether opposite expressions are appropriate or whether there is a third case.
    .import is not new. Look at http://www.columbia.edu/~jpl2136/zotfile.html. With this option you can decide whether Zotfile creates a link (.import=false) to the renamed file (which means that the file remains in the dest_dir) or whether the file is imported so that Zotero stores a copy if it. Imported pdfs can be synced but they are in not easily accessible through the file system.
This discussion has been closed.