Zotfile Reader - Zotero plugin to work with tablet (iPad, Android tablets etc)

ZotFile Reader
Streamlines process to work with Zotero and your pdf reader applications on tablet computers (iPad, Android tablets etc).

https://addons.mozilla.org/en-US/firefox/addon/zotfile-reader/

DONATION
If you are using this add-on, please consider a donation
http://www.columbia.edu/~jpl2136/zotfile.html


DESCRIPTION
ZotFile Reader is a Zotero plugin that streamlines the process to work with Zotero and your pdf reader applications on tablet computers (iPad, Android tablets etc). Building on my zotfile plugin, this add-on pushes attachments to a folder that syncs with your pdf app (such as a dropbox folder),renames them according to the zotfile rules, automatically checks whether these pdfs have been annotated, and prompts the user whether s/he wants to pull them back into zotero. It also adds a tag to the attachments on the 'reader' so that these attachments can easily be listed through a saved search. In particular, it offers the following functions:

1) Move Attachment File to Reader
This function moves the attachment to the reader destination folder specified by the user, renames it according to the zotfile rules, changes the attachment so that it links to the new location, adds the tag '_READ', and adds a time stamp. The folder should be set in the preference dialog and should be a folder that syncs with your tablet (dropbox...). The tag allows the user to easily find the attachments on the reader (you can also add a saved search for this tag). The time stamp is added to the attachment note field so that zotfile reader can recognize when the attachment was modified (e.g. annotated).

2) Remove Attachment File from Reader
This function pulls the attachment from the reader and moves it back to it's original place using zotfile's options. The function is similar to zotfile's 'Rename Existing Attachments' function but it also removes the tag.

3) Scan Reader (accessible through Zotero Actions -> Scan Reader...)
This functions checks whether any of the attachments on the tablet were updated using the time tag and asks the user whether s/he wants to a) pull the attachment, b) update the time tag, or c) do nothing.

Background and Foreground Mode
In background mode, zotfile reader leaves all zotero attachment files where they are and moves copies of the file to the destination folder when you push attachments and replaces the zotero attachment file with this copy when you pull files. Everything happens in the background. The mode is useful when you want to sync the attachment files in your zotero library across multiple computers or when you index your attachments. The disadvantage is that there are two copies of the file.
The foreground mode moves the attachment file to the destination folder and links to this file from within zotero so that there always is only one copy of your files. You can not, however, sync linked attachments to the zotero server.

GETTING STARTED
Here are the steps to get started:
1) install zotfile 1.6.2 from
https://addons.mozilla.org/en-US/firefox/addon/zotfile
2) (optional) change the zotfile renaming options so that you like them.
3) Set up some folder that syncs with you tablet. One possibility is to get a dropbox account, which syncs with PDF Expert, iAnnotate, etc.
4) install zotfile reader from
https://addons.mozilla.org/en-US/firefox/addon/zotfile-reader/
5) change the destination folder in the zotfile reader options to the folder set up under 3.
6) (optional) add a saved search with the tag '_READ'
7) please donate: http://www.columbia.edu/~jpl2136/zotfile.html

REQUIREMENT
This add-on requires zotfile 1.6.2 or higher: https://addons.mozilla.org/en-US/firefox/addon/zotfile


THIS ADD-ON IS UNDER DEVELOPMENT SO..
This add-on is still actively developed. The next big thing I plan is to integrate the extraction of annotations from pdf files. Because this add-on is still in an early stage, things might substantial change. Although I will put some efforts into the transition from one to the next version, you might have to enter some preferences again or (which is very unlikely) move your attachments again.
«1345
  • How are annotations currently stored by the apps that provide annotations?
  • Most apps save them in the standard PDF format so that they are compatible with Preview/Adobe Reader but the annotation are not added as notes. So the best solution would be to extract the annotations using javascript...
  • Joscha, I am really excited about your plugin. Will the file name change take effect in the Dropbox folder as well? I'd really love it if my file names were the same everywhere, and I dig the format you have created to adjust them in Zotero.
  • yes, the plugin uses zotfile's renaming rules to rename and move the file to the dropbox or whatever folder. There is a separate sub-folder option though.
    You can already check it out, by the way. It's just an early version but should be fully functional.
  • edited July 15, 2011
    Cool, that's fast!

    Some questions though about this 0.1 version. I hope you don't mind. If I choose "Push file to reader" it seems to (1) copy the file to my Dropbox folder and (2) delete the attachment and link the file. Then when I choose "Pull file from reader" it gets the attachment back in my library.

    While this works, it seems to presuppose that I'm fine with a linked item during the time that the PDF is on my reader. But there's a problem: when I sync and access my library on another computer, the PDF is not there and the link may or may not work (depending on whether I have my Dropbox folder in the exact same location). Is there a way around this?

    So there are two important downsides to the current implementation:
    1. The whole business of removing attachment from library, linking it, and (when pulling) re-adding it takes a lot of time presumably because this is done via Zotero (and Zotero tries to index the PDF, etc.). Just copying and overwriting files behind the scenes would probably be a lot faster and hence more usable.
    2. It makes it less attractive to just copy a whole bunch of pdfs and ebooks onto my reader, because while they're on my reader, they're not (necessarily) available in my library on all my computers.
  • One other comment:

    It would be nice to have a way of syncing the changes without pulling the file back. (I.e. pulling removes the file from my readings folder, but often I just want to update and read on.)

    But that is something for later. Right now it is already a nice tool for organising required readings; the more important things to fix would be the workflow-related issues in my previous post.
  • okay, I see the problem. I am neither syncing nor indexing my pdfs... I could provide the following two (or three) user scenarios, which I think would solve the problem:

    1) Leave attachment file in original location and move copy of file to the reader (your suggestions)
    2) Move attachment file from current location to the reader and change attachment type to linked pointing at the new location (current behavior)
    3) Move copy of file to the reader and add an additional linked attachment pointing at the new location.

    Also, can you tell me when zotfile causes zotero to index the pdf? Does that happen when I change an attachment from 'linked' to 'imported'? Does it happen for 'imported' to 'linked'? I am now only taking about the 'Rename Existing Attachment' function.
  • Yes, user scenario 1 looks perfect for my kind of workflow, which also involves syncing across multiple clients.

    As far as I can tell, the pdf indexing happens when the attachment is changed from "imported" to "linked" (i.e. when I do a "push") and also when it is changed from "linked" to "imported" (when I do a "pull"), though I'm less sure about the latter. In both these cases there is a lot of harddisk activity and a timeout in exactly the same way as when I import a new PDF by drag and drop, that's how I connected the dots.
  • I just installed the plugin and my pdfs are not syncing with my Dropbox folder at all, even when I choose "Push file to reader." However, I'm not sure what you mean when you say "There is a separate sub-folder option though." My source folder and destination folder are the same folder in my Dropbox. Do you have anyway for me to troubleshoot this in preferences?

    What's more, when I choose "Attach New File(s)", moreover, it attaches every file from the source folder to the entry--should that be happening?
  • edited July 15, 2011
    @Roxy075: Make sure you don't confuse the Zotfile core with the Zotfile Reader addon. "Attach New File(s)" is a Zotfile option that has nothing to do with the reader integration. All the reader integration is supposed to do is pushing files to a designated folder (which can be inside your Dropbox) and pulling them back into Zotero if they've changed.
  • @Mark, I've just installed both, so I'm working to get everything streamlined at present, but I see your point.

    Right now I am having to officially "pull" all of the pdfs from my Dropbox folder (to get them to become links) in order to get the files to return to Dropbox, and I can't seem to get it to do more than one file at a time :/ When I highlight a group, it only pulls the first one... any tips?
  • @ mark: I am going to add this.

    @ roxy: Batch renaming should work both for zotfile and zotfile reader and you do not need to 'pull' them when you never pushed them. You can just use the zotfile's rename existing attachment function.
  • @joscha: my problem isn't batch renaming, it's batch pulling. They have all been renamed, but now are solely stored in in Zotero and are no longer in my Drobox folder. How do I get them back in the folder with their new names, without pulling one by one?
  • @ mark: Here is a new version with an implementation of two modes (I call them back- and foreground mode). You will have to start again with pushing files to your reader (the best would be if you pull all of them back before you start with this version). Can you test whether it works and report here (maybe also the foreground mode even when you don't plan to use it)?

    http://www.columbia.edu/~jpl2136/zotreader_02.xpi

    @roxy: Batch pulling should work but I still have the impression that there are some misunderstandings. There is nothing to pull if all of your attachments are stored in zotero. Push means to move from the current attachment location to the zotfile READER destination folder (which might be dropbox). Pull means to move them back from the zotfile READER destination folder to the location defined by the zotfile destination folder (which is different from the zotfile READER destination folder). So pulling does not make any sense when they are all stored in Zotero.
  • @Josha It turned out I had an error in the destination folder path and it was failing quietly. Because my source folder and destination folder are the same (my dropbox folder), moreover, it was strangely "pushing" files when I pressed "pull" (though never allowing me to do this batch) and when I pressed "push to reader," nothing happened--all because of an incorrect path. Ideally there should be some sort of error message for if you have the path wrong.

    Thanks for the responses and for the plugin, I think I've finally got things sorted.
  • edited July 18, 2011
    Background pushing and pulling works great; no time-lag! Clever solution, too (I never realised you could add notes to attachments directly like that). Foreground mode works still seems to work the same, including the long delays associated with detaching and re-attaching the item.

    A common use case that is not addressed is the following: (1) I'm pushing a book to my reader (2) I make annotations (3) I want to update my Zotero copy with the annotations, but (4) I also want to keep reading the book on my reader. This would be a case of updating the item without pulling it back. I think it's pretty common for people to (want to) do this.

    Conceivably, this would be a third (middle) option in the Zotfile Reader menu: something like "Sync changes". Presumably, this should also work the other way around: if I update my Zotero copy while the file is checked out, pulling it back should warn me that the file will be overwritten. Right now it doesn't. (But I recognize this is a minority case: if I push a book to my reader, I should be reasonably expected to update it only there until I pull it back in.)
  • Is the _READ tag customizable? What is it used for?
  • Updating the copy of the file in Zotero is currently possible with the 'Scan Reader...' function (Zotero->Actions->Scan Reader). It's not pushing changes from the copy in zotero to the reader. Maybe I will add some warning if the att file in zotero was also updated.
    I will consider adding another right click menu item although the menu becomes quit populated with zotfile & zotreader stuff.

    The _READ tag has two functions: 1) 'Scan Reader' finds the items on the reader based on that tag, and 2) you can add a saved search to zotero, which allows you to quickly and easily list all the attachments on the reader. Currently, you have to do that manually but I might add some code so that the saved search is added automatically when you install the plugin.
    There is a hidden option to change the tag. It's hidden because changing it will not update the tag for the attachments you have already pushed. So the option should be used with caution.
  • edited July 19, 2011
    Sweet! I overlooked the Scan Reader function. In that case, scratch my proposal. I agree that the context menu is already quite full.

    This seals the deal for me. I'm going to donate. Though before releasing it to a 1.0 version, consider the following (mainly cosmetic) changes:

    1. It is currently possible to push multiple times, creating multiple copies of the file. Pulling back only retrieves the last copy, leaving redundant copies hanging around. Would probably be better to overwrite on a double push, or warn/ask.

    2. Pushing/pulling on top level items already works (great). And it handles multiple PDFs -- even better! If you know how to generate a saved search it would be cool to do so. In any case, include this in the instructions for adding one: Check the options "only show top level items" and "include parent and child items" to show not the PDFs but the top level items.

    3. Let "Scan reader" also pop up information when nothing was done (maybe via of those nice Zotfile information popups in the bottom right corner).

    4. Also, consider rebranding it and (perhaps) repackaging it, either as one extension integrated with ZF, or as a separate extension that will run on its own. I think this is going to attract a lot of attention. For it to be an extension (ZF reader) of an extension (ZF) of an addon (Zotero) is a bit too far-fetched. Maybe "Zotero Reader" is a good general name?

    5. Graceful handling of changing the folder and/or subfolder settings. The default works well enough but once you've done it differently it is somewhat difficult to change.
  • edited July 19, 2011
    6. Use relative Dropbox URLs, with the system-specific part coded in the Zotero Reader settings. This would probably involve sth like a %Dropbox% variable in the PDF text field in background mode, rather than a hardcoded URL as it is now.

    I.e. on PC1 my Dropbox is F:\Dropbox, on PC2 my Dropbox is D:\Work\Dropbox . Allow me to set the Dropbox URL in the settings and take it from there, so that pushing, pulling, and Scan Reader work wherever my Dropbox happens to be (once I have configured that right).

    7. Grey out "Pull from reader" if an item is not on the reader. Grey out "Push to reader" if an item is on the reader (partly addresses point 1 above).
  • And if I would rank these suggestions according to importance/urgency, it would be 1+7 - 6 - 3 - 5 - 2 - 4.
  • Hi, here is the next version. It includes three new things:
    - Pulling in background mode is now more intelligent. There are now four scenarios: a) Only the file on the reader has changed -> File in Zotero is replaced, and tag is removed; b) Only the file in Zotero has changed -> File on reader is deleted, and tag is removed; c) Both file have changed -> User is prompted; d) Neither the zotero file nor the reader file has changed -> File on reader is deleted, and tag is removed.
    - zotreader now checks whether an item has already been pushed based on the _READ tag. It only allows pushing if the item currently is NOT on the reader so that multiple pushs are avoided. Currently, the user is not prompted whether s/he wants to replace the file on the reader, which I might add later.
    - non intrusive alert if 'Scan Reader' did not find any modified files

    Can someone, presumably mark, test this?

    http://www.columbia.edu/~jpl2136/zotreader_03.xpi
  • All works as advertised, thank you! I like the non-intrusive notices. The pulling behaviour is nicely sophisticated now.

    One comment on pushing/pulling: literally disabling (greying out) either option depending on the status of the attachment would save one click. I do understand though that this would make for some extra checks while building the context menu.
  • yes, I agree. Just didn't know how to do it right away but will look into it at some point.

    Anyone here who knows something about java programming? I came across this
    http://ecomware.com/notes/itext-export-annotations
    and was wondering whether I can use it to extract annotations. I have no clue about java though and don't know how to get this running. I just need some external application that I can call from the plugin. So far I only found solutions that extract annotations, which are in a note and not highlighted text.
  • I was going through my bookmarked forum threads and found this one: http://forums.zotero.org/discussion/802/

    The user there has what says to be a fully-functional annotation importer, including highlighting, in the form of an import translator. Definitely worth exploring...
  • No, that actually does not work. It is only extracting highlights that are copied to a hidden note, which is only possible in Adobe Acrobat with an certain option and not with any other program or PDF App (as far as I know). Most programs I found (such as the Automator function in Mac OS) only do that and it's not helpful for my purposes. Maybe the java solution does that too but I don't know.

    Here is a quote from the discussion you linked to making that point:
    "However, for the Highlights to be extracted, one still has to use Acrobat's option to automatically copy highlighted content into a hidden note (and this is probably not going to change, since everything else would require me to basically rewrite pdf2txt in javascript)."
  • The plugin has now been approved by Mozilla. The next version 0.3 is already under review and can be downloaded in advanced from

    https://addons.mozilla.org/en-US/developers/addon/zotfile-reader/versions/

    New features in 0.3
    - More intelligent pulling in background mode
    - zotreader now checks whether an item has already been pushed based on the _READ tag.
    - non intrusive alert if 'Scan Reader' did not find any modified files
  • edited July 29, 2011
    Here is the first test version with 'project folders' - i.e. user specified subfolders. Project folders can be turned on and off through the zotfile preference window.
    This is a test version only and it would be good to get some feedback. Not only whether everything works (background and foreground mode?) but also about the general implementation. I am very open to suggestions. Does it make sense to have an extra window for the project folders? Would it make more sense to integrate the preferences in the main zotfile pref window? Should project folders be turned on by default? etc.

    http://www.columbia.edu/~jpl2136/zotreader_04b1.xpi

    Here is an screenshot that illustrates the new feature:
    http://www.columbia.edu/~jpl2136/zotreader-project-folders.png

    ps: project folders do not work in zotero tab might right now... i hope to fix that soon.
  • Here is a second version. Now everything should run smoothly in tab mode too. I have also added the possibility to push based on all the collection an item is in. So there are now two options for project folders"
    a) Project folders based on Zotero collections (automatically created from selected item)
    http://www.columbia.edu/~jpl2136/zotreader-project-folders.png
    b) User defined project folders
    http://www.columbia.edu/~jpl2136/zotreader-project-folders2.png

    Here is the new beta:
    http://www.columbia.edu/~jpl2136/zotreader_04b2.xpi

    Any comments? Any problems?
  • I really like this idea: http://www.columbia.edu/~jpl2136/zotreader-project-folders.png

    I have just installed the latest beta 0.4b2 and have three comments:
    1.The folder structure offered based on the collections is not in the expected order. Check your PNG above. I would expect that in the path the top collection should come first, then the subcollection, then subsubcollection etc.
    2.Would it be possible to push multiple attachments at once with an option to simply use the first offered folder or all offered folders?
    3.Would be great if this new collection->folder feature could also make it to the Zotfile plugin.
This discussion has been closed.