Zotfile no option to extract annotations

watdryhope
Hello,

I can no longer see an option for the latest version of Zotfile to extract annotations. I can't use the Zotero reader because it's not compatible with my Boox e-reader which I'm using to do my reading. Right now I've been making edits on the ebook reader and then reupping it to my library via Zoo for Zotero. I don't have an iPad.

Is there some update that has forbidden Zotfile to extract annotations? If not, can anyone suggest a workaround, such as some additional software that could extract annotations from an ordinary pdf? I'm right in the middle of exams so this is potentially a huge issue.
«1
  • dstillman
    edited January 26, 2022
    The Zotero beta provides the ability to extract annotations — it's just "Add Note from Annotations" in the items list context menu. You don't need to use the built-in PDF reader. The ZotFile feature (which didn't know anything about Zotero annotations and was causing a huge amount of confusion) was removed now that this functionality exists in Zotero.

    Zotero's annotation extraction is much more flexible, since it includes much more internal metadata in the notes: colors and citations can be toggled on and off in the notes and will be customizable in exports, rich UI can be displayed for various actions (opening the PDF and highlighting the annotation (when using the built-in reader), jumping back to the library item, restoring a deleted citation), it will be possible to automatically replace citations with pandoc-style citations for Markdown export, the citation format displayed in notes could be made customizable, etc.

    If there's any remaining annotation-related functionality that Zotero doesn't currently support that you found valuable (e.g., .pdfExtraction.NoteHtmlTagStart and similar), let us know.
  • watdryhope
    Ok thanks for this. I've worked out how to extract notes. But I notice that exporting to markdown (into Obsidian) makes the link to the document illegible. Whereas if I use zotfile and use the export to markdown the links are preserved. Is there any solution for this?

    Thanks!
  • dstillman
    What do you mean by "illegible"? Can you give an example?
  • watdryhope
    Sorry. I mean that the link doesn't copy across.

    When I copy across using the annotations pulled using Zotero Beta:

    "Their existence is fluid and often shifting, but nevertheless real. The concept of justice” (Young, 1990, p. 10)

    cf. using the export to markdown or simple copy and past from note into Obsidian.

    "Their existence is fluid and often shifting, but nevertheless real. The concept of justice" ([p. 10](zotero://open-pdf/library/items/PAGT8KWZ?page=10))
  • dstillman
    edited January 26, 2022
    It sounds like you're just copying from the note text? The Zotero beta has built-in Markdown export of notes with optional zotero://select and zotero://open-pdf links. You can trigger it via Quick Copy.
  • watdryhope
    Ok. Thanks. Trying to figure out how to update the beta on Arch linux.
  • kdb_research
    Thanks for this.

    It was useful to be able to insert html tags into the comments formatting in Zotfile.

    Is there a way to do similar formatting of the extracted notes in the new system?
  • dstillman
    @kdb_research: That's the main feature from ZotFile we know is still missing. Can you share some examples of the kind of formatting you have in mind?
  • kdb_research
    edited January 27, 2022
    Sure.

    In the config fields we can edit the html wrapping.

    I do this, but I'm sure there are better ways:

    extensions.zotfile.pdfExtraction.formatAnnotationNote;<p><b>%(content)</b> (<a href="%(uri)">note on p.%(page)</a>)</p>
    * which gives me notes in bold ...

    extensions.zotfile.pdfExtraction.formatAnnotationHighlight;<p>%(content) (%(cite))</p>
    * simplifies the outputs a little

    Zotfile seems to do a better job stripping out line breaks ;)
  • scholarium
    This is bad news. Please either allow zotfile extraction again or increase/remove the maximum length of notes. Due to all the metadata, the notes are bloated and do not have enough space for longer annotations. Zotfile was great for reading and annotating on iPad, whereas the Zotero iOS app always crashes on my iPad Pro. There was no confusion, zotfile-extracted notes coexisted peacefully with Zotero-extracted ones. I don't find much use in the extra features of Zotero annotations and dislike that I have lost the ability to edit the source code of notes. I guess many more users depended for their workflows on zotfile. Many thanks for your efforts in adding features to the App, but please understand the frustration.
  • dstillman
    @scholarium:

    First, if you're having a problem with notes in the beta, you should report it in a separate thread if you want it fixed. You appear to have posted one comment about this last March, with no follow-up after the fix we told you we'd be rolling out. If you can't sync a note created in the course of normal usage, that's just a bug.
    Zotfile was great for reading and annotating on iPad, whereas the Zotero iOS app always crashes on my iPad Pro.
    Again, if you're having trouble with the iOS beta you should report that in a new thread (though the latest iOS build is showing very few crashes), but this is a non sequitur. Nothing is preventing you from using non-Zotero PDF readers. The point here is just that ZotFile's annotation extraction is no longer necessary, even if you're using an external reader that saves embedded PDF annotations, because Zotero now does it itself.
    There was no confusion
    There was a massive amount of confusion [1] [2] [3] [4] [5] [6] [7] [8] [9]. (I stopped at that point, but there were more.)
    dislike that I have lost the ability to edit the source code of notes.
    This has nothing to do with ZotFile or annotation extraction. Zotero 6 will only include the new note editor, which is vastly more powerful and will allow for all sorts of advanced note-taking features that couldn't be done in an HTML-based editor. If there's something specific you're having trouble doing in the new note editor, again, you should report that in a separate thread.
  • scholarium
    edited January 28, 2022
    >> with no follow-up after the fix we told you we'd be rolling out. If you can't sync a note created in the course of normal usage, that's just a bug.

    It has neither been fixed since then, nor is it a bug. It is a decision to include powerful and advanced note-taking features that bloat the note files with an apparently proprietary format without increasing the character limitation that makes no sense for files that are no longer html. Source code editing was useful to manually remove occasional bloat from pasting to allow for syncing.

    I do not intend to use zotero as a note-taking app. It is most powerful as a reference manager that can host papers and annotations, and this primary use case is in danger if a large part of extracted annotations no longer sync due to a size limit that was meant for html files and never for a bloated proprietary format.
  • dstillman
    @scholarium: I have no idea why you’re making these bizarre, sweeping statements about technical issues that you have nothing to do with, but again, if you’re having trouble with something in the beta, you should report it in a new thread.
  • martynas_b
    @scholarium Could you create a new thread for that sync issue and explain how many annotations do you have approximately in a note that can't sync?
  • Fidcapist
    @dstillman May I ask what if any character limits there are for notes and for comments made to highlighted text. And, please confirm that highlights and comments are or are not earmarked for pdf extraction to notes that would show up below the main reference item as individual notes?

    Also, I have not been able to yet get the MS Word plugin installed to see how these highlights and comments work and show up there for citing as we write, which is critical, but am confused not to have yet found an "extract pdf highlights and comments as notes" button or facility

    Overall ― I feel the new note editor, pdf reader, colored tags, page no.s coming in on notes, and more powerful inline workflow features, et al., are tremendously more useful and intuitive using the new reader and editor, and am thankful for not needing a 3rd party reader any longer!!!

    I would personally like the schema for tag colors and highlight colors be more coordinated such that tags may be added by using a particular highlighted color, or define unique highlight colors that are assigned to tags also show up as highlight colors that when used auto-tag the note, though I appreciate also being able to more widely tag highlighted comments right in the Zotero pdf reader note sidebar.

    Clearly, there is are a lot of needs and scenarios to contemplate in framing how this whole system works that relate to down stream work flows and publishing purposes, and this is worthy of comprehensive and inclusive user discussion and inputs, but the possihilities look very promising and the raw feature capabilities look like they are moving in the right direction, IMHO.
  • dstillman
    May I ask what if any character limits there are for notes and for comments made to highlighted text.
    This isn't something you need to worry about. We'll handle the technical implementation. If you experience a problem in normal usage of Zotero, let us know in a new thread.
    please confirm that highlights and comments are or are not earmarked for pdf extraction to notes that would show up below the main reference item as individual notes?
    I don't know what you mean by this. The point of this thread is just that Zotero's own "Add Note from Annotations" function in the items list context menu is a direct replacement for ZotFile's "Extract Annotations" option, while opening the door for many more features. And if you use the new built-in PDF reader, there are various other ways to add annotations to notes as well.

    For other questions or feature requests about the new PDF reader or note editor, please start new threads.
  • juanof9
    Hi! The new update seems to have killed zotfile whilst not adopting some of its key functionalities.

    Among the things I've noticed:

    - Extracts no longer reference to highlight color
    - Its misinterpreting the role of the underscore, which is quite central to my notation system (adds a \ and cuts my notes
    "\_ " --> See https://pbs.twimg.com/media/FKZsaIEVQAUqoRI?format=jpg&name=large

    Notation no longer creates a visual distinction between comment and original text, which was useful when scrolling. See thread here for visual contrast https://twitter.com/juanof9/status/1483240141904699392

    In the future, perhaps allow both Zotero and plugins to coexist and the community to discuss before gutting an add-on and its functionalities! It creates a lot of problems
  • dstillman
    edited January 31, 2022
    @juanof9: To be clear, Zotero absolutely adopts ZotFile's main annotation functionality, and extends it significantly, with the potential for many more advanced features going forward. It doesn't yet implement some esoteric features configurable in ZotFile with hidden settings. As I said above, we're happy to discuss making those things possible in Zotero.
    Extracts no longer reference to highlight color
    This is incorrect. The annotation color is included in the annotation metadata, which is why you can toggle annotation colors on and off. If you're saying you want to be able to include the literal name of the color in a Markdown export, we could consider making that available as part of a template system for Markdown export.
    Its misinterpreting the role of the underscore, which is quite central to my notation system (adds a \ and cuts my notes
    I'm not totally understanding this, but please start a new thread for this with more details.
    Notation no longer creates a visual distinction between comment and original text, which was useful when scrolling.
    I'm not exactly sure what those screenshots are showing, but we don't have anything to do with mdnotes output. If there's specific output you'd like in the built-in Markdown note export, let us know in a new thread. But this would likely be addressed by the same sort of template system for Markdown export that I mention above.

    Within the Zotero note editor, you can toggle annotation colors on and off, which clearly differentiates them from the text. A separate template option for the format of annotations as they're added to notes is also a possibility.
    In the future, perhaps allow both Zotero and plugins to coexist and the community to discuss before gutting an add-on and its functionalities!
    ZotFile is unmaintained, and has been for years. The only reason it still runs on current versions of Zotero is that we submitted a large patch a couple years ago to keep it working. Since it was never updated for the Zotero 6 beta, it had a major data-loss bug for the entire course of the beta — it would delete Zotero annotations if you renamed a PDF that contained any — and was causing a massive amount of user confusion with the new PDF reader. And it was about to be disabled altogether in Zotero 6. We submitted patches to fix these things, which is the only reason ZotFile will be available in Zotero 6 at all.

    The mdnotes plugin also hasn't been updated for Zotero 6, and it may not be.

    So I'd ask for a little patience here. There might be a brief time before there's a direct replacement for some of the hidden features of ZotFile and/or mdnotes, but we're committed to supporting and improving on the workflows those plugins enabled. If you can't live without a particular feature in the meantime, you can continue using Zotero 5.0, under which ZotFile works exactly the same as before.
  • warguelles
    edited February 1, 2022
    @dstillman I would like to make a suggestion.

    The options for working with the beta version of Zotero today are significantly more important and with many more possibilities than they were when the first beta version was released last March. Despite some updates, the initial "support text" does not disclose all the current possibilities. Some of the additions and updates are scattered in different threads in this forum. I understand the large amount of work you are developing now, but would it be possible that the "support text" to be completed with all the new options you have included in the latest versions?

    Thank you so much!
  • dorner
    Can I ask for some clarification? The OP indicated that the Zotfile extract annotation feature was no longer available, but didn't specify which version of Zotero they were using. Has it only been disabled in the beta version? Because I am using Zotero 5.0.96.3 and the Zotfile function is still there.
  • dstillman
    @dorner: Yes, it's removed for Zotero 6 (the current beta), since that's the version with the new PDF reader that includes the replacement annotation extraction functionality.
  • dstillman
    @warguelles: I've updated the preview page with some newer details, but we'll be updating the PDF reader documentation when Zotero 6 ships.
  • warguelles
    @dstillman: Thank you very much!!
  • dstillman
    I would personally like the schema for tag colors and highlight colors be more coordinated such that tags may be added by using a particular highlighted color
    @Fidcapist: See this thread for some questions around colors and tags.
  • KMZahra
    Hello!

    Very very new to Zotero, I made the move so that I could continue to use my tablet for reading and annotating papers.

    I downloaded Zotfile in order to rename and move my PDFs to Dropbox and create linked copies in Zotero but I've spent several hours trying to figure out how to extract annotation using Zotfile (which it says it can do and lots of websites on google also talk about).

    It's only through this thread that I've figured out that this is no longer available on the latest version of Zotero as it's been built in (then it took a little while to find it within Zotero itself).

    Maybe I've not found it yet cause I'm new, but a clearer (google-able) guide to that feature would be useful?

    Taking some getting used to but I'm liking it so far, thanks!!


  • dstillman
    @KMZahra: You're using Zotero beta software, so it hasn't yet been fully documented. But this page explains all the ways to add annotations to notes using Zotero:

    https://www.zotero.org/support/pdf_reader#adding_annotations_to_notes

    ("Add Annotations to Note" is in the same items-list context menu where the ZotFile option was.)
  • droitalaville
    Hello Zoteros,

    Thanks for incorporating the Zotfile Extract Annotation feature directly into Zotero! I am using a non-markdown based writing programm for my thesis (Scrivener). I was relying on the hyperlinks in the notes created by Zotfile, which were recognized by Scrivener. I'm not super savvy with all of this, but I haven't found a way to export/(quick-)copy the notes created by Zotero in a similar way. Is this/will this be possible?

    Thank you!
  • dstillman
    edited February 11, 2022
    @droitalaville: How were you copying to Scrivener before? Just by copying and pasting from the note? And you're referring to the links back to the PDF in Zotero?

    We'll add that as an option ("Include Zotero Links") to the HTML export-to-disk option for notes, similar to Markdown, for the next beta. Later on, we should be able to make that and other options available when using Quick Copy.
  • droitalaville
    Yes, I was simply copying and pasting from the note, which was simple enough. Exactly, after the annotations had been extracted to the note, the reference at the end of an annotation served as the link text for a link back to the PDF (and the specific page). This stayed the same after being pasted into Scrivener (or Word, etc). It would be very useful if this could also be the case with the native Zotero extractions in whichever way. Thank you very much!
  • tre2017
    I have a similar issue as @droitalaville – the new Pdf reader/note system is really well thought out, but I still use Apple Notes to further proces some of my notes, so I also await "include Zotero links" for Quick Copy and HTML export.

    My current workaround to maintain a link to the pdf is to select the Pdf file > right click > Copy select item links (setup with Zutilo I think) and paste it into the annotation note before copypasting into Notes.
This discussion has been closed.