Enhancement: Simplify Update-Process

[schuhwerk]

I wrote a Userscript to help me learn things I frequently search (here).
I'm using githug as a database and CrowdAnki to import from there. This works great, thank you for working on this very convenient tool!

A little Nitpicking/Enhancement/Question: I imagined the library to (auto-)update from git. If I understand it correctly, I always have to (re-)enter the repo-url. Wouldn't It be nice to have an (auto-)update button somewhere?

[aplaice]

Thanks for the kind words!

That sounds like a cool userscript!

---

I agree that the suggested enhancement might be very useful, and I think that it'd be quite straightforward to implement!

As a first rough implementation (no promises on exactly when!), what do you think about the following flow:

1. Menu item: File > CrowdAnki: Re-import all git repositories.

   (I'm not too happy about the name — it's a bit clunky, while still not fully informative.)

   (This will be CrowdAnki's 5th menu item, so we'll probably need a sub-menu soon...)

2. Pop-up window lists all decks that would be updated, states that any local changes to notes will be silently overwritten (see below) and asks for confirmation.

   Optionally (all these would likely futzing with Qt so slightly trickier):

   a. Have checkbox to "skip this window next time". (i.e. proceed directly to 3) (off by default) (with an override in the main config to stop skipping, again) (not sure if it's worth it, and non-obvious effects when combined with b and c...)

   b. Allow the user to choose which decks to import (each deck has a checkbox, all the checkboxes are selected by default).

   c. Have an option to remember the choices from b (i.e. next time the decks you unselect will be unselected (though still listed, so you can select them again) (on by default?)

3. Show the "Import Settings" dialog for each deck.

   (We probably will need to tweak it, to display the deck name in the dialog.)

   I guess that you would probably want this to not be shown, for a more streamlined experience? (Unfortunately, I'm not sure how to best do it, in a way that wouldn't risk other users shooting themselves in the foot with this.)

Overwriting local changes

When we import we currently silently overwrite any local changes (Anki's in-built import previously also did this, but it now does better, allowing one to either import only newer notes or always just overwrite) (unless you're using a "personal field").

This is not critical when the import is more "manual" (per-deck and more cumbersome :)), but it worries me more if the system becomes automatic.

We do ask for confirmation for each deck, anyway, so maybe an earlier, general warning isn't essential? OTOH in the current "Import Settings" dialog, the info that local changes are overwritten is only present indirectly ("Fields which will keep their existing value instead of being imported"). Maybe we should make the warning clearer here, rather than in the new "Re-import all decks" window?

---

I don't think that fully auto-updating, without user interaction (in the way Anki syncs on open/close) would work well without significant changes.

[schuhwerk]

Thanks für your detailed reply!

Menu item: File > CrowdAnki: Re-import all git repositories.
(I'm not too happy about the name — it's a bit clunky, while still not fully informative.)
(This will be CrowdAnki's 5th menu item, so we'll probably need a sub-menu soon...)

sheets2anki (and other plugins i know) use a submenu under tools (this is where I was searching the menu for CrowdAnki, too).

The name makes sense for me btw. :)
I don't have a deep understanding of the repo (sorry for my oversimplification below ;). I think your solution is currently good, because it prevents loss of user-data, which should have a high priority.

I'm not sure, how porcelain works, but most issues seem to go away with a typical "git-interaction" [pseudocode].

 - if no conflicts: no need for any UI or questions
 - if conflicts ask:
    - overwrite local
    - keep local

This way you also don't have to store settings about individual decks (i guess). What do you think?

[aplaice]

Thanks for your reply!

I'm not sure, how porcelain works, but most issues seem to go away with a typical "git-interaction" [pseudocode].

• if no conflicts: no need for any UI or questions
• if conflicts ask:
  • overwrite local
  • keep local

This way you also don't have to store settings about individual decks (i guess). What do you think?

Yes, ideally the workflow (for imports in general, not just automated ones) would be a typical "git pull and reconcile conflicts" — i.e. something like:

1. "Export" the user's current version of the deck into git (i.e. output the data in Anki's database into the CrowdAnki json) and git-commit.
2. Fetch the remote.
3. (Like you write) for each conflict show a 3-way diff (original, user-version, new remote-version) and ask the user what to do. (Per-note would probably be most convenient.)

Unfortunately, for a variety of reasons (see below), step 3 is tricky to get right, so we don't do it (yet?).

As a result, I think that for the time being, having prominent warnings (and especially prominent ones for semi-automated imports) is the only possible solution.

---

Background

Conflict resolution

(This section ended up longer than intended...)

Reasons why conflict resolution is tricky:

1. (minor) The CrowdAnki json output changes slightly between Anki versions.

   (This is due to changes in Anki's database (e.g. addition of new per-note-type config options) and the fact that we try to faithfully export all relevant info.)

   (This doesn't cause issues when importing/exporting CrowdAnki JSON files from other Anki versions, because the extra fields aren't essential, but it does mean that the diffs become additionally messy... (Many false positives if this isn't taken into account.))

2. Creating an intuitive UI for conflict resolution isn't trivial.

3. How do we deal with these conflict resolutions going forward, when the user chooses to keep their own changes?

   From a git point of view, it'd make most sense to structure this as the user having their own private "fork", with a set of "patches" (commits) on top of upstream, that would be rebased on top of upstream after each import. (Or maybe "squashing" all the user patches into a single commit?)

   Unfortunately, keeping a fork with your own changes quickly becomes ugly even when you're a programmer directly managing the codebase. Here, we'd be trying to abstract everything away from the user, so they never have to manually resolve git issues.

   What do we do display when the user changes something on top of their own changes (i.e. they change note X, import from upstream (resolve git conflict keeping their version), change note X again and import again)? (A 3-way diff (previous upstream, current upstream, and current user-version) is probably good enough, but it might be confusing (after all, the version that'd the user will have edited will have been the previous user-version, which we wouldn't be displaying).)

4. We'd also need to deal with changes to objects other than the notes themselves, such as:

   • The note models (front/back templates and CSS) — again 3-way diff, ideally with highlighting (red/green)...
   • Increasing/decreasing the number of fields per note (for a note model) — given how this affects large numbers of notes, I'm not 100% sure how it should be handled.
   • Various minor config options

None of these are insurmountable, and I'd love to have a CrowdAnki that dealt with all of this, but it's definitely not happening in the foreseeable future.

Modifications times

In the absence of a sophisticated git conflict resolution system, we could check for note modification times, and in that case warn more about the risk of overwriting changes. However:

1. Anki sometimes changes (or at least did in the past) the modification times even when you don't actually change the notes. (I think opening the notes, in the Anki "browser", was at some point sufficient.)
2. Note modification times wouldn't catch changes to note modes (templates and CSS).

Hence, we'd have both many false positives and false negatives.