Four changes to the n-gram caches of llama.cpp make drafting up to 41.6x faster, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.
  • eleitl@lemmy.zipEnglish
    2·
    27 days ago

    It’s interesting that neural graphs have locality at all. At least I find it surprising.