candle

Commit Graph

Author	SHA1	Message	Date
Laurent Mazare	7f0f83a7c1	Rotating kv cache positions (#2901 ) * Retrieve the current positions for rotating KV caches. * Add the function to the kv cache too. * More testing.	2025-04-15 23:09:26 +02:00
Laurent Mazare	d01207dbf3	Add a RotatingKVCache. (#2493 ) * Add a RotatingKVCache. * Add some KvCache tests. * Test the reset too. * More kv-cache testing. * More tests for the rotating kv-cache. * Improve the api for the rotating cache so that the whole src tensor gets returned when it's overlarge. * Handle contiguity + bugfix + use in mimi. * Add a way to test the mimi streaming mode. * Mimi streaming fixes. * More rotating kv-cache. * Fix the attn mask generation. * Handle the abs case. * Add some tests for the generated mask.	2024-09-23 13:14:32 +02:00

Author

SHA1

Message

Date

Laurent Mazare

7f0f83a7c1

Rotating kv cache positions (#2901 )

* Retrieve the current positions for rotating KV caches.

* Add the function to the kv cache too.

* More testing.

2025-04-15 23:09:26 +02:00

Laurent Mazare

d01207dbf3

Add a RotatingKVCache. (#2493 )

* Add a RotatingKVCache.

* Add some KvCache tests.

* Test the reset too.

* More kv-cache testing.

* More tests for the rotating kv-cache.

* Improve the api for the rotating cache so that the whole src tensor gets returned when it's overlarge.

* Handle contiguity + bugfix + use in mimi.

* Add a way to test the mimi streaming mode.

* Mimi streaming fixes.

* More rotating kv-cache.

* Fix the attn mask generation.

* Handle the abs case.

* Add some tests for the generated mask.

2024-09-23 13:14:32 +02:00

2 Commits