naive_speculate.infer.kvcache.dynamic¶

Define DynamicCache, implementing KVCache.

`DynamicCache` ¶

DynamicCache wraps huggingface's DynamicCache.

DynamicCache implements KVCache and Sequence.

Attributes:

Name	Type	Description
`cache`	`DynamicCache`	The underlying huggingface DynamicCache instance.

__getitem__(index: int) -> KVState

__getitem__(index: slice) -> tuple[KVState, ...]

Retrieve kv states from the given layer index or indices.

If kv states for a certain layer do not exist, corresponding keys and values will be empty tensors.

Parameters:

Name	Type	Description	Default
`index`	`int \| slice`	The layer index or slice to retrieve.	required

Returns:

Type	Description
`KVState \| tuple[KVState, ...]`	KVState \| tuple[KVState, ...]: The kv state(s) of the specified layer(s).