naive_speculate.config.internal¶
Framework facing interface, defining internal configuration options for dependency manager.
DraftConfig
¶
Bases: BaseModel
Configuration for setting up a drafter.
Attributes:
| Name | Type | Description |
|---|---|---|
sample_strategy |
SampleStrategy
|
The sampling strategy for drafting. |
num_draft_tokens |
int
|
Number of tokens to draft in each iteration. |
infer |
InferenceConfig
|
Configuration about inference. |
InferenceConfig
¶
Bases: BaseModel
Configuration for assembling an inferencer.
Attributes:
| Name | Type | Description |
|---|---|---|
model_name |
str
|
HuggingFace model name. |
kvcache_type |
KVCacheType
|
The type of key-value cache to be used. |
inferencer_type |
InferencerType
|
The type of inferencer to be used. |
SpeculateConfig
¶
Bases: BaseModel
Configuration for setting up a speculative decoder.
Attributes:
| Name | Type | Description |
|---|---|---|
draft |
DraftConfig
|
Configuration for the drafter. |
verify |
VerifyConfig
|
Configuration for the verifier. |
VerifyConfig
¶
Bases: BaseModel
Configuration for setting up a verifier.
Attributes:
| Name | Type | Description |
|---|---|---|
verify_strategy |
VerifyStrategy
|
The verification strategy. |
infer |
InferenceConfig
|
Configuration about inference. |
internalize_config(user_config)
¶
Translate external user specified configuration into SpeculateConfig.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
user_config
|
UserSpeculateConfig
|
User specified configuration. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
SpeculateConfig |
SpeculateConfig
|
Specific configuration for internal use. |