Documentation
¶
Index ¶
- func PullModel(ctx context.Context, model string, ...) error
- type Tokenizer
- type TokenizerModelMappings
- type TokenizerOption
- func TokenizerWithCustomModels(models map[string]string) TokenizerOption
- func TokenizerWithFallbackModel(model string) TokenizerOption
- func TokenizerWithHTTPClient(client *http.Client) TokenizerOption
- func TokenizerWithModelMap(models map[string]string) TokenizerOption
- func TokenizerWithPreloadedModels(models ...string) TokenizerOption
- func TokenizerWithToken(token string) TokenizerOption
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Tokenizer ¶
type Tokenizer interface {
// CountTokens counts the number of tokens in the given prompt using the specified model.
// When to Use:
// - When you need the token count but not the actual tokens.
// - For validating prompt length against model limits (e.g., before API calls).
CountTokens(modelName, prompt string) (int, error)
// Tokenize tokenizes the given prompt using the specified model.
Tokenize(modelName, prompt string) ([]int, error)
// AvailableModels returns a list of available models that can be used for tokenization.
// This method is useful when you need to know which models are available for tokenization.
AvailableModels() []string
// OptimalTokenizerModel returns the optimal model for tokenization based on the given model.
// This is useful when the basedOnModel is not available in the list of available models.
// The implementation is based on the tokenizer model mappings.
// Logic flow:
// - Checks for exact matches in configured models.
// - Falls back to substring matches (e.g., phi3 → phi-3).
// - Uses a fallback model (default: llama-3.1) if no match is found.
OptimalTokenizerModel(basedOnModel string) (string, error)
}
Tokenizer represents an interface for tokenizing text using a specific model.
func NewTokenizer ¶
func NewTokenizer(opts ...TokenizerOption) (Tokenizer, error)
NewTokenizer creates a new tokenizer instance with the specified options. This implementation uses the tokenizer model mappings to determine the optimal model. It does not determine the optimal model dynamically instead it uses the default and/or provided mappings. Additional options can be provided if needed. All models will be loaded from the default model URLs, if not found on disk. the default download path is "~/.libollama" ensure the Host has enough disk space. tokenisation is done using the downloaded .gguf format models via the ollama/ollama/llama tokenizer. When a model is already used once it will be cached in memory. Concurrency: Methods are thread-safe, but model downloads block other operations. Performance: Tokenization is fast, but model loading incurs initial latency (mitigated by preloading). REMEMBER: The performance of tokenization is highly dependent on the model phi would be able to have 60.39 MB/s while tiny only 11.10 MB/s.
Example usage:
// Initialize with a custom model and preload
tokenizer, _ := libollama.NewTokenizer(
libollama.TokenizerWithCustomModels(map[string]string{"custom": "https://example.com/custom.gguf"}),
libollama.TokenizerWithPreloadedModels("llama-3.1"),
)
// Determine the optimal model for a user-provided name
model, _ := tokenizer.OptimalTokenizerModel("llama3-14b")
fmt.Printf("Using model: %s\n", model) // Output: Using model: llama-3.1
tokens, _ := tokenizer.Tokenize(model, "Hello, world!") fmt.Printf("Tokens: %v\n", tokens)
type TokenizerModelMappings ¶
TokenizerModelMappings represents a mapping between a canonical model name and its substrings.
type TokenizerOption ¶
type TokenizerOption func(*ollamatokenizer) error
func TokenizerWithCustomModels ¶
func TokenizerWithCustomModels(models map[string]string) TokenizerOption
Add or override model URLs without replacing the defaults. This allows to expand or update the model URLs.
func TokenizerWithFallbackModel ¶
func TokenizerWithFallbackModel(model string) TokenizerOption
TokenizerWithFallbackModel Changes the fallback model (default: llama-3.1).
func TokenizerWithHTTPClient ¶
func TokenizerWithHTTPClient(client *http.Client) TokenizerOption
Use a custom HTTP client (e.g., for proxies or timeouts).
func TokenizerWithModelMap ¶
func TokenizerWithModelMap(models map[string]string) TokenizerOption
TokenizerWithModelMap Replaces the default model URLs entirely.
func TokenizerWithPreloadedModels ¶
func TokenizerWithPreloadedModels(models ...string) TokenizerOption
TokenizerWithPreloadedModels Downloads the model and preloads models into memory. Use this to make the first tokenizer usage more responsive. Or to ensure the models are downloaded without errors.
func TokenizerWithToken ¶
func TokenizerWithToken(token string) TokenizerOption
TokenizerWithToken sets the API token for downloading restricted models. The token will be used in the Authorization header for requests to huggingface.co. Useful for huggingface. Get your token from https://huggingface.co/settings/tokens
Source Files