lightbench.loaders

The lightbench.loaders package contains modules for loading and interfacing with various model backends, including OpenAI, Mistral, and LLaMA. It also includes utilities for generation and general loading logic.

Loaders

class lightbench.loaders.generation.Generation(response, inference_time=0, ttft=0, peak_memory_usage=0, avg_power_usage=0)

Bases: object

class lightbench.loaders.loader.LLMServiceLoader

Bases: ABC

Abstract class for LLM loaders. Provides a common interface for generating text, and cleaning up resources.

abstractmethod cleanup()

Perform cleanup of model resources and release memory.

abstractmethod generate(prompt, max_tokens: int) → str

Generate text based on the given prompt and max tokens.

abstractmethod is_local() → bool

Returns true if using a local model, false if using an API.

abstractmethod name() → str

Returns the name of the model.