Skip to main content

chunk_text

toak_rs::text_chunker

Function chunk_text 

Source 
pub fn chunk_text(text: &str, config: &ChunkerConfig) -> Vec<TextChunk>
Expand description

Chunks text into overlapping segments based on token count