topai_semantic_cache
Store and fetch semantically matching LLM completions to skip redundant inference calls and cut latency and token spend before or after generation.
Instructions
Performs sub-5ms semantic vector cache lookups ('get') and completions storage ('set') for LLM prompts to eliminate redundant inference latency and token spend. Use before invoking an expensive upstream LLM to check if an equivalent prompt was already answered, or right after generation to cache the completion. Do not use for compressing raw text length within a single uncached prompt (use topai_token_slimmer instead). Idempotent, non-destructive edge cache operation.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes | Operation mode: 'get' to retrieve a semantically matching cached response, or 'set' to store a new prompt-response pair. | |
| prompt | Yes | The user prompt or query string used as the semantic lookup key. | |
| response | No | The LLM completion text to persist in cache (required when `action` is 'set', omitted when `action` is 'get'). |