When constructing an application that utilizes a substantial language model (LLM), the input prompt generally consists of two segments. The text describes a structure consisting of two components. The first component, referred to as the fixed part, establishes the context by providing information such as instructions, reference documents, or conversation history.
The second component, known as the variable part, is designed to hold the user’s input, which may vary depending on the situation. For instance, consider a customer service chatbot. As a support agent for AnyCompany, every request initiates with the same phrase: “You are a support agent for AnyCompany.
Here are our policies, as per your input: The commandments above could be 3,000 tokens. It is possible that the customer’s query at the end is 50 tokens long. This suggests that your model is repeatedly processing the initial 3,000-token sequence in numerous requests.
Frameworks such as vLLM and TensorRT-LLM provide a solution for this issue. The computed key-value (KV) pairs for previously encountered prompt prefixes are stored in their cache.
