Caching LLM responses¶

This notebook demonstrates how to use Cassandra for a basic prompt/response cache.

Such a cache prevents running an LLM invocation more than once for the very same prompt, thus saving on latency and token usage. The cache retrieval logic is based on an exact match, as will be shown.

In [1]:

from langchain.cache import CassandraCache

from langchain.cache import CassandraCache

In [2]:

from cqlsession import getCQLSession, getCQLKeyspace
astraSession = getCQLSession()
astraKeyspace = getCQLKeyspace()

from cqlsession import getCQLSession, getCQLKeyspace astraSession = getCQLSession() astraKeyspace = getCQLKeyspace()

Create a CassandraCache and configure it globally for LangChain:

In [3]:

import langchain
langchain.llm_cache = CassandraCache(
    session=astraSession,
    keyspace=astraKeyspace,
)

import langchain langchain.llm_cache = CassandraCache( session=astraSession, keyspace=astraKeyspace, )

In [4]:

langchain.llm_cache.clear()

langchain.llm_cache.clear()

Below is the logic to instantiate the LLM of choice. We choose to leave it in the notebooks for clarity.

In [5]:

from llm_choice import suggestLLMProvider

llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'VertexAI', 'OpenAI' ... manually if you have credentials)

if llmProvider == 'VertexAI':
    from langchain.llms import VertexAI
    llm = VertexAI()
    print('LLM from VertexAI')
elif llmProvider == 'OpenAI':
    from langchain.llms import OpenAI
    llm = OpenAI()
    print('LLM from OpenAI')
else:
    raise ValueError('Unknown LLM provider.')

from llm_choice import suggestLLMProvider llmProvider = suggestLLMProvider() # (Alternatively set llmProvider to 'VertexAI', 'OpenAI' ... manually if you have credentials) if llmProvider == 'VertexAI': from langchain.llms import VertexAI llm = VertexAI() print('LLM from VertexAI') elif llmProvider == 'OpenAI': from langchain.llms import OpenAI llm = OpenAI() print('LLM from OpenAI') else: raise ValueError('Unknown LLM provider.')

LLM from VertexAI

In [6]:

%%time
SPIDER_QUESTION_FORM_1 = "How many eyes do spiders have?"
# The first time, it is not yet in cache, so it should take longer
llm(SPIDER_QUESTION_FORM_1)

%%time SPIDER_QUESTION_FORM_1 = "How many eyes do spiders have?" # The first time, it is not yet in cache, so it should take longer llm(SPIDER_QUESTION_FORM_1)

CPU times: user 19.9 ms, sys: 1.22 ms, total: 21.1 ms
Wall time: 1.43 s

Out[6]:

'Spiders have eight eyes.'

In [7]:

%%time
# This time we expect a much shorter answer time
llm(SPIDER_QUESTION_FORM_1)

%%time # This time we expect a much shorter answer time llm(SPIDER_QUESTION_FORM_1)

CPU times: user 4.83 ms, sys: 263 µs, total: 5.1 ms
Wall time: 124 ms

Out[7]:

'Spiders have eight eyes.'

In [8]:

%%time
SPIDER_QUESTION_FORM_2 = "How many eyes do spiders generally have?"
# This will again take 1-2 seconds, being a different string
llm(SPIDER_QUESTION_FORM_2)

%%time SPIDER_QUESTION_FORM_2 = "How many eyes do spiders generally have?" # This will again take 1-2 seconds, being a different string llm(SPIDER_QUESTION_FORM_2)

CPU times: user 5.92 ms, sys: 3.34 ms, total: 9.26 ms
Wall time: 852 ms

Out[8]:

'Spiders generally have eight eyes.'

Stale entry control¶

Time-To-Live (TTL)¶

You can configure a time-to-live property of the cache, with the effect of automatic eviction of cached entries after a certain time.

Setting langchain.llm_cache to the following will have the effect that entries vanish in an hour:

In [9]:

cacheWithTTL = CassandraCache(
    session=astraSession,
    keyspace=astraKeyspace,
    ttl_seconds=3600,
)

cacheWithTTL = CassandraCache( session=astraSession, keyspace=astraKeyspace, ttl_seconds=3600, )

Manual cache eviction¶

Alternatively, you can invalidate cached entries one at a time - for that, you'll need to provide the very LLM this entry is associated to:

In [10]:

%%time
llm(SPIDER_QUESTION_FORM_2)

%%time llm(SPIDER_QUESTION_FORM_2)

CPU times: user 4.36 ms, sys: 105 µs, total: 4.47 ms
Wall time: 120 ms

Out[10]:

'Spiders generally have eight eyes.'

In [11]:

langchain.llm_cache.delete_through_llm(SPIDER_QUESTION_FORM_2, llm)

langchain.llm_cache.delete_through_llm(SPIDER_QUESTION_FORM_2, llm)

In [12]:

%%time
llm(SPIDER_QUESTION_FORM_2)

%%time llm(SPIDER_QUESTION_FORM_2)

CPU times: user 10 ms, sys: 621 µs, total: 10.7 ms
Wall time: 964 ms

Out[12]:

'Spiders generally have eight eyes.'

Whole-cache deletion¶

As you might have seen at the beginning of this notebook, you can also clear the cache entirely: all stored entries, for all models, will be evicted at once:

In [13]:

langchain.llm_cache.clear()

langchain.llm_cache.clear()