How to Cache API Responses in Python (requests-cache)

Calling the same API over and over is slow and burns through your rate limit for no reason. Caching API responses in Python fixes that: the response is stored locally and, on the next call, it is read from disk in milliseconds. With the requests-cache library you get this with almost no change to the code you already have.

Prerequisites

• Python 3.8 or later installed.
• Basic knowledge of the requests library (how to do a GET).
• A public REST API to test against — we use https://api.github.com as the example.

Step 1: Install requests-cache

Install it with pip. requests-cache is an add-on to requests, so we install both.

pip install requests requests-cache

Step 2: Create a CachedSession

Instead of calling requests.get() directly, you create a CachedSession. It behaves exactly like a normal session, but stores every response in a SQLite file. The expire_after parameter defines how many seconds the stored copy stays valid.

from requests_cache import CachedSession

session = CachedSession(
    cache_name="api_cache",   # creates the api_cache.sqlite file
    backend="sqlite",
    expire_after=3600,        # the response is valid for 1 hour
)

response = session.get("https://api.github.com/repos/psf/requests")
print(response.status_code, response.from_cache)

On the first run you see 200 False: the response came from the network. On the second, you see 200 True — it came from the cache.

Step 3: Measure the speed gain

It is worth confirming the effect with a simple example. The difference between a network call and a cache read is typically hundreds of milliseconds versus a few milliseconds.

import time

for i in range(2):
    start = time.perf_counter()
    r = session.get("https://api.github.com/repos/psf/requests")
    elapsed = (time.perf_counter() - start) * 1000
    print(f"call {i+1}: from_cache={r.from_cache} ({elapsed:.0f} ms)")

Step 4: Control what gets cached

Caching everything is rarely what you want. A common mistake is ending up with an error response stuck in the cache. Restrict caching to 200 responses and GET methods, and set different lifetimes per endpoint.

session = CachedSession(
    cache_name="api_cache",
    expire_after=300,              # 5 minutes by default
    allowable_codes=[200],         # only cache successful responses
    allowable_methods=["GET"],     # never cache POST/PUT
    stale_if_error=True,           # if the API fails, use the old copy
    urls_expire_after={
        "*/rate_limit": 0,     # this endpoint is never cached
        "*/repos/*": 86400 # repository data: 1 day
    },
)

stale_if_error=True is the option that will save you most often: if the API is down or returns a 429 error, your script keeps working with the last good response instead of crashing.

Step 5: Force a refresh and clear the cache

Sometimes you really do need the latest value, or you want to tidy up the file. You can bypass the cache for a single request with refresh=True, and delete entries with the methods on the session.cache object.

from requests_cache import CachedSession

session = CachedSession("api_cache", expire_after=3600)

# bypass the cache for this request only and refresh the stored copy
r = session.get("https://api.github.com/repos/psf/requests", refresh=True)

# remove only the entries that have already expired
session.cache.delete(expired=True)

# wipe everything
session.cache.clear()

Verify the result

Three signs confirm it is working:

1. There is an api_cache.sqlite file in your script's folder.
2. The second call to the same URL returns r.from_cache == True.
3. The second call takes an order of magnitude less time.

print(len(session.cache.responses))   # number of stored responses
print(list(session.cache.urls())[:5]) # cached URLs

If from_cache keeps returning False, check that you are using session (and not requests.get) and that expire_after is not 0.

Conclusion

With half a dozen lines you now save calls, avoid 429 errors, and make your script far faster during development. The natural next step is to take this cache to production: swap the SQLite backend for Redis (backend="redis") so several processes can share it, and tune expire_after per endpoint. One question to close with: which of the endpoints you use really change from minute to minute — and which ones could you cache for a whole day without anyone noticing?