How to paginate a REST API in Python (requests)

When you request data from a REST API, you rarely get everything at once: to save resources, most APIs split the results into pages. If you read only the first page, you end up with incomplete data. The good news is that paginating a REST API in Python is simple with the requests library — all you need is a loop that goes through every page and keeps collecting the results. Let's see how, with practical examples.

Prerequisites

• Python 3.9 or later installed.
• The requests library installed (pip install requests).
• A REST API that returns data in pages, plus access to its documentation.
• Basic Python knowledge (dictionaries and while loops).

Step 1: Make the first request to the API

Before worrying about pages, make sure you can get one. With requests.get you make a request and, with raise_for_status, you make sure any network or authentication error is flagged straight away instead of going unnoticed. The timeout stops your program from hanging forever.

import requests

BASE_URL = "https://api.example.com/items"

resposta = requests.get(BASE_URL, params={"page": 1}, timeout=10)
resposta.raise_for_status()
dados = resposta.json()
print(dados)

Replace BASE_URL with your API's real endpoint. Printing the response lets you see the shape of the data — usually a dictionary with a list of records inside it.

Step 2: Understand how the API paginates

There are two very common pagination styles, and you need to know which one your API uses (its documentation will tell you). In the first, you pass the page number in a parameter such as ?page=2. In the second, each response includes the address of the next page in a field, often called next. APIs built with Django REST Framework, for example, return the fields results, next and count — the structure we will use in the examples.

Tip: before writing the loop, read the API documentation to find the page parameter's name, the maximum page size and the request-per-minute limit.

Step 3: Loop through every page by number

The idea is simple: you start at page 1 and keep asking for the next page until the API returns an empty list, which means there is no more data. A counter and a while loop are enough for this.

todos = []
pagina = 1

while True:
    resposta = requests.get(BASE_URL, params={"page": pagina}, timeout=10)
    resposta.raise_for_status()
    lote = resposta.json()["results"]
    if not lote:
        break
    todos.extend(lote)
    pagina += 1

print(f"Total de registos: {len(todos)}")

On each pass of the loop we request a page, add its records to the todos list with extend, and move the counter forward. When lote comes back empty, the break ends the loop.

Step 4: Follow the "next" field (cursor pagination)

If your API returns the address of the next page, you don't even need to count pages: you follow the next field until it is empty. This method is even more robust, because it is the API itself telling you the next step.

todos = []
url = BASE_URL

while url:
    resposta = requests.get(url, timeout=10)
    resposta.raise_for_status()
    dados = resposta.json()
    todos.extend(dados["results"])
    url = dados.get("next")

print(f"Total de registos: {len(todos)}")

As long as there is a next, the loop continues. When the last page has next set to null, get returns None, the while condition becomes false and the loop ends naturally.

Step 5: Avoid errors and respect the API

Going through many pages quickly can make the API block you for too many requests (the well-known 429 Too Many Requests error). A short pause between requests and a safety limit make your code more polite and safer against infinite loops.

import time

todos = []
pagina = 1
MAX_PAGINAS = 500

while pagina <= MAX_PAGINAS:
    resposta = requests.get(BASE_URL, params={"page": pagina}, timeout=10)
    resposta.raise_for_status()
    lote = resposta.json()["results"]
    if not lote:
        break
    todos.extend(lote)
    pagina += 1
    time.sleep(0.2)

print(f"Total de registos: {len(todos)}")

Check the result

How do you know you really captured every record? Many APIs include a field with the total, such as count. Just compare that number with the size of your list:

print(len(todos))
print(dados["count"])

If the two values match, pagination went through everything. Alternatively, confirm that the last page returned fewer records than the normal page size — a sign that you reached the end.

Conclusion

You now know how to paginate a REST API in Python in the two most common formats — by page number and by the next field — and how to protect your code with pauses and limits. The natural next step is to store the data from todos in a pandas DataFrame or a database for analysis. Which of your APIs will you paginate first?