Early access is open. Free sample datasets are live now. Download a sample

Pagination and Large Requests

Details

Docs navigation

Ten years of 1-minute bars for one stock is close to a million rows, and a busy day of trades can be several million. No API returns that in one response. This page explains the cursor-based API pagination we use, the page size limits, and the patterns that work for very large pulls, including when to stop paging and buy a file instead.

Early access. Describes the planned v1 API.

How it works

How paging works

1. Make the first request

Ask for the range you want. Add limit to set the page size.

Example (curl):

curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://api.marketdata.co/v1/stocks/bars?symbol=NVDA&interval=1m&start=2026-01-01&end=2026-07-01&limit=50000"

2. Check next_cursor

{
  "status": "ok",
  "count": 50000,
  "next_cursor": "eyJ0IjoiMjAyNi0wMy0xMlQxNTo0Mjo",
  "results": [ ... ]
}

If next_cursor is a string, there is more data. A null means you have everything.

3. Request the next page

Send the same request with cursor set to that value. Keep the other parameters identical.

Example (curl):

curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://api.marketdata.co/v1/stocks/bars?symbol=NVDA&interval=1m&start=2026-01-01&end=2026-07-01&limit=50000&cursor=eyJ0IjoiMjAyNi0wMy0xMlQxNTo0Mjo"

4. Loop until done

Example (Python):

import os
import requests
import pandas as pd

API = "https://api.marketdata.co/v1"
H = {"Authorization": f"Bearer {os.environ['MARKETDATA_API_KEY']}"}

def fetch_all(path, params):
    params = dict(params, limit=50000)
    frames = []
    while True:
        r = requests.get(f"{API}{path}", headers=H, params=params, timeout=60)
        r.raise_for_status()
        body = r.json()
        frames.append(pd.DataFrame(body["results"]))
        if body["next_cursor"] is None:
            break
        params["cursor"] = body["next_cursor"]
    return pd.concat(frames, ignore_index=True)

nvda = fetch_all("/stocks/bars", {"symbol": "NVDA", "interval": "1m", "start": "2026-01-01", "end": "2026-07-01"})

Details

Page sizes

Endpoint type Default limit Maximum limit
Bars (stocks, options, FX, crypto, futures, indices) 5,000 50,000
Trades and quotes (ticks) 5,000 50,000
Reference lists (tickers, contracts) 1,000 10,000
Corporate actions, filings, calendars 1,000 10,000

Results are sorted by time, oldest first. Pass sort=desc for newest first; the cursor follows the same order.

Details

Patterns for big pulls

Split by time as well as by page. For tick data, request one trading day per call and page within it. If something fails, you redo one day, not a month.

Run symbols in parallel, pages in sequence. A cursor depends on the page before it, so pages for one query must run in order. Different symbols can run side by side, up to your rate limit.

Write each page to disk. Append each page to a Parquet or CSV file as it arrives. A crash at page 37 should not lose pages 1 to 36.

Count your calls first. Rows divided by page size is the number of requests. One year of 1-minute bars for one stock is about 98,000 regular-session rows, or two pages at 50,000. A year of trades for a large stock can run to hundreds of pages.

Consider a file. If you need the same history for hundreds of symbols, a one-off dataset download is faster and cheaper. US stock 1-minute history is $5 per symbol as a file.

Watch out

Notes and gotchas

  • Cursors are tied to the query that produced them. Change any parameter and you get 400 invalid_cursor.
  • A cursor expires after 24 hours. Restart the query if a job was paused overnight.
  • A page can hold fewer rows than limit and still have a next_cursor. Check the cursor, not the row count.
  • range_too_large means one call spans too much for the interval (for example 20 years of 1s bars). Split the range into smaller windows.
  • Each page counts as one request against your rate limit.
  • CSV responses (format=csv) return the cursor in the X-Next-Cursor header, since the body has no room for it.

Keep reading

Next steps

Details

Was this page useful?

[Yes] [No]

Need a bulk pull we do not cover? Tell us on the custom data form.