API Paginator — Fetch All Pages

Difficulty: Medium · Topics: Pagination, Async, API, Iterator

Difficulty: Medium | Tags: API, Pagination, Async, Iterator

Description

Given an API endpoint that returns paginated results, implement a utility that automatically fetches all pages and returns the complete dataset. Support different pagination strategies.

Implement:

APIPaginator
  fetchAll(baseUrl, options) → List<Object>     — fetch all pages
  fetchPage(baseUrl, page) → PageResult          — fetch single page
  stream(baseUrl) → Iterator<Object>             — lazy page-by-page iterator

Pagination strategies:

Offset-based: ?offset=0&limit=25
Page-based: ?page=1&per_page=25
Cursor-based: ?cursor=abc123&limit=25

Examples

Example 1 — Offset pagination:

Input:
  baseUrl  = "https://api.example.com/users"
  strategy = "offset"
  pageSize = 25

  Page 1: GET /users?offset=0&limit=25   → {data: [...25 items], total: 73}
  Page 2: GET /users?offset=25&limit=25  → {data: [...25 items], total: 73}
  Page 3: GET /users?offset=50&limit=25  → {data: [...23 items], total: 73}

Output: List of all 73 users

Example 2 — Cursor pagination:

Input:
  baseUrl = "https://api.example.com/posts"
  strategy = "cursor"

  Page 1: GET /posts?limit=10        → {data: [...], next_cursor: "abc"}
  Page 2: GET /posts?cursor=abc&limit=10 → {data: [...], next_cursor: "def"}
  Page 3: GET /posts?cursor=def&limit=10 → {data: [...], next_cursor: null}

Output: All posts concatenated

Example 3 — Lazy streaming:

iterator = paginator.stream("https://api.example.com/records")
for item in iterator:
    process(item)    // fetches next page automatically when current exhausted

Constraints

Handle rate limiting (retry after Retry-After header).
Support configurable page size (default 25, max 100).
Stop fetching when: empty page, no next cursor, or offset >= total.
Timeout per request: 30 seconds.
Total max pages to prevent infinite loops: 1000.

Hints

Hint 1

Use a while loop that breaks when the termination condition for the pagination strategy is met.

Follow-up

Add concurrent page fetching (fetch pages 2-5 in parallel while processing page 1).
Add retry logic with exponential backoff per page.

Practice this problem

Solve API Paginator — Fetch All Pages in the browser