How Do Unified APIs Handle Pagination Differences Across REST APIs?

An engineering team agrees to build five native integrations to unblock a major enterprise deal. The initial proof of concept goes smoothly. The developer reads the API documentation, writes a quick HTTP GET request, maps the JSON response to the internal schema, and merges the pull request. Getting the first page of results is easy.

The illusion shatters the moment that integration hits production and attempts to sync an enterprise customer's historical data.

Salesforce uses cursor-based pagination with nextRecordsUrl. HubSpot returns a paging.next.after token. Zendesk is actively migrating from offset to cursor pagination. Jira uses startAt offsets. And that GraphQL-only API from Linear? It wraps cursors inside relay-style edges and pageInfo objects.

You write a generic while loop to fetch records. It works perfectly for the first provider. Then you connect the second, and the loop breaks. The first API expects a ?page=2 query parameter. The second expects an ?offset=100 parameter. The third returns a cursor string hidden inside a nested meta.paging.next JSON object. The fourth uses RFC 8288 Link headers. The fifth is a GraphQL endpoint that requires variables passed in a POST body.

This is the hidden operational cost of building integrations, and it is the reason engineering leaders eventually look for ways to normalize pagination and error handling across 50+ APIs without building it yourself. This article breaks down why REST API pagination is harder than it looks, the trade-offs between common pagination methods, and how a unified API abstracts the mess into a single interface your team codes against once.

The Hidden Complexity of REST API Pagination

Fetching the first page of data from any API is easy. Every junior developer can write a GET /contacts?limit=10 and parse the JSON. The pain starts on page two.

Each API vendor made independent design choices about how to signal "there's more data." Some return a next_cursor string. Some return a full URL in a Link header. Some give you a totalResults count and expect you to do arithmetic with offset and limit. Some bury pagination tokens inside deeply nested response objects. And a few return a has_more boolean that lies.

Companies globally use 106 SaaS applications on average, down from 112 in 2023 - a 5% drop since the 2022 peak of 130 apps, with usage falling 18%. When your product needs to pull data from even a fraction of those tools, you are looking at dozens of distinct pagination implementations to build, test, and keep running.

The real cost is not the initial implementation. The average annual integration maintenance cost usually runs between 10% and 20% of the initial development cost. For pagination specifically, the maintenance burden comes from silent breaking changes: a vendor switches from offset to cursor pagination (as Zendesk did), changes the field name of its cursor token, or starts enforcing stricter rate limits on deep pagination requests. Each of these changes requires your team to detect, diagnose, and fix integration-specific code.

Engineers spend weeks writing custom pagination loops, state management logic, and error handling for every single third-party provider. If an API times out on page 400 because of an inefficient offset query, the sync job fails, and the engineering team has to build custom retry logic just for that specific endpoint. Instead of maintaining dozens of bespoke pagination scripts, they need a single abstraction layer.

Offset vs. Cursor vs. Page: Why APIs Can't Agree on Pagination

Before looking at how an abstraction layer works, we have to understand why APIs paginate differently in the first place. The differences are not arbitrary. They stem from underlying database architectures, the era in which they were built, and the specific scaling challenges of the companies that built them.

Offset-Based Pagination

The client sends limit and offset parameters. The server skips offset rows and returns the next limit rows.

GET /api/contacts?limit=100&offset=10000

On the backend, this usually translates directly to a SQL query:

SELECT * FROM contacts ORDER BY created_at DESC LIMIT 100 OFFSET 10000;

APIs use this because it maps directly to SQL's LIMIT and OFFSET clauses. It is simple to implement, easy for clients to understand, and supports random page access.

But offset pagination degrades badly as datasets grow. When dealing with very large datasets, pagination using LIMIT OFFSET often experiences performance degradation because every time we request a new page, the database must scan the entire table from the beginning to find the appropriate data. Postgres must scan and discard 100,000 rows before returning the desired 10 rows. This can be highly inefficient and slow.

Beyond performance, offset pagination has a data consistency problem. If a new item is added to the beginning of the list while a user is paginating, every subsequent page will be shifted. An item the user saw on page 2 might reappear on page 3. In a sync job that takes two hours to paginate through a million records, any records created or deleted during that window cause the offset math to shift, resulting in missed or duplicated data.

Cursor-Based Pagination

Modern, high-volume APIs (like Stripe, Slack, HubSpot, and Shopify) almost exclusively use cursor-based pagination. Instead of passing an offset number, the client passes an opaque token that points to a specific position in the dataset.

GET /api/contacts?limit=100&after=eyJpZCI6OTg3Nn0=

The backend decodes this cursor and uses it in a WHERE clause:

SELECT * FROM contacts WHERE id > 9876 ORDER BY id ASC LIMIT 100;

Cursor pagination maintains high performance even with large datasets since it works by directly accessing the position marked by the cursor. The database jumps directly to the indexed position instead of scanning and discarding rows. Inserts or deletes before the current page do not affect the integrity of the current page or subsequent pages. Users will not see duplicate records or miss records they were supposed to see.

The trade-off: you lose random page access. Navigation is typically forward (and sometimes backward), but skipping directly to page 50 isn't possible. This makes cursor pagination less practical when accessing random pages.

Slack's engineering team published a detailed account of their migration that illustrates why the industry is moving in this direction. The size and scope of the data they expose via their APIs has changed dramatically since the product first launched. Endpoints designed around the expectation of returning several hundred records are now returning hundreds of thousands of records. To handle this growth, they had to rethink how they paginate data - from no pagination, to offset pagination, to a new cursor-based pagination scheme.

Page-Number and Keyset Pagination

Page-number pagination is simply offset pagination with the math hidden. A request for ?page=5&per_page=100 forces the backend to calculate the offset ((page - 1) * per_page) and execute the exact same inefficient SQL query. It shares all the same performance and consistency problems, plus it tempts UI designers to build "jump to page" controls that crumble under large datasets.

Keyset pagination, often confused with cursor pagination, uses existing ordered columns (like created_at or id) to paginate. This avoids the offset scanning penalty but requires the client to track the last seen value of the sort column, which becomes complex when sorting by non-unique fields.

Quick Reference: Pagination in the Wild

How Unified APIs Handle Pagination Differences Across Different REST APIs

A unified API handles pagination differences by acting as a translation layer between the client and the provider. It exposes a single, standardized interface (typically using limit and next_cursor parameters) and automatically translates these values into the specific pagination format required by the underlying REST API — whether that is offsets, page numbers, cursor tokens, or GraphQL variables.

When you use a unified API, your request always looks the same, regardless of whether you are querying Salesforce, Zendesk, or HubSpot:

GET /unified/crm/contacts?integrated_account_id=abc&limit=50&next_cursor=abc123xyz

And you always get back:

{
  "result": [{"id": "1", "name": "Jane Doe"}, {"id": "2", "name": "John Smith"}],
  "next_cursor": "dHJ1dG9fY3Vyc29yXzEyMw==",
  "prev_cursor": null,
  "result_count": 50
}

Same parameters. Same response shape. Whether the backend is HubSpot (cursor-based), Salesforce (SOQL query cursor), Pipedrive (page-number), or Zendesk (offset migrating to cursor) — your code does not change.

The unified API layer does the heavy lifting through a multi-step execution pipeline:

Step 1: Request Interception and Parameter Mapping

When the unified API receives your request, it identifies the target integration and loads the corresponding pagination configuration. It translates your limit into the provider's parameter — page_size, per_page, count, $top, first (GraphQL), or whatever the vendor decided to call it. It decodes your next_cursor (which it generated during the previous request) and translates it into the provider's pagination mechanism — an after token, a startAt offset, a page number, a Link header URL, or GraphQL cursor variables.

Step 2: Provider Execution

The unified API makes the raw HTTP request to the third-party provider using the translated parameters.

GET /api/v2/users.json?page=4&per_page=50

Step 3: Response Extraction and Normalization

When the provider returns the response, the unified API extracts the next page indicator — which could be in paging.next.after, meta.cursor, a response header, or pageInfo.endCursor nested three levels deep. The unified API encodes this provider-specific state into an opaque string and returns it to your application as a standardized next_cursor.

Your engineering team writes exactly one while (response.next_cursor) loop. The unified API absorbs the complexity of the 50 different ways the underlying APIs actually paginate.

sequenceDiagram
    participant App as Your Application
    participant UA as Unified API
    participant Config as Config DB
    participant P as Provider API
    
    App->>UA: GET /unified/crm/contacts?limit=20
    UA->>Config: Load integration config & pagination strategy
    Config-->>UA: Return JSON config (Strategy: Cursor)
    UA->>P: GET /crm/v3/objects/contacts?limit=20 (HubSpot)
    P-->>UA: {results: [...], paging: {next: {after: "abc"}}}
    UA->>UA: Extract cursor from paging.next.after
    UA-->>App: {result: [...], next_cursor: "encoded_abc"}
    
    App->>UA: GET /unified/crm/contacts?limit=20&next_cursor=encoded_abc
    UA->>UA: Decode cursor, map to provider format
    UA->>P: GET /crm/v3/objects/contacts?limit=20&after=abc
    P-->>UA: {results: [...], paging: {next: {after: "def"}}}
    UA-->>App: {result: [...], next_cursor: "encoded_def"}

The important insight: your application code only ever handles cursor-based pagination — even when the underlying provider uses offset, page numbers, or something entirely custom. The unified layer absorbs all the complexity.

The Truto Approach: Declarative Pagination Without Integration-Specific Code

This is where architectural approaches diverge sharply between unified API vendors. Most platforms handle pagination by writing provider-specific adapter code: a HubSpot pagination handler, a Salesforce pagination handler, a Zendesk pagination handler. Behind their unified facade, they maintain separate code paths for each integration: if (provider === 'hubspot') { handleHubspotPagination() } else if (provider === 'salesforce') { handleSalesforcePagination() }. This code-heavy approach is brittle. When a provider updates their API, the platform has to rewrite the adapter, run tests, and deploy new code.

Truto takes a radically different approach. The entire platform contains zero integration-specific code. The declarative pagination system in Truto's Unified Real-time API handles pagination entirely through JSON configuration and JSONata expressions.

The system supports six pagination strategies out-of-the-box:

Each integration's pagination behavior is defined in a configuration object, not in code. Here is what a cursor-based pagination config looks like:

{
  "pagination": {
    "format": "cursor",
    "config": {
      "cursor_field": "paging.next.after",
      "cursor_query_param": "after",
      "limit_param": "limit"
    }
  }
}

And an offset-based one:

{
  "pagination": {
    "format": "offset",
    "config": {
      "offset_param": "startAt",
      "limit_param": "maxResults",
      "start_offset": 0
    }
  }
}

The runtime engine reads this configuration and executes the appropriate pagination strategy without any conditional logic like if (provider === 'hubspot'). The same generic code path handles all six strategies for all integrations.

Because Truto handles pagination entirely through configuration, bug fixes to the pagination engine instantly apply to all 100+ integrations. If we optimize how the offset strategy calculates limits, every single integration using offset pagination immediately benefits. Adding a new integration does not require writing new code — it only requires adding a new JSON configuration record to the database.

The dynamic pagination strategy deserves special attention. Some APIs do not fit neatly into any standard pattern — they return pagination tokens in unusual locations, require computing the next token from response metadata, or use non-standard signals for "end of results." Truto handles these with JSONata expressions that can compute the next cursor, check whether more data exists, and determine where to inject the pagination parameter — all declared in configuration, not code.

An important nuance: pagination can also be disabled per-resource. If an integration's default config specifies cursor pagination but a specific endpoint (like "get a single record") does not paginate, the resource method config can override it with "pagination": null. This avoids sending unnecessary pagination parameters to endpoints that do not support them.

Simple integrations start at $2,000, but complex API projects can cost more than $30,000, especially when you have development and maintenance needs. Organizations need $50,000 to $150,000 yearly to cover staff and partnership fees. Pagination alone is not the entire cost — but it is the kind of per-provider logic that multiplies your maintenance burden across every integration you support.

Handling Edge Cases: GraphQL, Rate Limits, and Sync Jobs

Abstracting pagination is only half the battle. In production environments, pagination intersects with other API complexities that can easily break a naive sync loop.

GraphQL APIs That Only Speak Cursors

APIs like Linear, GitHub GraphQL, and others expose a single POST /graphql endpoint. There is no GET /contacts?page=2. Pagination is embedded in the GraphQL query variables, and building a pagination loop for a GraphQL API requires completely different logic than a REST API.

Truto's proxy layer maps REST-style pagination parameters to GraphQL variables using template-based request building. When your application sends GET /proxy/issues?limit=20&next_cursor=abc, the system constructs a GraphQL body:

{
  "query": "query($first: Int, $after: String) { issues(first: $first, after: $after) { nodes { id title } pageInfo { endCursor hasNextPage } } }",
  "variables": {
    "first": 20,
    "after": "abc"
  }
}

The response_path configuration extracts data.issues.nodes for the results and data.issues.pageInfo.endCursor for the next cursor. Your application sees standard REST pagination. The GraphQL complexity is entirely encapsulated in configuration.

When Pagination Hits Rate Limits

Paginating through large datasets means making many sequential API calls. When a third-party API rate-limits you mid-pagination — say, on page 45 — you need to handle it gracefully, or your data sync silently fails.

If you build integrations in-house, your pagination loop has to catch the error, parse the provider's specific rate limit headers (which might be Retry-After, X-RateLimit-Reset, or a custom JSON payload), pause execution, and retry the exact same page.

Truto normalizes rate limit responses across all integrations into a standard format. Whether the provider returns HTTP 429, a 200 with a rate-limit error in the body, or a custom header, the client always receives a standard 429 status with a Retry-After header in seconds. Your application's HTTP client only needs to know how to read one standard header to safely pause and resume the pagination loop.

Sync Jobs, Data Drift, and Bypassing Pagination Entirely

When syncing massive datasets, data drift is a serious concern. If a sync job takes two hours to paginate through a million records via offset pagination, any records created or deleted during that window shift the offset math, resulting in missed data. Truto mitigates this by allowing developers to combine real-time webhook ingestion with cursor-based pagination. The initial sync pulls the historical data, while webhooks keep the state aligned without requiring constant, deep pagination loops.

For scenarios where you need to query large datasets repeatedly — analytics, search, bulk reporting — paginating through a live API every time is wasteful and slow. Truto supports a concept called SuperQuery, where data is synced to a local datastore and queried with SQL-style filtering, sorting, and pagination. This bypasses the third-party API entirely for read operations, eliminating both pagination complexity and rate-limit concerns on bulk reads.

Build vs. Buy: Escaping the Integration Maintenance Trap

Let's be direct about the trade-offs. Using a unified API for pagination normalization means you are adding a dependency. Your API calls go through an intermediary layer. You have less control over raw request parameters. If the unified API vendor has a bug in their pagination config for a specific provider, you are blocked until they fix it.

Those are real costs. Here is the calculus on the other side.

If your product integrates with 10 CRMs, you are maintaining 10 distinct pagination implementations — each with its own parameter names, cursor extraction logic, end-of-results detection, and error handling. When Zendesk migrates from offset to cursor pagination (which they are actively doing), you need an engineer to notice, update, test, and deploy the change.

APIs evolve — fields are added, endpoints deprecate, payloads change under load — and customers surface edge cases you did not see in staging. Continuous monitoring detects anomalies early; SLOs define acceptable sync latency and error budgets; and runbooks guide first-line diagnosis. Support handles user questions and configuration issues, while engineering adapts to API changes, fixes defects, and tunes performance as data volume grows. In practice, teams should expect a steady maintenance cadence over the year.

Gartner expects public cloud end-user spending to eclipse the one trillion dollar mark before the end of this decade. Worldwide spending on public cloud services grew 20.4% to total $675.4 billion in 2024. This growth is driven by generative AI and application modernization. The SaaS ecosystem is expanding, not contracting. The number of APIs your product needs to integrate with is going up, and integrations are consistently ranked among the top priorities on global B2B buyers' lists — meaning missing integrations directly impact retention and sales.

The decision comes down to whether pagination and integration plumbing is your product's competitive advantage. For the vast majority of B2B SaaS companies, it is not. Your competitive advantage is the workflow you build on top of the data — the pipeline that syncs CRM contacts into your analytics engine, the automation that triggers when an HRIS record changes. Spending senior engineering time debugging why Pipedrive's page-number pagination returns duplicates when a contact is deleted mid-sync is not a good use of that talent.

What to Look For When Evaluating a Unified API's Pagination

If you are evaluating unified API providers (including Truto), here are the specific questions to ask:

• How many pagination strategies are supported? If a provider only handles cursor and offset, you will hit gaps with APIs that use Link headers, range-based pagination, or GraphQL relay-style pagination.
• Is pagination configuration declarative or code-based? Declarative (config-driven) approaches mean bug fixes apply globally. Code-based adapters mean each integration is a separate maintenance burden for the vendor.
• What happens during rate limits mid-pagination? Does the unified API surface a standard retry-after signal, or do you get provider-specific error formats?
• Can pagination be disabled per-endpoint? Not every endpoint on an API supports pagination. If the unified layer sends pagination parameters to a non-paginated endpoint, you will get unexpected errors.
• Is there a synced data option for bulk reads? Paginating through a live API to export 50,000 records is a bad experience for everyone. A unified API that supports pre-synced data gives you a better path for those use cases.
• How is the raw response preserved? Unified schema fields are useful, but you sometimes need the original provider response. Look for a remote_data or equivalent field that gives you access to the untransformed response.

Start Here, Not From Scratch

Pagination is one of those problems that looks trivial until you are maintaining it across 20+ API integrations. The inconsistency across providers is not an accident — it reflects different architectural eras, different database technologies, and different engineering teams making reasonable but incompatible choices.

Your options are: build a generic pagination abstraction layer internally (a significant engineering investment), live with the maintenance overhead of per-provider pagination code (the cost multiplies linearly with each new integration), or use a unified API that has already solved it.

If you are leaning toward the unified API path, Truto's declarative architecture means you get support for six pagination strategies, GraphQL-to-REST proxying, rate-limit normalization, and synced data queries — all without writing provider-specific code. That is the engineering leverage that lets your team focus on your actual product.