CRM Integration Implementation Recipes: Salesforce & HubSpot Architecture (2026)

Building a CRM integration to Salesforce or HubSpot looks trivial in a sandbox environment. You authenticate once, make a single HTTP request to a contacts endpoint, parse a JSON response, and render it in your UI. Then you push it to production with a hundred enterprise customer accounts behind it and watch HTTP 429s, REQUEST_LIMIT_EXCEEDED errors, broken pagination cursors, and __c custom field mismatches turn your sprint into a six-month maintenance project.

The market opportunity is massive. Fortune Business Insights reports the B2B SaaS market is expanding rapidly, projected to reach $634.39 billion in 2026 at a 27.54% CAGR. However, the European B2B CX Benchmark Report 2025-2026 indicates companies typically manage customer data across an average of 12 different systems. Salesforce and HubSpot sit at the center of that fragmented ecosystem, which means your integration is not just talking to a CRM - it is competing for API budget with every other middleware tool already installed in your customer's org.

This guide provides actionable architectural blueprints for building native-feeling integrations with Salesforce and HubSpot, specifically addressing the technical hurdles that cause integrations to fail at scale. If you want the wider context on why these projects drain engineering time, read Building Native CRM Integrations Without Draining Engineering in 2026. The rest of this post is your technical recipe book.

The "Just a Few API Calls" Trap: Why CRM Integrations Fail in Production

B2B SaaS CRM integration failure occurs when engineering teams scope third-party API connections based on vendor "Getting Started" guides, ignoring the production realities of strict rate limits, undocumented pagination behaviors, and highly customized data schemas.

A CRM integration is not a simple HTTP client. It is a distributed system, especially when architecting real-time CRM syncs for enterprise. When your product manager asks for a Salesforce integration, the initial engineering estimate usually covers the basic HTTP request path. That represents roughly 10% of the actual work. The remaining 90% involves managing OAuth token lifecycles, handling burst rate limits across tenants, building exponential backoff queues, and dealing with custom objects that break your static data models.

Three failure modes show up in production for almost every team:

• Burst exhaustion: Your sync runs fine at 10 customers and falls over at 100 because you treated rate limits as a single number instead of a token bucket.
• Custom schema drift: Half your enterprise accounts have custom objects or properties you have never seen, and your hardcoded field mappings start returning null for the data customers actually care about.
• Silent pagination loss: Cursor-based pagination breaks when a record is created or deleted mid-walk, and your incremental sync starts missing records without throwing a single error.

The rest of this article gives you a recipe for each of these, then explains the architectural shift that removes most of the work.

Bidirectional Sync Architecture: Real-Time vs Batch

Before diving into individual recipes, it helps to understand the full picture. The question most teams are really asking is: "How do I sync customer data bidirectionally between my app and HubSpot (or Salesforce) without things breaking?" The answer has three moving parts.

1. Inbound path (CRM → Your App): The CRM pushes change events to your webhook endpoint. Your app processes them, maps them to your internal data model, and writes them to your database.
2. Outbound path (Your App → CRM): When a record changes in your app, you push the update to the CRM's API. You need to prevent the resulting CRM webhook from echoing back and triggering a loop.
3. Repair sync (periodic reconciliation): A scheduled batch job that sweeps updatedAt windows in both directions, catching anything the real-time path missed - dropped webhooks, failed writes, or drift that accumulates over time.

Real-time sync (webhooks) gives you sub-minute latency for most changes. Batch sync (polling lastmodifieddate windows) gives you completeness guarantees. You need both. If you skip the repair path, you are not building sync - you are building a demo.

flowchart LR
    subgraph RT[Real-Time Path]
        direction TB
        A1[CRM Webhook] -->|contact.propertyChange| B1[Your Webhook Endpoint]
        B1 --> C1[Map + Write to App DB]
        D1[App Record Changed] --> E1[Push to CRM API]
        E1 --> F1[Store Outbound Fingerprint]
    end
    subgraph RP[Repair Path]
        direction TB
        G1[Scheduled Job] --> H1[Poll CRM<br>updatedAt > last_sync]
        H1 --> I1[Diff + Reconcile]
        G1 --> J1[Poll App DB<br>updated_at > last_sync]
        J1 --> I1
    end

Preventing Infinite Loops with Fingerprints

The echo problem is the single biggest trap in bidirectional sync. When your app writes a contact update to HubSpot, HubSpot fires a contact.propertyChange webhook back to your endpoint. If you process that event naively, it writes back to your app, which triggers another outbound push, creating an infinite loop.

The standard prevention pattern:

1. Before writing to HubSpot, compute a fingerprint (hash of the field values you are sending) and store it keyed by (record_id, direction).
2. When an inbound webhook arrives, compute the fingerprint of the incoming data.
3. If it matches a stored outbound fingerprint, it is an echo - acknowledge and skip.
4. If it does not match, it is a genuine change from a HubSpot user - process it normally.

This pattern works regardless of the CRM. It is the same for Salesforce outbound messages, Pipedrive webhooks, or any other provider. The recipes below cover the specific implementation details for each part of this pipeline. For a deeper walkthrough of the full bidirectional architecture, see How to Architect a Bidirectional HubSpot Sync (Without Infinite Loops).

Recipe 1: Navigating HubSpot API Rate Limits and Pagination

As covered in our 2026 architecture guide to building a HubSpot integration, HubSpot's API is generally developer-friendly, utilizing standard REST conventions and JSON payloads. However, its rate limiting strategy is aggressive. HubSpot's API is the textbook example of a token bucket model. Misread it as a simple "X requests per second" cap and you will burn your daily quota fighting 429s.

The Actual HubSpot Rate Limit Numbers (2026)

For public OAuth apps, the burst limit is 110 requests per 10 seconds per installed account. For private apps on Pro and Enterprise tiers, the burst limit is 190 requests per 10 seconds.

Daily quotas matter more than people think. Free/Starter plans allow 250,000 daily requests. Enterprise plans offer up to 1,000,000 daily, with add-ons available for more capacity.

The Search API is the silent killer. The CRM search API (/crm/v3/objects/contacts/search, etc.) is capped at 5 requests per second across all search endpoints. That is not per endpoint - it is shared across contacts, deals, companies, and every other object type. If you use search for deduplication checks on inbound writes, it - not the burst limit - becomes your bottleneck.

Implementing a Token Bucket Rate Limiter

To handle this, your architecture must decouple integration logic from direct HTTP execution. You need an outbound request queue governed by a token bucket algorithm.

graph TD
    A[Application Logic] -->|Enqueue Request| B(Redis Queue)
    B --> C{Token Bucket Check}
    C -->|Tokens Available| D[Execute HTTP Request]
    C -->|Tokens Depleted| E[Delay & Re-queue]
    D --> F{Response Status}
    F -->|200 OK| G[Process Data]
    F -->|429 Too Many Requests| H[Exponential Backoff]
    H --> B

A naive sleep(100) between requests leaves throughput on the table. The token bucket model means you can spike above the average rate temporarily. A well-designed client takes advantage of this: send requests as fast as you need to, but build in breathing room so the bucket recovers.

Here is a practical TypeScript implementation of a token bucket sized for HubSpot's public app tier:

// Token bucket for HubSpot - 110 burst, ~10 tokens/sec refill (public app)
class HubSpotRateLimiter {
  private tokens = 110
  private lastRefill = Date.now()
  private readonly refillPerMs = 10 / 1000
  private readonly capacity = 110
 
  async acquire(): Promise<void> {
    this.refill()
    if (this.tokens >= 1) {
      this.tokens -= 1
      return
    }
    const waitMs = Math.ceil((1 - this.tokens) / this.refillPerMs)
    await new Promise(r => setTimeout(r, waitMs))
    return this.acquire()
  }
 
  private refill() {
    const now = Date.now()
    this.tokens = Math.min(
      this.capacity,
      this.tokens + (now - this.lastRefill) * this.refillPerMs
    )
    this.lastRefill = now
  }
}

Pair this with exponential backoff and jitter on 429s. If you receive an HTTP 429 response, you must respect the Retry-After header. Pair that with a small randomized delay (0-9 seconds of jitter) before each outbound call so 10,000 enrollments don't align perfectly. If you continue to hammer the API without jitter, HubSpot will temporarily ban your OAuth application from making further requests for that specific tenant.

Reading HubSpot Rate Limit Headers for Adaptive Throttling

The static token bucket above works, but you can do better. HubSpot returns rate limit state in every response via headers. Reading these headers lets your client adapt in real-time instead of guessing.

The relevant headers:

Here is an adaptive rate limiter that syncs its internal state from HubSpot's response headers on every call:

class AdaptiveHubSpotLimiter {
  private remaining = 110
  private windowMs = 10_000
  private windowStart = Date.now()
 
  updateFromHeaders(headers: Headers) {
    const rem = headers.get('X-HubSpot-RateLimit-Remaining')
    const max = headers.get('X-HubSpot-RateLimit-Max')
    const interval = headers.get('X-HubSpot-RateLimit-Interval-Milliseconds')
    if (rem !== null) this.remaining = parseInt(rem, 10)
    if (interval !== null) this.windowMs = parseInt(interval, 10)
  }
 
  async acquire(): Promise<void> {
    if (this.remaining > 5) { // keep a 5-request safety buffer
      this.remaining--
      return
    }
    const elapsed = Date.now() - this.windowStart
    const waitMs = Math.max(0, this.windowMs - elapsed) + 100
    await new Promise(r => setTimeout(r, waitMs))
    this.windowStart = Date.now()
    this.remaining = 110 // reset to burst capacity
  }
}
 
// Usage: wrap every outbound HubSpot call
async function hubspotFetch(
  url: string, opts: RequestInit, limiter: AdaptiveHubSpotLimiter
) {
  await limiter.acquire()
  const res = await fetch(url, opts)
  limiter.updateFromHeaders(res.headers)
  if (res.status === 429) {
    const retryAfter = parseInt(res.headers.get('Retry-After') ?? '10', 10)
    await new Promise(r => setTimeout(r, retryAfter * 1000 + Math.random() * 1000))
    return hubspotFetch(url, opts, limiter) // retry once
  }
  return res
}

The key detail: always keep a safety buffer of 5-10 remaining requests. If you drain the bucket to zero, concurrent requests in flight will all 429 simultaneously. The buffer gives you room to decelerate gracefully.

Handling Cursor-Based Pagination and Batching

If you only learn one thing about HubSpot efficiency: batch endpoints count as one request. HubSpot's batch create/update endpoints let you process up to 100 records in a single API call. From a rate limit perspective, that is 100x more efficient than individual calls.

API pagination consistency is another major hurdle. HubSpot uses cursor-based pagination. When you request a list of contacts, the response includes a paging.next.after token.

{
  "results": [
    { "id": "123", "properties": { "firstname": "Alice" } }
  ],
  "paging": {
    "next": {
      "after": "NTI1Cg%3D%3D",
      "link": "?after=NTI1Cg%3D%3D"
    }
  }
}

Your integration must recursively or iteratively fetch pages using this after parameter until the paging.next object is null. Do not rely on offset-based pagination for HubSpot, as it is inefficient and HubSpot reorders results on deletes, meaning you will drop records. For more on bidirectional sync challenges, review our bidirectional HubSpot sync tutorial.

Special Case: CRM Search API Pacing

The CRM Search API deserves its own rate limiter. As noted above, the standard burst limit (110 requests / 10 seconds for public apps) does not apply to search endpoints. The Search API is capped at approximately 4-5 requests per second, shared across all search endpoints at the account level - contacts, deals, companies, tickets, everything.

This has three practical implications:

1. Separate your queues. Run search requests through a dedicated limiter with a 4 req/s ceiling, independent of your CRUD limiter. If you mix them in one queue, a burst of search calls will starve your other operations.
2. Maximize page size. The Search API supports limit up to 200 records per response. At 4 requests per second with 200 records each, you get 800 records/second throughput. At the default of 10 records per page, you get 40 records/second. That is a 20x difference.
3. Respect the 10,000-result cap. A single search query can only return 10,000 results total, even with pagination. If you need more, segment your queries by a range filter (e.g., createdate ranges) so each segment stays under 10,000.

// Dedicated search limiter - 4 req/s with 250ms spacing
class SearchApiLimiter {
  private lastCall = 0
  private readonly minGapMs = 250 // 4 req/s = 250ms between calls
 
  async acquire(): Promise<void> {
    const now = Date.now()
    const elapsed = now - this.lastCall
    if (elapsed < this.minGapMs) {
      await new Promise(r => setTimeout(r, this.minGapMs - elapsed))
    }
    this.lastCall = Date.now()
  }
}

If you use search for deduplication checks during writes (e.g., "does this email already exist?"), batch those checks before you start writing. Running interleaved search-then-write loops is the fastest way to exhaust your search budget.

Recipe 2: Conquering Salesforce Governor Limits and SOQL Queries

As we noted in our hands-on guide to building a Salesforce API integration, Salesforce is an entirely different beast compared to HubSpot. It is essentially a massive relational database exposed over the web. Salesforce integrations frequently fail in production not because of load, but because they violate strict governor limits.

The Limits That Actually Bite

The base daily quota varies by edition: Enterprise Edition starts at 100,000 calls per day plus 1,000 per user license. That sounds like a lot until you remember every middleware tool in the customer's org draws from the same pool.

The Daily API Request Limit is a soft limit, and your org is allowed to exceed it temporarily. The 403 error with REQUEST_LIMIT_EXCEEDED is what you actually need to handle when the system protection limit kicks in.

Concurrency is the real trap. Salesforce limits the number of concurrent long-running (20+ second) requests to 25 on production orgs. If the limit is exceeded, any new concurrent requests will not be processed. Long-running SOQL queries during a backfill will block your customer's other integrations.

Building Safe SOQL Queries and Keyset Pagination

To fetch data efficiently from Salesforce, you must use the Salesforce Object Query Language (SOQL). REST API endpoints exist for individual records, but bulk extraction requires dynamic SOQL construction.

SELECT Id, FirstName, LastName, Email, AccountId 
FROM Contact 
WHERE LastModifiedDate > 2026-01-01T00:00:00Z 
ORDER BY Id ASC 
LIMIT 2000

Notice the ORDER BY Id ASC. This is critical. Salesforce's standard offset pagination (OFFSET 2000) maxes out at 2,000 records. If you need to sync 50,000 contacts, standard offset pagination will hard-fail. You must implement "keyset pagination" - tracking the highest Id from the previous batch and using it in the WHERE clause of the next query (WHERE Id > 'Last_Seen_Id').

Never construct SOQL by string concatenation against user input. For dynamic filtering across customers with different field sets, build the query off the org's actual describe metadata, not your assumption of what fields exist:

async function buildContactQuery(sf: Connection, filters: Filter[]) {
  const describe = await sf.sobject('Contact').describe()
  const validFields = new Set(describe.fields.map(f => f.name))
  const where = filters
    .filter(f => validFields.has(f.field))
    .map(f => `${f.field} ${f.op} ${escape(f.value)}`)
    .join(' AND ')
  return `SELECT Id, FirstName, LastName, Email FROM Contact WHERE ${where} LIMIT 200`
}

Utilizing Bulk API 2.0 for Backfills

For any operation involving more than a few thousand records, you must abandon the standard REST API and use Salesforce Bulk API 2.0. This is an asynchronous API designed for large data volumes that allows enormous throughput: up to 150 million records per rolling 24 hours. Each batch can contain up to 10,000 records, and each 10k batch counts as only 1 call against the daily limit.

1. Create a Job: Send a POST request to define the object (e.g., Contact) and operation (e.g., query, insert, upsert).
2. Upload Data: If inserting, upload CSV data to the job endpoint.
3. Close the Job: Signal to Salesforce that the upload is complete.
4. Poll for Status: Continuously poll the job status endpoint until it reads JobComplete. Do not poll aggressively - check status every 30-60 seconds.
5. Retrieve Results: Fetch the successful records and the error logs.

This architecture requires your application to maintain durable state. You cannot hold an HTTP connection open waiting for a Bulk API job to finish. You must persist the Job ID to your database and use a background worker to poll for completion.

Recipe 3: Handling Salesforce Custom Objects and HubSpot Custom Properties

This is where most unified data models break. No two enterprise Salesforce instances are identical. If your integration assumes that a Contact object only has FirstName, LastName, and Email, it will break immediately upon deployment to a mid-market or enterprise customer.

The Salesforce __c Suffix Pattern

In Salesforce, when a customer creates a custom field for "Lead Source Detail", the API exposes it as Lead_Source_Detail__c. Every custom field and custom object in Salesforce ends in __c.

Your integration must first query the Salesforce metadata API to discover available fields for the authenticated tenant:

GET /services/data/v59.0/sobjects/Contact/describe

This endpoint returns a massive JSON payload detailing every field. You must filter the field list where custom === true (or match /__c$/), cache this schema on a TTL (24 hours is fine), and emit those fields as an open-ended map. Read our deep dive on how to handle custom fields and custom objects in Salesforce via API for specific UI mapping patterns.

HubSpot Dynamic Properties

HubSpot handles custom data differently. Instead of appending suffixes, HubSpot nests custom data within a properties object.

{
  "properties": {
    "firstname": "John",
    "custom_industry_vertical": "SaaS",
    "internal_scoring_metric": "85"
  }
}

Crucially, HubSpot will not return custom properties unless you ask for them by name in the GET request. That single gotcha causes more "missing data" support tickets than anything else. You must discover them via /crm/v3/properties/contacts and explicitly request them:

const defaultProps = new Set([
  'firstname', 'lastname', 'email', 'phone',
  'jobtitle', 'address', 'city', 'state', 'zip', 'country'
])
 
const customProps = allProperties
  .filter(p => !defaultProps.has(p.name) && !p.hubspotDefined)
  .map(p => p.name)
 
// Include them explicitly in the GET call
const url = `/crm/v3/objects/contacts?properties=${[...defaultProps, ...customProps].join(',')}`

Recipe 4: Webhook Subscription and HMAC Verification for Inbound Sync

The inbound leg of a bidirectional sync starts with CRM webhooks. When a contact, deal, or company changes in HubSpot, a webhook pushes the event to your endpoint. Getting this right requires correct subscription setup, signature verification, and idempotent processing.

Setting Up HubSpot Webhook Subscriptions

HubSpot supports webhook subscriptions through the Webhooks API (for developer apps) and through workflow actions (for Operations Hub / Data Hub). For a programmatic bidirectional sync, you want the Webhooks API approach.

Key setup steps:

1. Register your app in the HubSpot developer portal with the required scopes (e.g., crm.objects.contacts.read for contact events).
2. Configure a target URL - this must be an HTTPS endpoint that responds within 5 seconds.
3. Create subscriptions for the event types you need: contact.creation, contact.propertyChange, contact.deletion, deal.creation, deal.propertyChange, deal.deletion, etc.
4. Activate subscriptions - they start in an inactive state and must be explicitly activated.

HubSpot batches webhook events: a single POST to your endpoint can contain up to 100 events. Each event in the batch may relate to a different object or subscription type. Your handler must iterate through the array and process each event individually.

HubSpot v3 Signature Verification Checklist

Every inbound webhook from HubSpot includes an X-HubSpot-Signature-v3 header and an X-HubSpot-Request-Timestamp header. You must verify both before processing any payload.

The verification steps:

1. Check the timestamp. Reject the request if X-HubSpot-Request-Timestamp is more than 5 minutes old. HubSpot timestamps are in milliseconds, not seconds - this is a common source of bugs.
2. Build the signature input. Concatenate (UTF-8): requestMethod + requestUri + requestBody + timestamp. The request URI must match exactly what HubSpot sent - watch for reverse proxies or load balancers that rewrite URLs.
3. Compute the HMAC. Use HMAC SHA-256 with your app's client secret (not an API key or access token) as the key.
4. Base64-encode the HMAC result.
5. Compare the computed value to the X-HubSpot-Signature-v3 header using constant-time comparison to prevent timing attacks.
6. Verify the raw body. The HMAC is computed on the exact bytes HubSpot sends. If middleware parses or reformats the body before your verification code runs, the signature will not match.

import { createHmac, timingSafeEqual } from 'crypto'
 
function verifyHubSpotV3(
  method: string,
  uri: string,
  rawBody: string,
  timestamp: string,
  signature: string,
  clientSecret: string
): boolean {
  // Reject stale requests (timestamp is in milliseconds)
  const age = Date.now() - parseInt(timestamp, 10)
  if (age > 5 * 60 * 1000) return false
 
  const input = `${method}${uri}${rawBody}${timestamp}`
  const computed = createHmac('sha256', clientSecret)
    .update(input, 'utf8')
    .digest('base64')
 
  return timingSafeEqual(Buffer.from(computed), Buffer.from(signature))
}

Idempotent Event Processing

HubSpot does not guarantee exactly-once delivery. Webhooks are retried up to 10 times over 24 hours if your endpoint returns a non-2xx status. Your handler must be idempotent.

The simplest approach: store processed eventId values in a fast lookup (a database table with a unique index, or a cache with a 48-hour TTL). On each inbound event, check if the eventId has already been processed. If yes, respond 200 and skip. If no, process the event and record the eventId.

For propertyChange events specifically, compare the incoming propertyValue against your current record state rather than blindly overwriting. HubSpot does not guarantee event ordering, so a stale event could arrive after a newer one. Use the occurredAt timestamp within each event to determine the actual sequence.

Recipe 5: Deletion and Tombstone Strategies

Deletes are the most dangerous operation in a bidirectional sync. A missed deletion leaves ghost records. An overly aggressive deletion propagation can wipe data a user only meant to archive in one system.

CRM Deletion Events

HubSpot fires *.deletion webhook events (e.g., contact.deletion, deal.deletion) when a record is permanently deleted. The event payload contains the object ID but not the full record - the data is already gone by the time you receive the event.

Salesforce can notify you of deletions via outbound messages or by querying the getDeleted() endpoint on each object, which returns IDs deleted within a given time window.

Three strategies for handling deletions:

The recommended default is soft delete with tombstone. When you receive a contact.deletion event:

1. Look up the record in your database by the CRM ID.
2. If found, set deleted_at = now() and deleted_source = 'hubspot' (or 'salesforce').
3. Stop including this record in outbound sync runs.
4. Do not push a delete back to the CRM - the record is already gone there.

For the reverse direction (a user deletes a record in your app), you have a choice: propagate the delete to HubSpot via DELETE /crm/v3/objects/contacts/{id}, or simply stop syncing it. Propagating deletes is permanent and irrecoverable in HubSpot - there is no recycle bin API. Make this a configurable behavior per customer, not a hardcoded default.

Handling Bulk Deletes and GDPR

If a customer triggers a bulk deletion in HubSpot (e.g., a list cleanup), you may receive hundreds of contact.deletion events in a single batch webhook. Your handler needs to process these without overwhelming your database with individual DELETE queries. Batch your tombstone updates.

For GDPR data subject access requests, HubSpot provides a gdpr-delete endpoint. If you receive a deletion through that path, you must ensure full erasure in your system as well - tombstoning is not sufficient. Track the deletion source to distinguish compliance-mandated erasures from normal CRM housekeeping.

Recipe 6: Field Ownership and Conflict Resolution

In a bidirectional sync, two systems can update the same field on the same record at the same time. Without explicit ownership rules, you get last-write-wins chaos where changes silently overwrite each other.

The Decision Matrix

Before writing a single line of sync code, fill out this matrix for every field you plan to sync:

Three common conflict resolution strategies:

1. Field-level ownership (recommended). Each field has exactly one authoritative source. plan_tier always comes from your app. lifecycle_stage always comes from HubSpot. No conflicts possible because writes only flow in one direction per field.
2. Timestamp-based (last write wins). Compare updatedAt from both sides and take the newer value. Simple to implement but can lose data if clocks drift or events arrive out of order.
3. Prefer-source-unless-blank. Use one system's value by default, but accept the other system's value if the preferred source is empty. Good for initial data enrichment, bad for ongoing sync.

Field-level ownership eliminates the most common class of sync bugs. It is more restrictive than last-write-wins, but it means you never have to debug "who overwrote this field and when" at 2am.

Implementing Field-Level Ownership in Code

type SyncDirection = 'hubspot_to_app' | 'app_to_hubspot' | 'bidirectional'
 
const fieldOwnership: Record<string, {
  direction: SyncDirection
  conflictRule: string
}> = {
  email:            { direction: 'hubspot_to_app', conflictRule: 'always_hubspot' },
  lifecycle_stage:  { direction: 'hubspot_to_app', conflictRule: 'always_hubspot' },
  plan_tier:        { direction: 'app_to_hubspot', conflictRule: 'always_app' },
  mrr:              { direction: 'app_to_hubspot', conflictRule: 'always_app' },
  company_name:     { direction: 'bidirectional',  conflictRule: 'latest_updated_at' },
}
 
function shouldApplyInboundChange(
  field: string, inboundValue: unknown, currentValue: unknown
): boolean {
  const rule = fieldOwnership[field]
  if (!rule) return false // unknown fields are not synced
  if (rule.direction === 'app_to_hubspot') return false
  if (rule.direction === 'hubspot_to_app') return true
  // bidirectional: compare timestamps or apply conflict rule
  return true
}

Store this configuration in a database or config file, not in code. When a customer asks you to change which system owns a field (and they will), you want to flip a config value, not ship a code change.

Monitoring, Metrics, and Alerting Playbook

A bidirectional sync that runs without monitoring is a ticking bomb. You will not notice dropped records, silent 429s, or accumulating drift until a customer reports it - usually weeks later.

Metrics to Track

Reconciliation Job

Run a scheduled reconciliation job every 4-8 hours. The job:

1. Fetches records updated since the last reconciliation window from both the CRM and your app.
2. Compares field values for records that exist in both systems.
3. Logs discrepancies and optionally auto-corrects them based on your field ownership rules.
4. Reports sync.reconciliation.drift_count as a metric.

This is your safety net. If the real-time webhook path drops an event, the reconciliation sweep catches it within hours instead of weeks. If drift count trends upward over time, something in your real-time path is broken and needs investigation.

Building ERP Integrations Without Storing Customer Data

The recipes above assume you can persist synced records in your own database. For CRM data, that is usually fine. For ERP data - NetSuite, SAP S/4HANA, Oracle Fusion, Microsoft Dynamics 365 Finance & Operations - it often is not. Financial ledgers, employee PII, tax records, and payroll data trigger compliance obligations (SOX, GDPR, HIPAA for healthcare ERPs, PCI for anything touching card data) that make even short-lived storage a liability.

The answer is a zero data retention (ZDR) architecture: process customer data in memory, pass it through to its destination, and let it disappear when the request completes. Nothing on disk, nothing in queues, nothing in logs. Same design pattern applies whether you are building integrations from scratch or extending the CRM sync patterns above to also cover ERPs. This section is the hands-on playbook.

Overview: Zero Data Retention Recap

A zero data retention integration guarantees that no customer business data touches durable storage owned by the integration layer. Only three categories of state are persisted:

• Connection metadata: integrated account ID, tenant identifier, integration type, status.
• Encrypted credentials: OAuth tokens or API keys, encrypted at rest with a key you control (AES-GCM is the standard choice).
• Operational metadata: request timestamps, HTTP status codes, latencies, error categories - never payloads.

Everything else - contact records, journal entries, employee records, invoice line items, purchase orders - flows through the process without ever being written to a database, log file, or queue payload.

This is a natural fit for ERP integrations because:

1. Customer legal teams frequently block deployments that store financial or HR data in third-party systems.
2. Data residency requirements (EU, India, Australia, UAE) become trivial when no data is stored.
3. Breach blast radius is bounded to whatever is in memory at the moment of compromise, not years of historical records.
4. "Privacy by design" and "data minimization" clauses in GDPR Article 25 and CCPA are satisfied by architecture, not policy.

The trade-off: you lose the ability to add value in-flight through caching, denormalization, or cross-tenant analytics. You are a stateless proxy. For ERP data, that is exactly what enterprise buyers want.

In-Memory Processing Patterns: Streaming vs Full-Payload

There are two ways to move data through memory without persisting it. Choose based on payload size and downstream consumer capabilities.

flowchart LR
    subgraph FullPayload ["Full-Payload Buffering"]
        direction TB
        A1["Upstream API<br>(SAP / NetSuite)"] --> B1[Buffer entire response<br>in memory]
        B1 --> C1[Transform + validate]
        C1 --> D1[Push to caller<br>as single response]
    end
    subgraph Streaming ["Streaming Pipeline"]
        direction TB
        A2["Upstream API<br>(SAP / NetSuite)"] --> B2[Read response<br>chunk by chunk]
        B2 --> C2[Transform on the fly]
        C2 --> D2[Write to caller<br>as stream]
        D2 --> B2
    end

Full-payload buffering is simpler. You fetch the entire upstream response into memory, transform it, then hand the transformed result to your caller. Use it when:

• The payload is bounded (a single invoice, one employee record, one journal entry).
• The downstream requires the complete response before it can act (e.g., signing an entire document).
• You need to sort, group, or aggregate across the whole response.

Streaming is the safer default for ERP. Bulk exports from NetSuite SuiteQL or SAP OData can return hundreds of thousands of records. Buffering that in memory will OOM your process. In a streaming pipeline you read the upstream response chunk by chunk (JSON lines, CSV rows, or paginated batches), transform each chunk, and pipe it directly to the caller before the next chunk arrives.

// Streaming pipeline: no full payload ever lives in memory
async function streamSuiteQL(
  upstreamUrl: string,
  token: string,
  transform: (row: Record<string, unknown>) => Record<string, unknown>,
  writer: WritableStreamDefaultWriter<Uint8Array>
) {
  const res = await fetch(upstreamUrl, {
    headers: { authorization: `Bearer ${token}` },
  })
  if (!res.body) throw new Error('no body')
 
  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader()
  const encoder = new TextEncoder()
  let buffer = ''
 
  while (true) {
    const { value, done } = await reader.read()
    if (done) break
    buffer += value
 
    // Emit complete JSON lines as they arrive
    let newlineIdx: number
    while ((newlineIdx = buffer.indexOf('\n')) >= 0) {
      const line = buffer.slice(0, newlineIdx)
      buffer = buffer.slice(newlineIdx + 1)
      if (!line.trim()) continue
      const row = JSON.parse(line)
      const mapped = transform(row)
      await writer.write(encoder.encode(JSON.stringify(mapped) + '\n'))
      // row + mapped go out of scope on the next iteration - GC-eligible
    }
  }
  await writer.close()
}

The key discipline: do not accumulate. No const allRows = []; allRows.push(row). Every intermediate value must be eligible for garbage collection as soon as its transformed output is emitted. If you find yourself writing .reduce() or Array.from() on a stream, stop and ask whether the aggregation can happen on the caller's side instead.

Safe Memory and Timeout Limits, and Buffering Strategies

Zero data retention runtimes are usually stateless workers (serverless functions or short-lived containers) with strict caps. Design around these limits explicitly:

Concrete rules:

• Cap buffer sizes explicitly. If you must buffer, allocate a fixed-size buffer and reject anything larger with a clear error. Never use unbounded arrays.
• Set per-request memory watermarks. Emit a metric when memory usage crosses 70% of the ceiling. That is your signal to switch a fetcher to streaming.
• Timeout upstream calls aggressively. If your worker has a 30-second budget, give the upstream call 20 seconds and reserve 10 for downstream write plus overhead. A hung ERP query should not eat your entire runtime.
• Prefer chunked responses. Ask ERP APIs for paginated results with small page sizes (500-2000 rows) rather than a single mega-response. NetSuite SuiteQL supports LIMIT / OFFSET; SAP OData supports $top / $skip; Dynamics 365 supports $top with server-side paging.
• Backpressure the writer. When streaming to a caller, respect their consumption rate. If they slow down, you slow down. pipeTo in the Web Streams API handles this automatically; write() returns a promise you must await.

A useful pattern is the bounded buffer: allocate a ring buffer of, say, 5000 records. Read into the buffer, transform, flush to the writer, repeat. Memory usage stays flat regardless of total payload size.

async function boundedStream<T, U>(
  source: AsyncIterable<T>,
  transform: (row: T) => U,
  sink: (batch: U[]) => Promise<void>,
  bufferSize = 5000,
) {
  let buffer: U[] = []
  for await (const row of source) {
    buffer.push(transform(row))
    if (buffer.length >= bufferSize) {
      await sink(buffer)
      buffer = [] // old buffer becomes GC-eligible
    }
  }
  if (buffer.length) await sink(buffer)
}

Retry, Idempotency, and Reconciliation Without Persistence

The classic retry pattern says: on failure, persist the request, retry with backoff, and give up after N attempts. Zero data retention breaks that model - you cannot persist the request because the request contains customer data.

The workable pattern:

1. Retries stay in-process. For transient failures (5xx, connection reset, 429), retry with exponential backoff and jitter within the current worker invocation. Do not enqueue the payload for later.
2. Failures propagate up. If in-process retries exhaust, return an error to the caller with a correlation ID. The caller (who owns the data) can decide whether to retry with the original payload.
3. Idempotency keys are caller-supplied. The caller passes an idempotency key (typically a UUID derived from their own record ID plus a version). Your integration layer forwards it to the ERP if the ERP supports idempotency headers - SAP S/4HANA Cloud supports Idempotency-Key on many endpoints; NetSuite requires an application-level pattern using external IDs.
4. Reconciliation runs on the caller side. Because you do not store payloads, you cannot reconcile centrally. Instead, the caller compares its own database to the ERP directly via read-through calls, using your integration layer as a stateless proxy.

// In-process retry with jitter - no payload persistence
async function writeWithRetry<T>(
  fn: () => Promise<T>,
  opts = { maxAttempts: 4, baseMs: 500, capMs: 8000 },
): Promise<T> {
  let lastErr: unknown
  for (let attempt = 0; attempt < opts.maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (err) {
      lastErr = err
      if (!isRetryable(err) || attempt === opts.maxAttempts - 1) throw err
      const backoff = Math.min(opts.capMs, opts.baseMs * 2 ** attempt)
      const jitter = Math.random() * backoff * 0.3
      await new Promise(r => setTimeout(r, backoff + jitter))
    }
  }
  throw lastErr
}
 
function isRetryable(err: unknown): boolean {
  if (!(err instanceof Response)) return true // network error
  return err.status >= 500 || err.status === 429
}

For at-least-once semantics without persistence, rely on ERP-side idempotency. SAP's Idempotency-Key header lets you retry a POST safely - the server deduplicates based on the key. If the ERP does not support idempotency headers, use conditional writes (If-Match on ETags) or upsert semantics keyed by a business identifier (external ID, natural key). NetSuite's REST API supports externalId for exactly this reason.

Reconciliation without a local mirror: the caller runs periodic diffs by fetching filtered windows (lastModifiedDate > checkpoint) from the ERP through your proxy and comparing to their own records. Your job is to make sure the proxy is fast and predictable. You never see the reconciliation output.

Token Refresh and Secure Ephemeral Credential Handling

OAuth tokens are the one piece of state a zero-retention integration must persist. Access tokens expire (typically 30-60 minutes), so refresh tokens have to live on disk. The goal is to keep the credential footprint minimal and the plaintext lifetime as short as possible.

The pattern:

1. Encrypt at rest with AES-GCM. Refresh tokens and access tokens live encrypted in a single row keyed by integrated account ID. The encryption key is managed by a KMS, not a config file secret. Rotate the KMS key at least annually.
2. Decrypt only in memory. Plaintext tokens exist only for the duration of a single upstream request. They are never logged, never included in error messages, never returned in API responses.
3. Refresh proactively. Schedule a refresh 60-180 seconds before the access token expires, randomized within the window to spread load across accounts. This avoids the "everyone refreshes at :00" thundering herd.
4. Serialize concurrent refreshes. If two workers try to refresh the same token simultaneously, the second must wait for the first. Otherwise you burn refresh tokens - many providers rotate them on every refresh, and the older refresh token becomes invalid the moment the new one is issued.

sequenceDiagram
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant Mutex as Per-Account Mutex
    participant Store as Encrypted Credential Store
    participant IdP as ERP Identity Provider

    W1->>Mutex: acquire(account_id)
    Mutex-->>W1: locked
    W2->>Mutex: acquire(account_id)
    Note over W2,Mutex: Blocks - awaits<br>same in-flight refresh
    W1->>Store: read encrypted token
    Store-->>W1: ciphertext
    W1->>W1: decrypt in memory
    W1->>IdP: POST /token (grant=refresh_token)
    IdP-->>W1: new access + refresh
    W1->>Store: write new encrypted token
    W1->>Mutex: release
    Mutex-->>W2: unblocked with W1's result
    Note over W2: Uses fresh token,<br>no duplicate refresh

Per-account serialization is essential. A single mutex keyed by the integrated account ID ensures only one refresh runs per account at a time. Concurrent callers wait on the same in-flight refresh and receive its result rather than issuing duplicate token exchanges. Two different accounts refresh in parallel, so the mutex does not become a global bottleneck.

Here is a minimal proactive refresh flow in code:

async function getAccessToken(accountId: string): Promise<string> {
  const account = await loadAccount(accountId) // metadata + ciphertext only
  const token = decryptToken(account.encryptedToken)
 
  // Refresh if within 30 seconds of expiry
  if (Date.now() + 30_000 >= token.expiresAt) {
    return await refreshMutex.run(accountId, async () => {
      // Re-read inside the lock - another worker may have refreshed already
      const fresh = decryptToken((await loadAccount(accountId)).encryptedToken)
      if (Date.now() + 30_000 < fresh.expiresAt) return fresh.accessToken
 
      const newToken = await exchangeRefreshToken(fresh.refreshToken)
      await saveEncryptedToken(accountId, newToken)
      scheduleProactiveRefresh(accountId, newToken.expiresAt)
      return newToken.accessToken
    })
  }
  return token.accessToken
}

Additional hardening:

• Mark accounts for reauth on non-retryable errors. HTTP 401 or invalid_grant from the identity provider means the refresh token is revoked. Update the account status, fire a webhook so your customer knows, and stop retrying - no amount of backoff will fix a revoked grant.
• Retry only on transient failures. Network errors and 5xx responses warrant a retry, ideally scheduled a few hours later, not immediately. 4xx responses do not.
• Route refreshes through fixed egress IPs when required. Enterprise ERPs often restrict token endpoints to IP allowlists. Route refresh requests through a stable egress path so the allowlist does not have to change every deployment.
• Never return tokens in API responses. Even to your own admin UI. Redact them at the serialization boundary.

Observability Without Persistence: Tracing and Logging Best Practices

Traditional observability practices leak data. Full request/response logging captures customer payloads. Distributed tracing spans often serialize payloads into span attributes. Debug dumps write entire response bodies to disk. A zero data retention integration must be observable while remaining data-free.

Guidelines:

• Log shape, not content. Log { resource: 'invoice', operation: 'read', row_count: 2431, latency_ms: 812, status: 200 }. Never log { invoice_body: { ... } }.
• Redact by allowlist, not by regex. Configure your tracer with an allowlist of span attributes. Anything not on the allowlist is dropped, not passed through. Regex-based PII scrubbing always misses something - a field renamed by a customer, an unexpected nested object, an unusual encoding.
• Hash identifiers for correlation. When you need to trace a specific record through the pipeline, log a hash of its identifier, not the identifier itself. HMAC-SHA256 with a per-tenant salt gives you deterministic correlation without exposing values.
• Sample errors, not payloads. For a 500 response from the ERP, log the HTTP status, the response headers you know are safe, and a categorized error code. Do not include the response body.
• Structured error categories. Map upstream errors to a fixed vocabulary (upstream_timeout, rate_limited, auth_failed, validation_error, not_found). Categories are safe to store; raw error messages containing customer-specific detail are not.
• Bound retention on operational logs. Even payload-free logs should have a TTL. 30-90 days is typical. Anything longer needs a documented business reason.

// Safe structured logging - never touches payload data
function logRequest(ctx: {
  accountId: string
  resource: string
  operation: 'read' | 'write' | 'delete'
  rowCount?: number
  latencyMs: number
  status: number
  category?: string
}) {
  console.log(JSON.stringify({
    ts: new Date().toISOString(),
    account_hash: hmacSha256(ctx.accountId, PER_TENANT_SALT),
    resource: ctx.resource,
    op: ctx.operation,
    rows: ctx.rowCount,
    latency_ms: ctx.latencyMs,
    status: ctx.status,
    category: ctx.category,
  }))
}

For deeper debugging, offer customers an opt-in temporary payload capture with a short TTL (e.g., 1 hour), gated by explicit consent per debugging session, ideally with a customer-side approval workflow. Never make payload logging a default state that requires opt-out.

Anti-Patterns That Create Hidden Persistence

The failure mode for zero data retention is rarely intentional storage. It is accidental storage. Every one of these has shipped in a "no data storage" integration and had to be ripped out:

• Queue payloads as durable storage. If you enqueue a job with the customer payload embedded in the message, that payload lives in the queue's storage layer for hours or days. Queues are databases with a delivery model bolted on. Use them for control signals ("record X changed, go re-fetch it") but never for payloads.
• Trace attributes containing bodies. Setting span.setAttribute('response_body', JSON.stringify(payload)) sends that payload to your APM vendor's servers, where it is retained per their policy. Assume every attribute you set is stored somewhere for at least 30 days.
• Verbose error logs. catch (e) { console.error('Failed:', e, 'payload was:', payload) } writes the entire payload to your log aggregator. The exception category alone is usually sufficient.
• Debug dumps to object storage. "Just write the failing payload to blob storage so we can debug it Monday" is the single most common way zero-retention promises get broken. If you must capture failure payloads, capture only synthetic reproductions, not real customer data.
• Retry queues with payloads. Dead letter queues that store the original payload for later inspection are a persistence layer with a friendly name. Use control-plane retries (re-fetch and re-transform) rather than payload replay.
• HTTP access logs with query strings. Your load balancer or reverse proxy probably logs full request URIs by default. If your API accepts filters as query parameters (?email=jane@customer.com), those are now in your access logs. Move filters to POST bodies or scrub URIs at ingress.
• APM auto-instrumentation of database drivers. Some APM agents capture SQL statements with parameter values inlined. If your integration ever writes customer data to a working table (even briefly), the SQL trace will capture it. Configure the agent to redact parameter values.
• Backup and snapshot systems. If you have an encrypted credential store, its backups are also encrypted - but the backups can persist for years past your stated retention window. Set backup TTLs deliberately, not just relying on defaults.
• CDN or WAF request buffering. Some edge providers buffer request bodies for inspection. Confirm with your provider whether raw bodies are stored for any period, and for how long.
• Container filesystems. If a process writes to /tmp or a mounted volume for any reason, that data can survive process restarts on some platforms. Treat every filesystem write as a persistence event.

Audit for these systematically. Run a data-flow diagram for every code path that touches a customer payload and ask: "Where does this byte sequence exist five minutes from now?" If the answer is anything other than "in memory of an already-terminated process," you have accidental persistence.

Operational Runbook: Incident Triage When Upstream APIs Fail

Zero data retention integrations have a specific failure mode: when the upstream fails, you cannot fall back to a local cache because there is no local cache. Runbooks matter more, not less.

Symptom: Upstream 5xx spike (>10% error rate over 5 minutes)

1. Check the upstream provider's status page (NetSuite Status, SAP Trust Center, Dynamics Service Health).
2. If the provider is degraded, enable a customer-facing banner acknowledging the issue and disable non-critical outbound writes.
3. Confirm in-process retry logic is active. Retries should already be absorbing transient errors; if 5xx is escaping to callers, retries are broken.
4. Do not enqueue failed requests for later replay. Return errors to callers with the correlation ID so they can decide whether to retry.

Symptom: OAuth refresh failure rate > 1%

1. Check whether the failures are concentrated on a single tenant (revoked grant) or spread across tenants (identity provider outage).
2. For single-tenant failures: mark the account for reauth, fire the customer webhook, stop retrying.
3. For broad failures: confirm the identity provider status. If the IdP is down, active access tokens continue working until their natural expiry - do not force refreshes.
4. Alert if the count of accounts in needs_reauth grows by more than 5% in an hour.

Symptom: Memory usage crosses 80% of ceiling

1. Identify the fetcher responsible via per-integration memory metrics.
2. Confirm the fetcher is streaming, not buffering. If it is buffering, switch to streaming and re-deploy.
3. Reduce per-request page size for the affected resource temporarily.
4. If the workload is truly larger than memory allows, refuse the request with a clear error rather than OOMing silently.

Symptom: Rate limit exhaustion (429 spike)

1. Identify the affected tenant and integration.
2. Confirm the rate limiter is respecting Retry-After headers from the upstream.
3. If a single tenant is bursting, apply per-tenant throttling ahead of the upstream call.
4. Escalate to the customer if they need to purchase additional API capacity from the ERP vendor.

Symptom: HMAC verification failures on inbound webhooks

1. Confirm the shared secret hasn't rotated on the upstream side without your rotation.
2. Check whether a proxy or WAF is mutating the request body (JSON reformatting is the usual culprit).
3. Never disable verification to "work around" the issue. A failing verification is a signal, not a nuisance.

Recipe 7: How to Integrate Multiple Calendar Services (Google & Microsoft)

Calendar integrations follow the same architectural spine as CRM integrations - OAuth, webhooks, incremental sync, reconciliation - but the specifics diverge enough that most teams end up rebuilding the entire stack per provider. Google Calendar and Microsoft Graph dominate the enterprise market, and if you support one you almost always need the other. This recipe covers how to integrate multiple calendar services in a single unified pipeline: OAuth acquisition, webhook subscription and renewal, incremental sync via provider-specific tokens, and recovery when sync state is lost.

The short version: if you are asking "how do I integrate calendars from multiple services without a per-provider codebase?", you need three shared abstractions - a provider-agnostic OAuth store with per-account refresh coordination, a unified webhook receiver that handles Google's watch channels and Microsoft's subscription validation in one handler, and a sync state manager that treats syncToken (Google) and deltaLink (Microsoft) as the same concept behind an interface.

Quickstart: End-to-End Flow (OAuth → Webhook → Unified Endpoint)

The minimum viable calendar integration has three moving parts:

1. OAuth acquisition - user authorizes your app; you receive an access token, a refresh token, and calendar scopes.
2. Webhook subscription - you register a push channel with the provider so it notifies your endpoint on event changes.
3. Unified read endpoint - your callers query one API for events, availability, and free/busy; your service fetches from whichever provider owns the account.

sequenceDiagram
    participant User
    participant App as Your App
    participant Provider as "Google / Microsoft"
    participant Handler as Webhook Handler
    User->>App: Click "Connect Calendar"
    App->>Provider: OAuth authorize (scopes)
    Provider-->>App: authorization code
    App->>Provider: exchange code for tokens
    Provider-->>App: access + refresh token
    App->>Provider: POST watch / subscription
    Provider-->>App: channel_id + expiration
    App->>App: store watch, schedule renewal
    Provider->>Handler: change notification
    Handler->>Provider: GET events?syncToken=...
    Provider-->>Handler: incremental changes
    Handler->>App: normalized event payload

Scopes differ per provider:

• Google Calendar: https://www.googleapis.com/auth/calendar.events (read/write events) or https://www.googleapis.com/auth/calendar.readonly. Include access_type=offline and prompt=consent on the authorize URL to guarantee a refresh token.
• Microsoft Graph: Calendars.ReadWrite (delegated or application) and offline_access (required for a refresh token).

The Microsoft offline_access scope is the single most common mistake when adding Outlook support. Without it, you get an access token but no refresh token, and your integration silently breaks after the first token expires.

OAuth: Refresh and Concurrent-Worker Coordination

Both Google and Microsoft issue access tokens with lifetimes around one hour. Two realities force careful design:

• Refresh tokens can rotate. Google rotates on some flows; Microsoft rotates when tenants configure a rotation policy. Using an outdated refresh token after rotation returns invalid_grant and the account is dead until reauth.
• Concurrent workers refresh simultaneously. Two sync jobs and an inbound webhook can all hit the same expired token at once. Without coordination, three parallel refresh calls fire, only one wins, and the others burn tokens.

The fix is a per-account mutex around the refresh operation. First worker acquires the lock, refreshes, updates the store. Subsequent workers wait on the same in-flight promise and receive the fresh token without issuing a duplicate refresh.

type CalendarProvider = 'google' | 'microsoft'
const refreshLocks = new Map<string, Promise<string>>()
 
async function getCalendarAccessToken(
  accountId: string,
  provider: CalendarProvider,
): Promise<string> {
  const account = await loadAccount(accountId)
  const token = decryptToken(account.encryptedToken)
 
  // 30-second safety buffer before expiry
  if (Date.now() + 30_000 < token.expiresAt) return token.accessToken
 
  // Coalesce concurrent refreshes per account
  const existing = refreshLocks.get(accountId)
  if (existing) return existing
 
  const refresh = (async () => {
    try {
      const fresh = await exchangeRefreshToken(provider, token.refreshToken)
      // Providers that rotate refresh tokens return a new one; persist it
      await saveEncryptedToken(accountId, {
        accessToken: fresh.access_token,
        refreshToken: fresh.refresh_token ?? token.refreshToken,
        expiresAt: Date.now() + fresh.expires_in * 1000,
      })
      return fresh.access_token
    } catch (err) {
      if (isInvalidGrant(err)) await markAccountForReauth(accountId, err)
      throw err
    } finally {
      refreshLocks.delete(accountId)
    }
  })()
 
  refreshLocks.set(accountId, refresh)
  return refresh
}
 
async function exchangeRefreshToken(
  provider: CalendarProvider,
  refreshToken: string,
) {
  const endpoints = {
    google: 'https://oauth2.googleapis.com/token',
    microsoft: 'https://login.microsoftonline.com/common/oauth2/v2.0/token',
  }
  const body = new URLSearchParams({
    grant_type: 'refresh_token',
    refresh_token: refreshToken,
    client_id: process.env[`${provider.toUpperCase()}_CLIENT_ID`]!,
    client_secret: process.env[`${provider.toUpperCase()}_CLIENT_SECRET`]!,
  })
  const res = await fetch(endpoints[provider], {
    method: 'POST',
    headers: { 'content-type': 'application/x-www-form-urlencoded' },
    body,
  })
  if (!res.ok) throw new Error(`refresh failed: ${res.status}`)
  return res.json() as Promise<{
    access_token: string
    refresh_token?: string
    expires_in: number
  }>
}

In a multi-worker deployment across processes or machines, an in-memory Map is not enough. Back the lock with an external coordination primitive - a Redis lock with a short TTL, a database row lock (SELECT ... FOR UPDATE), or a durable single-writer key per account. The pattern is the same: one refresh in flight per account, everyone else waits on that result.

Schedule a proactive refresh 60-180 seconds before expiry with per-account jitter to avoid a thundering herd at the top of every hour.

Webhook Handling: Google Watch Renewal and Microsoft Validation

The two providers have completely different webhook models. Your handler must accommodate both.

Google Calendar (Watch Channels)

Google uses a "watch" channel that expires. Maximum lifetime is around 7 days for events.watch; you must renew before expiry or you stop receiving notifications. Google does not send a validation request when creating a channel - the response to your POST returns the channel details directly.

// Create a Google watch channel for a calendar
async function createGoogleWatch(
  calendarId: string,
  accessToken: string,
  webhookUrl: string,
): Promise<{ id: string; resourceId: string; expiration: number }> {
  const channelId = crypto.randomUUID()
  const res = await fetch(
    `https://www.googleapis.com/calendar/v3/calendars/${encodeURIComponent(calendarId)}/events/watch`,
    {
      method: 'POST',
      headers: {
        authorization: `Bearer ${accessToken}`,
        'content-type': 'application/json',
      },
      body: JSON.stringify({
        id: channelId,
        type: 'web_hook',
        address: webhookUrl,
        token: signChannelToken(channelId), // verify on inbound
      }),
    },
  )
  if (!res.ok) throw new Error(`watch failed: ${await res.text()}`)
  const data = await res.json()
  return {
    id: data.id,
    resourceId: data.resourceId,
    expiration: parseInt(data.expiration, 10), // ms since epoch
  }
}
 
// Inbound Google notification - body is empty; headers carry metadata
export async function handleGoogleNotification(req: Request) {
  const channelId = req.headers.get('x-goog-channel-id')
  const resourceState = req.headers.get('x-goog-resource-state')
  const channelToken = req.headers.get('x-goog-channel-token')
 
  if (!channelId || !verifyChannelToken(channelId, channelToken)) {
    return new Response('invalid channel', { status: 401 })
  }
  if (resourceState === 'sync') {
    // Initial sync ping when the channel is created - safe to ignore
    return new Response(null, { status: 200 })
  }
  // For any change notification, do an incremental fetch using stored syncToken
  await enqueueIncrementalFetch(channelId)
  return new Response(null, { status: 200 })
}

Renewal is your responsibility. Schedule a job that runs hourly to renew any channel expiring within 24 hours. Do not wait until the last minute - if renewal fails, you need buffer to retry or fall back to polling.

async function renewGoogleWatchesIfExpiring() {
  const soon = Date.now() + 24 * 60 * 60 * 1000
  const expiring = await db.watches.where('expiration').lt(soon).toArray()
  for (const w of expiring) {
    try {
      // Google has no in-place renewal - stop the old channel, create a new one
      await stopGoogleWatch(w.id, w.resourceId, w.accessToken)
      const fresh = await createGoogleWatch(
        w.calendarId, w.accessToken, w.webhookUrl,
      )
      await db.watches.update(w.id, { ...fresh, previousId: w.id })
    } catch (err) {
      log.warn('watch renewal failed, falling back to poll', {
        accountId: w.accountId, err,
      })
      await markForPollingFallback(w.accountId)
    }
  }
}

Microsoft Graph (Subscriptions)

Microsoft uses a "subscription" resource. Maximum lifetime is roughly 4230 minutes (~3 days) for calendar events. Unlike Google, Microsoft supports in-place renewal via PATCH to the existing subscription - you do not have to recreate it. Microsoft also requires a validation handshake during creation: Graph sends a POST to your notificationUrl with a validationToken query parameter, and your endpoint must echo the token as text/plain within 10 seconds.

// Handle both the validation handshake and change notifications
export async function handleMicrosoftNotification(req: Request) {
  const url = new URL(req.url)
  const validationToken = url.searchParams.get('validationToken')
 
  // Validation handshake: echo the token as text/plain within 10s
  if (validationToken) {
    return new Response(validationToken, {
      status: 200,
      headers: { 'content-type': 'text/plain' },
    })
  }
 
  const body = await req.json() as {
    value: Array<{
      subscriptionId: string
      clientState: string
      resource: string
      changeType: 'created' | 'updated' | 'deleted'
    }>
  }
 
  for (const notification of body.value) {
    if (!verifyClientState(notification.subscriptionId, notification.clientState)) {
      continue // silently drop spoofed notifications
    }
    await enqueueIncrementalFetch(notification.subscriptionId)
  }
  return new Response(null, { status: 202 })
}
 
// Create a subscription for /me/events
async function createGraphSubscription(accessToken: string, webhookUrl: string) {
  const expirationDateTime = new Date(Date.now() + 4230 * 60 * 1000).toISOString()
  const res = await fetch('https://graph.microsoft.com/v1.0/subscriptions', {
    method: 'POST',
    headers: {
      authorization: `Bearer ${accessToken}`,
      'content-type': 'application/json',
    },
    body: JSON.stringify({
      changeType: 'created,updated,deleted',
      notificationUrl: webhookUrl,
      resource: '/me/events',
      expirationDateTime,
      clientState: randomClientState(), // verify on inbound
    }),
  })
  if (!res.ok) throw new Error(`subscription failed: ${await res.text()}`)
  return res.json()
}
 
// Renew before expiry - in-place PATCH
async function renewGraphSubscription(subscriptionId: string, accessToken: string) {
  const expirationDateTime = new Date(Date.now() + 4230 * 60 * 1000).toISOString()
  const res = await fetch(
    `https://graph.microsoft.com/v1.0/subscriptions/${subscriptionId}`,
    {
      method: 'PATCH',
      headers: {
        authorization: `Bearer ${accessToken}`,
        'content-type': 'application/json',
      },
      body: JSON.stringify({ expirationDateTime }),
    },
  )
  if (!res.ok) throw new Error(`renewal failed: ${await res.text()}`)
  return res.json()
}

Two critical Microsoft-specific gotchas:

1. The validation endpoint must be reachable before you POST the subscription. If the endpoint is not yet deployed or is behind auth, subscription creation fails with a validation error. Deploy the endpoint first, then create subscriptions.
2. clientState is your only spoofing defense. Microsoft does not sign notifications. Set a random clientState per subscription, store it, and reject notifications where the incoming clientState does not match.

Polling as fallback

Webhooks fail. Channels expire. Renewals fail. You need a polling fallback that keeps sync running when push notifications are down. A minimal polling loop calls events.list (Google) or /me/calendarView/delta (Microsoft) every 5-15 minutes for accounts marked in polling fallback state. Do not poll every account by default - polling every calendar every 5 minutes will exhaust your API quotas fast. Reserve polling for accounts where push is broken, and re-attempt push subscription creation on a slower cadence.

Sync State Management: syncToken vs deltaLink and Recovery Recipe

Both providers offer incremental sync, but the tokens have different names, different lifetimes, and different failure modes. Handle them behind a single interface.

The recovery recipe when sync state is lost:

type SyncState = { provider: CalendarProvider; token: string | null }
 
async function incrementalFetch(
  accountId: string,
  calendarId: string,
): Promise<{ events: CalendarEvent[]; nextToken: string }> {
  const state = await loadSyncState(accountId, calendarId)
  try {
    if (state.provider === 'google') {
      return await googleIncremental(accountId, calendarId, state.token)
    }
    return await microsoftIncremental(accountId, state.token)
  } catch (err) {
    if (isSyncStateGone(err)) {
      // 410 Gone - the token is unusable. Do a full resync.
      log.warn('sync token expired, performing full resync', {
        accountId, calendarId,
      })
      await clearSyncState(accountId, calendarId)
      return await fullResync(accountId, calendarId, state.provider)
    }
    throw err
  }
}
 
function isSyncStateGone(err: unknown): boolean {
  if (!(err instanceof HttpError)) return false
  if (err.status !== 410) return false
  // Google: 'fullSyncRequired' - Microsoft: 'syncStateNotFound'
  return true
}
 
async function fullResync(
  accountId: string,
  calendarId: string,
  provider: CalendarProvider,
): Promise<{ events: CalendarEvent[]; nextToken: string }> {
  const events: CalendarEvent[] = []
  let pageToken: string | undefined
  let nextToken = ''
 
  do {
    const page = provider === 'google'
      ? await googleListEvents(accountId, calendarId, {
          pageToken, timeMin: sixMonthsAgo(),
        })
      : await graphCalendarViewDelta(accountId, {
          pageToken, startDateTime: sixMonthsAgo(),
        })
    events.push(...page.events)
    pageToken = page.nextPageToken
    // Only the final page carries the next sync token
    if (page.nextSyncToken) nextToken = page.nextSyncToken
  } while (pageToken)
 
  await saveSyncState(accountId, calendarId, { provider, token: nextToken })
  await reconcileAgainstLocalState(accountId, calendarId, events)
  return { events, nextToken }
}

Three subtle rules that catch most teams:

• Only the last page carries the next sync token. If pagination stops early (a caller cancels, a timeout fires), you lose the token and must full-resync next time. Always drain to the end before committing state.
• Bound your window on full resync. Do not fetch "all events ever" - most calendars have years of noise. Fetch the last 6-12 months plus all future events. Anything older is rarely useful and can double your API cost.
• Reconcile after a full resync, do not overwrite. The events returned by a full resync are the current state within your window; anything you had locally that is not in the new set could be genuinely deleted or simply outside the window. Diff before applying, and mark records outside the window as "unknown" rather than deleted.

For Google specifically, the first call in a fresh sync uses timeMin (and any other filters). Google will only return nextSyncToken on the final page of that initial walk; if you stop paginating early you get nothing usable. On subsequent calls you pass only syncToken - filters like timeMin cannot be combined with an active sync token.

Troubleshooting Checklist and Monitoring/Alerting

Calendar integrations fail in predictable ways. This checklist covers the top failure modes and the metric that catches each one.

Symptom: "Users report missing events"

1. Check the account's last successful incremental fetch timestamp. If it is more than 30 minutes ago, the push channel is likely broken.
2. Verify the watch/subscription is still active by GET'ing the resource. Google returns 404 if the channel is gone; Microsoft returns the subscription with its current expirationDateTime.
3. Confirm renewal jobs are running. Look at logs for the last successful renewal per account.
4. Fall back to polling for the affected account until the push channel is restored, and re-attempt push subscription creation.

Symptom: "OAuth suddenly fails with invalid_grant"

1. The refresh token has been revoked (user disconnected, changed password, or admin revoked). Mark the account for reauth and stop retrying.
2. Alternatively, the refresh token was rotated and you did not persist the new one. Audit your refresh handler for the rotation case.
3. For Microsoft, verify the offline_access scope is still in the consent - tenant admins can restrict scopes at the org level.

Symptom: "Webhook validation fails during subscription creation" (Microsoft only)

1. Confirm the notificationUrl is publicly reachable and returns 200 with the validationToken echoed as text/plain.
2. Verify the response returns within 10 seconds. Cold-start latency on serverless can exceed this - warm the endpoint or move it off cold-start-heavy runtimes.
3. Check the URL is HTTPS with a valid certificate. Self-signed certs and expired certs fail silently.

Symptom: "Sync token expired (410 Gone)"

1. Not an error condition on its own - trigger a full resync per the recovery recipe.
2. If 410s are frequent (> 1% of fetches over an hour), investigate why. Google sometimes invalidates tokens after backend changes; Microsoft invalidates after ~30 days of inactivity.

Symptom: "Duplicate events in the local store"

1. Check idempotency. Both providers can redeliver notifications. Deduplicate by the provider's event ID plus changeKey (Microsoft) or etag (Google).
2. Confirm you are not applying an inbound notification and a scheduled polling fetch to the same event without deduplication.

Symptom: "Watch/subscription renewal fails intermittently"

1. Confirm the access token used for renewal is fresh - a stale access token causes 401 on the renewal call.
2. For Google, ensure you stop the old channel before creating the new one, otherwise you may hit per-user channel limits.
3. For Microsoft, watch for 429s on the subscription endpoint - Graph applies its own throttle to subscription management calls.

Metrics to monitor across all calendar accounts:

Alert on all of these. A silent calendar integration is worse than a broken one: customers assume it works, base decisions on the data, and only notice weeks later when a meeting is missing.

How Truto Normalizes CRM Complexities Without Custom Code

Everything above is correct, and every team that builds it from scratch ends up writing roughly the same code. Building out token buckets, keyset pagination, SOQL query builders, and dynamic schema discovery for just two CRMs will consume months of engineering time. Adding Microsoft Dynamics, Zoho, and Pipedrive will consume the rest of your year.

After enough integrations, you start to see the pattern: the differences between HubSpot and Salesforce are mostly data, not logic. The HTTP client, the pagination loop, the OAuth refresh, the response shape - those should be one piece of code parameterized by configuration.

Truto's architecture eliminates this burden entirely. Through our Unified API, we handle these complexities generically, allowing you to ship new API connectors as data-only operations.

Generic Execution Pipelines and JSONata

Truto operates on a declarative configuration model. There is zero integration-specific code in the Truto runtime. No if (provider === 'hubspot') statements exist in our proxy layer.

Instead, integration behavior is defined as data. We use JSONata expressions to map unified requests into provider-specific formats. For example, when you send a unified request to filter contacts by email, Truto's mapping engine automatically translates that into a HubSpot filterGroups array or a Salesforce SOQL WHERE clause.

flowchart LR
    A[Unified Request<br>GET /unified/crm/contacts] --> B[Generic Mapping Engine]
    B --> C{Integration<br>Config}
    C -->|HubSpot| D[filterGroups +<br>cursor pagination]
    C -->|Salesforce| E[SOQL WHERE +<br>OFFSET / nextRecordsUrl]
    D --> F[Unified Response<br>same shape]
    E --> F

Here is a simplified example of how Truto uses JSONata to normalize a Salesforce response, automatically identifying custom fields by their __c suffix:

response_mapping: >-
  response.{
    "id": Id,
    "first_name": FirstName,
    "last_name": LastName,
    "email_addresses": [{ "email": Email }],
    "custom_fields": $sift($, function($v, $k) { $k ~> /__c$/i and $boolean($v) })
  }

This single expression maps standard fields while dynamically isolating any custom fields the specific Salesforce tenant has created, packaging them neatly into a custom_fields object in the unified response.

Standardizing API Rate Limits

Every API returns rate limit information differently. HubSpot uses X-HubSpot-RateLimit-Remaining, while others use RateLimit-Remaining or bury the limits in the response body.

Truto normalizes upstream rate limit info into standardized headers per the IETF specification:

• ratelimit-limit
• ratelimit-remaining
• ratelimit-reset

Dynamic Resource Routing

APIs often require different endpoints for different actions. For example, HubSpot has a standard endpoint for listing contacts, but requires a completely different search endpoint if you want to filter those contacts.

Truto supports dynamic resource routing natively. The platform evaluates the incoming request parameters and automatically routes the call to the correct upstream endpoint.

sequenceDiagram
    participant Client
    participant Truto Unified API
    participant Truto JSONata Engine
    participant Upstream CRM

    Client->>Truto Unified API: GET /unified/crm/contacts?email=test@example.com
    Truto Unified API->>Truto JSONata Engine: Evaluate query parameters
    Truto JSONata Engine-->>Truto Unified API: Route to Search Endpoint
    Truto Unified API->>Upstream CRM: POST /crm/v3/objects/contacts/search
    Upstream CRM-->>Truto Unified API: Raw Provider Response
    Truto Unified API->>Truto JSONata Engine: Apply Response Mapping
    Truto JSONata Engine-->>Truto Unified API: Normalized Unified JSON
    Truto Unified API-->>Client: 200 OK (Unified Schema)

This architecture ensures that your engineers write code against one predictable REST API, while Truto handles the chaotic reality of third-party SaaS platforms in the background.

Unified Webhooks for Bidirectional Sync

The inbound webhook handling described in Recipe 4 is exactly the kind of per-provider complexity that compounds across integrations. HubSpot uses X-HubSpot-Signature-v3 with HMAC SHA-256 over a specific concatenation. Salesforce uses its own verification scheme. Every CRM webhook has a different payload shape, different event type naming, and different retry behavior.

Truto's unified webhook layer normalizes all of this. Truto supports two inbound webhook ingestion patterns - account-specific and environment-integration fan-out - and uses JSONata-based configuration for provider-specific event normalization. The result is that your app receives a consistent event payload regardless of whether the source is HubSpot, Salesforce, or any other CRM.

Outbound delivery to your endpoint uses queue-backed processing with signed payloads. Truto signs every outbound webhook with HMAC-SHA256 and sends the signature in the X-Truto-Signature header, so your verification code is the same regardless of the upstream provider.

Go-Live Checklist

Before flipping your bidirectional CRM sync to production, walk through this list:

Authentication & Security

• OAuth 2.0 flow tested with token refresh (confirm your app refreshes tokens before expiry)
• Webhook endpoint is HTTPS-only
• v3 HMAC signature verification active and rejecting stale timestamps (> 5 minutes)
• Webhook secret stored encrypted at rest

Rate Limiting

• Token bucket limiter in place for general API calls (110/10s for public apps, 190/10s for private Pro/Enterprise)
• Separate rate limiter for CRM Search API (~4-5 req/s)
• Retry-After header respected on 429 responses
• Jitter added to retry delays

Sync Logic

• Field ownership matrix documented and implemented for every synced field
• Echo/loop prevention via outbound fingerprints verified with a live round-trip test
• Deletion handling strategy configured (hard delete, soft delete, or archive)
• Idempotency store in place for inbound webhook eventId deduplication
• Custom properties discovered dynamically (not hardcoded)

Pagination & Bulk

• Cursor-based pagination handling tested beyond 10,000 records
• Batch endpoints used for bulk operations (up to 100 records per batch call)
• Bulk API 2.0 (Salesforce) tested for backfills over 5,000 records

Monitoring

• 429 rate alerting configured (> 5% threshold)
• Webhook inbound silence alert (0 events for 10+ minutes)
• HMAC verification failure alerts active
• OAuth token refresh failure alerts active
• Reconciliation job scheduled (every 4-8 hours)
• Drift count baseline established

Operational Readiness

• Dead letter queue configured for failed webhook deliveries
• Manual reconciliation / full-resync trigger available
• Runbook documenting how to pause sync, replay events, and investigate drift

Strategic Wrap-Up and Next Steps

Building native CRM integrations requires far more than parsing a JSON response. To survive production environments, your architecture must account for strict burst rate limits, complex pagination models, and the reality of highly customized tenant schemas.

If your team is shipping a Salesforce or HubSpot integration this quarter, the priorities in order are:

1. Implement a token bucket and Retry-After-aware client for HubSpot before you write a single mapper.
2. Use Bulk API 2.0 for any Salesforce backfill over a few thousand records.
3. Discover custom fields dynamically - do not hardcode field lists.
4. Treat OAuth refresh as a separate failure domain with its own alerting.
5. Decide build-vs-buy explicitly. If you are heading toward 5+ CRMs, the cost curve of maintaining per-vendor code paths gets brutal fast.