OAuth at Scale: The Architecture of Reliable Token Refreshes

If you have ever built a Salesforce integration in a weekend, you know the feeling of triumph when that first 200 OK comes back. You store the access token, maybe toss the refresh token into a database column, and call it a day.

Then, three months later, your error logs light up.

Tokens are expiring mid-sync. Refresh requests are hitting race conditions because two background jobs tried to refresh the same token simultaneously. A customer changed their password, revoking all tokens, and your app is still hammering the API until you get rate-limited.

At Truto, we maintain connections to over 100 different SaaS platforms—from HRIS systems like Workday to CRMs like HubSpot. We process millions of API requests, and every single one relies on a valid, fresh credential.

We learned the hard way that managing OAuth token lifecycles is not a storage problem; it is a distributed systems problem.

Here is the engineering deep dive into how we architected a token refresh system that handles concurrency, proactive renewal, and graceful failure at scale.

The "Happy Path" is a Lie

The OAuth 2.0 spec provides a framework, but every provider implements it with their own chaotic flair.

• Expiry Times: Some tokens last 1 hour, some 24 hours, some never expire until used.
• Refresh Behavior: Some providers rotate the refresh token every time you use it (refreshing the refresh token). If you fail to capture the new one due to a network blip, you are locked out forever.
• Concurrency: If two processes try to refresh the same token at the exact same second, many providers will invalidate both requests, assuming a replay attack.

To handle this, we moved away from simple "check and refresh" logic to a multi-layered architecture involving proactive alarms, mutex locks, and self-healing state machines.

Layer 1: The Two-Pronged Refresh Strategy

Waiting for a token to expire before refreshing it is a recipe for latency and failed user requests. Conversely, refreshing it too aggressively wastes API quota. We use a hybrid approach: Proactive Alarms and Just-in-Time (JIT) Checks.

1. Proactive Refresh (The Background Worker)

Whenever a token is created or updated, we schedule work in our auth layer to run 60 to 180 seconds before the token expires - per account, not on a coarse global cron.

Why the randomization? To prevent thundering herds. If 10,000 accounts were connected at 9:00 AM, we don't want 10,000 refresh requests firing exactly at 9:59 AM. Spreading the load ensures stability.

When that scheduled refresh runs, it negotiates a new token, updates durable storage, and re-encrypts the new credentials.

2. Just-in-Time Safety Net

We cannot rely solely on background jobs. Clocks drift, schedulers miss edge cases, and sometimes a token expires faster than the provider claimed.

Before every single API request—whether it's a proxy call or a sync job—our infrastructure checks the token's validity. We use a 30-second buffer logic:

// Simplified logic: treat token as expired if it expires in the next 30s
if (token.expiresAt < (now + 30_seconds)) {
  await refreshCredentials();
}

This buffer is critical. Without it, a token might be valid when the check runs but expire 100ms later while the request is in flight to Salesforce.

Layer 2: Solving Concurrency with Mutex Locks

This is where most in-house integrations fail.

Imagine a scenario:

1. A scheduled sync job starts for Customer A.
2. Simultaneously, Customer A triggers a manual "Test Connection" in your UI.
3. Both processes see the token is about to expire.
4. Both processes fire a request to POST /oauth/token.

The Result: The provider receives two refresh requests. It processes the first, invalidates the old refresh token, and issues a new one. Then it processes the second request (using the now-invalid old refresh token), throws an error, and potentially revokes the entire grant for security reasons. Your customer is now disconnected.

The Solution: Per-account serialized refresh (mutex)

To prevent this, we wrap the refresh operation in a mutex (mutual exclusion) lock scoped to each integrated account.

Conceptually, each account has a single serializer for refresh work. When a refresh is requested:

1. The request attempts to acquire a lock for that specific integrated_account_id.
2. If no operation is in progress, it proceeds, arms a short watchdog timeout (e.g., 30s) so a stuck provider cannot block the account forever, and executes the refresh.
3. If a refresh is already running, the second request simply awaits the same in-flight operation and reuses its result.

This ensures that no matter how many concurrent callers need a fresh token, we only send one HTTP request to the provider. Everyone else gets the outcome of that single successful call.

Crucially, the lock is keyed by integrated_account_id, not global. Two refreshes for the same account are serialized. Two refreshes for different accounts run in parallel. This preserves throughput while eliminating the race condition.

Layer 3: Handling "Invalid Grant" and Re-Auth

Sometimes, refresh fails. The user might have uninstalled the app, changed their password, or an admin might have revoked the token.

When we receive a fatal error (like HTTP 400 invalid_grant or HTTP 401 Unauthorized), retrying is futile. We need to involve the human.

The needs_reauth State Machine

1. Detection: We catch invalid_grant errors specifically. Transient errors (HTTP 500s) trigger a retry with exponential backoff. Auth errors trigger a state change.
2. State Update: The account status is flipped from active to needs_reauth.
3. Notification: We fire a webhook event: integrated_account:authentication_error.

This allows our customers to listen for this event and immediately show a "Reconnect" banner in their UI.

Auto-Reactivation

We also support auto-reactivation. If an account is in needs_reauth but a subsequent API call succeeds (perhaps the user fixed the issue on the provider side, or it was a false positive from the API), we automatically flip the status back to active and fire a integrated_account:reactivated event.

Security at Rest

Storing thousands of access and refresh tokens requires paranoia. As part of our strict security standards, we never store tokens in plain text.

• Encryption: All sensitive fields (access_token, refresh_token, client_secret) are encrypted using AES-GCM before hitting the database.
• Masking: When developers list accounts via our API, these fields are masked. They are only decrypted inside the secure enclave of the refresh service just before being used.

How Truto's Unified Calendar API Works

Everything above - proactive refreshes, mutex locks, reauth detection - is the foundation that keeps Truto's Unified Calendar API running reliably. Here is the shape of it in one paragraph: you point your client at https://api.truto.one/unified/calendar/*, pass an integrated_account_id on every request, and Truto routes the call to the right provider (Google Calendar, Outlook, Calendly, Cal.com, and others), refreshes credentials if needed, applies the provider's rate-limit contract, maps the response into a normalized schema, and returns it in the same request cycle. No calendar data is cached or stored on our side.

That pass-through design means every single call depends on a valid OAuth token being available instantly. There is no fallback to a local cache if the token has expired. Calendar providers are a particularly demanding stress test for token management - they combine short-lived tokens (typically ~1 hour for both Google and Microsoft) with high-frequency access patterns like availability polling, event sync, and webhook-driven updates across potentially thousands of connected accounts.

The Unified Calendar API normalizes six core entities across providers:

• Calendars - the container for time-based entries ("Work", "Personal", "Team Holidays").
• Events - individual appointments, meetings, or blocked time slots. Full CRUD.
• Availability - calculated free/busy time windows for finding open slots without double-booking.
• EventTypes - pre-configured booking templates (highly relevant for Calendly and HubSpot Meetings) that define duration, location, and routing before an Event is formally booked.
• Contacts - attendees, organizers, or external guests associated with an Event.
• Attachments - files, agenda documents, or meeting materials linked to an Event.

You write your integration logic once against this schema; Truto handles the per-provider translation using JSONata expressions at runtime. If you switch a customer from Google Calendar to Outlook, no client code changes.

OAuth Flow and Required Scopes per Provider

Each calendar provider has its own OAuth configuration, scopes, redirect behavior, and token lifetime. Truto handles the full lifecycle - authorization redirect, token exchange, encrypted storage, and proactive refresh - so your application never touches provider credentials directly.

The end-to-end flow looks like this:

sequenceDiagram
    participant User
    participant YourApp as Your App
    participant Truto
    participant Provider as "Calendar Provider (Google/Microsoft/etc.)"

    User->>YourApp: Click "Connect Calendar"
    YourApp->>Truto: Create link token
    Truto-->>YourApp: link_token
    YourApp->>Truto: Redirect user to /oauth/{provider}
    Truto->>Truto: Build authorize URL with scopes, PKCE, state
    Truto->>Provider: 302 to provider's authorize endpoint
    User->>Provider: Grant consent
    Provider->>Truto: Redirect to /oauth/{provider}/callback with code
    Truto->>Provider: POST /token (code, client_id, client_secret, PKCE verifier)
    Provider-->>Truto: access_token + refresh_token + expires_in
    Truto->>Truto: Encrypt tokens (AES-GCM), schedule refresh alarm
    Truto-->>YourApp: Webhook: integrated_account:active
    YourApp-->>User: "Calendar connected"

Truto stores the state parameter with a 5-minute TTL, validates it on callback, and enforces PKCE (S256) where the provider supports it. If any step fails, the account never reaches active and no partial tokens are persisted.

Here are the scopes we recommend for each supported provider. You can trim these to the minimum your product actually needs when you bring your own OAuth app.

A few provider-specific notes:

• Google Calendar. Truto's default OAuth app is CASA Tier 2 certified - your customers can connect without hitting Google's unverified app warnings or user-cap limits. If you use read-only calendar data, drop calendar.events and calendar.calendars to reduce the consent prompt.
• Microsoft Outlook. Include offline_access explicitly, otherwise Microsoft will not issue a refresh token and every access token will expire in an hour with no way to renew. Truto handles both common, tenant-specific, and consumers authorization endpoints depending on the app registration.
• Calendly. The token endpoint returns a refresh token on every exchange but does not always rotate it. Truto's token merge logic preserves the previous refresh token if the provider omits it in the response, avoiding accidental lockout.
• Cal.com. No OAuth cycle at all. The API key is stored encrypted and injected into every request via the Authorization header. Skip Layers 1-3 mentally for this provider; the security and reauth logic still apply if the key is rotated or revoked.

Bring Your Own OAuth App

Truto provides a default OAuth app for each calendar provider so you can start building immediately. For production, you will likely want to use your own.

You supply your OAuth client credentials (client ID and secret) at the environment integration level - a pure configuration change, no code deployment. From that point, Truto uses your OAuth app for all authorization flows with that provider. The connection flow through Truto's Link SDK shows your app name and branding to the end user.

Why this matters for calendar integrations specifically:

• Your consent screen, your brand. Users see your app name when granting calendar access, not a third party's.
• Credential portability. If you ever switch providers, the OAuth tokens belong to your app. You migrate without forcing every customer to re-authenticate.
• Scope control. You decide exactly which calendar scopes to request. If your product only reads events, drop write scopes entirely.

The token refresh, encryption, and concurrency protections described in the layers above work identically regardless of whether you use Truto's OAuth app or your own.

Proactive Refresh and Mutex Logic in Calendar Practice

Layer 1 and Layer 2 are especially load-bearing for calendars because of how customers actually use them. A typical calendar-heavy workload looks like this:

• A background sync polls events every 5 minutes across every connected account.
• The end user opens a scheduling UI that fires 3-4 availability queries in quick succession.
• A provider webhook fires on an event change, which triggers a fetch to enrich the payload.

All three can happen inside the same second for the same account. Without a mutex, at least two of them will race on a token that is 40 seconds from expiry. With the per-account mutex:

1. The first caller to notice the token is within the 30-second expiry buffer acquires the lock and issues the refresh.
2. Callers 2 and 3 detect the in-flight refresh, await its promise, and reuse the resulting token.
3. The provider sees exactly one POST /token request, avoiding replay-attack heuristics that would revoke the grant.

Proactive refresh (the 60-180 second pre-expiry alarm) is what keeps you off the JIT path in the first place. On a healthy account, Layer 1 handles the refresh in the background before any request ever hits the 30-second buffer. Layer 2 is the fallback for the cases where an alarm was skipped, a token expired earlier than the provider's expires_in claim, or the account was just connected and the first sync ran before the initial alarm.

One detail worth calling out: new tokens are merged with existing tokens, not replaced. If Google returns a fresh access_token but omits refresh_token in the refresh response (which is common - many providers only issue the refresh token in the initial code exchange), the previous refresh_token is preserved. Blindly replacing the token object would silently lock every customer out on the second refresh.

Reauth UX and Error Handling for Calendar Apps

When a refresh finally does fail (revoked grant, changed password, deleted mailbox), the account status flips to needs_reauth and the integrated_account:authentication_error webhook fires. What you do with that event is the difference between a good calendar product and a broken one.

A solid reauth UX for calendars looks like this:

1. Subscribe to integrated_account:authentication_error. The webhook payload includes the integrated_account_id, the customer identifier, and a last_error message extracted from the provider's response (via error expressions).
2. Surface the state in your UI immediately. Do not wait for the user to notice broken sync. Show a banner or badge on any calendar view: "Your Google Calendar needs to be reconnected."
3. Provide a one-click reconnect. Re-open the Truto Link flow scoped to the same integrated_account_id. The user does not need to pick the provider again or re-enter identifiers.
4. Pause background work. Suspend polling for that account until you receive integrated_account:reactivated. Continuing to poll a needs_reauth account just burns rate-limit quota on 401 responses.
5. Handle auto-reactivation gracefully. If the user fixes the underlying issue on the provider side (for example, re-granting access from their Google security dashboard), the next successful API call will flip the account back to active and fire integrated_account:reactivated. Your UI should clear the reconnect banner on that event.

On the client side, treat every unified API response with an eye on two error shapes:

• HTTP 401 from Truto - the account has been marked needs_reauth. Do not retry. Route the user to reconnect.
• HTTP 429 from Truto - a normalized rate limit. Back off using Retry-After (see below).

Everything else (5xx, network errors) is safe to retry with exponential backoff.

Rate Limit Handling and Normalized Headers

Calendar providers signal rate limits in provider-specific ways. Google returns HTTP 429 with a Retry-After header. Microsoft Graph uses 429 with Retry-After plus custom RateLimit-* headers. Other providers get creative.

Truto normalizes all of this. Regardless of what the provider returns, your application sees the same headers:

If the provider signals a rate limit - even via a non-429 status code or a custom header - Truto always returns HTTP 429 to your application. One rate-limit handler covers Google, Microsoft, Calendly, and every other integration.

These headers are also returned on successful (2xx) responses, so you can monitor your rate limit headroom proactively without waiting to get throttled.

Retry and Backoff Strategy

When you receive a 429 from Truto, here is a practical client-side pattern:

async function callWithBackoff(
  request: () => Promise<Response>,
  maxRetries = 5
): Promise<Response> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await request();
    if (response.status !== 429) return response;
 
    const retryAfter = parseInt(
      response.headers.get('Retry-After') || '0', 10
    );
    const backoff = retryAfter > 0
      ? retryAfter * 1000
      : Math.min(1000 * Math.pow(2, attempt), 30_000);
 
    // Add jitter to avoid synchronized retry storms
    const jitter = Math.random() * 1000;
    await new Promise(r => setTimeout(r, backoff + jitter));
  }
  throw new Error('Max retries exceeded');
}

Key points:

• Always prefer Retry-After when present. The value comes from the provider, normalized to seconds by Truto, and reflects actual quota recovery timing.
• Fall back to exponential backoff when Retry-After is absent: 1s, 2s, 4s, 8s, 16s, capped at 30s.
• Add jitter if you have many workers hitting the same calendar provider. Without it, retries synchronize and create a second thundering herd.

Provider Capabilities: Proxy vs Unified

Not every calendar provider exposes the same operations. This matrix shows what is available through Truto's Unified Calendar API (normalized schema at /unified/calendar/*) vs the Proxy API (raw provider pass-through at /proxy/*):

"Unified + Proxy" means the operation is available through both the normalized Unified Calendar API and the raw Proxy API. The Proxy API gives you access to every provider-specific field and parameter. The Unified API normalizes responses into Truto's standard calendar schema so you can swap providers without changing client code. Operations marked "N/A" mean the provider does not support that concept at the API level.

Webhook Subscription Lifecycle and Renewal

Real-time calendar sync without a local data store hinges on two things: an OAuth token that stays fresh (Layers 1-3 above) and a live webhook subscription with the provider. Truto handles both, but the subscription side has different failure modes worth understanding, especially if you're building your own integration.

Provider webhook subscriptions expire. Google Calendar channels last up to 7 days for event resources, Microsoft Graph subscriptions cap at roughly 4230 minutes (~3 days) for calendar resources, and Calendly subscriptions do not expire but can be revoked. If your subscription lapses silently, your app stops receiving updates and you have no signal until someone notices the drift.

The lifecycle Truto manages on your behalf when you subscribe to record:created, record:updated, or record:deleted events on a calendar integrated account:

sequenceDiagram
    participant Truto
    participant Google as "Google Calendar"
    participant Microsoft as "Microsoft Graph"

    Note over Truto,Microsoft: Initial subscription
    Truto->>Google: POST /events/watch (channel expires <= 7 days)
    Google-->>Truto: channel id, resource id, expiry
    Truto->>Microsoft: POST /subscriptions (max 4230 min)
    Microsoft-->>Truto: subscription id, expirationDateTime

    Note over Truto,Microsoft: Renewal before expiry
    Truto->>Truto: Alarm fires ~24h before expiry
    Truto->>Microsoft: PATCH /subscriptions/{id} (new expirationDateTime)
    Microsoft-->>Truto: Updated expiration
    Truto->>Google: POST /events/watch (new channel, stop old one)
    Google-->>Truto: New channel id

    Note over Truto,Microsoft: Delivery
    Google->>Truto: change notification
    Microsoft->>Truto: change notification
    Truto->>Truto: Normalize to record:updated, fetch fresh resource
    Truto-->>Truto: Emit unified webhook to customer endpoint```

If you're managing subscriptions yourself, this is the minimum viable Microsoft Graph renewal loop:

```typescript
// Microsoft Graph subscription renewal
async function renewGraphSubscription(subscriptionId: string, accessToken: string) {
  // Cap at 4230 min for calendar resources; use 4200 for safety
  const newExpiration = new Date(Date.now() + 4200 * 60 * 1000).toISOString();

  const response = await fetch(
    `https://graph.microsoft.com/v1.0/subscriptions/${subscriptionId}`,
    {
      method: 'PATCH',
      headers: {
        Authorization: `Bearer ${accessToken}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ expirationDateTime: newExpiration }),
    }
  );

  if (response.status === 404) {
    // Subscription is gone. Recreate from scratch.
    return { recreate: true };
  }
  if (!response.ok) {
    throw new Error(`Renewal failed: ${response.status}`);
  }
  return response.json();
}

// Cron: run every 6 hours, renew anything within 24h of expiry
async function renewalCron() {
  const due = await db.query(
    `SELECT id, integrated_account_id
       FROM graph_subscriptions
      WHERE expires_at < NOW() + INTERVAL '24 hours'
        AND status = 'active'`
  );

  for (const sub of due) {
    const accessToken = await getFreshAccessToken(sub.integrated_account_id);
    try {
      const result = await renewGraphSubscription(sub.id, accessToken);
      if (result.recreate) {
        await recreateSubscription(sub);
      } else {
        await db.update('graph_subscriptions', sub.id, {
          expires_at: result.expirationDateTime,
        });
      }
    } catch (err) {
      await alertOps(`Renewal failed for ${sub.id}`, err);
    }
  }
}

Rules worth internalizing:

• Renew well before expiry. 24 hours of buffer means one failed cron run still leaves you time to recover. Renewing at expiry-minus-1-hour is asking for outages.
• Refresh the OAuth token before renewing. Graph rejects renewal calls made with a stale access token. Pipe through the same refresh path your API calls use.
• Handle 404 as "recreate from scratch." Microsoft garbage-collects expired subscriptions aggressively. There is no undo.
• Persist and validate clientState. You set it at creation. Every incoming notification echoes it back. If the value doesn't match your stored copy, drop the request; it isn't from Microsoft.
• Google needs a stop + create, not a renew. Google Calendar's watch channels can't be extended. You create a new channel before the old one expires, then stop the old one to prevent duplicate notifications.

Truto runs this loop internally per integrated account: subscriptions are renewed on a background schedule tied to expiry, clientState (and its equivalent for other providers) is validated on ingestion, and failed renewals emit the same integrated_account:authentication_error webhook you already listen to for token issues. You subscribe to Truto's unified record:* events once and get real-time notifications across every provider without touching subscription code.

Managing syncToken and deltaLink Safely

Webhooks tell you something changed. They rarely tell you what in full detail, and they don't cover the gap when your subscription was down or a notification was dropped. That's what syncToken (Google Calendar) and @odata.deltaLink (Microsoft Graph) are for: incremental sync cursors that let you pull only the changes since your last read, without asking the provider for the full event list.

The rules for these cursors are unforgiving:

1. Persist the cursor before processing the results, ideally in the same transaction. If you crash after processing but before saving, you reprocess the same window on the next run (safe with idempotent writes). If you crash after saving but before processing, you silently drop events forever. Always know which side of that trade-off you're on.
2. Never share cursors across scopes. Google issues one syncToken per calendar per request scope. If your first query included showDeleted=true or a specific field mask, every follow-up must use the exact same parameters. A different scope invalidates the token and returns HTTP 410 Gone.
3. Handle 410 Gone as a full-sync signal. Both Google and Microsoft eventually invalidate old tokens (Google after roughly 7 days of inactivity or on schema changes; Microsoft on major shape changes). When you get a 410, drop the cursor, run a bounded full sync, and store the new cursor from the response. Do not retry the same cursor.
4. Store per-calendar, not per-account. A user with 5 calendars has 5 sync cursors. One shared cursor per account will diverge fast.
5. Timestamp your last successful sync alongside the cursor. On 410, you'll use the timestamp to bound the full sync to a reasonable window instead of paginating the entire event history.

The Unified Calendar API exposes cursors through the Proxy API when you want to manage them yourself. For most workloads, Truto's ?updated_after=<timestamp> filter is easier and works uniformly across providers - it internally uses the right cursor mechanism per provider and returns normalized event objects, so you don't have to reason about syncToken vs deltaLink semantics.

Here's the recovery pattern:

async function incrementalSync(calendarId: string) {
  const state = await getCursorState(calendarId);
  // state = { cursor: string | null, lastSyncAt: ISO string | null }
 
  const params = new URLSearchParams();
  if (state.cursor) {
    params.set('syncToken', state.cursor);
  } else {
    // Fresh cursor: scope must match all future incremental calls
    params.set('showDeleted', 'true');
    params.set('singleEvents', 'true');
    if (state.lastSyncAt) params.set('updatedMin', state.lastSyncAt);
  }
 
  const res = await truto.get(
    `/proxy/google_calendar/calendars/${calendarId}/events?${params}`
  );
 
  if (res.status === 410) {
    // Cursor invalidated. Drop and full-sync from lastSyncAt.
    await clearCursor(calendarId);
    return incrementalSync(calendarId);
  }
 
  // Save cursor + timestamp first, then process events.
  await saveCursorState(calendarId, {
    cursor: res.data.nextSyncToken,
    lastSyncAt: new Date().toISOString(),
  });
  await processEvents(calendarId, res.data.items);
}

Because Truto does not store calendar data on its side, your app is the source of truth for what's been processed. Cursors let you keep that state consistent without ever paging through the full event history a second time. Combine cursor-based incremental sync with webhook-triggered fetches and you get sub-second freshness on active calendars without storing a mirror on Truto.

Per-Provider Rate-Limit Headers and Backoff Recipes

The normalized ratelimit-* headers Truto returns cover most workloads. When you need provider-specific detail (say, to optimize a batch job around Microsoft's per-app quota rather than the per-user quota), the Proxy API surfaces the raw headers. Here is what each calendar provider signals and how to respond:

The recommended default backoff, applied uniformly through Truto's normalized 429, adds a circuit breaker on top of the earlier snippet:

const BACKOFF = {
  initialDelayMs: 1_000,
  maxDelayMs: 60_000,
  maxRetries: 5,
  jitterMs: 500,
};
 
const BREAKER = {
  // Trip after N consecutive 429s in a rolling window per provider
  threshold: 10,
  windowMs: 60_000,
  cooldownMs: 5 * 60_000,
};
 
async function callWithBreaker(
  provider: string,
  request: () => Promise<Response>
) {
  if (isCircuitOpen(provider)) {
    throw new Error(`Circuit open for ${provider}`);
  }
 
  for (let attempt = 0; attempt < BACKOFF.maxRetries; attempt++) {
    const response = await request();
 
    if (response.status < 400) {
      recordSuccess(provider);
      return response;
    }
    if (response.status === 429 || response.status === 503) {
      recordThrottle(provider);
      if (shouldOpenCircuit(provider)) openCircuit(provider);
 
      const retryAfter = parseInt(
        response.headers.get('Retry-After') || '0',
        10
      );
      const backoff = retryAfter > 0
        ? retryAfter * 1000
        : Math.min(
            BACKOFF.initialDelayMs * 2 ** attempt,
            BACKOFF.maxDelayMs
          );
      const jitter = Math.random() * BACKOFF.jitterMs;
      await sleep(backoff + jitter);
      continue;
    }
    return response; // 4xx other than 429: don't retry
  }
  throw new Error(`Max retries exceeded for ${provider}`);
}

Recommended thresholds worth committing to production:

• Initial delay: 1000ms. Anything less contributes to the problem.
• Max delay: 60000ms. Longer than this and you should alert instead of continuing to retry.
• Circuit breaker trip: 10 consecutive throttles in 60 seconds for a given provider. Cool off for 5 minutes.
• Jitter: 0-500ms randomized. Prevents synchronized retry storms across your worker fleet.
• Max retries: 5. Beyond that, kick the failure to a dead-letter queue and page an operator.

For high-throughput sync workloads, pair the breaker with a token bucket per provider to preemptively throttle before you hit the provider's limit. Truto normalizes rate limits at the response layer; the token bucket is your defense at the request layer.

Recovery Strategy After Token Loss or Failed Renewals

Things break. A cloud provider has an outage. A refresh token gets rotated and you miss the new value due to a network hiccup. A user revokes access from their Google security dashboard. Your recovery playbook determines whether these are minor annoyances or customer-visible incidents.

The failure modes worth planning for:

1. Refresh token silently rotated but not persisted. Some providers issue a new refresh token on every use and invalidate the old one. If your storage write fails after a successful refresh, the next attempt uses the invalid old token and locks the account out.

Mitigation: Persist the new token in the same transaction that records the refresh outcome. If the write fails, treat the refresh as failed and let the user reconnect. Truto's token merge logic preserves the previous refresh token when the provider omits one in the response, but a newly issued refresh token must be committed before the response leaves the refresh service.

2. Webhook subscription expired while your renewal cron was down. Result: silent data drift. Your app thinks it's in sync because no webhooks are erroring, but nothing is arriving.

Mitigation: Alarm on webhook event volume dropping unexpectedly per environment (compare current hour against a trailing 7-day baseline). On recovery, drop your cursor and run a bounded incremental sync from the last known-good timestamp. Truto's auth-related webhooks cover token loss; the volume-drop signal is your subscription health check.

3. Provider revoked access; user hasn't noticed. Result: your app throws 401s that your retry logic dutifully backs off on until an operator wakes up.

Mitigation: On integrated_account:authentication_error, immediately pause all background work for that account and surface a "Reconnect required" banner. Do not retry auth calls in the background; you'll only generate more 401s. The account state is a hard signal - respect it.

4. Cursor invalidated (410 Gone) during a critical window. Result: incremental sync stops and you might reprocess or miss events depending on how you handle the fallback.

Mitigation: On 410, drop the cursor, run a full list scoped to the last known-good timestamp (not from epoch), and store the new cursor. Combine with de-duplication on event ID at your write layer. Full re-syncs are expensive; bound them by time window whenever possible.

5. Clock skew between your infrastructure and the provider. Result: tokens marked expired that are still valid, or worse, tokens treated as valid after they've expired. Both waste quota and can cause 401 storms.

Mitigation: Sync clocks via NTP, use the 30-second expiry buffer from Layer 1, and always trust the provider's expires_in from the last refresh response, not a locally computed value.

6. Simultaneous refresh from multiple regions or replicas. Result: the same race condition the account-level mutex prevents, but at a different layer if you're not careful about where the lock lives.

Mitigation: Make sure the mutex is scoped to a coordination surface that all replicas share, not to a per-process in-memory lock. Truto's per-account serializer runs at the platform layer, so it holds across every worker instance that could initiate a refresh.

Recovery is fastest when your app pauses cleanly, waits for the human signal, and resumes from a known state. Aggressive retry-on-auth-failure is almost always the wrong instinct.

Monitoring, Metrics, and Operational Expectations

Treat OAuth infrastructure the same way you treat a payment gateway: instrument it, alert on it, and know exactly what "healthy" looks like. Whether you build it yourself or use Truto, these are the signals worth watching.

Webhook signals from Truto. These are the fastest way to see auth health across your customer base:

• integrated_account:authentication_error - track the rate over time, per provider. A sudden spike usually means a provider rotated something, changed their token endpoint, or updated their consent policy.
• integrated_account:reactivated - the recovery counter. A healthy system shows reactivations following errors within minutes to hours, not days.
• integrated_account:active - fresh connections, useful for growth dashboards.

Response-level signals. On every unified API response, check:

• HTTP status distribution (2xx vs 401 vs 429 vs 5xx) segmented by provider.
• ratelimit-remaining headroom. If you consistently see this dropping below 10% of ratelimit-limit, you are one traffic spike away from throttling.
• Latency percentiles (p50/p95/p99) per provider. Google and Microsoft normally sit under 500ms p95 for calendar reads; sustained increases usually correlate with provider-side incidents.

Metrics we recommend tracking (client-side):

Alerting thresholds worth setting:

• Auth error rate exceeds 2x the trailing 7-day baseline for a single provider - probable provider-side change.
• Any single account sends more than 5 authentication_error events in an hour - stop retrying and page an operator.
• 429 rate above 1% of total requests for any provider - back off and check your polling strategy.

What to expect operationally. Truto's refresh architecture is designed so that a healthy account never sees an expired token. In practice:

• Proactive refresh completes 60-180 seconds before expiry, so under normal conditions no client request ever waits on a refresh.
• Mutex-serialized refreshes mean a burst of 100 concurrent requests against a nearly-expired token results in exactly one refresh call to the provider, not 100.
• Failed refresh attempts against transient errors (HTTP 5xx) are retried automatically on a 3-hour cadence, capped so a permanent revocation doesn't loop forever.
• Non-retryable errors (HTTP 401, 403, invalid_grant) skip retry entirely, flip the account to needs_reauth, and emit the webhook within seconds.

We publish live availability on our status page and communicate incidents there. For contractual SLAs on your specific deployment, talk to us directly.

Monitoring and Alerting Checklist

Copy this into your runbook. If any of these are unchecked in production, you're flying blind on real-time calendar sync.

Auth health

• Alert if integrated_account:authentication_error rate exceeds 2x trailing 7-day baseline per provider.
• Alert if any single account emits >5 authentication_error events in one hour (retry loop suspected).
• Dashboard: current count of accounts in needs_reauth per provider, per customer.
• Dashboard: median time from authentication_error to reactivated (reconnect UX health).

Token refresh health

• Alert on p95 refresh latency above 2s sustained for 5 minutes.
• Alert on refresh failure rate above 1% per provider.
• Metric: proactive vs just-in-time refresh ratio (proactive should dominate).

Webhook subscription health

• Alert on webhook event volume dropping more than 50% below trailing 7-day baseline per environment.
• Alert on subscription renewal failures per provider.
• Metric: age distribution of active subscriptions (spot the ones close to expiry).

Rate-limit health

• Alert if 429 rate exceeds 1% of total requests per provider.
• Dashboard: ratelimit-remaining p10 per provider (worst-case headroom).
• Alert if the circuit breaker trips for any provider (indicates sustained throttling).

Sync cursor health

• Metric: 410 Gone rate per calendar (should be near zero on healthy accounts).
• Alert on 410 spikes (indicates cursor invalidation event, possibly provider-side).
• Metric: median incremental-sync window (should stay small on healthy accounts).

Data flow health

• Alert on webhook delivery failure rate above 5% for any of your subscribed endpoints.
• Alert on end-to-end latency (provider webhook to your app processing) exceeding 30s p95.

Because Truto doesn't store calendar data on its side, these signals live in two places: at Truto (auth state, refresh health, delivery health) and at your app (cursor state, processing health). Both matter. The Truto side is exposed via webhooks and log APIs; the app side is your responsibility to instrument.

Best Practices for High-Frequency Calendar Access

Calendar workloads punish naive polling. A scheduling assistant checking availability every minute across 5,000 accounts is 300,000 requests per hour before the user has even done anything. Here is how to keep that workload well-behaved.

Prefer webhooks over polling. For Google, Outlook, and Calendly, Truto normalizes provider webhooks into unified events (record:created, record:updated, record:deleted) delivered to your endpoint. Use these to invalidate a local mirror instead of re-polling every N minutes. Webhooks are dramatically cheaper on rate limit budget and lower-latency for the user.

Batch availability queries. If a user is looking at a week view, ask for the whole week in one call, not seven daily calls. Google's freebusy.query and Microsoft Graph's getSchedule both accept multiple calendars and long time windows.

Cache free/busy results with tight TTLs. Availability rarely changes within a 30-60 second window for the same user. A short in-memory cache in front of Truto cuts request volume without materially affecting freshness. Invalidate the cache on any Event webhook for that calendar.

Use updated_after filters for incremental sync. The Unified Calendar API supports incremental fetches. Poll ?updated_after=<last_sync> instead of paginating the full event list every cycle. Combine with sync tokens where the provider supports them (Google's syncToken is exposed via the Proxy API).

Spread background work. If you run a nightly sync across all accounts, jitter the start times over a window (e.g., 30 minutes) instead of firing them all at midnight. This mirrors the same reasoning behind our randomized 60-180 second refresh window: avoid synchronized load on both your infrastructure and the provider's.

Respect the account-level mutex. Because refreshes are serialized per account, extremely bursty traffic against a newly-expired token will queue behind a single in-flight refresh. This is by design and correct behavior. If you see latency spikes in the p99 during token expiry moments, that is the refresh flight completing and unblocking everyone. It should be sub-second in normal conditions.

Separate hot and cold reads. If your product mixes user-facing scheduling (needs to be fast) with background analytics (can tolerate lag), route them differently. Real-time scheduling should always hit the Unified API. Analytics can query a synced local copy or use the truto_super_query mechanism to hit pre-synced data without touching the provider at all.

Summary: The Checklist for Reliable Auth

If you are building this in-house, ensure your architecture covers these bases:

1. Buffer your expiry checks: Don't wait for 0 seconds remaining.
2. Serialize your refreshes: Never let two threads refresh the same token.
3. Handle revocation gracefully: Distinguish between "API is down" (retry) and "Token is dead" (alert user).
4. Secure your storage: Encrypt at rest, always.
5. Instrument everything: Auth failures should be a first-class metric, not a log line.

Or, you can offload this entirely. At Truto, we treat authentication infrastructure as a core product, so you can focus on the data, not the handshake.