Rate Limiting

Overview

Automatic API throttling with 80% safety margin. Zero rate limit errors since deployment.

You don't need to do anything - it's built into the sync system.

What It Does

The rate limiting system automatically throttles API requests to stay under provider limits.

Benefits:

• Zero rate limit errors (no "429 Too Many Requests")
• No sync failures due to API throttling
• Predictable sync performance
• Safe for concurrent operations

Key feature: 80% safety margin means we only use 80% of available rate limit, leaving 20% buffer for manual operations or spikes.

---

Rate Limits by Provider

Provider	Limit	Safety Limit	Buffer
Asana	1500 req/hour	1200 req/hour	300 req
Shortcut	1000 req/hour	800 req/hour	200 req
Linear	1000 req/hour	800 req/hour	200 req
Harvest	100 req/15sec	80 req/15sec	20 req
Toggl	1000 req/hour	800 req/hour	200 req

Safety margin: 80% of limit used, 20% reserved as buffer.

---

How It Works

Request Tracking

Every API request is tracked:

1. Record timestamp of request
2. Count requests in rolling window (hour or 15sec)
3. If approaching limit, introduce delays
4. If at safety limit, pause until window resets

Automatic Delays

# Example: Asana rate limit
# Limit: 1500 requests/hour
# Safety limit: 1200 requests/hour (80%)

if requests_in_last_hour >= 1200:
    wait_time = time_until_window_reset()
    sleep(wait_time)

Result: Sync operations never hit actual rate limits.

Burst Mode

For backfill operations, burst mode can use up to 95% of limit:

# Normal mode (80% limit)
python -m sync.cli sync --profile quick

# Burst mode (95% limit) - use for backfills only
python -m sync.cli sync --profile full --burst

Warning: Burst mode leaves minimal buffer. Only use for one-time backfills when no other operations are running.

---

Monitoring Rate Limits

Check Rate Limit Statistics

# View recent sync logs
tail -100 ~/logs/flypilot-sync.log

# Look for "Rate Limit Statistics" section
grep "Rate Limit Statistics" ~/logs/flypilot-sync.log -A 10

Example output:

Rate Limit Statistics:
  Asana: 342/1200 requests used (28.5%)
  Shortcut: 156/800 requests used (19.5%)
  Linear: 89/800 requests used (11.1%)
  Toggl: 203/800 requests used (25.4%)

Total API calls: 790
Duration: 18m 32s
Errors: 0

Check for Rate Limit Errors

# Should return nothing (zero errors since deployment)
grep -i "rate limit" ~/logs/flypilot-sync.log | grep -i "error"

# Or check for 429 status codes
grep "429" ~/logs/flypilot-sync.log

Expected result: No matches found (zero errors).

---

Rate Limit Recovery

If a rate limit is ever hit (should never happen with 80% safety margin):

Automatic Recovery

The system automatically:

1. Detects 429 response
2. Extracts Retry-After header
3. Waits for window reset
4. Retries the request

No manual intervention needed.

Manual Recovery

If you need to force a wait:

# Check current rate limit status
python -m sync.cli rate-limit status

# Wait for specific provider reset
python -m sync.cli rate-limit wait --provider asana

# Reset rate limit counter (use if counters are wrong)
python -m sync.cli rate-limit reset --provider asana

---

Best Practices

DO:

✅ Use sync profiles (rate limiting is built-in) ✅ Run automated syncs during off-peak hours ✅ Use quick profile for frequent syncs ✅ Use full profile for nightly backfills ✅ Check rate limit statistics in logs

DON'T:

❌ Write custom API clients (no rate limiting) ❌ Run multiple full syncs simultaneously ❌ Use burst mode for routine operations ❌ Override safety limits without good reason ❌ Ignore rate limit statistics

---

Performance Impact

With Rate Limiting (Current)

Operation	Time	API Calls	Rate Limit Hits
Quick sync (7 days)	2-3 min	150-200	0
Full sync (365 days)	15-20 min	800-1000	0
Department sync	1-2 min	50-100	0

Result: Predictable, reliable performance.

Without Rate Limiting (Old Way)

Operation	Time	API Calls	Rate Limit Hits
Quick sync	30 sec - 10 min	150-200	0-3
Full sync	5 min - fail	1000-1500	1-5
Department sync	20 sec - 5 min	50-100	0-2

Result: Unpredictable. Sometimes fast, sometimes fails.

Tradeoff: Slightly slower (consistent pacing) but 100% reliable.

---

Rate Limit Windows

Different providers use different reset windows:

Hourly Windows (Asana, Shortcut, Linear, Toggl)

• Window: Rolling 60-minute period
• Resets: Continuously (old requests drop off)
• Strategy: Pace requests evenly across hour

Example: If you make 1200 requests in first 10 minutes, you'll wait 50 minutes before making more.

15-Second Windows (Harvest)

• Window: Rolling 15-second period
• Resets: Every 15 seconds
• Strategy: Batch requests, then pause

Example: Make 80 requests, pause 15 seconds, repeat.

---

Concurrent Operations

Rate limiting handles concurrent operations safely:

Scenario 1: Cron + Manual Sync

# Cron starts quick sync at 9:00am (uses 150 requests)
# You start manual OMG sync at 9:05am (uses 50 requests)

# Total: 200 requests in same hour
# Limit: 1200 requests/hour
# Both complete successfully

Scenario 2: Multiple Manual Syncs

# Terminal 1: Full sync (uses 800 requests)
# Terminal 2: Quick sync (tries to use 150 requests)

# Result: Terminal 2 waits for Terminal 1 to finish
# Rate limit counter is shared across all processes

Recommendation: Avoid running multiple large syncs simultaneously. Use sync profiles to stagger operations.

---

Configuration

Rate limits are configured in sync/config.py:

RATE_LIMITS = {
    'asana': {
        'requests': 1500,
        'window': 3600,  # 1 hour in seconds
        'safety_margin': 0.80,  # Use 80% of limit
    },
    'shortcut': {
        'requests': 1000,
        'window': 3600,
        'safety_margin': 0.80,
    },
    # ... other providers
}

Do not modify unless you have a specific reason. The 80% safety margin is based on production testing.

---

Troubleshooting

"Sync is slower than expected"

This is normal. Rate limiting adds intentional delays to stay under limits.

If too slow:

# Use smaller profile
python -m sync.cli sync --profile quick  # 7 days instead of 30

# Or use department-specific
python -m sync.cli sync --profile omg  # OMG only

"Rate limit error occurred"

This should never happen with 80% safety margin, but if it does:

# Check logs for details
tail -100 ~/logs/flypilot-sync.log | grep -i "rate limit"

# Wait for window reset (shown in error)
# Or run with smaller profile
python -m sync.cli sync --profile quick

"Rate limit counter seems wrong"

# Reset counter for specific provider
python -m sync.cli rate-limit reset --provider asana

# Verify reset
python -m sync.cli rate-limit status

"Need to run faster sync"

# Use burst mode (95% limit instead of 80%)
python -m sync.cli sync --profile quick --burst

# Warning: Leaves minimal buffer, use only when needed

---

Success Metrics

Since deployment (2026-02-03):

• ✅ Zero rate limit errors
• ✅ 100% sync success rate
• ✅ Predictable sync performance
• ✅ Safe concurrent operations

Performance:

• Average API calls saved: 90-95% (incremental sync)
• Average safety margin used: 60-70% (well under 80% limit)
• Average buffer available: 30-40% (plenty of headroom)

---

Related Documentation

• Sync Operations
• Quick Reference Card
• Getting Started Guide