Skip to main content

Sync Operations

Overview

Get data from Asana/Shortcut/Linear without writing custom API clients. Production-ready system with 10 pre-configured sync profiles.

Quick Start​

# Activate environment
cd ~/GitHub/flypilot && source .venv/bin/activate

# List all available profiles
python -m sync.cli profiles

# Run a quick sync (7 days, incremental)
python -m sync.cli sync --profile quick

# Test first with dry-run
python -m sync.cli sync --profile quick --dry-run

10 Sync Profiles​

Comprehensive Profiles​

standard (default)​

python -m sync.cli sync --profile standard
  • Description: Standard sync with recent activity (default)
  • Departments: All (LRND, OMG, FP)
  • Days: 30
  • Syncs: Tasks, time, comments, attachments
  • Completed tasks: No
  • Use case: Daily routine sync with balanced scope

quick​

python -m sync.cli sync --profile quick
  • Description: Fast sync of active tasks and recent time entries
  • Departments: All (LRND, OMG, FP)
  • Days: 7
  • Syncs: Tasks, time, comments, attachments
  • Completed tasks: No
  • Use case: Quick updates during business hours (Mac Studio cron: every 30 min)

full​

python -m sync.cli sync --profile full
  • Description: Complete sync including historical data
  • Departments: All (LRND, OMG, FP)
  • Task days: 365
  • Time days: 90 (Toggl API limit)
  • Syncs: Tasks, time, comments, attachments, completed tasks
  • Max file size: 50MB
  • Use case: Comprehensive nightly sync (Mac Studio cron: 6am daily)

Department-Specific Profiles​

omg​

python -m sync.cli sync --profile omg
  • Description: OMG Interactive department only
  • Providers: Shortcut (tasks) + Toggl (time)
  • Days: 30
  • Use case: When working exclusively on OMG client issues

lm​

python -m sync.cli sync --profile lrnd
  • Description: Learned Media department only
  • Providers: Asana (tasks) + Harvest (time)
  • Days: 30
  • Use case: When working exclusively on LRND client issues

fp​

python -m sync.cli sync --profile fp
  • Description: FLYPILOT department only
  • Providers: Linear (tasks) + Toggl (time)
  • Days: 30
  • Use case: When working exclusively on FLYPILOT internal work

Entity-Specific Profiles​

tasks-only​

python -m sync.cli sync --profile tasks-only
  • Description: Tasks, projects, and clients only - no time entries
  • Departments: All (LRND, OMG, FP)
  • Days: 90
  • Syncs: Tasks, comments, attachments
  • Use case: Refresh task data without pulling time entries

time-only​

python -m sync.cli sync --profile time-only
  • Description: Time entries only - no tasks
  • Departments: All (LRND, OMG, FP)
  • Days: 90
  • Syncs: Time entries only
  • Use case: Update billing data without fetching task changes

Backfill Profiles​

content-backfill​

python -m sync.cli sync --profile content-backfill
  • Description: Backfill comments and attachments for existing tasks
  • Departments: All (LRND, OMG, FP)
  • Days: 365
  • Syncs: Comments, attachments only (no tasks or time)
  • Max file size: 100MB
  • Use case: One-time or periodic enrichment of historical tasks

historical​

python -m sync.cli sync --profile historical
  • Description: Full historical sync (2 years of tasks)
  • Departments: All (LRND, OMG, FP)
  • Task days: 730
  • Time days: 90 (Toggl API limit)
  • Syncs: Everything including completed tasks
  • Max file size: 100MB
  • Use case: Initial setup or major data refresh

Incremental Sync​

Incremental by Default

All syncs use incremental mode by default to minimize API calls and improve performance (90-95% fewer API calls).

How It Works​

  • Tasks and time entries fetch only records modified since last_successful_sync_at
  • Clients and projects always perform full sync (API limitation - no modified_at filter)
  • Timestamp tracked per source in sync_state table in Supabase

Failsafe Mechanism​

  • Uses last_successful_sync_at (not last_sync_at) to determine the sync window
  • If a sync fails, the next sync will retry from the last successful point
  • Prevents data gaps when syncs fail partway through

Override Incremental Mode​

# Force full sync (disable incremental)
python -m sync.cli sync --profile quick --force
python -m sync.cli sync --profile quick --full    # Alias for --force

Rate Limiting Integration​

All sync operations automatically respect rate limits:

ProviderRate LimitSafety Margin
Asana1500 requests/hour80% (1200 used max)
Shortcut1000 requests/hour80% (800 used max)
Linear1000 requests/hour80% (800 used max)
Harvest100 requests/15sec80% (80 used max)
Toggl1000 requests/hour80% (800 used max)

Result: Zero rate limit errors since deployment.

Mac Studio Cron Schedule​

All automated jobs run on the Mac Studio. Logs are written to ~/logs/.

JobScheduleProfileDescription
Quick Sync*/30 9-18 * * 1-5quickEvery 30 min, 9am-6pm, Mon-Fri
Full Sync0 6 * * *fullDaily at 6am

Quick Sync uses incremental mode (default) for fast updates during business hours. Full Sync runs --force to ensure complete data integrity daily.

Checking Logs​

# View recent sync activity
tail -100 ~/logs/flypilot-sync.log

# Check for errors
grep -i error ~/logs/flypilot-sync.log

# View rate limit statistics
tail -100 ~/logs/flypilot-sync.log | grep "Rate Limit Statistics"

# Check incremental sync performance
grep "Incremental" ~/logs/flypilot-sync.log

Data Flow​

Source Systems                    Supabase                    Consumers
─────────────────────────────────────────────────────────────────────────────
Asana (LRND tasks) ──┐
Shortcut (OMG tasks) ──┼──▶ UNIFIED DATA LAKE ──▶ Linear (visibility)
Linear (FP tasks) ──┤  (source of truth)      ──▶ Dashboards
Harvest (LRND time) ──┤                          ──▶ Reports
Toggl (OMG/FP time) ──┘                        ──▶ Agents

All external data flows through Supabase as the central data lake.

Common Use Cases​

Daily Team Updates​

# Quick sync during business hours
python -m sync.cli sync --profile quick

Department-Specific Work​

# Working on OMG projects today
python -m sync.cli sync --profile omg

# Working on LRND projects today
python -m sync.cli sync --profile lrnd

Billing Reports​

# Get time entries without task overhead
python -m sync.cli sync --profile time-only

Task Status Updates​

# Get task updates without time entries
python -m sync.cli sync --profile tasks-only

Initial Setup​

# First-time data load (2 years of history)
python -m sync.cli sync --profile historical

Troubleshooting​

"Rate limit error"​

# Wait for reset (shown in error)
# Or use smaller profile
python -m sync.cli sync --profile quick  # 7 days instead of 30

"Sync taking too long"​

# Use incremental mode (default)
python -m sync.cli sync --profile quick

# Or use department-specific profile
python -m sync.cli sync --profile omg  # OMG only

"Missing recent data"​

# Check last sync time in logs
tail -100 ~/logs/flypilot-sync.log | grep "last_successful_sync"

# Force full sync if needed
python -m sync.cli sync --profile quick --force

"Comments not syncing"​

# Use content-backfill profile
python -m sync.cli sync --profile content-backfill

Configuration​

Profile configuration is stored in config/sync-profiles.yaml.

To add a new profile, edit the YAML file and restart the sync CLI.