trek-ingestor

Multi-user IMAP → Trek bridge: forward your booking confirmations to a shared mailbox; the daemon fans them out to each user's Trek account based on the forwarded From: header.

Two processes:

• daemon (trek-ingestor-daemon) — polls IMAP, parses each new message, routes by From: to a registered user, calls Trek's MCP API.
• web (trek-ingestor-web) — FastAPI admin UI for registering users, managing their From: aliases, and running the per-user Trek OAuth flow.

Both ship as container images (codeberg.org/cdamian/trek-ingestor and codeberg.org/cdamian/trek-ingestor-web) built by the Forgejo workflow on push to main.

Setup

cp .env.example .env
# edit .env with IMAP settings, TREK_INGESTOR_WEB_TREK_URL, and
# ANTHROPIC_API_KEY (optional, enables LLM fallback).
uv sync --extra web

Run the web app, log in via Trek OAuth, add at least one sender alias:

uv run trek-ingestor-web --host 127.0.0.1 --port 8000
# then open http://127.0.0.1:8000/login and click "Continue to Trek"

The "Continue to Trek" button bounces you to your Trek instance's OAuth consent screen:

There is no separate registration step — the first successful OAuth login creates the user row, keyed by Trek's OIDC sub. The same Trek user always resolves to the same row across logins.

Run the daemon:

uv run trek-ingestor-daemon                 # POLL_INTERVAL or 60s
uv run trek-ingestor-daemon --interval 300  # override

SIGINT / SIGTERM shut it down cleanly between polls.

How injection works

For each unseen IMAP message:

1. Parse the message's From: header. If it doesn't match any registered alias, drop the message, ledger it as empty, mark \Seen.
2. Run the message through the parser stack (kitinerary → LLM fallback). The first parser returning a non-empty result wins.
3. Open an MCP session for the matching user via connect_trek_for_user. Look up an existing Trek trip whose dates overlap (coalesce); attach, or create a new trip.
4. Map every recognised reservation type (trains, flights, rental cars, boats/cruises, lodging, restaurants, events) to Trek's tools and call them. Endpoints with coordinates also become pinned Places on the map.

Outcomes recorded in the SQLite ledger ($XDG_DATA_HOME/trek-ingestor/ledger.sqlite, keyed by RFC 5322 Message-ID):

• Trip created — created row + \Seen.
• No reservations — empty + \Seen (cheap dedup on re-poll).
• Unrouted — empty + \Seen (no matching alias).
• MCP / network error — no ledger, no flag; retry on next poll.

Emails where no parser finds anything become a todo on the matched user's "Trek-ingestor inbox" trip (configurable per user via TREK_INGESTOR_INBOX_TITLE).

Parsers

The parser stack lives in trek_ingestor.parsers:

1. kitinerary — KDE's kitinerary-extractor, deterministic per provider. Set KITINERARY_EXTRACTOR to override the binary path; the parser also checks $PATH and the standard Fedora / Debian locations.
2. LLM fallback (optional) — Claude API. Activates when ANTHROPIC_API_KEY is set in .env. Covers long-tail providers kitinerary doesn't know (ibis/Accor, Vueling, etc.). Set TREK_INGESTOR_LLM_MODEL to override the default (claude-sonnet-4-6).

Adding a parser is a new file under src/trek_ingestor/parsers/ plus a line in default_parsers(); no other code changes.

Container images

The Forgejo build job publishes codeberg.org/cdamian/trek-ingestor:latest (daemon, from Containerfile) and codeberg.org/cdamian/trek-ingestor-web:latest (web app, from Containerfile.web) on every push to main, after lint/test/audit pass. The daemon image installs Fedora's kitinerary package so the deterministic parser stack works inside the container.

Build locally:

podman build -t trek-ingestor -f Containerfile .
podman build -t trek-ingestor-web -f Containerfile.web .

Both images set XDG_DATA_HOME=/var/lib and persist state under /var/lib/trek-ingestor/ — users.sqlite (registered users + tokens) and ledger.sqlite (message dedup). Mount the same volume into both containers so the web app's writes are visible to the daemon.

Both images run as the app user (UID/GID 10001) rather than root. A named volume picks up the in-image ownership automatically; a host bind mount must be writable by UID 10001 — chown 10001:10001 /your/data/dir once before the first run.

Running both containers

# Shared state. A named volume works; a bind mount of
# ~/.local/share/trek-ingestor:Z works too.
podman volume create trek-ingestor-data

# Web app: must come up first so you can log in before the daemon
# has anything to route on.
podman run --rm -d \
  --name trek-ingestor-web \
  -p 8000:8000 \
  --env-file .env \
  -e TREK_INGESTOR_WEB_REDIRECT_URI=http://localhost:8000/auth/callback \
  -e TREK_INGESTOR_WEB_SECRET_KEY="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')" \
  -v trek-ingestor-data:/var/lib/trek-ingestor \
  codeberg.org/cdamian/trek-ingestor-web:latest

# Daemon: same volume, same .env. It will read `users.sqlite` to
# resolve `From:` addresses to user OAuth tokens.
podman run --rm -d \
  --name trek-ingestor \
  --env-file .env \
  -v trek-ingestor-data:/var/lib/trek-ingestor \
  codeberg.org/cdamian/trek-ingestor:latest

TREK_INGESTOR_WEB_REDIRECT_URI and the URL you actually open in the browser must match scheme + host + port exactly — Trek echoes the redirect_uri back at OAuth callback, and the browser only sends the OAuth session cookie back to the origin it was set on. http://localhost:8000 on both sides is the simplest setup; behind a reverse proxy, set it to the public HTTPS URL.

TREK_INGESTOR_WEB_SECRET_KEY should be set explicitly (and stable across restarts) — when omitted, the web app generates a one-shot key at startup and every restart invalidates every active session.

The web app does not authenticate visitors itself — put it behind a reverse-proxy auth (Authelia / basic-auth / VPN) if it's reachable from anywhere untrusted.

If a user's refresh token expires the daemon raises a transient error and the user re-runs the OAuth flow at /login. The daemon never starts an interactive browser dance.

To avoid the periodic re-login chore, each user can switch to a long-lived machine token: in Trek, Settings → Integrations → New OAuth client, tick Machine client (client_credentials), grant trips:write + reservations:write, and paste the resulting client_id and client_secret into the Background ingestion section on /me. The web app validates against Trek before saving (both that the credentials work and that the required scopes are granted); the daemon then mints a fresh access token per poll via /oauth/token instead of refreshing.

Smoke test

After both containers are up:

1. Open http://localhost:8000/login and complete the Trek OAuth flow.

2. On /me, add the email address you'll forward booking confirmations FROM as an alias. The web app sends a confirmation link to that address; click it. Until you do the alias stays pending and the daemon won't route mail from it. Use Resend if the message doesn't arrive (rotates the token and resends via SMTP).

   

3. Forward a real booking email through your shared mailbox from that address.

4. podman logs -f trek-ingestor — you should see a routing: line matching your alias, a parser hit, and created trip id=….

Required environment

• IMAP_HOST, IMAP_PORT, IMAP_USE_SSL, IMAP_USER, IMAP_PASSWORD, IMAP_FOLDER — the shared mailbox the daemon polls.
• IMAP_PROCESSED_FOLDER — optional; when set (e.g. Processed), each message the daemon finishes with (created / empty / unrouted / already-ledgered) is moved out of IMAP_FOLDER into this folder. The folder is created on first use. Unset = leave messages in place with \Seen (the prior behaviour). Transient failures stay put so the next poll retries.
• TREK_INGESTOR_WEB_TREK_URL — the Trek instance the web app authenticates against, e.g. https://trek.example. Fixed in env; users don't pick it. The app fails to start if it's unset.
• TREK_INGESTOR_WEB_REDIRECT_URI — publicly reachable callback URL, e.g. https://trek-ingestor.example/auth/callback. Must match what the user opens in the browser (scheme + host + port).
• TREK_INGESTOR_WEB_SECRET_KEY — secret for signing the post-login session cookie and the CSRF tokens. Generate with python -c 'import secrets; print(secrets.token_urlsafe(32))'. Set it explicitly so sessions survive restarts.
• TREK_INGESTOR_WEB_COOKIE_SECURE — optional; force the Secure flag on outbound cookies. Defaults to auto (https redirect URI → on, http → off). Set to 1 behind a TLS-terminating reverse proxy whose inner redirect URI is http-shaped.
• SMTP_SERVER, SMTP_PORT, SMTP_USER, SMTP_PASSWORD, SMTP_FROM — outbound SMTP for alias-confirmation mail. Brevo's transactional tier (smtp-relay.brevo.com:587, free 300/day) is the reference target; any STARTTLS host works. SMTP_FROM is the visible From: and must be verified at the provider — it doesn't have to match SMTP_USER. Set SMTP_USE_SSL=1 for implicit TLS on port 465 instead. The web app refuses to add an alias (502) when these aren't all set.
• ANTHROPIC_API_KEY — optional; enables the LLM fallback parser.
• POLL_INTERVAL — daemon poll interval in seconds (default 60).
• TREK_INGESTOR_DB / TREK_INGESTOR_USERS_DB — override SQLite paths.

Per-user state (Trek OAuth identity + tokens or machine client_credentials, display email, aliases) lives in users.sqlite — managed via the web app, not env.