Sr. AI Engineer (Agent Analytics)

Companies are deploying agents faster than they can measure them. The missing layer is trusted, KPI-mapped analytics. You'll help build that layer—from raw traces to business action—owning core systems that will process billions of agent events.

What you'll do

• Ship v1→vN of ingestion, classification, labeling, evals, and analytics services (stream + batch; multi-tenant; privacy-first).
• Design the metrics layer: durable schemas, KPI definitions, versioning, lineage, and auditability.
• Create observability & drift detection for behaviors, data quality, and KPI misalignment—plus auto-backfills/remediation.
• Integrate with agent/observability tools (Langfuse/LangSmith/Helicone), warehouses (Snowflake/BigQuery/ClickHouse/Postgres), and BI (Looker/Metabase/Hex/Mode).
• Work directly with design partners to turn ambiguous business goals into measurable rubrics, then automate them.
• Set engineering guardrails: privacy, PII redaction, tenant isolation, SLA/SLOs, migrations, testing, and incident playbooks.

Must-haves

• Deep hands-on experience with LLM/agent stacks (tool-calling, function executors, retrieval) and their traces (prompts, tool calls, state, outcomes).
• Fluent in Python and/or TypeScript, and comfortable building data/analytics services and APIs.
• Think like an analytics engineer: SQL, dimensional modeling, metrics layers (dbt or equivalent), warehouse-first design.
• Have shipped eval systems (LLM-as-judge, rubric/HITL) and know how to productionize them.
• Understand embeddings/vector search and use retrieval-style signals for classification and labeling.
• Care deeply about privacy/security (PII handling, tenant isolation, audit logs) and operability (observability, SLOs).

Nice-to-haves

• Event pipelines/queues (Kafka/Redpanda, Redis/BullMQ), orchestrators (Temporal/Prefect/Airflow).
• Product analytics chops (cohorts, funnels, attribution) and experimentation (canaries, bandits) for agent policy changes.
• Experience with Langfuse/LangSmith/Helicone, ClickHouse/Snowflake/BigQuery/Postgres, and BI tooling.