Forward Deployed AI Engineer - Assistant Vice President

Citi · New York New York United States · posted 3 Nov 2026

✓ Checked open yesterday on Citi's careers site

Full-timeIT & Software

About the job

The Applications Development Senior Programmer Analyst — Forward Deployed Engineer (C12) is an intermediate-level software engineering position responsible for constructing automated evaluation harnesses, curating regression testbeds, optimizing prompts and output schemas, and automating testing pipelines within embedded LOB delivery pods. The overall objective of this role is to ensure the reliability, accuracy, and safety of production AI agents by building rigorous evaluation suites in Agentflow Studio and driving automated CI/CD test execution. Responsibilities: • Conducts tasks related to the construction, calibration, and execution of automated evaluation test suites using Agentflow Studio, measuring agent task completion, tool call correctness, latency, and hallucination metrics. • Curates and maintains domain-specific golden datasets, synthetic benchmarks, and regression testbeds in collaboration with LOB domain experts and senior engineers. • Optimizes prompt templates, system instructions, few-shot examples, and structured JSON output parsers (Pydantic/JSON Schema) to maximize deterministic agent behavior. • Authors and maintains automated test execution and deployment pipelines within Lightspeed CI/CD, enforcing continuous evaluation gates prior to staging and production rollout. • Utilizes in-depth specialty knowledge of Python applications development to analyze complex agent errors, evaluate system performance, and recommend advanced coding and prompt solutions. • Recommends and implements security measures and Lakera Guardrail policies in post-implementation analysis of agent usage to ensure safe functionality. • Monitors and controls evaluation test cycles, providing operational support, performance metrics, and prompt diagnostic reporting to business stakeholders and squad leads. • Has the ability to operate with a limited level of direct supervision, exercising independence of judgement and autonomy while acting as an evaluation SME to pod team…

Read the full description on Citi's site ↗

Skills

Upload your CV on LitCareer to see which of these you match.

Similar jobs

Data Scientist
EssilorLuxottica · Dallas, TX, US, 75204 · 17 Nov 2026
✓ Checked open today
Product Manager - Warehouse Management (m/f/d)
MediaMarktSaturn Retail Group · Ingolstadt, DE, 85053 · 17 Nov 2026
✓ Checked open today
Senior Engineering Manager
Ingersoll Rand · Hampton, NJ, US, 08827 · 17 Nov 2026
✓ Checked open today
A Shift - SAP Coordinator
Mahle · Olive Branch, MS, US, 38654 · 17 Nov 2026
✓ Checked open today
Product Manager
Mahle · Farmington Hills, MI, US, 48335 · 17 Nov 2026
✓ Checked open today
Expert Data Analyst
Volvo Group · Greensboro, NC, US, 27409 · 17 Nov 2026
✓ Checked open today

Collected from Citi's official careers site by LitCareer. Always apply on the employer's own site; LitCareer never charges you to apply.