LLMOps Engineer

About the role

The LLMOps Engineer will join Project A, ALX’s AI learning platform, to turn streams of LLM data into reliable assessments of product performance. Working with Anthropic engineers and a cross‑functional AI team, you will design and maintain evaluation frameworks that ensure the platform delivers effective learning experiences.

Key responsibilities

• Build and run evaluation suites for each LLM product, covering distinct failure modes and ensuring regression testing on every release.
• Maintain cost‑effective evaluation pipelines as the product portfolio expands.
• Own the reporting loop, presenting findings from evaluations and platform data to the team and stakeholders and flagging unintended model behaviour.
• Steward core datasets, including classified customer‑support data, and collaborate with the AI Product Manager on instrumentation.

Required profile

• Junior‑to‑mid level engineer eager to grow into full ownership of the LLMOps function.
• Ability to work closely with Anthropic engineers and a cross‑functional team of AI engineers, product managers and data scientists.
• Focus on measurement and evaluation rather than model hosting or serving.

Required skills

• Solid proficiency in Python.
• Strong data‑analysis mindset, comfortable shaping and analysing datasets.