Mohamed bin Zayed University of Artificial Intelligence
Machine Learning Engineer — Reinforcement Learning
$150,000 to $450,000 a year
About the job
About the Institute of Foundation Models
We are a dedicated research lab for building, understanding, using,
and risk-managing foundation models. Our mandate is to advance
research, nurture the next generation of AI builders, and drive
transformative contributions to a knowledge-driven economy.
As part of our team, you’ll have the opportunity to work on the
core of cutting-edge foundation model training, alongside
world-class researchers, data scientists, and engineers, tackling
the most fundamental and impactful challenges in AI
development. You will participate in the development of
groundbreaking AI solutions that have the potential to reshape
entire industries. Strategic and innovative problem-solving skills
will be instrumental in establishing MBZUAI as a global hub for
high-performance computing in deep learning, driving impactful
discoveries that inspire the next generation of AI pioneers.
The Role
We’re looking for an RL infrastructure engineer to help extend and
scale our end-to-end RL training systems. You’ll work side-by-side
with world-class researchers and engineers to:
• Extend distributed training frameworks (e.g., DeepSpeed, FSDP,
FairScale, Horovod)
• Integrate rollout generation, reward computation, trajectory
processing, policy updates, and weight synchronization
• Build robust config + launch systems across multi-node,
multi-GPU clusters
• Own experiment tracking, metrics logging, and job monitoring
for external visibility
• Improve training system reliability, maintainability, and
performance
Hands-on experience developing end-to-end RL infrastructure is
required. Strong infrastructure and systems experience is what we
value most.
Key Responsibilities
• Distributed Framework Ownership – Extend or modify training
frameworks (e.g., DeepSpeed, FSDP) to support new use cases and
architectures.
• Training & Inference Integration – Connect training workers
with rollout generation engines (e.g., vLLM, SGLang), including
trajectory exchange and policy-weight synchronization.
• Launch Config & Debugging – Create and debug multi-node
launch scripts with flexible batch sizes, parallelism strategies,
and hardware targets.
• Metrics & Monitoring – Build systems for experiment
tracking, job monitoring, and logging usable by collaborators and
researchers.
• Infra Engineering – Write production-quality code and tests for
ML infra in PyTorch or JAX; ensure reliability and maintainability
at scale.
• RL Pipeline Development – Implement reward/verifier integration
and trajectory processing, and validate log probabilities, token
masks, and loss inputs with researchers.
• RL Execution & Recovery – Coordinate rollout and training
workers, including checkpoint/restart and failure recovery; track
policy versions and sample staleness when using asynchronous
execution.
Qualifications
Must-Haves:
• 5+ years of experience in ML systems, infra, or distributed
training
• Experience modifying distributed ML frameworks (e.g.,
DeepSpeed, FSDP, FairScale, Horovod)
• Strong software engineering fundamentals (Python, systems
design, testing)
• Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and
debugging skills (e.g., NCCL/GLOO)
• Ability to implement algorithms across GPUs/nodes based on
mathematical specs
• Experience working on an ML platform/ infrastructure, and/or
distributed inference optimization team
• Experience with large-scale machine learning workloads (strong
ML fundamentals)
• Hands-on experience developing an LLM RL pipeline, with
ownership across rollout/inference and distributed training
integration.
• Working knowledge of policy optimization methods such as PPO or
GRPO, sufficient to implement and debug sampling, log
probabilities, and policy updates.
Nice-to-Haves:
• Exposure to mixed-precision training (e.g., bf16, fp8) with
accuracy validation
• Familiarity with performance profiling, kernel fusion, or
memory optimization
• Open-source contributions or published research (MLSys, ICML,
NeurIPS)
• CUDA or Triton kernel experience
• Experience with large-scale pre-training
• Experience building custom training pipelines at scale and
modifying them for custom needs
• Deep familiarity with training infrastructure and performance
tuning
• Experience with asynchronous RL, multi-turn or tool-using
rollout environments, or custom reward/verifier systems.
$150,000 - $450,000 a year
Compensation
$150,000 – $450,000/year + discretionary performance bonus +
benefits
The posted salary range represents the company’s good faith
estimate of the base compensation for this position upon hire. The
actual base compensation offered may vary within this range
depending on individual qualifications, including but not limited
to relevant skills, experience, education, certifications, and
specific business needs. In addition to base salary, total
compensation includes an annual discretionary performance bonus and
comprehensive benefits.
Benefits Include
• Comprehensive medical, dental, and vision benefits
• Bonus
• 401K Plan
• Generous paid time off, sick leave and holidays
• Paid Parental Leave
• Employee Assistance Program
• Life insurance and disability
We may use artificial intelligence (AI) tools to support parts of
the hiring process, such as reviewing applications, analyzing
resumes, or assessing responses and identifying potential
inconsistencies or verification signals in application materials
based on available information. These tools assist our recruitment
team but do not replace human judgment. Final hiring decisions are
ultimately made by humans. If you would like more information about
how your data is processed, please contact us.
About the Institute of Foundation Models
We are a dedicated research lab for building, understanding, using,
and risk-managing foundation models. Our mandate is to advance
research, nurture the next generation of AI builders, and drive
transformative contributions to a knowledge-driven economy.
As part of our team, you’ll have the opportunity to work on the
core of cutting-edge foundation model training, alongside
world-class researchers, data scientists, and engineers, tackling
the most fundamental and impactful challenges in AI
development. You will participate in the development of
groundbreaking AI solutions that have the potential to reshape
entire industries. Strategic and innovative problem-solving skills
will be instrumental in establishing MBZUAI as a global hub for
high-performance computing in deep learning, driving impactful
discoveries that inspire the next generation of AI pioneers.
The Role
We’re looking for an RL infrastructure engineer to help extend and
scale our end-to-end RL training systems. You’ll work side-by-side
with world-class researchers and engineers to:
• Extend distributed training frameworks (e.g., DeepSpeed, FSDP,
FairScale, Horovod)
• Integrate rollout generation, reward computation, trajectory
processing, policy updates, and weight synchronization
• Build robust config + launch systems across multi-node,
multi-GPU clusters
• Own experiment tracking, metrics logging, and job monitoring
for external visibility
• Improve training system reliability, maintainability, and
performance
Hands-on experience developing end-to-end RL infrastructure is
required. Strong infrastructure and systems experience is what we
value most.
Key Responsibilities
• Distributed Framework Ownership – Extend or modify training
frameworks (e.g., DeepSpeed, FSDP) to support new use cases and
architectures.
• Training & Inference Integration – Connect training workers
with rollout generation engines (e.g., vLLM, SGLang), including
trajectory exchange and policy-weight synchronization.
• Launch Config & Debugging – Create and debug multi-node
launch scripts with flexible batch sizes, parallelism strategies,
and hardware targets.
• Metrics & Monitoring – Build systems for experiment
tracking, job monitoring, and logging usable by collaborators and
researchers.
• Infra Engineering – Write production-quality code and tests for
ML infra in PyTorch or JAX; ensure reliability and maintainability
at scale.
• RL Pipeline Development – Implement reward/verifier integration
and trajectory processing, and validate log probabilities, token
masks, and loss inputs with researchers.
• RL Execution & Recovery – Coordinate rollout and training
workers, including checkpoint/restart and failure recovery; track
policy versions and sample staleness when using asynchronous
execution.
Qualifications
Must-Haves:
• 5+ years of experience in ML systems, infra, or distributed
training
• Experience modifying distributed ML frameworks (e.g.,
DeepSpeed, FSDP, FairScale, Horovod)
• Strong software engineering fundamentals (Python, systems
design, testing)
• Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and
debugging skills (e.g., NCCL/GLOO)
• Ability to implement algorithms across GPUs/nodes based on
mathematical specs
• Experience working on an ML platform/ infrastructure, and/or
distributed inference optimization team
• Experience with large-scale machine learning workloads (strong
ML fundamentals)
• Hands-on experience developing an LLM RL pipeline, with
ownership across rollout/inference and distributed training
integration.
• Working knowledge of policy optimization methods such as PPO or
GRPO, sufficient to implement and debug sampling, log
probabilities, and policy updates.
Nice-to-Haves:
• Exposure to mixed-precision training (e.g., bf16, fp8) with
accuracy validation
• Familiarity with performance profiling, kernel fusion, or
memory optimization
• Open-source contributions or published research (MLSys, ICML,
NeurIPS)
• CUDA or Triton kernel experience
• Experience with large-scale pre-training
• Experience building custom training pipelines at scale and
modifying them for custom needs
• Deep familiarity with training infrastructure and performance
tuning
• Experience with asynchronous RL, multi-turn or tool-using
rollout environments, or custom reward/verifier systems.
$150,000 - $450,000 a year
Compensation
$150,000 – $450,000/year + discretionary performance bonus +
benefits
The posted salary range represents the company’s good faith
estimate of the base compensation for this position upon hire. The
actual base compensation offered may vary within this range
depending on individual qualifications, including but not limited
to relevant skills, experience, education, certifications, and
specific business needs. In addition to base salary, total
compensation includes an annual discretionary performance bonus and
comprehensive benefits.
Benefits Include
• Comprehensive medical, dental, and vision benefits
• Bonus
• 401K Plan
• Generous paid time off, sick leave and holidays
• Paid Parental Leave
• Employee Assistance Program
• Life insurance and disability
We may use artificial intelligence (AI) tools to support parts of
the hiring process, such as reviewing applications, analyzing
resumes, or assessing responses and identifying potential
inconsistencies or verification signals in application materials
based on available information. These tools assist our recruitment
team but do not replace human judgment. Final hiring decisions are
ultimately made by humans. If you would like more information about
how your data is processed, please contact us.