Master's Thesis Expert-in-the-Loop AgentOps/LLMOps Pipeline for Systems Engineering Process Agents

Tasks

In this thesis, an AgentOps/LLMOps pipeline will be designed and implemented to support engineering process agents with expert-in-the-loop feedback. The work addresses the challenge that current generative-AI agents in systems engineering (SE) — used for tasks such as requirements derivation, test-case generation, and subsystem decomposition — often produce inconsistent or untraceable outputs when deployed as black boxes.

The pipeline will embed domain experts into iterative feedback loops, capturing their corrections and context, and using that feedback to selectively adapt the system. The approach will operationalise both forward flow (methods/data → agent output → user feedback → iteration) and backward flow (user feedback → classification/context engineering → selective LLM adaptation/test-update), minimising unnecessary retraining and focusing on context engineering and selective adaptation.

The main tasks of the thesis include:

• Conducting a literature and state-of-the-art review on MLOps, LLMOps, and AgentOps, with focus on engineering process agents and expert-in-the-loop systems.
• Designing and implementing an open-source AgentOps/LLMOps pipeline tailored for SE use-cases.
• Empirically comparing adaptation strategies (context engineering, parameter-efficient fine-tuning, hybrid) on two SE tasks:
  (a) requirements derivation for subsystems,
  (b) test-case derivation from requirements.
• Measuring precision/recall, semantic correctness, expert corrections per iteration, and cost-benefit metrics.
• Quantifying the value of expert feedback in iterative agent improvement.

Qualification

• Solid programming skills, preferably in Python (experience with other languages such as C++/Java is a plus)
• Fundamental knowledge of machine learning / AI, ideally including large language models, prompt engineering, retrieval-augmented generation (RAG), and context engineering
• Basic understanding of software engineering concepts (APIs, data models, version control, testing) and cloud development (Azure preferred)
• Analytical skills to design experiments, measure performance metrics, and interpret results
• Ability to work independently, structure complex problems, and document results in scientific English
• Interest in operationalising AI agents and integrating expert feedback loops

Nice to have:

• Experience with context engineering, RAG, or fine-tuning LLMs.
• Interest in systems engineering processes and tools.
• German skill >B1