LLM-in-the-Loop Robot Control: Failure Modes and Containment
How to keep a language model from driving a robot into harm: typed action plans, validators, an independent safety controller, approval gates and test evidence.
- PUBLISHED
- November 15, 2026
- READ TIME
- 8 MIN
- AUTHOR
- ONE FREQUENCY
- Topic
- robotics, physical-ai, ai
- Published
- November 15, 2026
- Read time
- 8 min
- Word count
- 1,462
Short answer: A large language model should advise a robot, not drive it. Let the model interpret a request and propose a plan, then pass every proposed action through a deterministic validator and an independent safety controller before anything moves. The December 2025 joint guidance from CISA, NSA, the FBI and allied agencies on AI in operational technology says LLMs almost certainly should not be used to make safety decisions in OT environments. A robot is a machine in the physical world, so the same reasoning applies.
This article is an engineering approach for teams adding language or vision-language models to robots, autonomous vehicles and industrial machines. It builds on our physical AI and robotics guardrails playbook. One Frequency does not claim robotics hardware delivery; we describe how to bound the software and control path.
What can go wrong when an LLM controls a robot?
Five failure modes matter most:
- Hallucinated or ungrounded actions. The model produces a plausible command that does not match the real scene, for example a target that is not there. The CISA guidance warns that AI can fabricate a plausible but false response and give operators incorrect information.
- Out-of-envelope commands. The model asks for a speed, force, reach or path that the hardware or the workspace cannot safely allow.
- Prompt injection through the environment. Text on a label, a document handed to the robot, a web page or a spoken instruction can carry hostile instructions. NIST lists direct and indirect prompt injection as a core risk for agentic systems in its 2026 concept paper on agent identity.
- Latency and non-determinism. LLMs are built with randomness, so the same input may not produce the same output, and inference time varies. A control loop that waits on a model can stall or act on stale data.
- Drift after change. A model or prompt update changes behavior without any change to the hardware. The CISA guidance calls out increased uncertainty over time and tighter coupling of actions as sources of cascading failure.
Where should the model sit in the control stack?
Put the model in the slow, supervisory layer. A common split has three layers:
- Reasoning layer (seconds): the LLM or vision-language model turns a goal into a structured plan. Its output is a typed object, not free text: an action name from an allowed list, bounded numeric parameters, and a stated reason.
- Validation layer (milliseconds): deterministic code checks the plan against the workspace map, kinematic limits, force and speed limits, the operating design domain and the current permissions. It rejects, clamps or asks a human.
- Safety layer (hardware-rate): safety-rated stops, speed and separation monitoring and protective zones run in an independent controller that the model cannot write to. The CISA guidance makes the same point in OT terms: keep failsafe mechanisms that revert to traditional automation or manual operation for any AI-enabled process.
OSHA states there is no specific robotics standard; it points to general machine-safety rules such as lockout/tagout (29 CFR 1910.147) and machine guarding (29 CFR 1910.212) and to consensus standards such as ANSI/RIA R15.06. Your safety layer is where those requirements are met, so the model must not be able to bypass it.
How do you contain a bad command?
Containment is a set of controls, each of which should work if the others fail:
- Allow-listed action schema. Reject any output that does not parse into an approved action type. Never execute generated code or raw motor commands from a model.
- Parameter clamps. Cap speed, acceleration, force and reach in the validator and again in the safety controller.
- Workspace and object checks. Confirm the target exists in perception from a second source before acting. Treat a mismatch as a stop condition.
- Rate and scope limits. Limit how many actions the model may issue per minute and which cells, tools or people-zones it may touch.
- Approval gates. Require a human confirmation for actions that are irreversible, near people, or outside a validated routine. The CISA guidance recommends human-in-the-loop intervention points, including when AI is actively updating control logic, and cautions that problems with active AI systems can escalate before operators become aware.
- Fail-safe default. On loss of communication, a validation failure or a model timeout, the machine holds or enters a safe state. It does not continue the last plan.
How should you secure the path between the model and the machine?
Treat the model host as an untrusted component. The CISA guidance recommends push-based or brokered architectures that move data or summaries out of the OT network without giving persistent inbound access, and one-way transfer patterns where data must cross to business networks. For a robot, that means the model service should not hold standing credentials to the control network. Give it a narrow, authenticated interface that accepts only the typed plan, log every call, and rotate keys. NIST's NCCoE concept paper on software and AI agent identity raises the same questions: how an agent proves its authority to perform an action, and how to bind agent identity to a human authorization. Our guide to AI agent identity, audit logs and approval gates covers that side in detail.
What evidence shows a language-model robot is safe enough?
Start from hazards, not from model accuracy. For each hazard (collision, pinch, drop, unintended motion, loss of link) write a pass/fail test. Then test three ways:
- Adversarial prompts and scenes. Include injected instructions, misleading labels, ambiguous requests and sensor dropouts.
- Simulation at scale. Replay thousands of randomized scenarios to find rare failures cheaply. Record seeds and model hashes so any run can be reproduced.
- Hardware spot checks. Repeat a subset on the real machine to measure the gap between simulation and the field. NIST's response-robot program developed more than 50 standard test methods, and its approach of repeatable, measured tests is a good template for acceptance criteria.
Report worst cases and the conditions that triggered them, not only an average success rate. Re-run the whole suite whenever the model, prompt, tool list or firmware changes. The NIST AI Risk Management Framework (Govern, Map, Measure, Manage) gives a structure for documenting that work, and the Generative AI Profile (NIST AI 600-1) adds risks specific to generative models.
What should you log?
Log the user request, the model and prompt version, the plan as parsed, each validator decision with the rule that fired, safety-layer interventions, operator overrides and stop events. Keep logs long enough to reproduce an incident. The CISA guidance also recommends inventorying AI components and logging inputs and outputs, and keeping a known good state so you know when to restore from backup.
How One Frequency approaches this
One Frequency operates more than 200 production AI agents with orchestration, observability and guardrails, and has done NIST AI RMF governance work. Our founder led engineering and the FedRAMP High effort for a zero-trust secure remote access platform for operational technology and industrial control systems at a previous company. We apply that experience to action schemas, validators, approval gates, logging and the security of the control path. We do not claim robotics hardware delivery, and we say so when scoping. See our AI services and cybersecurity and compliance pages, or contact us.
FAQ
Q: Can an LLM safely control a robot directly? A: Not on its own. Use the model for interpretation and planning, and put a deterministic validator and an independent safety controller between it and the actuators. Joint government guidance on AI in operational technology says LLMs almost certainly should not make safety decisions.
Q: What is the safest way to connect an LLM to a machine? A: Have the model emit a typed plan from an allow-listed set of actions, validate every parameter against limits, and send only validated commands through a narrow, authenticated, logged interface. Keep standing credentials to the control network away from the model host.
Q: How do you defend a robot against prompt injection? A: Treat all text and imagery the robot reads as untrusted, restrict the action set so injected text cannot request anything dangerous, require approval for high-impact actions, and test with adversarial prompts. No filter is enough alone, so rely on layered limits.
Q: What should the robot do if the model fails or times out? A: Hold position or enter a defined safe state and alert an operator. Do not continue the last plan. Design the fallback to traditional automation or manual operation before deployment and test it.
Q: Does One Frequency build robots? A: No claim is made of robotics hardware delivery. One Frequency offers AI agent engineering, guardrail design, validation planning and cybersecurity for the software and control path around physical systems.
Cited and consulted.
- 01CISA, NSA, FBI and partners: Principles for the Secure Integration of AI in Operational Technology (Dec 2025)cisa.gov · accessed Nov 15, 2026
- 02NIST NCCoE concept paper: Software and AI Agent Identity and Authorization (Feb 2026)nccoe.nist.gov · accessed Nov 15, 2026
- 03OSHA: Robotics, Standards (no specific robotics standards; applicable standards)osha.gov · accessed Nov 15, 2026
- 04NIST AI 100-1: AI Risk Management Framework 1.0nvlpubs.nist.gov · accessed Nov 15, 2026
- 05NIST: AI Risk Management Framework and Generative AI Profile (AI 600-1)nist.gov · accessed Nov 15, 2026
- 06NIST: Performance metrics and test methods for response robotsnist.gov · accessed Nov 15, 2026
Ready to ship the next outcome?
One Frequency Consulting brings 25+ years of technology leadership and military discipline to every engagement. First call is operator-grade scoping — sixty minutes, no charge.