In an era where AI agents operate autonomously, one of the biggest concerns is “runaway behavior.” Straying from the topic, generating inappropriate content, or producing hallucinations (plausible errors)—how can we prevent these risks?NVIDIA’s answer to this challenge is patent application US 2024/0354319 A1, “Runtime alignment of language models in conversational AI systems,” which we will explore in depth in this article.
In this series so far, we’ve examined patents that protect an agent’s “runtime infrastructure,” “core,” “training,” “collaboration,” and “construction.”This article addresses a new layer distinct from those—the “safety rails (guardrails).” It is also an interesting case in that it corresponds to “NeMo Guardrails,” which NVIDIA has released as open source. A patent attorney specializing in AI intellectual property will explain the details while citing actual claims.
💡 Key Point: This article is part of the AI Agent Patent Series. For comparisons of various companies, please see the Anthropic Strategy edition or the Japan, U.S., and Europe Case Studies edition.
Table of Contents
| Item | Content |
|---|---|
| Publication Number | US 2024/0354319 A1 |
| Title of the Invention | Runtime Alignment of Language Models in Conversational AI Systems and Applications |
| Publication Date | October 24, 2024 |
| Filing Date / Priority Date | April 20, 2023 |
| Applicant | NVIDIA Corporation |
| Inventors | Razvan Dinu, Jonathan M. Cohen, Christopher M. Parisien, Traian-Eugen Rebedea |
| Number of Claims | 20 (3 independent: Claims 1, 11, and 19) |
| Status | Published Application (Under Examination) |
| Related Technology | NeMo Guardrails / Colang |
Large Language Models (LLMs) are powerful, but their outputs cannot be fully controlled. Risks such as straying from the topic, making inappropriate statements, or asserting untrue information (hallucinations) pose significant barriers to integrating AI into actual services and business operations.
The impact of runaway behavior is particularly severe in AI agents that act autonomously. “How to keep AI within safe limits” is just as important as “how to make AI smarter.” This patent describes technology that solves this latter challenge.
Conventional methods for ensuring desirable AI behavior have relied on “re-training the model itself,” such as fine-tuning or RLHF. However, these approaches are costly and time-consuming, making them unsuitable for frequent adjustments.
The approach of this patent is fundamentally different. The specification explicitly states that “output can be controlled at runtime without training or retraining the language model.” The core idea is to leave the model as is and place a control layer (guardrails) “around” it.
💡 Key Point: The design of “controlling the model from the outside without changing it” offers the advantage of quickly and flexibly making AI safer simply by changing settings. Eliminating the need for the resource-intensive process of retraining—this technical benefit is a key point supporting the claim of inventive step.
So, how is control exercised at runtime? The process described in Claim 1 consists of the following three steps.
The key point is to first convert the input into a standardized semantic representation called the “canonical form.” This allows diverse inputs—such as “Tell me the weather” and “What’s the weather like today?”—to be consolidated into a common format, enabling them to be matched against a predefined dialog flow (i.e., a script of permitted behaviors).
The specification explains that a “formal (conversational or natural language) modeling language” is used to define dialogue flows and canonical forms. This corresponds to NVIDIA’s “Colang.”
Colang is a language that allows users to describe rules such as “If the user says this, the AI behaves that way,” as well as permitted dialogue flows (flows and subflows) and structured programming syntax. Its key feature is that it enables the AI’s “code of conduct” to be written explicitly, much like code.
According to the patent specification, the guardrails function at both the ingress and egress points.
| Stages | Role |
|---|---|
| Ingress | Controls user input. Blocks inappropriate or dangerous input at the ingress point |
| Egress | Verification of model output. Blocking undesirable output at the egress |
By installing guardrails on both the “input side” and the “output side,” we ensure safety through multiple layers. This single framework allows us to address both defense against malicious inputs—such as prompt injection (input)—and the prevention of harmful or erroneous outputs (output).
US 2024/0354319 A1 | Claim 1 (Original Text / English)
A method comprising: generating, based at least on a user input, a canonical form that comprises a constrained semantic representation of the user input; determining, based at least on the canonical form, a dialog flow that controls output of a language model; and performing one or more operations to execute the dialog flow to generate an output.
Reference Translation by a Patent Attorney (Japanese)
Claim 1 is notable for being quite concise and broad, even compared to the other patents we’ve examined in this series. It is condensed into three steps: “generate a canonical form → determine a dialog flow → execute.”
💡 Key Point: Broad claims are powerful if granted, but they also carry a higher risk of rejection based on prior art. This application is pending (A1), and the scope within which this concise Claim 1 will ultimately be granted depends on the outcome of future examination.In practice, the standard approach is to balance risk and coverage by combining broad independent claims (1, 11, and 19) with dependent claims that include specific limitations.
In addition to Claim 1 (method), Claims 11 and 19 are also filed as independent claims.
If we list the patents examined in this series as “components of an AI agent,” the positioning of this patent becomes clear.
| Layer | Representative Patents (This Series) | What Is Protected |
|---|---|---|
| Learning | OpenAI VPT (US 11,887,367) | Method for Learning Operations |
| Main Unit/Execution | Anthropic (US 12,430,150, etc.) | Runtime and Agents |
| Cooperation | OpenAI/Salesforce | Multi-Agent |
| Development | OpenAI/Anthropic | Agent Creation |
| Safety (This Article) | NVIDIA (US 2024/0354319) | Guardrails and Output Control |
💡 Key Point: Not only “intelligence” but also “safety and control” are independent subjects of patent protection. As societal demands for AI governance grow, patents related to safety layers are expected to become even more important in the future. If your company possesses proprietary control and safety technologies, it is well worth considering patenting them.
What’s interesting is that NVIDIA has released “NeMo Guardrails,” the technology covered by this patent, as open source. You might wonder, “Why file a patent if it’s open source?” but this is not a contradiction.
Since Claim 1 is concise and broad, a potential point of contention is whether it will be deemed an “abstract idea (organization of information or application of rules)” under the Alice/Mayo tests. The key issue is the extent to which specific processes—such as canonical form, dialogue flow, and runtime control—will be recognized as technical implementations; the application is currently under examination.
The specification describes specific data processing steps—such as converting input into canonical form and determining and executing dialogue flows—making it a structure that readily supports a claim of patent eligibility as a software-related invention. The key factors for establishing inventive step are the effect of “eliminating the need for retraining” and the specific mechanism of the guardrails.
This configuration can be easily positioned as a technical solution to the technical problem of “safe control of LLM output” and is likely to be evaluated as a technical feature even under the COMVIK approach.
① Secure rights for “safety and control” as well. Not only the intelligence of the AI (the model) but also the technology for controlling and ensuring the safety of its output is an important subject for patent protection.
② Highlight “no need for retraining” as a technical benefit. The benefits of reduced costs and time serve as strong evidence of inventive step.
③ Specify unique description languages and formats. “Mechanisms for describing behavior,” such as Colang, provide a way to move beyond abstract concepts.
④ Open source and patents can coexist. Consider combining a dissemination strategy (OSS) with a defense strategy (patents).
⑤ Balance broad independent claims with specific dependent claims. Since broad claims pending in court carry risks, it is important to build a multi-layered defense.
Why not secure intellectual property rights for your company’s AI, including safety and control technologies?
Patent attorneys with deep expertise in the IT, software, and AI fields provide comprehensive support—from claim drafting that incorporates AI safety and governance technologies, to free assessments of patentability, ensuring compatibility with open-source strategies, and filing strategies in Japan, the U.S., and Europe.
Schedule a Free Initial Consultation IT & AI Intellectual Property ServicesQ. What kind of patent is US 2024/0354319 A1?
A. This is a U.S. patent application by NVIDIA covering “guardrail” technology that controls and secures the output of large language models (LLMs) at “runtime.” It converts user input into a “canonical form” (a constrained semantic representation) and uses that to determine and execute an “interaction flow” that controls the output.A key feature is its ability to control the behavior of an LLM without retraining it. The application was published on October 24, 2024, and is currently under examination. It corresponds to NVIDIA’s open-source “NeMo Guardrails.”
Q. What are “Guardrails”?
A. It is a mechanism that “constrains and guides” AI outputs to prevent them from becoming inappropriate, harmful, or off-topic. Just as guardrails on a road prevent vehicles from veering off course, it keeps LLM outputs within a desirable range. This patent achieves this through runtime control via dialogue flow, rather than model retraining.
Q. What does “control without retraining” mean?
A. Conventional methods for changing AI behavior (such as fine-tuning or RLHF) require retraining the model itself, which is costly and time-consuming. The method described in this patent leaves the model as-is and instead places a control layer (guardrails) “around” it to control the output at runtime. The advantage is that safety can be enhanced quickly and flexibly simply by changing settings.
Q. What is “canonical form”?
A. It is a standardized “constrained semantic representation” derived from user input (e.g., summaries, definitions of intent). By normalizing diverse phrasing into a common form, it becomes easier to match the input against predefined dialogue flows.
Q. Can AI safety and control technologies be patented?
A. Yes, they can. As demonstrated by this patent, if specific mechanisms such as canonical form, dialogue flows, and runtime control are described, patent protection can be pursued in Japan, the United States, or Europe. AI safety and governance are fields that will become increasingly important in the future and represent a promising area for securing intellectual property rights.
Important Note Regarding This Article: This article provides a general explanation of technology and legal systems based on published patent application bulletins. US 2024/0354319 A1 is a published application currently under examination and has not yet been granted. Since the claims may be amended, the final scope of protection has not yet been determined.The cited claims, abstract, and description are based on published application data (Google Patents, FreePatentsOnline, etc.); however, for legally significant purposes (FTO, infringement analysis, filing, etc.), please be sure to verify the USPTO official transcript and the latest processing information, and consult with an expert for a case-by-case review.The Japanese translation is provided for reference purposes only; the official text is the original English version.