In a nutshell, OpenAI’s approach, as outlined in this patent, is to “have the AI learn by watching a large number of videos of humans performing these operations.” Moreover, it employs a sophisticated mechanism that automatically labels unlabeled videos using a proprietary method. This is a contrasting example that pursues the same goal via a path entirely different from Anthropic’s “Computer Use” patents.A patent attorney specializing in AI intellectual property will explain this while citing the actual claims.
💡 Key Point: This article is part of the AI Agent Patent Series. Anthropic’s contrasting approach is explained in the Data Flow Patent and the Core Computer Use Patent.
Table of Contents
| Item | Content |
|---|---|
| Patent Number | US 11,887,367 B1 |
| Title of the Invention | Using machine learning to train and use a model to perform automatic interface actions based on video and input datasets |
| Registration Date | January 30, 2024 |
| Filing Date / Priority Date | April 19, 2023 |
| Applicant | OpenAI OpCo, LLC |
| Inventors | Bowen Baker, Jeffrey Clune, et al. (9 in total) |
| Number of Claims | 20 (3 independent claims: Claims 1, 13, and 20) |
| Common Name | VPT (Video PreTraining) |
| Status | Granted Patent |
The most straightforward way to teach an AI screen operations is to provide it with a large amount of correct data (labeled data)—such as “Press this key when this screen appears”—and have it learn from that. However, manually annotating videos to indicate “when, which key, and which click was made” incurs enormous costs.
On the other hand, there are countless videos on the internet showing people operating software and games. The problem is that these videos lack operation labels. The technical challenge addressed by this patent is how to utilize this “massive volume of unlabeled videos.”
At the core of this patent’s solution is the Inverse Dynamics Model (IDM).
The processing flow is as follows.
💡 Key Point: The key is to use the IDM—created from a “small amount of ground-truth data”—as a “labeling machine” to instantly convert a massive amount of unlabeled video into training data. This dramatically reduces the cost of manual annotation and paves the way for training computer-based agents at scale.
Claim 1 explicitly states that the IDM’s predictions are “based on a non-causal combination of past and future information.” This is the technical highlight.
In reality, the “policy model” that the AI uses to make real-time decisions can only see the past (the future has not yet occurred = causal). However, since IDM is a post-processing step for labeling, it can reference both the frames before and after a given point in time—including future frames.
💡 Key Point: Thanks to the condition that it “can look into the future,” IDM can predict “which action was performed at that moment” with far greater accuracy.For example, based on future information such as “the cursor moved to the right in the next frame,” it can estimate with high accuracy that “the mouse was just moved to the right.” Utilizing this “non-causality” for labeling is the intellectual core of this invention.
According to the specification, the actions generated by the model (automatic interface actions) include keystrokes, button presses, touchscreen inputs, joystick movements, mouse clicks, scrolling, and mouse movements. Application areas cited include video games, various processing applications, web browsers, spreadsheets, and file explorers.
In other words, the goal is to enable AI to perform any operation a human would perform on a GUI without human intervention. This is precisely the foundational technology for “computer use” agents.
US 11,887,367 B1 | Claim 1 (Original Text / English)
A method for training a machine learning model to perform automated actions, comprising: receiving unlabeled digital video data; generating pseudo-labels for the unlabeled digital video data, wherein the generating comprises: receiving labeled digital video data; training a first machine learning model including an inverse dynamics model (IDM) using the labeled digital video data; and generating at least one pseudo-label for the unlabeled digital video data, wherein: the at least one pseudo-label is based on a prediction, generated by the IDM, of one or more actions that mimic at least one timestep of the unlabeled digital video data, and the prediction of the one or more actions is generated based on a non-causal combination of past information and future information within the unlabeled digital video data, the past and future information being relative to one or more reference frames within the unlabeled digital video data; adding the at least one pseudo-label to the unlabeled digital video data to form pseudo-labeled digital video data; and further training the first machine learning model or a second machine learning model using the pseudo-labeled digital video data to generate at least one additional pseudo-label for the unlabeled digital video.
Reference Translation by a Patent Attorney (Japanese)
| Claims | Technical Meaning | Reason for Inclusion |
|---|---|---|
| IDM (Inverse Dynamics Model) | Inferring User Actions from Previous and Next Screens | Moving Beyond Abstract Ideas to Concrete Algorithms |
| Non-causal (past + future) prediction | Referencing Future Frames to Improve Accuracy | The Core of the Invention’s Originality and Inventive Step |
| Pseudo-labeling | Converting Unlabeled Videos into Training Data | Solution to the Technical Challenge (Annotation Cost) |
| Further Training | Training the Operational Model Using Pseudo-Data | A Complete Processing Workflow as a Learning Method |
This patent consists of a total of 20 claims and follows the standard structure for software patents, with independent claims organized into three categories: method (Claim 1), system (Claim 13), and non-transitory computer-readable medium (Claim 20).This structure covers different potential infringers: those who implement the method, those who manufacture or use the device, and those who distribute the program.
Although both OpenAI and Anthropic share the same goal of “having AI operate a PC,” they have obtained patents using different technical approaches. A comparison of the patents from both companies, as examined in this series, clearly highlights their differing strategies.
| OpenAI (this article) | Anthropic (Previously Discussed) | |
|---|---|---|
| Representative Patents | US 11,887,367 B1 (VPT) | US 12,430,150 B1, among others |
| Subject of Protection | Training Method for Learning Operations from Videos | Agent Execution Framework, DSL, and Data Flow |
| Approach | Large-scale unlabeled video data + automatic labeling via IDM | Runtime Architecture and Dedicated Language |
| Technical Origins | In-house Research (VPT) | Acquisition of Adept |
| Analogy | Mastering “How to Teach (Learning Methods)” | Mastering “How to Operate (Execution System)” |
💡 Key Point: OpenAI focuses on patenting “how to train (training methods and data),” while Anthropic focuses on “how to operate (execution architecture).”This is a prime example of how companies in the same product sector secure patents based on their respective areas of strength. When filing your own patent applications, it’s crucial to determine “which layer to compete in.”
Since machine learning training methods can be viewed as abstract mathematical techniques, the Alice/Mayo test becomes a key issue.Since this case involves specific algorithms—such as IDM, non-causal prediction, and pseudo-labeling—along with specific actions like keystrokes and mouse operations, it is easier to argue that it provides a “concrete solution to a technical problem,” and the patent is currently registered.
In Japan, as well, training methods and pre-trained models are patentable. However, according to the Japan Patent Office’s AI case law, “simply replacing human work with AI” is deemed to lack inventive step.This case features a clear technical innovation—"utilizing non-causality for labeling"—and a clear benefit—"a dramatic reduction in annotation costs," making it easy to argue for inventive step.
While mathematical methods themselves are excluded, this application clearly demonstrates a technical contribution to the technical challenge of “efficiently converting unlabeled video into training data,” and its structure makes it easy to evaluate as a technical feature even under the COMVIK approach.
① Patent the training methods and data creation processes. Not only the model itself, but also “how to train it” and “how to create training data” can form the basis of strong patents.
② Highlight your unique technical ingenuity. Place specific ideas that effectively solve problems—such as “the use of non-causal prediction”—at the center of your claims.
③ Move away from abstract mathematics. Describe specific algorithms and specific outputs (such as mouse and keyboard operations) to ensure patent eligibility.
④ Compete at the level where your company excels. Even if you share the same goal as competitors, it’s an effective strategy to secure rights at the level where your company excels (e.g., learning methods, execution systems, data, etc.).
Why not secure patent rights for your company’s AI, including its learning methods and data?
Patent attorneys with deep expertise in the IT, software, and AI fields provide comprehensive support—from claim drafting that includes learning methods, data, and models, to free assessments of patentability, and application strategies in Japan, the U.S., and Europe.
Schedule a Free Initial Consultation IT & AI Intellectual Property ServicesQ. What kind of patent is US 11,887,367 B1?
A. It is a U.S. patent registered by OpenAI that protects a method for training AI to perform “automated screen operations.”It is a technology that uses an “Inverse Dynamics Model (IDM)”—trained on a small amount of labeled data—to automatically label (pseudo-label) vast amounts of unlabeled video (such as footage of people operating software), and then trains an operation model using that large dataset.It corresponds to the method OpenAI calls “VPT (Video PreTraining)” and was registered on January 30, 2024.
Q. What is an “Inverse Dynamics Model (IDM)”?
A. It is a model that estimates “what actions were performed” during a given time interval based on the video frames (observations) before and after that point. Whereas conventional AI (policy models) predict “what to do next,” IDM estimates “what was done” retrospectively.Because it can reference not only past frames but also future frames, it achieves high accuracy in estimating user actions and can generate large volumes of labels at low cost.
Q. Why is “learning from videos” important?
A. There are countless videos on the internet showing people operating software and games. However, these videos do not include action labels indicating “when and which keys were pressed.”This invention has paved the way for training computer-use agents cost-effectively and at scale by training IDM on a small amount of labeled data and automatically labeling vast amounts of unlabeled video.
Q. How is this different from Anthropic’s “computer use” patent?
A. The approaches are fundamentally different. Anthropic (originating from Adept) protects the “agent execution framework, DSL, and data flow.”In contrast, OpenAI’s patent protects a “training method for teaching an agent to perform operations from video.” It is interesting to note that, despite addressing the same challenge of “having AI operate a PC,” the two companies have obtained patents based on different technical approaches.
Q. Can a learning method (training method) also be patented?
A. Yes, they can. As demonstrated by this patent, if you clearly describe specific algorithms (IDM, non-causal prediction, pseudo-labeling) and the technical effects they produce (low-cost, large-scale labeling), it is possible to obtain patent protection in Japan, the U.S., or Europe.Data creation methods and training methods are the source of AI’s competitive advantage and are important subjects for patent protection.
Note regarding this article: This article provides a general explanation of the technology and system based on published patent applications. Although US 11,887,367 B1 is a registered patent, the actual scope of protection is determined by the wording of each claim, the doctrine of equivalents, and historical information.The cited claims, abstract, and specification are based on published patent bulletin data (such as FreePatentsOnline); however, for legally significant purposes (FTO, infringement analysis, invalidation, patent applications, etc.), please be sure to verify the USPTO official transcript and the latest prosecution history, and consult a specialist for an individual review.The Japanese translation is provided for reference purposes only; the official text is the original English version.