Recording someone at a computer produces plenty of data and very little explanation. The sequence captures clicks, keystrokes, windows, and screen changes, but it does not reliably say where one task ends, why an action was necessary, or which steps form an intentional loop. The problem becomes harder when a person alternates between writing, searching, repairing an error, and answering an unrelated request.
A new paper on task model induction tries to bridge that gap. Instead of summarizing a session as one continuous list of steps, the method separates interleaved activities, infers the objectives they serve, and reconstructs the procedure that organized them. Its aim is not merely to describe what happened. It is to produce a representation that a person can audit and another agent can reuse.
This distinction matters because a demonstration is not an instruction. A recording contains the path that was actually taken, including detours and failed attempts. A useful instruction must explain what should be achieved and how to recognize sequences, repetitions, and stopping conditions.
From raw events to recognizable activity
The method is called Task Model Induction, or TMI. It starts with screenshots and mouse or keyboard events. A vision-language model compares the screen before and after each event to describe what changed. Events are grouped into semantic actions and then into activities associated with a local objective.
A click at a coordinate carries no intent on its own. Visual context, the artifact that changed, and adjacent events can turn it into something recognizable, such as opening a file, revising a form, or validating a change. The system moves from interface signals to units of work that can be compared.
The next stage does not assume that the entire session belongs to one workflow. Each activity joins the closest latent task or starts a new one. Identifiers for files, applications, and entities help recover the same work after an interruption or a window switch. A final consolidation pass attempts to merge fragments pursuing the same objective.
This is different from a linear summary. If someone repairs a chart, answers a message, and returns to the chart, chronology blends two kinds of work. TMI tries to preserve two separate sequences even when they occupy alternating spans of the recording.
Knowing why and knowing in what order
The paper’s central move is to construct two complementary models.
The first is an objective model. It decomposes the main purpose into subgoals while keeping every leaf grounded in an observed activity. The second is a procedure model. It represents control flow through sequence and observable forms of repetition, such as performing an action for each item or repeating a cycle until a condition is met.
Keeping these views separate avoids two familiar mistakes. A goal tree may explain why the work happened while missing an operational loop. A procedure may preserve the order of actions while flattening a shift between different objectives. The method builds both structures independently and then reconciles them, moving boundaries until goal and control flow provide a compatible account.
Consider a person creating accounts for several participants, testing each login, and correcting configuration until access works. The objective is to provision dependable access. The procedure contains a loop over accounts and another cycle of repair until validation passes. A recording contains the actions; a task model tries to expose the grammar underneath them.
Promising numbers from a controlled test
The authors evaluate the method with 38 human work sessions spanning 15 tasks in five professional domains, as well as trajectories from coding agents. To test whether interleaved tasks could be separated, they cut originally continuous sessions and combined the segments synthetically. In that setting, grouping reached 0.974 on the Adjusted Rand Index, a measure of agreement corrected for chance.
In the structural evaluation, step descriptions reached 74.9% accuracy under one judge, compared with 30.3% for the workflow-summary baseline. When the induced models were used to generate reusable skills, held-out task accuracy rose from 14.29% to 18.57% over the strongest baseline, a 30% relative gain.
Those results support a narrow conclusion: representing objectives and procedure together was more useful than giving the agent either a raw demonstration or a phase-based summary. They do not show that the method can already understand any office routine. The main interleaving test recombines sessions that were originally recorded one at a time, and part of the structural assessment relies on other language models as judges. The study establishes a direction and measures its components; it does not establish a universal deployment-ready system.
Operational knowledge can carry secrets
The most important boundary appears in the paper’s own privacy section. Screenshots and keyboard events may contain personally identifiable information, credentials, private messages, customer data, and internal details. If that material becomes a skill distributed to other agents, sensitive information may survive even after the original recording is gone.
The authors recommend redacting private content before induction, but leave the effect of that redaction on quality to future work. This is not peripheral. An identifier that helps the system recognize one task across several applications may be exactly what a privacy policy requires it to remove.
For a team, the practical consequence is to treat task-model creation as a new stage of data processing. Consent, minimization, retention, access control, and review of the induced artifact must come before the result becomes documentation or an agent capability. Learning how work happens does not automatically grant permission to preserve everything that appeared while it happened.
Observation and teaching need an engineering layer between them
Computer-use agents can learn a great deal from real demonstrations. The paper shows why storing the trajectory is not enough: a trace mixes tasks, preserves accidental execution details, and leaves the relationship between goals and procedures implicit. Turning it into reusable knowledge requires segmentation, inference, formal structure, and reconciliation.
It also requires deliberate forgetting. The best task model is not the one that remembers every pixel. It is the one that preserves the necessary logic and discards what should never become an instruction. A screen records steps. Teaching the work begins when we can separate the intention worth reusing from the information that needs to disappear.