DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning
This paper discusses a new approach to improve how models understand and reason about images by allowing them to interact with visual content directly.
Content & Liability Disclaimer
This article and its accompanying video are automated summaries derived from the original research paper by Unknown authors. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.
The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.
This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.
- 1 This gap is critical, as the core objective for advanced reasoning models is to solve challenging, real world problems.
- 2 Crucially, beginning in this phase, the VLM consumes editor-produced visual tokens rather than ground-truth visual context, and the LM objective is conditioned on the editor's outputs.
- 3 The phase-1 language modeling objective then averages over all text tokens across the corpus and sums over examples, segments, and token positions:.
- 4 Finally, we unfreeze the LLM backbone while maintaining the Phase 2 objective.
Introduction
Recent progress shows that integrating step-by-step reasoning into VLMs has substantially improved their performance on complex tasks . However, current VLMs often exhibit a “thinking over see-ing” tendency : while they can generate lengthy and seemingly coherent reasoning traces, these traces are frequently detached from the actual visual input.
To address this, OpenAI introduced a new axis for VLM reasoning with “thinking with images” .
Instead of merely generating textual reasoning traces that overlook visual content, this approach enables models to actively interact with images through an explicit mechanism.
Yet, these efforts have largely been confined to limited scenarios such as jigsaw puzzles and mazes.
While promising, these paradigms struggle with grounding noise and limited precision, impeding the generation of consistent and controllable traces (Figure 1 ).
Research Question
The phase-1 language modeling objective then averages over all text tokens across the corpus and sums over examples, segments, and token positions:. Crucially, beginning in this phase, the VLM consumes editor-produced visual tokens rather than ground-truth visual context, and the LM objective is conditioned on the editor’s outputs.
Finally, we unfreeze the LLM backbone while maintaining the Phase 2 objective.
This gap is critical, as the core objective for advanced reasoning models is to solve challenging, real world problems.
Methodology
In both cases, the core mechanism relies heavily on precise coordinate regression. Compared to the prior approach that edits images in a highly compressed latent space, our method preserves richer semantic information: the editor operates directly on visual tokens with explicit conditioning on action embeddings.
Study Design
To better understand the effect of our method, we conduct an in-depth comparison against the base model Qwen2.5-VL-7B.
When breaking down results by task category, consistent patterns emerge: the most reliable gains appear in tasks involving geometry and counting, with particularly striking improvement on MathVision reaching 5.3 points.
How PDFdigest Helps You Understand Research
Instant Paper Analysis
Get structured summaries and key findings from dense PDFs in seconds.
Visual Explanations
Turn complex methods, figures, and results into clearer visual breakdowns.
AI-Powered Q&A
Ask focused questions and get answers grounded in the paper.
Results & Findings
In many cases, the models misinterpret critical details in the image or even hallucinate content that is not present , suggesting that their reasoning is driven more by linguistic priors than by genuine visual perception . By zooming, cropping, and performing image-level manipulations, VLMs are encouraged to ground their reasoning in actual visual evidence.
- In many cases, the models misinterpret critical details in the image or even hallucinate content that is not present , suggesting that their reasoning is driven.
- By zooming, cropping, and performing image-level manipulations, VLMs are encouraged to ground their reasoning in actual visual evidence.
- This paradigm represents a shift from “thinking over seeing” to “thinking through seeing,” enabling models to analyze visual information more deeply, more thoroughly, and ultimately achieve.
- Following such an idea, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception and reasoning.
- VILASR defines a closed set of drawing operations and trains the model to decide when to invoke each of them.
Despite their differences, these approaches share a common limitation: the supported action space remains relatively restricted, and they inevitably rely on accurate spatial grounding, which remains challenging: curated data seldom yield perfectly accurate annotations, and end-to-end reinforcement learning rollouts are.
To overcome the constraints of a limited action space and to expand the model’s “thinking space,” another line of work makes a conceptual leap from execution to imagination, aiming to unify gen-.
Practical Applications
While such data may seem scarce, prior work has demonstrated its efficacy in enhancing VLM perception and reasoning across structured domains and beyond . It is worth noting that while programmatic edits serve as a strong reference, they are illustrative rather than absolute ground truth, as natural language instructions may admit multiple valid implementations.
First, the dataset is generated exclusively from code, which may limit the approach’s applicability to broader, open-world domains.
Code Editor
The Code Editor section presents a question regarding finding overlap areas and provides a source code snippet related to visual question answering.
Solver
The Solver section outlines the steps required to ensure accuracy in visual tasks, including the use of tools for zooming in on specific areas.
Frequently Asked Questions
Crucially, beginning in this phase, the VLM consumes editor-produced visual tokens rather than ground-truth visual context, and the LM objective is conditioned on the editor’s outputs. This gap is critical, as the core objective for advanced reasoning models is to solve challenging.
It is worth noting that img2code is an extremely challenging task, as it requires the model to faithfully capture all fine-grained details in an image using programmatic language. This low success rate is attributable to the task’s sensitivity; even a minor error.
By zooming, cropping, and performing image-level manipulations, VLMs are encouraged to ground their reasoning in actual visual evidence. Following such an idea, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception.
Therefore, the final overlapping area of the two circles is about 17.65 square meters. NONE 17.65 . First, the dataset is generated exclusively from code, which may limit the approach’s applicability to broader, open-world domains.
Despite their differences, these approaches share a common limitation: the supported action space remains relatively restricted, and they inevitably rely on accurate spatial grounding, which remains challenging: curated data seldom yield perfectly accurate annotations, and end-to-end reinforcement learning rollouts are similarly error-prone.
This paper discusses a new approach to improve how models understand and reason about images by allowing them to interact with visual content directly.