Symbolic Graphics Programming with Large Language Models

Introduction

Large language models (LLMs) excel at understanding and generating code, but their ability to create symbolic graphics programs (SGPs)—which produce images remains underexplored. SVGs, as a widely used SGP format, provide an ideal testbed for evaluating LLMs in this area. We address two main research questions:

(1) how well can LLMs draw using SGPs?
To address the first question, we introduce SGP-GenBench, a comprehensive benchmark for evaluating LLMs' ability to generate SGPs from three perspectives: object, scene, and composition.

(2) how can we improve their ability to generate SGPs?
To improve performance, we propose a reinforcement learning approach that uses similarity scores between visual encoder outputs and input text descriptions as reward signals. Our experiments show that RL significantly boosts the symbolic graphics programming abilities of LLMs.

SGP as Visual Synthesis

Symbolic graphics programming (SGP) aims to generate structured graphics code, such as SVG, from natural language instructions. Unlike conventional text-to-image models that produce pixel-based images, SGP translates prompts into interpretable, parametric programs, enabling precise and controllable visual synthesis.

SVG is widely used and serves as a natural bridge between language and vision, making it ideal for evaluating LLMs' ability to ground semantics from prompt to code. SGPs are both:

(a) PARAMETRIC - allowing exact geometric and appearance control
(b) PROCEDURAL - supporting hierarchical and compositional scene construction.

These properties enable fine-grained, interpretable, and flexible visual generation, and allow us to directly analyze the complexity and structure of generated scenes.

SGP-GenBench

Construction

• - Scene generation: Evaluated on COCO-val, covering 80 object categories with complex scene descriptions. We sample 1,024 images for efficient and representative assessment.
• - Object generation: Assessed on SGP-Object-val, a set of 930 single-object examples with captions, focusing on the model's ability to render individual objects accurately.
• - Compositional generation: Tested on SGP-CompBench with 3,200 prompts, targeting attribute binding, spatial relationships, and numeracy using 80 common objects for compositional evaluation.

Method

Given a text description, we sample a group of SVG codes from the model and render them as images. Each SVG code is scored by the alignment between the rendered image and the text description. The advantages are calculated based on the scores, and used for updating the model.

Discussion of Method

Distilling from vision models. We use external vision models (e.g., DINO, CLIP) to define rewards, guiding the LLM to align its generations with strong visual and semantic priors. This implicit distillation improves both visual understanding and cross-modal alignment.

Training without ground truth SGPs. Our RL method learns directly from images using feedback from vision models, removing the need for paired image-program data and enabling scalable, diverse training.

Aligning language and vision. The RL process ensures that generated SGPs are consistent with both linguistic input and visual model judgments, grounding language in visual evidence and enhancing cross-modal reasoning.

Results

Overview of SGP-GenBench

• SGP-GenBench reflects general model abilities. The ranking of models on our benchmark aligns well with their perceived general capabilities, especially in code generation. For example, Claude 3.7 Sonnet Thinking generally outperforms o3, which in turn surpasses Gemini 2.5 Pro Preview, followed by open-source systems like DeepSeek-R1 and Qwen-2.5-7B. This consistent ordering suggests that SVG generation is a reliable indicator of broader model competence.
• Closed-source models remain strongest. Frontier systems achieve the best results not only on compositional reasoning tasks such as attribute binding and numeracy, where Claude 3.7 Sonnet Thinking reaches 90.5 on color binding and 89.4 on numeracy, but also on scene and object fidelity, where Gemini 2.5 Pro Preview attains the top DINO object score of 0.653 and strong VQA scene performance of 0.554.
• Our RL-trained model substantially narrows the gap. The RL post-trained Qwen-2.5-7B raises its overall compositional score from 8.8 to 60.8, outperforming all other open-source counterparts such as DeepSeek-R1 and QwQ-32B. It also achieves the best VQA score across all models at 0.596, slightly higher than Claude 3.7 Sonnet Thinking, demonstrating that reinforcement learning enables open-source models to approach the closed-source frontier.

Object & Scene

Click any column header to sort ascending/descending. Bold indicates the best in each group.

Composition

Click any column header to sort ascending/descending. Bold indicates the best in each group.

Analysis

Reinforcement Learning vs. Best-of-N Sampling

How SGP Generation Capabilities Evolve during Training

Strategy 1: Decomposition into basic components.

Strategy 2: Contextual optional details.

Bibtex

@article{chen2025sgp,
  title={Symbolic Graphics Programming with Large Language Models},
  author={Chen, Yamei and Zhang, Haoquan and Huang, Yangyi and Qiu, Zeju and Zhang, Kaipeng and Wen, Yandong and Liu, Weiyang},
  journal={arXiv preprint arXiv:2509.01000},
  year={2025}
}