LAMP: Language-Assisted Motion Planning for Controllable Video Generation

Prompt

A large ship is slowly sailing to the right across a calm, open ocean under a soft golden sunset sky. The camera moves in a smooth counterclockwise quarter arc around the ship, gradually pulling away while keeping the vessel in focus. Gentle waves ripple around the hull, and the sunlight reflects off the water’s surface, creating a warm, cinematic shimmer. As the camera retreats, the vastness of the ocean and the ship’s steady motion evoke a sense of tranquility and scale.

Generated Trajectory

Generated Video

Abstract

Video generation has achieved remarkable progress in visual fidelity and controllability, enabling conditioning on text, layout, or motion. Among these, motion control -- specifying object dynamics and camera trajectories -- is essential for composing complex, cinematic scenes, yet existing interfaces remain limited. We introduce LAMP that leverages large language models~(LLMs) as motion planners to translate natural language descriptions into explicit 3D trajectories for dynamic objects and (relatively defined) cameras. LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories. We construct a large-scale procedural dataset pairing natural text descriptions with corresponding motion programs and 3D trajectories. Experiments demonstrate LAMP's improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives, establishing the first framework for generating both object and camera motions directly from natural language specifications. Code and data will be released.

Method Overview

A learned LLM acts as a motion planner, generating symbolic motion programs in a DSL format from textual descriptions of object and camera motion. These programs are deterministically converted into 3D trajectories, which are used to condition a pretrained video generator.

Video Generation Process

Prompt

Inside a commercial airplane cabin, a man dressed in a flight attendant uniform walks steadily down the aisle, holding a large bird’s nest carefully in front of him. Behind him, another man wearing a Santa hat follows closely, adding a subtle touch of humor and surreal contrast. The camera faces them from the front, performing a smooth dolly-backward motion that leads the subjects down the aisle, maintaining a stable, cinematic flow with minimal shake. Soft overhead cabin lighting casts natural highlights on the uniforms and seats, while the narrow perspective of the aisle enhances the depth and focus on their unusual yet composed procession.

DSL

tail_track follow_style=soft follow_axis=z mirror_axis=no dolly=out_0.2 amp=all_0.5 lead=lead object=none jitter=none ease=in_out dutch=0

Generated Trajectory

Guidance Video

Generated Video
Prompt

A child rides a blue tricycle straight down a hallway, mostly centered and proceeding forward without notable interactions on either side; the camera follows from behind, moving forward with a noticeably unsteady motion and visible shake.

DSL

tail_track follow_style=hard follow_axis=z mirror_axis=no dolly=in_0.3 amp=all_1.5 lead=none object=none jitter=high ease=out dutch=0

Generated Trajectory

Guidance Video

Generated Video
Prompt

A biker rides straight ahead along a wet road through a misty landscape without interacting with anything, while the camera moves backward to lead from the front with a slightly unsteady motion that exhibits some shaking.

DSL

tail_track follow_style=soft follow_axis=z mirror_axis=no dolly=out_0.2 amp=all_0.8 lead=lead object=none jitter=low ease=in_out dutch=0

Generated Trajectory

Guidance Video

Generated Video
Prompt

A woman walks forward slowly through a dimly lit hallway, focused on something ahead, as the camera retreats to lead her from the front, its backward move slightly unsteady and showing some shaking.

DSL

tail_track follow_style=soft follow_axis=z mirror_axis=no dolly=out_0.2 amp=all_0.8 lead=lead object=none jitter=low ease=in_out dutch=0

Generated Trajectory

Guidance Video

Generated Video
Prompt

An airplane rolls forward along the runway, remaining the central focus without other interactions, while the camera smoothly tracks backward from the front and cranes upward around it so that the airplane gradually becomes smaller in the frame, all with very smooth, steady, shake-free motion.

DSL

orbit_track dir=ccw deg=180 plane_axis=x spiral=out_0.2 dutch=0 jitter=none ease=in_out object=none vertical_angle=aerial

Generated Trajectory

Guidance Video

Generated Video
Prompt

A person skates straight forward, staying centered with no significant interactions, while the camera tail-tracks from behind, moving forward consistently and smoothly panning right to keep them centered; as the follow continues the subject grows larger in the frame, with only minimal shaking.

DSL

tail_track follow_style=soft follow_axis=z mirror_axis=no dolly=in_0.2 amp=all_0.5 lead=none object=right jitter=none ease=in_out dutch=0

Generated Trajectory

Guidance Video

Generated Video
Prompt

A man in a denim jacket walks toward the right through a green park, while another man sits casually on a bench behind him. The camera advances forward with a soft, steady dolly while panning right to maintain focus on the walking subject. The background trees and benches drift gently across the frame, highlighting the quiet, continuous flow of his motion through the scene.

DSL

rotation_track rot_axis=pan push=in_0.2 local_offset=x_0.1 jitter=none ease=in_out dutch=0 vertical_angle=none

Generated Trajectory

Guidance Video

Generated Video
Prompt

A woman stands motionless in a lush garden filled with greenery and soft natural textures. The camera makes a full orbit around her, keeping her form perfectly centered while the environment slowly rotates behind her. The consistent distance and smooth circular motion create a serene sense of spatial completeness and contemplation.

DSL

orbit_track dir=cw deg=360 plane_axis=y spiral=no dutch=0 jitter=none ease=in_out object=none vertical_angle=none

Generated Trajectory

Guidance Video

Generated Video
Prompt

A ceramic vase is standing on a wooden table in a softly lit room. The camera slowly moves upward and slightly to the right, smoothly tracking the vase as it remains centered in the frame. Natural light from a nearby window creates gentle reflections and shadows on the vase’s surface. The background gradually shifts, revealing subtle details of the tabletop and surrounding space, giving the shot a realistic and elegant cinematic feel.

DSL

rotation_track rot_axis=full push=no local_offset=no truck_right_1_0.5 pedestal_up_1_0.3 goes_in_1_0.2 jitter=none ease=in_out dutch=0

Generated Trajectory

Guidance Video

Generated Video
Prompt

A bear ambles toward a dense forest, its movements heavy and grounded. A bear ambles toward a dense forest, its movements heavy and grounded. The camera follows along a smooth quarter arc, keeping the animal framed slightly off-center as trees gradually fill more of the background. The gentle curve of motion captures both the bear’s mass and the quiet transition from open space to shadowed woodland.

DSL

orbit_track deg_90

Generated Trajectory

Guidance Video

Generated Video
Prompt

A wild tiger runs straight toward the viewer across dense grass, muscles flexing beneath its fur. A wild tiger runs straight toward the viewer across dense grass, muscles flexing beneath its fur. The camera leads the motion, moving slightly backward to maintain distance while keeping the predator centered. The framing captures the animal’s power and momentum, amplifying the intensity as it closes in on the lens.

DSL

tail_track lead dolly_out_0.1

Generated Trajectory

Guidance Video

Generated Video
Prompt

An astronaut walks forward inside a metallic spacecraft corridor lined with glowing panels and narrow passageways. An astronaut walks forward inside a metallic spacecraft corridor lined with glowing panels and narrow passageways. The camera follows a gentle vertical counterclockwise arc, gradually moving closer as reflections shift across the surfaces. The smooth, controlled motion reinforces the feeling of calm focus and precision within the confined environment.

DSL

orbit_track plane_axis_z deg_360 spiral_in_0.1

Generated Trajectory

Guidance Video

Generated Video
Prompt

A playful cat chases a small dog as they both run to the right across a grassy backyard under bright daylight. A playful cat chases a small dog as they both run to the right across a grassy backyard. The camera remains fixed in position, but smoothly pans right to track their movement across the frame. The cat’s quick, agile strides contrast with the dog’s energetic sprint, creating a lively sense of motion. Grass and dust subtly blur in the foreground, and the natural lighting enhances the realistic, cheerful tone of the chase scene, giving it a cinematic and playful atmosphere.

DSL

rotation_track pan_only

Generated Trajectory

Guidance Video

Generated Video
Prompt

A helicopter rises steadily into the sky, its rotors blurring as it gains height. A helicopter rises steadily into the sky, its rotors blurring as it gains height. The camera performs a full horizontal circle around it, matching its altitude and maintaining continuous focus throughout the orbit. The surrounding air and distant clouds subtly shift with the rotation, emphasizing the machine’s ascent and the vastness of open space.

DSL

orbit_track deg_360

Generated Trajectory

Guidance Video

Generated Video
Prompt

An elderly woman climbs a flight of stairs slowly, her hand resting on the railing. An elderly woman climbs a flight of stairs slowly, her hand resting on the railing. The camera stays fixed in position but tilts downward slightly, observing her progress from above with patient steadiness. Each step she takes draws attention to her gradual ascent, the composition emphasizing effort and persistence in motion.

DSL

rotation_track tilt_only look_offset_y_-0.1

Generated Trajectory

Guidance Video

Generated Video
Prompt

A large ship is slowly sailing to the right across a calm, open ocean under a soft golden sunset sky. The camera moves in a smooth counterclockwise quarter arc around the ship, gradually pulling away while keeping the vessel in focus. Gentle waves ripple around the hull, and the sunlight reflects off the water’s surface, creating a warm, cinematic shimmer. As the camera retreats, the vastness of the ocean and the ship’s steady motion evoke a sense of tranquility and scale.

DSL

orbit_track deg_90
spiral_out_0.5

Generated Trajectory

Guidance Video

Generated Video

Comparisons

Prompt

A skateboarder stands at the top of a steep concrete ramp, crouched low in anticipation. As they launch forward, the board glides smoothly down the slope, their body staying compact and balanced with precise, controlled movements. The camera trucks slightly to the left, capturing the dynamic motion and shifting perspective, then tilts downward quickly as the skateboarder picks up speed. As they approach the bottom of the ramp, the camera moves backward in sync, tracking their descent and preparing for the landing. The motion is fluid and stable, with minimal shake, creating a cinematic, high-energy shot that captures the momentum, balance, and focus of the skateboarder’s ride.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

Inside a commercial airplane cabin, a man dressed in a flight attendant uniform walks steadily down the aisle, holding a large bird’s nest carefully in front of him. Behind him, another man wearing a Santa hat follows closely, adding a subtle touch of humor and surreal contrast. The camera faces them from the front, performing a smooth dolly-backward motion that leads the subjects down the aisle, maintaining a stable, cinematic flow with minimal shake. Soft overhead cabin lighting casts natural highlights on the uniforms and seats, while the narrow perspective of the aisle enhances the depth and focus on their unusual yet composed procession.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A child rides a blue tricycle straight down a hallway, mostly centered and proceeding forward without notable interactions on either side; the camera follows from behind, moving forward with a noticeably unsteady motion and visible shake.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A skateboarder glides along a sidewalk, performing a stunt that transitions into a crouched position while moving predominantly to the left. The camera dollies forward in a tail-follow motion, maintaining pursuit from behind while adding a subtle pan to the right during the stunt. As the skateboarder progresses, they gradually appear smaller in the frame, and the camera movement retains a slightly unsteady, natural shake, enhancing the dynamic, street-level realism of the scene.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A biker rides straight ahead along a wet road through a misty landscape without interacting with anything, while the camera moves backward to lead from the front with a slightly unsteady motion that exhibits some shaking.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A woman walks forward slowly through a dimly lit hallway, focused on something ahead, as the camera retreats to lead her from the front, its backward move slightly unsteady and showing some shaking.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

An airplane rolls forward along the runway, remaining the central focus without other interactions, while the camera smoothly tracks backward from the front and cranes upward around it so that the airplane gradually becomes smaller in the frame, all with very smooth, steady, shake-free motion.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A person skates straight forward, staying centered with no significant interactions, while the camera tail-tracks from behind, moving forward consistently and smoothly panning right to keep them centered; as the follow continues the subject grows larger in the frame, with only minimal shaking.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A coffee cup rests motionless in the center of a table as the camera, placed high above, moves forward and downward to approach it, then slightly tilts upward to level the view. The smooth descent reveals the tabletop’s texture and the warm tones of the surrounding environment. The movement feels natural but faintly handheld, creating the sense of a careful yet human observation of a static object.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A man in a denim jacket walks toward the right through a green park, while another man sits casually on a bench behind him. The camera advances forward with a soft, steady dolly while panning right to maintain focus on the walking subject. The background trees and benches drift gently across the frame, highlighting the quiet, continuous flow of his motion through the scene.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A woman stands motionless in a lush garden filled with greenery and soft natural textures. The camera makes a full orbit around her, keeping her form perfectly centered while the environment slowly rotates behind her. The consistent distance and smooth circular motion create a serene sense of spatial completeness and contemplation.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result
Prompt

A ceramic vase is standing on a wooden table in a softly lit room. The camera slowly moves upward and slightly to the right, smoothly tracking the vase as it remains centered in the frame. Natural light from a nearby window creates gentle reflections and shadows on the vase’s surface. The background gradually shifts, revealing subtle details of the tabletop and surrounding space, giving the shot a realistic and elegant cinematic feel.

Our Result

Vace Without Guidance Result

GPT-DSL Result

GPT Trajectory Result

Iterative Results

Example 1

1

Initial prompt

LLM input

A person is walking forward - Camera makes a clockwise quarter arc around the person.

DSL: orbit_track deg_90 cw

2

Refinement

LLM edit

“Make it 60 degree.”

DSL: orbit_track deg_60 cw

3

Final tweak

LLM edit

“Add a spiral out move to this 60 degree arc.”

DSL: orbit_track deg_60 cw spiral_out_0.3
Example 2

1

Initial prompt

LLM input

Woman goes right and in - Camera tracks the woman with huge delay.

DSL: tail_track follow_style_lazy

2

Refinement

LLM edit

“while lazily following, it can only move on x-axis.”

DSL: tail_track follow_style_lazy follow_axis_x

3

Final tweak

LLM edit

“also it moves slightly less than woman horizontally.”

DSL: tail_track follow_style_lazy follow_axis_x amp_x_0.8
Example 3

1

Initial prompt

LLM input

Man rushes to right while turning extensively left - Camera pans to track the man while slightly trucking right.

DSL: tail_track follow_style_lazy follow_axis_x amp_x_0.8 dutch_cw_15

2

Refinement

LLM edit

“Move camera more to rightward.”

DSL: rotation_track trucks_right_1_0.1

3

Final tweak

LLM edit

“After moving extensively right, it should slightly pedestal up.”

DSL: rotation_track truck_right_1_0.5 pedestal_up_2_0.1
Example 4

1

Initial prompt

LLM input

Dog is running right - Camera remains static while yawing left.

DSL: yaw_25

2

Refinement

LLM edit

“Add small right move while keeping yaw right.”

DSL: near_right yaw_25

3

Final tweak

LLM edit

“Add small pedestal up and tilt down after moving right and yawing right.”

DSL: near_right yaw_25, near_up tilt_-40
Example 5

1

Initial prompt

LLM input

A car goes forward - Camera slightly goes forward.

DSL: in

2

Refinement

LLM edit

“Make forward move faster.”

DSL: far_in

3

Final tweak

LLM edit

“After forward move camera goes backward while yawing left.”

DSL: far_in, far_out yaw_-30

Multi Object Results

Prompt

Nighttime quiet road illuminated by vehicle headlights and street lamps. On the right, a sleek black motorcycle close to the camera speeds forward, engine humming, LED beam cutting the asphalt. On the left, far in the background, a large yellow construction truck drives toward the camera, headlights casting wide warm cones. Motorcycle moves away, truck grows larger, paths visually crossing. Camera rises gently and drifts forward, framing both vehicles, reflections on wet pavement, emphasizing contrast between fast, close motorcycle and slow, distant truck.
Prompt

Misty medieval forest. Three riders in formation on a narrow woodland path: king on a white horse in the center, two warriors slightly ahead on gray horses flanking him. Steady forward ride. Dense trees, drifting fog, pale sun shafts. Camera traveling with them at rider level, then slowly rising upward while moving forward, revealing treetops but keeping riders centered. Noble, atmospheric, cinematic.
Prompt

Highway at early evening. Right lane: sleek gray sedan speeding right→left, kicking subtle dust. Left lane: bright yellow Lamborghini racing left→right with warm orange reflections. Rugged rocky terrain with orange stones, light dust in air. Camera rises and pushes in, widening the view while keeping both cars in frame, revealing long road and dramatic warm shadows.
Prompt

Mystical realm with swirling ambient magic. Armored knight on horseback riding forward; camera follows closely from behind, a few meters back. Ahead: massive stone arcane gateway with glowing runes, flanked by two lion statues wrapped in blue-gold swirling magic. Dust rising under hooves, floating light particles around the portal. Knight gallops toward the glowing gateway as camera glides forward with him.
Prompt

Dusty highway in a rocky orange landscape at late afternoon. Left side: old weathered metal service booth, stationary. Right side: slightly rusted gray sedan speeding diagonally toward the booth, kicking up dust. Warm sun, long shadows. Camera pulls back and rises slowly, keeping both car and booth in frame, revealing rocky terrain and drifting sunlit dust. Cinematic.

Long Video Generation

Prompt
A spacious, old stone hall with warm sunlight, tall archways, and a dusty stone floor. Ball first drops and bounces repeatedly until settling, then it drops with a last bounce and rolls forward, finally it continues rolls across the floor. Camera first tilt tracks the ball, then it tilts and moves forward to follow the ball, finally it arcs smoothly around the ball’s right side.

Object DSL

far_down, up, down, near_up
near_down, near_up in, near_down in, in
in

Camera DSL

rotation_track tilt_only low_angle
in tilt:-15, in tilt: 10
orbit_track deg_120

Generated Trajectory

Generated Video
Prompt
A warm, sunlit farmyard filled with drifting dust and a nostalgic, golden glow. Rooster first walks confidently across the ground, then it flaps its wings to hover slightly in midair, finally it bobs rhythmically and stretches its neck to crow. Camera first follows from behind and arcs to the left, then it stays steady with minimal reactive movements, finally it performs a smooth, deliberate push-in.

Object DSL

right near_in, right in
right near_in, right near_up
left near_up, left near_down, left up

Camera DSL

orbit_track deg_90 cw
rotation_track goes_in
rotation_track push_in

Generated Trajectory

Generated Video
Prompt
A dense, humid Amazonian rainforest alive with dappled sunlight, massive ferns, ancient trees, and colorful birds flitting through the upper canopy. The magnificent Bengal tiger first walks purposefully toward the viewer from the left, then it continues its steady advance through the foliage, finally it moves slightly away before performing an upward standing. Camera first pulls backward while tracking the tiger, then it executes a smooth arc to the left around the animal’s front, finally it slightly rises to a high overhead perspective to capture the tigers' standing.

Object DSL

out
out
right near_out, near_up

Camera DSL

tail_track object_left ease_in
orbit_track deg_60 cw ease_out
rotation_track goes_out pedestal_up

Generated Trajectory

Generated Video

References

[GenDoP] Auto-regressive Camera Trajectory Generation as a Director of Photography, ICCV 2025.

@misc{zhang2025gendopautoregressivecameratrajectory, title={GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photography}, author={Mengchen Zhang and Tong Wu and Jing Tan and Ziwei Liu and Gordon Wetzstein and Dahua Lin}, year={2025}, eprint={2504.07083}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2504.07083}, }

[ET] The Exceptional Trajectory: Text-to-camera-trajectory generation with character awareness, ECCV 2024.

@misc{courant2024etexceptionaltrajectoriestexttocameratrajectory, title={E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness}, author={Robin Courant and Nicolas Dufour and Xi Wang and Marc Christie and Vicky Kalogeiton}, year={2024}, eprint={2407.01516}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2407.01516}, }

[Director3D] Real-world Camera Trajectory and 3D Scene Generation from Text, NeurIPS 2024.

@inproceedings{NEURIPS2024_89566c18, author = {Li, Xinyang and Lai, Zhangyu and Xu, Linning and Qu, Yansong and Cao, Liujuan and Zhang, Shengchuan and Dai, Bo and Ji, Rongrong}, booktitle = {Advances in Neural Information Processing Systems}, doi = {10.52202/079017-2391}, editor = {A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang}, pages = {75125--75151}, publisher = {Curran Associates, Inc.}, title = {Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text}, url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/89566c18d5b3e9836e8e16fde010b41d-Paper-Conference.pdf}, volume = {37}, year = {2024} }

[CCD] Cinematographic Camera Diffusion Model, Eurographics 2024.

@misc{jiang2024cinematographiccameradiffusionmodel, title={Cinematographic Camera Diffusion Model}, author={Hongda Jiang and Xi Wang and Marc Christie and Libin Liu and Baoquan Chen}, year={2024}, eprint={2402.16143}, archivePrefix={arXiv}, primaryClass={cs.GR}, url={https://arxiv.org/abs/2402.16143}, }

[VACE] All-in-One Video Creation and Editing

@misc{jiang2025vaceallinonevideocreation, title={VACE: All-in-One Video Creation and Editing}, author={Zeyinzi Jiang and Zhen Han and Chaojie Mao and Jingfeng Zhang and Yulin Pan and Yu Liu}, year={2025}, eprint={2503.07598}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2503.07598}, }

Citation

[LAMP] M. B. Kizil et al., LAMP: Language-Assisted Motion Planning for Controllable Video Generation, arXiv 2025.

@misc{kizil2025lamplanguageassistedmotionplanning, title={LAMP: Language-Assisted Motion Planning for Controllable Video Generation}, author={Muhammed Burak Kizil and Enes Sanli and Niloy J. Mitra and Erkut Erdem and Aykut Erdem and Duygu Ceylan}, year={2025}, eprint={2512.03619}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2512.03619}, }