Turning Ordinary Video Into a 3D Scene With LingBot-Map — Without Buying a New GPU

A recently released open-source project called “LingBot-Map” (https://github.com/Robbyant/lingbot-map) can reconstruct a three-dimensional representation of a scene from an ordinary video or a sequence of images.

Instead of requiring a dedicated depth camera or LiDAR scanner, LingBot-Map analyzes the changing viewpoint in the video. It estimates the camera’s movement, calculates depth and reconstructs the visible environment as a coloured 3D point cloud.

The project includes:

• An interactive browser-based 3D viewer
• Direct processing of video files or image folders
• Streaming and windowed processing modes
• Camera-pose and trajectory estimation
• Depth and confidence predictions
• Sky masking for outdoor footage
• An offline pipeline that can render a point-cloud fly-through as an MP4

The developers describe it as a feed-forward 3D foundation model designed for streaming reconstruction. They report approximately 20 frames per second at a working resolution of 518 × 378 pixels on suitable hardware, although actual performance will depend heavily on the GPU and the options selected.

This is not the same as pressing a button and receiving a polished, watertight 3D model. Its primary output is a reconstructed point cloud, depth information and estimated camera path. Converting that into a clean, textured mesh suitable for editing, printing or game development would require additional processing.

Even so, it is an impressive way to turn a simple walk-through video into a navigable 3D representation of a real place.

Why running it locally is difficult

LingBot-Map is open source, but that does not mean it will run comfortably on an ordinary computer.

The provided installation instructions are built around:

• Linux
• Python 3.10
• PyTorch with CUDA 12.8
• An NVIDIA GPU
• FlashInfer for its optimized attention system
• NVIDIA Kaolin for the full offline rendering pipeline
• Additional custom CUDA extensions for rendering

The core interactive demo can fall back from FlashInfer to PyTorch’s native attention system, but the overall project remains strongly designed around NVIDIA CUDA. The offline renderer is even more firmly tied to NVIDIA because it uses Kaolin and custom-compiled CUDA code.

That makes my Radeon RX 6700 XT unsuitable without significant porting work. It might eventually be possible to adapt part of the reconstruction model to AMD’s ROCm platform, but that would be a software-development project of its own. It is not currently a straightforward installation.

Video length also creates a substantial memory problem. Although LingBot-Map performs streaming inference internally, the supplied demo first extracts and preprocesses the selected frames. Its current code can then move the full image tensor onto the GPU before inference begins. Long videos can therefore consume several gigabytes of video memory before the model, attention cache and reconstruction data are counted.

Community reports suggest that modified versions can operate within 8GB of GPU memory, but the standard software is better treated as a 16GB-to-48GB GPU workload. For a relatively painless first attempt, 24GB of GPU memory is a sensible target. Longer sequences and the full rendering pipeline are better suited to 48GB.

Buying an NVIDIA card specifically for occasional experiments would be difficult to justify. Renting one by the hour is far more practical.

Renting a GPU instead

Cloud GPU services let users temporarily rent a remote Linux computer equipped with an NVIDIA GPU.

This is different from calling a hosted artificial-intelligence API. LingBot-Map is not simply a model that accepts a request and returns a small response. It is an application with:

• Source video files
• Large model checkpoints
• Python dependencies
• CUDA libraries
• Extracted image frames
• An interactive web viewer
• Optional rendering tools
• Large output files

The appropriate service is therefore a persistent GPU virtual machine, often called a GPU pod, rather than a serverless inference endpoint.

Services such as “RunPod” (https://www.runpod.io/), “Vast.ai” (https://vast.ai/), “Lambda Cloud” (https://lambda.ai/service/gpu-cloud) and similar providers rent GPU-equipped Linux systems by the hour. The exact prices and available hardware change continuously.

For this project, a useful hierarchy is:

GPU class	GPU memory	Suitability
RTX A4000 or similar	16GB	Small experiments with careful settings
RTX 3090, RTX 4090 or A5000	24GB	Good starting point
RTX A6000 or A40	48GB	Better for longer or less constrained work
RTX 6000 Ada or L40S	48GB	Faster 48GB option
A100 or H100	80GB	Powerful but probably unnecessary for initial testing

RunPod currently groups its 24GB RTX 3090, A5000 and L4 systems as cost-effective small-to-medium inference hardware. Its A6000 and A40 systems provide 48GB at a higher rate, while faster 48GB cards such as the L40S and RTX 6000 Ada cost more. RunPod bills pods by the second, with storage charged separately, and directs users to its deployment console for current prices.

A 24GB RTX 3090-class pod is probably the best first choice. It provides enough memory to determine whether the reconstruction quality is useful without paying for premium data-centre hardware.

If that runs out of memory or requires too many compromises, moving the same setup to a 48GB A6000-class pod should be straightforward.

What to select when creating the rented system

A reasonable first LingBot-Map pod would have:

• An NVIDIA GPU with at least 24GB of VRAM
• Ubuntu Linux
• A CUDA development image
• CUDA 12.8 compatibility
• At least 32GB of system RAM
• Preferably 64GB or more system RAM
• At least 50GB of persistent storage
• Preferably 100GB if processing several videos
• SSH access
• A way to expose or tunnel port 8080

A development image is preferable to a minimal runtime image. FlashInfer may compile optimized kernels when first used, and the offline renderer explicitly builds CUDA extensions. A development image includes the compiler and headers needed for this.

Persistent storage is also important. A rented GPU instance can disappear when stopped or deleted. The repository, model checkpoint, source videos and reconstructed output should be placed on a persistent volume if they need to survive between sessions.

A basic installation

Once connected to the rented Linux machine, the initial setup would look roughly like this:

git clone https://github.com/Robbyant/lingbot-map.git
cd lingbot-map

conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map

pip install torch==2.8.0 torchvision==0.23.0 \
 --index-url https://download.pytorch.org/whl/cu128

pip install -e ".[vis]"

pip install --index-url https://pypi.org/simple \
 flashinfer-python

The model checkpoint is available through the project’s “Hugging Face repository” (https://huggingface.co/robbyant/lingbot-map).

The project offers several checkpoints. The “lingbot-map-long” model is intended for longer sequences and larger scenes, while the standard “lingbot-map” checkpoint provides a more general balance.

A model can be downloaded using the Hugging Face command-line utility:

hf download robbyant/lingbot-map lingbot-map-long.pt \
 --local-dir checkpoints

Start with a small test

The first experiment should not be a 20-minute 4K video.

LingBot-Map resizes its input to a much smaller working resolution, so uploading extremely high-resolution video provides little benefit while increasing transfer and preprocessing time. A short, steady walk-through with good lighting and visible surface detail is a better test.

The first run could sample five frames per second and stop after 300 frames:

python demo.py \
 --model_path checkpoints/lingbot-map-long.pt \
 --video_path input.mp4 \
 --fps 5 \
 --first_k 300 \
 --num_scale_frames 2 \
 --camera_num_iterations 1 \
 --keyframe_interval 2

These settings are intentionally conservative:

• “–fps 5” samples five frames from each second of video.
• “–first_k 300” limits the test to 300 extracted frames.
• “–num_scale_frames 2” reduces the initial memory peak.
• “–camera_num_iterations 1” reduces camera refinement from its normal four iterations.
• “–keyframe_interval 2” stores only every second frame in the attention cache.

This should be treated as an installation and feasibility test rather than a final-quality reconstruction.

If it succeeds, the frame limit and camera iterations can be increased. Longer videos can use windowed processing:

python demo.py \
 --model_path checkpoints/lingbot-map-long.pt \
 --video_path input.mp4 \
 --fps 10 \
 --mode windowed \
 --window_size 128 \
 --overlap_keyframes 8 \
 --keyframe_interval 4 \
 --num_scale_frames 2

Windowed mode processes long sequences in overlapping sections instead of maintaining one continuously growing context.

Viewing the reconstruction

The interactive demo starts a browser-based viewer on port 8080.

On the rented machine it will report an address similar to:

http://localhost:8080

That address refers to the remote system, not the user’s local computer. There are two common ways to reach it.

The first is to expose TCP port 8080 through the GPU provider’s interface. The second, and generally safer, method is an SSH tunnel:

ssh -L 8080:localhost:8080 user@remote-server

After establishing the tunnel, the viewer can be opened locally at:

http://localhost:8080

The interface then displays the reconstructed point cloud and camera trajectory while the computation continues to run on the rented GPU.

Moving files to and from the pod

Input video can be uploaded with “scp”, SFTP, the provider’s file manager or an object-storage service.

For example:

scp input.mp4 user@remote-server:/workspace/lingbot-map/

Completed predictions, videos or point-cloud exports can be copied back in the opposite direction:

scp -r user@remote-server:/workspace/lingbot-map/output ./

Large files may be easier to move through cloud storage, but direct transfer is sufficient for initial experiments.

Controlling the cost

The GPU itself is usually not the expensive part of a small experiment. Forgetting to shut it down is.

A sensible process is:

1. Create the pod.
2. Install the software.
3. Store the environment and model on persistent storage.
4. Upload one short video.
5. Run the test.
6. Download the result.
7. Stop or terminate the pod immediately.

Persistent storage may continue to incur a smaller charge after the GPU is stopped. It is worth checking whether the provider distinguishes between stopping a pod, deleting the pod and deleting its storage volume.

A successful first experiment may require only a fraction of an hour of GPU computation, although installation, model downloading and CUDA compilation can add setup time. Once the environment is preserved, later sessions should be faster to start.

What kind of video should work best?

The system needs visual evidence of both camera movement and stable scene structure.

Good input should generally have:

• Slow, deliberate camera motion
• Substantial overlap between consecutive views
• Good lighting
• Sharp frames
• Detailed surfaces
• Limited motion blur
• Mostly stationary objects
• A path that occasionally looks back toward previously viewed areas

Poor input would include:

• Rapid turns
• Heavy motion blur
• Featureless white walls
• Large reflective surfaces
• Crowds or moving vehicles dominating the frame
• Strong exposure changes
• Digital zoom
• Long periods with almost no camera movement

A phone video recorded while walking slowly around a room, building, garden or other fixed environment should be a reasonable starting point.

Is it worth trying?

LingBot-Map is interesting because it reduces 3D reconstruction to a video-capture problem and a GPU-processing problem. There is no specialized scanner to purchase and no permanent high-end GPU requirement.

The hardware requirements are substantial, but cloud rental changes the decision. Instead of spending hundreds or thousands of dollars on an NVIDIA GPU, it is possible to rent the appropriate hardware long enough to answer the important question:

Does the reconstruction provide useful results for my own videos and projects?

That can be determined inexpensively with one short recording, a 24GB GPU pod and an hour or less of hands-on experimentation.

The project is available here: https://github.com/Robbyant/lingbot-map