OPEN-SOURCE EXTENSION

pi-vision: eyes for pi agents

Ask your agent about a PDF, a scan, a photo, a video or a recording. It reads the file, answers, and cites the page, second or spot behind each answer. At the end it writes you a report.

Watch it runRead the README

Examples

Each example is a folder with a question and its files. You can run any of them with the commands below.

A 1906 frame of Market Street with Jev's boxes from the run: a streetcar or cable car, a horse carriage or wagon and an automobile.market-street

Every vehicle on Market Street

A 1906 film from the front of a cable car on San Francisco's Market Street, four days before the earthquake. The agent logs every vehicle that crosses in front of the camera, with the second it passes and a box around it, then counts each kind.

Watch the run README
A 1939 World's Fair frame with Jev's box from the run around the golden robot.fair-robot

A robot in a long film

A 1939 film of nearly an hour. When does a golden robot first appear, and where is it in the frame?

Watch the run README
The foot of a Delaware certificate with Jev's boxes from the run on the seal and the signature.document-check

Document check

A certified copy of a Delaware filing, and a test copy with every mark painted out. For each copy you get the marks it has, where they are, and a box on each.

Watch the run README
A Landsat image of Lake Mead in July 2003, captioned with the run's finding: the water fell most between the May 2000 and July 2003 images. Landsat imagery courtesy of the U.S. Geological Survey.lake-mead

Lake Mead from space

Thirteen Landsat images of Lake Mead, from 1984 to 2024. When did the water fall most, and where? The agent boxes each shoreline the water has left since 1984 and names the first image it is dry in.

README
The lunar surface just after touchdown, inside the run's chapter 6, Lunar landing, 11:00 to 13:55.video-chapters

A film, in chapters

NASA's 1969 Apollo 11 film, cut into chapters. Each chapter gets a title, a start and end time, a line on what happens, and a frame.

README
The US flag on the Moon in NASA's Apollo 11 film, with Jev's box from the run: American flag.hear-and-find

Hear and find

When is "The Eagle has landed" said during the descent, and when does the flag first show on screen? You get both times, the frame, and a box around the flag.

README
Page 317 of NASA's Apollo 11 transcript, where the commander says the Eagle has landed, captioned with the run's answer: touchdown at 102:45:40 GET.apollo-descent

The Apollo 11 descent

NASA's typed transcript, twelve photos and the descent film, turned into a minute-by-minute account of the landing. Each line cites its page, photo or film moment.

README
Boulder Dam releasing jets from its outlet valves, inside the run's chapter 13, 32:01 to 34:52.boulder-chapters

Boulder Dam, in chapters

A 1930s government film about building Boulder Dam, cut into chapters. The film has no chapter cards, so the agent finds the breaks itself.

README

Reading a replay

A replay opens as a chat. On the left you see what the agent thought and did, with Jev's answers in pink. On the right is the report it wrote.

What it adds to your agent

The read tool keeps its name and arguments. It now reads PDFs, Office files, spreadsheets, images and video, and it tells the model when it cannot read something.

Skills

pi-vision adds no new tool. The agent writes code, and five skills show it how.

Run it

git clone https://github.com/Parcha-ai/agentrun && cd agentrun
npm ci --ignore-scripts && npm run build
npm run demo:vision -- examples/document-check          # scripted: no key, no network
npm run demo:vision -- examples/hear-and-find --live    # real models, your keys

A live run needs GEMINI_API_KEY (or another provider's key), TYPESAFE_API_KEY for Jev, and FAL_KEY for speech. The agent's code calls ffmpeg, ffprobe, pdftoppm and pdftotext.

The agent runs shell commands as your user, so run it in a container or a VM.

To ask your own question, make a folder with question.md and inputs.json, or pass --input.

Packages

Node 22.19 or later, ESM only. Install with @beta. Apache-2.0.