My daughter rides horses. One day she asked me something about one of her riding videos, so I said let's upload it to an LLM and see what it says. The analysis was good enough that she demanded an app she could upload videos to. While building it, I discovered something I suspect most people sending media to vision models do not know: the exact same frame is worth a completely different number of tokens depending on whether you sent it inside a video or as an image.
The original version was a tweet in Hebrew.
How many tokens does a frame actually cost?
In Gemini 3 the numbers look like this, controlled by media_resolution:
- Image: 280 / 560 / 1120 tokens at low / medium / high. There is also an
ultra_highlevel worth 2240, which can only be set on the individual part. - Frame inside a video: 70 tokens at low and also at medium, 280 at high.
ultra_highdoes not exist for video at all.
Here is the important part: video's ceiling, 280, is exactly the floor for images. The most detailed frame a video can give you equals an image at its lowest setting. And with defaults, when you touch nothing, a video frame is sampled at 70 tokens while an image gets 1120. That is 16x, on the same pixels.
The model does not miss the small details because it is not smart enough. It misses them because you sent it a heavily downsampled version of what you watched.
Whoever uploads video because it is more convenient pays for that convenience in analysis quality, and usually without knowing, like I did not know.
Should you send video or frames?
The question to ask before building anything: what am I actually trying to analyze?
If the signal is motion, sequence, change over time, or audio, video is the right format and there is nothing to argue about. Individual frames simply do not carry that information.
But if what you are looking for is clearly visible in a single frame, and in my case it is exactly that, the angle of a heel, the line of a back, the position of a hand over a jump, then you do not need video. You need the right frames. Send them as images and each one gets 4 to 16 times more visual budget. Same model, same request, completely different analysis.
What do media_resolution and fps control?
You have control in both directions:
- Video has two dials.
media_resolutiondecides whether each frame is worth 70 or 280 tokens, andfpsinvideoMetadatadecides how many frames get sampled at all. The default is one frame per second, and you can go up to 24. If the moment you care about is fast, the default sampling can skip it entirely. - Images use the same
media_resolutionwith levels worth 280, 560, 1120 or 2240 tokens, and you can set it per part. That means you can send the three critical frames at high and the rest of the context at low, in the same request.
Why extract frames on the client?
The part that saved me the most headache: frame extraction and encoding can run entirely on the client. The phone already holds the video and already knows how to decode it. Instead of uploading a raw 4K clip and running ffmpeg on a server, I upload only the frames I actually need.
That removes the whole server-side media pipeline: the CPU bill that grows with volume, the amusing codec and rotation-metadata bugs, the user's data plan, and the upload itself, which is usually the most fragile step in the chain. Worth checking before you build a media worker you do not actually need.
What is the catch?
None of this is free. More tokens means more money and more latency, and frames give up sequence and audio. But it should be a decision, not a default you forgot to check. I chose frames because in my case the small details are the signal.
One last tip: always set media_resolution explicitly, even when the default suits you. You want it to be something you decided, not something you discover on the invoice. It is the same discipline as picking the smallest model that does the job, which I wrote about in run the weakest model that still does the job: capability budgets are choices, and unexamined defaults are where the quality and the money quietly leak.
Getting these decisions right early is a large part of what I do when I build AI products and agents for clients. The API always works on the default settings. The product works when someone asked what the model actually needs to see.
FAQ
How many tokens does an image cost in Gemini 3?
It depends on media_resolution: an image costs 280 tokens at low, 560 at medium, 1120 at high, and 2240 at ultra_high, which can only be set on the individual part. A frame inside a video costs 70 tokens at low and medium and 280 at high, and ultra_high does not exist for video at all.
Should I send video or individual frames to a vision model?
Decide by the signal you are analyzing. If it is motion, sequence, change over time, or audio, send video, because single frames do not carry that information. If what you are looking for is visible in a single frame, send the right frames as images: each one gets 4 to 16 times more token budget than it would inside a video, so the model sees far more detail.