zenlm/ml-fastvlm
1
0
Fork 0
mirror of https://github.com/zenlm/ml-fastvlm.git synced 2026-07-26 22:30:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

Rebrand README

Browse Source
This commit is contained in:
z
2026-02-26 19:41:21 -08:00
parent 2503f14eba
commit 21ac6bd3ae
1 changed files with 18 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+18 -84
View File
@@ -1,94 +1,28 @@
# FastVLM: Efficient Vision Encoding for Vision Language Models
# FastVLM
This is the official repository of
**[FastVLM: Efficient Vision Encoding for Vision Language Models](https://www.arxiv.org/abs/2412.13303). (CVPR 2025)**
Fast vision-language model architecture research. Part of the Zen LM ecosystem.
[//]: # (![FastViTHD Performance](docs/acc_vs_latency_performance.png))
<p align="center">
<img src="docs/acc_vs_latency_performance.png" alt="Accuracy vs latency figure." width="400"/>
</p>
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
### Highlights
* We introduce FastViTHD, a novel hybrid vision encoder designed to output fewer tokens and significantly reduce encoding time for high-resolution images.
* Our smallest variant outperforms LLaVA-OneVision-0.5B with 85x faster Time-to-First-Token (TTFT) and 3.4x smaller vision encoder.
* Our larger variants using Zen-7B LLM outperform recent works like Cambrian-1-8B while using a single image encoder with a 7.9x faster TTFT.
* Demo iOS app to demonstrate the performance of our model on a mobile device.
## Overview
<table>
<tr>
<td><img src="docs/fastvlm-counting.gif" alt="FastVLM - Counting"></td>
<td><img src="docs/fastvlm-handwriting.gif" alt="FastVLM - Handwriting"></td>
<td><img src="docs/fastvlm-emoji.gif" alt="FastVLM - Emoji"></td>
</tr>
</table>
FastVLM explores efficient architectures for vision-language models, focusing on reducing computational overhead while maintaining strong multimodal understanding.
## Getting Started
We use LLaVA codebase to train FastVLM variants. In order to train or finetune your own variants,
please follow instructions provided in [LLaVA](https://github.com/haotian-liu/LLaVA) codebase.
We provide instructions for running inference with our models.
## Features
### Setup
```bash
conda create -n fastvlm python=3.10
conda activate fastvlm
pip install -e .
```
- Efficient vision-language model architecture
- Reduced computational overhead vs standard VLMs
- Strong multimodal understanding
- Research reference implementation
### Model Zoo
For detailed information on various evaluations, please refer to our [paper](https://www.arxiv.org/abs/2412.13303).
## Related
| Model | Stage | Pytorch Checkpoint (url) |
|:-------------|:-----:|:---------------------------------------------------------------------------------------------------------------:|
| FastVLM-0.5B | 2 | [fastvlm_0.5b_stage2](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_0.5b_stage2.zip) |
| | 3 | [fastvlm_0.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_0.5b_stage3.zip) |
| FastVLM-1.5B | 2 | [fastvlm_1.5b_stage2](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_1.5b_stage2.zip) |
| | 3 | [fastvlm_1.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_1.5b_stage3.zip) |
| FastVLM-7B | 2 | [fastvlm_7b_stage2](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_7b_stage2.zip) |
| | 3 | [fastvlm_7b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_7b_stage3.zip) |
To download all the pretrained checkpoints run the command below (note that this might take some time depending on your connection so might be good to grab ☕️ while you wait).
```bash
bash get_models.sh # Files will be downloaded to `checkpoints` directory.
```
### Usage Example
To run inference of PyTorch checkpoint, follow the instruction below
```bash
python predict.py --model-path /path/to/checkpoint-dir \
--image-file /path/to/image.png \
--prompt "Describe the image."
```
### Inference on Apple Silicon
To run inference on Apple Silicon, pytorch checkpoints have to be exported to format
suitable for running on Apple Silicon, detailed instructions and code can be found [`model_export`](model_export/) subfolder.
Please see the README there for more details.
For convenience, we provide 3 models that are in Apple Silicon compatible format: [fastvlm_0.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_0.5b_stage3_llm.fp16.zip),
[fastvlm_1.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_1.5b_stage3_llm.int8.zip),
[fastvlm_7b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_7b_stage3_llm.int4.zip).
We encourage developers to export the model of their choice with the appropriate quantization levels following
the instructions in [`model_export`](model_export/).
### Inference on Apple Devices
To run inference on Apple devices like iPhone, iPad or Mac, see [`app`](app/) subfolder for more details.
## Citation
If you found this code useful, please cite the following paper:
```
@InProceedings{fastvlm2025,
author = {Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, Hadi Pouransari},
title = {FastVLM: Efficient Vision Encoding for Vision Language Models},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2025},
}
```
## Acknowledgements
Our codebase is built using multiple opensource contributions, please see [ACKNOWLEDGEMENTS](ACKNOWLEDGEMENTS) for more details.
- [zen-vl](https://huggingface.co/zenlm/zen-vl-8b-instruct) — Zen vision-language models
- [jin](https://github.com/zenlm/jin) — Multimodal understanding framework
- [Zen LM](https://github.com/zenlm) — Full model family
## License
Please check out the repository [LICENSE](LICENSE) before using the provided code and
[LICENSE_MODEL](LICENSE_MODEL) for the released models.
See LICENSE file.
*Part of the [Zen LM](https://zenlm.org) ecosystem by [Hanzo AI](https://hanzo.ai)*