diff --git a/README.md b/README.md index 40cd26a..0e9693a 100644 --- a/README.md +++ b/README.md @@ -1,94 +1,28 @@ -# FastVLM: Efficient Vision Encoding for Vision Language Models +# FastVLM -This is the official repository of -**[FastVLM: Efficient Vision Encoding for Vision Language Models](https://www.arxiv.org/abs/2412.13303). (CVPR 2025)** +Fast vision-language model architecture research. Part of the Zen LM ecosystem. -[//]: # (![FastViTHD Performance](docs/acc_vs_latency_performance.png)) -

-Accuracy vs latency figure. -

+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -### Highlights -* We introduce FastViTHD, a novel hybrid vision encoder designed to output fewer tokens and significantly reduce encoding time for high-resolution images. -* Our smallest variant outperforms LLaVA-OneVision-0.5B with 85x faster Time-to-First-Token (TTFT) and 3.4x smaller vision encoder. -* Our larger variants using Zen-7B LLM outperform recent works like Cambrian-1-8B while using a single image encoder with a 7.9x faster TTFT. -* Demo iOS app to demonstrate the performance of our model on a mobile device. +## Overview - - - - - - -
FastVLM - CountingFastVLM - HandwritingFastVLM - Emoji
+FastVLM explores efficient architectures for vision-language models, focusing on reducing computational overhead while maintaining strong multimodal understanding. -## Getting Started -We use LLaVA codebase to train FastVLM variants. In order to train or finetune your own variants, -please follow instructions provided in [LLaVA](https://github.com/haotian-liu/LLaVA) codebase. -We provide instructions for running inference with our models. +## Features -### Setup -```bash -conda create -n fastvlm python=3.10 -conda activate fastvlm -pip install -e . -``` +- Efficient vision-language model architecture +- Reduced computational overhead vs standard VLMs +- Strong multimodal understanding +- Research reference implementation -### Model Zoo -For detailed information on various evaluations, please refer to our [paper](https://www.arxiv.org/abs/2412.13303). +## Related -| Model | Stage | Pytorch Checkpoint (url) | -|:-------------|:-----:|:---------------------------------------------------------------------------------------------------------------:| -| FastVLM-0.5B | 2 | [fastvlm_0.5b_stage2](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_0.5b_stage2.zip) | -| | 3 | [fastvlm_0.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_0.5b_stage3.zip) | -| FastVLM-1.5B | 2 | [fastvlm_1.5b_stage2](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_1.5b_stage2.zip) | -| | 3 | [fastvlm_1.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_1.5b_stage3.zip) | -| FastVLM-7B | 2 | [fastvlm_7b_stage2](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_7b_stage2.zip) | -| | 3 | [fastvlm_7b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_7b_stage3.zip) | - -To download all the pretrained checkpoints run the command below (note that this might take some time depending on your connection so might be good to grab ☕️ while you wait). - -```bash -bash get_models.sh # Files will be downloaded to `checkpoints` directory. -``` - -### Usage Example -To run inference of PyTorch checkpoint, follow the instruction below -```bash -python predict.py --model-path /path/to/checkpoint-dir \ - --image-file /path/to/image.png \ - --prompt "Describe the image." -``` - -### Inference on Apple Silicon -To run inference on Apple Silicon, pytorch checkpoints have to be exported to format -suitable for running on Apple Silicon, detailed instructions and code can be found [`model_export`](model_export/) subfolder. -Please see the README there for more details. - -For convenience, we provide 3 models that are in Apple Silicon compatible format: [fastvlm_0.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_0.5b_stage3_llm.fp16.zip), -[fastvlm_1.5b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_1.5b_stage3_llm.int8.zip), -[fastvlm_7b_stage3](https://ml-site.cdn-apple.com/datasets/fastvlm/llava-fastvithd_7b_stage3_llm.int4.zip). -We encourage developers to export the model of their choice with the appropriate quantization levels following -the instructions in [`model_export`](model_export/). - -### Inference on Apple Devices -To run inference on Apple devices like iPhone, iPad or Mac, see [`app`](app/) subfolder for more details. - -## Citation -If you found this code useful, please cite the following paper: -``` -@InProceedings{fastvlm2025, - author = {Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, Hadi Pouransari}, - title = {FastVLM: Efficient Vision Encoding for Vision Language Models}, - booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, - month = {June}, - year = {2025}, -} -``` - -## Acknowledgements -Our codebase is built using multiple opensource contributions, please see [ACKNOWLEDGEMENTS](ACKNOWLEDGEMENTS) for more details. +- [zen-vl](https://huggingface.co/zenlm/zen-vl-8b-instruct) — Zen vision-language models +- [jin](https://github.com/zenlm/jin) — Multimodal understanding framework +- [Zen LM](https://github.com/zenlm) — Full model family ## License -Please check out the repository [LICENSE](LICENSE) before using the provided code and -[LICENSE_MODEL](LICENSE_MODEL) for the released models. + +See LICENSE file. + +*Part of the [Zen LM](https://zenlm.org) ecosystem by [Hanzo AI](https://hanzo.ai)*