Llama3-V emerges as a groundbreaking multimodal model, leveraging the prowess of Llama3 to outshine competitors, including GPT3.5 and, in several benchmarks, GPT4. This innovation introduces a cost-effective and efficient approach to model training, with expenses capped at $500. Llama3-V boasts a 10-20% performance improvement over Llava, the leading open-source model for multimodal understanding, and holds its ground against significantly larger models like GPT4v, Gemini Ultra, and Claude Opus.
The model’s architecture is ingeniously designed to integrate visual information with textual content. Utilizing the SigLIP model for image embedding, it aligns these embeddings with textual tokens through a projection block featuring two self-attention blocks. This alignment facilitates the creation of a joint input representation, which is then processed by Llama3. The model’s efficiency is further enhanced by strategic optimizations, including a caching mechanism and MPS/MLX optimizations, which streamline the training and inference processes.
Llama3-V’s training framework is both innovative and resourceful, employing a precomputed embedding approach from SigLIP and focusing on updating the projection matrix during pre-training. Supervised fine-tuning further refines the model’s performance, ensuring that Llama3-V not only sets a new standard for multimodal models but does so with unprecedented cost-efficiency and scalability.
Read more at Medium…