OpenAI’s recently announced GPT-4V, an AI model capable of understanding both text and images, has generated significant excitement in the field of artificial intelligence. However, concerns about its flaws and potential risks have prompted the development of alternative open source projects. These projects aim to provide similar functionalities while addressing some of the limitations of GPT-4V. Let’s take a closer look at two of these challengers and how they compare.
Key Takeaway
OpenAI’s GPT-4V, while promising, has its limitations and potential risks. Open source projects like LLaVA-1.5 and Adept offer alternative multimodal models that aim to provide similar functionalities with certain improvements. These projects address the limitations of GPT-4V and offer more accessible options for developers to experiment with.
The Power of Multimodal Models
Unlike models that focus solely on text or images, multimodal models like GPT-4V can combine both modalities to enhance their capabilities. For example, these models can provide instructions that are easier to understand through visual demonstrations, such as repairing a bicycle. Additionally, they can go beyond simple image recognition and offer suggestions based on the content of images, like recommending recipes using ingredients from a photographed refrigerator.
However, along with their potential benefits, multimodal models also introduce new risks. OpenAI initially delayed the release of GPT-4V due to concerns that it could be used for unauthorized identification of individuals in images. Furthermore, GPT-4V has been found to have significant flaws, including an inability to recognize hate symbols and a tendency to exhibit discrimination against certain demographics, sexes, and body types, as pointed out by OpenAI itself.
Open Source Alternatives
Despite the risks, both companies and independent developers have been actively working on open source projects to create alternative multimodal models. While these models may offer a slightly different feature set compared to GPT-4V, they can still accomplish many, if not most, of the same tasks.
One such project is LLaVA-1.5, a collaboration between researchers from the University of Wisconsin-Madison, Microsoft Research, and Columbia University. LLaVA-1.5 enables users to ask questions about images, similar to GPT-4V. The project aims to make it easier for developers to get started with multimodal models by ensuring compatibility with consumer-level hardware. Unlike the more resource-intensive GPT-4V, LLaVA-1.5 can be run on a GPU with less than 8GB of VRAM.
Another noteworthy multimodal model is being developed by Adept, a startup focused on autonomous web and software navigation. Their model, Fuyu-8B, is not designed to compete directly with LLaVA-1.5, but instead aims to showcase Adept’s in-house advancements and gather feedback from the developer community. Fuyu-8B specifically focuses on understanding unstructured data, such as user interfaces, charts, and diagrams.
While these open source alternatives offer exciting possibilities, they also come with their own limitations. LLaVA-1.5, as tested by researchers, demonstrated strengths in object detection and contextualizing images but struggled with more complex scenarios involving multiple objects or text recognition. On the other hand, Fuyu-8B has shown promise in image understanding and data extraction but may still carry some similar flaws present in GPT-4V.
Nevertheless, the release of these open source projects provides a more accessible avenue for developers to experiment with multimodal models. By embracing the open source approach, these projects encourage the community to build upon their foundations and explore a wide range of use cases. However, it is essential for developers to carefully consider the potential risks and limitations involved in using these models.
