Nvidia has introduced a new tool that enables owners of GeForce RTX 30 Series and 40 Series cards to run an AI-powered chatbot offline on a Windows PC. This tool, named Chat with RTX, offers users the ability to customize a GenAI model similar to OpenAI’s ChatGPT by connecting it to documents, files, and notes for querying.
Key Takeaway
Nvidia’s Chat with RTX tool enables users to run AI-powered chatbots offline on Windows PCs, offering customization of GenAI models and support for various text-based models.
Customizable GenAI Model
Chat with RTX allows users to type queries, such as “What was the restaurant my partner recommended while in Las Vegas?” The tool then scans local files specified by the user and provides the answer with context. It defaults to AI startup Mistral’s open source model but also supports other text-based models, including Meta’s Llama 2. However, users should be aware that downloading all the necessary files may consume a significant amount of storage, ranging from 50GB to 100GB, depending on the selected model(s).
Supported Formats and Features
Currently, Chat with RTX supports text, PDF, .doc, .docx, and .xml formats. Users can point the app at a folder containing any supported files to load them into the model’s fine-tuning dataset. Additionally, the tool can take the URL of a YouTube playlist to load transcriptions of the videos in the playlist, allowing the selected model to query their contents.
Limitations and Considerations
Despite its capabilities, Chat with RTX has certain limitations. It does not remember context, meaning it won’t consider previous questions when answering follow-up questions. Nvidia also acknowledges that the relevance of the app’s responses can be affected by various factors, including question phrasing, model performance, and the size of the fine-tuning dataset. As a result, Chat with RTX is more of a toy than a production tool.
