API company Kong has announced the launch of its open source AI Gateway, an extension of its existing API gateway. This new gateway is designed to make it easier for developers and operations teams to integrate their applications with one or more large language models (LLMs) and access them through a single API. Alongside this release, Kong is also introducing several AI-specific features, including prompt engineering and credential management.
Key Takeaway
Kong’s new open source AI Gateway simplifies the integration of large language models (LLMs) into applications, providing developers with the ability to access them through a single API.
AI as an Additional Use Case for APIs
Kong co-founder and CTO Marco Palladino emphasized that AI is an additional use case for APIs. He highlighted that APIs are driven by various use cases such as mobile and microservices, with AI being the latest addition. Palladino pointed out that AI usage involves consuming AI through APIs, fine-tuning AI, and managing AI itself. As AI adoption grows, so does the usage of APIs.
Addressing Data Privacy Concerns
Palladino also addressed the growing concern of data privacy in AI usage. He noted that organizations are apprehensive about data leaks and are increasingly interested in running AI models locally while using the cloud as a backup. However, managing credentials for accessing cloud-based models, controlling and logging traffic, and managing quotas are immediate challenges that need to be addressed.
Enhancing Developer Productivity
Kong’s goal with the AI Gateway is to enable developers to be more productive when building on AI. By leveraging the gateway, developers can consume one or more LLM providers without having to modify their code. The gateway currently supports Anthropic, Azure, Cohere, Meta’s LLaMA models, Mistral, and OpenAI.
Enabling New Use Cases with AI-Specific Features
The Kong team believes that by incorporating AI-specific features into the API gateway, they can enable new use cases or simplify the implementation of existing ones. The AI response and request transformers allow developers to modify prompts and their results in real-time, enabling tasks such as automatic translation or the removal of personally identifiable information.
Additionally, prompt engineering is deeply integrated into the gateway, allowing businesses to enforce their guidelines on top of these models and providing a central point for managing these guidelines and prompts.
Continued Evolution of Kong’s Platform
Kong’s journey began almost nine years ago with the launch of the Kong API Management Platform. Over time, the company has evolved, with the Kong Gateway now serving as the core of its platform and powering the new AI Gateway. Current Kong users can seamlessly upgrade their installations to access the new AI features.
Notably, the new AI features are currently available for free, with Kong planning to introduce paid premium features in the future. However, the team emphasized that monetization is not the primary focus of this initial release.
