As the demand for large language models (LLMs) continues to rise, business leaders and IT decision-makers are faced with the challenge of launching and running these AI-powered chatbots. DeepInfra, a promising new company founded by former engineers at IMO Messenger, aims to address this challenge by providing a cost-effective solution. In an exclusive announcement to VentureBeat, DeepInfra revealed its plans to set up LLM chatbots on private servers and offer them to customers at an aggressively low rate of $1 per 1 million tokens in or out. This is significantly lower compared to the prices offered by OpenAI’s GPT-4 Turbo and Anthropic’s Claude 2.
The demand for DeepInfra’s services is reflected in its recent $8 million seed round, led by A.Capital and Felicis. With this funding, DeepInfra plans to offer a range of open source model inferences to customers, including Meta’s Llama 2 and CodeLlama, as well as customized versions of these models. The company’s goal is to provide value on the inference side of AI, as much attention has been focused on training large language models, while the importance of reliable and efficient inferencing has been overlooked.
The Challenge of Concurrent Users and Resource Optimization
One of the challenges in serving a model to multiple users simultaneously is optimizing the usage of hardware resources. The computation and memory bandwidth required for generating tokens in large language models pose challenges when handling multiple users. DeepInfra’s co-founder and CEO, Nikola Borisov, explained that their prior experience in running large fleets of servers with connectivity around the world helped them develop efficient infrastructure to serve millions of people. This experience allows DeepInfra to optimize server space and maximize resource efficiency, resulting in significantly lower costs for its customers.
DeepInfra’s low-cost solution has the potential to disrupt the market, as cost is often a limiting factor in implementing AI and LLMs. Small-to-medium sized businesses (SMBs) in particular can benefit from DeepInfra’s affordability, allowing them to leverage LLM technology in their applications and experiences. With a close eye on the open source AI community, DeepInfra plans to capitalize on the advancements and new models being released in order to continually enhance their offerings.
Data Privacy and Security: DeepInfra’s Commitment
In addition to cost savings and performance, DeepInfra also prioritizes data privacy and security. Borisov emphasized that they do not store or use any user prompts, ensuring that customer data is immediately discarded once the chat window is closed. This commitment to privacy and security will likely appeal to enterprises with strict data protection requirements.
Overall, DeepInfra’s innovative approach to launching and running AI language models is poised to revolutionize the industry. By offering a cost-effective solution and prioritizing performance, efficiency, and security, DeepInfra aims to empower businesses of all sizes to harness the power of AI and enhance their operations.
