DataStax, a leading commercial vendor behind the open-source Apache Cassandra database, has released a new data API to simplify the development of generative AI retrieval augmented generation (RAG) applications. The API aims to bridge the gap between DataStax and purpose-built vector databases, enabling developers to access the database using Python and JavaScript programming languages.
Enhancing RAG Applications with Vector Database Capability
Vector database capability plays a critical role in enabling RAG applications, which combine large language models (LLMs) with data platforms to generate highly accurate and customized results. While DataStax has previously incorporated vector capabilities into its AstraDB cloud database, developers were still required to work with the Cassandra Query Language (CQL) as the primary method for querying the data. The new data API eliminates this limitation by providing developers the flexibility to use Python and JavaScript, narrowing the gap between DataStax and dedicated vector databases like Pinecone.
“There has been a kind of tug of war between the native vector databases that don’t support any other query type other than vectors and the hybrid databases that have very robust query models,” said Ed Anuff, Chief Product Officer at DataStax.
By introducing the data API, DataStax aims to reduce the impedance mismatch between developers’ requirements and the capabilities provided by the database. Since the introduction of vector capabilities in AstraDB, approximately half of all new users have utilized the cloud database to build gen AI applications. However, developers faced challenges in using their preferred programming languages, namely Python and JavaScript, to access AstraDB. Previously, developers building AI applications would have to rely on the Cassandra Query Language (CQL), which demanded extensive data modeling knowledge and did not optimize queries for vector data.
The new data API automates vectorization, simplifies the interface in Python and JavaScript, and optimizes performance by efficiently storing and indexing vector data at the database level,” explained Ed Anuff.
The new data API presents a simplified JSON-based data format for developers, eliminating the need for complex data modeling knowledge. By leveraging the underlying Cassandra data architecture, which is designed around high-performance primitives, the API enables deep integration with the database, resulting in improved overall query performance.
JVector Search Engine and Open-Source Commitment
Alongside the data API, DataStax has also developed the JVector search engine as part of AstraDB. JVector is an open-source embedded vector search engine that utilizes the DiskANN algorithm, an optimized version of the approximate nearest neighbor search (ANN) algorithm. The JVector engine enhances the relevancy and recall capabilities of AstraDB, outperforming other vector databases.
DataStax is committed to open-source collaboration and aims to make its vector work, including JVector and the data API, available to the Cassandra open-source community and AstraDB customers. By doing so, DataStax aims to provide developers with the easiest path to leverage these technologies and make informed choices when selecting cloud services.
“We’re very strongly committed to making stuff available to open source ecosystems,” emphasized Ed Anuff. “We also just want to make sure that if you’re just the developer trying to figure out what cloud service you should use, that you’ve got the easiest path for that.”
