Writer, a San Francisco-based startup, has been making significant strides in the AI field. Despite not receiving as much media attention as other prominent AI companies, Writer’s family of in-house language models (LLMs) called Palmyra have been gaining recognition for their effectiveness in enterprise use cases.
Impressive Performance in Benchmarking
Writer’s Palmyra models have caught the attention of industry leaders and clients. Companies like Accenture, Vanguard, Hubspot, and Pinterest have chosen Writer’s platform powered by Palmyra models for their creativity and productivity needs.
Recently, Stanford HAI’s Center for Research on Foundation Models conducted benchmarking and introduced a new benchmark called HELM Lite. The benchmark focuses on in-context learning, which involves learning a new task from a small set of examples presented within the prompt during inference.
While GPT-4 emerged as the top performer in the benchmark, Palmyra’s X V2 and X V3 models delivered unexpected results despite being smaller models. Percy Liang, director of the Stanford Center for Research on Foundation Models, acknowledged Palmyra’s strong performance in machine translation, where it landed in first place.
Palmyra X from Writer is doing EVEN BETTER than the classic benchmark. We aren’t just the top model in the MMLU benchmark, but the top model in production overall — close second only to the GPT-4 previews that were analyzed. And across translation benchmarks — a NEW test — we’re #1.” – May Habib, Writer CEO
Addressing Enterprise Needs
In an interview with VentureBeat, Writer CEO May Habib highlighted the economic challenges faced by enterprises when implementing large models like GPT-4, which is trained on 1.2 trillion tokens.
“Generative AI use cases [in 2024] are now actually going to have to make economic sense. Enterprises are building use cases on a GPT model and then ‘two or three months later the prompts don’t really work anymore because the model has been distilled, because their own serving costs are so high.'”
Habib emphasized the need for smaller models that align with enterprise budgets and can handle curated training data effectively. She also stressed the importance of real-world enterprise use cases and practitioners, citing Stanford HAI’s benchmarking efforts as being more relevant and practical compared to traditional leaderboards.
Writer was co-founded by May Habib and Waseem AlShikh in 2020 initially as a tool for marketing teams. Building on their expertise in NLP and machine translation from their previous venture Qordoba, founded in 2015, Writer launched Palmyra models in 2023.
The Palmyra lineup consists of Palmyra-Small with 128 million parameters, Palmyra-Base with 5 billion parameters, and Palmyra-Large with 20 billion parameters. To cater to the enterprise market, Writer introduced Knowledge Graph in May 2023, allowing companies to connect business data sources to Palmyra and self-host models based on Palmyra.
“When we say full stack, we mean that it’s the model plus a built-in RAG solution,” explained Habib. “AI guardrails on the application layer and the built-in RAG solution is so important because what folks are really sick and tired of is needing to send all their data to an embeddings model, and then that data comes back, then it goes to a vector database.”
Writer’s new graph-based approach to RAG for building digital assistants grounded in a customer’s data has garnered attention. Habib believes that smaller specialized models are easier to manage and more cost-effective for enterprises.
When questioned about the value of proprietary data in the world of LLMs, Habib pointed out that medical LLM models surpassed GPT-4 in Stanford HAI’s HELM Lite leaderboard. However, she stressed that once a specialized model surpasses the state of the art, factors like inference and cost become crucial for enterprise adoption.
