Lyzr is an Antler backed startup, which builds building blocks & Generative AI powered products. NeoAnalyst, one solution from Lyzr, is an AI data analyst platform, which enables business leaders to interact with data through natural language queries. The existing solution was built on GPT-4. The target audience for the platform being enterprise business users, they faced multiple challenges in enterprise deployments, as they scaled:
- Enterprise customer preferences for an open LLM model over OpenAI’s GPT series
- Ensure the entire platform remains compliant with:
- GDPR regulations, ensuring user data privacy and protection.
- SOC2 standards, guaranteeing security, availability, processing integrity, confidentiality, and privacy of customer data.
Additionally, the NeoAnalyst product required the implementation of an agent-based model for facilitating interactions between multiple agents that collectively provide AI data analyst functionality.
Post evaluation by the GoML Data Scientists, LLaMA2 was the target LLM for migration, which met the above-mentioned criteria. The migration process involved transitioning the existing chatbot system from using ChatGPT’s capabilities by leveraging the enhanced features of LLaMA2.
The overall scope identified by the goML’s GenAI team was broken into 3 parts:
- Migrate core components from GPT-4 to a hosted version of LLaMA2.
- Ensure that the LLaMA2 model, specifically the 7B version, undergoes instruct tuning or fine-tuning.
- Ensure that the LLaMA2 model’s output accuracy and performance either matches or surpasses that of the existing OpenAI LLM models.
There was a need for fine-tuning the LLaMA2-7b chat open-source model & the entire platform was designed to be hosted natively on AWS using a serverless architecture, specifically employing Lambda-based microservices.