Agentic Conversations (formally mlops.community)

Latest Episodes:

Collaboration and Strategy // Vin Vashishta // #176 Sep 18, 2023

Show notes

MLOps Coffee Sessions #176 with Vin Vashishta, Collaboration and Strategy.

// Abstract

From the significance of technical strategists to the crucial role of product managers with a deep understanding of data and AI products, Vin shares invaluable insights on fostering collaboration, driving strategy, and maximizing the potential of data within organizations. Join us as we explore the importance of becoming multipliers in our fields, the power of effective strategy in leveraging data, and the opportunities that lie in the generative AI era.

// Bio

Vin's background is in applied data science. He is the founder of V Squared, one of the oldest and smallest data science consulting companies in the world. They help businesses monetize data and AI. Vin is the author of From Data to Profit. He teaches technical strategy and data, and AI product management.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://www.datascience.vin/

From Data to Profit: How Businesses Leverage Data to Grow Their Top and Bottom Lines book:https://www.amazon.com/Data-Profit-Businesses-Leverage-Bottom/dp/1394196210/⁠

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Vin on LinkedIn: https://www.linkedin.com/in/vineetvashishta/

Timestamps:

[00:00] Vin's preferred coffee

[00:14] Takeaways

[02:02] Please like, share, and subscribe to our MLOps channels!

[02:28] Recent ideas of Vin

[05:09] Understanding the business value of any project [08:30] Generative AI making things faster

[14:29] Strategy in practice

[20:19] Practicality and Credibility of Strategists

[22:42] Coming soon!!! LLMs in Production Conference Panel Part III

[27:48] Becoming a Multiplier

[29:03] The AI Product Manager

[35:12] Successful monetization and integration of technologies

[37:48] Justifying the ROI of LLMs

[44:59] Adding that extra value

[49:52] Read Vin's book linked above!

[50:35] Wrap up

Ux of an LLM User Panel // LLMs in Production Conference Part II Sep 15, 2023

Show notes

#180 with LLMs in Production Conference part 2 Ux of a LLM User Panel, Misty Free, Dina Yerlan, and Artem Harutyunyan hosted by Innovation Endeavors' Davis Treybig. // Abstract Explore different approaches to interface design, emphasizing the significance of crafting effective prompts and addressing accuracy and hallucination issues. Discover some strategies for improving latency and performance, including monitoring, scaling, and exploring emerging technologies. // Bio Misty Free Misty Free is a product manager at Jasper, where she focuses on supercharging marketers with speed and consistency in their marketing campaigns, with the power of AI. Misty has also collaborated with Stability and OpenAI to offer AI image generation within Jasper. She approaches product development with a "jobs-to-be-done" mindset, always starting with the "why" behind any need, ensuring that customer pain points are directly addressed with the features shipped at Jasper. In her free time, Misty enjoys crocheting amigurumi, practicing Spanish on Duolingo, and spending quality time with her family. Misty will be on a panel sharing her insights and experiences on the real-world use cases of LLMs. Davis Treybig Davis is a partner at Innovation Endeavors, an early-stage venture firm focused on teams solving hard technical & engineering problems. He personally focuses on computing infrastructure, AI/ML, and data. Dina Yerlan Head of Product, Generative AI Data at Adobe Firefly (family of foundation models for creatives). Artem Harutyunyan Artem is the Co-Founder & CTO at Bardeen AI. Prior to Bardeen, he was in engineering and product roles at Mesosphere and Qualys, and before that, he worked at CERN. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠Website: https://www.angellist.com/venture/relay Foundation by Isaac Asimov: https://www.amazon.com/Foundation-Isaac-Asimov/dp/0553293354 AngelList Relay blog: https://www.angellist.com/blog/introducing-angellist-relay --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Davis on LinkedIn: https://www.linkedin.com/in/davistreybig/ Connect with Misty on LinkedIn: https://www.linkedin.com/in/misty-miglorin/ Connect with Dina on LinkedIn: https://www.linkedin.com/in/dinayerlan/ Connect with Artem on LinkedIn: https://www.linkedin.com/in/artemharutyunyan/

From Virtualization to AI Integration // Lamia Youseff // # 175 Sep 12, 2023

Show notes

MLOps Coffee Sessions #175 with Lamia Youseff, From Virtualization to AI Integration.

// Abstract

Lamia discusses how both Fortune 500 companies and SMBs lack the knowledge and capabilities to identify which use cases in their systems can benefit from AI integration. She emphasizes the importance of helping these companies integrate AI effectively and acquire the necessary capabilities to stay competitive in the market.

// Bio

By way of an introduction, Dr. Lamia Youseff has been working in AI / ML for ~25 years, first in academia (MIT, Stanford, UCSB), then large tech (Google, Microsoft, Apple, and Facebook), and most recently with startups in Generative AI. She is currently the executive director of JazzComputing, a Visiting Research Scientist at Stanford University in Computer Science and AI, and a research affiliate with MIT Computer Science and Artificial Intelligence Lab (CSAIL). Dr. Youseff earned her Ph.D. in computer science by studying computationally intensive workloads (such as AI / ML and HPC / Scientific Codes) and has built/led several AI teams as an executive and leader at large tech companies over the years (Google, Facebook, Microsoft, and Apple). She also earned her Master's in business management, strategy, and leadership from Stanford Graduate School of Business (GSB), where she is a guest lecturer today. Dr. Youseff regularly writes and speaks about AI and Machine Learning evolution at CIO/CTO/CEO summits.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

⁠--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Lamia on LinkedIn: https://www.linkedin.com/in/lyouseff/

Timestamps:

[00:00] Lamia's preferred coffee

[01:12] Takeaways

[03:00] Please like, share, and subscribe to our MLOps channels!

[03:20] Lamia's background

[09:52] Getting into Google Cloud

[13:10] The Google Cloud project

[16:38] The world before Kubernetes

[19:25] Evolution of virtualization

[23:20] Cloud evolution

[28:13] Kubernetes enables the ecosystem

[32:38] Multiple systems for machine learning

[34:40] Standardization for a greater good

[39:50] Complexity and pain points of ML in production

[46:26] JazzComputing

[50:33] Bridging gaps in AI implementation and investment

[51:19] Wrap up

LLM on K8s Panel // LLMs in Conference in Production Conference Part II Sep 08, 2023

Show notes

MLOps Coffee Sessions #178 with LLMs in Production Conference part 2 LLM on K8s Panel, Manjot Pahwa, Rahul Parundekar, and Patrick Barker hosted by Outerbounds, Inc.'s Shrinand Javadekar. // Abstract Large Language Models require a new set of tools... or do they? K8s is a beast and we like it that way. How can we best leverage all the battle-hardened tech that K8s has to offer to make sure that our LLMs go brrrrrrr. Let's talk about it in this chat. // Bio Shrinand Javadekar Shri Javadekar is currently an engineer at Outerbounds, focussed on building a fully managed, large-scale platform for running data-intensive ML/AI workloads. Earlier, he spent time trying to start an MLOps company for which he was a co-founder and head of engineering. He led the design, development, and operations of Kubernetes-based infrastructure at Intuit, running thousands of applications, built by hundreds of teams and transacting billions of $$. He has been a founding engineer of the Argo open-source project and also spent precious time at multiple startups that were acquired by large organizations like EMC/Dell and VMWare. Manjot Pahwa Manjot is an investor at Lightspeed India and focuses on SaaS and enterprise tech. She has had an operating career of over a decade within the space of fintech, SaaS, and developer tools spanning various geos such as the US, Singapore, and India. Before joining Lightspeed, Manjot headed Stripe in India, successfully obtaining the payment aggregator license, growing the team from ~10 to 100+, and driving acquisitions in the region during that time. Rahul Parundekar Rahul has 13+ years of experience building AI solutions and leading teams. He is passionate about building Artificial Intelligence (A.I.) solutions for improving the Human Experience. He is currently the founder of A.I. Hero - a platform to help you fix and enrich your data with ML. At AI Hero, he has also been a big proponent of declarative MLOps - using Kubernetes to operationalize the training and serving lifecycle of ML models and has published several tutorials on his Medium blog. Before AI Hero, he was the Director of Data Science (ML Engineering) at Figure-Eight (acquired by Appen), a data annotation company, where he built out a data pipeline and ML model serving architecture serving 36 models (NLP, Computer Vision, Audio, etc.) and traffic of up to 1M predictions per day. Patrick Barker Patrick started his career in Big Data back when that was cool, then moved into Kubernetes near its inception. He has put major features into the Kubernetes API and built several platforms on top of it. In recent years he has moved into AI, with a focus on distributed machine learning. He is now working with a startup to reshape the world of AI agents. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠Website: https://www.angellist.com/venture/relay Foundation by Isaac Asimov: https://www.amazon.com/Foundation-Isaac-Asimov/dp/0553293354 AngelList Relay blog: https://www.angellist.com/blog/introducing-angellist-relay --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Shri on LinkedIn: https://www.linkedin.com/in/shrijavadekar/ Connect with Manjot on LinkedIn: https://www.linkedin.com/in/manjotpahwa/ Connect with Rahul on LinkedIn: https://www.linkedin.com/in/rparundekar/ Connect with Patrick on LinkedIn: https://www.linkedin.com/in/patrickbarkerco/

Harnessing MLOps in Finance // Michelle Marie Conway // #174 Sep 05, 2023

Show notes

MLOps Coffee Sessions #174 with Michelle Marie Conway, Harnessing MLOps in Finance: Bringing Statistical Models to Life for Positive Impact, co-hosted by Stephen Batifol.

// Abstract

Michelle Marie Conway joins hosts Stephen Batifol and Demetrios to share their insights and experiences in the tech industry. Michelle emphasizes the importance of constant learning and adaptation in the rapidly changing tech industry. They discuss the need to stay up to date with the latest documentation, understand code logic, and be mindful when writing code. Michelle also reflects on their experiences as one of the few women in their university math class and often being the only woman on their team in the workplace. They discuss the need for more girls to pursue STEM subjects in schools and the importance of allies in the workplace. Additionally, Michelle explores the benefits and challenges of AI tools, sharing their experiences with tools like Gen AI and ChatGPT. While AI tools enhance productivity, Michelle also acknowledges the limitations of these tools in more technical tasks and the continued reliance on developer resources. This episode offers valuable insights into the importance of continuous learning, gender diversity in STEM, and the potential of AI tools in the field of MLOps.

// Bio

As an Irish woman who relocated to London after completing her university studies in Dublin, Michelle spent the past 12 years carving out a career in the data and tech industry. With a keen eye for detail and a passion for innovation, she has consistently leveraged my expertise to drive growth and deliver results for the companies she has worked for. As a dynamic and driven professional, Michelle is always looking for new challenges and opportunities to learn and grow, and she's excited to see what the future holds in this exciting and ever-evolving industry.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links⁠

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Stephen on LinkedIn: https://www.linkedin.com/in/stephen-batifol/

Connect with Michelle on LinkedIn: https://www.linkedin.com/in/michelle-conway-40337432

Timestamps:

[00:00] Michelle's preferred coffee

[02:04] Takeaways

[05:18] Please like, share, and subscribe to our MLOps channels!

[06:18] Michelle's journey in tech

[07:49] Engineering best practices

[09:38] Getting comfortable with the hump

[11:22] Clean coding fundamentals

[13:29] Working with the people

[14:09] GCP migration

[18:00] GCP migration length of journey

[18:38] Moving data focus

[19:18] Effectiveness of running 2 systems

[21:00] Dealing with discrepancies

[22:15] Using Nexus

[24:04] Migrating data from Teradata to BigQuery, strict security

[28:48] Hiring new people

[30:56] Securely managing financial data with millions of customers

[32:30] When things go wrong

[35:08] Finding the root cause

[36:28] Dealing with the producers' problems

[40:46] Rapid tech evolution, constant learning

[44:44] Teaching Python, using Gen AI for tasks

[46:34] Dealing with LLMs use cases

[49:15] Dealing with stakeholders and MLOps teams

[51:17] Having a translator

[52:18] Being a woman in the tech industry

[55:11] Encourage more girls in STEM, support women

[56:36] Women in the conversation on tech and female representation

[1:03:49] Wrap up

MLOps vs. LLMOps Panel // LLMs in Conference in Production Conference Part II Sep 01, 2023

Show notes

MLOps Coffee Sessions #176 with MLOps vs. LLMOps Panel, Willem Pienaar, Chris Van Pelt, Aparna Dhinakaran, and Alex Ratner hosted by Richa Sachdev. // Abstract What do MLOps and LLMOps have in common? What has changed? Are these just new buzzwords or is there validity in calling this ops something new? // Bio Richa Sachdev A passionate and impact-driven leader whose expertise spans leading teams, architecting ML and data-intensive applications, and driving enterprise data strategy. Richa has worked for a Tier A Start-up developing feature platforms and in financial companies, leading ML Engineering teams to drive data-driven business decisions. Richa enjoys reading technical blogs focussed on system design and plays an active role in the MLOps Community. Willem Pienaar Willem is the creator of Feast, the open-source feature store and a builder in the generative AI space. Previously Willem was an engineering manager at Tecton where he led teams in both their open source and enterprise initiatives. Before that Willem built the core ML systems and created the ML platform team at Gojek, the Indonesian decacorn. Chris Van Pelt Chris Van Pelt is a co-founder of Weights & Biases, a developer MLOps platform. In 2009, Chris founded Figure Eight/CrowdFlower. Over the past 12 years, Chris has dedicated his career optimizing ML workflows and teaching ML practitioners, making machine learning more accessible to all. Chris has worked as a studio artist, computer scientist, and web engineer. He studied both art and computer science at Hope College. Aparna Dhinakaran Aparna Dhinakaran is the Co-Founder and Chief Product Officer at Arize AI, a pioneer and early leader in machine learning (ML) observability. A frequent speaker at top conferences and thought leader in the space, Dhinakaran was recently named to the Forbes 30 Under 30. Before Arize, Dhinakaran was an ML engineer and leader at Uber, Apple, and TubeMogul (acquired by Adobe). During her time at Uber, she built several core ML Infrastructure platforms, including Michelangelo. She has a bachelor’s from UC Berkeley's Electrical Engineering and Computer Science program, where she published research with Berkeley's AI Research group. She is on a leave of absence from the Computer Vision Ph.D. program at Cornell University. Alex Ratner Alex Ratner is the co-founder and CEO at Snorkel AI, and an Affiliate Assistant Professor of Computer Science at the University of Washington. Prior to Snorkel AI and UW, he completed his Ph.D. in CS advised by Christopher Ré at Stanford, where he started and led the Snorkel open source project, and where his research focused on defining and forwarding the concept of “data-centric AI”, the idea that labeling and developing data is the new center of the AI development workflow. His academic work focuses on data-centric AI and related topics in data management and statistical learning techniques, and applications to real-world problems in medicine, science, and more. Previously, he earned his A.B. in Physics from Harvard University. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Richa on LinkedIn: https://www.linkedin.com/in/richasachdev/ Connect with Willem on LinkedIn: https://www.linkedin.com/in/willempienaar/ Connect with Chris on LinkedIn: https://www.linkedin.com/in/chrisvanpelt/ Connect with Aparna on Twitter: https://www.linkedin.com/in/aparnadhinakaran/ Connect with Alex on Twitter: https://www.linkedin.com/in/alexander-ratner-038ba239/

Building Cody, an Open Source AI Coding Assistant // Beyang Liu // #173 Aug 29, 2023

Show notes

MLOps Coffee Sessions #173 with Beyang Liu, Building Cody, an Open Source AI Coding Assistant.

We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O

// Abstract

Root about the development of Cody, an open-source AI coding assistant. Cody empowers developers to query and comprehend code within codebases through the integration of robust language model capabilities. Sourcegraph tackles the intricacies of understanding intricate codebases by creating comprehensive code maps and employing AI for advanced search functionalities. Cody harnesses the potential of AI to offer features such as code exploration, natural language queries, and AI-powered code generation, augmenting developer productivity and code comprehension.

// Bio

Beyang Liu is the CTO and Co-founder of Sourcegraph. Prior to Sourcegraph, Beyang was an engineer at Palantir Technologies, building large-scale data analysis tools for Fortune 500 companies with large, complex codebases. Beyang studied computer science at Stanford, where he discovered his love for compilers and published some machine learning research as a member of the Stanford AI Lab.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related

Links⁠Website: https://beyang.com

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Beyang on LinkedIn: https://www.linkedin.com/in/beyang-liu/

Timestamps:

[00:00] Beyang's preferred coffee

[00:19] Takeaways

[01:25] Please like, share, and subscribe to our MLOps channels!

[01:48] Beyang background before Sourcegraph

[03:10] War stories

[04:30] Technological tool solution

[06:41] Landscape change in the past 10 years

[09:32] Code search engine evolution

[16:28] Vector databases

[17:40] Actual tech breakdown

[19:52] Incorporating AI into products amid organizational challenges

[25:39] Breaking down Cody

[28:04] Context fetching

[30:44] AI replicating human code understanding?

[36:22] Key for software creation

[40:26] Speak the language

[42:20] Leveraging LLMs

[44:18] Low code, no code movement

[47:54] Reliability issues amongst agents

[53:12] LLMs used in code and chat generation

[56:12] Dealing with rate limits and followers or failovers

[57:33] Unnecessary comparison

[1:00:26] Wrap up

Evaluation Panel // Large Language Models in Production Conference Part II Aug 25, 2023

Show notes

MLOps Coffee Sessions #174 with Evaluation Panel, Amrutha Gujjar, Josh Tobin, and Sohini Roy hosted by Abi Aryan. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O // Abstract Language models are very complex thus introducing several challenges in interpretability. The large amounts of data required to train these black-box language models make it even harder to understand why a language model generates a particular output. In the past, transformer models were typically evaluated using perplexity, BLEU score, or human evaluation. However, LLMs amplify the problem even further due to their generative nature thus making them further susceptible to hallucinations and factual inaccuracies. Thus, evaluation becomes an important concern. // Bio Abi Aryan Machine Learning Engineer @ Independent Consultant Abi is a machine learning engineer and an independent consultant with over 7 years of experience in the industry using ML research and adapting it to solve real-world engineering challenges for businesses for a wide range of companies ranging from e-commerce, insurance, education and media & entertainment where she is responsible for machine learning infrastructure design and model development, integration and deployment at scale for data analysis, computer vision, audio-speech synthesis as well as natural language processing. She is also currently writing and working in autonomous agents and evaluation frameworks for large language models as a researcher at Bolkay. Amrutha Gujjar CEO & Co-Founder @ Structured Amrutha Gujjar is a senior software engineer and CEO and co-founder of Structured, based in New York. With a Bachelor of Science in Computer Science from the University of Washington's Allen School of CSE, she brings expertise in software development and leadership to my work.

Connect with Amrutha on LinkedIn to learn more about her experience and discuss exciting opportunities in software development and leadership.

Josh Tobin

Founder @ GantryJosh Tobin is the founder and CEO of Gantry. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel.

Sohini Roy

Senior Developer Relations Manager @ NVIDIASohini Bianka Roy is a senior developer relations manager at NVIDIA, working within the Enterprise Product group. With a passion for the intersection of machine learning and operations, Sohini specializes in the domains of MLOps and LLMOps. With her extensive experience in the field, she plays a crucial role in bridging the gap between developers and enterprise customers, ensuring smooth integration and deployment of NVIDIA's cutting-edge technologies. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Abi on LinkedIn: https://www.linkedin.com/in/abiaryan/ Connect with Amrutha on LinkedIn: https://www.linkedin.com/in/amruthagujjar/ Connect with Josh on LinkedIn: https://www.linkedin.com/in/josh-tobin-4b3b10a9/ Connect with Sohini on Twitter: https://twitter.com/biankaroy_

FrugalGPT: Better Quality and Lower Cost for LLM Applications // Lingjiao Chen // #172 Aug 22, 2023

Show notes

MLOps Coffee Sessions #172 with Lingjiao Chen, FrugalGPT: Better Quality and Lower Cost for LLM Applications.

This episode is sponsored by QuantumBlack.

We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O

// Abstract

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade that learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g., GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

// Bio

Lingjiao Chen is a Ph.D. candidate in the Computer Science department at Stanford University. He is broadly interested in machine learning, data management, and optimization. Working with Matei Zaharia and James Zou, he is currently exploring the fast-growing marketplaces of artificial intelligence and data. His work has been published at premier conferences and journals such as ICML, NeurIPS, SIGMOD, and PVLDB, and partially supported by a Google fellowship.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links⁠Website: https://lchen001.github.io/

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance paper: https://arxiv.org/abs/2305.05176

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Lingjiao on LinkedIn:

Timestamps:

[00:00] Lingjiao's preferred coffee

[00:35] Takeaways

[02:41] Sponsor Ad: Nayur Khan of QuantumBlack

[05:27] Lingjiao's research at Stanford

[07:51] Day-to-day research overview

[10:11] Inventing data management inspired abstractions research

[13:58] Agnostic Approach to Data Management

[15:56] Frugal GPT

[18:59] Just another data provider

[19:51] Frugal GPT breakdown

[26:33] First step of optimizing the prompts

[28:04] Prompt overlap

[29:06] Query Concatenation

[32:30] Money saving

[35:04] Economizing the prompts

[38:52] Questions to accommodate

[41:33] LLM Cascade

[47:25] Frugal GPT saves cost and improves performance

[51:37] End-user implementation

[52:31] Completion Cache

[56:33] Using a vector store

[1:00:51] Wrap up

Building LLM Products Panel // LLMs in Production Conference Part II Aug 18, 2023

Show notes

MLOps Coffee Sessions #172 with LLMs in Production Conference part 2 Building LLM Products Panel, George Mathew, Asmitha Rathis, Natalia Burina, and Sahar Mor Using hosted by TWIML's Sam Charrington. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O // Abstract There are key areas we must be aware of when working with LLMs. High costs and low latency requirements are just the tip of the iceberg. In this panel, we hear about common pitfalls and challenges we must keep in mind when building on top of LLMs. // Bio Sam Charrington Sam is a noted ML/AI industry analyst, advisor and commentator, and host of the popular TWIML AI Podcast (formerly This Week in Machine Learning and AI). The show is one of the most popular Tech podcasts with nearly 15 million downloads. Sam has interviewed over 600 of the industry’s leading machine learning and AI experts and has conducted extensive research into enterprise AI adoption, MLOps, and other ML/AI-enabling technologies. George Mathew George is a Managing Director at Insight Partners focused on venture-stage investments in AI, ML, Analytics, and Data companies as they are establishing product/market Fit. Asmitha Rathis Asmitha is a Machine Learning Engineer with experience in developing and deploying ML models in production. She is currently working at an early-stage startup, PromptOps, where she is building conversational AI systems to assist developers. Prior to her current role, she was an ML engineer at VMware. Asmitha holds a Master’s degree in Computer Science from the University of California, San Diego, with a specialization in Machine Learning and Artificial Intelligence. Natalia Burina Natalia is an AI Product Leader who was most recently at Meta, leading Responsible AI. During her time at Meta, she led teams working on algorithmic transparency and AI Privacy. In 2017 Natalia was recognized by Business Insider as “The Most Powerful Female Engineer in 2017”. Natalia was also an Entrepreneur in Residence at Foundation Capital, advising portfolio companies and working with partners on deal flow. Prior to this, she was the Director of Product for Machine Learning at Salesforce, where she led teams building a set of AI capabilities and platform services. Prior to Facebook and Salesforce, Natalia led product development at Samsung, eBay, and Microsoft. She was also the Founder and CEO of Parable, a creative photo network bought by Samsung in 2015. Natalia started her career as a software engineer after pursuing Bachelor's degree in Applied and Computational Mathematics from the University of Washington. Sahar Mor Sahar is a Product Lead at Stripe with 15y of experience in product and engineering roles. At Stripe, he leads the adoption of LLMs and the Enhanced Issuer Network - a set of data partnerships with top banks to reduce payment fraud. Prior to Stripe he founded a document intelligence API company, was a founding PM in a couple of AI startups, including an accounting automation startup (Zeitgold, acq'd by Deel), and served in the elite intelligence unit 8200 in engineering roles. Sahar authors a weekly AI newsletter (AI Tidbits) and maintains a few open-source AI-related libraries (https://github.com/saharmor). // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Our TOPPODCAST Picks

Follow Us

Stay Connected

Agentic Conversations (formally mlops.community)

Related Podcasts

Links

Stay Connected