Engineering

Last month, engineering leads at Duolingo held our first “engineerication.” (I got the idea from Stripe.) Basically, we took a week off from our regular jobs to sit together and code in order to see what our engineers do daily, get back into the tech stack to see what can be improved. 👩💻 Turns out, a lot has changed in the years since I last wrote a line of code 😅Some things have gotten much easier. Codespaces, for example, allows engineers to easily create a coding environment in the cloud instead of on their local machines. And GitHub Copilot auto-suggests chunks of code. But there were 3 big takeaways I had for how we can improve our engineers’ experiences: 1. On the first day, I set out to do a few small code changes. But as a new engineer, you can’t just sit down and make a change immediately, even a tiny one. You have to set up your environment and figure out where the code needs to go. Then you have to build the code, test it, get it reviewed, and push it live to production. All of this takes time—sometimes, too much time. I want to make it easy for new software engineers to be productive right away without having to set up their development environment or having to learn a lot of arcane tools, processes, and magical incantations. 2. When I started at Duolingo nine years ago, we had a few GitHub repositories. Now we have dozens of microservices. This means that even a small code change may require updates to several repositories. If you’re not regularly working in a given repo, this is much harder, especially if it’s one that your team doesn't own. More standardization is needed to make it easier to code across our various microservices and client repos (we’re starting to do this, but more can be done!) 3. Code review turnaround time slowed us down. We averaged about a 24-hour turnaround, but when making atomic (easily revertable) pull requests, lots of PRs stack up and eventually lead to merge conflicts, etc. In practice, I could see engineers cutting their turnaround time by making PRs that are less atomic, which leads to messier reverts and a harder time diagnosing bugs. Whenever software engineers are tempted to take shortcuts because the current process is too slow or painful, that’s an opportunity for improvement. Overall, taking a week to go hands-on with the code is a great way to build better processes and teams. It was an incredibly valuable week that helped us visualize the steps we can take to remove pain points and speed up some unnecessarily slow processes. 🐢 (One more learning: Next time, I’m going to invite one of our software engineers to pair program with me 😅) Help us develop new products and processes! We're hiring: https://lnkd.in/eSJYjYPG #engineering #processimprovement #leadershipidea

Lesson #5: Build Platforms, Not Tools. Create a Platform Advantage One of the most strategic things that a tech company can do is build a platform advantage. This creates a durable moat for a business. It also creates a built-in incentive for the market to keep coming back to you. So let’s study the concept of a Platform Advantage. A platform can bring several benefits to a business. 1. Foundation On Which Others Build Value: A platform is a foundation on top of which other players in the ecosystem build value. An example of this is application developers building apps for the iOS or Mac or Windows platform. 2. Platform Takes a Minority of the Available Economic Advantage: Typically, a great platform is built in a way that the ecosystem enjoys a large part of the economic opportunity availed by the platform. Going back to IOS or Windows, the revenue captured by all the app developers building on iOS is far greater than what Apple makes on iOS devices or what Microsoft makes on Windows. 3. Decrease in Incremental Effort: A platform advantage is defined as something where value can be realized with far less incremental effort for every subsequent addition for the user/customer. A great example of this is our Meraki platform. It started as a switching platform, but then the management plane was able to also manage cameras in the environment and people just decided to add them because the simplicity of managing everything is that much easier. 4. Sum is Greater than the Parts: The value of a good platform is always greater than the sum of the piece parts. In the security business at Cisco, we used to operate much more as a holding company. Several different products. Several different ways to use and manage the products. Each run by a General Manager. This created a disjointed experience for customers and incongruent objectives between the teams and the customer. And it didn’t take advantage of the breadth of the offering. As we built out the Cisco Security Cloud, all of this started to come together. There is a common design language, a common policy engine, a common set of policy objects, cohesion and predictability in how each component of the platform behaves, etc. Adding a new product to your environment has very low marginal effort. All the piece parts are well integrated. 5. Ecosystem Advantage: A platform delivers an ecosystem advantage. If you think of the Google ecosystem vs the Apple ecosystem, people aren’t making purchase decisions based on every small feature that gets added. When the iPhone 15 is released, only decision I make is whether to upgrade from my iPhone 14 to the newer version. What I’m not doing is evaluating the camera on the Google Pixel. That’s because I also use the Apple Watch, and the iPad and the Mac and the AppleTV and the VisionPro and iTunes and Apple News and they all work in perfect harmony with each other. The platform advantage leverages the power of the ecosystem. Net net, build platforms.

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

Want happy developers? Prioritize these 3 things. Flow. Feedback. Cognitive Load. A very insightful 2023 study published by the Association for Computing Machinery breaks optimal developer experience down into these three practical dimensions: 1. Flow state. Developers do their best work when they can get into the zone. Anything that creates interruptions means that they're taken out of flow state and unable to do their best problem solving. 2. Feedback loops. Devs like to have short feedback loops. Problem solving is inherently an iterative exercise and you want fast iterations. The faster you get feedback, the faster you can move onto the next iteration. Devs get frustrated if code is sitting there for weeks before moving into production. 3. Cognitive load. Even though a developers job is to code, there are many other things they have to deal with that's irrelevant to solving their code problem — e.g. learning the internals of Kubernetes or having to figure out a deployment script someone wrote 5 years ago. All of these things create cognitive load that slows them down. Agree/disagree? What other factors need to be highlighted? #DevelopersExperience #DevEx #Developer

In the last three months alone, over ten papers outlining novel prompting techniques were published, boosting LLMs’ performance by a substantial margin. Two weeks ago, a groundbreaking paper from Microsoft demonstrated how a well-prompted GPT-4 outperforms Google’s Med-PaLM 2, a specialized medical model, solely through sophisticated prompting techniques. Yet, while our X and LinkedIn feeds buzz with ‘secret prompting tips’, a definitive, research-backed guide aggregating these advanced prompting strategies is hard to come by. This gap prevents LLM developers and everyday users from harnessing these novel frameworks to enhance performance and achieve more accurate results. https://lnkd.in/g7_6eP6y In this AI Tidbits Deep Dive, I outline six of the best and recent prompting methods: (1) EmotionPrompt - inspired by human psychology, this method utilizes emotional stimuli in prompts to gain performance enhancements (2) Optimization by PROmpting (OPRO) - a DeepMind innovation that refines prompts automatically, surpassing human-crafted ones. This paper discovered the “Take a deep breath” instruction that improved LLMs’ performance by 9%. (3) Chain-of-Verification (CoVe) - Meta's novel four-step prompting process that drastically reduces hallucinations and improves factual accuracy (4) System 2 Attention (S2A) - also from Meta, a prompting method that filters out irrelevant details prior to querying the LLM (5) Step-Back Prompting - encouraging LLMs to abstract queries for enhanced reasoning (6) Rephrase and Respond (RaR) - UCLA's method that lets LLMs rephrase queries for better comprehension and response accuracy Understanding the spectrum of available prompting strategies and how to apply them in your app can mean the difference between a production-ready app and a nascent project with untapped potential. Full blog post https://lnkd.in/g7_6eP6y

Yesterday I had the pleasure of working with leaders and teachers from L’Anse Creuse School District outside of Detroit for one of our Train-the-Trainer Institutes. We had a great time digging into all things GenAI!   Our 1-day institute focuses on two key PD sessions: Introduction to Generative AI for Educators and Prompting 101. We work to upskill the new trainers on foundational concepts of GenAI, before equipping them with strategies to turnkey this work in their school. In our Prompting 101 session we focus on strategies for getting the best out of popular and powerful free GenAI tools like ChatGPT, Claude, and Gemini.   What's great is there are many different prompt frameworks out there for educators to use - including our 5S Framework: Set the scene (priming), be Specific, Simplify language, Structure output, and Share feedback. We also break down a good prompting in the following four steps: 1.      Clarity is Key   Explicitly state what you would like the model to do. The more specific your prompt, the more accurate and tailored the AI's response will be. General prompts will result in general responses. 2. Pick the Right Prompting Technique You may be able to get what you need from one well-structured prompt (one-shot prompting), but there are other techniques too. You can provide examples in your prompt to guide the AI's responses (few-shot prompting), or cut down your requests into steps (chain-of-thought prompting). 3.      Provide Context   The chatbot is called a "context window" for a reason! Give AI as much necessary background information as possible. This will help it prepare a response that fits your needs.   4.      Format Matters   A well-structured prompt guides the AI in understanding the exact nature of your request. Use clear and direct language, and structure your prompt logically.   So what does that look like in practice for a one-shot prompt?   An OK prompt for educators might look like this:   “Create a lesson plan about multiplying fractions for 5th graders”   A better prompt would look like:   “Act as an expert mathematician and a teacher skilled in designing engaging learning experiences for upper elementary students. Design a lesson plan about multiplying fractions for 5th grade students.”   And an even more effective prompt would be:   “You are an expert mathematician and teacher skilled in Universal Design for Learning. Design an accessible lesson plan about multiplying fractions for 5th grade students interested in soccer. The lesson should include a hands-on activity and frequent opportunities for collaboration. Format your response in a table.”   We take this approach every time we create on of our more than 100 customizable prompts in our Prompt Library. You can check out or complete prompt library here: https://lnkd.in/evExAZSt. AI for Education #teachingwithAI #promptengineering #GenAI #aieducation #aiforeducation

My friend Chip has done it again!!!! Just finished reading AI Engineering! Not going to lie, this is not your typical AI book. It's dense, it's opinionated in the best way, and it's exactly what we needed in the middle of all the noise around GenAI This book is not about “how to prompt better” or “10 tools to build with ChatGPT”. It’s a real engineering guide. You want to understand why RAG works or when to use finetuning over prompt engineering? This book breaks it down The chapters that hit hard for me: 1. Evaluation. Most people don’t talk about how tough it is to evaluate LLMs. Chip dives deep into perplexity, cross entropy, exact match, embedding-based similarity, and even using LLMs to judge other LLMs. There's nuance here. She lays out the limitations, and it’s not sugarcoated. If you're building anything beyond a toy demo, this stuff is critical 2. Prompt Engineering. Way beyond “add examples to your prompt”. Talks about context windows, system prompts, chaining reasoning steps, prompt versioning, and even how to defend against jailbreaks and prompt injection. Real talk for anyone putting a model in front of users 3. RAG and Agents. RAG gets the technical treatment it deserves. Vector stores, retrieval strategies, failure modes, ways to optimize latency — all in there. On the agent side, I appreciated that she didn’t oversell it. Agents can be powerful, sure, but they also fail in weird ways and we’re still figuring them out. This section felt honest 4. Finetuning. The memory math. Quantization. PEFTs. When to merge models. If you’ve ever struggled with GPU limits or ran into model bloat, this chapter hits home. This isn’t “click this button to fine-tune” — it’s “here’s what’s actually going on” 5. Inference optimization. If you’ve worked with LLM latency, you know the pain. This book doesn’t gloss over it. It talks about how to cache, how to route requests, model optimization tricks, service-level tricks, and tradeoffs around hosting vs. calling APIs What I liked most, it’s not trying to hype up AI. It’s showing how to actually build with it. It doesn’t assume you’re at a FAANG company with unlimited infra. It assumes you’re trying to ship real stuff, today, under real constraints And I genuinely believe every engineer building production AI systems should read it. It’s not a light read. It’s a reference manual. And yeah, it’s going to sit on my desk for a long time Chip — hats off. You didn’t write a trend-chasing book. You wrote a field guide for the ones actually building #aiengineering #theravitshow

🩺 RAG and Fine-Tuning: Precision and Personalization in AI 🩺 Consider a highly skilled radiologist with decades of training (Fine-Tuning). This training allows them to accurately interpret medical images based on patterns they've mastered. However, to provide the best diagnosis, they need your specific patient data (RAG), such as images from a recent CT scan. Combining their expertise with this personalized data results in a precise and personalized diagnosis. In AI, Fine-Tuning is similar to the radiologist’s extensive training. It involves adjusting pre-trained models to perform specific tasks with high accuracy. This process uses a large dataset to refine the model’s parameters, making it highly specialized and efficient for particular applications. Retrieval-Augmented Generation (RAG) works like the personalized patient data. RAG integrates external, real-time information into the model’s responses. It retrieves relevant data from various sources during inference, allowing the model to adapt and provide more contextually accurate outputs. How They Work Together: Fine-Tuning: ✅ Purpose: Customizes the base model for specific tasks. ✅ Process: Uses a labeled dataset to refine the model’s parameters. Outcome: Produces a highly accurate and efficient model for the task at hand. RAG: ✅ Purpose: Enhances the model with real-time, relevant information. Process: During inference, it retrieves data from external sources and integrates this data into the model’s responses. ✅ Outcome: Provides contextually relevant and up-to-date outputs, improving the model’s adaptability. Combining Fine-Tuning and RAG creates a powerful AI system. Fine-Tuning ensures deep expertise and accuracy, while RAG adds a layer of real-time adaptability and relevance. This combination allows AI models to deliver precise, contextually aware solutions, much like a skilled radiologist providing a personalized diagnosis based on both their expertise and the latest patient data. #dataengineering #AI #MachineLearning #RAG #FineTuning #DataScience #ArtificialIntelligence

Some of the best AI breakthroughs we’ve seen came from small, focused teams working hands-on, with structured inputs and the right prompting. Here’s how we help clients unlock AI value in days, not months: 1. Start with a small, cross-functional team (4–8 people) 1–2 subject matter experts (e.g., supply chain, claims, marketing ops) 1–2 technical leads (e.g., SWE, data scientist, architect) 1 facilitator to guide, capture, and translate ideas Optional: an AI strategist or business sponsor 2. Context before prompting - Capture SME and tech lead deep dives (recorded and transcribed) - Pull in recent internal reports, KPIs, dashboards, and documentation - Enrich with external context using Deep Research tools: Use OpenAI’s Deep Research (ChatGPT Pro) to scan for relevant AI use cases, competitor moves, innovation trends, and regulatory updates. Summarize into structured bullets that can prime your AI. This is context engineering: assembling high-signal input before prompting. 3. Prompt strategically, not just creatively Prompts that work well in this format: - “Based on this context [paste or refer to doc], generate 100 AI use cases tailored to [company/industry/problem].” - “Score each idea by ROI, implementation time, required team size, and impact breadth.” - “Cluster the ideas into strategic themes (e.g., cost savings, customer experience, risk reduction).” - “Give a 5-step execution plan for the top 5. What’s missing from these plans?” - “Now 10x the ambition: what would a moonshot version of each idea look like?” Bonus tip: Prompt like a strategist (not just a user) Start with a scrappy idea, then ask AI to structure it: - “Rewrite the following as a detailed, high-quality prompt with role, inputs, structure, and output format... I want ideas to improve our supplier onboarding process with AI. Prioritize fast wins.” AI returns something like: “You are an enterprise AI strategist. Based on our internal context [insert], generate 50 AI-driven improvements for supplier onboarding. Prioritize for speed to deploy, measurable ROI, and ease of integration. Present as a ranked table with 3-line summaries, scoring by [criteria].” Now tune that prompt; add industry nuances, internal systems, customer data, or constraints. 4. Real examples we’ve seen work: - Logistics: AI predicts port congestion and auto-adjusts shipping routes - Retail: Forecasting model helps merchandisers optimize promo mix by store cluster 5. Use tools built for context-aware prompting - Use Custom GPTs or Claude’s file-upload capability - Store transcripts and research in Notion, Airtable, or similar - Build lightweight RAG pipelines (if technical support is available) - Small teams. Deep context. Structured prompting. Fast outcomes. This layered technique has been tested by some of the best in the field, including a few sharp voices worth following, including Allie K. Miller!

⭐️ Generative AI Fundamentals 🌟 In the Generative AI development process, understanding the distinctions between pre-training, fine-tuning, and RAG (Retrieval-Augmented Generation) is crucial for efficient resource allocation and achieving targeted results. Here’s a comparative analysis for a practical perspective: Pre-training:📚 • Purpose: To create a versatile base model with a broad grasp of language. • Resources & Cost: Resource-heavy, requiring thousands of GPUs and significant investment, often in millions. • Time & Data: Longest phase, utilizing extensive, diverse datasets. • Impact: Provides a robust foundation for various AI applications, essential for general language understanding. Fine-tuning:🎯 • Purpose: Customize the base model for specific tasks or domains. • Resources & Cost: More economical, utilizes fewer resources. • Time & Data: Quicker, focused on smaller, task-specific datasets. • Impact: Enhances model performance for particular applications, crucial for specialized tasks and efficiency in AI solutions. RAG:🔎 • Purpose: Augment the model’s responses with external, real-time data. • Resources & Cost: Depends on retrieval system complexity. • Time & Data: Varies based on integration and database size. • Impact: Offers enriched, contextually relevant responses, pivotal for tasks requiring up-to-date or specialized information. So what?💡 Understanding these distinctions helps in strategically deploying AI resources. While pre-training establishes a broad base, fine-tuning offers specificity. RAG introduces an additional layer of contextual relevance. The choice depends on your project’s goals: broad understanding, task-specific performance, or dynamic, data-enriched interaction. Effective AI development isn’t just about building models; it’s about choosing the right approach to meet your specific needs and constraints. Whether it’s cost efficiency, time-to-market, or the depth of knowledge integration, this understanding guides you to make informed decisions for impactful AI solutions. Save the snapshot below to have this comparative analysis at your fingertips for your next AI project.👇 #AI #machinelearning #llm #rag #genai