Chen Wang

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

You want to do Machine Learning on large scale Marketing solutions ? Check below 🚀

3

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

I'm happy to share that our paper, "Distributed-Memory Parallel Algorithms for Sparse Matrix and Sparse Tall-and-Skinny Matrix Multiplication," has been accepted for presentation at SC24! (https://lnkd.in/gp6txyc2) Our algorithm achieves 5x performance improvement over 2D and 3D SUMMA (https://lnkd.in/gzUhf2gV). Here are related papers we've previously published: Scalable Node Embedding Algorithms Using Distributed Sparse Matrix Operations (https://lnkd.in/g4ybbybG) Distributed Sparse Random Projection Trees for Constructing K-Nearest Neighbor Graphs (https://lnkd.in/gvuyhmw2)

62

3 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

Some folks from my team will be in Barcelona later this month to present at the 2024 KDD Workshop on "Causal Inference and Machine Learning in Practice." They will be presenting our paper titled "Practical Marketplace Optimization at Uber Using Causally-Informed Machine Learning." This work delves into the technical and business challenges of budget allocation for marketplace levers at Uber. These levers are instruments such as driver incentives and rider promotions. Understanding the impact of budget changes on the market and estimating cost efficiency are crucial for setting forward-looking budgets. Our paper describes an end-to-end machine learning and optimization procedure to automate budget decision-making across cities where Uber operates. This end-to-end process includes: * An advanced deep learning (DL) estimator based on S-Learner that leverages extensive user experimental and temporal-spatial observational data. * A novel tensor B-Spline regression model to enforce efficiency shape control while maintaining the sophistication of DL models’ response surface. * Solving high-dimensional optimization problems using Alternating Direction Method of Multipliers (ADMM) and primal-dual interior point convex optimization. All of this has significantly improved Uber’s resource allocation efficiency. Kudos to the team for their hard work on this and for capturing it for an outside audience! https://lnkd.in/gFkbAbCY Zhongbo (Bobby) Chen / Siyu Chen / Jason Dowlatabadi / Yu Xuan H. / Vinayak Iyer / Uday Mantripragada / Rishabh Narang / Apoorv Pandey / Zijun Qin / Abrar Sheikh / Hongtao Sun / Jess Sun / Matthew Walker / Kaichen Wei / Chen Xu / Jingnan Y. / Allen Zhang / Guoqing Zhang, CFA, FRM

99

1 Comment

My team is looking for new Research Scientists to join in Meta's London office. Suitable for recent PhD graduates and postdocs. For any questions, please reach out to Daniel or to me. Some recent works from our team: • 𝗜𝗻𝗳𝗼𝗦𝗵𝗮𝗽 (NeurIPS 2023), a SHAP extension to explain uncertainty estimates: https://lnkd.in/d4DPic5j • 𝗧𝗖𝗘 (UAI 2023), a metric to quantify probability calibration error that is suitable for extreme class imbalance: https://lnkd.in/dHbbQ6Nf • 𝗨𝗖𝗕-𝗘𝗨 (Data Mining & Knowledge Discovery journal 2024), an Active Learning method that accounts for non-response in label requests: https://lnkd.in/ejKSuNq9 • 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲-𝗪𝗲𝗶𝗴𝗵𝘁 𝗦𝗮𝗺𝗽𝗹𝗶𝗻𝗴 (under review), an Active Learning method with provable convergence guarantees that can handle industry scale data volume: https://lnkd.in/eQYnD4HB • 𝗢𝗻𝗹𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗯𝗲𝘀𝘁 𝗮𝗰𝘁𝗶𝗼𝗻 𝗾𝘂𝗲𝗿𝗶𝗲𝘀 (under review), a theoretical analysis of an online learning model that is applicable to industry scale content moderation systems: https://lnkd.in/eTCKAXFt

74

Go check out Brevitas at the links below and start experimenting with MX and all the latest quantization algorithms!

11

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

LLM Research Papers: The 2024 List (Sebastian Raschka, December 2024) "It’s been a very eventful and exciting year in AI research. This is especially true if you are interested in LLMs. I had big plans for this December edition and was planning to publish a new article with a discussion of all my research highlights from 2024. I still plan to do so, but due to an accident and serious injury, I am currently unable to work at a computer and finish the draft. But I hope to recover in the upcoming weeks and be back on my feet soon. In the meantime, I want to share my running bookmark list of many fascinating (mostly LLM-related) papers I stumbled upon in 2024. It’s just a list, but maybe it will come in handy for those who are interested in finding some gems to read for the holidays. And if you are interested in more code-heavy reading and tinkering, My Build A Large Language Model (From Scratch) book is out on Amazon as of last month. In addition, I added a lot of bonus materials to the GitHub repository." https://lnkd.in/e7d6z4Vj

2

Check out our latest blog to see how our Notification Team harnesses the power of Graph Neural Network (GNN) models to create personalized restaurant recommendations for user notifications.

63

1 Comment

Lightweight large language model (LLM) might be a reality soon Recently I came across the paper “Scalable MatMul-free Language Modeling” where Rui-Jie Zhu et al propose a scalable matrix-multiplication-free model capable of similar performance when compared to state-of-the-art transformers while reducing computational resources. This is achieved, among other things, by modifying the self-attention to use element-wise Hadamard products instead of matrix multiplication. If those finding are confirmed the possibility of fasters, energy efficient models with less GPU requirement become real to the point that these models could reside in the end-user machine. Scalable MatMul-free Language Modeling” https://lnkd.in/gtu55sFF

10

Many researchers are building agents to improve the capabilities of LLMs at long-range tasks, like writing a pull request for an issue in a code repository. Typical agents use the LLM in a loop, possibly with human feedback, where previous answers are an input to the LLM and are used to inform next actions. Other promising agents use tree searches, where multiple trains of thought are sampled in parallel. MOLLEO is a recent algorithm that caught my eye, as the authors wrapped an LLM in an evolutionary algorithm to generate SMILES strings. An evolutionary algorithm goes like this. Start with a group of candidates and calculate their fitness with an oracle. Then, sample two parents proportional to the fitness. Combine the two parents with a Crossover operator to create an offspring and then edit the offspring with a Mutation operator. Remove the lowest fitness members of the population and repeat the process. MOLLEO uses an LLM to do the Crossover and Mutation. Interestingly, evolutionary algorithms with LLMs benchmark better than without the LLM, and better than alternatives such as reinforcement learning. Paper in the comments 👇

15

3 Comments

Google "Quantum error correction below the surface code" unedited version available in Nature: https://lnkd.in/dqMWEYUB Older version, with more supplemental material (and open access), in arxiv:

13

1 Comment

Building a Japanese NLP Parser with Laravel, Vue.js, and Python 💻 皆さん、こんにちは！👋 As part of my journey to enhance my language skills and dive deeper into Natural Language Processing (NLP), I’ve just completed a project called Japanese NLP Parser. This project helps break down Japanese sentences into morphemes — the smallest units of meaning in a language. Since Japanese is written without spaces between words, parsing sentences is an essential first step in any NLP task. This parser simplifies Japanese sentence structures to improve language learning, translation tools, and chatbots. 🔧 Tech Stack: ✅ Laravel for the backend ✅ Vue.js for the frontend ✅ Python (SudachiPy) for Japanese text parsing 🌟 Features I’ve implemented: 🔸 Flashcards to help users learn new words and their meanings 🔸 Quiz Mode with random questions and multiple-choice answers 🔸 Shareable Results to let users share their quiz achievements The Quiz Mode also includes: ✔️ Randomized questions ✔️ Different question types (dictionary form, reading form, part of speech) ✔️ Timer for each question to enhance engagement 🚀 The application is now live and accessible online! Feel free to try it out: https://lnkd.in/gSPBGggY 📂 GitHub Repository: If you're curious about the code behind this project, check out the repo here: https://lnkd.in/gi8QHW3Z 💬 Why this project? I’ve always been fascinated by Japanese culture, language, and technology. As I prepare to build a career in Japan, this project not only helps me improve my Japanese skills but also strengthens my knowledge in NLP, web development, and interactive UI design. 📝 Next steps: In future updates, I plan to add: 🔹 Learning Mode to track user progress 🔹 Custom quizzes for vocabulary practice 🔹 Progress tracking and achievements もし興味があれば、ぜひ試してみてください！😊 Feel free to reach out if you'd like to discuss more about NLP, Japanese language learning, or web development. I'd love to connect and share ideas! #NLP #MachineLearning #Laravel #VueJS #Python #JapaneseLanguage #WebDevelopment #SudachiPy #LearningJourney #CareerInJapan #FullStackDevelopment #GitHub #LinkedInSharing

7

I first joined Amazon in December of 2005. Even though I left for two years, my "Amaversary" still coincides with the end of the year and a natural moment for reflection. As I reached my 17 year mark this December, I also had the privilege of working with Amazon Careers to share some of my journey and thoughts about individual contributor technical leader and people manager career paths. In my ruminating, I put together far more content than could be used for a single article. So, I've decided to post here as a series of Q&As - in case others might find my observations, experiences, and perspectives interesting or useful. Below is the first of 6. 1. 𝘽𝙧𝙞𝙚𝙛𝙡𝙮 𝙨𝙝𝙖𝙧𝙚 𝙮𝙤𝙪𝙧 𝙘𝙖𝙧𝙚𝙚𝙧 𝙟𝙤𝙪𝙧𝙣𝙚𝙮; 𝙬𝙝𝙖𝙩 𝙢𝙤𝙩𝙞𝙫𝙖𝙩𝙚𝙙 𝙮𝙤𝙪 𝙩𝙤 𝙩𝙖𝙠𝙚 𝙤𝙣 𝙙𝙞𝙛𝙛𝙚𝙧𝙞𝙣𝙜 𝙘𝙖𝙧𝙚𝙚𝙧 𝙥𝙖𝙩𝙝𝙨, 𝙄𝘾 𝙫𝙨 𝙢𝙖𝙣𝙖𝙜𝙚𝙧? I started my career as a software engineer 25 years ago. I've worked for startups, enterprise companies, and 7 different organizations over 17 years and two tours at Amazon. I've written firmware for robotic camera mounts; built highly scalable distributed systems; designed restaurant marketing, ordering, logistics, and operations software; created automated test harnesses for mobile games; tested HD DVD encoding; helped launch grocery delivery to customers around the globe - just to name a few of the many cool things I've had the privilege to work on. I've been an entry-level software engineer, a technical lead of teams, a software manager on a couple of occasions, chief software architect reporting to a CTO, and (for the last few years) Amazon Principal Engineer directly supporting a VP with about 200 engineers within my remit. I first became a manager because I mistakenly thought it was the only way to continue growing as a leader, and in many ways I failed. I learned some very humbling lessons about leadership through that experience, which I used to be a better individual contributor technical leader on my path to Senior Engineer. My second time as manager was because my organization had a need that I could fill. I had grown enough as a leader that this time I was much more successful. I said yes to an opportunity that started with managing a small team through delivery of a complex and critical project. Once I grew a high performing team that delivered that initial project, I took on more scope and expanded into multiple teams. After a couple of years of success growing and investing in my teams so that we could deliver greater value to our customers and business, I decided to prioritize being close to my family. For me, at that time, it meant giving up my people management role and returning to the role of an individual contributor technical leader that supports a globally distributed organization. Even though I really enjoyed my time as a people manager, there is no better job for me than that of a Principal Engineer. #amazon #careerdevelopment #technicalleadership #peoplemanagement

91

3 Comments

🚀 Excited to share my latest in-depth blog post exploring a cutting-edge AI framework that combines Hyperdimensional Memory Matching (HDMM), Neuro-Symbolic Context Mapping (NSCM), Evolutionary Reinforcement Scoring (ERS) powered by Genetic Algorithms (GAs) and Linear Regression for advanced feature prioritization and decision-making. This framework isn’t just another iteration of AI — it’s a paradigm shift. By leveraging hyperdimensional computing for high-capacity memory representation, symbolic reasoning for human-like interpretation and evolutionary algorithms for dynamic optimization it creates a system that’s scalable, interpretable, and robust. Key highlights: 📈 HDMM enables unmatched scalability for high-dimensional data encoding and matching. 🧠 NSCM bridges symbolic reasoning with data-driven models for enhanced decision-making. ⚙️ ERS with GAs optimizes features in a way that’s both adaptive and efficient. 🧮 Linear Regression integrates seamlessly for actionable insights and predictions. This isn’t just about better AI — it’s about creating systems that mimic cognitive processes, adapt dynamically, and deliver transparent and interpretable results. Let’s push the boundaries of what’s possible in AI! 💡 Read the full article here #ArtificialIntelligence #MachineLearning #HyperdimensionalComputing #NeuroSymbolicAI #GeneticAlgorithms #Innovation #FutureOfAI

1