【中&英】《生成人工智能手册》Generative AI Handbook: A Roadmap for Learning Resources

输入“/”快速插入内容

👍
【中&英】《生成人工智能手册》Generative AI Handbook: A Roadmap for Learning Resources

2024年6月20日创建

😋

作者：William Brown X@willccbb v0.1 (June 5, 2024)

Introduction 介绍

This document aims to serve as a handbook for learning the key concepts underlying modern artificial intelligence systems. Given the speed of recent development in AI, there really isn’t a good textbook-style source for getting up-to-speed on the latest-and-greatest innovations in LLMs or other generative models, yet there is an abundance of great explainer resources (blog posts, videos, etc.) for these topics scattered across the internet. My goal is to organize the “best” of these resources into a textbook-style presentation, which can serve as a roadmap for filling in the prerequisites towards individual AI-related learning goals. My hope is that this will be a “living document”, to be updated as new innovations and paradigms inevitably emerge, and ideally also a document that can benefit from community input and contribution. This guide is aimed at those with a technical background of some kind, who are interested in diving into AI either out of curiosity or for a potential career. I’ll assume that you have some experience with coding and high-school level math, but otherwise will provide pointers for filling in any other prerequisites. Please let me know if there’s anything you think should be added!​

本文档旨在作为学习现代人工智能系统背后的关键概念的手册。考虑到人工智能最近的发展速度，确实没有一个好的教科书式的资源来快速了解 LLMs 或其他生成模型的最新和最伟大的创新，但互联网上有大量关于这些主题的优秀解释资源（博客文章、视频等）。我的目标是将这些资源中的“最好的”组织成教科书式的演示文稿，它可以作为满足个人人工智能相关学习目标的先决条件的路线图。我希望这将是一份“活文件”，随着新的创新和范式不可避免地出现而进行更新，并且理想情况下也是一份可以从社区投入和贡献中受益的文件。本指南针对的是那些具有某种技术背景、出于好奇或潜在职业而有兴趣深入研究人工智能的人。我假设您有一些编码和高中数学水平的经验，但否则将提供填写任何其他先决条件的指导。如果您认为有什么需要补充的，请告诉我！​

The AI Landscape 人工智能前景

As of June 2024, it’s been about 18 months since ChatGPT was released by OpenAI and the world started talking a lot more about artificial intelligence. Much has happened since: tech giants like Meta and Google have released large language models of their own, newer organizations like Mistral and Anthropic have proven to be serious contenders as well, innumerable startups have begun building on top of their APIs, everyone is scrambling for powerful Nvidia GPUs, papers appear on ArXiv at a breakneck pace, demos circulate of physical robots and artificial programmers powered by LLMs, and it seems like chatbots are finding their way into all aspects of online life (to varying degrees of success). In parallel to the LLM race, there’s been rapid development in image generation via diffusion models; DALL-E and Midjourney are displaying increasingly impressive results that often stump humans on social media, and with the progress from Sora, Runway, and Pika, it seems like high-quality video generation is right around the corner as well. There are ongoing debates about when “AGI” will arrive, what “AGI” even means, the merits of open vs. closed models, value alignment, superintelligence, existential risk, fake news, and the future of the economy. Many are concerned about jobs being lost to automation, or excited about the progress that automation might drive. And the world keeps moving: chips get faster, data centers get bigger, models get smarter, contexts get longer, abilities are augmented with tools and vision, and it’s not totally clear where this is all headed. If you’re following “AI news” in 2024, it can often feel like there’s some kind of big new breakthrough happening on a near-daily basis. It’s a lot to keep up with, especially if you’re just tuning in.

截至 2024 年 6 月，自 OpenAI 发布 ChatGPT 以来已经过去了大约 18 个月，世界开始更多地谈论人工智能。此后发生了很多事情：Meta 和 Google 等科技巨头发布了自己的大型语言模型，Mistral 和 Anthropic 等较新的组织也被证明是有力的竞争者，无数初创公司开始在他们的 API 基础上构建，每个人都在争夺强大的 Nvidia GPU、论文以极快的速度出现在 ArXiv 上、由 LLMs 驱动的物理机器人和人工程序员的演示在流传，聊天机器人似乎正在寻找进入在线生活的各个方面的方法（在不同程度上）的成功）。与LLM竞赛同时进行的，通过扩散模型生成图像也取得了快速发展； DALL-E 和 Midjourney 正在展示越来越令人印象深刻的结果，这些结果常常在社交媒体上难倒人们，随着 Sora、Runway 和 Pika 的进步，高质量视频生成似乎也指日可待。关于“AGI”何时到来、“AGI”到底意味着什么、开放模型与封闭模型的优点、价值一致性、超级智能、存在风险、假新闻和经济的未来，一直存在争论。许多人担心自动化会导致工作岗位流失，或者对自动化可能推动的进步感到兴奋。世界在不断发展：芯片变得更快，数据中心变得更大，模型变得更智能，上下文变得更长，能力通过工具和愿景得到增强，但目前尚不完全清楚这一切的发展方向。如果您在 2024 年关注“人工智能新闻”，您通常会感觉几乎每天都会发生某种重大的新突破。有很多事情需要跟上，尤其是当你刚刚收听时。

With progress happening so quickly, a natural inclination by those seeking to “get in on the action” is to pick up the latest-and-greatest available tools (likely GPT-4o, Gemini 1.5 Pro, or Claude 3 Opus as of this writing, depending on who you ask) and try to build a website or application on top of them. There’s certainly a lot of room for this, but these tools will change quickly, and having a solid understanding of the underlying fundamentals will make it much easier to get the most out of your tools, pick up new tools quickly as they’re introduced, and evaluate tradeoffs for things like cost, performance, speed, modularity, and flexibility. Further, innovation isn’t only happening at the application layer, and companies like Hugging Face, Scale AI, and Together AI have gained footholds by focusing on inference, training, and tooling for open-weights models (among other things). Whether you’re looking to get involved in open-source development, work on fundamental research, or leverage LLMs in settings where costs or privacy concerns preclude outside API usage, it helps to understand how these things work under the hood in order to debug or modify them as needed. From a broader career perspective, a lot of current “AI/ML Engineer” roles will value nuts-and-bolts knowledge in addition to high-level frameworks, just as “Data Scientist” roles have typically sought a strong grasp on theory and fundamentals over proficiency in the ML framework du jour. Diving deep is the harder path, but I think it’s a worthwhile one. But with the pace at which innovation has occurred over the past few years, where should you start? Which topics are essential, what order should you learn them in, and which ones can you skim or skip?

随着进展如此之快，那些寻求“参与行动”的人自然倾向于选择最新最好的可用工具（截至撰写本文时，可能是 GPT-4o、Gemini 1.5 Pro 或 Claude 3 Opus），取决于你问的是谁）并尝试在它们之上构建一个网站或应用程序。当然还有很大的空间，但这些工具会很快发生变化，对底层基础知识有深入的了解将使您更容易地充分利用您的工具，并在新工具推出时快速使用它们，并评估成本、性能、速度、模块化和灵活性等方面的权衡。此外，创新不仅仅发生在应用层，Hugging Face、Scale AI 和 Together AI 等公司通过专注于开放权重模型的推理、训练和工具（等等）而站稳了脚跟。无论您是想参与开源开发、从事基础研究，还是在因成本或隐私问题而无法使用外部 API 的环境中利用 LLMs，了解这些事情在hood，以便根据需要调试或修改它们。从更广泛的职业角度来看，当前的许多“AI/ML 工程师”角色除了高级框架之外还将重视具体知识，就像“数据科学家”角色通常寻求对理论和基础知识的牢固掌握一样对当前的 ML 框架过于熟练。深入研究是一条更艰难的道路，但我认为这是一条值得的道路。但随着过去几年创新的步伐，您应该从哪里开始呢？哪些主题是必不可少的，您应该按什么顺序学习它们，以及您可以浏览或跳过哪些主题？

The Content Landscape 内容格局

Textbooks are great for providing a high-level roadmap of fields where the set of “key ideas” is more stable, but as far as I can tell, there really isn’t a publicly available post-ChatGPT “guide to AI” with textbook-style comprehensiveness or organization. It’s not clear that it would even make sense for someone to write a traditional textbook covering the current state of AI right now; many key ideas (e.g. QLoRA, DPO, vLLM) are no more than a year old, and the field will likely have changed dramatically by the time it’d get to print. The oft-referenced Deep Learning book (Goodfellow et al.) is almost a decade old at this point, and has only a cursory mention of language modeling via RNNs. The newer Dive into Deep Learning book includes coverage up to Transformer architectures and fine-tuning for BERT models, but topics like RLHF and RAG (which are “old” by the standards of some of the more bleeding-edge topics we’ll touch on) are missing. The upcoming “Hands-On Large Language Models” book might be nice, but it’s not officially published yet (available online behind a paywall now) and presumably won’t be free when it is. The Stanford CS224n course seems great if you’re a student there, but without a login you’re limited to slide-decks and a reading list consisting mostly of dense academic papers. Microsoft’s “Generative AI for Beginners” guide is fairly solid for getting your hands dirty with popular frameworks, but it’s more focused on applications rather than understanding the fundamentals.

教科书非常适合提供一系列领域的高级路线图，其中“关键思想”集更加稳定，但据我所知，确实没有公开的 ChatGPT 后“人工智能指南”和教科书-风格的全面性或组织性。目前还不清楚有人编写一本涵盖当前人工智能现状的传统教科书是否有意义；许多关键想法（例如 QLoRA、DPO、vLLM）都还不到一年的时间，到印刷时该领域可能已经发生了巨大变化。经常引用的《深度学习》一书（Goodfellow 等人）已经有近十年的历史了，并且只粗略地提到了通过 RNN 进行语言建模。较新的《深入深度学习》一书涵盖了 Transformer 架构和 BERT 模型的微调，但 RLHF 和 RAG 等主题（按照我们将要涉及的一些更前沿主题的标准来看，这些主题已经“过时”））缺失。即将出版的《大型语言模型实践》一书可能很不错，但它尚未正式出版（现在可以通过付费专区在线获取），并且可能不会免费。如果您是斯坦福大学的学生，CS224n 课程似乎很棒，但如果没有登录，您只能使用幻灯片和主要由密集的学术论文组成的阅读列表。微软的“面向初学者的生成式人工智能”指南相当可靠，可以帮助您熟悉流行的框架，但它更侧重于应用程序而不是理解基础知识。

The closest resource I’m aware of to what I have in mind is Maxime Labonne’s LLM Course on Github. It features many interactive code notebooks, as well as links to sources for learning the underlying concepts, several of which overlap with what I’ll be including here. I’d recommend it as a primary companion guide while working through this handbook, especially if you’re interested in applications; this document doesn’t include notebooks, but the scope of topics I’m covering is a bit broader, including some research threads which aren’t quite “standard” as well as multimodal models.

我所知道的与我的想法最接近的资源是 Github 上 Maxime Labonne 的 LLM 课程。它具有许多交互式代码笔记本，以及用于学习基本概念的资源链接，其中一些与我将在此处包含的内容重叠。我建议您在阅读本手册时将其作为主要配套指南，特别是如果您对应用程序感兴趣的话；本文档不包括笔记本，但我涵盖的主题范围更广泛，包括一些不太“标准”的研究线程以及多模式模型。

Still, there’s an abundance of other high-quality and accessible content which covers the latest advances in AI — it’s just not all organized. The best resources for quickly learning about new innovations are often one-off blog posts or YouTube videos (as well as Twitter/X threads, Discord servers, and discussions on Reddit and LessWrong). My goal with this document is to give a roadmap for navigating all of this content, organized into a textbook-style presentation without reinventing the wheel on individual explainers. Throughout, I’ll include multiple styles of content where possible (e.g. videos, blogs, and papers), as well as my opinions on goal-dependent knowledge prioritization and notes on “mental models” I found useful when first encountering these topics.​

尽管如此，还有大量其他高质量且易于访问的内容涵盖了人工智能的最新进展——只是并非全部井井有条。快速了解新创新的最佳资源通常是一次性博客文章或 YouTube 视频（以及 Twitter/X 线程、Discord 服务器以及 Reddit 和 LessWrong 上的讨论）。我编写本文档的目标是提供一个导航所有这些内容的路线图，将其组织成教科书式的演示文稿，而无需重新设计各个解释器。在整个过程中，我将尽可能包含多种风格的内容（例如视频、博客和论文），以及我对目标相关知识优先顺序的看法以及我在第一次遇到这些主题时发现有用的“心理模型”注释。​

I’m creating this document not as a “generative AI expert”, but rather as someone who’s recently had the experience of ramping up on many of these topics in a short time frame. While I’ve been working in and around AI since 2016 or so (if we count an internship project running evaluations for vision models as the “start”), I only started paying close attention to LLM developments 18 months ago, with the release of ChatGPT. I first started working with open-weights LLMs around 12 months ago. As such, I’ve spent a lot of the past year sifting through blog posts and papers and videos in search of the gems; this document is hopefully a more direct version of that path. It also serves as a distillation of many conversations I’ve had with friends, where we’ve tried to find and share useful intuitions for grokking complex topics in order to expedite each other’s learning. Compiling this has been a great forcing function for filling in gaps in my own understanding as well; I didn’t know how FlashAttention worked until a couple weeks ago, and I still don’t think that I really understand state-space models that well. But I know a lot more than when I started.​
我并不是以“生成式人工智能专家”的身份来创建这份文档，而是以最近有过在短时间内熟悉这些主题的经验的人的身份来创建这份文档。虽然我从 2016 年左右开始就一直在人工智能领域工作（如果我们将运行视觉模型评估的实习项目算作“开始”），但我直到 18 个月才开始密切关注 LLM 的发展之前，随着 ChatGPT 的发布。大约 12 个月前，我第一次开始使用开放重量 LLMs。因此，在过去的一年里，我花了很多时间筛选博客文章、论文和视频，以寻找其中的精华；希望本文档是该路径的更直接版本。它也是我与朋友进行的许多对话的精华，我们试图找到并分享处理复杂主题的有用直觉，以加快彼此的学习速度。编译这篇文章对于填补我自己的理解空白也是一个很大的推动作用。直到几周前我才知道 FlashAttention 是如何工作的，而且我仍然认为我还没有真正理解状态空间模型。但我比刚开始的时候知道了很多。​

Resources 资源

Some of the sources we’ll draw from are:

我们将借鉴的一些来源是：

•

Blogs: 博客：

◦

Hugging Face blog posts

拥抱脸博客文章

◦

Chip Huyen’s blog Chip Huyen 的博客

◦

Lilian Weng’s blog 翁莉莲的博客

◦

Tim Dettmers’ blog 蒂姆·戴特默斯的博客

◦

Towards Data Science 走向数据科学

◦

Andrej Karpathy’s blog 安德烈·卡帕蒂的博客

◦

Sebastian Raschka’s “Ahead of AI” blog

Sebastian Raschka 的“Ahead of AI”博客

•

YouTube: YouTube：

◦

Andrej Karpathy’s “Zero to Hero” videos

安德烈·卡帕蒂 (Andrej Karpathy) 的“从零到英雄”视频

◦

3Blue1Brown videos 3Blue1Brown 视频

◦
Mutual Information 互信息​

◦
StatQuest 统计任务​

•

Textbooks 教科书

◦

The d2l.ai interactive textbook

d2l.ai 互动教科书

◦

The Deep Learning textbook

深度学习教科书

•

Web courses: 网络课程：

◦

Maxime Labonne’s LLM Course

Maxime Labonne 的 LLM 课程

◦

Microsoft’s “Generative AI for Beginners”

微软的“初学者生成式人工智能”

◦

Fast.AI’s “Practical Deep Learning for Coders”

Fast.AI 的“程序员实用深度学习”

•
Assorted university lecture notes​
各种大学讲义​

•
Original research papers (sparingly)​
原创研究论文（少量）​

I’ll often make reference to the original papers for key ideas throughout, but our emphasis will be on expository content which is more concise and conceptual, aimed at students or practitioners rather than experienced AI researchers (although hopefully the prospect of doing AI research will become less daunting as you progress through these sources). Pointers to multiple resources and media formats will be given when possible, along with some discussion on their relative merits.​

我经常会参考原始论文的关键思想，但我们的重点将是更简洁和概念性的说明性内容，针对的是学生或从业者，而不是经验丰富的人工智能研究人员（尽管希望进行人工智能研究的前景会更好）当您在这些来源中取得进展时，就会变得不那么令人畏惧）。如果可能的话，将给出多种资源和媒体格式的指示，并对其相对优点进行一些讨论。​

Organization 组织

This document is organized into several sections and chapters, as listed below and in the sidebar. You are encouraged to jump around to whichever parts seem most useful for your personal learning goals. Overall, I’d recommend first skimming many of the linked resources rather than reading (or watching) word-for-word. This should hopefully at least give you a sense of where your knowledge gaps are in terms of dependencies for any particular learning goals, which will help guide a more focused second pass.​

本文档分为几个部分和章节，如下和侧栏中所列。我们鼓励您跳到对您的个人学习目标最有用的部分。总的来说，我建议首先浏览许多链接的资源，而不是逐字阅读（或观看）。希望这至少能让您了解在任何特定学习目标的依赖性方面您的知识差距在哪里，这将有助于指导更有针对性的第二遍。​