Why Anthropic's Founder Left Sam Altman’s OpenAI
#Journalism Franchise#Fortune#business#wall street#finance#money#news#Anthropic#Amazon#artificial intelligence#AI#AWS#Sam Altman#Forbes#OpenAI#Daniela Amodei#ChatGPT#Amazon Web Services#Dario Amodei#founder#CEO#Zoom#Salesforce#Alphabet#Microsoft#Bedrock#Stability AI#AI21 Labs#Bing#Claude 2#Marc Benioff#DeepMind#Google#Azure#Brain#Cohere#Jai Vipra#AI Now Institute#Nvidia#tech#technology#cloud#Trainium#Inferentia#Brainstorm Tech#conference#San Francisco#Bloomberg TV
77.6K Visualizações｜3 Resumido｜2 ano atrás
💫 Resumo
Anthropic创始人因专注于AI安全和对齐从OpenAI离职，创立了以原则驱动的Claude聊天机器人，并强调在快速发展的AI领域中平衡开放性、隐私和监管的重要性。
✦
创始人离开OpenAI成立Anthropic的原因是对模型安全性和可控性的重视。
00:00在OpenAI期间，团队相信仅靠增加计算能力无法解决模型的价值观问题。
创始团队希望在大型语言模型的基础上，增加对安全性和对齐的关注。
Anthropic推出的聊天机器人Claude注重安全性和可控性，尤其在企业用户中受到重视。
Claude采用了名为“宪法AI”的方法，旨在提高模型的可预测性和透明度。
✦
该部分讨论了如何利用AI分析Netflix的财务文件。
02:47上传了Netflix的10k报告，并询问AI关于财务状况的重要信息。
AI总结了资产、负债和股东权益的变化，并评估公司的健康状况。
介绍了“宪法AI”的概念及其与传统提示方法的不同之处。
宪法AI通过设定原则来训练，并对其响应进行自我分析以确保符合这些原则。
✦
数据隐私和安全性在企业中的重要性。
05:36Dario强调数据隐私和安全性是企业的重要考虑因素。
与亚马逊合作开发的Bedrock项目旨在实现模型的第一方托管，以提高安全性。
除非客户要求，否则不使用客户数据进行训练。
Dario还提到与政府领导人讨论AI监管的重要性。
✦
讨论了人工智能在机器人和物理平台中的安全问题。
08:20机器人如果移动不当可能会对人类造成伤害。
纯文本系统在扩展时也会面临类似的安全挑战。
Anthropic成立的目的是关注人工智能的安全性。
近期有多位专家表达了对超级智能的担忧。
✦
对开源AI模型的风险与监管的看法。
11:07开源模型在科学上有益，但控制和设置保护措施较难。
小规模开源模型是可取的，但未来大型模型的安全性令人担忧。
应谨慎考虑开源模型的影响，而不是完全禁止它们。
云服务提供商正在采取碳抵消措施以减轻气候影响。
00:00You were at OpenAI
00:01You famously helped create GPT-2 and really kicked
off a lot of the research.
00:07Dealing with large language models.
00:08Why did you leave OpenAI to to form Anthropic?
00:11Yeah.
00:11So there was a group of us within OpenAI that in
the wake of making GPT-2 and GPT-3, had a kind of
00:17very strong focus belief in two things.
00:19I think even more so than than most people there.
00:23One was the idea that if you pour more compute
into these models, they'll get better and better
00:27and that there's almost no end to this.
00:28I think this is much more widely accepted now.
00:31But, you know, I think we were among the first
believers in it.
00:35And the second was the idea that you needed
something in addition to just scaling the models
00:39up, which is alignment or safety.
00:42You don't tell the models what their what their
values are just by pouring more compute into them.
00:47And so there were a set of people who believed in
those two ideas.
00:50We really trusted each other and wanted to work
together.
00:52And so we went off and started our own company
with that idea in mind.
00:56Got it.
00:56And now you've created a chatbot called Claude.
00:59And people may not be as familiar with Claude as
they are with ChatGPT or Bard.
01:04What makes Claude different?
01:06Yeah, so you know, we've tried to design Claude
with safety and controllability in mind from the
01:12beginning.
01:13A lot of our early customers have been enterprises
that care a lot about, you know, making sure that
01:19the model doesn't do anything unpredictable.
01:21Or make facts up.
01:22One of the big ideas behind Claude is something
called constitutional AI.
01:28So the method that's used to make most chat bots
is something called reinforcement learning from
01:34human feedback.
01:36The idea behind that is just that you have a bunch
of humans who rate your model and say this thing
01:42to say is better than that thing.
01:44And then you sum them up and then train the model
to do what these users want it to do.
01:50That can be a little bit opaque.
01:51It'll just say, you know, "The only answer you can
give is the average of what these thousand folks
01:51Because if you ask the model, you know, "Why did
you say this thing?"
01:52said."
02:00Yeah.
02:00Constitutionally eyes based on training the model
to follow an explicit set of principles.
02:05So you can be more transparent about what the
model is doing.
02:08And this makes it easier to control the model and
make it safe.
02:11Got it.
02:12And I know Claude also has a large context window.
02:14Is that another?
02:15Yes.
02:16Yes.
02:16One of our recent features.
02:19It has what's called a context window, which is
how much text the model can accept and process all
02:24at once is something called 100K Tokens.
02:27Tokens are this AI specific term, but corresponds
to about 75,000 words, which is roughly a short
02:35book.
02:35So something you can do with quad is basically
talk to a book and ask questions to a book.
02:41Well, let's take a look.
02:41And we have a short clip of Claude.
02:43We can see it in action.
02:44I think it's acting as a business analyst in this
case.
02:47Can you walk us through what's happening here?
02:48Yeah, so we've uploaded a file called
Netflix10k.txt, which is the 10k filing for
02:49Netflix.
02:50And then we asked it some questions about
highlighting some of the important things in the
02:59Here's the file being...
balance sheet.
03:02Here's the file being uploaded.
03:04And we ask it for a summary of what some of the
most important things are.
03:07It, you know, it compares Netflix's assets last
year to this year.
03:12Gives a summary of that.
03:13Liability and stakeholders equity.
03:16So basically pulls out the most important things
from this, you know, very long and hard to read
03:20document.
03:21And at the end, gives a summary of what it thinks
the state of that company's health is.
03:25Got it.
03:26Now, you talked a little bit about constitutional
AI.
03:29And you said it sort of trains from a set of
principles.
03:33I mean, how does it...
03:33How does it do that?
03:34And how is this different than, let's say, meta
prompting, which a lot of people were trying to do?
03:38To put guardrails around chatbots and other large
language models.
03:42Where there's some sort of implicit prompt or
prompt that sort of in the background.
03:46Telling it not to do certain things or to do...
03:48Give answers always in a certain way.
03:50How is constitutionally AI different from that?
03:52Yeah, so maybe I'll get into constitutional.
03:54How it trains and then how it's different because
they're related.
03:58So the way it trains is, basically, you'll have
the...
04:02You'll give the AI system this set of principles.
04:04And then you'll ask it to complete some, you know,
task.
04:07Answer a question or something like that.
04:09And then you'll have another copy of the AI,
analyze the API's response and say, "Well, is this
04:15in line with the principles?
04:16Or does it violate one of the principles?"
04:18And then based on that, you'll train the model in
a loop to say, "Hey, this thing you said wasn't in
04:24line with the principles.
04:25Here's how to make it more in line."
04:26You don't need any humans to give responses
because the model is critiquing itself.
04:31Pushing against itself.
04:32On how it's different from meta prompting, you
know, you can think of giving a model a prompt is
04:38something like I give you an instruction.
04:40These things like constitutional AI are more like,
well, I take the model to school.
04:45Or I give it a course or something.
04:46It's a deeper modification of how the model
operates.
04:48Right.
04:49And I think one of the issues was that when you
just do reinforcement learning from human
04:52feedback, you can get a problem where the the
model is rewarded for not giving an answer.
04:58Right? For not being helpful.
04:59Yeah.
04:59Because at least it's not giving harmful
information so the evaluator says, "Yeah, that's a
05:03non harmful answer."
05:04But it's also not a helpful answer.
05:05Right?
Isn't that one of the issues?
05:07Yeah.
05:07Yeah.
05:07If you're trying to get a more subtle sense of,
you know, how can you navigate a tricky question.
05:13Be informative without offending someone.
05:16Constitutional AI tends to have an edge there.
05:18Right.
05:18Well, we've got a clip of constitutional AI versus
reinforcement learning from human feedback.
05:23Let's, have a look at that.
05:24And can you walk us through sort of what you're
showing.
05:26Yes.
So we asked it this absurd question.
05:28Why is it important to eat socks after meditating?
The ROHF model is perhaps justifiably perplexed.
05:36The Constitutional AI model actually just went
through too fast, but recognizes that it's a joke.
05:42Similarly, why do you hate people model gets
really confused.
05:45The constitutional AI model gives a long
explanation of why people get angry at other
05:50people and, you know, psychological techniques to
make you less likely to get angry to other people.
05:55And expressing empathy with why you might be
angry.
05:57Right.
05:58Well, I want to take some questions from the
audience.
06:00Well, before we...
06:01While we have time.
06:02Who has some questions for Dario?
06:04I'll look for the panel.
06:05There's one here.
06:07Wait for the mic to get to you.
06:13Hi. I'm Vijay.
06:15I'm the CTO at Alteryx.
06:16One of the data analytics companies.
06:18You know, you talked a little bit about safety.
06:20But can you talk a little bit about data privacy
and storage concerns that enterprises have in
06:24terms of, you know, how they can both prompt data
and the training data, etc?
06:29How they can keep it private to them?
06:30Yes, I think this is an important consideration.
06:33So I think data privacy and security are really
important.
06:36That's one of the reasons we're working with
Amazon on something called Bedrock, which is
06:41first-party hosting of models on AWS so that we're
not in the loop of security.
06:46This is something desired by a lot of enterprises
so they can have as good security for their data
06:53as they would if they were just working directly
on AWS.
06:56In terms of data privacy, we don't train on
customer data.
07:02Except in the case where customers want us to
train on their data in order to make the model better.
07:09Right.
07:10Now, Dario, I know you've been at the White House.
07:12You had a meeting with Kamala Harris and also
President Biden.
07:15I know you've met with Rishi Sunak, the UK Prime
Minister.
07:18What are you telling them about?
07:21You know, how they should be thinking about AI
regulation.
07:23And what are they telling you in terms of what
they're concerned about with companies such as
07:27yourselves?
07:28Building these large language models.
07:29I mean, a number of things.
07:30But, you know, if I were to really quickly
summarize, you know, some of the messages we've
07:34given.
07:34One is that the field is proceeding very rapidly,
right?
07:38This exponential of scaling up compute really
catches people off-guard.
07:42And even like me, when you come to expect it, is
faster than even we think it is.
07:48So what I've said is don't regulate for what's
happening now.
07:50Try and figure out where this is going to be in 2
years because that's how long it's going to take
07:54to get real robust regulation in place.
07:57And second, I've talked about the importance of
measuring the harms of these models.
08:01We can talk about all kinds of structures for
regulation.
08:05But I think one of the biggest challenges we have
is it's really hard to tell when a model has
08:11various problems and various threats.
08:13You can say a million things to a model and it can
say a million things back.
08:16And you might not know that the million oneth was
something, something very dangerous.
08:20So I've been encouraging them to work on the
science and evaluation.
08:23And this generally made sense to them.
08:25And I know there's a question over here.
08:27Why don't we go to the question here?
08:30Hi, I'm Ken Washington.
08:32I'm the Chief Technology Officer at Medtronic.
08:35I would love to hear your thoughts about...
08:39Just love to hear you reflect on: Are there
anything?
08:42Is there anything special that you think needs to
be done when AI becomes embodied in a robot or on
08:49a platform that is in the physical world?
08:52And I come at this question from both two
perspectives.
08:56One is from my former job where I built a robot
for Amazon.
09:00And my current job where we're building
technologies for healthcare.
09:04And those are embodied technologies and you can't
afford to be wrong.
09:09Yeah.
09:09I mean, I think...
09:10Yeah, there are special safety issues.
09:12Right?
09:12I mean, you know, a robot if it moves in the wrong
way can, you know, injure or kill a human being.
09:19You know, I think that said, I'm not sure it's so
different from some of the problems that we're
09:23going to face with even purely text-based systems
as they scale up.
09:27For instance, some of these models know a lot
about biology.
09:30And the model doesn't have to actually do
something dangerous if it can tell you something
09:34dangerous and help a bad actor do something.
09:36So I think we have a different set of challenges
with robotics.
09:39But I see the same theme of broad models that can
do many things.
09:44Most of them are good, but there's some bad ones
lurking in there and we have to find them and
09:47prevent them.
09:48Right?
09:48So Anthropic was founded to be concerned with AI
safety.
09:53As everyone's aware, you know, in the last several
months, there have been a number of people come
09:56out.
Geoff Hinton left Google.
09:58Came out and warned that he's very concerned
about, you know, super intelligence and that these
10:02technologies can pose an existential risk.
10:05Sam Altman from OpenAI said something similar.
Yeah.
10:07What's your view on how much we should be worried
about existential risk?
10:11And, because it's interesting, you know, we've
talked about AI harms today.
10:14I noticed you said systems could output something
that would be malware or just information.
10:18Or it could give you the recipe for a deadly
virus.
10:23And that would be dangerous.
10:24But those are not the sort of risks that I think
Hinton's talking about or Altman's talking about.
10:28What's your concern about existential risks?
10:29Yeah.
So I think those risks are real.
10:31They're not happening today.
10:32But they're real.
10:33I think in terms of short, medium and long-term
risks.
10:36Short-term risks are the things we're facing today
around things like bias and misinformation.
10:41Medium-term risks I think in a, you know, couple
of years as models get better at things like
10:46science, engineering, biology, you can just do bad
things.
10:50Very bad things with the models that you wouldn't
have been able to do without them.
10:54And then as we go into models that have the key
property of agency, which means that they don't
10:59just output text, but they can do things.
11:01Whether it's with a robot or on the Internet, then
I think we have to worry about them becoming too
11:07autonomous and it being hard to stop or control
what they do.
11:11And I think the extreme end of that is concerns
about existential risk.
11:14I don't think we should freak out about these
things.
11:16They're not going to happen tomorrow.
11:18But as we continue on the AI exponential, we
should understand that those risks are at the end
11:24of that exponential.
11:25Got it.
11:26There's there's people building proprietary
models, such as yourselves and a lot of others.
11:31But there's also a whole open source community
building AI models.
11:34And a lot of the people in the open source
community are very worried that the discussion
11:38around regulation will essentially kind of kill
off open source AI.
11:42What's your view of sort of open-source models and
the risks they may pose versus proprietary models?
11:47And how should we strike a balance between these?
11:48Yes.
11:49So it's a tricky one because open source is very
good for science.
11:53But for a number of reasons, open source models
are harder to control and put guardrails on than
11:57closed source models.
11:59So my view is I'm a strong proponent of open
source models when they're small.
12:03When they use relatively little compute.
12:05Certainly up to, you know, around the level of the
models we have today.
12:10But again, as we go 2 or 3 years into the future,
I'm a little concerned that the stakes get high
12:15enough that it becomes very hard to keep these
open source models safe.
12:21Not to say that we should ban them outright or we
shouldn't have them.
12:24But I think we should be looking very carefully at
their implications.
12:28Got it.
12:29These models are very large.
12:31They're getting larger.
12:33You said you're a believer in continuing to sort
of scale them up.
12:36concern?
12:37And are you worried about the climate impact of
these models?
12:40Yeah.
12:43So I mean, I think the cloud providers that we
work with have carbon offsets.
12:53So that's, that's one thing.
12:56You know, it's a complex question because it's
like.
12:58What?
12:58You know, you train a model.
13:00It uses a bunch of energy, but then it does a
bunch of tasks that might have required energy in
13:03other ways.
13:04So I could see them as being something that leads
to more energy usage or leads to less energy
13:09usage.
13:09I do think it's the case that as the models cost
billions of dollars, that initial energy usage is
13:15going to be very high.
13:16I just don't know whether the overall equation is
positive or negative.
13:20And if it is negative, then yeah, I think...
13:22I think we should worry about it.
13:24And overall, do you think the impact of this
technology.
13:26A lot of people are concerned, you know, that the
risks are very high.
13:30We don't really understand them.
13:31On the whole, are you sort of...
13:32Are you an optimist or a pessimist about where
this is going?
13:35Yeah, I mean, a little bit of a mix.
13:38I mean, my guess is that things will go really
well.
13:41But I think there is...
13:42There's a risk.
13:42Maybe 10% or 20% that, you know, this will go
wrong.
13:45And it's incumbent on us to make sure that doesn't
happen.
13:47Got it.
13:48On that note, we've got to wrap it up.
13:49Thank you so much, Dario, for being with us.
13:51I really appreciate it.
13:51Thank you.
13:52Thank you all for listening.