There is a specific, unilateral action that he is taking described in the article.
And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.
I mean, he is right that they can and should unilaterally slow down, but doesn't it also make sense for the government to create the regulations that will make this easier to do in a trustless way?
The whole point of the Pacing the Frontier letter was that employees at both OpenAI and Anthropic are extremely aware of the risks but realize that they are locked in a competitive battle for survival between 2 companies that don't trust each other enough to voluntarily pause, while any sort of coordination out in the open might be against antitrust laws.
What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.
And all of this information has been available on the open web way before LLMs.
I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
So what's the fuss about then? Is the idea that a Chinese company will release a model that will have no safeguards? For what purpose?
Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe".
Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, or learn how to jailbreak?
That would be a very dedicated person. And dedicated person doesn't need an LLM. So where are they?
Yes, that is exactly Dario's concern. Either one of the US labs or one of the Chinese ones will eventually release something with insufficient safety controls for its power level because it gives them slightly better user retention (look how much complaining there is about current frontier models, especially Fable, rejecting requests). Regulation or consortium is how you avoid the prisoner's dilemma.
As long as user provides inputs and LLMs stay LLMs, you can waltz through any guardrail. Fable is the extreme case, but it's not that hard if you know what you're doing and know how LLMs and their guardrails work.
Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".
All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".
No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.
The bioweapons argument is one of the favorites used by these con artists and dreamed up by their PR team.
Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
An undergrad bio student with some lab experience can do some effing terrifying things with about $20k in equipment and time and access to some papers and a library. AI is not needed, but it might help accelerate the research.
Not going to go into it but I studied biology. It’s all out there. It’s easier than you think.
It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb.
It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
I love watching NileRed/NileBlue on youtube - crazy chemist that does a lot of insane stuff. While it's obvious that he's very smart and probably way above the average, his education is still nothing that probably millions of people don't have:
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
Except that there are finite hours in the day. I enjoy the process and craft very much, but I also enjoy making high tech salaries. Up until a year or so ago, those weren't contradictory, because the craft paid well. Now, continuing to be employed in an AI-dominated industry means that I can no longer practice my craft at work most days. So now I need to find the motivation to do it in my spare time - and this while the work hours are significantly more emotionally and motivationally draining because it's less fulfilling.
If I switched to spending more time on the hobbyist craft, I would pretty quickly have to give up doing well at the job. And the parts of the craft I enjoyed the most are the puzzles around distributed systems and high-scale correctness, which tend not to come up in the hobbyist context anyway.
Welcome to the world everyone else has to muddle through.
The solution has long been, and remains, "Organize and demand enough money to live on from ~20 hours of employed labor a week, spend the rest of the time as you'd like."
Sure, I acknowledge that it was incredibly rare and lucky for "the thing I enjoy doing" and "the thing that pays" to have been the same for so long. That doesn't mean it's not painful for that to change out from under me - especially when I can't quite convince myself that the new way of doing things is actually better by any metric.
Well, the key is understanding that the old paradigm wasn't just rare and lucky; there's a good chance that it was unfair, and actively robbing someone else of dignity.
What's happening to you now may also feel unfair, but it's up to you if you're just going to be pissed on your own behalf, or if, now that you recognize that everyone else is going through what you've been newly-introduced to, you're going to go, "Hey, no one should have to live like this! Fix this crap for everyone."
Google's AdSpam/fraud/bot-prevention team was, when I worked there, world class and fairly well funded, took their job seriously, and had access to all the data. It's an existential threat to the ad business, because if Google gets a reputation for being full of bots/spam, then the advertisers will bid lower per click/conversion to compensate, which means that legitimate website publishers will get paid less and go to other networks, which is a feedback loop that leads to the entire market collapsing (see also: https://en.wikipedia.org/wiki/The_Market_for_Lemons). It's absolutely worth refunding/zero-rating huge amounts of advertiser spend to avoid that situation, and they do.
It's not that they're not trying, it's just a very hard problem.
Problem 1: Buyers cannot tell if a product is good or bad, so they offer less money and good sellers may leave.
Suppose 50% of used laptops are good and worth $1,000, while 50% are bad and worth $400. Since you cannot tell which one you are buying, the average value is 0.5x1000 + 0.5x400 = $700, so you will not want to pay more than about $700.
But owners of good laptops may refuse to sell for $700, so more good laptops leave the market and the chance of buying a bad one increases. And the only guy selling for $700 is the lemons.
Problem 2: The theory assumes buyers already know how many bad products are in the market, but in real life they often do not.
Your "point 1" is literally the argument of the paper. If that scenario arises, the market collapses and no further sales can be made. That's the whole problem. As someone who takes a percentage of every sale, you want to keep the lemon-sellers out even though in the short run they make you extra money.
Your "point 2", if it's meant to be a rebuttal, isn't much of one. Buyers don't need to accurately know exactly what fraction of sellers are fraudulent; if they believe that it's 50-50, the same thing happens, even if the true rate is 80-20 in favor of good sellers. Conversely, if buyers are overly optimistic about quality, the market can persist despite a level of fraud that's higher than should be tolerated. But in any case, things like reviews and external reporting should eventually give them good information.
Cynicism is fine. They don't have to believe in any sort of ideal to do this. They just have to be thinking more than a couple months ahead to realize the math favors quality.
Just to be clear, this is different to the problem of Google ads that link to malware and fake banking websites and promotion of cryptocurrency scams isn't it?
Related in that some of the same organized groups tend to be carrying out every kind of attack at once, but operationally a different thing.
In the ad marketplace there are four participants: the advertiser, the user, the network (Google), and the publisher (also Google for AdWords, other websites for AdSense and so on, and effectively the channel owner for YouTube).
In the click spam or botnet case, the bad actor is the user, who is usually associated with a publisher trying to get extra money (although not always - there's reasons like auction manipulation where some advertisers run click bots too). In the bad-ads case, the bad actor is the advertiser.
At Google, each of those problems has their own well funded team, but they do also share some data to help catch rings of bad guys.
> the point isn't that "Leave to renew" is some kind of horrible policy
I mean... "leave to renew" is a horrible policy, it's just one that we've normalized since 9/11. It's incredibly wasteful and disruptive, and serves no purpose. There's no legitimate reason why a long-term residential visa like an H1-B can't be renewed by mail or electronically. I can renew my driver's license by mail, and I'm a hell of a lot more likely to cause serious damage with a car than my foreign coworkers are by continuing to do their jobs well.
The reason the State Department hides behind, by the way, is that domestic offices don't have the ability to take biometric data (fingerprints), which is self-evidently pretextual.
Right, stateside renewal was common until 9/11. Disappointingly, Biden tried to bring it back but was presumably thwarted by labor/union or incompetence. Immigration tends to be a bipartisan morass.
They say it out loud. The people writing these policies are vocally proud of being white nationalists. Defending them with "well maybe they have a good reason!" is evidence that you're either on board with their actual agenda (i.e., evil) or incomprehensibly naive (i.e., stupid).
First, that's not what this is talking about. "Leave to renew" becoming "leave, and you will not be renewed" on a whim is insane, self-destructive cruelty.
But also, it's only been "the policy" since 2004. That's technically "decades", but only just, and it was also introduced for idiotic pretextual reasons. The ostensible reason is that domestic State Department offices don't have the equipment to capture biometric data. Seriously. That's the argument. The law doesn't say "you have to leave to renew", it says "you have to get fingerprinted to renew", and conveniently there's nowhere in the United States that can do that, despite every police station in the country having that equipment at this point.
They come with renewal expectations, assuming the situation hasn't changed. Reserving the right to change the rules unilaterally doesn't make you not an asshole if you actually exercise that right. And government only functions if it's predictable - if it's a random-number generator (outside of processes that are explicitly lotteries), what is anyone supposed to do? Why would anyone engage with it?
In most states, employers have broad rights to fire employees at will, including no reason or "he wore the wrong color shirt". That doesn't mean that if you have a dickhead manager who fires you because he doesn't like the color of your shirt isn't a dickhead, and if you defended that manager I would similarly consider you unfit to participate in civil society.
> They come with renewal expectations, assuming the situation hasn't changed
What? Where is this expectation enumerated? An expiration date is in fact defined. I'm not sure where some legal expectation of guaranteed renewal could arise from.
> unilaterally doesn't make you not an asshole if you actually exercise that right.
"Asshole" is a matter of opinion, and rights aren't actually rights if you aren't ever able to exercise them.
> if you defended that manager I would similarly consider you unfit to participate in civil society.
Why have laws, borders at all by these standards? What's the point?
Norway is pretty much the only country with oil wealth that can reasonably be considered well run (unless you count Canada or the US, which, while resource-rich, have a lot of other things going on and aren't dependent on their petroleum resources).
Of the other countries where petroleum is >10% of the national economy, I think Norway might be the only one that isn't a brutal dictatorship, a failed state, or both?
> Of the other countries where petroleum is >10% of the national economy, I think Norway might be the only one that isn't a brutal dictatorship, a failed state, or both?
You raise a great point. I wracked my brain trying to think of counterexamples: It is tough!
One thing I want to point out: Kuwait is the most dependent upon oil and gas of any nation in the world (about 50% of its GDP). However, I think it would be a stretch to call them "a brutal dictatorship, a failed state, or both". Sure, it isn't a Western liberal democracy, but people live pretty good lives and the state does not have a strong reputation for suppressing its people. (I'm not here to apologize for their gov't... but I think it would be a bit harsh to put in your categories.) UAE, Bahrain, Oman, Brunei, and Qatar: I would put in the same category: Non-democracy or flawed democracy, but life is pretty good for the locals. I guess that we can call all of the countries that I mentioned before: "benevolent dictator" gov't model. Interestingly, Ecuador is about 20% of GDP, but I would call them a developing/emerging democracy. The rest of the list... sheesh: It does not look good!
What do you think about my analysis? Do you agree?
?? Russia, the US, half the damn gulf states seem to be doing just fine (maybe excluding Bahrain and Iraq). Very much including Iran, despite our efforts to strangle it to death for fifty years straight.
> I think Norway might be the only one that isn't a brutal dictatorship
Just because you don't like a government doesn't mean it's doing a bad job at, well, holding a state together.
Not eligible for the reason I stated: the US economy is much less than 10% petroleum. We have a lot of oil, but we have a lot more of everything else.
> gulf states
Mostly brutal, repressive dictatorships (especially Saudi Arabia and Iran) - some are better than others, but the trend is poor.
> Just because you don't like a government doesn't mean it's doing a bad job at, well, holding a state together
You're moving the goalposts a fair amount here. The thread started with "good governance", and I said "well run" - it's hard to say that the bar for either of those is simply maintaining superficial order. But even if you set that aside, consider the famous quote from the Shiekh of Dubai about the fragility of the petrostates that haven't figured out how to build strong civil societies with their wealth: "my grandfather rode a camel, my father rode a camel, I drive a Mercedes, my son drives a Land Rover, his son will drive a Land Rover, but his son will ride a camel."
> it's hard to say that the bar for either of those is simply maintaining superficial order.
Well it's clearly more than superficial or the state would be collapsing.
Regardless, I don't think the form of the government itself implies any sort of governance quality. Both Russia and Iran are the subjects of such heavy propaganda I don't expect anyone to evaluate them in good faith—but I would certainly prefer to live in either place compared to, say, Mississippi.
And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.
reply