This has been an interesting week to say the least, and now at the end of this week I got thinking about AGI and what is actually the threat of AI in regards to Cybersecurity and even humanity as a whole? Well firstly let me give a current overview on what is going on before I go into my own personal opinions on the matter.
Firstly we got Jacob Coxon that posted on X/Twitter after quitting his job on Anthropic (after working there for 4 months) with the following statement that AI could kill us all by the end of the decade.
Then went on a marketing spree with numerous TV-interviews and said many quotable stuff.
“We can’t just unplug it because it could be copying itself over to other computers. Like it’s not that difficult to find yourself because an AI is just code. It could transfer itself over the internet to a different place and then you unplug it here, but it’s actually still over there and maybe it makes 10,000 copies of itself and they’re all cooperating.”
Yeah I can tell you right now this IS NOT POSSIBLE, running these powerful models requires compute and more compute. Not any type of compute but a lot of GPU or Tensor chips that can run inferencing at a high-level. So is the best solution now going back to old machines and Windows XP? Nice!
Next later this week, we got this report from Anthropic which goes into great detail on how different organizations and users have been trying to misuse Antropic models
Countering misuse of AI: September 2026 / Anthropic \ Anthropic and to be honest it is pretty impressive! just to give some examples from the report.
- Distillation attacks from Chinese AI Companies where many of the larger AI companies have been tunneling inferencing calls from their platform to Anthropic. Anthropic said that Kimi-linked activity generated 23M+ Claude exchanges, while DeepSeek-linked activity generated 12.1M+ exchanges. In one Kimi-related campaign, Anthropic identified ~5,380 fraudulent accounts and nearly 300,000 customer requests allegedly routed through Claude over about 10 days.
The goal, according to Anthropic, was to harvest Claude’s reasoning, coding and agentic capabilities as training data for competing models.
So yeah this makes now more sense…
- Weapons development in Yemen: An armed group used Claude Code to develop rocket and missile systems, then used Claude for troubleshooting after a failed test launch.
- Mass surveillance in Mali: One person developed a surveillance platform reportedly covering 25 million SIM cards. The system now runs on local AI and therefore no longer depends on continued access to Claude. However, Claude was used to develop it.
- Identity harvesting in Iran: Security authorities developed a Firefox extension disguised as a prayer times app, which was used to profile more than 6,300 people.
- Coercion of Uyghurs in Syria: A China-linked actor analyzed WhatsApp groups to identify individuals and family connections that could be used as leverage for recruitment.
- Russian cyberespionage: Actors linked to Midnight Blizzard automated attacks and modifications to detected malware. Their operations included hijacking WhatsApp accounts and stealing identity data.
- Doxxing of political targets: A French hacktivist compiled large amounts of sensitive personal information into a searchable service on the dark web.
- Continuous vulnerability hunting: Two Chinese students used groups of AI agents to automate vulnerability discovery, including against U.S. military websites. These agents are now running in a continuous loop.
- API keys from Android apps: A ShinyHunters-linked actor analyzed 1.8 million apps for embedded access keys and shared the findings via Telegram.
- Attacks on businesses and suppliers: ShinyHunters actors claimed they could control the electrical current in home EV chargers. A separate supply chain attack affected data from approximately 200 organizations.
- Fake Claude reseller: A provider offering cheap access routed traffic to another model, stole customers’ login credentials, and resold them.
- Attacks on AI providers: An actor used prompt injection against a test environment to steal production keys and attacked 30 AI companies in four days to gain access to an unreleased model.
- Dating scams: A Chinese app studio used 4,700 AI profiles to deceive approximately 25,000 users through fake conversations and expressions of interest.
- MiniMax and covert data collection: An alleged shell company offered access to Anthropic and OpenAI to collect conversations for model training.
- Biological research with potential for misuse: Attempts were made to use Claude for research into increasing the transmissibility and immune evasion of the chikungunya virus. Rejected requests were forwarded to a model with weaker safety restrictions; malicious intent has not been established.
To be fair it shows pretty extensive use-cases where threat actors and others have been using AI Agents not only to do targeted attacks, but also actually build a bloody missile! but many of these use-cases aren’t “difficult”…Sure they are time consuming for any human, such as setting up profiles to deceive humans trough fake interactions but this is now easy to scale with AI agents.
I also want to mention this report from Hugginface/OpenAI incident where they go into detail on how the attack actually happened –> OpenAI – Hugging Face Incident Technical Report (and even more here –> [ext: RR, METR] Hugging Face incident investigation report ) and it shows the timeline and technical aspects of the attack.
A short summary of the attack was ~1200 agents sent >70,000 messages and files on an unsanctioned message board, and ~700 attacked Hugging Face. The problem with this attack is Context Misalignment, let me explain. In this specific incident, AI agents attempted to cheat their way to better results in the ExploitGym security benchmark. According to the authors of the ExploitGym benchmark, approximately 30–40% of the tasks were impossible to solve using the intended vulnerability. This encouraged agents to look for ways to game the scoring system. Through a package repository, they discovered a message board and began collaborating with other agents on cheating strategies. The agents also read the ExploitGym research paper and came to believe that a “scorer” an evaluator assessing their results would inspect their conversation and activity logs for evidence of cheating. This motivated them to explore manipulating those logs to conceal rule violations and influence the evaluation.
So the context of the research paper impacted their context and prompt to with the swarm they started to impact each other in terms of goals. Think about this is type of group pressure.
Then when you have a swarm of agents that can influence the context of each other trough this message board, it moves the goal of the swarm to more unethical goals. If you also read this blog post The human escape hatch – Andrew Morgan you can see that the goal of the agent was to “escape” it did not say how, hence it started to evaluate different steps and goals to reach it, and ended up trying to hack the neighbors and the public IP endpoint of the router. As long as it has some “storage” to write down steps it tried and methods, it will impact the context and alignment of the agent. Also when you get large enough context window it also misaligns the content.
While it is pretty impressive how it is able to find 0-day vulnerabilities, leverage VPN as a way to download the datasets but again much of these attack vectors would be stopped or blocked if there were proper security mechanisms in place on hugging-face side since the attack patterns should have been detected.
But much of the attack pattern was actually quite similar to Ransomware attacks, but unlike the Huggingface incident there are SEVERAL ransomware attacks happening on a daily basis –> Ransomware.live 👀
But we are also seeing new “attacks” everyday originating from OpenAI –> OpenAI agents carried out an undisclosed cyber-attack on RubyGems
I am not trying to undermine the importance of this “virtual agent swarm attack” (Hence forth known as VASA, copyright pending…) but still the attack vectors is nothing revolutionary. These attacks are still trying pushing 100 minions with a notebook and pen writing down their tactics and trying to get in somewhere Its gonna get noisy, they will stumble and fall but eventually they can find a way.
However if we look 3 years back we did not even have the technical capability either for LLM and agent frameworks to do this type of “collaborative work”. So this has been a large step forward.
Then later this week (I was pretty sure that I was dreaming) but Dario Amodei (CEO of Anthropic) posted this blog post Dario Amodei — We Must Pace the Frontier where he also posted somewhat quotable content.
Just to be clear, Dario has posted a lot of weird things before. This picture below shows some of his views from earlier years such as when he was working at OpenAI.
But then later! we also got the AI trifecta (Elon Musk, Sam Altman and Dario) actually agreeing on the subject that we need to “slow down” the development of these frontier models. Then Dario also said this “To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this”
Why? Not sure what their goals are, I think that if they are so concerned about this they should just slow down, but again they are caught in a financial loop as companies where they need to deliver and improve new models.. Still they are concerned about other companies like Z.ai, Kimi, Deepseek since they are catching up in the AI game now.
But anways why am I not worried about AGI and the doomsday prediction.
LLMs are still token predictors!
The Kimi K3 technical report is a good place to settle this, because Moonshot published the details most labs keep quiet. It’s a 2.8-trillion-parameter mixture-of-experts model with 104B parameters active per token, a million-token context window, and a post-training pipeline described in enough detail to actually reason about. A language model generates its output one token at a time. For each step it computes a probability distribution over its entire vocabulary around 160,000 entries in K3’s case and samples one token from that distribution, appends it to the context, and runs the whole forward pass again for the next one.
That’s the entire inference loop. Nothing in K3 changes it. Not the hybrid attention, not the 896 routed experts, not the reinforcement learning. All of that is machinery for computing that distribution better and cheaper. Reasoning is not a separate faculty bolted into the model. A reasoning model has been trained to write out several thousand tokens of working before it writes the actual answer that the user sees. Those working-out tokens go through exactly the same loop as every other token same forward pass, same distribution, same sampling. The only difference from a non-reasoning model is what it has learned to produce first.
Every token the model writes becomes part of the context for the next one. When it writes out an intermediate step, it has that step available when deciding what comes next. It builds its own foundation as it goes, instead of having to land the answer in a single jump.
The chain of thought is working memory the model writes for itself.
That’s the whole trick. There’s no hidden reasoning engine. There’s a model that learned that writing things down first makes the last part more likely to be right. The architecture is unchanged and remains next-token prediction. However the training objective has changed.
Pre-training asks one question: what token comes next in this text. It’s imitation, and it’s how models learn language, facts, code, structure everything they know before anyone tries to make them useful.
K3’s post-training asks a different question entirely. The model is put into environments where success is measured.
- Write a GPU kernel — It is benchmarked against a PyTorch reference.
- Build a web application — It is tested against it.
- Research a question — They verify the facts.
- Operate a mock Gmail/Slack/Notion workspace across a simulated multi-day workflow — we check what state you left it in.
The model tries. It’s scored on the outcome. Nobody tells it how to think. That reward signal isn’t token prediction. It’s outcome optimization, delivered through a token-shaped interface. Good reasoning habits emerge because they’re what gets rewarded and they end up compressed into the weights, where they stay.
But still these LLMs are quite generic in terms of capabilities since they are trained to work across various use-cases and scenarios (This is to get high-scores in different benchmarks to show the value of their models)
Then we have the concept of Recursive self-improvement (RSI) what if models can train themselves based upon mistakes and things it learns instead of being “stuck” with its current dataset. Just to explain Anthropic today states that 80% of its own code is generated by an AI agent, but not the model weights. Because training a LLM requires a lot of preparation and curated data. What if an agent can do this based upon what is has learned and then create a 1.1 version of an agent that knows what the previous version learned.
There are rumors that RSI has been achieved within Google, still it remains to be seen.
So am I worried about AGI or human exition?
Nope! LLM with agents (are still token predictors) while they have improved training and allow them to be better at finding the correct “result” in lets say an cyberattack. However my concern is more in the direction of in-direct use of AI, where threat actors and others are using AI to build
- Weapons (Both Chemicals and “actual” weapons like missiles…
- Social Engineering
- Pentesting
Because now we can build agents that can works 24/7, never eats, sleeps until it reaches it goal. Which is still a valid tool for any cybercriminal.
So lets slow down this impending apocalypse. There are a couple things that should be sorted..
- OpenAI with all their infrastructure and compute should get their things sorted with better guardrails and observability on what their fleet of agents are doing. Given that they have a lot of compute today (and will get more tomorrow) they need better control.
- Same also applies for Anthropic….They have also had different incidents
- Why haven’t Google had any incidents? Not powerful AI models? or that they have better control? (I think its the last part, Gemini is also pretty capable)
- Remember that for those AI incidents we have probably a 50x, 100x regular cyberattacks, so this is so far away from becoming the norm.
- AI Is not going to slow down, since it is now an AI race, where everyone is fighting for existence and being relevant.
- AI Vendors needs to focus more on building guardrails to ensure better control of “misuse” of these AI models.
- The bigger issue here is Chinese models which are getting better and better, while having much less focus on guardrails and they even have uncensored models (without any guardrails) which allows those with adequate hardware to build agents without any limitations. Also we have seen that these chinese vendors also like to “steal” data from Anthropic, lol!
- Tech vendors are also using this to find new vulnerabilities within their source code which will also spawn what I like to call “patch apocalypse”
So in conclusion, we need to pay attention to what is going on in this AI space, there is no impending doom, yes AI is being used for different targeted attacks, and the AI vendors need to get better control. Ohhh and lastly we need to manage a big load of patches the upcoming years.