As AI systems increasingly claim to “know” their users, the risks around trust, manipulation, and decision-making are growing just as fast. In this post, I explore whether AI truly understands individuals—or whether we are mistaking prediction for insight. For anyone working in governance, risk, or cybersecurity, the distinction isn’t just academic; it’s a critical emerging threat. Link to Substack post Can AI really get to know you? How well could an AI ever really know you? This week, I’ve been experimenting with using AI as a kind of coach — a patient, curious thought partner — and exploring what it seems to understand about me ...

Before an AI system can answer a query, write a paragraph of text, recommend a movie, or drive a car, it must first learn - and that process is called AI training. AI training involves teaching an artificial intelligence model to make accurate predictions or decisions. It learns by analysing large volumes of training data, identifying patterns, and adjusting its internal settings like numerical weights to improve its predictions. Throughout training the AI compares its predictions to known correct answers, refining itself over millions of cycles to reduce errors. However it’s important to remember: AI doesn’t understand its tasks the way a human would. It is simply refining its pattern recognition. ...

Authors: Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean Published 3 April 2025 AI 2027 report We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution. https://ai-2027.com/ Topics covered AI Agents (Unimpressive and error-prone at first, but starting to replace some employee tasks and companies inserting into workflows regardless. And getting better) Autonomous Coding AI Research (AIs creating better AIs) After being trained to predict internet text, the model is trained to produce text in response to instructions. This bakes in a basic personality and “drives.” For example, an agent that understands a task clearly is more likely to complete it successfully; over the course of training the model “learns” a “drive” to get a clear understanding of its tasks. Other drives in this category might be effectiveness, knowledge, and self-presentation (i.e. the tendency to frame its results in the best possible light). ...

Prompt engineering is a term coined in the 2020s to describe the practice of creating and crafting effective prompts to guide the behaviour and output of large language models (LLMs). Most people ask simple questions and can get simple answers. Somewhat like in the real world, the art of asking questions will bring the best results. Effective prompt engineering involves understanding the model’s capabilities - as well as its limitations - to achieve the most optimal results. ...

(4 hour) March 2025 interview with William MacAskill, author of What We Owe the Future. Introducing his new paper - Preparing for the Intelligence Explosion - 11 March 2025 Summary takeaway - if we’re about to have 100 years of progress happening in the space of 10 years, how do we prepare? Topics Software intelligence explosion (43m) Hardware intelligence explosion (45m), e.g. AI robots doing everything Space governance (1h21m50s) Very optimised education and training (2h7m45s) Training for politicians AI as a coach, career, productivity, life, health coach (2h11m53s) Dealmaking AI (2h12m48s) Further reading Moorhouse, F., MacAskill, W. (2025). Preparing for the Intelligence Explosion. Available at: Forethought.org Related pages Book summary - What We Owe The Future, by Will MacAskill Discuss on X Interview with Will MacAskill (March 2025) on AGI preparedness https://t.co/hiGmQ9hp4S #agipreparedness ...