I spent years thinking AI was something close to magic. Then I started building with it — and the curtain fell hard.
I remember the exact feeling. I was in school, sitting at my laptop, and I had just typed a question into ChatGPT for the first time. Not the app — this was before the app existed. Just the website, a blinking cursor, and an answer that came back so naturally it made my chest feel weird.
I’d been using Google Assistant for years. “Hey Google, what’s the weather.” “Hey Google, set a timer.” It always felt like talking to a very obedient robot with the personality of a filing cabinet. ChatGPT felt like talking to someone who had actually read something.
I didn’t understand what I was looking at. I just knew it felt different. And that feeling — that breathless, oh this changes everything feeling — stayed with me for a long time.
Then I became a developer. And developers, eventually, look under the hood.
“The moment I touched the API for the first time, everything I believed about AI started quietly falling apart.”
The first thing that broke was memory. I was building a small project, making API calls, and I noticed something strange: the model had no idea what I’d asked it two messages ago — unless I sent that message again myself. It wasn’t remembering anything. It was stateless. A complete blank slate with every single call.
I genuinely thought I’d made a mistake. I hadn’t. That’s just how it works.
Every time you have a “conversation” with an AI, it isn’t remembering you. It’s re-reading the entire conversation from scratch before it replies. The thing that feels like memory is just the model being fed its own previous answers as new input. It has no internal state. No recollection. No sense of you as a person who existed five minutes ago.
Think about what that means for a second. Every reply you get is from a model that just woke up, read your entire chat history in one gulp, and then responded.
Then I learned about the context window — which I had assumed was some clever memory system. Nope. It’s just the maximum amount of text the model can read at one time. That’s it. A limit, not a feature. Once you go over it, the earlier parts of your conversation fall off the edge and disappear forever.
These two realizations back to back were disorienting. But the third one was the one that really got me.
AI is, at its core, a next-token predictor. It was trained on an enormous amount of human text — books, articles, code, Reddit arguments, Wikipedia, everything — and what it learned was essentially: given this sequence of words, what word is most likely to come next? That’s the engine underneath everything. All the apparent intelligence, all the clever answers, all the code it writes — it all comes from that one mechanism, repeated billions of times.
“I used to be afraid of AI taking over the world. Now I understand it well enough to know it can’t remember the question it just answered.”
And this is where I’ll be honest with you: I think the term “Artificial Intelligence” is doing a lot of heavy lifting that the technology hasn’t earned. These aren’t intelligent systems. They’re extraordinarily sophisticated pattern-matching machines. The intelligence isn’t in the model — it’s in the hundreds of millions of humans whose writing the model was trained on. When AI gives you a brilliant answer, it’s not thinking. It’s reflecting humanity’s knowledge back at you, remixed and compressed.
That doesn’t make it useless. It makes it genuinely remarkable in a different, less cinematic way. Having instant access to the compressed output of millions of books and websites is actually powerful. But it has a hard ceiling: it cannot know what no human has written. If the answer doesn’t exist somewhere in its training data, it will either fail or — more dangerously — make something up with complete confidence.
Hallucination isn’t a bug they forgot to fix. It’s a natural consequence of how the system works. A next-token predictor doesn’t know when it doesn’t know something. It just keeps predicting.
Magic Until You Read the Spec
I remember the first time I wired up an MCP integration. The model reads a list of available tools, picks the right one, sends it a structured message, gets data back, and folds that into its response. When it worked, it felt like watching the AI reach out and touch the real world. Then I read the spec. It's an elegant API integration pattern — genuinely useful, genuinely well-designed. But the "autonomy" people talk about isn't in the model. It's in the plumbing someone else built around it. The model is still just predicting tokens. It just has better tools to predict with.
AGI is the part I find hardest to have an opinion on — not because I haven't thought about it, but because the more I understand current systems, the less certain I am about the path from here to there. What exists today is very good at one specific thing. General intelligence might need something architecturally different, not just more of the same scaled up. I don't know. I don't think anyone does. That uncertainty, to me, is more honest than a confident five-year prediction.
“Understanding something deeply usually makes you more humble about it — not more certain.”
Here’s what I actually believe after all of this: AI is impressive and it’s limited, and both of those things are true at the same time. The hype machine wants you to pick a side — either it’s going to save civilization or it’s going to destroy it. The boring, accurate answer is that it’s a powerful tool built on a surprisingly simple idea, and most of the people talking loudest about it have never made a single API call.
I’m not afraid of AI anymore. I’m not in awe of it either. I’m just using it — with my eyes open, knowing what it is and what it isn’t.
That feels like a better place to be.
Also available on Medium.