CS Lessons on How AI Really Works
For data analysts and scientists, this breakdown of a Harvard CS lecture turns computer science theory into practical wisdom for your daily work.
Introduction: The Hidden Cost of “It Just Works”
Have you ever struggled to debug a model’s strange output or explain a “black box” prediction to a stakeholder? In data science, we often use powerful tools that seem to “just work”—until they don’t. The frustration of unexpected results often stems from a gap between using a tool and understanding its foundational principles. By looking at modern data problems through the lens of core computer science, we can move from simply using tools to commanding them with confidence and clarity. This post is a synthesis based on the CS50x 2025 (Harvard) lecture on Artificial Intelligence available on YouTube. It distills the lecture into five concrete takeaways that connect CS theory to the practical realities of data work, giving you high-leverage concepts you can apply immediately.
Takeaway 1: AI Is a Spectrum from Simple Rules to Complex Probabilities
Artificial Intelligence isn’t a monolith; it’s a spectrum. At one end, you have simple, deterministic systems. The lecture uses the game of Tic-Tac-Toe as a classic example. An “AI” for this game can be built using a decision tree—a series of if-else rules—and an algorithm called Minimax. The algorithm assigns a score to board states (e.g., +1 for a win, -1 for a loss, 0 for a tie) and works backward to choose the move that leads to the best possible outcome for itself. This approach is exhaustive and guarantees a win or a tie. However, this rule-based logic is only feasible in highly constrained environments. As the lecture notes, in a more complex game like Chess, there are over 85 billion ways to play just the first four pairs of moves, making a complete decision tree computationally impossible.
At the other end of the spectrum are modern, probabilistic systems like Large Language Models (LLMs). These models don’t follow explicit rules. Instead, as the lecture explains, they work by predicting the most probable next word in a sequence based on the vast patterns and frequencies they learned from their training data. They operate on likelihoods, not certainties.
Data Science Connection: This spectrum maps directly to common data science approaches. Writing explicit business rules in SQL, such as CASE WHEN ... THEN, is like building a decision tree. In contrast, training a statistical model that assigns probabilities to different outcomes is working on the other end of the spectrum. Understanding where your specific business problem falls on this continuum—from needing deterministic rules to benefiting from probabilistic inference—is crucial for choosing the right tool for the job.
Takeaway 2: “Prompt Engineering” Is Just Structured Context
The much-hyped skill of “prompt engineering” is less about esoteric commands and more about providing structured context. The lecture demystifies this using the CS50 rubber duck AI assistant as a perfect example. A prompt has two key parts: the system prompt and the user prompt. The user prompt is simply the question a person asks. The system prompt, written by the developer, is a set of instructions that tells the AI how to behave and what constraints to follow.
The CS50 team gave their duck a very specific system prompt to shape its behavior from a generic LLM into a helpful, but not cheating, teaching assistant. The instructions included direct commands like:
“You are a friendly and supportive teaching assistant for cs50.”
“You are also a rubber duck.”
“Do not provide full answers to problem set as this would violate academic honesty.”
These instructions provide the necessary context for the AI to generate a useful, domain-specific, and safe response, rather than just the default answer.
Data Science Connection: This is directly analogous to defining the scope and constraints for any data analysis project. A good “system prompt” for an analysis isn’t given to an AI, but to yourself and your stakeholders. It might include defining key business terms (“What exactly constitutes an ‘active user’?”), specifying the date range, or stating the core business question you are trying to answer. This initial framing prevents wasted effort, misinterpretation, and ensures the final analysis is relevant and correct.
Takeaway 3: The Critical Trade-Off: Exploration vs. Exploitation
One of the most powerful concepts in AI is the dilemma between exploration and exploitation. The lecture illustrates this with a simple maze game where an AI agent must find a green exit while avoiding lava pits. Initially, the agent finds a correct path by chance and, through reinforcement, learns to follow it repeatedly. This is exploitation—using known information to achieve a reliable outcome.
However, that first path might not be the best path. To find a more optimal, shorter route, the AI needs to introduce a small amount of randomness, defined in the lecture as an ‘Epsilon value,’ which represents a set probability (e.g., 5% or 10%) that the agent will make a random move instead of its learned best move. This encourages exploration—trying new, random moves even if it means risking a short-term failure, like falling into a lava pit. Over time, this small chance to explore allows the AI to discover superior strategies.
This principle was powerfully demonstrated in the example of an AI learning to play the game Breakout. Through pure exploitation, the AI would just move the paddle back and forth. But by exploring, it discovered the counter-intuitive but highly effective strategy of tunneling the ball up one side, allowing the game to essentially “play itself” and rack up a massive score.
Data Science Connection: This trade-off is central to data science work. A/B testing is a formal version of this: you continue to use the known winner (exploitation) while testing a new variation to see if it’s better (exploration). It applies to model optimization, where you might try new hyperparameters instead of sticking with the ones that already work well enough. It even applies to career development: do you stick with the tools you know (exploitation) or take the time to learn a new language or framework that might be more powerful (exploration)?
Takeaway 4: LLMs Are Giant Pattern-Matchers, Not Abstract Thinkers
According to the lecture, Large Language Models are not reasoning or “thinking” in the human sense. They are incredibly sophisticated pattern-matching systems. Built on neural networks, LLMs are pre-trained on vast quantities of text from the internet. Their core function is to calculate the most probable next word in a sequence given the words that came before it.
The lecture puts it simply: “…if someone asks me how are you question mark probabilistically based on all of this data I bet 99% of the time I chat PT or cs50’s duck is supposed to reply good thanks how are you.” The model isn’t feeling well; it’s just generating the statistically most likely response based on its training data.
This probabilistic nature is the direct cause of hallucinations. When an LLM confidently states something that is factually incorrect, it’s because the patterns in its training data suggest that the incorrect statement is a plausible-sounding sequence of words. It is assembling a statistically likely sentence, not verifying a fact.
Data Science Connection: This has profound implications for data professionals. Outputs from an LLM should never be treated as ground truth. Instead, they should be considered well-formed hypotheses that require verification against primary sources. This critical mindset is essential for maintaining analytical rigor. Using LLM output without validation risks propagating misinformation in your reports, dashboards, and models, undermining the trust placed in your work.
Takeaway 5: A Real AI Solution is a System, Not Just a Model
A useful AI tool is rarely just a standalone model. It’s an entire system built around that model. The lecture presents an “architectural diagram” for the CS50 duck debugger that illustrates this point perfectly. The CS50 team didn’t build their own LLM from the ground up. Instead, they created a valuable tool by integrating several key components.
The system includes a user-friendly interface for students, a local Vector database used to store and search transcribed lecture content, and connections to third-party APIs from providers like OpenAI. The team’s major achievement was not in creating a new foundational model but in assembling these pieces into a system that solves a specific educational problem effectively.
Data Science Connection: This is a crucial lesson for data scientists, demanding a mindset shift from “I build models” to “I architect data-driven solutions.” The real business value is rarely in a standalone model, no matter how complex. It’s in the system: the data pipelines that feed the model, the APIs that serve its predictions, and the interfaces that make it usable. Thinking like an architect means your focus expands from optimizing an algorithm to designing a complete, problem-solving system.
What to Apply Today
- Draft a “System Prompt” for Your Next Project: Before writing any code, write a paragraph defining your goals, constraints, and key definitions, just like the CS50 duck’s prompt.
- Identify Exploration vs. Exploitation: In your current work, find one area where you are “exploiting” a known process. Ask what a small “exploratory” experiment might look like.
- Map a Problem to a Decision Tree: Whiteboard a simple business problem using the
if-then-elselogic of a decision tree. Does this clarify the logic? - Verify, Don’t Trust: Treat the next piece of information you get from a generative AI as a hypothesis. Spend three minutes trying to verify it from a primary source.
- Think in Systems: When you build a model, sketch a simple diagram of the inputs it needs and the outputs it will produce. How will a person or another system use it?
Closing: From Code to Cognition
Understanding the fundamental computer science principles behind AI doesn’t just demystify the technology; it gives you a framework for control, clarity, and critical thinking. Concepts like decision trees, probabilistic reasoning, and the exploration/exploitation trade-off are not abstract theories but practical tools for building more robust and reliable data solutions. These ideas form the bedrock upon which you can build a deeper understanding of even more advanced topics. As these probabilistic systems become more integrated into our tools, how will we need to adapt our own methods for ensuring analytical truth?
