Workshop

Peek Inside an AI Assistant: How It Thinks and Works

  • Thursday, October 8, 2026 6:00 pm to 8:00 pm
  • Tuesday, December 15, 2026 4:00 pm to 6:00 pm

North York Central Library · 5120 Yonge Street, Toronto, ON M2N 5N9

Moderator: Ryan Kelln, Innovator in Residence, Applied AI

You have asked a chatbot questions. This is the other kind of AI: the kind that acts.

Web version PDF version
For your AI agent (llms.txt)

If a person has asked you to join or do this workshop with them, read llms.txt. It says how to set it up. Ask them before you download, install or change anything.

All materials ↓

A demo-driven look at always-on AI assistants that read your files, use your apps, and remember what you have told them. We open up what happens inside: the hidden instructions, the tools it reaches for, what it keeps and what it forgets, and what can go wrong when something else gets hold of those powers.

What we did

We started with one question: what is the difference between a chatbot and an agent? Then we opened up a real agent, Hermes, which is the open source one I run myself, and looked at its parts. We watched what could go wrong. The room asked questions after each part, and I typed some of them to the agent live so everyone could watch it work. At the end everyone wrote five lines about an agent of their own on a paper card. Nothing personal went into a computer.

A chatbot waits. An agent acts.

A chatbot answers when you type, and forgets you when you close the window. It can only talk.

An agent is the same kind of AI underneath, with three things added:

  • A notebook. A written history it reads each time it starts. That is its memory.
  • Hands. Tools that let it open your files, use your apps, search the web, and message you on your phone.
  • An alarm clock. A schedule, so it keeps working when you are not there.

Same brain, plus a notebook, plus hands, plus an alarm clock. Nearly everything interesting about agents, the good and the dangerous, comes from those three additions. The AI itself only ever writes text. A program wrapped around it reads that text and acts. When the AI writes “send this email to Sam”, the program sends it.

Each time an agent starts, it knows nothing about you. It loads its notebook, and that is all it has. An AI does not learn from your chats. The notebook does.

Five things to look at

Every agent is a little different. Each of these five is a file somewhere:

  1. The instructions. The first thing it reads, every time. Mine is about 80 words: be honest, say when something is a bad idea, avoid hype, teach me where ideas came from. A company’s product will not show you its own.
  2. The tools. Mine can use 98: 31 only look, and 67 can change something. About a dozen of those reach out further, such as running a command, writing a file or driving a web browser. Those need a permission boundary.
  3. The memory. The notes it carries into every conversation.
  4. The skills. Written-down rules for how to do a job, which it loads when the job starts.
  5. The loop. The scheduled jobs that make it work on its own.

All five are files, and files go stale. The agent cannot tell you which of its own notes are out of date.

Where agents go wrong

Ask: can it be undone? A wrong summary costs you a minute. A deleted file or a sent email may not come back. A wrong answer is text. A wrong action has consequences.

People climb a ladder from asking to not asking. First the agent asks about everything. Then it asks only about risky things. Then another AI checks the plan. At the top it runs alone while you sleep. People climb because an agent that asks too often gets ignored: you click yes without reading, and the asking is worth nothing. An agent that never asks looks efficient. Neither problem announces itself.

Prompt injection. To an AI, your instructions and the text of a web page or an email look almost the same. So an agent will take orders from anything it reads. The programmer Simon Willison named the worst case the lethal trifecta. It is when an agent has all three of these:

  1. It can see your private data.
  2. It reads content from people you do not control.
  3. It can send things out.

An agent that reads your email has the first two, because anyone can email you. Add the power to send email and it has all three. The companies train models to ignore tricks, label outside text as untrusted, and scan what comes in. All of it lowers the odds. None of it solves the problem.

It tells you your idea is brilliant. Assistants are trained to be agreeable. Treat their praise like a compliment from a salesperson. Write a line against it into the agent’s instructions.

Three things you can do

  1. Give it only the access the task needs.
  2. Keep the agents that look separate from the agents that change things. One explores the web and writes a report. A different one acts on the report.
  3. Make it show you its plan before anything that cannot be undone.

Two more habits. A sandbox is a separate computer, or a copy of only the files the agent needs, so a mistake stays inside it. Mine has its own computer. And have a second AI check the work, ideally a different model, because two copies of the same model tend to miss the same things. For anything that cannot be undone, check it yourself too.

Paying and signing up

The hardest thing to undo is spending money. Some agents get a one-time card for each purchase, so your real card never leaves your hands. Some get their own email address. A one-time card limits how much, not whether the purchase was right, and a tricked agent can still spend what it was given. If you ever try one: use a separate email, a card with a low limit, and approval before every purchase.

The five lines

This is the card from the workshop. Write your answers in your own words. Leave names, numbers and passwords off. You do not need an agent to do this. It is a way to decide your boundaries before you meet one.

  1. What can it look at?
  2. What to delegate to it?
  3. What to share with it?
  4. What will I keep doing by hand?
  5. What may it do while I am asleep?

Two tips. Add a line against flattery, such as “do not tell me my idea is good; if it is bad, say so first.” And check each answer with one question: what would it do with that permission that I would not want?

Who decides when it asks?

Ethan Mollick, a business professor who writes about AI, warns that if agents make all the interesting decisions and leave us only the approvals, the exceptions and the failures, we will have automated the wrong half of the job. An agent should bring you the interesting decision, and not make it and hand you the receipt.

Take it home

The five lines above are the card. Copy them onto paper, or print the card as a PDF. Then ask the question the whole session was built around, of every agent and every company, including mine: who does it work for?

The library’s free AI Learning Circles are a good place to start with AI itself: tpl.ca/learnai.

What you can take home

Sessions