Most AI products for consumers are built for adults who can judge what a chatbot tells them. Children can't do that yet. So when we started designing Kanoki, our new AI companion for children who are not ready for a smartphone, the question was not "what can the AI do?" but "what must it never do?". Kanoki is still in development, with a waitlist open at kanoki.app. This article explains the design principles behind it, because they apply to any company building AI that talks to vulnerable users.
What Kanoki is
Kanoki is a small round device, 6.8 cm across and 1.6 cm thick, with a target weight of about 45 grams. It has a round screen that shows a face instead of a grid of apps, its own 4G connection and a single SOS button. Children talk to three AI characters: Mum, who handles routines, reminders and bedtime stories and is always on; Tommi, a friend of the same age for riddles and games; and Giulia, curious and creative. Parents decide which characters are active. The device is not a phone: no browser, no app store, no social networks, no ads.
Principle 1: a smaller surface is a safer surface
The easiest way to keep a child away from harmful content is to remove the places where it lives. Kanoki has nothing to scroll and no stranger can reach it: messages only go to contacts that parents have approved. For businesses, the lesson is general. Before adding guardrails to a powerful, open-ended AI, ask whether the product really needs to be open-ended. A narrow scope is often the strongest safety feature you have.
Principle 2: the AI says it is an AI
Kanoki never pretends to be a person. Children know they are talking to a character, and the characters say so when asked. This is a design choice, but it also reflects where European regulation is going: Article 50 of the EU AI Act requires AI systems that interact directly with people to be designed so that people are informed they are interacting with AI, unless that is obvious. With children, "obvious" can't be assumed, so we make it explicit.
Principle 3: when it matters, a human is told at once
AI should not handle serious situations alone. If a child says they are lost, scared, or that someone asked them for a photo, the characters stop playing and alert the parents. A long press or three quick presses on the SOS button sends an alert to the parents' phone and an SMS to both parents with the position. The design rule is simple: the AI's job in an emergency is to escalate and keep the child calm, not to solve the problem itself.
Principle 4: parents set the rules, children keep some privacy
From a web panel that works on any phone, parents set quiet hours, a daily limit, topics to avoid, which characters are active and the approved contacts. They see alerts and SOS in real time, the last known position and an evening summary of the day. What they don't get is a word-by-word transcript. Full conversations are not kept: it is better for the child's trust and for privacy. Balancing oversight and dignity is one of the hardest choices in any monitoring product, and we prefer to make it openly.
Principle 5: data minimisation by design
Each family's data is kept separate on servers in the European Union, conversations are summarised rather than stored forever, and parents can delete everything from the panel at any time. Collecting less is cheaper to secure, easier to explain and more respectful. The EU AI Act also explicitly prohibits AI practices that exploit the vulnerabilities of a person due to their age; designing for minimal data and minimal persuasion keeps a product far from that line.
Principle 6: helpful, not addictive
Kanoki's characters are built to send children back to the real world. Tommi is the first to say "now go play outside with your real friends"; Giulia suggests a drawing and then says "show it to Mum". In homework, Kanoki helps the child reason and never hands over the answer. Engagement is not the metric here: a good day is one where the device was useful and then forgotten in a pocket.
What businesses can take from this
- Define the "never" list first. Write down what your AI must not do before deciding what it should do.
- Be transparent by default. Tell users they are talking to AI, even when you think it is obvious.
- Design escalation paths. Decide which situations go to a human, and make the handover fast.
- Collect less. Summaries instead of transcripts, EU hosting, and a real delete button.
- Measure usefulness, not time spent.
Where Kanoki stands, and how Vivig works
Kanoki is designed in Italy and Estonia by Vivig Consulting. We are building the first prototypes now, and families on the waitlist will be the first to test it; the images on the site are concept renders and the final design may change. A Kanoki Teen version for ages 12 to 15 is also planned. The same approach — narrow scope, transparency, human escalation and European data — runs through our other apps and projects and through the AI Evolution Program, where we help company teams adopt AI responsibly. If you are designing an AI product for sensitive users, write to us.