Fringe: the words for what they love

Live demo: https://fringe-aac.vercel.app · Built for the Qloo Agentic Hackathon · License: MIT

Fringe is an AI agent that builds personal AAC (augmentative and alternative communication) boards from Qloo's taste graph. It gives people who can't rely on speech the words for the shows, songs, games, books, foods and places they love, so they can do what most talk is for: connect with people.

One sentence from a parent or speech therapist ("loves Bluey, Taylor Swift, Encanto, Minecraft, mac and cheese") becomes a working, talking board of more than 60 picture buttons in about 5 to 10 seconds. Every button traces back to a Qloo entity or the caregiver's own words.

The problem

• About 4 million Americans can't reliably meet daily communication needs with natural speech (Beukelman & Mirenda, cited by the National Academies, 2017). Many speak through AAC: tablets and picture boards that turn a tap into a voice.

• Those boards are built around needs: eat, drink, toilet, pain. But adults rank social closeness as their first reason to communicate (Light's purposes of communicative interaction), and closeness runs on fringe vocabulary: the personal words for a favourite show, a band, a team or a local place.

• Today, fringe vocabulary is added by hand, one button at a time, by a parent or a speech-language pathologist (SLP) with a full caseload. It goes stale as soon as tastes change.

• That staleness is a known cause of abandonment. In a survey of 275 AAC specialists, not maintaining or adjusting the system and poor fit were among the factors most often linked to people giving up on AAC (Johnson, Inglebret, Jones & Ray, AAC 22(2), 2006).

• About 30% of autistic children remain minimally verbal even after years of intervention (Tager-Flusberg & Kasari, Autism Research, 2013). And AAC doesn't stop speech: in a review of 27 cases, none spoke less and 89% spoke more (Millar, Light & Schlosser, JSLHR, 2006).

• And the fit is cultural. A review of research on culturally and linguistically diverse AAC users found AAC practice is shaped by Western, Euro-American norms (Kulkarni & Parmar, AAC 33, 2017). Speech therapists have asked the open-source AAC app Cboard for country-specific tiles since 2022 (cboard#1267): alfajores in Argentina, queque in Chile.

An AAC board without culture is technically functional. A board that knows what someone loves is how they make friends. That gap is the one this hackathon asks about.

Related MCP server: Semesters MCP Server

What Fringe does

Why it needs Qloo

The language model handles language: parsing the caregiver's note and phrasing buttons. Qloo decides what goes on the board. The curation step may only choose from ids Qloo returned, and a verifier drops anything else.

Running the LLM-only control on Maya's note produced mostly restatements of the note (9 of 12 items), plus events that can't be verified ("Mac and Cheese Cooking Class for Kids at Central Market"). It produced no pictures, no checkable places and no audience signal. Qloo supplies the parts an AAC button actually needs:

What we tried and left out, on purpose. signal.demographics.age on the main expansion pulled an Indian 12-year-old's board towards US youth titles (Kota Factory and Shreya Ghoshal gave way to The Crystal Maze), so age is handled by content ratings and demographics instead; the age signal is used only for snacks. /v2/trending in the hackathon snapshot ends in June 2026 and is flat week to week, so it can't say what is rising now; Fringe uses each entity's popularity percentile to show how widely known a topic is ("an easy opener with anyone") instead.

Architecture

flowchart LR
  A[Caregiver note] --> U[Understand<br/>Groq gpt-oss-120b]
  U --> R[Resolve<br/>Qloo /search]
  R --> E[Expand<br/>Qloo /v2/insights x6<br/>+ age guard rails]
  R --> L[Localise<br/>Qloo /v2/tags + place insights]
  R --> H[Hot-spots<br/>Qloo urn:heatmap → place by geohash]
  R --> S[Snacks<br/>Qloo brand insights]
  E --> D[Read the room<br/>Qloo urn:demographics]
  H --> C
  S --> C
  E --> C[Curate + phrase<br/>Groq, closed set of Qloo ids]
  L --> C
  D --> V[Verify + assemble]
  C --> V
  V --> B[(Board<br/>core words + folders)]
  B --> P[Talk with partner<br/>Qloo compare + joint insights + audiences<br/>+ explainability + demographics]
  B --> X[Export .obz]
  B --> T[This or that<br/>Qloo pairs: picks as signal,<br/>another domain for range]
  T --> B
  B --> K[Keep it current<br/>Groq tool calling: add_interest, more_like,<br/>find_places, remove_buttons, add_phrases → Qloo]
  B --> N[Without-Qloo control<br/>Groq only, checked against Qloo]

• Next.js 16 (App Router, TypeScript) on Vercel. Agent runs stream as NDJSON, so the studio shows every step live.

• Qloo client (src/lib/qloo.ts): typed wrapper with stable cache keys, an in-memory TTL cache with in-flight de-duplication, timeouts, retry on 429/5xx, and response trimming (akas and other verbose fields never reach the LLM).

• LLM (src/lib/llm.ts): Groq's OpenAI-compatible API in JSON mode. Falls through gpt-oss-120b → gpt-oss-20b → qwen3.8-27b on rate limits. Every LLM step has a deterministic fallback, so the demo never dead-ends.

• Refine agent (src/lib/refine.ts): a bounded tool-calling loop (Groq function calling, up to 4 turns) with five tools, including more_like, which turns the buttons a person taps most into the Qloo signal. The model plans; each tool hits Qloo and reports back, so the model can react. The reply shown to the user is built from what the tools actually did, never from the model's own claims.

• Agent (src/lib/agent.ts, src/lib/bridge.ts): explicit steps with self-correction (widening a search when a name resolves to a non-boardable type, relaxing a too-strict rating filter, trying progressively broader food tags). Candidates go to the LLM under short handles rather than UUIDs, which saves tokens and prevents mangled ids.

• Device UI (src/components/Device.tsx): sentence strip, Web Speech output, Fitzgerald-coded core words, picture folders, caregiver review mode.

• OBF export (src/lib/obf.ts): root board with fixed core words, linked to one sub-board per folder.

Use Fringe from your own agent (MCP)

Fringe is also an MCP server (Streamable HTTP, stateless), so an existing agent such as Claude, an AAC app's assistant or a care-coordination bot can give its users culturally grounded vocabulary:

claude mcp add --transport http fringe https://fringe-aac.vercel.app/api/mcp

Tests

npm test   # vitest, 26 tests: age guard rails (ESRB, film/TV ratings), name matching, fallback parsing and planning, place filters, curation handles, core vocabulary, Open Board Format export

Beyond unit tests, Fringe was stress-tested black-box against the live deployment with 55 scenarios written from this README: typos ("Bluee", "Tayler Swift"), notes with only themes or only foods, notes in Spanish, prompt injection, a child whose family names M-rated games, unknown cities, Lagos and Tokyo, ages 3 to 100, a 6,000-character note, abusive refine requests, and every MCP error path. Plus 16 browser tests (switching persona mid-build, double-clicking Build, reload, export, review queue, 375px mobile). Things it found and fixed:

• Qloo search isn't typo-tolerant: "Bluee" came back as Blue Eye Samurai (TV-MA). The model now corrects spelling and the resolver only accepts close name matches.

• Mixing a theme tag with entity signals derails results (Bluey + "dinosaurs" returned adult dramas), so themes are a filter alongside favourites and a signal only on their own.

• When the language model is rate-limited, every step still finishes: a plain parser, a rule-based planner for "Tastes change", phrase templates, and Hindi and Spanish phrase patterns.

• Favourites rated above the person's age (film, TV and game ratings from Qloo) stay on the board for the caregiver to review, but never steer suggestions.

Run it locally

git clone https://github.com/Arnav020/fringe && cd fringe
npm install
cp .env.example .env.local   # add QLOO_API_KEY (hackathon key) and GROQ_API_KEY
npm run dev                  # http://localhost:3000

• QLOO_API_KEY: a Qloo hackathon key. The client calls https://hackathon.api.qloo.com with the X-Api-Key header.

• GROQ_API_KEY: a free key from console.groq.com.

No Qloo response data is committed to this repository. All Qloo data is fetched live and cached only in server memory.

Try it as a judge

1. Open https://fringe-aac.vercel.app/studio?p=maya. The agent builds Maya's board live; watch the steps on the right.

2. Tap pictures and core words to hear them spoken and build a sentence. Tap the info button on any picture: Qloo affinity, which of Maya's favourites led to it (explainability), how widely known it is, and who it lands with.

3. Scroll the left column to Where Maya's taste lives: the Qloo heatmap of Austin with her places pinned on it.

4. Open Talk with Grandpa Joe → build. You get the shared themes, how much each topic draws on each person's favourites, both age groups' affinity bars, and a new folder on the board.

5. Open This or that: tap one of two pictures six times. The pairs follow her choices; then add the picks to the board.

6. Open Without Qloo → run the LLM-only control and compare.

7. In "Tastes change", tap "She's obsessed with dinosaurs now" and watch the tool-calling agent update the board. After tapping a few pictures, a "More like what Maya taps" chip appears.

8. Turn on Caregiver review, keep or remove suggestions, then Export (.obz).

9. Try Aarav (12, Mumbai): his board knows Chhota Bheem, Motu Patlu, Lagaan, Shreya Ghoshal and the cricket grounds near him. Turn on Speak Hindi. Then Leo (16, eye-gaze user, Chicago) and Ruth (71, aphasia, Boston), or build your own.

Responsible design

• Fringe is a vocabulary tool, not a medical device. Boards are reviewed by the people who know the user.

• Personas are illustrative, not real individuals.

• Kids' boards use content-rating filters (and ESRB ratings for games), outing-only place categories, and a deterministic deny-list under the curation prompt for names a parent wouldn't expect on a child's board.

• Taps are counted only in the browser. Nothing is sent until a caregiver asks for "more like what she taps", and then only the button names.

• No accounts and no analytics. Board data stays in the user's browser.

License

MIT © 2026 Arnav Joshi. Taste data by Qloo Taste AI™.