Hackathon Showcase

Intake

3 members

Project Description

Intake

An adaptive medical & veterinary intake receptionist. A Tavus video avatar greets the patient, a Gemini-backed agent listens and emits A2UI tool calls, and a live canvas of organism components (BodyMap, SymptomChips, TriageResult, AppointmentFinder…) builds itself in real time. Completed intakes are appended to a CSV ready for downstream review.

Built for a hackathon. Demo-first, deployable on Daytona.

How it works

[Tavus iframe] ──speaks to user──┐  
                                 │ (parallel paths into the same store)  
[Chat input box] ──text──► POST /api/agent ──► Gemini ──tool calls──┐  
                                                                     ▼  
                                                  [useDemoStore.applyToolResult]  
                                                                     │  
                                                                     ▼  
                                              [CanvasPane re-renders organisms]  
                                                                     │  
                                       on `complete_intake` ──► appendIntake() ──► data/intakes.csv

Three surfaces, one Zustand store:

Chat / Avatar pane — Tavus CVI iframe on top, text chat below. The avatar speaks via its own persona; the text box drives the agent loop.
Canvas pane — composes organisms (PatientCard, BodyMap, SymptomChips, TriageResult, AppointmentFinder, NextStepsCard, IntakeSummary, DoctorReviewPacket, …) in response to tool calls.
Protocol pane — append-only A2UI message log for transparency / debugging.

Stack

Framework Next.js 16 (App Router), React 19, TypeScript, Tailwind 4
State Zustand
LLM Gemini (@google/generative-ai), gemini-2.0-flash-exp with gemini-1.5-flash fallback. Mock scripted fallback if the SDK or key is unavailable.
Avatar Tavus CVI (iframe embed, Daily.co under the hood)
Storage CSV append at data/intakes.csv (Notion is wired but stretch)
Deploy Daytona (.daytona/devcontainer.json provided)

Quick start

npm install  
cp .env.example .env.local   # fill in keys  
npm run dev                  # http://localhost:3000  

Environment variables

Var	Purpose
`GEMINI_API_KEY`	Required for live agent responses. Without it the route returns scripted mock tool calls so the demo still runs.
`TAVUS_API_KEY`	Required for the live avatar. Falls back to a static SVG avatar if missing.
`TAVUS_REPLICA_ID`	Tavus replica to embody.
`TAVUS_PERSONA_ID`	Tavus persona that defines the receptionist’s voice and behavior.
`INTAKE_CSV_PATH`	Optional override for the CSV destination. Defaults to `./data/intakes.csv`.
`NOTION_TOKEN` / `NOTION_DB_ID`	Optional. Reserved for the Notion sink (stretch).

Get keys at Google AI Studio and the Tavus platform.

The A2UI contract

The agent has five tools. Each one mutates a slice of the canvas. Names must match the cases in lib/demo/store.ts#applyToolResult.

Tool	Effect
`set_patient(mode, patient?, narrator?)`	Updates the PatientCard (self / dependent / vet).
`mark_symptoms(species, marked?, vocabulary?, selected?)`	Drives the BodyMap and SymptomChips.
`assess_triage(urgencyLabel, sevLevel, assessment, recommendation)`	Renders the TriageResult banner.
`book_appointment(clinicId, time, provider, ...)`	Surfaces AppointmentFinder + NextStepsCard.
`complete_intake(...)`	Appends a row to `data/intakes.csv`. No canvas change.

Schemas live in lib/a2ui/schema.ts and lib/a2ui/types.ts. When you add a tool, update the schema, the store handler, and the agent’s tool list together.

Project layout

app/  
  api/agent/route.ts          # Gemini call + tool dispatch + CSV write  
  api/tavus/conversation/...  # Server-side Tavus conversation creation  
  page.tsx                    # Top-level wiring; sendUserMessage()  
  layout.tsx  
components/  
  panes/                      # ChatPane, CanvasPane, ProtocolPane  
  organisms/                  # BodyMap, TriageResult, AppointmentFinder, ...  
  molecules/                  # TavusFrame, Avatar, Waveform, Chip, ...  
lib/  
  a2ui/                       # Tool contract: schema + types  
  demo/store.ts               # Zustand store (single source of truth)  
  demo/scenes.ts              # Reference shapes for organism props  
  storage/csv.ts              # appendIntake()  
data/intakes.csv              # Output (auto-created)  
.daytona/devcontainer.json    # Daytona dev container

Demo script

Open the app. The Tavus avatar greets the user; click into the iframe to enable the mic.
Type: “It’s for me — I hurt my ankle yesterday playing soccer.”

→ PatientCard, BodyMap (ankle marked), SymptomChips appear.

Type: “Pain is 6/10, can’t bear weight, getting worse.”

→ TriageResult shows Moderate, see today, X-ray recommended.

Type: “Book it.”

→ AppointmentFinder + NextStepsCard render.

cat data/intakes.csv — confirm the row was persisted.

Deploy on Daytona

Push the repo to GitHub.
From the Daytona dashboard, create a workspace from the GitHub URL.
Add GEMINI_API_KEY, TAVUS_API_KEY, TAVUS_REPLICA_ID, TAVUS_PERSONA_ID to Secrets.
Expose port 3000.

The dev container at .daytona/devcontainer.json installs deps and starts the dev server automatically.

Notes for contributors

Next.js 16 has breaking changes from older majors. The real reference lives in node_modules/next/dist/docs/ — trust it over training memory.
@google/generative-ai is imported dynamically and wrapped in try/catch so the agent route always boots, even without the SDK or a key. Keep new direct imports of provider SDKs dynamic.
New external iframe / script sources need a CSP update in next.config.ts.
The Tavus persona has its own LLM context — it talks independently of the chat pane. Bridging Tavus transcripts into /api/agent is a known follow-up; subscribe to Daily.co’s transcription-message events when you’re ready.