🎓

Think & Speak

AI English Learning Platform

Empowering multilingual classrooms with real-time translation, phoneme-level pronunciation feedback, intelligent assessment, and personalized mastery tools.

🔧 React + TypeScript ⚡ Next.js 🧠 Azure Speech SDK 📡 WebSocket Real-Time 🌍 Trilingual: EN / 粵 / 普

Teacher Workshop Presentation · Click anywhere or press → to advance · Press 📝 for speaker notes

01 / 15

THINK & SPEAK™

Complete Learning Ecosystem

One Platform. One Learning System.

📝

Assessment

Personalized learning profile builder capturing age, proficiency, interests, motivation, and learning style to power adaptive content delivery.

📚

Course Adventure

5-task lesson structure: Video → Comprehension → Glossary → Download → Speaking. Gamified progression with clear milestones and achievements.

🎤

Conversation Role Play

Structured academic speaking contexts — presentations, debates, discussions — with phoneme-level pronunciation feedback and 4D scoring.

📖

Read & Speak

Daily Reading with AI personalization + Personalized Script upload and practice. Pronunciation comparison with AI samples.

🧠

Vocabulary Builder

10 progressive levels (First Words → Extreme Words), 5 tasks per word: Learn, Spelling I/II, Usage, Dictation. Live phoneme feedback.

🔄

Live Translation

Real-time bilingual subtitles (English ↔ Mandarin/Cantonese). Smart note capture, auto-flashcard generation, session reuse.

Supplementary Modules: 🤖 AI Tutor (GPT + RAG) 🏛️ Group Discussion — DSE Paper 4 🎯 Reading Comprehension — Exam Mode 📊 Dashboard & Analytics

02 / 15

THINK & SPEAK™

Live Translation Engine

Breaking Language Barriers — Turning Every Class into a Learning Ecosystem

🌐 Real-Time Bilingual Subtitles

English ↔ Cantonese (zh-HK / zh-Hant) with Cantonese phonology
English ↔ Mandarin (zh-CN / zh-Hans) with Simplified characters
Intelligent auto-routing: EN→yue-HK, ZH→EN by default
Azure Speech Translation Recognizer: en-US source, zh-CN + yue-HK targets
Intermediate "recognizing" events enable real-time subtitle streaming
Final "recognized" events populate session history for vocabulary extraction

📐 Technical Architecture

WebSocket communication layer (NEXT_PUBLIC_WS_ROOM_URL endpoint)
Exponential backoff reconnection: max 3 attempts, 10-second delays
Auto-refresh token cycle every 10 minutes (autoRefetchRecognitionTokenTimeoutRef)
Chat channel (translations) separated from Room channel (presence updates)
Message objects: originalText, translations Record<string,string>, timestamp, senderId
10-second polling for room metadata and attendance rosters

📝 Smart Note Capture

Students highlight and save important sentences during live sessions
Personal remarks with AutoComplete suggestions: Key Point, Don't Understand, Exam Material, Review Later
Unicode-aware tokenization: /\p{L}/gu regex preserves multilingual boundaries
Numeric filtering — excludes number-only strings from vocabulary capture
Auto-flashcard generation from captured vocabulary

🎯 Classroom Management

Three-state attendance: Attending (WebSocket), Invited (API), Available
Room status machine: NOT_STARTED → STARTED → IN_PROGRESS → COMPLETED
Real-time engagement heatmap: question/understand signals
Broadcast announcements and material distribution during class
Differentiated grouping for targeted instruction

The Complete Learning Loop: 👂 Listen → 📋 Capture → 🔄 Review → 🎓 Master

03 / 15

THINK & SPEAK™

Group Discussion — DSE Paper 4

100% HKDSE Paper 4 Format Replication with AI-Powered Exam Simulation

10 min

Part A: Preparation

Stimulus documents + talking points

8 min

Part A: Discussion

Timed group conversation with AI

10×60s

Part B: Individual

Individual response with AI evaluation

🎯 Four Discussion Phases

1. Opening
Topic introduction and position establishment

2. Exploration
Idea development and evidence exchange

3. Challenge
Counter-argumentation and critical thinking

4. Synthesis
Conclusion and consensus building

📊 Assessment Metrics

Interruption Count: Turn-taking etiquette measurement (critical DSE criterion)
Phase-Based Analysis: Evaluation across all four discussion phases
Part A + B Separation: Mirrors official DSE scoring structure
Rank Badge: Overall performance level indicator

🤖 Six AI Persona Roles

🎯

Facilitator

Guides discussion flow

⚡

Starter

Initiates topics

🔧

Builder

Develops ideas

🔥

Challenger

Questions assumptions

🔗

Connector

Links concepts

💡

Clarifier

Explains complexities

DSE mode requires 3–5 AI personas (per exam specifications). Teachers can queue specific persona interventions.

🎤 Voice Processing Pipeline

🎙️ Speech

→

☁️ Azure SDK

→

📝 Transcribe

→

🌐 Translate

→

💬 Subtitles

Azure Cognitive Services: enterprise-grade phoneme-aware ASR
2-second silence debounce for natural conversation flow
Accumulated voice ref for multi-utterance processing
Interrupt tracking: records where AI was interrupted for assessment

Four Learning Tracks: 🎤 Public Speaking | 🏛️ Debate & Argumentation | 👔 Leadership Communication | 💬 English Fluency

04 / 15

THINK & SPEAK™

Phoneme-Level Pronunciation Assessment

Four-Dimensional Scoring — Not Just Right/Wrong, But Exactly Which Sound Needs Work

92

Accuracy

78

Fluency

95

Complete

87

Pronunc.

🔬 Assessment Pipeline

🎙️ Record
16kHz stereo

→

✅ Validate
Blob + duration

→

☁️ Upload
FormData → API

→

🔬 Analyze
Forced alignment

→

📊 Score
4D metrics

📊 Four-Dimensional Scoring

Accuracy: Phonetic precision and phoneme alignment with target pronunciation
Fluency: Rhythm, speed, prosody, natural pause placement
Completeness: Lexical coverage — all words in target phrase pronounced
Pronunciation: Overall quality metric combining all factors

🎨 Phoneme-Level Color Coding

The importance of environmental protection cannot be overstated .

✓ > 70

Native-like

⚡ 40-70

Needs work

✗ < 40

Mispronounced

🔤 IPA Phoneme Breakdown

phonemeScores: [
  { phoneme: "/θ/", score: 85, feedback: "Good" },
  { phoneme: "/æ/", score: 72, feedback: "Needs practice" },
  { phoneme: "/ɛ/", score: 91, feedback: "Excellent" },
  { phoneme: "/ɪɹ/", score: 68, feedback: "Retract /ɹ/" }
]

Distinguishes /θ/ vs /s/, /æ/ vs /ɛ/ — known Cantonese speaker difficulties
Click any word → phoneme-level breakdown panel
Words < 50 auto-added to vocabulary practice queue

🎮 Mastery Tracking

Archive threshold: 80-90% (deliberate practice theory)
practiceCountMax enforcement — quality over quantity
Chronological practice records with date-fns parsing
Disabled "Mastered" button creates closure satisfaction

05 / 15

THINK & SPEAK™

AI Tutor & Reading Comprehension

Intelligent Learning Companion + HKDSE Exam Preparation

🤖 AI Tutor — Dual Model System

Model.GPT — AI Tutor

General English conversation practice
Everyday vocabulary and fluency
Casual discussions
Pronunciation improvement focus

Model.RAG — AI Debate Tutor

HKDSE-aligned academic content
Structured argument frameworks
Evidence evaluation training
Rebuttal skill development

🗣️ Interaction Modes

Text-to-Text: Type questions, receive written responses
Text-to-Speech: Type questions, hear AI audio answers
Speech-to-Text: Speak, receive transcribed responses
Speech-to-Speech: Full voice conversation mode

🏗️ Conversational UX

Contextual auto-scroll: mimics human tutor maintaining eye contact
Activity state awareness: prevents accidental "Clear History" during AI explanation
Conversation history as learning portfolio — analytics logging before clear
Interruptibility: Skip button (opacity-0 → opacity-100 on hover) — students can interject

📖 Reading Comprehension

⏱️ Exam Simulation Mode

Timed reading tasks for speed and focus
Exam-authentic passages (HKDSE aligned)
Real exam pressure training
15-minute timed sessions

🤖 Guided Learning Mode

AI Coach available during practice
Mentor-style strategic hints
Build exam thinking, not just answers
Three-tiered hint system (0.55→0.35→0.22 opacity)

📝 Words & Vocabulary Builder

Ten Progressive Levels

First Words Extreme Words

5 Tasks per Word: Learn & Read → Spelling I → Spelling II → Word Usage → Picture Dictation
Auto-collect from: Lessons, readings, role play, live translation
Category filtering: Subject, Theme, Difficulty, Custom Tags
Your Library: Mastered words with scores above 80

"Read Smarter. Think Deeper. Score Higher."

06 / 15

THINK & SPEAK™

Assessment, Dashboard & Progress Tracking

See Progress Clearly — Personalized Learning Starts with Understanding the Learner

🎯 Intelligent Assessment

Captures Age group & learning stage
Captures English proficiency level
Captures Interests & preferred topics
Captures Learning motivation & learning style
Captures Academic & subject curiosity
Powers personalization across all platform modules

Personalized learning starts with understanding the learner.

🔐 Authentication & Access

Dual-role: Teacher (admin, progress, assignments) | Student (dashboard, vocabulary, practice)
Registration: Invitation Code, URL Link, QR Code — School Program data segregation
Security: 6-digit verification codes, RSA encryption, optional PII (K12 compliance)
State: localStorage persistence via useLocalStorage hook — crash recovery

📊 Student Dashboard

127

Practice Sessions

342

Words Mastered

48h

Time Spent

8

Courses Completed

🎙️ Speaking Scores (Sample)

Accuracy92%

Fluency78%

Completeness95%

Pronunciation85%

📱 Cross-Device Sync

Resume discussions from any device (mobile ↔ desktop ↔ tablet)
Synchronized vocabulary flashcards everywhere
Continue timed practice sessions seamlessly
Redux state + localStorage hybrid persistence architecture

07 / 15

THINK & SPEAK™

Technical Architecture Deep-Dive

Enterprise-Grade Infrastructure for Reliable Classroom Deployment

⚡

Frontend Layer

React + TypeScript — type-safe component development
Next.js — optimized page loading for classroom environments
Redux — centralized real-time session state management
Ant Design — UI component foundation with custom CSS Modules
next/dynamic — code-splitting for modal components
Barrel exports — tree-shaking via Index.tsx patterns

📡

Communication Layer

WebSocket — real-time channels via NEXT_PUBLIC_WS_ROOM_URL
Azure Speech SDK — enterprise-grade phoneme-aware ASR
Chat + Room channels — separated for translation vs presence
Exponential backoff — max 3 reconnect attempts, 10s delays
Token auto-refresh — 10-minute cycle via timeout refs
Polling — 10s intervals with failure-count ceilings

☁️

Backend Services

Azure Speech Translation Recognizer — en-US → zh-CN + yue-HK
RSA encryption — PEM key pairs for token security
API endpoints: callAddScript, useCallGetTaggedWordList, callGetWordDetails
Multi-tenant — school hierarchy with pageSize: 10000
Room lifecycle — RoomStatus state machine management
Export — session transcripts for downstream processing

🎨

Iconography System

Modular SVG — standalone components accepting className + style props
currentColor stroke — dynamic theming based on context
aria-hidden="true" — screen reader accessibility compliance
Scale variants: 16×16 (dense) → 40×40 (primary actions)
Note icons: IconNote (34px), IconMyNotes (16px), IconNotebook (14px)
Learning icons: IconVocabulary, IconReading, IconGrammar

🎭

Animation & UX

LinesAnimation: SVG SMIL (GPU-accelerated 60fps)
9-second loop — aligned with attention span refresh cycles
3 opacity layers: 0.55 (explicit), 0.35 (hints), 0.22 (subtle)
nodRocket keyframe — ARI mascot "nods" during AI processing
WCAG 2.1 AA — accessibility compliance across all components
Responsive: useBreakpoints hook for desktop ↔ mobile

🛡️

Security & Data

Redux (ephemeral) + localStorage (persistent) hybrid strategy
RSA genKeyPair + decryptFromServer — Azure token encryption
School data segregation: invitationType === 'School Program'
K12 privacy: optional PII fields, minimal data collection
Session history — Message objects with translation caching
Production: app.thinkandspeak.com | UAT: uat-app.thinkandspeak.com

08 / 15

THINK & SPEAK™

Student Learning Interface — AttendantSidebar

The Personal Knowledge Base: Vocabulary Capture, Script Creation & Flashcard Mastery

📋 Session History & Intelligent Capture

Personal record of classroom discourse via sessionHistoryList
Unicode-aware tokenization: /\p{L}/gu preserves multilingual boundaries
Numeric filtering: excludes number-only strings
Timestamp formatting: HH:mm:ss (today), MM-dd (this year), full dates (archival)
getWordEntity / hasWordEntity with useMemo performance optimization
Session history words: green highlighted with sessionHistoryWordWithEntity CSS class

📦 Dual-Channel Capture

📜 VOCABULARY CHANNEL

Words → Flashcards
Pronunciation practice focus
IPA transcription inherited
Auto-tag "To Practice"

📓 SCRIPT CHANNEL

Phrases → My Notebook
Contextual review focus
≥5 words auto-add (400ms debounce)
Source message linkage

🏷️ Metacognitive Annotation

Key Point Don't Understand Exam Material Review Later To Practice

AutoComplete suggestions via draftingInput field — builds self-awareness about learning patterns over time.

🃏 FlashCardDialog — Spaced Repetition

Communication

/kəˌmjunɪˈkeɪʃən/

fn (noun)

🔄 Review ✓ Master

Step 7 of 15 | Mastered: 42 / Total: 128

Ternary states: Need Review → Active → Mastered
Z-index + rotateY 180° for depth perception (card stack metaphor)
Gradient: linear-gradient(124.76° from #00A1FF) — premium game feel
Audio playback with animated ripple effects on mic button

➕ AddScriptDialog — Input to Output

Batches session history items → concatenated practice scripts
180-character minimum for substantive practice passages
Read-only textarea preserves authentic classroom discourse
React Hook Form validation, next/dynamic for code-splitting
Bridges ephemeral translation layer → durable content management

Complete Pipeline: Words Encountered→ Words Module Bank→ Vocabulary Trainer→ Your Library

09 / 15

THINK & SPEAK™

Gamification Psychology — ARI Mascot Feedback

Affective Filter Reduction Through Emotional AI Feedback

OOPS

0-50%

😢 Encouraging ARI — "Keep trying, you're learning!"

GOOD EFFORT

51-60%

👍 Hopeful ARI — "You're making progress!"

KEEP IT UP

61-70%

👋 Encouraging ARI — "You're getting there!"

WELL DONE

71-90%

🎉 Happy ARI — "Great pronunciation!"

AMAZING

91-100%

🥳 Excited ARI — "Perfect score!"

🧠 Affective Filter Reduction

Krashen's Affective Filter Theory: anxiety blocks language acquisition
Friendly non-textual mascot feedback replaces harsh red X marks
Blue ARI mascot (brand color) creates emotional safety
Encouraging over critical response framing at every score tier
Non-judgmental retry prompts maintained across all tiers
Safe learning environment without human teacher availability

📊 Phoneme-to-Emotion Integration

Technical scores (0-100) → emotional tier conversion
Consistent feedback language across all assessment types
Each phoneme-level result maps to mascot response
mascotScoreDefn.level property (0-4) maps to tiers
Students learn to associate scores with feelings, not just performance
Emotional literacy development support

🎮 Level Progression

Level 0-4 maps to Course Adventure progression system
Students "level up" through mascot evolution
Visual progression reinforces learning journey over time
"AMAZING" (91-100) creates aspirational targets
"OOPS" (0-50) prevents shame spirals — encourages retry
Long-term motivation architecture with achievement milestones

🎯 Thinking Lab — Critical Thinking

Distinct vertical for debate & critical thinking
Entry point for ARI Debate capabilities
External bridge: nexus-ari-ai.replit.app
RAG model for HKDSE-aligned academic content
Structured arguments, group discussions, exam simulations
Speech & Debate Course Adventure integration

🖥️ UI Design Philosophy

Cognitive Load: 24px font (desktop), 18px (mobile) for bilingual EN+ZH
75% max-width mimics optimal reading line length (~66 chars)
AI Presence: nodRocket animation during processing — mimics human cues
Interruptibility: Skip button opacity-0 → 100 on hover — student agency
ChatFooter: 64px desktop / 48px mobile for thumb-friendly recording
Login: 540px centered card / full-width mobile — one-handed registration

"An AI that doesn't just teach English, but understands the learner's context,
accommodates their device constraints, and adjusts its personality
to maintain the optimal affective state for language acquisition."

10 / 15

THINK & SPEAK™

Knowledge Retrieval & Historical Analysis

Every Class Becomes a Structured, Reusable Learning Asset

🔍 Semantic Search

segments = splitIgnoreCases(preview, searchValueTrimmed);
segments.map((v, i) =>
  i % 2 === 0
    ? {v}
    : {v}
)

±50 character context window around matches
Preserves vocabulary usage context for exam prep
Visual highlight on matched terms (orange background)
Example: "The student practiced the vocabulary from the previous lesson"

📚 Room-Based Aggregation

Filter Chat

→

Sort DESC

→

Slice ×3

dayjs.createTime.valueOf() chronological sorting
3-message preview limit per room — scannable results
Room title + date + total message count display
"View All" navigation to full conversation thread

📓 Learning Journal — Session Previews

English Oral Practice — Unit 4

March 15, 2024 · Teacher: Ms. Chan

47 msgs

"Today we'll practice environmental vocabulary..." → "Let's discuss climate policy impacts on..."

HKDSE Prep — Paper 3 Discussion

March 12, 2024 · Teacher: Mr. Wong

62 msgs

"What are the advantages and disadvantages of..." → "In your opinion, should the government..."

Vocabulary Workshop — Presentation Skills

March 8, 2024 · Teacher: Ms. Lee

28 msgs

"Key presentation vocabulary: furthermore, moreover..." → "Practice using these words in context..."

Students jump back to specific classroom discussions — finding that one conversation that explained a difficult concept before exams.

11 / 15

THINK & SPEAK™

Read & Speak + Course Adventure

Structured Learning Paths — Gamified Progression from Assessment to Academic Achievement

📖 Read & Speak Module

📰 Daily Reading

AI selects readings based on assessed topics
Dynamic text complexity adjustment to student level
Reading passage with recording interface overlay
AI Coach available during reading tasks

📝 Personalized Scripts

Teachers upload scripts in bulk
Students upload own content for practice
Listening + recording functions
Pronunciation comparison with AI sample

⏱️ Practice Tracking

Practice history with scores tracked for visualization
Best voice recordings for playback and self-evaluation
Progress bar: total practice hours used and remaining
Time tracking across all modules

📊 Read & Speak Dashboard

24

Scripts

12h

Practice Time

+15%

Improvement

🎮 Course Adventure

Five-Task Lesson Structure

🎥

1. Lesson Video

Short video introducing key topic or skill

✅

2. Comprehension Check

Multiple choice questions about lesson content

📋

3. Drag & Drop Glossary

Match terms with definitions to reinforce vocabulary

📥

4. Glossary Download

Downloadable PDF of key words and definitions

🎤

5. Speaking Exercise

Practice speaking related sentences with phoneme feedback

📈 Skills Developed

🎧 Listening 🗣️ Speaking 📖 Reading 📝 Vocabulary ✍️ Writing 🧠 Critical Thinking

Progressive disclosure: icons shift from full opacity (available) to 30% opacity (locked) — guides students through learning routes without explicit lock indicators.

Clear objectives, milestones, and achievements — make learning feel purposeful, motivating, and game-like — not repetitive.

12 / 15

THINK & SPEAK™

Security, Data & Three-Layer Personalization

Privacy-by-Design for K12 Environments — GDPR-Aligned Architecture

🔐 Three-Layer Personalization

Layer 1: Assessment-Driven Onboarding

Invitation code + role-based routing → School Program / Individual / Trial learning track assignment. Captures age, proficiency, interests, motivation, style.

Layer 2: Adaptive AI Interaction

Dual-model system (GPT vs RAG) + input mode flexibility (text/audio). Adapts to proficiency — beginners type, advanced use speech-to-speech. Dashboard personalization.

Layer 3: Affective Feedback Loops

ARI mascot utils + Message components with 5-tier scoring system. Animated responses create emotionally safe environment. Engagement without human teacher availability.

🛡️ Registration & Data Handling

School Data Segregation: invitationType === 'School Program' → schoolName disabled + readOnly
Prevents: wrong institution enrollment, roster exposure
K12 Privacy: sourceOfReferral strictly optional, no validation rules
Minimal PII collection for minors — GDPR-aligned design
6-digit verification codes (maxLength: 6) with email delivery status
"Check email and spam box" guidance for secure password flows

🔐 Security Architecture

RSA encryption: PEM key pairs (genKeyPair + decryptFromServer)
Azure token encryption during transit — prevents interception
Redux (ephemeral) for session state — disappears on browser close
localStorage (persistent) via useLocalStorage hook — crash recovery
API sync for durable long-term storage
Export generation with isLoadingExport state management

🔧 Production & UAT Environments

Production URL

app.thinkandspeak.com

UAT URL

uat-app.thinkandspeak.com

Teacher Account

test_teacher@seechange-edu.com

Student Account

test_student1@seechange-edu.com

Password

Aa123456

13 / 15

THINK & SPEAK™

Benefits for Teachers

Empower Educators — Reduce Administrative Burden While Maximizing Student Outcomes

⚡

Real-Time Engagement Monitoring

See student understanding instantly through live feedback signals — question/understand indicators, comprehension heatmap, attendance tracking. Adjust teaching pace in real time based on quantitative participation metrics.

🎯

Differentiated Instruction

Group students for targeted learning tasks. Three-state attendance model (Attending/Invited/Available) supports cohort-based content delivery. Teachers can target entire schools, specific classes, or individual students.

📢

Broadcast & Material Distribution

Send announcements and assign materials during class. Sticky messages persist on screen while conversation flows. StickyMessage interface with position: 'top' | 'bottom' anchoring for differentiated instructions.

📈

Comprehensive Analytics

Track progress across modules and students. Dashboard shows: total courses, lessons completed, time spent, vocabulary progress, speaking scores. Export session transcripts for institutional reporting. Progress made visible — stay motivated naturally.

💡

AI-Powered Hints & Mentor Support

AI Coach provides tailored explanations considering previous performance patterns. Three-tiered hint system (explicit → hints → subtle) adapts to student needs. Mentor-style support without giving away answers — builds independent thinking.

🏗️ Engineering Quality for Institutional Deployment

Performance

SVG icons prevent FOIT. GPU-accelerated 60fps animations. Code-splitting via next/dynamic.

Maintainability

TypeScript strict mode. CSS Modules. Component-level state isolation. Barrel exports.

Scalability

Multi-tenant. 10,000+ students. Concurrent sessions. RSA token encryption.

Accessibility

WCAG 2.1 AA. Keyboard navigation. Scalable SVGs. Cross-browser tested.

14 / 15

THINK & SPEAK™

🚀

Ready to Transform Your Classroom?

Teach with Confidence. Learn without Limits.

Join thousands of educators using AI-powered tools to help students succeed. A personalized English learning platform — built to understand learners, support growth, and drive real outcomes.

🌐 Production

URL

app.thinkandspeak.com

Teacher

test_teacher@seechange-edu.com

Student

test_student1@seechange-edu.com

Password

Aa123456

🧪 UAT Environment

URL

uat-app.thinkandspeak.com

Teacher

test_teacher@seechange-edu.com

Student

test_student1@seechange-edu.com

Password

Aa123456

"Breaking Barriers. Building Knowledge. Make Every Class Unforgettable."

15 / 15

THINK & SPEAK™

Think & Speak

Complete Learning Ecosystem

Live Translation Engine

Group Discussion — DSE Paper 4

Phoneme-Level Pronunciation Assessment

AI Tutor & Reading Comprehension

Assessment, Dashboard & Progress Tracking

Technical Architecture Deep-Dive

Student Learning Interface — AttendantSidebar

Gamification Psychology — ARI Mascot Feedback

Knowledge Retrieval & Historical Analysis

Read & Speak + Course Adventure

Security, Data & Three-Layer Personalization

Benefits for Teachers

Ready to Transform Your Classroom?

📝 SPEAKER NOTES