Desktop AI tutor
Clicky for Windows
Hold a shortcut and ask about anything on screen. Clicky talks you through it and draws directly over the app to show exactly where to look or click.
Capture, transcription and provider setup now overlap, while the overlay stays responsive through slow devices and network calls.
- Demo: Hold the shortcut, ask a question and get an explanation over the app already on screen.
- Architecture: Independent capture, audio and provider work overlaps before the final answer is painted.
- Knowledge map: The Obsidian graph makes app-specific context visible instead of burying it in an opaque database.
- Provider settings: Provider choices are explicit and API keys stay in Windows Credential Manager.
- PyCon Top 3: Top 3 at the OpenAI-backed PyCon Singapore 2026 Hackathon, with S$2K+ in prizes.
What it does
Clicky starts when a user holds a shortcut and asks a question about the current screen. It captures the screen, transcribes the question, talks the user through the next step and draws directly over the current app to show where to look or click.
The result
Clicky finished Top 3 at the OpenAI-backed PyCon Singapore 2026 Hackathon, with S$2K+ in prizes. Clicky now has 100+ Windows installs. It also has 431 passing tests across the parts desktop software usually breaks: global shortcuts, audio devices, installers, provider credentials and overlay state.
The latency work
Clicky originally waited for speech-to-text, then captured the screen, loaded memory, called the model and finally started speaking. I changed it so transcription, screen capture, app memory and knowledge lookup all start when the user releases the hotkey.
As soon as the model finishes one sentence, Clicky sends it to speech instead of waiting for the full answer. A second worker fetches the next sentence while the first plays. On the measured multi-sentence path, that cut the wait before Clicky starts speaking from 3.7 seconds to 1.7.
Why the memory is plain text
Each Windows app gets a plain Markdown file that the user can open or edit. SQLite indexes those files. I skipped a vector database because inspectable memory mattered more than semantic search at this scale.