Bilal Irfan
Work

Sketchnote

Show your coding agent the problem instead of typing it out.

2026 · Website · Source on GitHub

Overview

Press a hotkey, talk through a bug, and draw or click on the screen. One Ctrl+V then pastes timestamped notes and annotated screenshots into your coding agent. It's a GNOME Shell extension with a Rust service behind it, installed with one command. Speech is transcribed on your own machine by default, so nothing leaves it unless you paste it.

How a capture becomes notes

Capture

You circle a footer, click a button, and say what's wrong with each.

  1. Record

    The screen, the mic, and every stroke, pin and click with its time and position.

  2. Transcribe

    whisper.cpp turns the speech into punctuated text with a timestamp for every word.

  3. Match

    Each sentence goes to the mark it overlaps most in time. A click nobody talked about stays in the notes but gets no screenshot.

    • stroke → #1
    • click, no speech → unnumbered
    • click + speech → #2
  4. Screenshots

    A keyframe for each mark, cropped around the activity with the annotations drawn in, laid out as one numbered contact sheet.

Pasted notes

#1 (0:04) “This footer has too much text, let's cut it down.” (0:08) [clicked] #2 (0:11) [clicked] “This button should open the settings instead.”

The contact sheet is pasted right after the text, so the agent gets both from one key press.

An illustrative example in the real output format.

Results

Install
One command, no Rust toolchain needed
Transcription
~10 s for ~21 s of speech, on CPU
Shipping
21 PRs, 2 releases, 31 tests

Highlights

Built with

Rustwhisper.cppGNOME ShellD-Busffmpeg