• 

  icon

Inspiration

What it does

The user wants me to rewrite the Flowiki project story in English, following the same structure (Inspiration, What it does, How we built it, Challenges, Accomplishments, What we learned, What's next) with Markdown and LaTeX support for math formulas. Let me translate the previous story.

Flowiki Project Story

Inspiration

Flowiki was born from a long-standing tension: the "paper-like feel" of tablet handwriting and the "compute power" of AI tools have always been forced to live apart.

Existing note apps fall into two camps. One camp perfects the "feels like real paper" experience (GoodNotes, Noteshelf), but AI is reduced to pasting plain text. The other camp perfects AI (NotebookLM, ChatGPT), but you must switch into a chat window to use it — and the notes you wrote by hand become second-class citizens.

The scene we wanted was simple: a student takes notes by hand on a canvas, writing $\int_a^b f(x)\,dx$, and AI continuation appears on the next line in their own handwriting style. A researcher drags a PDF onto the canvas, lassoes a question, and the answer lands next to the notes with clickable citations. An engineer taps into the chart gallery mid-lecture and pulls up a noisy Gaussian peak fit with Levenberg-Marquardt.

After studying W3C InkML's digital ink semantics, NotebookLM's "controlled sources + verifiable citations" principle, LabPlot's scientific plotting capability matrix, and Google's Material 3 Expressive research framework, we decided to compress all of these capabilities into a single Android tablet note workflow — no window switching, no copy-paste, no cloud dependency for core compute.

What it does

Flowiki is a hybrid-canvas note app for Android tablets (package com.feng.suixinji). It places all of the following onto a single page coordinate system (default $1000 \times 1400$ units):

• Stylus writing: captures $x, y, t, p, \tau, \phi$ and tool type; pressure pen, highlighter, eraser, lasso, shapes
• Hybrid canvas elements: ink, text, tables, images, stickers, formulas, charts (PlotElement), masking tape, attachments, audio
• AI Studio: streaming Markdown, KaTeX-style formulas ($...$ inline, $$...$$ block), flowcharts, whiteboard tool calls
• BYOK model interface: users bring their own Chat / Image / Embedding endpoints; credentials are protected by Android Keystore, no developer keys shipped
• Local WebView search: queries go to the user's chosen search engine and the app fetches result pages directly
• Scientific chart gallery: 10 categories, 69 templates — from basic line plots to noisy Gaussian peak LM fitting, Savitzky-Golay smoothing, arPLS baseline correction
• Chart digitization (Datapicker): pick an image element on the canvas → calibrate two axis reference points → tap-to-sample → generate a data column and build a chart

Everything is offline-first: canvas rendering, numerical algorithms, formula typesetting all run on-device. AI is a pluggable capability, not a prerequisite.

How we built it

Tech stack

• Language: Kotlin (client, tests, build scripts all Kotlin DSL — no other language runtime)
• UI: Jetpack Compose + Compose Material 3 1.5.0-alpha23 (directly using MaterialExpressiveTheme, MotionScheme.expressive(), HorizontalFloatingToolbar, and other official Expressive APIs)
• Handwriting: AndroidX Ink + custom PressurePenEngine
• Recognition: ML Kit Digital Ink Recognition
• Networking: OkHttp + SSE decoder
• Build: Gradle Kotlin DSL + Version Catalog, minSdk 28 / targetSdk 36 / compileSdk 37

Key architectural decisions

1. Four-layer ink data — semantically aligned with InkML but stored as a versioned binary structure rather than XML:

• Immutable raw points
• Smoothed render path
• Per-tile raster cache
• Hidden Unicode text + text span ↔ stroke IDs mapping for search, RAG, and accessibility

2. Unified canvas element model — CanvasElement is a sealed interface. Every element (ink, text, formula, chart, sticker…) shares the same withFrame / withLocked / withRotation / withZIndex / encodeElement / decodeElement contract. When a new element type is added, the compiler forces us to exhaustively cover all when branches across five files (we call this the "exhaustive when checklist").

3. PlotElement renders through one pipeline in three places — canvas element, editor preview, and gallery card all consume the same PlotRenderSpec: composition phase uses TextMeasurer for text layout → DrawScope for pure drawing. PNG/PDF export goes through the same draw function.

4. All numerical algorithms are hand-written in pure Kotlin — this is the project's lifeline. LabPlot is a C++/Qt app; we explicitly forbid porting any C++ code. Every algorithm is a pure function verified by JVM unit tests:

• Polynomial least squares: normal equations $(X^\top X)c = X^\top y$ + Gaussian elimination with partial pivoting. $x$ is centered and scaled via $\tilde{x} = (x - \bar{x})/\sigma_x$ before solving, then coefficients are mapped back — this is the key to numerical stability. $R^2 = 1 - SS_{res}/SS_{tot}$.
• Levenberg-Marquardt (~150 lines): solves $(J^\top J + \lambda \cdot \text{diag}(J^\top J))\delta = J^\top r$, with Jacobian via forward-difference numerical approximation. On success $\lambda /= 3$, on failure $\lambda *= 5$; convergence criterion $|\Delta\chi^2|/\chi^2 < 10^{-9}$.
• FFT: iterative bit-reversal radix-2 Cooley-Tukey + Bluestein chirp-z for arbitrary lengths. Unit tests compare against hand-computed DFT ($n=8$ exact; $n=1000$ random sequence $|FFT - DFT| < 10^{-9}$).
• arPLS baseline: iteratively solves $(W + \lambda D^\top D)z = Wy$, where $D^\top D$ is a pentadiagonal symmetric positive-definite banded matrix solved with bandwidth-2 banded Cholesky (~60 lines). Weight update $w_i = 1/(1 + \exp(2(d_i - (2\sigma_d - m))/\sigma))$.

5. FormulaEngine.kt — a native LaTeX typesetting engine written to replace KaTeX WebView. Fully offline, no CDN dependency.

6. Material 3 Expressive design system — theme name Ink & Paper, primary seed deep ink-indigo #4F4B83. Font stack is Google Sans Flex + Noto Sans SC (Google Sans Flex metadata doesn't cover Han, so Noto Sans SC is an explicit, testable fallback — never a silent system fallback).

Challenges we ran into

1. Non-ASCII path + JBR encoding conflict

The project path c:\Users\feng\Documents\随心记 contains Chinese characters. The Gradle JVM test worker decodes paths as UTF-8 by default and crashes wholesale. We had to explicitly inject:

.\gradlew.bat assembleDebug "-Dorg.gradle.jvmargs=-Dfile.encoding=GBK -Xmx4g"

It took a full afternoon to pin this down.

2. JBR + AGP 9.1.0 + compileSdk 37 test environment conflict

testDebugUnitTest fails under this combination, but assembleDebug is completely fine. It's a toolchain version coupling issue, not a code issue. We chose to run unit tests where the environment allows and rely on APK install verification otherwise, rather than stacking workarounds.

3. Material 3 Expressive 1.5 is still alpha

MaterialExpressiveTheme, MotionScheme.expressive(), and the new LoadingIndicator all live in 1.5.0-alpha23. We explicitly accepted this risk: Version Catalog pins the version, all experimental APIs are opted into only within core/designsystem, and business features never depend on experimental symbols directly. Every upgrade must pass a full regression.

4. The "small pitfalls" of Pixel Tablet debugging

• The device has Google Pinyin installed, and adb shell input text ASCII gets hijacked by the IME — UI verification has to go through template taps, not typing
• In PowerShell, adb exec-out screencap -p > file corrupts the output as UTF-16; screenshots must be redirected through Bash
• The test device serial 51101HFH80E6RC has to be hardcoded in scripts

5. Rendering math formulas inside notes

We originally rendered block-level formulas via WebView + KaTeX, which caused duplicate loading, transparent background gaps, and theme color drift. The final solution: IncrementalMarkdownEngine does block-level formula recognition, cleanMathLiteral() strips delimiters, KatexMath injects theme-aware text color, and JavascriptInterface enables height auto-fit. Test cases $E = mc^2$ and $\int_a^b x^2\,dx$ must both render correctly.

6. Initial values for multi-peak fitting

LM is a local optimizer — bad initial values mean no convergence. Our strategy:

• Gaussian peak: $\mu$ = peak position, $a$ = peak height, $\sigma$ = FWHM $/ 2.355$, $b$ = baseline median
• Exponential/power: linearize to get initial values, then refine with LM
• Failure fallback: return null, UI shows "fit did not converge, please adjust initial values / model" — never silently commit wrong parameters

Accomplishments that we're proud of

1. LabPlot's full capability matrix, replicated on a tablet note app

All seven milestones shipped and verified by install:

Milestone	Scope
M1	Data foundation (columnar container + CSV + codec)
M2	Render engine + editor skeleton + gallery launch
M3	Regression analysis (19 fit models + 8 weight types + statistics $R^2$/adjusted $R^2$/RMSE/MAE/AIC/BIC)
M4	Signal processing (FFT/Hilbert/convolution/correlation)
M5	Data workflow (formula columns + themes + PNG/PDF/SVG export)
M6	Statistics module (t/ANOVA p-values + control charts + heatmaps)
M7	Chart digitization + hand-written xlsx/ODS parser

2. Gallery expanded from 38 to 69 templates

Ten categories (basic, functions, physics, chemistry, statistics, engineering, finance, mathematics, signal, quality control), each template rendered as a live thumbnail by PlotEngine. The new gaussianFitTemplate (noisy Gaussian + LM fit) and smoothingTemplate ($\sin$ + noise + triangular weighted moving average) demonstrate end-to-end analysis directly.

3. Zero C++ dependency, end to end

LabPlot is C++/Qt; xlsx parsing is POI's domain. We:

• Hand-wrote xlsx read-only parsing with ZipInputStream + XmlPullParser (~300 lines, no POI)
• Hand-wrote all numerical algorithms in pure Kotlin
• Replaced KaTeX WebView with FormulaEngine.kt

APK size stays controllable, and startup doesn't depend on any runtime network fetch.

4. The "exhaustive when checklist" engineering discipline

CanvasElement is a sealed interface. When PlotElement was added, the compiler forced us to fill in branches across five files: UnifiedDocument.kt, ElementLayerOrder.kt, UnifiedEditorScreen.kt, UnifiedCanvas.kt, and PlotElementContent. This mechanism ensures that "adding a new element type" can never leave a silent missing branch.

5. Three render sites, one code path

Canvas element, editor preview, gallery card, and PNG/PDF export all consume the same PlotRenderSpec. One change, four surfaces synchronized.

What we learned

Engineering

1. Offline-first is not a technical choice — it's a product positioning. Under mainland China network conditions, "WebView + CDN" and "runtime library fetch" are both unreliable. Rewriting KaTeX, numerical algorithms, and xlsx parsing as pure on-device Kotlin felt like doubling the workload, but it's actually the baseline of product usability.

2. Sealed interface + compiler-enforced exhaustiveness is the best tool for managing canvas element type expansion. Safer than abstract classes + runtime type checks, more readable than reflection.

3. Centered scaling in numerical algorithms is not an optimization — it's a necessity. Solving the Vandermonde system directly on raw $x$ numerically collapses at higher orders. Centering via $\tilde{x} = (x - \bar{x})/\sigma_x$ first makes 9th-degree polynomials converge stably.

4. Material 3 Expressive 1.5 alpha is usable, but only with an isolation layer. We centralize all @ExperimentalMaterial3ExpressiveApi opt-ins in core/designsystem, and business features depend only on stable symbols — if an alpha API breaks, only the adapter changes, not the business pages.

5. Gradle JVM encoding must be declared explicitly. With a non-ASCII path, -Dfile.encoding=GBK is not optional.

Algorithms

1. Levenberg-Marquardt's initial values are everything. Linearize for initial values + LM refinement is an order of magnitude more stable than pure LM. For multi-peak fitting, letting users tap peak positions on the preview as initial values is more reliable than auto-guessing.

2. arPLS pentadiagonal banded Cholesky is tens of times faster than a general solver. $D^\top D$ is sparse; general matrix factorization is both slow and numerically unstable. Banded decomposition is ~60 lines of code but pays off enormously.

3. Bluestein chirp-z is the key to arbitrary-length FFT. Radix-2 only handles $n = 2^k$, but real data has arbitrary length. Bluestein converts a length-$n$ DFT into a length-$2^k$ convolution, reusing the radix-2 kernel.

4. The incomplete beta function $I_x(a, b)$ is the cornerstone of statistical testing. p-values for t-tests and F-tests both depend on it. A continued-fraction expansion (following Numerical Recipes 6.4) is ~50 lines and accurate enough.

Product

1. AI should enter from the cursor, blank space, or lasso — not take over the canvas. Surfacing candidates when the user pauses mid-sentence respects the act of "writing" far more than a persistent chat window.

2. Provenance is non-erasable. AI-generated content can visually match the user's handwriting, but origin = AI + model version + profile version must be stored permanently. This is not a technical detail — it's product ethics.

3. "Verbatim transcription" and "AI organizes into notes" must be two distinct modes. Letting AI silently rewrite spoken words destroys the evidentiary value of the recording.

What's next for Flowiki

1. Personal handwriting L2/L3

L1 (real variant replay for seen characters) is usable today. Next:

• L2 personal glyph completion: generate same-style glyphs for unseen common characters, referencing EasyFont's 775-character optimized set
• L3 online trajectory generation: directly generate vector strokes with stroke order, velocity, and pressure, supporting per-stroke erasure and replay. Builds on DeepWriting, Write Like You, and ICLR 2025's OLHWG, but requires in-house R&D for Chinese coverage and on-device performance

2. Full RAG evidence layer

MVP currently has BYOK + local WebView search. Next:

• Self-owned SourceVersion + Chunk + Citation evidence master data
• Hybrid keyword + dense retrieval + rerank, golden set $\text{Recall}@10 \geq 90\%$
• Fine-grained citation locators (PDF bbox / slide page / table cell range / audio-video timecode)

3. Long-stream ASR

Cloud Speech streaming requires proactive reconnection roughly every 4 minutes 30 seconds per stream. We need: sequence numbers + monotonic timestamps + audio bridging + idempotent final segments + full recording review, so 60–90 minute recordings neither drop nor duplicate segments.

4. E-ink DisplayProfile adaptation

Currently targets normal LCD/OLED tablets. Next, introduce DisplayProfile rendering configuration:

• colorMode: fullColor | reducedColor | grayscale
• refreshClass: highRefresh | standard | eInk
• motionLevel: full | reduced | none
• Vendor e-ink SDKs confined to integration/eink, never leaking into business logic or the document model

5. Flashcard and quiz generation

Following NotebookLM's principles: controllable sources, verifiable evidence, materials convertible into learning objects. Every card contains explanation + evidence[]; when a source is deleted, derived cards become unsearchable.

6. Collaboration and sync

Local-first + operation-based merge. Concurrent conflicts on the same object preserve both copies rather than silently overwriting. AI jobs use idempotency keys so retries never duplicate inserts.

---

Flowiki = Flow + wiki . We want tablet notes to regain the natural feel of "writing," while AI, audio, and document libraries become native capabilities on the canvas — not external tools that require switching windows.

Built With

• ai
• llm
• mlkit
• notebook