This is feedback from side, lack of time form my side i could'nt upload it
What worked:
- Tool-discovery on CPU - The embedding strategy (bge-small-en-v1.5) delivered 40.9% real savings with zero GPU. This is the killer feature: developers can adopt compression without infrastructure changes.
- The /api/test health endpoint - GET /api/test returning {gpu_available, message} was essential during the outage. Made it trivial to confirm when the server was back vs. local config issues.
- Graceful degradation - When the 4B server echoed inputs above the ceiling, it returned valid content (the original) rather than erroring. The system stayed usable.
What broke:
- Undocumented input ceiling - The 4B compressor has a ~4k-token (~8k-char) ceiling and silently echoes above it. This is fine behavior but needs to be in the docs, ideally with a response header (X-Paritok-Skipped: input_too_large) so the proxy can log/chunk intelligently.
- Proxy doesn't pre-chunk - It hands whole tool-result segments to the server. For real CI logs (15k+ tokens), this guarantees echo → 0% end-to-end. A max_chunk_size config with smart sentence-boundary splitting would fix it.
What's missing:
- Compression metadata in the response - Return {original_tokens, compressed_tokens, model_version, skipped, skip_reason} in the /compress response body. Right now the only signal that compression happened is comparing lengths. Telemetry/debugging need the actual numbers.
Log in or sign up for Devpost to join the conversation.