The Problem
I wanted to monitor discussions around products, bugs and trends across communities.
Examples:
Reddit
Hacker News
GitHub Issues
Forums
Most solutions rely on:
OpenAI embeddings
Pinecone
Weaviate
Expensive vector search
For a side project, that cost didn't make sense.
My Goal
I wanted something that could:
Run on a cheap VPS
Process thousands of posts
Group similar discussions
Generate summaries
Without paying for embeddings.
The Approach
Instead of embeddings, I used:
Post
↓
Tokenize
↓
TF-IDF Vector
↓
Cosine Similarity
↓
Cluster
The implementation ended up surprisingly simple.
The Hardest Part Wasn't Coding
The hardest problem was choosing a similarity threshold.
I tested:
0.15
0.20
0.25
0.30
0.35
Results:
0.35
too strict
almost every thread became its own cluster
0.15
too loose
unrelated discussions merged together
0.25
good enough
This single number took longer to tune than the rest of the project.
Architecture
Crawler
↓
BullMQ
↓
TF-IDF Engine
↓
Cluster Engine
↓
PostgreSQL
↓
Groq Summary Generator
Stack:
Node.js
PostgreSQL
Redis
BullMQ
Groq
Vanilla JS
Why I Chose Groq
I didn't need GPT-4 quality.
I needed:
fast summaries
low cost
easy API
Groq was sufficient.
What Surprised Me
For discussion monitoring, TF-IDF works better than people expect.
Most conversations contain repeated terms:
rate limits
api quota
billing
token cost
Those keywords alone are often enough to group discussions accurately.
Current Results
Example topic:
OpenAI Rate Limits
Generated clusters:
API throttling complaints
Increase request discussions
Azure OpenAI alternatives
Workarounds and solutions
The summaries are generated automatically.
Future Plans
I'm considering adding:
Hacker News
GitHub Issues
RSS feeds
Telegram alerts
Trend detection
Feature request extraction
Repository
GitHub:
https://github.com/melyx-id/discussion-radar
I'd love feedback from people building monitoring or intelligence tools.
