TG
All posts
2 min read

Retiring Elasticsearch: Cutting Logging Costs by 80% with SQLite

At 5-10 GB of logs a day, an ELK cluster costs more in JVM heap and shard balancing than the search is worth. SQLite FTS5 and Grafana replaced it.

Elasticsearch is the right answer at scale. At 5 to 10 GB of logs per day it is not — you are paying for JVM heap, shard balancing, and cloud storage to run a distributed search engine over a volume that fits comfortably on one disk.

Faced with a logging bill that had outgrown what it bought, I retired Elasticsearch and replaced it with SQLite behind FastAPI.

SQLite is not a toy

It is the most widely deployed database engine in the world, and the two features that matter here are already in it. The FTS5 virtual table provides full-text search over structured log data, and Write-Ahead Logging mode allows concurrent reads and writes without the whole database locking — which is the objection people usually raise before they have tried it.

LOGGING WITHOUT A CLUSTERAPPLICATION SERVICESINGESTSTORE5-10 GB per dayQUERY AND RECOVERYMicroservicesuser · order · paymentElasticsearchJVM heap · shard balancingRETIREDFastAPIasync Uvicorn workersSQLite (.db file)WAL mode · FTS5 virtual tableGrafana dashboardsearch · trends · latencyObject storageS3 · Supabase, 24h RPO1. HTTP POST /logs2. batch insert, async3. FTS5 query by trace IDfile streamnumbered happy pathbackup / retired component
Batch inserts through async workers into one file — the throughput comes from WAL mode, not from adding nodes.

How a log line becomes a trace

  1. The user, order and payment services POST JSON log payloads to /logs, each carrying application logs, events and a trace ID.
  2. FastAPI, running asynchronous Uvicorn workers, validates the JSON schema and the trace ID, then performs a high-speed batch insert. Batching is what makes a single-file database keep up with several services at once.
  3. A Grafana dashboard queries the FTS5 virtual table to search and filter logs by trace ID, and to chart error rates, latency and event trends.

The database file is streamed to object storage on a schedule, which is the entire disaster recovery story.

What the pivot actually bought

No network latency on query. The database is a file on the same NVMe SSD as the API, so a trace lookup costs a disk read rather than a round trip to a cluster.

Trace correlation that survived the migration. We enforced standardised JSON structures and indexed trace IDs across all microservices, so following a request through the stack works as well as it did in Kibana — the dashboard changed, the capability did not.

Recovery by file copy. High availability came from simplicity rather than clustering: backing up the logs is streaming one .db file to S3 or Supabase, which meets a 24-hour RPO without a replica set.

The pivot eliminated JVM tuning, removed the need for dedicated master and data nodes, and cut logging infrastructure compute spend by exactly 80%.

GitHub
LinkedIn