Data platform
Public Data Search System
The source can change tomorrow. This is built for that.
It collects thousands of public records from a website nobody here controls, has an AI read and label every one of them, keeps its own copy of the pictures, and turns the whole pile into a search box that answers in a second.
This one was built for a client. The name on it is theirs to give out, not ours.
Where it runsOn a rented server, as four programs that start themselves again on their own if the machine restarts.
Who is this for?
A team whose daily work depends on watching what gets published somewhere public, and who were doing it by opening tabs one at a time.
What the numbers mean
Written down and frozen, so two parts of the system cannot drift apart without somebody noticing.
One of them exists only to notice when the other three stop.
A year from now whoever inherits this can read why it was built this way instead of guessing.
Including one that follows a single record all the way from collection to the search results.
How it actually works
Everything here comes from a website owned by somebody else, who can redesign it on a Tuesday without telling anyone. That one fact shaped everything else.
So we kept the part that does the collecting completely separate from everything else. It is its own little program, and the only thing that passes between the two is one agreed format that both sides have to stick to. When the source changes, you swap that one small program and nothing else moves.
Every picture that arrives is copied onto storage this system controls, rather than pointing back at the original. A link to someone else's server is a link that will go dead, and a page of dead images looks like a broken product.
A language model reads each record and labels it, so a person searching gets a shortlist instead of ten thousand rows.
The system runs as four separate programs on the server. One of them does nothing but check, every few minutes, that the other three are still alive. When a piece stops, a machine notices it, not a customer.
If your business depends on a website you do not own, the question is not whether it breaks. It is whether you hear about it before your customers do.
We build things like this. Tell us what you are stuck on.
What it is built with
- TypeScript
- Fastify
- Next.js
- PostgreSQL
- Redis
- Python