From before Craftosphere. Between 2020 and 2022 our founder was the lead engineer and architect of this system at a large technology company. The story is anonymized: no product, no company, and details changed where they don't matter to the lesson.

Stories Earlier work, 2020 to 2022

Rebuilt while running

A monolith that archived tens of terabytes, moved piece by piece to microservices while it stayed live

By Published

A system archived and searched tens of terabytes of sensitive personal data. It was a backend monolith with a few insecure services next to it, an outdated web frontend, and a desktop client that had to receive the encryption key. I led a team of ten that moved it to microservices over about a year and a half, piece by piece, while it stayed live and kept getting new features. The old system topped out at about 2 TB archived a month. The new one archived 11 TB, and we never found its limit.

Archived a month
about 2 TB before, 11 TB after, limit never reached
Uptime
99.99% through the migration, measured for the client SLAs
Team and time
ten people, about a year and a half

Before

The system took in data files, sometimes several terabytes each, archived them and made them searchable. The data included personal information, so it was encrypted. The backend was a monolith, with a few separate services next to it that were full of security holes. Users worked in an outdated single-page web frontend and a Swing desktop application. To work, the desktop client had to receive the encryption key, which should never leave a server.

It didn't scale horizontally at all, and the business needed new features that the old structure made hard to add. Some small, well-defined tasks should have run in hundreds of instances at once while the rest of the system didn't need to, and we couldn't know in advance which tasks those would be because it depended on the load. Some data structures were also badly designed.

The developers disliked working in the old codebase. The modernization was my proposal and a team decision, and management backed it because I didn't plan it as a big-bang rewrite. We would replace the system piece by piece, following the strangler fig pattern, and it would stay live the whole time.

What was built

We started with the frontend, because the user experience was the worst part. Together with the new frontend we built an API layer: a contract the frontend relied on, behind which we could hide the services and every change to them. As we carved out the services from the monolith one by one, the frontend didn't notice. Rebuilding the frontend also mapped the user journeys, so we drew the boundaries along what the users actually did.

Once the new frontend was live, the desktop client wasn't needed any more, and the encryption key stayed on the server. The frontend received the data already decrypted. The old system was fully synchronous. The new one was event-driven where possible, with Kafka between the services. Each service scaled horizontally on its own, so a task that needed hundreds of instances could get them without scaling the rest of the system.

The single document database was complemented by Postgres, Redis, and object storage. We migrated the badly designed data structures to optimized ones. Each kind of data had a single owner, and every read and write went through it. The unit was the data, not the database: a collection could belong to one service, and in a shared collection, one service could own some document types and another service the rest.

The desktop client used to upload the data on a single thread. In the new system, separate services chunked the files, orchestrated the upload and did it, server to server, into object storage.

After

The old system had topped out at about 2 TB archived a month. The new one archived 11 TB a month, and we never reached its limit. Uptime was 99.99% for the whole time I worked on it, measured by the monitoring behind the client SLAs. Queries that used to take minutes took seconds, and reports that took hours ran in the background while the application stayed responsive. Cloud costs went down, too.

The developer experience improved, too. The rules for each kind of data were enforced in one place: the service that owned it. Inconsistent states became hard to create. Deployments got quicker and safer, people worked on different features without stepping on each other's toes, and a new feature usually had an obvious service to go into. When it didn't, we decided in planning where it belonged.

The lesson

I cut service boundaries where there is evidence for them. Here we had plenty of real usage to learn from, and parts with very different scaling needs, so the boundaries were visible. On a new system I start with a modular monolith instead, because I don't know yet which boundaries are worth a network call. Microservices bring the problems of distributed systems, so I only go there for a reason: for example, when many teams work on one system, or when some parts have different scalability requirements.

The modular part matters. The original monolith was a tangled mess, with no modules to extract. In a modular monolith, with each module behind an adapter, a module can later move into its own service without the others noticing.

Where would your system split?

A short conversation about your architecture and what slows change down. You leave with a recommendation, which may be to keep the monolith.

Book a call How it works