Migrating Mapify search to PostgreSQL without a hard cut-over
Yingchen Liu
How I designed full-text mind-map search and moved index ownership from S3 to PostgreSQL with fallback, background migration and user-transparent rollout.
- PostgreSQL
- Search architecture
- Data migration
01
A mind map is a searchable tree, not just a file name
Mapify is an AI app that turns documents, videos, recordings and web pages into visual mind maps.
Each map contains a hierarchy of topics and nested ideas, so the useful search target is often a concept deep inside the tree rather than the title of the saved document. Search must return the right map, preserve enough context to explain the match and stay responsive across a large personal library.
S3 was well suited to storing the complete generated document, but object storage is not a full-text search engine. PostgreSQL could hold a searchable projection of the relevant text and maintain an index, while S3 remained the safe document source during migration. That distinction—source document versus searchable projection—is the foundation of the rollout described here.
02
The problem
Mapify users needed to find ideas inside a growing library of mind maps, not merely locate a file by title. The existing S3-backed representation was useful for document storage but not a strong foundation for full-text query, relevance and operational migration state.
A hard cut-over would have made indexing failures visible as missing results. It also would have coupled search launch to the complete migration of existing content.
Search has to see inside the map
This is why a file name is not enough. One map holds a deep tree of topics. The searchable copy has to include those topics, not only the title of the saved document.
03
Search the ideas without making people wait for a migration
Architecture
The full mind map stays in file storage. A database copy is what search uses, and only after that map has been indexed. Until then, search falls back instead of hiding the map.
The move
Where does a map live during the move?
File storage keeps the complete map. PostgreSQL holds a searchable copy, built in the background.
- 01
Keep the original
The saved mind map stays in file storage. Search does not require that file to be thrown away first.
- 02
Copy the searchable text
A background job reads the topics inside the map, not only the file name, and writes them into PostgreSQL.
- 03
Record whether it finished
Each map has a migration state. Operations can see a failure instead of discovering a missing result.
The search
What does the person see?
Search never assumes every old map has already been copied.
- 01
Search the copy when it is ready
A finished map is found through the database index, including short words and numbers.
- 02
Fall back when it is not
A map that has not been indexed, or whose indexing failed, is still found through the safe original.
- 03
Drop deleted maps
Deleting a map removes it from the results. A failed index is retried a limited number of times.
How I implemented it
The work crossed four repositories: the web experience, search backend, database package and operations dashboard. I added the full-text search vector and index checks, built indexing and migration endpoints, exposed migration errors for operations, and implemented responsive search states in the product.
The edge cases mattered as much as the first query: short terms and numbers needed explicit handling, deleted maps had to disappear from results, image-heavy result lists needed careful loading, and transient indexing failures required bounded retry rather than an infinite background loop.
04
The decision
I designed PostgreSQL as the searchable projection while keeping S3 as a safe source during migration. Search and migration were separate concerns: the indexer could progress in the background, and the read path could use explicit state and fallback instead of assuming every item had migrated successfully.
The decisions that shaped the system
- 01
Add a projection before replacing storage
PostgreSQL became the searchable representation while S3 remained the safe document source during the transition.
- 02
Design fallback first
Explicit migration state lets an unmigrated or failed record use the safe path instead of disappearing from search.
- 03
Operate the migration as a product
Index checks, bounded retries, deletion cleanup and visible errors make partial progress supportable rather than exceptional.
05
What improved
Mapify gained content-level search without forcing users through a migration ceremony. Operations could see indexing state and retry failures, while the product could remain useful during partial migration. PostgreSQL also placed query behaviour, deletion consistency and search-vector maintenance in one inspectable data path.
06
What I learned
A migration is safer when the new store is introduced as a projection with explicit completeness, not declared the source of truth on day one. Designing the fallback first made partial progress a supported state rather than an emergency.
Takeaways
- Make migration completeness explicit and observable.
- Keep fallback behaviour during partial indexing.
- Test search semantics beyond happy-path words.