Database Comparison — FiWorks Labs

Overview Comparison

Aspect	MongoDB	DuckDB	Redis
Type	Document store (NoSQL)	Analytical RDBMS (OLAP)	In-memory key-value store
Data Model	JSON/BSON documents	Columnar relational tables	Key-value, hashes, sets, sorted sets
Query Language	MQL	SQL (PostgreSQL dialect)	Commands + RediSearch
Architecture	Client-server	Embedded (in-process, like SQLite)	Client-server
Storage	Row-oriented on disk (BSON)	Columnar on disk	In-memory (optional persistence)
Best For	OLTP, flexible schemas	OLAP, analytics, batch	Caching, ultra-low-latency CRUD
Concurrency	Multi-user, multi-connection	Single-writer, multi-reader	Single-threaded event loop
Scaling	Horizontal (sharding)	Vertical only	Horizontal (Redis Cluster)
Transactions	Multi-document ACID	Full ACID	MULTI/EXEC (limited)
Joins	Limited (`$lookup`)	Full SQL joins	None
Memory	Configurable cache	Streams from disk	All data must fit in RAM
Footprint	~500MB+	~20MB library	~10MB binary
License	SSPL	MIT	RSALv2 + SSPLv1

When to Use Each

MongoDB

Operational/transactional workloads (high-throughput R/W)
Flexible, evolving schemas
Real-time applications with low-latency CRUD
Horizontal scaling beyond a single machine
Geospatial queries (built-in geo indexes)
Multi-user concurrent access

DuckDB

Analytical queries (aggregations, joins, window functions)
Embedded analytics — no separate server
ETL / data transformation (Parquet, CSV, SQL)
Single-machine analytics on billions of rows
Zero configuration, just link the library
Cost-sensitive — no server infrastructure

Redis

Ultra-low-latency CRUD — sub-millisecond reads/writes
Caching layer for frequently accessed data
Pub/Sub and event-driven architectures
Session / state management
Rate limiting, counters, leaderboards (atomic operations)

Performance Characteristics

Workload	MongoDB	DuckDB	Redis
Point lookups by key	Fast (indexed)	Acceptable	Fastest (sub-ms)
Insert single records	Fast	Acceptable	Fastest (sub-ms)
Bulk inserts	Good	Very fast	Fast (pipelining)
Full table scans	Slow	Very fast (columnar)	Impractical at scale
Complex aggregations	Moderate	Very fast	Very limited
Joins	Slow (`$lookup`)	Very fast	Not supported
Concurrent writers	Excellent	Single writer	Single-threaded but fast
Geo queries	Good (2dsphere)	Limited	Good (GEOSEARCH)
Complex filtered queries	Good	Very fast	Limited (needs RediSearch)

Redis as a MongoDB Replacement for Current State?

Where Redis Wins

Raw speed — 10–100x faster than MongoDB for simple key-based CRUD
Simpler operations — no query planner overhead, no disk I/O for reads
Built-in pub/sub — natural fit for NGSI-LD notification triggering
Geo queries — GEOSEARCH supports radius and bounding box natively
Lower operational overhead — simpler to run than MongoDB

Where Redis Falls Short

RAM cost — millions of entities with rich JSON-LD gets expensive
Complex NGSI-LD queries — the q parameter's nested attribute filters are hard to map
No native joins — cross-entity-type queries need application logic
Persistence — RDB/AOF not as robust as MongoDB's journaling
Transactions — MULTI/EXEC is not full ACID

Key Tradeoffs

MongoDB trades analytical power for operational flexibility and horizontal scale
DuckDB trades multi-user concurrency for extreme single-machine analytical speed
Redis trades query richness and dataset size for raw speed
They are complementary rather than competing

Recommendation for the FiWorks NGSI-LD Broker

Role	Best Candidate	Rationale
Current state (simple access)	Redis	Fastest CRUD by entity ID
Current state (complex queries)	MongoDB	Rich query engine for `q` filters
Current state (hybrid)	Redis + MongoDB/DuckDB	Speed + query power + durability
Temporal / TRoE	DuckDB or TimescaleDB	Analytical queries over time-series data

Verdict: Hybrid Architecture

The ideal FiWorks NGSI-LD Broker database architecture is not a single technology choice but a layered approach:

Redis as the hot cache for current entity state — sub-millisecond GET/PATCH by entity ID, built-in pub/sub for notification triggering
DuckDB (embedded) for complex queries and temporal data — full SQL power without a separate server process
Optional MongoDB for deployments needing horizontal scaling, durability guarantees, or rich q-filter support beyond what Redis offers

DuckDB's embedded nature (in-process, like SQLite) is particularly interesting — it eliminates the network roundtrip to a database server entirely, which aligns perfectly with the C broker's performance-first philosophy.