Architecturally, the biggest scalability bottleneck in Marconi's design at the moment is the dependency on FIFO semantics for queuing operations.
Each queue maps to a particular shard. This sharding design helped Marconi overcome its first scaling bottleneck, which was being able to handle many messages to multiple queues. Now, since queues can live on different storage nodes, it's possible to scale out a Marconi deployment.
However, there's still FIFO semantics to contend with. The FIFO invariant is enforced by the storage layer by taking advantage of atomic commit semantics when posting messages. This marker is unique to each queue. This is done so that messages are claimable in the order that they arrived. If messages would try to post concurrently, then one of those POST operations would "fail", causing an internal retry that isn't exposed to the connecting client. By fail, I mean, the operation is retried shortly after.
Fortunately, storage driver implementations for Marconi need not honor the FIFO property. If a particular workload does not require FIFO, then it becomes much easier to scale out such a deployment.
I can envision a future feature involving queue flavors where users submitting requests to Marconi can annotate their queue at creation time with attributes they care about. For example:
``` PUT /v1/queues/fifo_queue
{ "fifo": true, }
PUT /v1/queues/fast_queue
{ "persist": false, "fifo": false }
PUT /v1/queues/make_it_last
{ "persist": true, "fifo": false } ```
I've identified the flavors persist and fifo so far for choosing storage shards automatically. An example of a persistent storage flavor that offers FIFO is Marconi's reference mongodb implementation, that can be deployed with replica sets for added reliability. An example of a storage driver that's the polar opposite of that is my marconi-redis work-in-progress. Since the data is maintained in memory, it can be lost at any time.
It's all about configuration and letting the user choose what they need. I hope to see more of this in the future of Marconi.