Lately, I've been putting some thoughts around the bandwidth, storage and performance in Marconi.
A queuing service is not actually about queues, it's about messages. If you think about it, the real resource that travels from one endpoint to the other are messages. There are all kind of things that happen from the moment a message is sent and the moment it is received but at the very end, the message is all that matters.
Queuing services' reliability exists just because no one don't one to loose messages. Message routing exists because it is important to be able to tell the message where it should go. Filters exist because not every node wants to get all messages, they need to be selective. I could go on with this but I think I made my point.
With all that in mind, the resource that we need to improve and make sure it's as lightweight as possible during the whole transmission period, is the message.
In Marconi, it's possible to configure the maximum size allowed per message. This is a huge benefit for users because they can tune their setup based on their needs. However, something that is not being taken under consideration is that not all storage would work well with all kind of message sizes. For example, SQS has a maximum message size that is not configurable, which means that someone willing to bake Marconi with SQS won't be able to use a maximum size for messages that is bigger than SQS's. Now, bare with me. I know that example is crazy and that most probably no one, ever, will do that. Just take what it is, a crazy example.
Thankfully, Marconi has support for shards. It allows users to deploy Marconi and have several backends backing it up concurrently. With shards, it's possible to tell Marconi what storage to use for each queue. Although this is great, I don't think it's enough.
There's one thing that we need before we can call this feature useful for-real™, though:
Routers help with directing messages to their final destination. In the case of Marconi, they'll help with putting each message into the right storage. The idea is to make routers configurable at every possible - and reasonable - level. Routers should allow users to control the message flow based on the source, some headers, rate, queue flavors, size, etc. Most of the existing technologies allow users to do that already, nothing new so far.
Now, what if we take advantage of the routing support and we move it down to a field level.
Imagine a very write-heavy queue with a fairly high message size - documents, for example - sharing a store with other queues. In a scenario like that, other queues may be penalized by the queue with big messages. We could argue this saying that it could be possible to dedicate a cluster for that specific queue and leave the other for the most lightweight queues. However, for the sake of the argument, lets say that is not possible.
A possible solution for that would be to split the message into 2 separate resources, one containing the body and the other the rest of the message. The later would be routed to the shared storage backend whereas the former would be routed to a dedicated storage for 'big things'. This will keep the main storage lighter and it won't penalize other queues. The body will be kept in a separate storage and it'll be retrieve as soon as it has to be sent to the client.
All that may sound a bit weird. Why would you do that instead of putting the big message in the storage that is good for 'big messages' in first place?.
One reason I could think of is that not all stores will be good for big messages and not all stores will be good for 'fast messaging'. For example, if you've Marconi deployed on top of an AMQP broker, I wouldn't suggest sending huge messages through the broker but that doesn't mean you shouldn't be able to take advantage of the performances of the broker. In a case like this, keeping splitting the message could make sense. It'd be very simple to store the message's body into swift and send the rest of the message to the broker.
The idea is definitely not bullet proof, it's just an idea, but I'd definitely like to explore it a bit more.