We are currently working hard on a partitions feature that will ease the scale of Marconi server's horizontally. Not exactly rocket science but definitely and exiting feature to have.
I really like Marconi's architecture. It has the right level of abstraction and customization. Users can deploy Marconi based on their needs and current infrastructure.
I wonder what other people think about it.
The migration to testr has revealed interesting things about nosetests and Marconi tests themselves.
Quee's API has been split in 2 separte drivers. This means it is necessary to run an 'admin' node to expose the admin API. The admin API exposes things like stats, health, shards management, etc.
We're now thinking about having schema versions defined on the server side that can be consumed by both the client and transports. The idea is to also be able to expose that through Marconi's home page, have extensions and whatnot. Here's the blueprint for that:
https://blueprints.launchpad.net/marconi/+spec/cross-transport-api-spec
This is the second year Marconi applies as a project for the OPW program. The last year it didn't gather much attention, probably because it was really really young. Now that it has grown quite a bit, It caught the attention of 2 women that are willing to work on it.
AMAZING!
I'm impress with how many things to do, discuss and plan we still have in Marconi. The number of tasks keeps growing, new ideas come out of the blue and the whole team sticks together working on it!
Great work guys!
We've been triaging some ideas as a daily routine in the last couple of days. This ideas come either from ourselves or external users that have shown interest in Marconi.
Some of those ideas are really valid, others have been rejected mostly because they'd have broken Marconi's conceptual integrity.
Here's the list of ideas. Link
Wow, it's Wednesday and I can't still believe how great the last OpenStack summit was. We've amazing news and a bunch of new ideas. As far as Marconi is concerned, this is going to be a very busy development cycle.
We - alcabrera and myself - had a great pair reviewing session yesterday. We went through the sharding patches and I now feel more confident about letting them land.
There's still some more work to do there, but they're in a better shape now.
I started working on a websocket transport for Marconi. It's pretty cool to see it perfectly fitting into the architecture and being able to use it without much effort. The code is here.
As for now, it just proxies Marconi's public wsgi transport. The goal is to use this transport as another reference for the work going on the API layer.
Lately, I've been putting some thoughts around the bandwidth, storage and performance in Marconi.
A queuing service is not actually about queues, it's about messages. If you think about it, the real resource that travels from one endpoint to the other are messages. There are all kind of things that happen from the moment a message is sent and the moment it is received but at the very end, the message is all that matters.
Queuing services' reliability exists just because no one don't one to loose messages. Message routing exists because it is important to be able to tell the message where it should go. Filters exist because not every node wants to get all messages, they need to be selective. I could go on with this but I think I made my point.
With all that in mind, the resource that we need to improve and make sure it's as lightweight as possible during the whole transmission period, is the message.
In Marconi, it's possible to configure the maximum size allowed per message. This is a huge benefit for users because they can tune their setup based on their needs. However, something that is not being taken under consideration is that not all storage would work well with all kind of message sizes. For example, SQS has a maximum message size that is not configurable, which means that someone willing to bake Marconi with SQS won't be able to use a maximum size for messages that is bigger than SQS's. Now, bare with me. I know that example is crazy and that most probably no one, ever, will do that. Just take what it is, a crazy example.
Thankfully, Marconi has support for shards. It allows users to deploy Marconi and have several backends backing it up concurrently. With shards, it's possible to tell Marconi what storage to use for each queue. Although this is great, I don't think it's enough.
There's one thing that we need before we can call this feature useful for-real™, though:
Routers help with directing messages to their final destination. In the case of Marconi, they'll help with putting each message into the right storage. The idea is to make routers configurable at every possible - and reasonable - level. Routers should allow users to control the message flow based on the source, some headers, rate, queue flavors, size, etc. Most of the existing technologies allow users to do that already, nothing new so far.
Now, what if we take advantage of the routing support and we move it down to a field level.
Imagine a very write-heavy queue with a fairly high message size - documents, for example - sharing a store with other queues. In a scenario like that, other queues may be penalized by the queue with big messages. We could argue this saying that it could be possible to dedicate a cluster for that specific queue and leave the other for the most lightweight queues. However, for the sake of the argument, lets say that is not possible.
A possible solution for that would be to split the message into 2 separate resources, one containing the body and the other the rest of the message. The later would be routed to the shared storage backend whereas the former would be routed to a dedicated storage for 'big things'. This will keep the main storage lighter and it won't penalize other queues. The body will be kept in a separate storage and it'll be retrieve as soon as it has to be sent to the client.
All that may sound a bit weird. Why would you do that instead of putting the big message in the storage that is good for 'big messages' in first place?.
One reason I could think of is that not all stores will be good for big messages and not all stores will be good for 'fast messaging'. For example, if you've Marconi deployed on top of an AMQP broker, I wouldn't suggest sending huge messages through the broker but that doesn't mean you shouldn't be able to take advantage of the performances of the broker. In a case like this, keeping splitting the message could make sense. It'd be very simple to store the message's body into swift and send the rest of the message to the broker.
The idea is definitely not bullet proof, it's just an idea, but I'd definitely like to explore it a bit more.
BTW, just in case you didn't know this. We just released the first alpha version of marconiclient. wOOOOOOOOOOOt!
$ pip install 'python-marconiclient>=0.0.1a1'
Keep in mind it's an alpha version. Not full-featured and most likely a bit buggy.
The missing features are: