Fun with compression in Haskell:

Working on a Python tech-lab. It's going to talk about the Python ecosystem, Openstack, and how to make the most of it.

Licensed under CC-BY-SA 4.0 when it's done!

Infinitely sleeping, a little systems practice:

A little shifting exercise today:

```haskell shiftR :: [a] -> [a] shiftR xs = last xs : init xs shiftR [] = []

shiftL :: [a] -> [a] shiftL (x:xs) = xs ++ [x] shiftL [] = []

shiftBy :: ([a] -> [a]) -> [a] -> Int -> [a] shiftBy f xs n = head $ drop n $ iterate f xs

shiftByL :: [a] -> Int -> [a] shiftByL = shiftBy shiftL

shiftByR :: [a] -> Int -> [a] shiftByR = shiftBy shiftR ```

What's the development process?

With a few terminals open:

• emacs open in one window
• ghci open on another window

I do this:

1. Load module into ghci: :l shift
2. edit in emacs
3. reload in ghci :r after each edit

Once things were working, I moved to clean up the style using hlint.

Good future additions:

1. QuickCheck suite
2. Criterion benchmarks
3. Packaging: Setup.hs and shift.cabal

On watching: Conference Submissions and Presentations: Made More Effective with 10 Quick Tips: Matthew McCullough

How to Make an Effective Proposal

Frame Your Topic

• Teach a skill that you know very well and other are weak at
• Change opinions about something that is viewed as hot (or not)
• Share a story of how something worked out well
• Recount how a process didn't work for your team
• Tell you learned a hard lesson
• Do a technical deep dive; human friendly

Investigate the Audience

• Ask the organizers:
  • What's the demographic?
  • What background do they have?
  • What do they want to learn?
• Ask the attendees
  • Query previous attendees
• Adjust for maximum audience impact
  • What is the technical level of the audience?
  • What languages do the audience members primarily program in?
  • Diversity: what social factors make a difference? Age? Race? Etc.?
  • What educational background?
  • What languages will they know?

Craft a Story

Your talk is a story. Design it and build it with that in mind.

1. Exposition
2. Rising action
3. Complication
4. Climax
5. Falling Action
6. Resolution
7. Denouement

Write a Proposal

Think about the weight of your words. Make it more concise, but impactful.

• Brief, but impactful
• Title: 4 - 8 words (The Case for Haskell)
• Abstract: 2 - 4 sentences is best (tease the tech, the story)
• Exposition: Clarify what you're giving

Get inspiration from other proposals.

Market Your Talk

• Get the message out there; fill the seats!
  • Tweet
  • Lanyrd
  • wherever you can
  • Engage your network!

Design Your Talk

Different from implementation of your talk, e.g., slides and demos.

• Sketch your ideas
  • Tiny notebook is perfect
  • Encourages minimalism; get to the point!
• Linearize at the last possible moment
  • Collect all the beneficial pieces

Build The Talk

This can be deferred into very late into the process.

YOU are The Talk

Slides are merely a support - don't let them take away from the story. Let them be secondary to being

Practice The Talk

Less than 10% practice their talk even once. Thesis: quality/impact would increase greatly if more people took the time to review and rehearse.

Lower the stress; increase the impact.

Seed Satisfaction

What does this mean? Place friends in the front row. People who are rooting for you. They will ask questions, clap, root for you. It becomes infectious, spreading like a posi=virus across the rest of the room.

Just Talk

Time to make it happen! "The audience members are your friends". Why? Because they expect to learn something. They're here because they want to learn what you're teaching. Make it worth their while.

Feedback

"A responsible presenter has one more activity to do" - gather feedback. Why? Not only to improve, but also...?

Provide a feedback channel - this is your task. Give them a preferred means of speaking with you, especially a public channel to communicate with you.

Vent and direct the steam. Own your talk and ensure that people get what they wanted.

Looks nice. I'm pleased with this update to ThoughtStreams.

Watching: Brent Yorgey On Diagrams and Typeclassopedia

A few sections caught my eye.

"I think to be really successful with Haskell, you need to understand at least some of the materials on the Typeclassopedia..."

From here. I agree when Brent speaks on understanding the common abstractions. I've found that the more I learn to read and understand the abstractional conventions >>=, >>, <-, fmap, <*>, <$>, runThing, and more, the more sense reading Haskell APIs makes to me.

On design patterns, and how they're fundamentally a manifestation of an anti-pattern in software design. I found this ironic, but I understood the why of it. Design patterns exist because the languages that we've been using today fail to capture the abstractions that we want to use. Iterators, facades, etc. - it all connected pretty strongly with me.

"Design patterns are just libraries that your language isn't expressive enough to write."

The next section that got me thinking was on Teaching Haskell.

"Was there a consistent hurdle?".

Applicative functors and monads seemed to give trouble. This has been my experience so far with the little bit of programming I've been able to do so far. Sometimes, I'll get stuck trying to get something out of a Context, and what conventions exist for that. For example, using Data.Random and finding out how to plug the results of generating a RVar [ByteString] into a function that expects a [ByteString] was tricky.

I love the thoughts on teaching students how to move between different levels of abstraction. This is something that I missed while going through school, and had to more or less rough it out while working on my CS degree.

"What do you think about Haskell as a general purpose language for teaching...?"

It seems that reasoning about time/space usage in Haskell is tricky. I can understand this. It would probably be difficult to convey some basic concepts on estimating the complexity of an algorithm in Haskell.

On the other hand, I think an unfortunate trade-off is made here. Haskell makes it easy to understand how to cleanly express algorithms and programs. I think this sacrifice in delaying the introduction of algorithmic complexity and space usage is worth it. I've also heard that Chris Okasaki does a great job of introducing how to reason about complexity in a lazy setting in his Purely Functional Data Structures.

Wrap Up

Agree? Disagree? Fork this ThoughtStream or catch me on @cppcabrera. Share your thoughts!

Took me some time to figure out how to do a basic HTTP request in Haskell. Using http-client made it pretty easy, ultimately. Here's the result:

```haskell import Network.HTTP.Client

main = do req <- parseUrl "http://google.com" mgr <- newManager defaultManagerSettings httpLbs req mgr ```

Be warned: that'll output quite a bit of data to the console.

Fun, fun.

Seems there's quite a bit of power hiding behind the simple http-client interface: data streaming, TLS support, keep-alive, etc.

Trust isn't easy.

It doesn't matter whether it's in relationships, secure system design, or what have you - it's the same problem.

No secrets. Be honest - to yourself, especially! Be aware.

There's no shortcuts, and doing the above still isn't 100%, both for gaining trust or for gauging the trustworthiness of another.

Architecturally, the biggest scalability bottleneck in Marconi's design at the moment is the dependency on FIFO semantics for queuing operations.

Each queue maps to a particular shard. This sharding design helped Marconi overcome its first scaling bottleneck, which was being able to handle many messages to multiple queues. Now, since queues can live on different storage nodes, it's possible to scale out a Marconi deployment.

However, there's still FIFO semantics to contend with. The FIFO invariant is enforced by the storage layer by taking advantage of atomic commit semantics when posting messages. This marker is unique to each queue. This is done so that messages are claimable in the order that they arrived. If messages would try to post concurrently, then one of those POST operations would "fail", causing an internal retry that isn't exposed to the connecting client. By fail, I mean, the operation is retried shortly after.

Fortunately, storage driver implementations for Marconi need not honor the FIFO property. If a particular workload does not require FIFO, then it becomes much easier to scale out such a deployment.

I can envision a future feature involving queue flavors where users submitting requests to Marconi can annotate their queue at creation time with attributes they care about. For example:

``` PUT /v1/queues/fifo_queue

{ "fifo": true, }

PUT /v1/queues/fast_queue

{ "persist": false, "fifo": false }

PUT /v1/queues/make_it_last

{ "persist": true, "fifo": false } ```

I've identified the flavors persist and fifo so far for choosing storage shards automatically. An example of a persistent storage flavor that offers FIFO is Marconi's reference mongodb implementation, that can be deployed with replica sets for added reliability. An example of a storage driver that's the polar opposite of that is my marconi-redis work-in-progress. Since the data is maintained in memory, it can be lost at any time.

It's all about configuration and letting the user choose what they need. I hope to see more of this in the future of Marconi.

On Mindfuless and Navigating the Present

Flavio said on his ThoughtStream:

...being fully aware of what's happening won't help with making what will happen next any better.

I don't agree here fully. I'll share a story-ish to elaborate.

There have been many times in the past where I've allowed myself to throw tantrums (even as an adult!). I didn't stop to try to understand what or why I've been feeling. In the present, however, I've learned techniques to navigate what I'm feeling and to manuever through these difficult situations that are overflowing with emotional energy.

In order to do that, I needed to be as aware as I could be not only of what I was feeling, but also of what: 1) my wife was feeling, 2) about what I wanted, 3) and what I wanted to happen. I can't control the final outcome all on my own, but I am able to influence my own actions.

I'm not sure if this is called mindfulness, that ability to navigate one's self, but it's helped me. By pairing it up with writing like this, as well as some private journaling, I've made my own progress in an area that would freak me out previously!

On the Relationship Between Now, Then, and Previously

More from Flavio:

Is there actually such a thing as 'present' ?

"People like us, who believe in physics, know that the distinction between past, present, and future is only a stubbornly persistent illusion" - Einstein

I love this concept - and I agree! It's something I've discussed with people close to me and that I'd like to think on further. So here are some thoughts:

Most of our society at this time is ruled by deadlines and schedules. We have to get to school at 7:30am or we'll be late. We have to delivery this software by Dec. 27th or it'll be late. If you get there early, you'll have more time.

Stressful. That's the first word that comes to mind. We're placing unnecessary constraints on ourselves in order to increase the likelihood of some event happening.

I've lived most of my life in this framework of time: deadlines, schedules, and appointments. Even my personal endeavors were tainted by these concepts, e.g., "I should only play Diablo 2: Median XL for the next two hours if I hope to finish this homework that is due tomorrow*".

I think the two things that have suffered the most in my life as a result are my ability to reach for a personal balance and my ability to identify what's really important to me.

Balance depends on choosing the things that you want to carry and commiting to them in such a way that you don't drop them. It's identifying that there's a subset of all the things that matter to you enough to keep them in mind. It's difficult, both keeping them balanced amongst themselves and protecting your commitment to them from external pressures.

Identifying what's really important comes with the prerequisite that you know yourself well enough to choose what you want. There's dangers here, too. The psyche is very vulnerable to subtle influences (Check out Thinking Slow and Fast for more on this!). The day to day influence of marketing, especially that which crops up every 10 minutes in standard cable television is poison. Without paying mind to these influences, you'll slowly be influenced to desire something that you didn't choose for yourself.

So that's the crux of it for me. Reducing the importance of time in life is powerful. I say reducing rather than eliminating, because I still live in a schedule-bound society.

10 Ways to Incorporate Haskell into a Modern, Functional, CS Curriculum

I had some time to spare while I was waiting for dinner to be ready. Dinner was being prepared by the local China Gate, and it would probably take on the order of 10 to 15 minutes. So I sat down in the lobby with a notepad in hand.

The topic on my thoughts: how could Haskell be incorporated into a modern CS curriculum? I decided to run as radically as I could with this, writing a rough draft of what such a curriculum might look like.

I won't make any arguments for or against functional programming here. I refer readers instead to papers like this one and this one, talks like this one or this one, and books like this one. This is an exercise in ideation.

Without further ado, let's begin:

0. Haskell for Introductory Computer Science

Imagine this is the first time you're learning programming. You've never been exposed to mutation, to functions, to compiling, algorithms, or any of the details of architecture. Where do we start?

Four weeks of Haskell. Enough to get through the first few chapters of LYAH and be able to start thinking about recursion. End each week with a simple exercise, for example, upper-casing a list of strings, upper-casing only the first letter of every string that is longer than 2 characters - little things to build confidence. The Udacity Introduction to Computer Science course has many appropriate ideas for beginning level exercises.

With that introductory period out of the way, now's the time to show why computer science is relevant! Take the time to show case the areas: operating systems, networking, algorithms, programming languages, cryptography, architecture, hardware, and more. Make it relevant:

• Operating systems: Why are there so many? What does it do? How does my application (email, browser, Steam) run from beginning to end?
• Algorithms: How do you figure out if two characters have collided in a video game? How do you sort a list of address contacts alphabetically by last name?
• Networking: How do you send an instant message or an email to a friend on the other side of the world?
• Programming Languages: As with operating systems.

There are many applicable introductory exercises here that can set the pace for future courses.

1. Haskell for Basic Algorithms

This one, and the latter algorithms course on this "Top 10 List", deserve special attention.

Algorithms are fundamentally pure constructs. You give them a well-defined input, and receive a well-defined output

Take plenty of time to provide weekly exercises. Teaching sorting algorithms, trees, string matching algorithms, and more will be a delight here, I predict.

It's also a good time to introduce basic run-time analysis, e.g., Big O notation.

This is also a beautiful time to introduce QuickCheck in conjunction with invariants.

2. Haskell for Data Structures

Very similar to the basic algorithms course, except now we teach students about some basic ways to organize data. Lists, vectors, trees, hash maps, and graphs - these should be enough to keep most students (and practitioners) well-equipped for years!

QuickCheck and frequent programming exercises will do well here.

If an advanced version of this course is desired, I highly recommend starting from here to brainstorm a variant: Purely Functional Data Structures

3. Haskell for Networking

This can be very low-level (OSI network stack, TCP window buffering, etc.), it can be very high-level (HTTP, distributed systems), or some mix of the two.

I think the most important concepts students can come of this course with would be:

• Validating data at the boundaries and the dangers/usefulness of IO
• How to communicate with other systems
• How to write their own protocols, and why in general, they shouldn't reinvent the wheel

4. Haskell for Operating Systems

Teach them to ask - what is an operating system? How do I manage my resources?

It's worth surveying the concepts of: memory management, file systems, data persistence, concurrency, parallelism, process management, task scheduling, and possibly a bit more.

Great projects in this course include: 1) write your own shell, 2) write a simple, local task manager.

5. Haskell for Comparative Programming Languagues

Let's talk types, functions, composition, and problem solving using different approaches. Ideally, such a course would come after learning how to design solutions to mid-sized programming challenges.

After that, have students write an interpreter for a Lisp.

6. Haskell for Compiler Construction

More on: write your own language. This course should cover parsing, lexical analysis, type analysis, and the conversion from source to assembly.

7. Haskell for Advanced Algorithms

This one's going to be fun. Unleash the power of equational reasoning to put together a course that runs through: graph theory, network flow analysis, greedy algorithms, memoization, and more.

This would also be a great time to discuss how the price one pays for purity in regards to asymptotic performance, and how to overcome that, if necessary.

Also, an extended treatment of algorithmic analysis in the presence of laziness would be valuable here.

8. Haskell for Introductory Design of Programs

Really, this should be higher up in this list, and a very early course.

The goal of this course is to come out of it knowing how to:

• Get a big picture of what they're trying to build
• Break it down into the smaller pieces
• Iterate 'til they get the big picture running

It's a great time to teach some basic ideas for testing, how to experiment with the REPL, and how to take advantage of the type system for simple things.

On a more social level, it's a wonderful time to also guide students towards collaborative design, e.g., how to work together and make it fun and efficient.

9. Haskell for High-Performance Computing

This could be a very fun course. It affords the opportunity to allow students to run wild with simulations and experiments of their choosing, while learning about what it means to do high-performance computing in a functional language.

Given that, it should teach some basic tools that will be applicable to most or all projects. How does one benchmark well? What are the evils of optimization? What is over-optimization? When is optimization needed? What tools exist right now to harness parallelism in Haskell (Repa, Accelerate, etc.)? When is data distribution needed? Why is parallelism important? How is parallelism different than concurrency? How can the type system be wielded to help keep units (km/s, etc.) consistent across calculations?

I'd advocate for letting students choose their own projects built in parallel with the course taking place. A simple default is to optimize the much-lauded matrix multiply to larger and larger scales (distributed even, if they want to go so far!). Writing a collision detection engine for a simple game would be pretty interesting, as well.

Notably (in my opinion) absent topics:

• Hardware: CPUs, cache hierarchies, main memory, storage, interconnects
• Advanced data structures
• Cryptography
• Web development
• Type systems
• Cross-disciplinary development, e.g., Haskell for CS + [Physics, Chemistry, Biology, etc.]

These topics are absent from my list for no reason other than I didn't think of them 'til the list was done and articulated. There's so much one can learn and apply at the level of abstraction that computer science (and mathematics) affords that we could specialize further and further. For my own sake, I'm setting a limit. :)

Final Notes

I've just brain-stormed a curriculum in Haskell. There's a lot of details missing, but it's a working draft.

There's also other things to consider, beyond the technical aspects. Consider the social aspects. How we teach students to work together? How do we keep learning engaging and fun? How do we help students connect to the greater community of developers that exist outside of academia? How do we keep the lessons relevant to the lives that they lead? How do we encourage them to pursue their passions?

**Cross posted from my blog: Read, Review, Refactor

Marconi sharding is officially merged in! It's now possible to set up multiple storage nodes and partition queues amongst them. This makes it possible to scale Marconi quite a bit.

Thanks goes to everyone on the Marconi team and much of the team at the Atlanta Rackspace office for helping make this happen!

Marconi is growing - and could use help.

Before too much longer, we're going to be branching a notifications project off of the Marconi code base. That's to say, it'll grow along with queues, and might even be able to be launched with a unified API, but I see that it'll become it's own project some day.

Then there's the need to support more storage back ends. Currently, we support mongodb and sqlite. There's work started on sqlalchemy (awesome!) and you've probably already heard about my own efforts at redis support.

How about those transports? We only support WSGI/HTTP at the moment, implemented on top of the lean and lovely Falcon framework. We also have an extremely experimental websockets implementation available contributed by flaper87. I'm hoping to see a zeromq transport in the future. Then there's the upcoming nanomsg transport, as well. It's an exciting area, and I hope you'll join in and share your thoughts.

Features: there's many more planned. Better operational stats, queue quotas, queue flavors, Heat integration, Horizon integration, Tempest integration - just check out our blueprints page!

What's next on my plate? I hope to wrap up Marconi Redis, at least in beta form.

I've taken a little break from writing about Haskell, since I've been so deeply involved in Python for my day job.

Still, my reading hasn't slowed down. I've come to adore LYAH. Now that I have a much better understanding of monads and how Haskell solves problems, it's been much easier to digest the book. All of the first 12 chapters make a lot of sense to me, and it was fun being excited about the power of Applicatives. My favorite example so far has been the instance of Applicative for []. It leads to an expansion on all possible outcomes, which is awesome:

haskell (++) <$> ["ha","heh","hmm"] <*> ["?","!","."] ["ha?","ha!","ha.","heh?","heh!","heh.","hmm?","hmm!","hmm."]

All it took was three lines to define the Applicative instance for [], too!

haskell instance Applicative [] where pure x = [x] fs <*> xs = [f x | f <- fs, x <- xs]

I love that! The whole thinking behind typeclasses is very appealing to me.

Oh, which reminds me - the History of Haskell paper, though a bit long (55 pages, 2-columns), has been exactly what I've been looking for. It explains how all the features in Haskell, including various extensions, came to be, and what problem they solved. I highly recommend this paper if you're early in your Haskell education.

I've yet to make the time to write something in Haskell, but my learning and excitement hasn't come to a stop.

More on mindfulness, before continuing with the disconnect:

10 Ways to be More Mindful

1. Stop and breathe. Take in the moment and breathe.
2. Look around - who do you see? What do you see?
3. Stop thinking about how well you are doing. Do what you are doing.
4. Stop thinking about what others are thinking about.
5. Take a look at your hands. Take them in.
6. Don't think about yesterday or tomorrow - think about now.
7. Don't think about deadlines.
8. Instead of avoiding what bothers you, accept it.
9. Accept that there are things that you don't know - and that's okay.
10. Sit. Do nothing.

There is a connection between mindfulness and the disconnect that comes from working with the abstract. It's something I'm going to explore here. For me, there's a tension between the two that I'm very familiar with. More on that tension later.

Part of the goal of working with mindfulness is to be more aware. More aware of my internal processes, how I feel, what I want, and who is around me. I believe this is important for keeping myself honest. If I don't know what I want, or how to analyze my emotions and what they mean, then it's easy for me to lose myself, and it becomes harder to come back.

That's been my experience.

With the abstract and programming, I can dive into exciting and interesting projects for hours and hours. It's happened before, where my wife was trying to talk to me, and it was obvious I wasn't there. My mind was off trying to solve a distributed parallel problem, and she was wondering why I didn't have an opinion on what she was talking about. I was listening to her, and I was responding, but I wasn't communicating effectively. I wasn't there enough to actually process what she was saying and connect it to my own experiences and beliefs. This has caused me some strife.

That's the disconnect.

That's where mindfulness comes in. How do I stop for a moment and bring myself to the now?

I did the list above as an opening to this post as an exercise. I wanted to see how much I could come up with if I stopped to think about mindfulness. It wasn't my effort alone, to be honest. I went as far as 4 before I started reading an article about mindfulness to get the juices flowing. Mindfulness isn't something I'm very experienced with.

I feel anxious about being in the now because it goes against what I've been taught and what I've been raised with. The very act of mindfulness means that for a time, I am in the now. This implies that I'm not working on improving myself (an action towards the future), I'm not trying to solve any problems, and I'm not avoiding my weaknesses. I live for that constant improvement. Yet...

If I spend all my time improving myself, it comes at a cost.

That's why I'm interested in mindfulness.

Inspired by flaper87, I'll also be sharing my thoughts on this subject.

The abstract has a powerful appeal. It's too easy for me to get carried away and lose track of many others aspects of my life. I think this is something that needs to be fought against, because software isn't the only important thing in my life.

More thoughts on this later.

Marconi Redis still needs more work. Queues, shards, and catalogue are working. Messages are mostly working. Claims are not working at all.

More progress on Marconi Redis:

``` alejandro@rainbow-generator:~/Development/marconi-redis:[master %>]$ MARCONI_TEST_REDIS=1 tox -e py27 -- tests.unit.queues.storage.test_impl_redis:RedisShardsTests GLOB sdist-make: /home/alejandro/Development/marconi-redis/setup.py py27 inst-nodeps: /home/alejandro/Development/marconi-redis/.tox/dist/marconi-redis-backend-0.8.0.a37.gb18597a.zip py27 runtests: commands[0] | nosetests tests.unit.queues.storage.test_impl_redis:RedisShardsTests

RedisShardsTests test_create_replaces_on_duplicate_insert OK 0.10 test_create_succeeds OK 0.02 test_delete_nonexistent_is_silent OK 0.02 test_delete_works OK 0.02 test_detailed_get_returns_expected_content OK 0.01 test_drop_all_leads_to_empty_listing OK 0.01 test_exists OK 0.01 test_get_raises_if_not_found OK 0.01 test_get_returns_expected_content OK 0.01 test_listing_simple OK 0.02 test_update_raises_assertion_error_on_bad_fields OK 0.01 test_update_works OK 0.01 ```

Shard storage tests now pass!

It took some finagling, and implementing a list find command in Python. Since the Marconi API expects paginated results in alphabetical order (rather than lexicographical), Redis LIST commands had to be used, rather than SORTED SETS.

Correction

Marconi API expects lexicographical sorting.

Marconi Redis now supports catalogue storage!:

``` alejandro@rainbow-generator:~/Development/marconi-redis:[master %=]$ MARCONI_TEST_REDIS=1 tox -e py27 -- tests.unit.queues.storage.test_impl_redis:RedisCatalogueTests GLOB sdist-make: /home/alejandro/Development/marconi-redis/setup.py py27 inst-nodeps: /home/alejandro/Development/marconi-redis/.tox/dist/marconi-redis-backend-0.9.0.zip py27 runtests: commands[0] | nosetests tests.unit.queues.storage.test_impl_redis:RedisCatalogueTests

RedisCatalogueTests test_catalogue_entry_life_cycle OK 0.11 test_exists OK 0.01 test_get OK 0.01 test_get_raises_if_does_not_exist OK 0.01 test_list OK 0.02 test_update OK 0.01 test_update_raises_when_entry_does_not_exist OK 0.01

Slowest 1 tests took 0.11 secs: 0.11 RedisCatalogueTests.test_catalogue_entry_life_cycle

Ran 7 tests in 0.191s

OK ```

Marconi Redis is one step closer to being a proper backend for Marconi:

``` alejandro@rainbow-generator:~/Development/marconi-redis:[master %>]$ tox -e pep8 GLOB sdist-make: /home/alejandro/Development/marconi-redis/setup.py pep8 inst-nodeps: /home/alejandro/Development/marconi-redis/.tox/dist/marconi-redis-backend-0.8.0.a37.gb18597a.zip pep8 runtests: commands[0] | flake8 pep8: commands succeeded congratulations :)

```

``` RedisCatalogueTests test_catalogue_entry_life_cycle ERROR 0.02 test_exists ERROR 0.01 test_get ERROR 0.01 test_get_raises_if_does_not_exist ERROR 0.01 test_list ERROR 0.01 test_update ERROR 0.01 test_update_raises_when_entry_does_not_exist ERROR 0.01 RedisClaimTests test_claim_lifecycle ERROR 0.02 test_do_not_extend_lifetime ERROR 0.03 test_expired_claim ERROR 0.01 test_extend_lifetime ERROR 0.03 test_extend_lifetime_with_grace_1 ERROR 0.03 test_extend_lifetime_with_grace_2 ERROR 0.03 test_illformed_id ERROR 0.01 RedisDriverTest test_control_db_instance OK 0.00 test_data_db_instance OK 0.00 RedisMessageTests test_bad_claim_id OK 0.01 test_bad_id OK 0.01 test_bad_marker OK 0.01 test_claim_effects FAIL 0.03 test_expired_messages FAIL 0.01 test_get_multi FAIL 0.03 test_message_lifecycle OK 0.01 test_multi_ids OK 0.01 RedisQueueTests test_list_None OK 0.02 test_list_project OK 0.02 test_queue_lifecycle ERROR 1.23 test_stats_for_empty_queue OK 0.01 RedisShardsTests test_create_replaces_on_duplicate_insert ERROR 0.00 test_create_succeeds ERROR 0.00 test_delete_nonexistent_is_silent ERROR 0.00 test_delete_works ERROR 0.00 test_detailed_get_returns_expected_content ERROR 0.00 test_drop_all_leads_to_empty_listing ERROR 0.00 test_exists ERROR 0.00 test_get_raises_if_not_found ERROR 0.00 test_get_returns_expected_content ERROR 0.00 test_listing_simple ERROR 0.00 test_update_raises_assertion_error_on_bad_fields ERROR 0.00 test_update_works ERROR 0.00 Ran 40 tests in 1.679s

FAILED (errors=27, failures=3) ```

In short:

• Shards and Catalogue are not yet implemented (19/30 failures in 19 tests)
• Claims are all wonky (7/30 failures in 7 tests)
• Queues have some quirks (1/30 failures in 4 tests)
• Messages have some quirks (3/30 failures in 7 tests)

Support for both FIFO and non-FIFO Redis is baked in. All it takes is flipping one configuration option and it just works:

conf [queues:storage:driver:redis] fifo = True

More updates coming tomorrow.

Reading The Compromiseless Reconciliation of I/O and Purity

The argument that I/O actions are themselves pure is interesting, and makes sense. When thinking about IO as a series of instructions, it is true that the series of instructions used by the program each time will be the same.

I'm also pretty fond of the idea that Haskell is great at sequencing and composing these series of instructions.

Instructions as a data structure - very cool idea. Very nicely written.

Reading Haskell: Teaching Haskell in Academia and Industry (panel)

Disclaimer: I've yet to write any significant body of Haskell.

I was struck by the notion that most people that work on Haskell spend a lot of time iterating to get a program that doesn't type check to one that does. It reminds me a lot of Python cycles spent getting a program that doesn't pass unit tests to one that does.

This to me sounds like feedback even earlier in the process of development. Sometimes, I don't stop to write the unit tests before the implementation because I've found a good implementation flow. And then I go through a crash-debug-fix cycle for a little bit. After it works, I realize it would've been faster to just write the test in the first place. Trade-offs, ya know?

I wonder how much I'd be able to lean on the compiler to support me writing my ideas down first?

I tinkered with Yesod today. I'd love to write out an alternative implementation of the Marconi project (what I'm currently working on) that is based on Haskell rather than Python. Here's what I pulled together today: routes definition in Yesod!

```haskell {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE QuasiQuotes #-} {-# LANGUAGE TemplateHaskell #-} {-# LANGUAGE TypeFamilies #-} import Data.Text (Text) import Yesod

data App = App

mkYesod "App" [parseRoutes|

-- Fundamental routes: health, homedoc /v1 HomeR GET /v1/health/#Text HealthR GET HEAD

-- Dealing with queues /v1/queues/#Text QueuesR GET /v1/queues/#Text/#Text QueueItemR HEAD GET PUT DELETE /v1/queues/#Text/#Text/stats StatsR GET HEAD /v1/queues/#Text/#Text/metadata MetadataR GET PUT

-- Handling messages /v1/queues/#Text/#Text/messages MessagesR GET POST DELETE /v1/queues/#Text/#Text/messages/#Text MessageItemR GET HEAD DELETE PUT

-- Handling claims /v1/queues/#Text/#Text/claims ClaimsR POST
/v1/queues/#Text/#Text/claims/#Text ClaimItemR GET PATCH DELETE

-- Admin API /v1/shards ShardsR GET
/v1/shards/#Text ShardItemR GET HEAD PUT DELETE PATCH |] ```

It's pretty cool how the routes are typed and how the routes definition is rather self-documenting.

I've also put together a simple implementation of the health resource.

```haskell getHealthR :: Handler () getHealthR Project = return

headHealthR :: Handler () headHealthR Project = getHealthR Project ```

That's all for now. Next up, I'll try and implement another one of the simple resources, perhaps attempting to handle query parameters.

Watching We're Doing it All Wrong by Paul Phillips.

His comments on for-loops still being taught as the fundamental unit of iteration or operations on collections resonated with me pretty strongly. map, filter, fold, and friends get us much closer to expressing what we mean.

Let's get more functional.

Marconi sharding is getting very close:

1. Sharding core
2. Sharded listing of queues

Next up: caching, perf testing, functional testing

The journey towards sharding capability in Marconi is an epic one. There's been lots of help from everyone towards deploying it, testing it, and making sure that the reference implementation is solid.

It's going to be awesome when this feature is ready. Only a bit more to go, and it's likely to be ready before the Openstack Summit!

To overlap my two thought streams:

I wonder what the Marconi API implemented in Yesod would look like...?

Sounds like a hackday project to me. :)

Marconi sharding progresses a little bit more. Development is proceeding at a frantic pace!

Latest today: catalogue storage driver was merged to main line.

Listening to: Haskell Cast, Episode 3: Simon Peyton Jones on GHC

Commentary follows below.

Avoiding Success at All Costs

It becomes very difficult to change a language when it becomes embedded in many products. Early in the lifetime of Haskell, changes to the standard library were made more liberally. As Haskell becomes more successful, it will lose a little less nimble.

"It's not language changes that's most difficult to manage, it's library changes. It's one reason we recently established the core libraries committee."

GHC 7.8/7.10

• Closed Type Families
• Roles - "...if you don't use it, it won't bite you"
• Much more

Closed Type Families

GHC has supported type level functions. f Int = Char, gives us associated types. Introduced as a result of a paper comparing type-level programmability features across languages. Haskell was missing associated types.

With a closed-type family, you can give an order for equations. As a result, you need fewer equations.

Backpack: Improving the Module System

Rationale: solve Cabal hell by introduing a level of abstraction.

199x: Started with an ML-like module system as a notion, but turned out to be too much trouble in early language evolution, so went with simplest thing that could possibly work.

Fast forward: 2005. Needed something to indicate a minimum unit of distribution - a Cabal package. Cabal packages as a metamodule. The idea behind Backpack is how to make Haskell packages depend on APIs rather than concrete implementations. It's not about the Haskell language, perse, but rather a way to tackle the Cabal hell. It'd be a big change, because it would change how the distribution of Haskell packages.

Status: it's not yet implemented.

"Cabal hell is a real place, a very real place".

The Future of GHC

"How can more people get involved?"

GHC is everyone's compiler; it's time to work on this together. 14 new commiters have joined in since 2013.

New pieces that are growing: backend code generation, data parallel programming, template Haskell, parallel I/O manager, dynamic linking. There's a lot of room for more ideas.

On BDFLs - there has to be a way to make decisions where universal consensus isn't possible. On the compiler, SPJ will serve as a BDFL of sorts. The core libraries committee will serve as BDFL where library decisions are concerned.

Parallel Programming

Array fusion as a way to abstract away low-level details of vector operations, as well vector instructions.

If you want to program a parallel machine, start with a functional language. To make things actually run fast, it takes a lot of iteration.

There's also large scale concurrency in Haskell, like Cloud Haskell.

There's also small scale concurrency in Haskell, like forkIO and MVars. This is the process-level communication. There's an approach that uses LVars (lattice-based flows) that might introduce determinism in these types of calculations.

Purely functional parallelism: Repa, Data Parallel Haskell, Accelerate, etc.

GHC out of the box already supports parallelism.

Part of the parallelism problem is that someone can take (with a lot of effort) a program in a low-level language and access all the parallelism in a given machine. The challenge in a functional, high-level language is to find a way to do this cleanly and within a few constant factors of the speed of the low-level implementation.

Purity in Haskell

You can't screw around with side effects, because the ordering of effects is not obvious in a lazy language.

Static Guarantees

Being able to prove that pattern match failings don't exist would be great. There's some research being done at this point. We can be reasonably certain that a Haskell program won't segfault, but we're still at a point where a program might fail at runtime because there might be a failure to pattern match.

Current Research

Limitations of NewType and Resolving it With Roles

Getting a handle on newtype and coercions. It's possible to convert from an Age to and Int at runtime for free. It's not possible to convert from a [Age] to [Int] for free at runtime - this would require a map.

This is where Roles come in. It requires a new form of equality.

GHC 7.8 exposes some of these ideas in an experimental form.

Type-Level Naturals

Over time, hope to build more type level reasoning into GHC. Type-level naturals are being added, allowing to express ideas such a vector of length 3 and do basic operations on that. In future versions of GHC, there'll be support for more such operations.

Computing in School

Going beyond word processors and database management in schooling, and trying to reform ICT computer science curriculum as a "foundational discipline".

Functional programming is still a foreign concept to many teachers. A separate exercise from the Computing in School project would be to establish functional programming as a medium for carrying the concepts of computer science. However, less work is being done on this end because the goal is to get computer science into curriculums at all.

Some Final Commentary

Haskell is growing. I'm pretty excited about the developments in the realm of concurrency, myself. Given that I develop APIs as part of my day-to-day, having easy to use, parallelizable concurrency primitives in Haskell makes it look very tempting. Furthermore, having strong guarantees available at compile-time make me feel more comfortable that things will work when refactoring time comes around...

...and there will always be refactoring!

Today on Marconi: Strudels

Sharding news:

Patches are starting to get merged on the admin API side of things. We now have a clear distinction between control plane and data plane storage drivers. This distinction made abstractions easier to cope with further along.

There's also a more encompassing notion of an admin API instance. An admin instance contains all the routes/resources/powers that a public API instance has, plus access to control plane features. In the case of sharding, an admin instance allows an operator to register shards and investigate the state of the catalogue that maps queues to shards.

Next up: finish getting the admin API for sharding merged in, get the catalogue portion reviewed, and then put it all together!

This thought stream serves as my Haskell commentary center and my Haskell story.

Regarding ThoughtStream as a medium - it's lightweight enough so that I'm not tempted to try to add more to it than content (setting up a blog to get it just right can be rather time consuming), but has enough features that I don't feel limited. I know that if I want to drop in some Markdown to organize the content a wee bit better, I can do that.

At a later time, I'll speak to why I'm so interested in Haskell. I'm an enthusiast, an advocate, and a novice!

flaper87 and kgriffs are so in sync. It's how we reach consensus so quickly on new ideas - it's because they actually communicate telepathically!

In other news, the discussions around how to implement sharding are starting to come together. We've been putting together pieces, and as they've come to light, it's been getting easier to see what works and what doesn't.

Next up: storage controllers to manage shard registration and mapping of queues to shards!

Watching Haskell Talks: Alain Odea by FP Complete.

"Haskell make it possible to do these small things easily and these large things really really well."

Watching Haskell Talks: Alain Odea by FP Complete.

"[Yesod] The idea that with a type system you can get a secure web application..."

In which Alain tried to do XSS vulnerability in the Yesod-written application, and it wasn't easy. Interesting.

Watching Haskell Talks: Alain Odea by FP Complete.

"I didn't actually know there were real problems with Java... I felt that Java - coming into learning Haskell - I felt that Java was a really strong language... but as soon as I started learning Haskell, I felt like there were barriers everywhere.... lots of extra code that you have to write. It's hard to do certain kinds of abstractions in terms of functional decomposition and refactoring... It's just easier to write code, and it's easier to write good code."

"You have to write so fewer tests."

"The challenge I had with Erlang was having confidence that it was still working."

"The big challenge for me in going from more liberal languages is that ... Haskell requires you to write true functional programs. You can't just write to the disk, read from the network. You have to actually organize that... Haskell requires you to think differently."

Here's to testing how a stream with the same title interacts with an existing stream.

Is this how one "shares" a stream?