This week I have dived into the logging setup within systemd. Logging now feels like such a 'solved problem' within my systemd experiments. My sufficiently complex multi-machine vagrant setup can have makeshift log aggregation using journalctl, netcat, and socat.

log-server dot sh, running on my host machine:

  socat -u tcp-l:8888,reuseaddr,fork STDOUT

log-agent dot sh, running on each machine:

 DATE="`date '+%Y-%m-%d %H:%M:%S'`"
 journalctl -f --since="$DATE" | nc hostip 8888

This is, of course, not a solution specific to systemd or binary logging formats. However, journalctl seems to double as a simple logging agent itself; allowing streaming, filtering, cursors, and multiple output formats (json, single line, etc).

We can further extend log-agent, to include the hostname or other metadata:

 DATE="`date '+%Y-%m-%d %H:%M:%S'`"
  journalctl -f --since="$DATE" \
     | sed  -u "s,^,[ `hostname` ] ,g" \
     | nc hostip 8888

Finally, I would love to replace nc with socat everywhere, using OPENSSL: and OPENSSL-LISTEN to encrypt the logging traffic as it goes from log-agent to log-server.

fleetd and systemd are still confusing me. TIL: There are definitely more state to a systemd and/or fleetd unit than simply 'start' and 'stopped'.

• systemdActiveState: active/inactive/???
• systemdLoadState: loaded/launched/???
• systemdSubState: dead/running/failed/???

From what I gathered, a loaded/active/running state is what happens when a service is started as a dependency of another service. launched/active/running is what happens when a service is started manually with fleetctl start.

A launched/failed/failed can't simply be started, it must be loaded and started.

I am still trying to figure out how this effects the systemd.service Restart directive which I can not get to restart anything on non-zero exit codes.

I'm not 100% convinced many of the systemd features work alongside fleetd and docker. I was hopeful systemd would be my nanny process for ensuring crashed processes could be restarted, but I think it is time to introduce a new service into my test bed before I get too frustrated at systemd.

EDIT: Looks like Restart directive demands a timeout (RestartSecs), although I have not found the documentation that proves it.

My third Uber ride was interesting. I had asked the driver how long they had been driving today. I do not mean to pry I am always just curious about their day.

To obfuscate hours: My driver had been driving awhile and was planning on driving for a bit longer that day. I wondered when they were getting time to eat. A few seconds later the driver starts pulling out some snacks.

I could not help but feel like every attempt at opening their snack and every turn of the radio dial was followed by the merging of our car into the lane of oncoming traffic. Granted this was empty road but still very concerning to a nervous passenger (me).

No one was hurt.

Cost: maybe 7 dollars. Rated: 4/5 stars.

My hunch is right on coreos/fleet. It is spelled out directly on the fleet github page.... so less of a hunch and more of a direct and specific project design:

"fleet ties together systemd and etcd into a distributed init system. Think of it as an extension of systemd that operates at the cluster level instead of the machine level. This project is very low level and is designed as a foundation for higher order orchestration."

My first Uber ride was after a nights stay in a hospital. My friend and I ordered the ride on his phone. Since it was our first ride, we upgraded to a 'nicer' car. The driver was existing driver-for-hire with what looked like an existing taxi license.

The GPS was rough, our driver ended up pretty far from the hospital pick up area. To be fair, we had no idea which road we were on either so directing them was less than easy.

The ride itself was comfortable, had free water, got home safe. Was a little too worn out to notice anything interesting or exciting.

Regarding fleet/etcd/docker/confd: It seems very non-trivial to keep track of systemd service files, confd templates, and Dockerfiles. Definitely more-trivial than puppet based automation but very, very less trivial than it seems to be sold as.

Is there something I'm missing about how it all works together? Are we expected to build or use a system built on top of this, such as flynn or deis?

I just wrote a puppet manifest that applies a patch to a puppet ruby file on the destination system due to a bug in the version of augeas on the machine. Sidenote: Who do I call to get my devops license revoked?

Thoughtstreams is finally usable now that I know the only hangup was platform.twitter.com (blocked at work). Blocked it in noscript and now everything is speedy.

Looks like thoughtstreams allows you to create empty messages using <-- --> . Weird. They aren't vulnerable to simple XSS and the message body doesn't contain the html comments that were attempted to be added.

Lets go ahead and test this.

1 2 3 4