Sources of Truth

January 12th, 2013  |  Published in puppet, this mortal coil

I probably should have had the hell frustrated out of me by this, but I think I learned something on Wednesday. This post is going to be kind of a work post and kind of a process post. Things could get a little tangled up.

I’ve been working on the documentation for this thing at work called Hiera. What’s there right now is mostly not mine (I inherited the work from coworker Nick), but I did write the bits about using Hiera on the command line, and I wrote the JSON and YAML examples.

I’m not going to get too much into what Hiera is, except to say it has the potential to be a really awesome tool if used properly, and that it was a good pick to hand me for my first thing to work on because you can’t really get Hiera unless you get Puppet, and then — once you’re pretty sure you get them both — there’s room for even more clouds to part.

There are good ways to get Hiera and there are bad ways to get Hiera. I’m pretty sure, based on my own experience after thinking about Hiera for a number of weeks and coming at it from the perspective of someone who was learning all about Hiera at the same time he was learning all about Puppet, that newer Puppet users following a relatively normal path of learning about Puppet are at fairly high risk to get Hiera one of the bad ways. I know I spent a number of weeks getting Hiera one of the bad ways. If not “bad,” anyhow, “skewed.”

An Aside on Hiera and Truth

If you’re actually here because you’re interested in Hiera and are suddenly wondering what’s “bad” and what’s “good” and who the hell am I to say, and are you Doing It Wrong, I’ll offer this:

There are a few kinds of truth in the world. Some truth is local (or organizational) and some truth is universal:

Universal Truth Organizational Truth
What Debian calls its MySQL packages The password for your website’s database
What the name of the postfix service is on CentOS The name of the host your Postfix service is running on
The default NTP servers for Ubuntu The ntp servers your East Coast datacenter should be using

Just, you know, go with this. For our purposes, your special in-house MySQL package doesn’t count and doesn’t really change my point:

Hiera is sometimes sold as a way to remove a lot of conditional logic from your Puppet code, and it’s true that’s a good use. If, however, you’re removing conditional code that describes universal truths from your modules and classes, then moving it over into Hiera, you’re creating a pretty bad situation from a management point of view (you now have logic living in two places — your Hiera data sources and your module code) and you’re also living a pretty bad pattern from a community point of view, because your modules will be difficult to use on places like the awesome Puppet Forge: A chunk of their logic will live outside the confines of a standard Puppet module, making them largely unusable without requiring consumers to create or modify a Hiera hierarchy. That’s a drag.

Instead, you want to put organizational truth into Hiera and call it into your classes and modules via parameters. That means you can put that organizational truth in one place, potentially reuse it in a number of classes and modules, and you can share your modules without the pain of sanitizing them of organizational data every time you share them: That data is coming from Hiera data sources, which you aren’t sharing at all.

Now that I’m at the point in the process where I’m about ready to move on to writing a complete example, that’s what I’m thinking.

How We Got There

It took a while to get to where I could type that up. Time spent getting Puppet up and running in such a way that I could make sure I understood what constitutes normal Puppet behavior, time spent making my Puppet setup work reliably, time spent learning how to provide data to Hiera in YAML and JSON, time spent learning how the Hiera command line tool works to make sure I was testing my assumptions correctly, etc. etc. etc. Most recently, it took me on a detour to learn Vagrant so I could build an easily maintained and reproducible Puppet environment. It also took me time to learn about Puppet modules so I could write a few to work with Hiera.

After months, I was able to push up that page about how to use Hiera on the command line and drop a quick note to the Puppet Users group to let them know we’d made a little more progress. I’m pretty happy with that, because the command line tool is really useful for learning about Hiera outside even a complete Puppet environment and there have been a number of requests for even a bare hint of how to get going with Hiera.

The act of pushing that one page and letting people know about it caused another coworker to write and ask if I needed more help with the documentation. I’d started the day I got that mail thinking I was going to be working on one thing, but his mail got me to thinking about Hiera interacted with his particular concerns and how I’d had the beginnings of a discussion with another coworker about those issues, so I spent the next 2.5 hours trying to crystallize all my thinking and write up some notes not only on what I perceived the state of affairs to be, but how we might make it better.

After doing all that writing, which had started as a 1:1 response and ended up cc’ing somebody else, I paused for a moment to eat lunch, then opened my draft back up, re-read it, saw some things I felt less confident about, and did what I always do before sending a mail of more than a graf or two: I went back to the documentation and my notes and asked myself if that was really, really what I wanted to say. I discovered it really wasn’t, because I had things backwards and didn’t completely understand part of how Hiera interacts differently between a few different versions of Puppet.

So I deleted 90 percent of the email.

Time on a Discarded Mail Well Spent

We can’t always count on someone asking us a simple or friendly question at just the right time to trigger a response that ultimately helps us understand something we were kind of stuck or misguided about, right? So as I sat there staring at the email I’d just pared down to practically nothing, after spending hours thinking about it and staring at it and consulting manuals and looking through at least three git repos to write bits of it, I was tempted to think “why the fuck did I even write this fucking thing?” It felt like a waste of time, because I’d just spent a ton of time documenting questions and concerns that weren’t so much stupid as they were, perhaps, misguided.

Then I realized that in the process of figuring out how much I’d not gotten things right to start with, I’d learned a few new answers, understood a concept I had listed in my todos as a thing I needed to figure out, and I’d been spared writing an example that would have been teaching people A Bad Way to Do Things, provided it even got past the stage where it wasted someone else’s time reviewing it.

It didn’t take me long to think back to when I was blogging a lot more often about a lot of things, and how it would sometimes take me days from starting an entry to actually publishing it, and how sometimes entries were never, ever published in any form because a premise turned out to be too flawed for public consumption, or I decided I was just wrong.

I started out behaving very differently. Writing came naturally enough to me that I won praise and a few small awards without ever having to revise or reconsider what I was writing. I used to think that merely meant that the products I was best suited to create were written ones. That makes sense, even though I’ll confess I’ve never really thrown myself into the craft very deeply.

But I think it may also mean that the best thoughts I have are the ones produced because they were written down, thought through and reconsidered once or twice. I don’t trust many of my thoughts until I’ve taken the time to write them down and think about them. The thoughts I trust the very most are the ones I’ve prepared myself to stand by in public in written form.

So there’s the claim “I’m a writer” you can make from the standpoint of a vocation, profession or hobby; and there’s the claim “I’m a writer” you can make from the standpoint that writing is key to your good internal working order.

I can check both boxes at the moment, but the former can and has come and gone. The latter’s just a matter of truth. It took a few hours worth of email that never saw the light of day to remember it.

Leave a Response

© Michael Hall, licensed under a Creative Commons Attribution-ShareAlike 3.0 United States license.