Expecting Change

There is one universal truth about DevOps. When you first embrace DevOps—when you can declare the manifestation, when you embrace constant iteration and improvement, when you can encourage change rather than react to the consequences—your job will become bigger and greater than ever before. You can do more than ever before. Your environment will change.

You will change.

In many cases, a new Puppet implementation will enable months of rapid and exhilarating change. In some cases, it will be slow and progressive, and you won’t notice the depth of change until you look back and realize how far you’ve come. But no matter how wide and diverse the experience, no matter what your industry or background, everyone who walks down these roads experiences dramatic change.

Go out and talk to people who are using Puppet. Ask them to compare their job before and after Puppet. Ask them if they want their old job back. Watch the look on their faces.

It’s easy to say, “You’ll get so many benefits,” and “You’ll be able to accomplish more than ever before.” Those statements are absolutely true. Puppet will give you powerful tools. Your success with Puppet will change the way your job is viewed, and the scope of decisions you are involved in. Fast change can be good, but even slow and progressive change will redefine the landscape eventually. The questions asked of you will change.

Change will come to you, to your processes, and to your expectations. Expect change.

Controlling Rate of Change

Puppet provides powerful tools for managing changes on systems. However, only in really small environments do all servers apply at the same time. Most environments run the Puppet agent every 30 minutes. Nodes receive their configuration changes spread out over time. This means that at any point in time, there are two different configurations running.

In large-scale environments with multiple teams, the rate of change can be such that one set of changes is not yet done rolling out while another is starting to hit the first nodes. At any moment at my current job, there are between 3 and 7 module changes getting pushed to live nodes every second of the day. I’ve seen as many as 20 different active configurations in globally diverse Puppet environments, although 2 or 3 was by far the most common.

This can be good, as problems with a change can be spotted and reverted before it breaks all nodes. This can be bad when consistency is crucial.

Orchestration tools like MCollective are necessary to accelerate change and limit the inconsistent period to a few seconds. MCollective provides a framework to apply immediate changes that bring your nodes into sync. If you skipped over Chapter 30, listen to my advice that this is a crucial tool for managing change.

Tracking Change

Absolutely everyone should implement tracking of Puppet convergence results. This can be done in several ways:

• Parsing log messages provided by Puppet
• Tracking status from a dashboard that evaluates Puppet reports
• Processing Puppet node reports with a custom report handler

Parsing log messages with analysis tools like Splunk, Graylog, or Logstash is easy for basic uses (to capture failures) but difficult, time-consuming, and subject to evolving requirements to gather new statistics. It is considerably better to provide a standard data storage mechanism like PuppetDB, and enable users to store and retrieve their own data.

Dashboards provide a drop-in, immediately available overview of node results reported by nodes. Any of the dashboards reviewed here can provide a wall display useful for service desks to see happy green and problematic red immediately. Detailed analysis of the Puppet node reports depends on the features available in the dashboard selected.

PuppetDB stores and aggregates data about changes to nodes. All dashboards provide a web interface to review the data from PuppetDB. Futhermore, multiple reporting tools can utilize the same PuppetDB as a data source. PuppetDB’s independent data source provides a clear advantage for tracking change.

If your organization uses a different tool for tracking change, you may be able to leverage it for tracking change on your Puppet nodes. Create a custom report processor that extracts data from node reports and feeds it to the existing tool’s input mechanism. I have done this dozens of times with only a few hours effort required to make the Puppet data available in the organization’s existing toolset.

Benefits of Puppet Apply

In Part I, you interacted with Puppet only through puppet apply. You saw how to invoke it on demand to immediately evaluate the declared state, and apply the requested state.

There are numerous advantages to using puppet apply:

• Allows use by detached nodes without access to a shared server.
• Does not require TLS certificate signing before the node can apply changes.
• Can enforce policy in containerized microservices with short lifespans.
• Uses only local resources without depending on infrastructure for a shared service.
• Access to local system state and data sources unavailable to a remote server.
• Executes function calls on local system during the catalog build.
• Converges many nodes in parallel without “storming” a shared server.
• Enables simple code pushes without learning Puppet environments and architecture.
• NTP time synchronization is not essential unless you are also using MCollective.

The independent, local source for information works very well for some organizations. This implementation is very common among some of the largest Puppet customers with the highest total node counts.

Some of the advantages for puppet apply create requirements that must be satisfied outside of Puppet. Depending on your needs, these requirements can actually nullify the advantages:

• You’ll have to create a service to synchronize Puppet modules and data down to each node.
• It requires a constant development effort to push only the necessary modules and data to each node. In most organizations, all Puppet modules and data are pushed to every node for simplicity, which can make unrelated security-sensitive data available for anyone with (authorized or not) privileged access on the managed node.
• Making changes to the Puppet code—even a one-line attribute change—would require recycling all containers with static copies of the Puppet data.
• Every node requires access to every data source. This could require complex firewall or VPN infrastructure if remote or public nodes access private data sources.
• Security credentials required for data access will have to be synchronized out to every node that needs to query the data source to build a Puppet catalog.
• Having each node connect independently to data sources will consume significantly more resources on the central service than having the server connect and cache the response. Nodes remote from the resource will have significantly longer catalog build times due to latency of data access.

I’ve probably forgotten an advantage or disadvantage here, but I think this gives you an idea of the concerns.

Benefits of Puppet Server

In a server-centered configuration, the code and data only needs to reside on the Puppet server. The Puppet agent on each node submits its locally evaluated facts to the Puppet server, and receives back a catalog containing only the resources that should be applied locally.

There are numerous advantages to using a Puppet server, including the following:

• Requires no configuration of client nodes beyond the TLS certificate signing.
• Requires no presynchronization of data sources to utilize.
• Provides file synchronization services for applications.
• Receives only the resources and data applicable to that node. No manual work separating files is necessary to avoid placing unrelated data on the wrong nodes.
• Outsources the resource-intensive catalog build process to the servers, allowing the use of low-power, low-memory Puppet agents on network devices and embedded systems.
• Deploy code only to Puppet servers, rather than to every individual node.
• Evaluating change in a short-lived Canary environment can be done without a code deployment framework.
• The server caches the modules and data in short-term memory, whereas a node with other priorities will reload from disk on every run.
• Centralize storage of modified files and change reports.
• Puppet servers scale horizontally very easily.
• Puppet servers can be collocated near every node, or centralized globally depending on the need.
• Puppet servers run functions when building the catalog, which can request data from sources unavailable to every node.
• The Puppet service utilizes a single encrypted TCP connection from each node, making it easier to pass through a firewall and more secure than many other data access protocols (DB queries, LDAP, etc.).
• The server could utilize security credentials for data access that do not need to be synchronized to every node.
• Servers can be collocated near the data sources for fast catalog builds. The latency of one request from an Puppet agent to the server is much lower than having the node use independent queries that each have different connection setup and latency costs.

That’s quite a long list of advantages. It is easy to see why the use of a Puppet server is the easiest to implement, and is by far the most common solution (when you’re counting organizations, rather than nodes).

As with the other mode, the use of TLS authentication and the nature of centralized servers create some requirements that must be satisfied. Some of the disadvantages of using a Puppet server include:

• Requires either a manual key authorization, a pre-creation and synchronization of the node’s keys using a build system, or an automatic signing policy to add nodes to the server.
• TLS validation requires working DNS or synchronization of IP to name mappings.
• NTP time synchronization is required for successful TLS authentication.
• A badly performing Puppet server can slow down application of new resources, if nodes are unable to get a revised catalog for several connection attempts.
• It is not possible for every node to receive its catalog at the same moment in time unless the ratio of clients to servers is quite low.
• The Puppet server cannot access data sources that are only available to the node.
• A Puppet server caches custom functions, requiring an API call for it to reload a changed function.

There is one significant difference in the requirements of each method. The Puppet server can provide code to resolve any dependency other than network access from the node to the server. You can write a Puppet module that spreads out client connections to avoid storming the servers, for example. You can write custom facts that contain information only accessible by the client node.

Benefits Shared

Quite often a person will state that they use either a Puppet server or puppet apply because only that method provides certain functionality. This is typically followed by a dozen people pointing out that they use the other method to do the same thing. Both methods share a lot of advantages, especially in Puppet 4.

Let’s list the shared advantages so that you know what either can do, and what you can use no matter which method you select:

Puppet servers can be autoscaled too

If your environment is scaled on demand, you can also scale your Puppet servers on demand. In fact, as Puppet servers are always bound by CPU limitations, and response time degrades linearly according to the usage, I have found it easier to successfully build the auto-scaling rules than with almost any other service.

ENCs can be used by both a Puppet server and puppet apply

You can enable an exec terminus in either the [master] or the [user] section, depending on which way you invoke Puppet. This can be very useful if the node invokes an application to provide necessary data.

If you call back to a central service to gather node information, it’s very unlikely that it provides any advantage in speed and resilience over using a Puppet server.

You can always use puppet apply to evaluate local manifests as necessary

Many sites have the Puppet server synchronize out a directory of manifests to be used by puppet apply on demand by other deployment mechanisms (including MCollective). Some sites use the Puppet server as a file synchronization source, and puppet apply to build the catalogs on the node.

MCollective works perfectly fine with either solution

There is absolutely nothing about MCollective that is benefited by either solution, other than the synchronization of data issues already discussed.

Puppet agents can continue to apply cached catalogs

If the Puppet server is unavailable, the node can fall back and apply the last catalog it received. This is no different from using apply mode and having a file synchronization problem.

Summarizing the Differences

After having read five long lists, you’re probably wishing it was easier to understand. Unless you were able to spot concerns that are especially important, you might be looking for advice on which way to go.

There is no golden rule, but the most obvious use cases can be summarized as follows:

• A Puppet server makes it trivially easy to synchronize data to nodes. You need only run one command to submit a certificate request. All other required files can be supplied by the server to the node going forward.
• Local catalog build and evaluation can be used with distributed nodes lacking centralized servers, so long as another file distribution mechanism is available.
• It is more efficient and secure to authorize a few Puppet servers to a secured data set than to provide access for every remote node.
• puppet apply can make use of data sources accessible only by the node. This situation is rare, and confined almost exclusively to managed service providers with off-net equipment.
• If client nodes are underpowered or lack storage, you can outsource the compute and storage requirements to a Puppet server.
• For static container images with short lifespans, puppet apply avoids the need for TLS certificate authorization.

I think this clearly outlines the most common situations that make one method stand out over the other.

Pulp

Pulp: Juicy Software Repository Management provides the ability to inventory Puppet modules and serve them from the Pulp server. This includes both uploading modules to the Pulp server, as well as selectively downloading and keeping up-to-date modules served at Puppet Forge.

Instructions for installing and using Pulp can be found at http://www.pulpproject.org/.

Puppet Forge Server

The Puppet Forge Server serves modules from local module files and proxies to other Puppet forges.

Puppet Forge Server includes a web UI that looks very similar to the official Puppet Forge web page and provides a simple module search feature.

Instructions for installing and using the Puppet Forge Server can be found at https://github.com/unibet/puppet-forge-server.

Django Forge

Django Forge provides a Forge service using a Django implementation of third version (v3) of the JSON web services necessary to house (or mirror) a private, stand-alone version of the Forge.

Instructions for installing and using Django Forge can be found at https://github.com/jbronn/django-forge.

Indenting Heredoc

Heredoc format can suppress leading spaces, allowing you to maintain indentation in the code without this showing up to the user:

   else {
     if( $keeping_up_with_jones ) {
       $message_text = @(END)
         This is a bit of a rat race, don't you think $username?
         All this running about, for $effort and $gain,
         the results of which we can't see at all.
         |- END
     }
   }

Furthermore, you can tell Heredoc the syntax inside the block and it can validate the syntax for you. Geppetto will display it according to the appropriate syntax scheme:

$certificates = @(END:json)
   {
     "private_key": "-----RSA PRIVATE KEY----...",
     "public_key": "-----BEGIN PUBLIC KEY-----...",
     "certificate": "-----BEGIN CERTIFICATE-----..."
   }
   | END

Splaying Puppet Agent Cron Jobs

If you run Puppet from cron and use Puppet or any shared data sources, you can’t run them all at the same time or you will overload the data source. In situations with only a few hundred nodes, you can simply use the splay feature and the random distribution will be random enough:

cron { 'puppet-agent':
  ensure      => present,
  environment => 'PATH=/usr/bin:/opt/puppetlabs/bin'
  command     => 'puppet agent --onetime --no-daemonize --splay --splaylimit 30m",
  user        => 'root',
  minute      => 0,
}

In my experience, after a few hundred nodes you’ll find that a quarter of them end up in the same three minutes and you’ll experience the “thundering herd” phenomenon. At that point, you’ll want a mechanism to guarantee a more consistent spread of hosts.

The way to do this is to have the server calculate a different minute number for each node, and add the cron resource to its catalog with that minute. The number selection must be consistent, as random numbers will change on each connection.

The easiest way with zero development on your part is to use the fqdn_rand(minutes,string) function, which will always produce the same value between 0 and the minutes provided. If you don’t provide a string, it will use the current node name as salt. It is possible to supply any value (which doesn’t change on a node) as salt. Here are some of my favorite sources for the string:

• Any unique identifier for each node, such as a serial number.
• The node’s IP address. The unique fe80:: address is always good.
• Node certificate name or fully qualified domain name.
• A unique identifier for each node from an internal datastore.

It really doesn’t matter how you generate the source number for the node, so long as it is evenly distributed and does not change often. However you generate the number, add 30 for the second run. Place both of these numbers in a resource like so:

$minute = fqdn_rand( 30, $facts['certname'] )
cron { 'puppet-agent':
  ensure  => present,
  command => '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize',
  user    => 'root',
  minute  => [ $minute, $minute+30 ]
}

Cleaning Puppet Reports

The reports created by Puppet nodes are never removed by the server. You’ll have to add jobs to clean them up. Shown here is a resource for your server to clean up reports after 90 days:

cron { 'clean-puppetserver-reports':
  command => 'find /var/opt/puppetlabs/puppetserver/reports -ctime +90 -delete',
  user    => 'root',
  minute  => '2',
  hour    => '3',
}

While it’s not likely to fill up quite so fast, the same problem does exist for the node itself. Here’s a resource to apply to every node to clean up the reports once a month:

cron { 'clean-puppet-agent-reports':
  command => 'find /opt/puppetlabs/puppet/cache/reports -ctime +90 -delete',
  user    => 'root',
  day     => '1',
  minute  => '2',
  hour    => '3',
}

Adjust the timing of the runs and the retention length to meet your own needs. In many environments we centrally archive the reports on long-term storage, and delete reports older than two days on the node every night.

Trimming the File Bucket

The file backups made by Puppet on the node can be really useful. If Puppet only manages text files, you can likely leave the backups around for as long as your security policy permits.

If you are using Puppet to manage large files, you may need to trim back the archive periodically. Here’s a resource to remove files older than 3 days from the backup bucket. This example should be added to your puppet::agent class:

cron { 'clean-puppet-client-filebucket':
  command => 'find /opt/puppetlabs/puppet/cache/clientbucket -mtime +30 -delete',
  user    => 'root',
  day     => '1',
  minute  => '2',
  hour    => '3',
}

If you have enabled a filebucket on a Puppet server, you may need to trim back that bucket as well. Here’s a resource you could use in a puppet::server class to purge backups after 365 days:

cron { 'clean-puppetserver-filebucket':
  command => 'find /var/opt/puppetlabs/puppetserver/bucket -mtime +365 -delete',
  user    => 'root',
  day     => '1',
  minute  => '2',
  hour    => '3',
}

Adjust the timing and retention length to meet your own needs.

Hating on Params.pp

The Puppet Style Guide recommends placing all parameter default values inside params.pp. I completely agree with this logic for operating system defaults and values that must be derived through lengthy selectors and case logic.

Doing this for static values violates the idea of favoring readability. For example, a subclass that declares a single service should not use this design:

class apache_httpd::service (
  String $status = $apache_httpd::service_status,
) inherits apache_httpd::params {
  service { 'httpd':
    ensure => $status,
  }
}

That’s a tiny little module that fits on half of a page. To be forced to look in the params.pp class to find the value is annoying.

In larger modules, params.pp ends up crammed with many variables that satisfy the needs of a wide variety of subclasses. This can be a management nightmare all its own, especially as these variables are divorced from the classes that depend on them. Someone refactors one module and breaks another because they weren’t aware that the modules shared a data source.

I have found that patterns like the following can be significantly more readable in large modules with many declared resources. Static default values are there to read, and the source of the default value is likewise clear.

Define the params class as a wrapper class that includes multiple other classes named for the type of data provided:

# static wrapper for value parameters that includes multiple data sources
class mcollective::params() {
  include mcollective::os_defaults
  include mcollective::puppet_defaults
}

Then inherit from params in the class definition, but set the default value to the specific source class so that it’s self-documenting, as shown here:

class mcollective::client(
  # Derived these values from the OS
  $package      = $mcollective::os_defaults::client_package_name,
  $group_owner  = $mcollective::os_defaults::admin_group,

  # Derive these from the Puppet version
  $etcdir       = $mcollective::puppet_defaults::etcdir,


  # Common values used below
  $logger_type  = 'console',
  $log_level    = 'warn',
  $keeplogs     = '5',
  $max_log_size = '2097152',
)
inherits mcollective::params {

Your mileage may vary.

Disabling Environments

If you are truly hardcore about not wanting environment directories, then create the following file and place it at /etc/puppetlabs/code/environments/production/environment.conf:

# Remove environment directory from path
manifest = /etc/puppetlabs/code/manifests

# Just use /etc/puppetlabs/code/modules
modulepath = $basemodulepath

Tracking Providers

Something that I have really found useful is the definition of provides variables. The simplest example is when you support multiple types of web servers. Is it nginx or Apache? In that situation, you may want to build your application configurations to not be aware of which web server is used.

I’ve implemented this using Provides::Webserver hashes that can be dereferenced by other classes:

# poor innocent class with no knowledge of web server setup
file { 'config-file-for-other-service':
 ...
 path    => "${provides::webserver::sites_directory}/other_service.conf",
 require => Class[ $provides::webserver::class ],
 notify  => Service[ $provides::webserver::service ],
}

You can set these variables in Hiera using the hierarchy based around operating system, or cluster name, or anything your hierarchy makes available to you. Or you can create a provides module that has a webserver class that a module could declare to instantiate itself as the provider.

Breaking the Rules

Sometimes better judgment needs to take priority over following the rules. For example, the Puppet Style Guide says that every parameter should receive a default value such that the class can be successfully declared with no input values, and it should implement the most common use case.

That’s a great practice, and certainly applicable to installing most software packages where default configurations work fairly well.

However, some systems cannot be, or should never be, configured with default values. For instance, modules that create security key stores should never use default passwords. If the user fails to set an appropriate value, the key store can be opened by anyone who can read the default password from your code. Modules that configure routers or firewalls should likewise demand customized values before instantiating security-related infrastructure.

In these situations, it may be better to force a failure to ensure that the user reads the module documentation completely before using it. Announcing “I’m using a default value for this”—even with all caps and dozens of exclamation points—will get overlooked far more often than you can imagine.

Sometimes the right thing to do requires breaking the rules.

Working Good, Fast, Cheap

Puppet is a fantastic tool for ensuring that things are exactly as you declared them. MCollective provides a mechanism to make change happen fast, everywhere.

• Write quick, small manifests to achieve exactly one thing well with Puppet.

• Use best practices. Declare final state. Log useful information.

• Invoke puppet apply using MCollective for immediate response.

Cheap. Good. Fast. You can do it.

Choosing Fight or Flight

If the resource being evaluated is noncritical, it may be better to log an error but return successfully, to allow the remainder of the dependent Puppet resources to be applied. Can this resource wait until someone fixes the problem before this bit of functionality is enabled? Sometimes, this answer might be more acceptable than you think.

Likewise, if your site doesn’t have a good logging or alerting infrastructure, or the nodes are being built autonomically, perhaps a big explosion is the better path. Define your module as a dependency for a bunch of other modules. When your module fails to converge, the dependency structure will cause Puppet to skip over the other poor, innocent modules you’re holding hostage. It may be better to ensure that nothing works, than to allow the application to start half of the necessary services, and start causing errors visible to the customer.

Your mileage will vary, tremendously. Make it work for you.

Letting the Strings Pull You

If the puppets are alive, the marionette can learn from the strings.

During the convergence process, Puppet analyzes the state of resources to determine if change should be applied. Therefore, Puppet agents can provide a high amount of analytic data with a minimal effort for collection. PuppetDB provides one method of collecting this data.

Alarms (triggers) and state transitions (events) can flow upward from the managed nodes using queues or frameworks like MCollective to event management systems that respond to the input in an appropriate manner.

Leveraging Puppet for Small Changes

Do you have event handlers that log in to a node and take corrective action—say restarting a Tomcat service when it locks up? What happens when those simple commands fail? How do you know if it succeeded? Are the logs of these events available where you track other system changes?

How about leveraging the Puppet infrastructure to track these changes for you?

1. Write a small Puppet manifest that declares how the state should be. Yes, you can declare that the service should be stopped before the service is restarted.
2. Synchronize this manifest down to the related nodes.
3. Adjust the event handler to invoke puppet apply EventResponse.pp
4. Observe the response handler actions in the log entries added by Puppet.
5. Review the number of occurrences and other statistics in your Puppet dashboard.

Tossing Declarative to the Wind

Some days, sometimes, you just end up tossing declarative to the wind and implementing something imperative in Puppet.

You end up with a product for which you are paying for support, and the only supported method to install it is to run five commands in a row. You could totally wrap this up with your own installation with templates and such, but it could break on the next update and business or legal constraints limit your choices.

Because sometimes (as illustrated by the quote from Rob Nelson presented earlier), you have no choice but to get Stone Age about it:

If you try hard enough, you can make [Puppet] act like an imperative language. That would be akin to ripping the bottom out of your car, dressing in a leopard-print toga and a ratty blue tie, and driving to work with your feet.

When that day comes, you put on the leopard-print toga and you punch out something that violates every bit of common sense.

If you absolutely must implement something in a bunch of sequential exec statements, use the following attributes to make it as stateful as possible:

• Make use of timeout to prevent hung commands.
• Use returns to verify that the operation succeeded. Don’t use semicolons in the commands to avoid missing a failed step.
• Avoid using dependency chains and refreshonly to execute commands after a resource is updated. If anything in that chain fails, a person will have to manually revert each step of the chain before Puppet will process the notification again.
• Use the onlyif, unless, and creates attributes to determine if a command should be executed.
• If you can’t find an appropriate test, modify the previous command to write state somewhere that can be referenced by onlyif, unless, and creates attributes of the following statements.

If you absolutely must implement something in a bunch of sequential exec statements, make heavy use of timeout to prevent hangs, and creates and returns to verify that the operation succeeded. Use require and refreshonly to ensure commands aren’t run if the previous command failed. If you can’t find appropriate information written out by the previous step, write state somewhere that can be referenced by onlyif, unless, and creates attributes for the following statements.

I’ve inherited that Dino doll a few too many times. I’ve also written some extremely long imperative sequences that were stateful, and could handle being killed off at any step and then complete properly in the next run. If you have to run to work with bare feet, ensure you don’t have to answer your mobile phone en route.

Allowing Anyone to sudo puppet

It can be tempting to allow people to run Puppet any time they want using sudo. I mean why not—Puppet just talks to its server, and does what you tell it, right?

Well, look at these commands that a user could run:

sudo puppet apply pinky.pp

Because you’ve always wanted to be 0wned by Brain.

sudo puppet agent --server thebrain.local

Because burlap chafes me so?

sudo puppet cert clean --all

If you can find a chicken, 20 yards of spandex, and smelling salts at this hour…

You really want to do this using a small wrapper script that accepts very limited input.