Selecting a Hiera Backend

There are three data backends built into Hiera:

yaml_data

Parses YAML format files

json_data

Parses JSON format files

hocon_data

Parses HOCON format files

Creating Hiera Data

The built-in Hiera backends support five data types:

• String
• Number
• Boolean (true/false)
• Array
• Hash

Let’s review how to utilize these data types in each backend.

# string
agent_running: 'running'

# boolean
agent_atboot: true

# array
puppet_components:
  - facter
  - puppet

# a hash of values
puppet:
  ensure: 'present'
  version: '4.10.9'

# A variable lookup
hostname: %{facts.hostname}

puppet:
  ensure: 'present'
  version: '4.10.9'
  agent:
    running: 'running'
    atboot: true
  components:
    - 'facter'
    - 'puppet'

{
  "puppet": {
    "ensure": "present",
    "version": "4.10.9",
    "agent": {
      "running": "running",
      "atboot": true
    },
    "components": [
      "facter",
      "puppet"
    ]
  }
}

{
  "puppet": {
    # parameter values supplied to the puppet class
    ensure: present, # simple strings unquoted as in YAML
    "version": ${PUPPET_VERSION}, # environment variable

Version

The version setting must exist, and must be set to 5.

Defaults

The defaults hash can define a default backend, data path, and options for any level in the hierarchy that does not supply them. This can greatly reduce repetition within the file.

defaults:
  datadir: data          # relative directory path
  data_hash: yaml_data   # expect a hash of results from the YAML parser

Hierarchy

The hierarchy key defines an array containing the data sources in priority order to query for values. Each entry of the array defines a data source, format, and query options.

One of the most powerful features of Hiera is the interpolation of a node’s data, such as the hostname or operating system of the node, to select the source file location. In a larger enterprise, the data lookup hierarchy could be quite complex; however, I recommend the following for a good starting point:

1. Put default values in a file named common.yaml.
2. Put all operating system–specific information in a file named for the OS family as returned by Facter (e.g., RedHat.yaml, Debian.yaml, FreeBSD.yaml)
3. Put information specific to a single node within a file named the fully qualified domain name with a .yaml extension.

You would implement this hierarchy using the following configuration syntax (as you can see, we are interpolating data provided by Facter to choose which files will be read):

hierarchy:
  - name: "Node specific values"
    path: "fqdn/%{facts.fqdn}.yaml"

  - name: "OS-specific values"
    path: "os/%{facts.os.family}.yaml"

  - name: "common"
    path: "common.yaml"

All paths are relative to the datadir of the hierarchy entry, or the value provided in the defaults hash.

hierarchy:
  # Read files specific to the host from multiple paths
  - name: "Node specific values"
    paths: ["infra/fqdn/%{facts.fqdn}.yaml", "product/fqdn/%{facts.fqdn}.yaml"]

  # Read every YAML file in the OS directory
  - name: "OS-specific values"
    glob: "os/%{facts.os.family}/*.yaml"

  # Recursively read every YAML file in the service and provider directories
  - name: "Service value tree"
    globs: ["team1/%{facts.team1}/**/*.yaml", "team2/%{facts.team2}/**/*.yaml"]

  # Read values from an array of services 
  - name: "multiple services"
    mapped_paths: [%{facts.services_array}, name, "service/%{name}.yaml"]

  - name: "encrypted secrets"
    lookup_key: eyaml_lookup_key
    path: "secrets/%{facts.domain}.eyaml"
    options:
      pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7

Merge Strategy

In previous versions of Puppet, you could only set merge strategy globally using a single :merge_behavior configuration key. This was not flexible enough for even basic use cases. Hiera provides the ability to change the merge strategy on a per-lookup basis.

For single values, Hiera will proceed through the hierarchy until it finds a value (at which point, it will stop). For arrays and hashes, Hiera will merge data from each level of the hierarchy, as configured by the merge strategy selected for that lookup key. The following merge strategies are supported:

first (default)—formerly known as priority

Returns the first value found, with no merging. Keys found at the higher priority will return the value from that priority level without any recursion.

hash—formerly known as native

Merge keys only. The values from the higher priority match will be used exclusively, without any values from lower priority keys.

deep—formerly known as deeper

Recursively merge array and hash values. If a key exists at multiple levels, the lower-priority values that don’t conflict will be merged with the higher-priority values.

unique—formerly known as array

Flatten array and scalar values from all priority levels into a single list. Duplicate values will be dropped. Hashes will cause an error.

Puppet allows setting the merge strategy on a per-lookup basis, using the following two methods:

An entry in the lookup_options hash of the data source.

A key the lookup_options hash in the data source. The values assigned to the key are lookup options, including the merge strategy.

Options provided in the lookup() function

Any options used in a lookup() function call will override the lookup_options provided in the Hiera configuration file.

We’ll provide examples of using both methods in “Looking Up Hiera Data”.

Complete Example

Following is a complete example of a Hiera configuration file. This example is what we will use for the rest of testing code within this book. It enables YAML data input from /etc/puppetlabs/code/hieradata, which allows us to share the same Hiera data across all environments.

The following hierarchy prioritizes host-specific values, followed by operating system values, and defaulting to the lowest priority values common to every host:

---
version: 5
defaults:        # for any hierarchy level without these keys
  datadir: data  # directory name inside the environment
  data_hash: yaml_data

hierarchy:
  - name: "Hostname"
    path: "hostname/%{trusted.hostname}.yaml"

  - name: "OS-specific values"
    path: "os/%{facts.os.family}.yaml"

  - name: "common"
    path: "common.yaml"

Let’s go ahead and add this file now to your Puppet code directory:

[vagrant@client ~]$ cp /vagrant/etc-puppet/hiera.yaml /etc/puppetlabs/puppet/

Checking Hiera Values from the Command Line

You can utilize the hiera command-line tool to test lookups of Hiera data. Unfortunately, this tool doesn’t retrieve values in the same manner as Puppet:

[vagrant@client ~]$ hiera puppet::enabled
true
[vagrant@client ~]$ hiera puppet::status
running

Wait, weren’t these values changed for the client host? Yes, they were. But the hiera command-line tool doesn’t have facts and other configuration data available from Puppet. As a result, it didn’t properly interpret the hierarchy that uses filenames derived from facts. Thus, it has returned the only values it knows to find: the default values from the common.yaml file.

For this reason, it is significantly easier and more accurate to test Hiera values by using the lookup() function to query Hiera data, as follows:

[vagrant@client ~]$ puppet apply -e "notice(lookup('puppet::enabled'))"
Notice: Scope(Class[main]): false
Notice: Compiled catalog for client.example.com in environment production
Notice: Applied catalog in 0.01 seconds

Better yet, it is possible to test Puppet lookups directly from the command line without evaluating code:

[vagrant@client code]$ puppet lookup puppet::status
--- stopped
...

Performing Hiera Lookups in a Manifest

Let’s modify one of our manifests to utilize Hiera data. First, create a manifest that contains variables for the configuration of the Puppet agent service. Name it something like hierasample.pp:

# Always set a default value when performing a Hiera lookup
$status = lookup({ name => 'puppet::status',  default_value => 'running' })
$enabled = lookup({ name => 'puppet::enabled', default_value => true })

# Now the same code can be used regardless of the value
service { 'puppet':
  ensure => $status,
  enable => $enabled,
}

Execute this manifest utilizing the Hiera data we created:

[vagrant@client ~]$ sudo puppet apply /vagrant/manifests/hierasample.pp
Notice: Compiled catalog for client.example.com in environment production
Notice: /Stage/Main/Service[puppet]/ensure: ensure changed 'running' to 'stopped'
Notice: /Stage/Main/Service[puppet]/enable: enable changed 'true' to 'false'
Notice: Applied catalog in 0.07 seconds

This has stopped the Puppet service, and prevented it from starting at boot, due to the overrides we added for this specific hostname.

Now let’s comment out the host-specific override in hostname/client.yaml, and reapply the manifest:

[vagrant@client ~]$ rm /etc/puppetlabs/code/hieradata/hostname/client.yaml
[vagrant@client ~]$ sudo puppet apply hierasample.pp
Notice: Compiled catalog for client.example.com in environment production
Notice: /Stage/Main/Service[puppet]/ensure: ensure changed 'stopped' to 'running'
Notice: /Stage/Main/Service[puppet]/enable: enable changed 'false' to 'true'
Notice: Applied catalog in 0.07 seconds

Now that the host-specific override has been removed, the default values we placed in common.yaml are applied.

Testing Merge Strategy

For a more complex example, let’s test how data in hashes can be merged together. Using the example Hiera configuration we created in the previous section, define some users in the global common.yaml file, like so:

# common.yaml
users:
  jill:
    uid: 1000
    home: '/home/jill'
  jack:
    uid: 1001
    home: '/home/jack'

Let’s query this data source and confirm the global values. The following are the results for any system without any higher-priority overrides:

[vagrant@client ~]$ puppet lookup users
---
jill:
  uid: 1000
  home: "/home/jill"
jack:
  uid: 1001
  home: "/home/jack"

On the client system, we actually want home directories in /homes/ for some strange reason. In addition, we have a special local user, Jane. As we only want to change the user’s home directories, we might create a higher-priority hostname/client.yaml file with just the differences:

# hostname/client.yaml
users:
  jill:
    home: '/homes/jill'
  jack:
    home: '/homes/jack'
  jane:
    uid : 999
    home: '/homes/jane'

So let’s test this out now with a command-line lookup() query. We’ll compare the results for a default match with a match for the client machine, as shown here:

[vagrant@client ~]$ puppet lookup --node default users
---
jill:
  uid: 1000
  home: "/home/jill"
jack:
  uid: 1001
  home: "/home/jack"

[vagrant@client ~]$ puppet lookup users
---
jill:
  home: "/homes/jill"
jack:
  home: "/homes/jack"
jane:
  uid: 999
  home: "/homes/jane"

What happened to the user’s UIDs when run on the local client node? When searching for users, it found the jill and jack user keys in the hostname/client.yaml file. It accepted the value associated with that key, and ignored the unique sub-keys in the lower-priority global file. To get the same results with first or hash merging, you’d have to repeat all keys in the higher-priority file.

Let’s try a query that performs a recursive merge through both hashes and find unique keys at each level:

[vagrant@client ~]$ puppet lookup users --merge deep
---
jill:
  uid: 1000
  home: "/homes/jill"
jack:
  uid: 1001
  home: "/homes/jack"
jane:
  uid: 999
  home: "/homes/jane"

As you can see, this result merged hash keys from all levels in the hierarchy. When keys matched ("home" for instance), it chose the higher-priority value.

As we know that users should be merged from all levels in the hierarchy, let’s set the merge strategy using the lookup_options hash in common.yaml, as shown here:

# common.yaml
lookup_options:
  users:
    merge: deep

By placing this configuration in the data, the deep merge strategy will be used by default for lookups of the users key:

[vagrant@client ~]$ puppet lookup users
---
jill:
  uid: 1000
  home: "/homes/jill"
jack:
  uid: 1001
  home: "/homes/jack"
jane:
  uid: 999
  home: "/homes/jane"