Starting the dashboard VM

Start up the dashboard virtual instance. This boots a machine which is named dashserver as just noted:

learning-puppet4$ vagrant up dashboard
Bringing machine 'dashboard' up with 'virtualbox' provider...
==> puppetserver: Importing base box 'puppetlabs/centos-7.2-64-nocm'...
==> puppetserver: Matching MAC address for NAT networking...
==> puppetserver: Checking if box 'puppetlabs/centos-7.2-64-nocm' is up to date...
==> puppetserver: Setting the name of the VM: learning-puppet4_dashboard_144075009
...snip...

learning-puppet4$ vagrant ssh dashboard
[vagrant@dashserver ~]$ 

As this is a new virtual instance, install the necessary utilities and a text editor of choice, as you did back in Chapter 2:

$ sudo yum install rsync git vim nano emacs-nox

Ensuring dependencies

One of the reasons I recommend putting Puppet Dashboard on a separate server is because it has a tremendous amount of dependencies. We’re going to install many of them now.

Our first step is to install the EPEL package repository:

[vagrant@puppetmaster ~]$ sudo yum install -y epel-release

Puppet Dashboard is a Rails application, so we need to install Ruby, RubyGems, Rake, and Bundler:

[vagrant@dashserver ~]$ sudo yum -y install ruby ruby-devel rake rubygem-bundler
Loaded plugins: fastestmirror
…snip…

Installed:
 ruby.x86_64 0:2.0.0.598-25.el7_1      ruby-devel.x86_64 0:2.0.0.598-25.el7_1
 rubygem-bundler.noarch 0:1.3.1-3.el7  rubygem-rake.noarch 0:0.9.6-25.el7_1

Dependency Installed:
 libyaml.x86_64 0:0.1.4-11.el7_0       ruby-irb.noarch 0:2.0.0.598-25.el7_1
 ruby-libs.x86_64 0:2.0.0.598-25.el7_1 rubygem-bigdecimal.x86_64 0:1.2.0-25.el7_1
 rubygem-json.x86_64 0:1.7.7-25.el7_1  rubygem-io-console.x86_64 0:0.4.2-25.el7_1
 rubygem-psych.x86_64 0:2.0.0-25.el7_1 rubygems.noarch 0:2.0.14-25.el7_1
 rubygem-rdoc.noarch 0:4.0.0-25.el7_1  rubygem-thor.noarch 0:0.17.0-3.el7
 rubygem-net-http-persistent.noarch 0:2.8-5.el7

Complete!

We’ll need compilers and a JavaScript runtime installed to build native binaries for some extensions. Install the gcc-c++ and nodejs packages to get the necessary tools:

[vagrant@dashserver ~]$ sudo yum -y install gcc-c++ nodejs
Loaded plugins: fastestmirror
…snip…

Installed:
  gcc-c++.x86_64 0:4.8.3-9.el7             nodejs.x86_64 0:0.10.36-3.el7                                                                                                                                                                                                      

Dependency Installed:
  c-ares.x86_64 0:1.10.0-3.el7             libstdc++-devel.x86_64 0:4.8.3-9.el7
  libuv.x86_64 1:0.10.34-1.el7             v8.x86_64 1:3.14.5.10-17.el7
  http-parser.x86_64 0:2.0-4.20121128gitcd01361.el7

Dependency Updated:
  cpp.x86_64 0:4.8.3-9.el7                 gcc.x86_64 0:4.8.3-9.el7
  libgcc.x86_64 0:4.8.3-9.el7              libgomp.x86_64 0:4.8.3-9.el7
  libstdc++.x86_64 0:4.8.3-9.el7                     

Complete!

It also depends on having the MySQL, PostgreSQL, SQLite, and XML development libraries available. You can install them like so:

[vagrant@dashserver ~]$ sudo yum -y install mariadb-devel  \
                         libxml2-devel libxslt-devel sqlite-devel
Loaded plugins: fastestmirror
…snip…

Installed:
 libxslt-devel.x86_64 0:1.1.28-5.el7    libxml2-devel.x86_64 0:2.9.1-5.el7_1.2
 mariadb-devel.x86_64 1:5.5.44-1.el7_1  sqlite-devel.x86_64 0:3.7.17-6.el7_1.1

Dependency Installed:
  zlib-devel.x86_64 0:1.2.7-13.el7      libgcrypt-devel.x86_64 0:1.5.3-12.el7_1
  krb5-devel.x86_64 0:1.12.2-14.el7     libcom_err-devel.x86_64 0:1.42.9-7.el7
  libselinux-devel.x86_64 0:2.2.2-6.el7 libgpg-error-devel.x86_64 0:1.12-3.el7
  libsepol-devel.x86_64 0:2.1.9-3.el7   libverto-devel.x86_64 0:0.2.5-4.el7
  libxslt.x86_64 0:1.1.28-5.el7         openssl-devel.x86_64 1:1.0.1e-42.el7.9
  pcre-devel.x86_64 0:8.32-14.el7       xz-devel.x86_64 0:5.1.2-9alpha.el7
  keyutils-libs-devel.x86_64 0:1.5.8-3.el7

Dependency Updated:
  e2fsprogs.x86_64 0:1.42.9-7.el7        e2fsprogs-libs.x86_64 0:1.42.9-7.el7
  krb5-libs.x86_64 0:1.12.2-14.el7       libcom_err.x86_64 0:1.42.9-7.el7
  libgcrypt.x86_64 0:1.5.3-12.el7_1.1    libxml2.x86_64 0:2.9.1-5.el7_1.2
  libss.x86_64 0:1.42.9-7.el7            openssl.x86_64 1:1.0.1e-42.el7.9
  openssl-libs.x86_64 1:1.0.1e-42.el7.9  pcre.x86_64 0:8.32-14.el7
  sqlite.x86_64 0:3.7.17-6.el7_1.1       xz.x86_64 0:5.1.2-9alpha.el7
  xz-libs.x86_64 0:5.1.2-9alpha.el7

Complete!

Installing Apache for Puppet Dashboard

To run the dashboard under Passenger, we’ll use Apache httpd to provide the base web service. Install Apache httpd, the Apache development tools, and the TLS module as follows:

[vagrant@dashserver ~]$ sudo yum install -y httpd httpd-devel mod_ssl
Loaded plugins: fastestmirror
...snip...

Installed:
 httpd.x86_64 0:2.4.6-31.el7.centos    httpd-devel.x86_64 0:2.4.6-31.el7.centos
 mod_ssl.x86_64 1:2.4.6-31.el7.centos

Dependency Installed:
 apr.x86_64 0:1.4.8-3.el7              apr-devel.x86_64 0:1.4.8-3.el7
 apr-util.x86_64 0:1.5.2-6.el7         apr-util-devel.x86_64 0:1.5.2-6.el7
 cyrus-sasl.x86_64 0:2.1.26-17.el7     cyrus-sasl-devel.x86_64 0:2.1.26-17.el7
 expat-devel.x86_64 0:2.1.0-8.el7      httpd-tools.x86_64 0:2.4.6-31.el7.centos
 mailcap.noarch 0:2.1.41-2.el7         openldap-devel.x86_64 0:2.4.39-6.el7
 libdb-devel.x86_64 0:5.3.21-17.el7_0.1

Complete!

Installing Passenger for Puppet Dashboard

Phusion provides a Yum repo with Passenger binaries. They don’t provide a release RPM, just the repo configuration file. Download and install it with the following commands:

[vagrant@dashserver ~]$ curl -sSLo passenger.repo \
     https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
[vagrant@dashserver ~]$ sudo mv passenger.repo /etc/yum.repos.d/passenger.repo

Now install Passenger and the Passenger module for Apache:

$ sudo yum install -y passenger mod_passenger
Loaded plugins: fastestmirror
...snip...

Installed:
  passenger.x86_64 0:5.0.16-8.el7             mod_passenger.x86_64 0:5.0.16-8.el7

Dependency Installed:
  rubygem-rack.noarch 1:1.5.2-4.el7

Start up Apache and confirm that the Phusion Passenger configuration is correct:

$ sudo systemctl enable httpd
ln -s '/usr/lib/systemd/system/httpd.service'
      '/etc/systemd/system/multi-user.target.wants/httpd.service'
$ sudo systemctl start httpd
$ sudo passenger-config validate-install --validate-apache2 --auto
 * Checking whether this Passenger install is in PATH... ✓
 * Checking whether there are no other Passenger installations... ✓
 * Checking whether Apache is installed... ✓
 * Checking whether the Passenger module is correctly configured in Apache... ✓

Everything looks good. :-)

At this point, the system is prepared for the dashboard to be installed.

Configuring Firewall for Puppet Dashboard

Next, we need to adjust the firewall on the server. Puppet servers will connect to the dashboard to deliver reports. System administrators and DevOps engineers will also browse the results using a web browser:

[dashboard ~]$ sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
success
[dashboard ~]$ sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
success
[dashboard ~]$ sudo firewall-cmd --permanent --zone=public --add-port=3000/tcp
success
[dashboard ~]$ sudo firewall-cmd --reload
success

Enabling the Puppet agent on Puppet Dashboard

At this point, we should add the dashserver node to the Puppet server, to take advantage of the configuration management available. Connect to the server using this command:

[vagrant@dashserver ~]$ puppet agent --test --server=puppet.example.com
Info: Creating a new SSL key for dashserver.example.com
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for dashserver.example.com
Info: Certificate Request fingerprint (SHA256): 4A:25:F1:3A:B5:DC:0F:64:CC...
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled

When the command succeeds, you’ll need to log in to the Puppet server and sign the certificate, as done in Part III and shown here:

[vagrant@puppetserver ~]$ puppet cert --sign dashserver.example.com
Notice: Signed certificate request for dashserver.example.com
Notice: Removing file Puppet::SSL::CertificateRequest dashserver.example.com
 at '/var/opt/puppetlabs/puppetserver/ssl/ca/requests/dashserver.example.com.pem'

Rerun the puppet agent command on the dashboard server, and your node will receive the Puppet configuration you built in Part II.

Installing Puppet Dashboard

Create a puppet-dashboard user account. You can place the home directory anywhere you want, but /usr/share/puppet-dashboard and /opt/puppet-dashboard are the most common places. For the following example, I have chosen the home directory in /opt/puppet-dashboard. Give useradd the -m option to create the home directory, for reasons I’ll explain shortly:

[vagrant@dashserver ~]$ sudo useradd -d /opt/puppet-dashboard -m puppet-dashboard

At this point, we want to become the dashboard user. All further installation steps will be done as the dashboard user. We want all the files installed to be owned by the dashboard user, and operating as that user guarantees that.

The first thing we’ll need to do is fix the permissions and remove all files from the application root. Run the following commands:

[vagrant@dashserver ~]$ sudo su - puppet-dashboard
[puppet-dashboard@dashserver ~]$ chmod 755 /opt/puppet-dashboard
[puppet-dashboard@dashserver ~]$ rm .[a-z]*
[puppet-dashboard@dashserver ~]$ ls -la
total 0
drwx------  2 puppet-dashboard puppet-dashboard  6 Aug 29 06:38 .
drwxr-xr-x. 4 root             root             61 Aug 29 06:38 ..

The directory must be empty or the following git clone operation will fail. As the dashboard user we’re going to clone the puppet-dashboard Git repository into this directory:

$ git clone https://github.com/sodabrew/puppet-dashboard.git ./
Cloning into '.'...
remote: Counting objects: 24226, done.
remote: Total 24226 (delta 0), reused 0 (delta 0), pack-reused 24226
Receiving objects: 100% (24226/24226), 13.54 MiB | 5.41 MiB/s, done.
Resolving deltas: 100% (12058/12058), done.

At this point, we have a complete installation of Puppet Dashboard in an appropriate installation directory. Now run bundler to install the Ruby gems upon which the Dashboard depends. As you’ll see, it installs many, many gems. Bundler will install the dependencies within the dashboard user’s home directory, and not pollute the system paths:

[puppet-dashboard@dashserver ~]$ bundle install --deployment --without postgresql
Fetching gem metadata from https://rubygems.org/...........
Using rake (0.9.6) 
Installing i18n (0.6.11) 
Installing multi_json (1.10.1) 
Installing activesupport (3.2.21) 
 …snip…
Installing uglifier (1.3.0) 
Installing will_paginate (3.0.7) 
Your bundle is complete! It was installed into ./vendor/bundle

We have installed Puppet Dashboard and its dependencies into the /opt/puppet-dashboard directory. Next, we’ll configure the dashboard and initialize the database.

Configuring Puppet Dashboard

To configure Puppet Dashboard, you’ll need to modify two configuration files. The first file is /opt/puppet-dashboard/config/database.yml. You can copy this from the example and then edit it, or simply create it in the format shown here:

[puppet-dashboard@dashserver ~]$ cd config/
[puppet-dashboard@dashserver config]$ cp database.yml.example database.yml
[puppet-dashboard@dashserver config]$ $EDITOR database.yml
[puppet-dashboard@dashserver config]$ cat database.yml
production:
  adapter: mysql2
  database: dashboard_production
  username: dashboard
  password: the password you specified for the dashboard user
  encoding: utf8
[puppet-dashboard@dashserver config]$ chmod 0440 database.yml

As you can see, the database config file contains the database type, username, and password. The chmod command ensures that it is only readable by the dashboard user.

The second file is /opt/puppet-dashboard/config/settings.yml. You should copy this much more extensive file from the example and then edit it:

[puppet-dashboard@dashserver config]$ cp settings.yml.example settings.yml
[puppet-dashboard@dashserver config]$ $EDITOR settings.yml

There are only five settings you must change, which I’ve shown here (obviously you should change the names to match your own domain if you aren’t using example.com to learn):

# CN that is used in Dashboard's certificate.
cn_name: 'dashboard.example.com'

# Hostname of the certificate authority.
ca_server: 'puppet.example.com'

# Hostname of the inventory server.
inventory_server: 'puppet.example.com'

# Hostname of the file bucket server.
file_bucket_server: 'puppet.example.com'

disable_legacy_report_upload_url: true

If you want the dashboard to show times in a specific time zone, uncomment this line and set the appropriate environment:

# Uncomment the following line to set a local time zone.  Run
# "rake time:zones:local" for the name of your local time zone.
#time_zone: 'Pacific Time (US & Canada)'

You can add custom links to the top bar of the dashboard by uncommenting and editing the following lines (this can be useful to direct people to other useful services):

# Adds optional custom links in the top bar of puppet dashboard.
# Example that will add two links:
#custom_links:
# -
#   href: /link/href
#   title: The title that will be displayed
# -
#   href: /node_groups/1
#   title: Special group

Puppet Dashboard can be used as an ENC. This allows you to assign configuration classes (modules) to hosts from the dashboard web interface. If you only intend to use the dashboard to review node status and reports, you should enable the read_only_mode setting. If you intend to use the dashboard as an ENC, this setting must be set to false:

# Disables the UI and controller actions for editing nodes, classes,
# groups and reports.  Report submission is still allowed
enable_read_only_mode: true

Finally, delete this next-to-last line with the secret token, as you’ll be generating a unique one to use:

# Generate a new secret_token for production use!
secret_token: 'b1bbd28f6f9ebfc25f09da9bff4....

Generate a new secret key and add it to the configuration file:

[puppet-dashboard@dashserver config]$ cd ../
[puppet-dashboard@dashserver ~]$ export RAILS_ENV=production
[puppet-dashboard@dashserver ~]$ echo "secret_token: \
  '$(bundle exec rake secret)'" >> config/settings.yml

Defining the Puppet Dashboard schema

There is a Rake task that populates the MySQL database with the appropriate schema (database tables and columns) for Puppet Dashboard. This task produces a lot of output that may not make a lot of sense. However, it will complain loudly if it fails for any reason:

[puppet-dashboard@dashserver ~]$ bundle exec rake db:setup
dashboard_production already exists
-- create_table("delayed_job_failures", {:force=>true})
   -> 0.0104s
-- create_table("delayed_jobs", {:force=>true})
   -> 0.0019s
-- add_index("delayed_jobs", ["failed_at", "run_at", "locked_at", 
 …snip…
-- initialize_schema_migrations_table()
   -> 0.0153s
-- assume_migrated_upto_version(20141217071943,
  ["/home/vagrant/puppet-dashboard/db/migrate"])
   -> 0.0215s

Now that the database has been defined, you can precompile all static assets for higher performance:

[puppet-dashboard@dashserver ~]$ bundle exec rake assets:precompile
/usr/bin/ruby /opt/puppet-dashboard/vendor/bundle/ruby/bin/rake
  assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets

Connecting Puppet Dashboard to Puppet Server

Generate a unique TLS key and certificate request using the following commands. The cert:request command will submit the CSR to the Puppet CA:

[puppet-dashboard@dashserver ~]$ bundle exec rake cert:create_key_pair
[puppet-dashboard@dashserver ~]$ bundle exec rake cert:request

You may get the following error at this point:

[puppet-dashboard@dashserver ~]$ bundle exec rake cert:request
rake aborted!
404 "Not Found"
/opt/puppet-dashboard/lib/puppet_https.rb:34:in 'put'
/opt/puppet-dashboard/lib/tasks/install.rake:50:in 'block (2 levels) in <top>'
Tasks: TOP => cert:request
(See full trace by running task with --trace)

When the command succeeds, you’ll need to log in to the Puppet server and sign the certificate, as done in Part III and shown here:

[vagrant@puppetserver ~]$ puppet cert --sign dashboard.example.com
Notice: Signed certificate request for dashboard.example.com
Notice: Removing file Puppet::SSL::CertificateRequest dashboard.example.com
  at '/var/opt/puppetlabs/puppetserver/ssl/ca/requests/dashboard.example.com.pem'

Return to the dashboard server and execute the following request to download and store the signed certificate. The dashboard will also retrieve the CA’s certificate and the CA revocation list:

[puppet-dashboard@dashserver ~]$ bundle exec rake cert:retrieve
[puppet-dashboard@dashserver ~]$ ls -1 certs/
dashboard.ca_cert.pem
dashboard.ca_crl.pem
dashboard.cert.pem
dashboard.private_key.pem
dashboard.public_key.pem

At this point, Puppet Dashboard is an authorized client of the Puppet server and can submit requests to it.

We are done taking action as the dashboard user, so exit back to our normal vagrant user login. As you won’t need to log in as this user again, we can disable the shell for it at this time:

[puppet-dashboard@dashserver ~]$ exit
[vagrant@dashserver ~]$ sudo usermod -s /bin/nologin puppet-dashboard

Enabling the Dashboard Rails service

Now we’ll configure Apache to start up Puppet Dashboard running as a Rails application under Phusion Passenger. First, Apache comes with several configuration files that provide a helpful defaults for setting up Rails applications. We don’t need and won’t be using this configuration, so let’s remove them:

[vagrant@dashserver ~]$ cd /etc/httpd/conf.d
[vagrant@dashserver conf.d]$ sudo rm ssl.conf welcome.conf userdir.conf

Install an Apache configuration file from the learning-puppet4 repo. This file contains a complete, self-standing service definition for the Puppet Dashboard service:

[vagrant@dashserver conf.d]$ sudo cp /vagrant/etc-puppet/dashboard.conf ./

Restart Apache to pick up the changes. Here’s what it looks like when it restarted successfully:

$ sudo systemctl restart httpd
$ sudo systemctl status httpd
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: active (running) since Thu 2015-08-27 05:24:38 UTC; 5s ago
  Process: 14661 ExecStop=/bin/kill -WINCH ${MAINPID} (status=0/SUCCESS)
 Main PID: 14666 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─14666 /usr/sbin/httpd -DFOREGROUND
           ├─14693 Passenger watchdog
           ├─14696 Passenger core
           ├─14703 Passenger ust-router
           ├─14714 /usr/sbin/httpd -DFOREGROUND
           ├─14715 /usr/sbin/httpd -DFOREGROUND
           ├─14716 /usr/sbin/httpd -DFOREGROUND
           ├─14717 /usr/sbin/httpd -DFOREGROUND
           └─14718 /usr/sbin/httpd -DFOREGROUND

dashboard.example.com systemd[1]: Starting The Apache HTTP Server...
dashboard.example.com systemd[1]: Started The Apache HTTP Server.

There are two logs that may contain errors from the dashboard initialization. Check the Apache error log for Passenger startup failures, and your syslog daemon log for messages from the running Puppet Dashboard service. The following command can be useful to view both logs when you are debugging problems:

[vagrant@dashserver ~]$ sudo tail -f /var/log/messages /var/log/httpd/error_log

Viewing the dashboard

At this point, you should open a browser and connect to the dashboard. If you are running the Dashboard on a Vagrant node, you will need to add the node’s IP address to the hosts file on your personal computer.

On a Mac or Linux system, add the following line to /etc/hosts. On a Windows machine, add the same line to C:\Windows\System32\drivers\etc\hosts:

192.168.250.7  dashboard.example.com

Now open up a browser and connect to http://dashboard.example.com/.

You should see a screen that looks exactly like Figure 27-1.

Figure 27-1. Puppet Dashboard with no node reports

There’s no information here, as no nodes are reporting here yet. Let’s configure the Puppet server to send node reports to the Dashboard.

Sending node reports to the dashboard

Enable the dashboard as a target for node reports on your Puppet server. Do so by adding http to the report processors, and a reporturl setting pointing to the dashboard in /etc/puppetlabs/puppet/puppet.conf:

[master]
  reports = store, http
  reporturl = https://dashboard.example.com:3000/reports/upload

Restart the Puppet server, and it will start forwarding node reports to the dashboard. For Puppet Server the command would be:

[vagrant@puppetserver ~]$ sudo systemctl restart puppetserver

For the deprecated Puppet master, you would restart Apache:

[vagrant@puppetmaster ~]$ sudo systemctl reload httpd

Now you should run Puppet on a client node. After the run has finished, check back and you should see the message shown in Figure 27-2 on the top left of the console.

Figure 27-2. One node report has been received

A pending task means that a report has been received and is waiting to be processed. If you see this message, node report forwarding is working properly. If you do not see this on the dashboard, check the system log on your Puppet server for messages about the report submission:

[vagrant@puppetserver ~]$ grep puppet /var/log/messages

Enabling worker processes

Puppet Dashboard uses background worker processes to process node reports. We need to enable a service to run these workers. Adjust the configuration files for this service like so:

[vagrant@dashserver ~]$ cd /opt/puppet-dashboard/ext/redhat
[vagrant@dashserver redhat]$ sed -e 's#usr/share#opt#' \
          puppet-dashboard.sysconfig | sudo tee /etc/sysconfig/puppet-dashboard
[vagrant@dashserver redhat]$ echo "WORKERS=$(grep cores /proc/cpuinfo \
          | cut -d' ' -f3)" | sudo tee /etc/sysconfig/puppet-dashboard-workers

This has adjusted the system configuration files for the dashboard installation path, and selected the correct number of workers. Now enable and start the puppet-dashboard-workers service like so:

[vagrant@dashserver ~]$ sudo systemctl enable puppet-dashboard-workers
ln -s '/etc/systemd/system/puppet-dashboard-workers.service'
  '/etc/systemd/system/multi-user.target.wants/puppet-dashboard-workers.service'
[vagrant@dashserver ~]$ sudo systemctl start puppet-dashboard-workers

Now that the worker processors are enabled, the pending tasks should reduce to zero. You should also see some information in the node reports.

If Background Tasks says 1 new failed task, click it. Do you see a message about an attribute that was supposed to be an array, like the one shown in Figure 27-3?

Figure 27-3. Background task failure

If so, then you should download and apply the patch from Puppet Dashboard Pull Request 329, which contains a fix for processing reports from Puppet 4 nodes. Then restart the workers service:

[vagrant@dashserver ~]$ sudo -u puppet-dashboard bash -l
[puppet-dashboard@dashserver vagrant]$ cd
[puppet-dashboard@dashserver ~]$ curl -sL \
   http://github.com/sodabrew/puppet-dashboard/pull/329.diff | patch -u -p1
patching file app/models/report.rb
[puppet-dashboard@dashserver ~]$ exit
[vagrant@dashserver ~]$ sudo systemctl restart puppet-dashboard-workers

Viewing nodes by status

The home page of Puppet Dashboard provides an overview of the convergence history of Puppet nodes that connect to the Puppet server.

In the toolbar on the left, the Nodes widget contains a count of all nodes with reports, and another count of the number of nodes that last reported each status, as shown in Figure 27-4.

Figure 27-4. Puppet Dashboard: Nodes widget

In the main display, you’ll find a graphic display of run status over the last 30 days, as shown in Figure 27-5.

Figure 27-5. Puppet Dashboard: Run status graph

Beneath it you’ll find a list of nodes and the last time they reported (Figure 27-6).

Figure 27-6. Puppet Dashboard: Node list

Just above the list of nodes is a tab bar of node statuses. Selecting one of these tabs will limit the list to show you only nodes that last reported the selected status (Figure 27-7).

Figure 27-7. Puppet Dashboard: Node status tabs

Reviewing a node’s history

If you click on a node name, you’ll be taken to the node’s overview page (Figure 27-8). This contains a graph of the node’s Puppet run (catalog evaluation) status over 30 days, a graph of the run time, and a table showing the status from every stored run report for that node.

Figure 27-8. Node overview

In the preceding figure, you’ll notice that four inspections were reported, which contain the current status of all resource attributes for which auditing was enabled. The report was created by running puppet inspect on the node. The dashboard provides an interface to browse through both run reports and audit inspections.

Analyzing a node’s run report

If you click on the datestamp next to a node name, you’ll see an overview of that node’s Puppet catalog evaluation (Figure 27-9). This report includes detailed statistics about the number of resources, the changes made during the run, and the time intervals of each stage of the Puppet run.

Figure 27-9. Puppet run report

Finding node failures

Puppet Dashboard can help you identify problems with nodes. If a Puppet agent convergence fails on a node, the node will show up in the red Failed count on the left.

You can see a list of all nodes whose most recent Puppet run failed by clicking Failed in the widget bar on the left, or by clicking Failed on the Status tab bar in the main display (Figure 27-10).

Figure 27-10. Node status: Failed

You can see a list of all failed Puppet runs, current and previous, by clicking on Reports at the top right, and then Failed in the tab bar shown on that page (Figure 27-11).

Figure 27-11. Run reports: Failed

Identifying out-of-sync nodes

The Puppet Dashboard can help you identify nodes that are out of sync with policy. If a node has not sent a Puppet report in a few hours, the node will show up in the grey Unreported count on the left.

If you click on Unresponsive in the node status widget, you’ll find a report of nodes that have not reported recently (Figure 27-12).

Figure 27-12. Node status: Unresponsive

There are a number of reasons why nodes may not be reporting:

• The node has a networking problem.
• The node has been shut down.
• Puppet has been prevented from running on the node.

Unfortunately, Puppet Dashboard can’t identify why the node hasn’t sent a report to the Puppet server recently. Someone will need to investigate this situation.

Setting the environment for a node

Click on the client.example.com node name in the main screen. Then click the Edit button near the top right of the title bar for the node, shown in Figure 27-13.

Figure 27-13. Puppet Dashboard: Node title bar

On the following screen (Figure 27-14), you have the option to set the node’s environment, to assign classes to the node, or to define parameters (top-level variables) for the node.

Figure 27-14. Puppet Dashboard: Edit node

While it can be useful to assign classes or parameters to individual nodes, you will generally want to do this with a node group, as described shortly. At this time, write a description for the node and select the test environment for it.

Click Update to save your changes.

Repeat this process for each node you wish to add a description for, or assign to the test environment.

Adding classes to the dashboard

Under Classes in the left sidebar, click the “Add class” button. On the following page, enter the name of a class available on the Puppet server. Add a description for the dashboard display, then click Create (Figure 27-15).

Figure 27-15. Puppet Dashboard: Add node class

To make use of the module created in Part II, enter puppet::agent as the class name.

Repeat this process for each class you wish to assign to nodes. You may wish to add mcollective::server to the class list, for example.

Creating node groups

Under Groups in the left sidebar, click the “Add group” button. On the following page enter a name you’d like to use for a group of nodes (Figure 27-16).

Figure 27-16. Puppet Dashboard: Add node group

Enter a description for dashboard users to understand the meaning of the group. As you can see from this screen, a group can contain:

• Nodes that are included in the group
• Other groups that are included in the group
• Classes that apply to all nodes in the group
• Parameter keys and values provided with the group

For practice, you can create a group named Clients, with a node of client.example.com and classes puppet::agent and mcollective::server. For testing purposes, add a parameter named findme with any value you like.

When you are done, click Create. You will see a filtered status view that contains only the results from nodes within the group (Figure 27-17).

Figure 27-17. Puppet Dashboard: View Clients group

Installing the node classifier

To enable ENC lookups from the dashboard’s nodes and classes, install the external_node query script on the Puppet server. You can copy this file from the dashboard instance, or download it directly from the GitHub repository:

[vagrant@puppetserver ~]$ sudo mkdir -p /opt/puppet-dashboard/bin
[vagrant@puppetserver ~]$ sudo chown vagrant /opt/puppet-dashboard/bin
[vagrant@puppetserver ~]$ cd /opt/puppet-dashboard/bin
[vagrant@puppetserver bin]$ curl -sL -o external_node \
    https://github.com/sodabrew/puppet-dashboard/raw/master/bin/external_node
[vagrant@puppetserver bin]$ chmod 755 external_node

There are two ways to make the node classifier work correctly:

• Edit the script to contain the correct dashboard URL and filenames.
• Provide the Dashboard URL and the server’s TLS key and certificate filenames with environment variables.

I have found that editing the script (or installing it with a customized template using Puppet) is easiest. You can edit it as follows:

[vagrant@puppetserver ~]$ $EDITOR /opt/puppet-dashboard/bin/external_node

Adjust the following settings near the top of the script:

DASHBOARD_URL = "https://dashboard.example.com:3000"

# These settings are only used when connecting to dashboard over https (TLS)
CERT_PATH = "/etc/puppetlabs/puppet/ssl/certs/puppetserver.example.com.pem"
PKEY_PATH = "/etc/puppetlabs/puppet/ssl/private_keys/puppetserver.example.com.pem"
CA_PATH   = "/etc/puppetlabs/puppet/ssl/certs/ca.pem"

Testing class and parameter assignment

Before enabling the ENC for the Puppet server, test that ENC node lookup is working properly with the dashboard. The following test query should retrieve the data configured in the Dashboard for the client node:

$ sudo /opt/puppet-dashboard/bin/external_node client.example.com
---
name: client.example.com
environment: test
classes:
  mcollective::server: {}
  puppet::agent: {}
parameters:
  findme: It works!

As you can see, the output contains the environment configured with the node, plus the classes and parameters assigned to the Clients group. The group name itself does not appear in the output, as this is a user interface feature internal to the dashboard.

If you are using an unmodified external_node script, then you may need to set environment variables when performing the query:

[vagrant@puppetserver ~]$ cd /var/opt/puppetlabs/puppetserver
[vagrant@puppetserver puppetserver]$ sudo /usr/bin/env \
  CA_PATH=ssl/certs/ca.pem \
  CERT_PATH=ssl/certs/puppetserver.example.com.pem \
  PKEY_PATH=ssl/private_keys/puppetserver.example.com.pem \
  PUPPET_DASHBOARD_URL=https://dashboard.example.com:3000 \
  /opt/puppet-dashboard/bin/external_node client.example.com

Test out the results with different hostnames. When the output is what you expect, enable the node classifier, as described next.

Querying the dashboard’s ENC

Enable the Puppet Dashboard as an ENC using the following settings in the [master] section of /etc/puppetlabs/puppet/puppet.conf on the Puppet server:

[master]
  node_terminus = exec
  external_nodes = /opt/puppet-dashboard/bin/external_node

Alternatively, pass the Puppet server’s key and certificate to the script using environment variables as follows:

[master]
  node_terminus = exec
  external_nodes = /usr/bin/env CA_PATH=$ssldir/certs/ca.pem
    CERT_PATH=$ssldir/certs/puppetserver.example.com.pem
    PKEY_PATH=$ssldir/private_keys/puppetserver.example.com.pem
    PUPPET_DASHBOARD_URL=https://dashboard.example.com:3000
    /opt/puppet-dashboard/bin/external_node

After making these changes, you’ll need to restart the Puppet server:

[vagrant@puppetserver ~]$ sudo systemctl restart puppetserver

From this point forward, the dashboard will be queried for the environment, classes, and parameters (node variables) of each node. The response from the dashboard will be used along with the results of Puppet Lookup to configure the node.

Eliminating certificate errors

If you are familiar with Apache configuration, you may have observed that the Puppet Dashboard service has been configured to answer on both TCP port 3000 and port 443. This configuration enhances security and improves the user experience.

The Puppet Dashboard service on port 3000 has special rules to allow only connections that utilize TLS certificates signed by the Puppet certificate authority. The Puppet server must provide its own TLS certificate when communicating with the dashboard.

When users connect to view node status, they will connect to the standard TLS port of 443. If their browser is given a certificate signed by the Puppet certificate authority (which is not authorized as a trusted CA in any browser), they will receive a certificate error from their browser.

You can provide a better experience by obtaining a TLS certificate from a certificate authority trusted by web browsers, the same as you would do for any public website. Generate an TLS key and CSR, and install the key and signed certificate on the dashboard server.

Starting at line 23 in /etc/httpd/conf.d/dashboard.conf are the keys and certificate settings for the VirtualHost handling users on port 443. Comment out or remove the lines referencing the Puppet keys and uncomment the lines referencing the new key and certificate. Following is an example that shows a key and certificate installed in the RedHat standard TLS directories:

  # This cert was signed by the Puppet CA and will produce a browser warning
  #SSLCertificateFile      /opt/puppet-dashboard/certs/dashboard.cert.pem
  #SSLCertificateKeyFile   /opt/puppet-dashboard/certs/dashboard.private_key.pem
  # Change to keys and certificates issued by a trusted certificate authority
  # to avoid browser errors
  SSLCertificateFile      /etc/pki/tls/certs/dashboard.crt
  SSLCertificateKeyFile   /etc/pki/tls/private/dashboard.key

Optimizing the database

The dashboard database is a normal MySQL database. You can back it up, restore it, and optimize it using standard tools and processes.

There is a special job that optimizes the dashboard’s database schema. You should configure this to run from the dashboard user’s cron once a month. Edit the dashboard user’s cron like so:

[vagrant@puppetserver ~]$ sudo crontab -u puppet-dashboard -e

The contents of the crontab should contain something like this:

MAILTO=you@example.com
RAILS_ENV=production

* 3 1 * * /bin/bundle exec rake db:raw:optimize

This is the bog-standard Unix crontab format you can find documented everywhere. The operational line here will run the optimize job on the first day of the month at 3 a.m., and email the results to you. You should adjust the email address and timing of the optimization to whatever you desire.

Pruning old reports

Reports will remain in the database forever unless they are pruned. To keep the database size from outgrowing the space available, you should run the following job to prune old reports after they are no longer useful.

As in the previous example, configure this to run from the dashboard user’s cron once a month. Add the following line to the dashboard user’s crontab:

MAILTO=you@example.com
RAILS_ENV=production

* 4 * 0 * /bin/bundle exec rake reports:prune upto=3 unit=mon

This runs the prune job every Sunday at 4 a.m., and emails any output to you. As written, it will expire any report older than three months. You should adjust this parameter based on the disk storage available, and your organization’s data retention requirements.

Rotating logs

While most messages go to the system logger, some low-level log messages are stored in the log/ subdirectory of the installation directory. To keep these files from growing too large, install the dashboard’s logrotate file, which tells the nightly logrotate process how to trim these files.

The only necessary change to this file is to alter the installation path to match where Puppet Dashboard was installed:

$ cd /opt/puppet-dashboard/ext/redhat
$ sed -e 's#usr/share#opt#' puppet-dashboard.logrotate \
    | sudo tee /etc/logrotate.d/puppet-dashboard

This will keep the dashboard’s logfiles from getting out of control.

Ensuring a high-performance dashboard

Once you have finished testing and wish to deploy Puppet Dashboard in production, there are two issues to consider for performance: available RAM and fast storage. The dashboard tends to be write-heavy, generally doing 5:1 writes versus reads at most sites.

The dashboard server only needs sufficient RAM and for the database to be stored on the fastest disks available. Some of the best ways to meet these requirements are as follows:

Physical server

Mirror the root and operating system volumes for redundancy on small, stable disks. Put the MySQL database on a RAID 0+1 or RAID-6 high-speed disk array.

Virtual server

Put the operating system on the regular shared storage volumes used by other virtualized systems. Place the MySQL database on direct attached storage using VMWare’s Raw Device Mapping or OpenStack Block Storage.

Cloud server

Put the operating system on the regular instance volumes (such as AWS GP2 EBS volume). Place the MySQL database on direct attached SSD if possible (e.g., AWS R3 instance), or a remote drive with dedicated bandwidth and high IOPS commitment (e.g., AWS C4 instance with an EBS Provisioned IOPS volume).

Keep in mind this is all relative. If you have fewer than 100 nodes, almost any server will suffice. If you have thousands of nodes and an information radiator querying the dashboard every minute, you’ll want to spend significant effort tuning the server appropriately.

Sizing the dashboard database

The data store size is one of the most important considerations when building a production Puppet Dashboard service. Running out of disk space can cripple the dashboard, and rebuilding a MySQL database on a new volume is nobody’s idea of a fun Friday night.

Thankfully, there is a formula you can use to size your dashboard pretty much exactly as required.

On your Puppet server, you will find node reports in the $vardir/reports directory. These reports contain messages for each resource processed during the Puppet run. The more resources that are applied to a client node, the larger the report file will be.

The dashboard server will be storing every node report for as long as you tune the report database to retain the information. In practice, the MySQL data store uses slightly more space than the compact YAML file in the reports directory.

Using this information, calculate the disk space for the MySQL database by multiplying all of the following pieces of information and rounding up:

• Number of nodes
• Size of the reports
• Puppet runs per day (48 by default)
• Number of days before data is purged
• Average increase in MySQL data versus YAML file = 1.2

In a very stable environment where all nodes apply similar amount of Puppet resources, and very few ad hoc Puppet runs are done out of schedule, it is possible to target the disk space very closely:

In reality, it’s not usually so simple, for a variety of reasons:

• Ongoing work adds new nodes on a regular basis.
• Autoscaled environments have a dynamic number of nodes.
• Different node types have vastly different numbers of resources applied to them, creating a large discrepancy in report sizes.

To give you a concrete example, at one customer we had these node types:

• More than 10,000 globally distributed nodes running Puppet four times a day, and managing only 120 resources.
• A little more than 200 servers in the core data center running Puppet the standard 48 times a day, each with more than 5,000 managed resources.
• An on-demand testing platform that built and tore down complete cluster sets. Nodes in the cluster ran Puppet exactly twice, with more than 14,000 resources applied between the two runs before being destroyed at the end of the test.

To scale the dashboard appropriately in this environment, we did three separate calculations, including a reasonable estimate for the maximum on-demand runs in a single day, and expected growth for servers and remote sites. That estimate was accurate enough that it is still running with average 80% space utilization on the same size disk cluster two years later.

Not every situation is so easy to predict. Last year I worked with a company that has two very large peak periods within the year, during which time the production deployment tripled in size. The saving grace is that during these rush periods the number of ad hoc convergence runs drops to nearly zero, as nonemergency deployments are limited during these periods. All of the reporting systems were scaled to handle peak load, and thus ran with extra capacity outside of the rush periods.