A simple puppet test framework with Docker

I've been looking at the puppet testing setups (rspec etc) and while I may be missing the point, they don't appear to meet my immediate requirements.

I have a *lot* of puppet manifests that have been coded to allow for complex server sets, for example a server A may have 10+ configuration sets applied, server B 20+ etc.   A complicating factor is that these configuration sets (manifests) will be sourced from different parts of the business, so I've been looking for a testing solution I can run on the CI server (bamboo) to test these assorted manifesto's before they hit and potentially break in production.

Enter Docker.

I've included this simple example at: https://github.com/agronaught/puppettest

$puppetroot/modules  -- puppet modules.$puppetroot/scripts   -- scripts for automating the use of puppet.

$puppetroot/docker   -- docker test scripts.

The modules directory contains, oddly enough, puppet modules.  For this example I've included one of the sudo modules from the puppet forge and my test manifest:

$puppetroot/modules/sudo -- sudo manifests from puppet forge
$puppetroot/modules/testbase -- the test manifests

In this example the test manifest itself is very simple:

class testbase {

  class { '::sudo': 

    purge => true,

    config_file_replace => false,

  }

  sudo::conf { "root-notty": 

    content => "Defaults:root !requiretty"

  }

}

it essentially installs sudo and then adds a config to allow the root account to run sudo in a batch script.

The docker script is also very simple:

# DOCKER-VERSION 1.0.1

FROM centos:6.4

MAINTAINER Jason Ball <jason@ball.net>





# Install yum repo for puppet and install it

RUN rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

RUN yum install -y puppet





# Mount the puppet modules...

RUN mkdir -p /tmp/puppet

ADD . /tmp/puppet



RUN /usr/bin/puppet apply -e 'class { "::testbase": }'  --verbose --modulepath=/tmp/puppet/modules

This container is based on the publicly available centos image and:

• installs puppet from the puppet repository
• mounts the manifests to be tested under /tmp/puppet
• runs the 'testbase' manifests on the container

There is a high level script to run the docker build and any test scripts if finds against the resultant image. This should allow for test scripts to be run against the resultant image to ensure configurations load, check for errors/failures, test new features, etc.

#!/bin/bash



##

# housekeeping

##

function cleanup {

    rm Dockerfile

    echo "Stopping autotest docker"

    docker ps -a | grep autotest | awk '{print $1}' | xargs docker rm 

    docker images | grep autotest | awk '{print $3}' | xargs docker rmi

}

#trap cleanup EXIT



#

# Run the scripts

#

echo "Running Autotest"

echo "Building test image"

cp ./docker/autotest/Dockerfile .

docker build --no-cache=true --tag="autotest" .



echo "Running Test Scripts"

for script in `find docker -name "test*.sh" | xargs`; do

    echo "Running $script..."

    $script

    status=$?

    if [ $status -ne 0 ]; then

        echo "ERROR: Script returned non zero status: $status -- error in $script"

        break

    fi

done

exit $status

The rest should be fairly self explanatory, the only prerequisite is that docker is setup on the system thats going to run the scripts.

The run for the docker build:

$ ./scripts/run_autotest.sh Running AutotestBuilding test imageSending build context to Docker daemon 878.1 kBSending build context to Docker daemon Step 0 : FROM centos:6.4 ---> 539c0211cd76Step 1 : MAINTAINER Jason Ball <jason@ball.net> ---> Running in a48a2627ec29 ---> cbda1f07a5beRemoving intermediate container a48a2627ec29Step 2 : ---> Running in 1b3869f27a98 ---> 9c047488909aRemoving intermediate container 1b3869f27a98Step 3 :  ---> Running in 9e04aab7d891 ---> 9b83b773effeRemoving intermediate container 9e04aab7d891Step 4 : RUN rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm ---> Running in c3c1f0fe0f6ewarning: /var/tmp/rpm-tmp.tSV15D: Header V4 RSA/SHA1 Signature, key ID 4bd6ec30: NOKEYRetrieving http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpmPreparing...                ##################################################puppetlabs-release          ################################################## ---> d325d0e86a97Removing intermediate container c3c1f0fe0f6eStep 5 : RUN yum install -y puppet ---> Running in 67e098975729Loaded plugins: fastestmirrorSetting up Install ProcessResolving Dependencies--> Running transaction check---> Package puppet.noarch 0:3.6.2-1.el6 will be installed--> Processing Dependency: facter >= 1:1.7.0 for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: ruby >= 1.8.7 for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: hiera >= 1.0.0 for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: ruby >= 1.8 for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: ruby-rgen >= 0.6.5 for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: ruby(selinux) for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: /usr/bin/ruby for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: rubygem-json for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: ruby-shadow for package: puppet-3.6.2-1.el6.noarch--> Processing Dependency: ruby-augeas for package: puppet-3.6.2-1.el6.noarch--> Running transaction check---> Package facter.x86_64 1:2.1.0-1.el6 will be installed--> Processing Dependency: dmidecode for package: 1:facter-2.1.0-1.el6.x86_64--> Processing Dependency: pciutils for package: 1:facter-2.1.0-1.el6.x86_64--> Processing Dependency: virt-what for package: 1:facter-2.1.0-1.el6.x86_64--> Processing Dependency: which for package: 1:facter-2.1.0-1.el6.x86_64---> Package hiera.noarch 0:1.3.4-1.el6 will be installed---> Package libselinux-ruby.x86_64 0:2.0.94-5.3.el6_4.1 will be installed--> Processing Dependency: libselinux = 2.0.94-5.3.el6_4.1 for package: libselinux-ruby-2.0.94-5.3.el6_4.1.x86_64---> Package ruby.x86_64 0:1.8.7.352-13.el6 will be installed--> Processing Dependency: ruby-libs = 1.8.7.352-13.el6 for package: ruby-1.8.7.352-13.el6.x86_64--> Processing Dependency: libruby.so.1.8()(64bit) for package: ruby-1.8.7.352-13.el6.x86_64---> Package ruby-augeas.x86_64 0:0.4.1-3.el6 will be installed--> Processing Dependency: augeas-libs >= 0.8.0 for package: ruby-augeas-0.4.1-3.el6.x86_64--> Processing Dependency: libaugeas.so.0(AUGEAS_0.10.0)(64bit) for package: ruby-augeas-0.4.1-3.el6.x86_64--> Processing Dependency: libaugeas.so.0(AUGEAS_0.1.0)(64bit) for package: ruby-augeas-0.4.1-3.el6.x86_64--> Processing Dependency: libaugeas.so.0(AUGEAS_0.11.0)(64bit) for package: ruby-augeas-0.4.1-3.el6.x86_64--> Processing Dependency: libaugeas.so.0(AUGEAS_0.8.0)(64bit) for package: ruby-augeas-0.4.1-3.el6.x86_64--> Processing Dependency: libaugeas.so.0(AUGEAS_0.12.0)(64bit) for package: ruby-augeas-0.4.1-3.el6.x86_64--> Processing Dependency: libaugeas.so.0()(64bit) for package: ruby-augeas-0.4.1-3.el6.x86_64---> Package ruby-rgen.noarch 0:0.6.5-2.el6 will be installed---> Package ruby-shadow.x86_64 1:2.2.0-2.el6 will be installed---> Package rubygem-json.x86_64 0:1.5.5-1.el6 will be installed--> Processing Dependency: rubygems for package: rubygem-json-1.5.5-1.el6.x86_64--> Running transaction check---> Package augeas-libs.x86_64 0:1.0.0-5.el6_5.1 will be installed---> Package dmidecode.x86_64 1:2.12-5.el6_5 will be installed---> Package libselinux.x86_64 0:2.0.94-5.3.el6 will be updated--> Processing Dependency: libselinux = 2.0.94-5.3.el6 for package: libselinux-utils-2.0.94-5.3.el6.x86_64---> Package libselinux.x86_64 0:2.0.94-5.3.el6_4.1 will be an update---> Package pciutils.x86_64 0:3.1.10-2.el6 will be installed--> Processing Dependency: pciutils-libs = 3.1.10-2.el6 for package: pciutils-3.1.10-2.el6.x86_64--> Processing Dependency: libpci.so.3(LIBPCI_3.1)(64bit) for package: pciutils-3.1.10-2.el6.x86_64--> Processing Dependency: libpci.so.3(LIBPCI_3.0)(64bit) for package: pciutils-3.1.10-2.el6.x86_64--> Processing Dependency: libpci.so.3()(64bit) for package: pciutils-3.1.10-2.el6.x86_64---> Package ruby-libs.x86_64 0:1.8.7.352-13.el6 will be installed--> Processing Dependency: libssl.so.10(libssl.so.10)(64bit) for package: ruby-libs-1.8.7.352-13.el6.x86_64--> Processing Dependency: libcrypto.so.10(libcrypto.so.10)(64bit) for package: ruby-libs-1.8.7.352-13.el6.x86_64--> Processing Dependency: libreadline.so.5()(64bit) for package: ruby-libs-1.8.7.352-13.el6.x86_64---> Package rubygems.noarch 0:1.3.7-5.el6 will be installed--> Processing Dependency: ruby-rdoc for package: rubygems-1.3.7-5.el6.noarch---> Package virt-what.x86_64 0:1.11-1.2.el6 will be installed---> Package which.x86_64 0:2.19-6.el6 will be installed--> Running transaction check---> Package compat-readline5.x86_64 0:5.2-17.1.el6 will be installed---> Package libselinux-utils.x86_64 0:2.0.94-5.3.el6 will be updated---> Package libselinux-utils.x86_64 0:2.0.94-5.3.el6_4.1 will be an update---> Package openssl.x86_64 0:1.0.0-27.el6_4.2 will be updated---> Package openssl.x86_64 0:1.0.1e-16.el6_5.14 will be an update--> Processing Dependency: make for package: openssl-1.0.1e-16.el6_5.14.x86_64---> Package pciutils-libs.x86_64 0:3.1.10-2.el6 will be installed---> Package ruby-rdoc.x86_64 0:1.8.7.352-13.el6 will be installed--> Processing Dependency: ruby-irb = 1.8.7.352-13.el6 for package: ruby-rdoc-1.8.7.352-13.el6.x86_64--> Running transaction check---> Package make.x86_64 1:3.81-20.el6 will be installed---> Package ruby-irb.x86_64 0:1.8.7.352-13.el6 will be installed--> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package            Arch     Version                Repository             Size================================================================================Installing: puppet             noarch   3.6.2-1.el6            puppetlabs-products   1.3 MInstalling for dependencies: augeas-libs        x86_64   1.0.0-5.el6_5.1        updates               309 k compat-readline5   x86_64   5.2-17.1.el6           base                  130 k dmidecode          x86_64   1:2.12-5.el6_5         updates                73 k facter             x86_64   1:2.1.0-1.el6          puppetlabs-products    89 k hiera              noarch   1.3.4-1.el6            puppetlabs-products    23 k libselinux-ruby    x86_64   2.0.94-5.3.el6_4.1     base                   99 k make               x86_64   1:3.81-20.el6          base                  389 k pciutils           x86_64   3.1.10-2.el6           base                   85 k pciutils-libs      x86_64   3.1.10-2.el6           base                   34 k ruby               x86_64   1.8.7.352-13.el6       updates               534 k ruby-augeas        x86_64   0.4.1-3.el6            puppetlabs-deps        21 k ruby-irb           x86_64   1.8.7.352-13.el6       updates               314 k ruby-libs          x86_64   1.8.7.352-13.el6       updates               1.6 M ruby-rdoc          x86_64   1.8.7.352-13.el6       updates               377 k ruby-rgen          noarch   0.6.5-2.el6            puppetlabs-deps       237 k ruby-shadow        x86_64   1:2.2.0-2.el6          puppetlabs-deps        13 k rubygem-json       x86_64   1.5.5-1.el6            puppetlabs-deps       763 k rubygems           noarch   1.3.7-5.el6            base                  207 k virt-what          x86_64   1.11-1.2.el6           base                   24 k which              x86_64   2.19-6.el6             base                   38 kUpdating for dependencies: libselinux         x86_64   2.0.94-5.3.el6_4.1     base                  108 k libselinux-utils   x86_64   2.0.94-5.3.el6_4.1     base                   81 k openssl            x86_64   1.0.1e-16.el6_5.14     updates               1.5 M
Transaction Summary================================================================================Install      21 Package(s)Upgrade       3 Package(s)
Total download size: 8.3 MDownloading Packages:--------------------------------------------------------------------------------Total                                           1.4 MB/s | 8.3 MB     00:06     warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEYRetrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6Importing GPG key 0xC105B9DE: Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org> Package: centos-release-6-4.el6.centos.10.x86_64 (@febootstrap/$releasever) From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6warning: rpmts_HdrFromFdno: Header V4 RSA/SHA512 Signature, key ID 4bd6ec30: NOKEYRetrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabsImporting GPG key 0x4BD6EC30: Userid : Puppet Labs Release Key (Puppet Labs Release Key) <info@puppetlabs.com> Package: puppetlabs-release-6-10.noarch (installed) From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabsRunning rpm_check_debugRunning Transaction TestTransaction Test SucceededRunning TransactionWarning: RPMDB altered outside of yum.  Updating   : libselinux-2.0.94-5.3.el6_4.1.x86_64                        1/27   Installing : 1:dmidecode-2.12-5.el6_5.x86_64                             2/27   Installing : virt-what-1.11-1.2.el6.x86_64                               3/27   Updating   : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64                  4/27   Installing : augeas-libs-1.0.0-5.el6_5.1.x86_64                          5/27   Installing : libselinux-ruby-2.0.94-5.3.el6_4.1.x86_64                   6/27   Installing : 1:make-3.81-20.el6.x86_64                                   7/27   Updating   : openssl-1.0.1e-16.el6_5.14.x86_64                           8/27   Installing : compat-readline5-5.2-17.1.el6.x86_64                        9/27   Installing : ruby-libs-1.8.7.352-13.el6.x86_64                          10/27   Installing : ruby-1.8.7.352-13.el6.x86_64                               11/27   Installing : 1:ruby-shadow-2.2.0-2.el6.x86_64                           12/27   Installing : ruby-rgen-0.6.5-2.el6.noarch                               13/27   Installing : ruby-irb-1.8.7.352-13.el6.x86_64                           14/27   Installing : ruby-rdoc-1.8.7.352-13.el6.x86_64                          15/27   Installing : rubygems-1.3.7-5.el6.noarch                                16/27   Installing : rubygem-json-1.5.5-1.el6.x86_64                            17/27   Installing : hiera-1.3.4-1.el6.noarch                                   18/27   Installing : ruby-augeas-0.4.1-3.el6.x86_64                             19/27   Installing : pciutils-libs-3.1.10-2.el6.x86_64                          20/27   Installing : pciutils-3.1.10-2.el6.x86_64                               21/27   Installing : which-2.19-6.el6.x86_64                                    22/27   Installing : 1:facter-2.1.0-1.el6.x86_64                                23/27   Installing : puppet-3.6.2-1.el6.noarch                                  24/27   Cleanup    : libselinux-utils-2.0.94-5.3.el6.x86_64                     25/27   Cleanup    : libselinux-2.0.94-5.3.el6.x86_64                           26/27   Cleanup    : openssl-1.0.0-27.el6_4.2.x86_64                            27/27   Verifying  : ruby-libs-1.8.7.352-13.el6.x86_64                           1/27   Verifying  : which-2.19-6.el6.x86_64                                     2/27   Verifying  : libselinux-2.0.94-5.3.el6_4.1.x86_64                        3/27   Verifying  : 1:ruby-shadow-2.2.0-2.el6.x86_64                            4/27   Verifying  : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64                  5/27   Verifying  : puppet-3.6.2-1.el6.noarch                                   6/27   Verifying  : pciutils-libs-3.1.10-2.el6.x86_64                           7/27   Verifying  : ruby-augeas-0.4.1-3.el6.x86_64                              8/27   Verifying  : virt-what-1.11-1.2.el6.x86_64                               9/27   Verifying  : compat-readline5-5.2-17.1.el6.x86_64                       10/27   Verifying  : ruby-rdoc-1.8.7.352-13.el6.x86_64                          11/27   Verifying  : 1:facter-2.1.0-1.el6.x86_64                                12/27   Verifying  : pciutils-3.1.10-2.el6.x86_64                               13/27   Verifying  : 1:make-3.81-20.el6.x86_64                                  14/27   Verifying  : rubygems-1.3.7-5.el6.noarch                                15/27   Verifying  : 1:dmidecode-2.12-5.el6_5.x86_64                            16/27   Verifying  : ruby-rgen-0.6.5-2.el6.noarch                               17/27   Verifying  : ruby-irb-1.8.7.352-13.el6.x86_64                           18/27   Verifying  : rubygem-json-1.5.5-1.el6.x86_64                            19/27   Verifying  : openssl-1.0.1e-16.el6_5.14.x86_64                          20/27   Verifying  : augeas-libs-1.0.0-5.el6_5.1.x86_64                         21/27   Verifying  : libselinux-ruby-2.0.94-5.3.el6_4.1.x86_64                  22/27   Verifying  : ruby-1.8.7.352-13.el6.x86_64                               23/27   Verifying  : hiera-1.3.4-1.el6.noarch                                   24/27   Verifying  : libselinux-2.0.94-5.3.el6.x86_64                           25/27   Verifying  : libselinux-utils-2.0.94-5.3.el6.x86_64                     26/27   Verifying  : openssl-1.0.0-27.el6_4.2.x86_64                            27/27 
Installed:  puppet.noarch 0:3.6.2-1.el6                                                   
Dependency Installed:  augeas-libs.x86_64 0:1.0.0-5.el6_5.1                                            compat-readline5.x86_64 0:5.2-17.1.el6                                          dmidecode.x86_64 1:2.12-5.el6_5                                                 facter.x86_64 1:2.1.0-1.el6                                                     hiera.noarch 0:1.3.4-1.el6                                                      libselinux-ruby.x86_64 0:2.0.94-5.3.el6_4.1                                     make.x86_64 1:3.81-20.el6                                                       pciutils.x86_64 0:3.1.10-2.el6                                                  pciutils-libs.x86_64 0:3.1.10-2.el6                                             ruby.x86_64 0:1.8.7.352-13.el6                                                  ruby-augeas.x86_64 0:0.4.1-3.el6                                                ruby-irb.x86_64 0:1.8.7.352-13.el6                                              ruby-libs.x86_64 0:1.8.7.352-13.el6                                             ruby-rdoc.x86_64 0:1.8.7.352-13.el6                                             ruby-rgen.noarch 0:0.6.5-2.el6                                                  ruby-shadow.x86_64 1:2.2.0-2.el6                                                rubygem-json.x86_64 0:1.5.5-1.el6                                               rubygems.noarch 0:1.3.7-5.el6                                                   virt-what.x86_64 0:1.11-1.2.el6                                                 which.x86_64 0:2.19-6.el6                                                     
Dependency Updated:  libselinux.x86_64 0:2.0.94-5.3.el6_4.1                                          libselinux-utils.x86_64 0:2.0.94-5.3.el6_4.1                                    openssl.x86_64 0:1.0.1e-16.el6_5.14                                           
Complete! ---> 3929c2fd7335Removing intermediate container 67e098975729Step 6 : RUN mkdir -p /tmp/puppet ---> Running in 1862953608fa ---> 4eaf5c96f9a3Removing intermediate container 1862953608faStep 7 : ADD . /tmp/puppet ---> c47c7a0a6743Removing intermediate container 0e2be2ebd336Step 8 : RUN /usr/bin/puppet apply -e 'class { "::testbase": }'  --verbose --modulepath=/tmp/puppet/modules ---> Running in e728af515dffInfo: Loading facts in /tmp/puppet/modules/stdlib/lib/facter/puppet_vardir.rbInfo: Loading facts in /tmp/puppet/modules/stdlib/lib/facter/root_home.rbInfo: Loading facts in /tmp/puppet/modules/stdlib/lib/facter/pe_version.rbInfo: Loading facts in /tmp/puppet/modules/stdlib/lib/facter/facter_dot_d.rbWarning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaultsNotice:secondsWarning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.   (at /usr/lib/ruby/site_ruby/1.8/puppet/type.rb:816:in `set_default')Info: Applying configuration version '1404712257'Notice: /Stage[main]/Sudo::Package/Package[sudo]/ensure: createdNotice: /Stage[main]/Sudo/File[/etc/sudoers.d/]/mode: mode changed '0750' to '0550'Notice: /Stage[main]/Testbase/Sudo::Conf[root-notty]/File[10_root-notty]/ensure: createdInfo: /Stage[main]/Testbase/Sudo::Conf[root-notty]/File[10_root-notty]: Scheduling refresh of Exec[sudo-syntax-check for file /etc/sudoers.d/10_root-notty]Notice: /Stage[main]/Testbase/Sudo::Conf[root-notty]/Exec[sudo-syntax-check for file /etc/sudoers.d/10_root-notty]: Triggered 'refresh' from 1 eventsInfo: Creating state file /var/lib/puppet/state/state.yamlNotice: Finished catalog run in 2.17 seconds ---> 0abda7c7aed6Removing intermediate container e728af515dffSuccessfully built 0abda7c7aed6

And the test scripts:

Successfully built 0abda7c7aed6
Running Test Scripts
Running docker/autotest/test_sudo.sh...
Running docker/autotest/test_sudo_bad.sh...
sudo: unknown user: baduser
sudo: unable to initialize policy plugin
ERROR: Script returned non zero status: 1 -- error in docker/autotest/test_sudo_bad.sh

Nice and simple and easily automated in Bamboo.   In my case I have an auto test dock for each logical group of manifests representing a production environment and this approach effectively allows me to perform a sociability test on scripts prior to hitting production.

The test scripts can be coded using whatever tool set your happiest with.

Managing users with Puppet (aka creating a custom resource type)

I have to admit I've become something of a puppet fanatic.  I now have a policy of no changes to systems outside of puppet, no exceptions, and for the most part that works well.   The only exceptions I have at this stage are significant file system changes, and they don't happen all that often.

It takes a while to get your head around puppet and how to make use of the features it provides, one that I really like is user management where centralised authentication isn't an option.

Puppet provides a convenient resource type for managing users which can be used to create your initial template:

$/usr/local/bin/puppet resource user FOO

 user { 'FOO':

  ensure           => 'present',

  comment          => 'User FOO',

  gid              => '56666',

  groups           => ['support'],

  home             => '/home/FOO',

  password         => '<its a secret>',

  password_max_age => '99999',

  password_min_age => '0',

  shell            => '/bin/bash',

  uid              => '56666',

}

This is an easy way to create an initial user resource for central management, however I've often found that I want to be able to manage more than the basic account entries.   What if we want to install config files and manage permissions for each user in a generic way ?

The simple answer is to create our own resource type.  For example I want to be able to manage users, their group memberships, uid/gid and ssh key essentially a subset of the default user resource.   I also want to enforce file permissions and install some 'standard' config files when user accounts are created.

So we define a custom virtual resource type 'account':

class virtual::define::account {
}
define account::virtual ($uid,
                         $realname,
                         $unixgroups,
                         $key,
                         $ensure) {
  case $osfamily {
    /RedHat/: {
      case $ensure {
        /present/: {
          user { $title:
            ensure => present,
            uid => $uid,
            gid => $title,
            groups => $unixgroups,
            membership => inclusive,  
            shell => '/bin/bash',
            home => "/home/${title}",
            comment => $realname,
            managehome => true,
            require => Group[$title],
          }
          group { $title:
            gid    => $uid,
          }
          file { "/home/$title":
            ensure => directory,
            owner => $title,
            group => $title,
            mode => "0750",
            require => [ User[$title], Group[$title] ],
          }
          file { "/home/$title/.ssh":
            ensure => directory,
            owner => $title,
            group => $title,
            mode => "0700",
            require => [ User[$title], Group[$title], File["/home/$title"] ],
          }
    file { "/home/$title/.ssh/authorized_keys":
            ensure => file,
            content => $key,
            owner => $title,
            group => $title,
            mode => "0600",
            replace => false,
            backup => false,
            require => [ User[$title], Group[$title], File["/home/$title/.ssh"] ],
          }
          exec { "${title}-etc-skel":
            command => "/bin/cp /etc/skel/.* /home/${title}/; chown -R ${title}:fxsupport /home/${title}",
            require => [ User["${title}"], File["/home/$title"] ],
            unless => "/usr/bin/test -f /home/$title/.bash_profile",
          }
          file_line{"$title-umask":
            ensure => "present",
            path => "/home/$title/.bash_profile",
            line => "umask 0022",
            require => [ User["$title"]],
          }

          file { "/home/$title/.deploytool":
            ensure => file,
            source => "puppet:///modules/virtual/default-deploytool",
            owner => $title,
            group => $title,
            mode => "0600",
            replace => "true",
            backup => "true",
            require => [ User[$title], Group[$title], File["/home/$title"]],
          }
        } # present
        /absent/: {
          user { $title: ensure => absent, uid => $uid, gid => $title, before => Group["$title"]}
          group { $title: gid  => $uid, ensure => absent }
          file { "/home/$title": ensure => absent, force => true }
        } # absent
        default: {
        }
      } # case $ensure
    } # case RedHat
 default: {
    }
  }
}

I put this class into the 'modules/virtual/manifests/define/account.pp' directory.   Once created I can define the actual user accounts in 'modules/virtual/manifests/account.pp:

 

class virtual::accounts {    include virtual::define::account
            @account::virtual { 'FOO':                uid => 501,                realname => 'User FOO',                unixgroups => ["support"],                key => "<ssh key>",                ensure => present,        }}

Declaring a virtual account resource just as you would the standard 'user' resources.

Getting puppet to create the accounts uses the same mechanism you would use for a 'user' resource.  For example to create all user accounts in with the 'support' group:

class usermgmt::createaccounts {
  case $::osfamily {
    /RedHat/: {
        include virtual::groups
        include virtual::accounts
        realize(Group['support'])
        Account::Virtual <| unixgroups == 'support' |>
    } # Redhat
    default: {}
  } # case
}

And your done.   For bonus points you can use the 'resources' class to purge any accounts that are not explicitly managed by puppet, although do use care with this one as the results could be disastrous if your not careful...

    resources { 'user':

            purge => true

    }

Don't just disable SELinux!!

SELinux is a good thing, a *very* good thing for improving the security of your linux box.   Unfortunately most people seem to simply 'turn it off' rather than fix seemingly trivial problems, the following is an example.

On a new RHEL6.4 server install, SELinux is enabled by default in a 'enforcing' 'targeted' mode.  This means that the 'standard' daemons on the system are protected by a predefined security scheme configured by Redhat.    This is a good thing as even if there is a remote exploit for the daemon, the operating system can enforce access rules and limit or prevent anybody from actually doing anything with it.   Best of all, it's preconfigured, out of the box, ready to use.

Unfortunately it doesn't always work as expected and people turn it off.   In this instance I was informed that people couldn't use authorized_keys to login to the server.

Checking /var/log/audit/audit.log provided the following clue:

type=USER_AUTH msg=audit(1391466473.340:95114): user pid=60103 uid=0 auid=59418 ses=6119 subj=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="l059418" exe="/usr/sbin/sshd" hostname=? addr=10.28.11.195 terminal=ssh res=failed'

  and in /var/log/secure

sshd[60103]: debug1: Could not open authorized keys '/home/user/.ssh/authorized_keys2': Permission denied

You could run the above audit error via audit2allow and import the policy, but it won't work as these rules already exist and then give up in disgust.  I'm not going to go into specifics for selinux as there are a heap of sites on the subject, the fix here was simply to fix the tags on the files so they have the correct security context.

$restorcon -Rv /home

And try again. 

Now, that wasn't difficult and was found with a single search of Google.  Better still spend a little time and learn about SELinux as I am, it will be worth the effort.

 

OpenStack

I seem of have a thing about ignoring my blogs, not a good thing really.

In case you have been asleep one of the more exciting developments over the past year or two is 'OpenStack', simply put it provides the ability to create your own stack of AWS like services and it's is (going to be) the hot product over the next few years.

I attended the recent LCA (Linux Conference Australia) and attended a number of sessions on openstack.  The best part is these sessions are available for anybody to watch, so here they are ;)


http://mirror.linux.org.au/linux.conf.au/2014/Monday/220-OpenStack_CI_See_how_OpenStack_runs_a_massively_scalable_testninfrastructure_in_the_open_-_James_E._Blair.mp4

http://mirror.linux.org.au/linux.conf.au/2014/Tuesday/128-The_OpenStack_Project_and_Moving_to_a_Foundation_-_Paul_Holland_Hewlett-Packard_Company.mp4

http://mirror.linux.org.au/linux.conf.au/2014/Tuesday/130-OpenStack_and_the_network_is_there_a_better_way_-_Iain_Robertson_Brocade.mp4

http://mirror.linux.org.au/linux.conf.au/2014/Tuesday/237-OpenStack_at_Canonical_-_Brad_Marshall.mp4

http://mirror.linux.org.au/linux.conf.au/2014/Thursday/78-Rapid_OpenStack_Deployment_for_Novices_and_Experts_Alike_-_Florian_Haas.mp4

http://mirror.linux.org.au/linux.conf.au/2014/Friday/106-How_OpenStack_Improves_Code_Quality_with_Project_Gating_and_Zuul_-_James_Blair.mp4

And while your at it, lots of really good sessions to watch:

http://mirror.linux.org.au/linux.conf.au/2014/

CHROOT Environments - JailKit

A colleague recently asked about chroot environments and how to set these up, it sounded like a good topic for this overly neglected blog so here we go.    On the neglect, sincere apologies as time has been at a premium.  I'm starting a new role next week and hope to reclaim some form of work/life balance which will allow me to work on my own stuff.

So lets start with something simple ;)

CHROOT

A chroot is a simple mechanism available on most (?all?) unix platforms for securing access to system resources.   In this instance a process (or account) has restricted access to the system by creating a 'jail', which contains only those directories, files and binaries that process (or account) is allowed to access.  While the chroot is a simple mechanism, setting up a chroot can be a fairly complicated process which is made much simpler through the use of the 'jailkit' package.

This is very useful for protecting some services such as 'bind' so that in the event of a remote compromise there is limited access to the system for the supposed hacker.   It's also useful for providing remote access to people (partners, businesses etc) where allowing access to the entire system is not ideal.

An example is an 'sftproot', restricting user access to only be able to copy files to a select set of directories.   This functionality can be enforced via command scripts in an aithorized_keys file, I find the chroot to be more secure and faster to setup.   There are also a number of sites that outline how to configure this manually and I'm sure you can google these, in this instance I'm going to look at using the 'jailkit' to automate the setup of an sftp account.

The jailkit package has been around for a while and is still being actively maintained:

http://olivier.sessink.nl/jailkit/

Pre-assembled packages are available from multiple locations, if you use any of the Redhat or Centos distributions you can find these at http://pkgs.repoforge.org/jailkit/.

For this example imagine the following scenario:

Widget Inc requires a login for one of it's partner firms so that files can be exchanged between the companies.  These files will be automatically transferred using 'scp/sftp' with certificate based authentication.  Care needs to be taken to ensure this account is secured to prevent access to any other system functionality or resources. 

It's a simple but fairly common scenario.   It's worth noting that I have used the same setup for automated backups via rsync and it works well but that would be a different discussion.

In this instance the base system is:

• OS: Centos 6.3
• Chroot User: widgetinc
• Chroot Path: /opt/sftproot/widgetinc

Start by installing the 'jailkit' from the above URL:

$rpm -ivh http://pkgs.repoforge.org/jailkit/jailkit-2.11-1.el6.rf.x86_64.rpm

You can inspect the package contents via:

$rpm -ql jailkit

and read the man pages at your leasure.   To create the sftproot we need to create the config file section in /etc/jailkit/jk_init.ini:

[scp]
comment = ssh secure copy
paths = /usr/bin/scp
includesections = netbasics, uidbasics
devices = /dev/urandom

[sftp]
comment = ssh secure ftp
paths = /usr/lib/sftp-server, /usr/libexec/openssh/sftp-server, /usr/lib/misc/sftp-server, /usr/libexec/sftp-server
includesections = netbasics, uidbasics
devices = /dev/urandom, /dev/null
# on solaris
#paths = /usr/lib/ssh/sftp-server

To initialise the 'jail' with support for both 'scp' and 'sftp':

$mkdir /opt/sftproot

$chown root:root /opt/sftproot

$chmod 0755 /opt/sftproot

$jk_init -j /opt/sftproot scp

$jk_init -fj /opt/sftproot sftp

$jk_init -fj /opt/sftproot jk_lsh

$mkdir /opt/sftproot/home

The Jail environment is ready for use, create the user account:

$jk_addjailuser /opt/sftproot widgetinc

You can now set a password on the 'widgetinc' user ($passwd widgetinc) or drop any ssh keys in /opt/sftproot/home/widgetinc/.ssh/authorized_keys and start using the account.

An important part I _did_ forget..

Add a section to /etc/jailkit/jk_lsh.ini for the user.  Also add it to /opt/sftproot/etc/jailkit/jk_lsh.ini if the chroot has already been created:

[widgetinc]
paths=/usr/lib,/usr/bin
executables=/usr/lib/sftp-server,/usr/bin/scp
allow_word_expansion=0
umask=002

Test with 'scp' and 'sftp'.  Any attempt to get an interactive shell should result in the session being dropped.

Nice, simple, and reasonably secure ;)

J.