CHROOT Environments - JailKit

A colleague recently asked about chroot environments and how to set these up, it sounded like a good topic for this overly neglected blog so here we go.    On the neglect, sincere apologies as time has been at a premium.  I'm starting a new role next week and hope to reclaim some form of work/life balance which will allow me to work on my own stuff.

So lets start with something simple ;)

CHROOT

A chroot is a simple mechanism available on most (?all?) unix platforms for securing access to system resources.   In this instance a process (or account) has restricted access to the system by creating a 'jail', which contains only those directories, files and binaries that process (or account) is allowed to access.  While the chroot is a simple mechanism, setting up a chroot can be a fairly complicated process which is made much simpler through the use of the 'jailkit' package.

This is very useful for protecting some services such as 'bind' so that in the event of a remote compromise there is limited access to the system for the supposed hacker.   It's also useful for providing remote access to people (partners, businesses etc) where allowing access to the entire system is not ideal.

An example is an 'sftproot', restricting user access to only be able to copy files to a select set of directories.   This functionality can be enforced via command scripts in an aithorized_keys file, I find the chroot to be more secure and faster to setup.   There are also a number of sites that outline how to configure this manually and I'm sure you can google these, in this instance I'm going to look at using the 'jailkit' to automate the setup of an sftp account.

The jailkit package has been around for a while and is still being actively maintained:

http://olivier.sessink.nl/jailkit/

Pre-assembled packages are available from multiple locations, if you use any of the Redhat or Centos distributions you can find these at http://pkgs.repoforge.org/jailkit/.

For this example imagine the following scenario:

Widget Inc requires a login for one of it's partner firms so that files can be exchanged between the companies.  These files will be automatically transferred using 'scp/sftp' with certificate based authentication.  Care needs to be taken to ensure this account is secured to prevent access to any other system functionality or resources. 

It's a simple but fairly common scenario.   It's worth noting that I have used the same setup for automated backups via rsync and it works well but that would be a different discussion.

In this instance the base system is:

• OS: Centos 6.3
• Chroot User: widgetinc
• Chroot Path: /opt/sftproot/widgetinc

Start by installing the 'jailkit' from the above URL:

$rpm -ivh http://pkgs.repoforge.org/jailkit/jailkit-2.11-1.el6.rf.x86_64.rpm

You can inspect the package contents via:

$rpm -ql jailkit

and read the man pages at your leasure.   To create the sftproot we need to create the config file section in /etc/jailkit/jk_init.ini:

[scp]
comment = ssh secure copy
paths = /usr/bin/scp
includesections = netbasics, uidbasics
devices = /dev/urandom

[sftp]
comment = ssh secure ftp
paths = /usr/lib/sftp-server, /usr/libexec/openssh/sftp-server, /usr/lib/misc/sftp-server, /usr/libexec/sftp-server
includesections = netbasics, uidbasics
devices = /dev/urandom, /dev/null
# on solaris
#paths = /usr/lib/ssh/sftp-server

To initialise the 'jail' with support for both 'scp' and 'sftp':

$mkdir /opt/sftproot

$chown root:root /opt/sftproot

$chmod 0755 /opt/sftproot

$jk_init -j /opt/sftproot scp

$jk_init -fj /opt/sftproot sftp

$jk_init -fj /opt/sftproot jk_lsh

$mkdir /opt/sftproot/home

The Jail environment is ready for use, create the user account:

$jk_addjailuser /opt/sftproot widgetinc

You can now set a password on the 'widgetinc' user ($passwd widgetinc) or drop any ssh keys in /opt/sftproot/home/widgetinc/.ssh/authorized_keys and start using the account.

An important part I _did_ forget..

Add a section to /etc/jailkit/jk_lsh.ini for the user.  Also add it to /opt/sftproot/etc/jailkit/jk_lsh.ini if the chroot has already been created:

[widgetinc]
paths=/usr/lib,/usr/bin
executables=/usr/lib/sftp-server,/usr/bin/scp
allow_word_expansion=0
umask=002

Test with 'scp' and 'sftp'.  Any attempt to get an interactive shell should result in the session being dropped.

Nice, simple, and reasonably secure ;)

J.

A little Linux history to start with

It's difficult to believe just how pervasive Linux is these days, I still recall the late 90's where the linux movement was still trying to gain traction and any large commercial user was big news.   Of course, most companies were running Linux in some form, the IT department simply neglected to tell  the business ;)

Then one fine day the following appeared on slashdot:

The Standish Group: I Bet the Farm on Linux

Mar 18, 1999, 21:46 (28 Talkback[s])

A Worldwide Contest

The Standish Group will award a plush cozy blanket to the selected IT executive in a medium or large company anywhere on the planet who can document a successful deployment of any mission-critical application on the Linux operating system.

This luxurious, deep-pile cotton blanket is suitable for curling up into the fetal position and sucking your thumb when your T-1 lines go down. Or when you just feel like schlepping around the office with a dog named Snoopy. HURRY. Enter now to claim this coveted, one-of-a-kind prize. This one-time offer won't last long. Enter NOW while supply (one blanket) lasts. You do not have to be present to win.

Here are the hastily-devised contest rules:

1) Only commercial organizations with annual revenues exceeding $250M are eligible. (Sorry, high-tech companies, non-profit and government agencies are ineligible.)(Sorry, Standish Group employees, you're not eligible either.)

2) The winning organization must be running at least one mission-critical production application on Linux. We define a mission-critical application as an ATM network, a purchasing system, an airline reservation system, an order processing system or any other OLTP system. Task uncritical applications such as general ledger, accounts payable, e-mail, word processing and general office applications don't qualify. Don't bother.

3) We must be able to interview the senior IT executive at the company to verify details of the winning system.

4) Only the senior IT executive at the company can receive this prestigious award.

In the unlikely case of multiple entries, the decision of a panel of distinguished judges will be based upon an aerodynamic random selection method by means of a semi-precious metal to determine the winner. Decision of the judges is final.

To enter, fill in the entry form, including a brief description of your Linux-based mission-critical system. All entries must be submitted by April 1, 1999. (Just kidding. Entries are due in by May 1, 1999.)

Editor's note: The contest is currently closed. Links to the rules and entry form have been taken down at the request of the Standish group.

I was working at Corporate express leading the electronic commerce team at the time.  The original system had been developed by the late Anthony Rumble, Grant Parnell and Myself with a number of other noteworthy people working on the project over the years.   We entered the 'competition', little did we expect we would actually win the thing:

Announcing the Winner of The Standish Group's

"I Bet the Farm on Linux" Contest

Friday, May 7, 1999

The Standish Group is pleased to announce the winner of our "I Bet the Farm on Linux" contest. We created the contest to find a commercial organization with annual revenues exceeding $250 million anywhere on the planet that had successfully deployed a mission-critical application on the Linux operating system.

The winner is Corporate Express Australia Ltd., the leading supplier of office products and services to large Australian corporations and government departments.

The A$342-million publicly traded company has built and deployed a Linux-based e-commerce order processing system, which generates more than 10% of the company's annual sales. (CE-A is a division of the $3.7 billion Corporate Express, Inc., global provider of office and computer products and services in 300 locations worldwide.)

Elke Roewekamp, CE-A's application development manager and leader of the 4-person e-commerce group that built and operates the system, will receive the grand (and only) contest prize -- a cozy, cotton blanket emblazoned with the words "I Bet the Farm on Linux."

The "I Bet the Farm on Linux" contest was created to supplement our research into the adoption of the Linux OS in corporate computing environments.

Our research confirmed that Linux is being used in mission-critical applications, however sparingly. Its market penetration is chiefly on departmental servers, application servers, network servers, e-mail servers, FTP servers, print servers, file servers, Web servers, e-commerce software development, and very small enterprises. It has a reputation for reliability that puts Windows NT to shame. It is cheap. It boasts a string of vendor endorsements. Commercial 24 x 7 technical support is available from HP, IBM and others. These are good things, because they increase the confidence level in Linux for corporate IT directors.

While Linux is suitable for certain applications, can it handle "bet the farm" mission-critical production applications? A manufacturing shop floor production system, for example, or a financial ATM network? We wanted to know. Hence, the contest, which was posted on our Standish Web site from March 15 to April 30, 1999. When the contest closed, we had received entries from companies in the U.S., Australia and Brazil.

In early 1997, Corporate Express Australia Ltd. introduced NetExpress, a business-to-business Linux-based e-commerce order processing system. The company's corporate and governmental customers (CE has no retail operation) can select from 11,000 different products, all available electronically through NetExpress. Earlier this year, CE-A added online stock availability and secure credit card encryption to its e-commerce application.

CE-A launched NetExpress with an initial investment of less than A$50,000 and only one customer. Two years later, NetExpress -- with a total investment of less than A$200,000 -- serves 1,000 customers, processes 800-900 orders per day, and generates A$4 million in sales per month, or over 10% of the company's annual revenue. Roewekamp expects Web sales to double to A$8 million per month by year-end.

NetExpress runs entirely on Linux. (So does CE-A's EDI system, which generates another A$6 million per month via the Web.) For NetExpress, Roewekamp selected Red Hat Linux 5.2 and installed the OS on three Intel-based servers -- one for the database, a second for the Web browser, and a third for catalog image storage.

Roewekamp says low startup costs, bulletproof reliability, and a worldwide community of developers and technical support resources were the three key reasons why CE-A chose Linux. No other operating system was considered.

NetExpress runs 24 x 7, and Roewekamp says her expectations of Linux were "really high." She has not been disappointed. System availability is 99.99% or higher, she says.

"NetExpress has proven to be a market differentiator for us, allowing us to gain a one- or two-year advantage on our major competitors," she says. "Linux lets us concentrate on the business benefits we deliver to our customers rather than continually maintaining or working with the constraints of the more traditional operating systems."

Currently deployed in Australia and New Zealand, NetExpress is being considered for deployment by CE's European offices.

For a demonstration of NetExpress, go to www.ce.com.au. There you can open a free demo account and test all the functions of the system.

I still have the 'security blanket' at home.  Not much of a claim to fame, but worth remembering there was a time when running linux was seen as 'out there'.

Welcome to Updated Admin

As with all blogs this one is starting with a simple idea.  Lets see if I can keep on top of this one ;)

I have been working with Unix based systems for most of the past 20 years, and have worked professionally with Linux for the last 16 years.  Early on I was involved in implementing the first recognised mission critical system on Linux (standish group) and These days I work on high frequency trading (HFT) and low latency systems where tuning every microsecond counts and make use of every trick I can find to achieve these times.

In my journeys across the linux landscape I find there is a lot of seriously cool stuff that has been developed for and added to the core system.  The part that surprises me is that few people even know about this 'cool stuff', even fewer make use of it even when it's applicable.   Some simple examples being cgroups, schedulers, capabilities, security, and many many more.

The intent of this blog is to being some attention to these new kernel and system facilities, why they are important, how and when to use them.   Hopefully this will be of benefit to some other Linux dinosaurs out there.