Geek projects, tidbits and things I'm working on at the moment…

This is the second part of my FTPES, vsftpd & CentOS article. In the first part I walked you thru how I built a certificate, sent the CSR off to my CA and finally, modified the vsftpd.conf file. In this part I’ll show you how to test the service via command-line so you can actually see the certificate and how to punch a hole thru your FW because the data part of your FTP session is now encrypted.

First, let’s talk about verifying the configuration. One of the obvious things you can do is open up a command prompt session and attempt to log into your ftp server. Once connected, does it accept an anonymous connection? Was that what you wanted? Also, was the plain jane ftp command allowing you to log in or did you get a 331 error, “Non-anonymous sessions must use encryption.” If you’ve configured it correctly it shouldn’t allow you to login at all. So how do you test it using encryption? By using openssl w/ s_client of course! Use…

openssl s_client -starttls ftp -connect yourserver.example.com:21

This will allow you to not only log into your server using encryption via command-line but also verify that you’ve got the proper certificate & certificate chain installed. I personally tried connecting w/ out the starttls option but wasn’t successful (instead I get a “SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol” error message. I beleive this is because I’m using vsftpd in explicit mode not impicit mode.) The openssl s_client command can also be used for a number of other encrypted services for debugging certificates which is extremely helpful!

So you’ve verified that FTPES does work on the local LAN but doesn’t seem to work thru your FW. This is where passive ftp comes into play because your FTP traffic is now encrypted and your FW can’t do an inspection to determine which port is going to be used next and open it up for you ahead of time. Most FWs come with some sort of FTP inspect feature but we just killed it because that data is totally encrypted now and it’s unable to sniff the FTP traffic and sense which port needs to be opened. You won’t have to turn on passive FTP because it’s on by default with vsftpd. You should specify the port range and I also turned on pasv_address too:

pasv_enable=YES
pasv_addr_resolve=YES
pasv_address=yourserverhere.example.com
pasv_min_port=63000
pasv_max_port=65535

Once done, this will force your vsftpd server to use a port range of 63000 to 65535 for data connections. Alot of the commands I used above can be found w/in the man page for vsftpd.conf (along with their default values!)

Now you just need to open up that same range on your FW device (NOTE: Each FW is going to be different!) On my Cisco Adaptive Security appliance (ASA) I’m defining an object-group first for the port range like such:

object-group service ftp_passive tcp
  description ftp passive range
  port-object range 63000 65535

Then I use that port range in my outside ACL for inbound traffic like such:

access-list outside_access_in extended permit tcp any host YOUR_PUBLIC_IP object-group ftp_passive
.
.
.

Just be sure to use your public IP address for ‘YOUR_PUBLIC_IP’ in the above ACE. Now when it comes to ACL rules there’s tons of different ways to allow traffic so I won’t go into much more detail here other than how I punched a hole thru my FW to allow the encrypted vsftpd data channel traffic through.

That’s about it folks. Remember, if you’re having problems with your configuration, break it down into simple pieces and troubleshoot it that way versus trying to eat the entire problem at once!

-Q

Oh, almost forgot. Here’s a couple links that might be helpful!

http://en.wikipedia.org/wiki/Ftp
http://en.wikipedia.org/wiki/FTPES
http://it.toolbox.com/blogs/unix-sysadmin/troubleshooting-ssltls-mail-services-29266
http://www.wowtutorial.org/tutorial/26.html
http://vsftpd.beasts.org/
http://www.openssl.org/docs/apps/s_client.html
http://blogs.iis.net/robert_mcmurray/archive/2008/11/10/ftp-clients-part-2-explicit-ftps-versus-implicit-ftps.aspx

At home I typically use Linux whenever possible and feel pretty familiar working with Openssl when it comes to generating different certificates, making a certificate signing request (CSR) and what the different files mean. But when it comes to Windows boxes, I’ve generally used the certificate wizard like everyone else. Today I had an issue with moving a wildcard certificate from one windows box to another because you can’t just look in the file system to find the private key for the public key your certificate authority (CA) just finished issuing to you from the CSR you built previously.

Essentially, you’ve got to use the Windows mmc console, add the certificate Snap-in and export the certificate’s private key & public key to the new server via a PFX file which is password protected. Once you’ve moved it. You can simply select the new certificate at the Bindings option or use the certificate wizard to bring it in. I won’t go into lots of details but will show you the link I found here: http://www.sslshopper.com/move-or-copy-an-ssl-certificate-from-a-windows-server-to-another-windows-server.html

So, for some time now I’ve worked w/ HTTPS and SSL and have had no real issues. Today however, I finally got a chance to put multiple SSL sites on one IIS 6.0 server which wasn’t very intuitive. HTTPS is funny in that the host header data that a web server needs to have access to is encrypted and can’t be used until it’s decrypted. Course for it to decrypt the packet, it must know which certificate to use and very quickly we’ve got a circular issue. Long story short, web servers CAN host multiple SSL websites so long as the sites are variations of  <something>.example.com and you use an apporpriate wild card certificate that will cover all the different sites on your HTTPS web server (such as an SSL certificate issued to *.example.com to cover all of the variations of <something>.example.com.) Ya, I know, confusing.

So while I was applying my craft to a Windows Server 2003 R2 box running IIS 6.0 I quickly encountered an error when I tried to put another website on port 443. Error was, “Cannot register the URL prefix https://*:443/ for site ‘<your site identifier here>’. The necessary network binding may already be in use. The site has been deactivated.” I believe it was event ID 1007 in the event viewer system logs. God I love logs 🙂

Quick search on Google reveals you’ve got to go command-line for this one by executing cscript.exe like so:

cd C:\Inetpub\AdminScripts
cscript.exe adsutil.vbs set /w3svc/Identifier/SecureBindings ":443:host header"

.
.
.
.

You can find your site identifier inside of IIS for the site you’re trying to attach on port 443 and just use your site’s FQDN for the host header field.

See this link for a more thorough walk thru:
http://www.digicert.com/ssl-support/configure-iis-host-headers.htm

So this past weekend I went to visit my Aunt & Uncle in Harrisonburg, VA. While there, my Aunt tells me she can’t send email thru my server which I was fairly certain was once again, “user error.” Come to find out. It was a misconfiguration on my part! Ah ha!

Long story short, I had reconfigured my email server about two months ago and added a DNSBL (zen.spamhaus.org) to my sendmail features and it was blocking her from relaying email thru my sendmail server. I simply had to add…

FEATURE(`delay_checks')

… to my sendmail.mc file and restart the service. (NOTE: On CentOS 5.0 if you restart the sendmail service it will check your sendmail.mc & sendmail.cf files and do a ‘make -C /etc/mail’ for you if the sendmail.mc is newer.)

Now, Sendmail will wait on the DNSBL and give the user a chance to authenticate first which will allow them to then bypass the DNSBL. Seems her residential comcast IP Address was added to zen.spamhaus.org which is commonly done to blocks of residential user IP address spaces to prevent spam.

Ask anyone that knows me and they’ll tell you I love Linux. In particular, I love Red Hat’s flavor of Linux. I’ve still got my first Linux book “The Complete Red Hat 5.2 Installation Guide” and while I don’t consult it first for all problems I do remember thumbing thru its pages when trying to configure those tulip.o drivers and how to change my X Windows config file settings (NOTE: I still to this day get tripped up on configuring X Windows!) So that having been said, it’s no wonder I’m using CentOS and have been for years since Red Hat has stopped releasing a free desktop variety for users to use (I know they’ve got Fedora but that product lines’ release cycle is too aggressive for my tastes.) 

  So that having been said, I love CentOS. I’ve been using it at home for a while now and whenever possible do my best to stick w/ the stock rpms so updates go smoothly. I’ve recently been trying to get vsftpd daemon to use encryption while still jailing my end users in their home directories and here’s what I found out. Vsftpd doesn’t natively do sftp but it does do ftpes pretty well. So I’m going to walk thru generating a private key, making a certificate signing request, adding the public key from your third party certificate authority (CA) to their certificate bundle to form a chained public key and finally, configuring vsftpd to do ftpes.

First, I logged into the box and grabbed superuser.

su -

Then changed directory to the location of the certificates.

cd /etc/pki/tls/certs/

Next, I ran openssl to generate my private key w/o a passphrase! (NOTE: If you decide to use a passphrase a number of services won’t auto start on a power failure which means you’ll need to run home, start the service manually again and enter in the passphrase so the service can use the certificate. Some services can be configured to manually put the passphrase in for you but that’s lots of effort.)

openssl genrsa -out myserver.mydomain.private.key 2048

Had I wanted to generate a private key w/ a passphrase it would have been…

openssl genrsa -des3 -out myserver.mydomain.private.key 2048

You’ll notice at the end of the command I put 2048. That stands for how many bits this certificate is generated with. A number of CAs will require now a days that you use a 2048 bit key versus the old standby of 1024 bit ( the CA I use GoDaddy is one of them.)

The very next thing we need to do is lock down your private key. This is used to encode encrypted communications to the end user who will now use your public key to decrypt the data. So let’s make it readable only by root.

chmod 600 myserver.mydomain.private.key

Once that’s done, it’s time to generate our certificate signing request (CSR) that needs to be passed to your CA so they can generate a public certificate for you.

openssl req -new -key myserver.mydomain.private.key -out myserver.csr

At this point you’ll be prompted for answers to a series of questions which will be placed in your CSR. The big thing you need to get right is your Common Name (CN) question. Make sure the CN is the server name that matches up w/ your DNS entry for this server. Folks are going to use DNS to access your server so the certificate needs to match up with what they are going to use to hit your server across the internet. It’s interesting to note, that w/ GoDaddy, pretty much all of the other fields you fill in get thrown out in the generated public certificate they send back to you so don’t get too freaked out with putting your location information in there. Again, this is with GoDaddy. Other CAs may publish that information w/ the public certificate.

After you’ve submitted your CSR to your CA. You’ll have to go thru the whole domain verification process which can be its’ own little tap dance. I know when our company had their domain names register w/ someone else besides GoDaddy it took 24 to 48 hours for whois verification and whatnot. Once we moved our domain names to GoDaddy the verification process took less that 5 minutes. Coincidence? Business strategy? I don’t know but I’m sure glad we moved our domains over to GoDaddy.

Okay, so you’ve finally gotten your certificate from your CA. Yay! Now comes the fun of lumping that certificate w/ the included CA bundle they sent you and configuring your vsftpd daemon.

First, concatenating the two files together into a combined public certificate file. Why? Cause vsftpd doesn’t have an option for the certificate chain file your CA gave you w/ your public certificate.

cat myserver.crt CA_bundle.crt > myserver.mydomain.public.key

This will put both files together into a combined file which will have all certificates. Now we can configure vsftpd and be done.

Open up the vsftpd.conf file w/ vi.

vi /etc/vsftpd/vsftpd.conf

When looking at your vsftpd.conf file I started at the beginning and slowly worked my way thru turning off features I knew I didn’t need. One of the first was anonymous_enabled. It defaults to enabled unless you uncomment the line and tell it NO. Most everything else I left as is. It’s interesting to note, that the SSL features you’re looking for in the config file aren’t present so you’ll need to add them manually to the file. Here’s what I’ve put in mine. Most of these commands by the way are covered in your man page for vsftpd.conf so take a look there for a more detailed explanation.

ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
rsa_cert_file=/etc/pki/tls/certs/myserver.mydomain.public.key
rsa_private_key_file=/etc/pki/tls/certs/myserver.mydomain.private.key

This essentially turns on SSL, turns off anonymous SSL, forces both data & logins to go over SSL and only allows TLS encryption. It also points to the location of your public & private keys.

This concluded part 1. Part 2 will talk about the implications of encrypting your FTP traffic and how it will mess w/ your FW rules at your border device.

-Q

Links I found useful while doing my install:
http://wiki.vpslink.com/Configuring_vsftpd_for_secure_connections_%28TLS/SSL/SFTP%29

http://en.wikipedia.org/wiki/FTPS

http://vsftpd.beasts.org/vsftpd_conf.html

This post is out of sequence but I told myself I’d throw it up while it’s still fresh. About a month ago I redid my entire email server from scratch when I started having some hardware issues. This post is on one of those bug-a-boos of switching from a physical server to a virtual world. TIME. Seems keeping vmguests synchronized with the correct time is an issue and wrecks havoc upon Dovecot (even when vmware tools are installed!) So, one of solutions was to not only install the vmware tools on my vmguest but to also install ntpd. This is good and bad…

  See, ntpd checks the time rather frequently and all it takes is a sync of > 75 seconds and Dovecot kills itself. Thankfully, it left a message in the log file.

dovecot: Time just moved backwards by 109 seconds. This might cause a lot of problems, so I’ll just kill myself now. http://wiki.dovecot.org/TimeMovedBackwards: 1 Time(s)

The URL provided in the error message was very illuminating and gave some recommendations from there. I installed the script that used lsof and was pretty much good to go. I did a couple of tests to ensure it wasn’t just restarting it once per minute then did the old crontab -e command and placed the script in the root’s cron jobs file stored at /var/spool/cron/root. Life is good. I’ve always been a big fan of netstat but lsof also gives some very valuable information on tracking down rogue issues.