Category Archives: Server

Certbot and apache

I promised a blog post detailing changes I needed to make to my apache config in order to move BRBcoffee to Https, but in hindsight there isn’t much to write about it, it’s basically just a refactor.

Certbot, the tool from EFF (written in Python, yay!) that gets ssl certs from Let’s Encrypt, doesn’t work with monolithic conf files with multiple hosts. I run all my projects on the same server, routing traffic based on the site address using apache’s VirtualHost directive. It used to look like this:

<VirtualHost *:80>
    DocumentRoot "/var/www/blog"
    ServerName blog.brbcoffee.com
    # Some more directives
</VirtualHost>
<VirtualHost *:80>
    DocumentRoot "/var/www/archive"
    ServerName archive.brbcoffee.com
    # Some more directives
</VirtualHost>
<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyRequests Off
    ServerName cv.brbcoffee.com
    ProxyPass / http://localhost:5000/
    ProxyPassReverse / http://localhost:5000
</VirtualHost>

So what you need to do is rip all of that out, you don’t want it in there. In their place you want this:

IncludeOptional conf.d/*.conf

Depending on your packager, it may be that this directive is already somewhere in your httpd.conf file. If it is, great, just leave it be. After that you want to take each VirtualHost that you ripped out of the main httpd.conf, and place them in individual files, like so:

<VirtualHost *:80>
    DocumentRoot "/var/www/blog"
    ServerName blog.brbcoffee.com
    # Some more directives
</VirtualHost>

blog.conf

The configuration doesn’t change, it just needs to be in separate file for certbot to be able to do its thing. You see, after certbot goes out and gets your certificates it needs to add some rules to each vhost for redirecting traffic to ssl, which I guess they didn’t want to write a lot of ugly parsing code to be able to do in a program that really isn’t about that (although it should be trivial with BeautifulSoup.

Anyway, before, running certbot –apache probably didn’t work, it got the certs for you, but couldn’t edit your conf to do what we want. Now, when you run it, it’ll complete just fine. If you chose to set all traffic to go to https, it will add three redirect lines to your conf files, and it will create a new file as well, in my case, blog-le-ssl.conf. It’s identical to the old conf file, except that it is on port 443, and that it checks that mod_ssl is loaded. All of this is stuff we could have done ourselves, of course, but it’s a convenience thing.

So that’s all there is to it. Refactor your httpd.conf by separating each virtualhost into a different file, and run certbot –apache again.

Https

Welcome to Https BRBcoffee, and thank you to Let’s Encrypt and certbot for making it a breeze, mostly.
Tomorrow there will be a blog post up about the changes I needed to make to my Apache configuration in order to get certbot to play nice.

For now, glory in that green padlock! And don’t go to the archive if you want it to stay green. WordPress doesn’t automatically update image links, so I’ll need to fix that at some point. The CV though I had no trouble with, even though it’s a Flask app that Apache just redirects traffic to. Good job, Apache, good job, python!

Configuring a linux firewall

So you’ve got your Linux server going, it’s configured the way you want it , and everything is coming up roses. You try to ping it, but the server doesn’t seem to exist. You’ve been blocked by some of the best/most insane firewall in the galaxy: iptables. A firewall’s job is to block unwanted traffic, and iptables takes its job seriously. By default, it. drops. everything. Got a http request incoming? Psh, drop that packet. I don’t care if you’ve got apache running. FTP request? Same story. Mysql? Nope.

A cat shoving things off a desk
This is iptables

Ssh is usually fine though, so we can log in and edit the rules. Iptables rules are added to rule chains. The only chain we’re interested in is the INPUT chain for now; We want to be able to receive http requests to our server, ssh connections, and nothing else. We’ll also want to allow existing connections to persist. These are the switches we’ll be using (you can find all these in the manpages, of course, but some are in the iptables-extensions manpage).

  • -F flushes the rulechains. This means exactly what you’d think.
  • -A [chain] adds a rule to the specified chain.
  • -I [chain] [number] same as -A but inserts rule at a given point in the chain.
  • -i [interface] specifies the interface the rule will act on. If you don’t specify this, the rule will act on all interfaces, including loopback (more on this later).
  • -p [protocol] specifies whether the rule is for tcp, udp, or whathaveyou.
  • --dport [port] further narrows down the packets to look at by checking which port they’re headed for.
  • -m [match] this is an extension that looks at the type of traffic the packet belongs to. We use it with:
  • --state [state], which asks a different module called conntrack whether the connection is INVALID, NEW, ESTABLISHED, RELATED, or UNTRACKED. This is magic, I have only a vague understanding of how it works.
  • -j [policy] says whether to accept or drop the packet.

Alright, let’s get to it. You can think of iptables as a sieve, where every rule along the way checks out a packet and decides whether to keep it or discard it. If the rule doesn’t match the packet, it moves further down the sieve to be checked by other rules. Therefore, at the end of it, if the packet doesn’t match any of our rules, we will just discard it. A good policy for internet traffic is that if you don’t know what it is, don’t touch it. Every rule we add gets added last in the chain/sieve.

A script demonstrating the use of iptables

And that’s it. We’ve configured our firewall. It will reset every time you reboot your server, but that isn’t often. I just keep a script like the one above to reconfigure it. You can get NetworkManager to configure it for you on boot, but I don’t really see the point unless you reboot your server all the time, which, I mean, why would you do such a thing?

Plugin rot and how to do regex replaces in mysql

I’ve spent a few hours over the past week restoring my old blog, which can now be reached at brbcoffee.com/archive/.  This will be the final chapter in that saga.

My old blog is sort of old. It started out on MySpace, moved to Google’s BlogSpot platform, then moved to Tumblr for a while because I got some new authors on board. Finally we moved to a WordPress solution.  From there on BRBcoffee has been on WordPress, but it’s moved around hosts like an alcoholic goes from bar to bar trying to get his fix. I’ve employed a bunch of plugins to do various things for me, from importing and exporting posts to providing language switching capabilities to advanced caching to image scaling to… you get the picture. This is all fine and dandy, but it does pose a few problems when you move around like I do.

  1. The image scaling edits your wp_posts database table, naturally. When you move, the scaled images don’t necessarily come with, resulting in a lot of broken links.
  2. The translation plugin just has it’s own tags in that same database table, and keeps two posts in one field. When the plugin isn’t there anymore, or isn’t compatible with your WordPress version, the tags become visible on the site, and you get two languages at once.
  3. Plugins are often developed by lone wolf devs, and they are never up to date when you check up on them years later.

So, we need to clean up the mess they’ve created, and we don’t install Yet Another Plugin™ to try to fix the problem. We need to dig into our database and fix things ourselves. There’s only one problem: Mysql is made for storing, finding, updating, removing, etc. It isn’t the tool for the job when you need to do complex things like searching for a pattern and replacing only the part of the string that matches that pattern. It can be done, I’m sure, but why would you do it when there are better tools for the job? Here’s how I do it:

  1. Export the table you need to edit with
    mysqldump -u USER -p DATABASE TABLE > dumpfile.sql
  2. Do work on the dumpfile with specialized tools like sed, awk, or even vim when in a pinch.
  3. Empty the table before reimporting the modified dump with
    mysql -u USER -p DATABASE -e "delete from TABLE;"
  4. Finally reimport the table with
    mysql -u USER -p DATABASE < dumpfile.sql

Mysql also comes with a tool called mysqlimport, but I had a problem with it, so I just used the standard mysql command in the end.  So what do those switches mean?

  • -p means this user has a password, ask me for it before doing anything
  • -u USER means use this username. Pretty obvious.
  • -e means execute. Execute the following quoted code.

The only weird thing is that -u takes an argument, but -p doesn’t. It looks like DATABASE should be the argument to -p, but it isn’t. DATABASE is simply the database you want to do work on. In my case, it was called “archive,” while yours might be “blog,” “sales_records,” or even “naughty_movies.” That’s commandline switches for you. In my opinion, a single dash should be reserved for switches, double for arguments, like so:

  • -p [ask me for a password]
  • --user <name> [I'm logging in as name, how quaint]

Oh well, you get used to it. That’s how you should do many complex updates on a migrated database. This isn’t for when you want do do one small change on a live system. The 3rd step removes all records from the specified table. Your system will practically be offline until you’ve reimported the changed data.

If you wish this were simpler, you’re not alone. Behavior like this has been a feature request in mysql since 2007. Go there and click affects me, and maybe in another 7 years we’ll have it built in. Till then, use the method described above, and for heaven’s sake, don’t let the plugin rot get your blog too. If you can do it without a plugin, do it without a plugin.