The premise of Real Metal Hosting is simple.
Say goodbye to expensive + slow *Cloud + VPS Servers* (virtual machine instances).
Say hello to cheap + obscenely fast *Real Metal Servers*.
This is the process I suggest for my clients. My focus here is to create
an infrastructure that is decoupled from me. In other words, if a bus hits
me, then my clients will have to replace me + their business will still continue.
For the DIYers (Do It Yourself-ers), follow my checklist. You can also contract me
to implement my checklist on your site. Please be aware, my checklist requires you
run a dedicated server with complete root access via ssh. This is a must. No exceptions.
Machines Monthly machine leases run from $100 - $250ish. These machines
are capable of easily running 100K/hour uniques along with
1000s of individual sites/domains.
$ 497.00 External site audit - Speed + Security + SEOity.
$1250.00 Provision a new server machine + Apache SSL/HTTP1.1 Stack,
MariaDB (MySQL that works) + PHP + WordPress + working WP Caching.
$2500.00 Provision a new server machine + w/optimized Apache SSL HTTP/2.0 Stack.
Note: mod_spdy used, till true HTTP/2.0 available. Includes installing
SSL cert + testing for one domain.
$1000.00 10 Hour Prepaid Retainer ($100/hour) for Malware Cleansing, site migration,
Bricked Site recovery, Security Hardening, Blocking "High Value Attacks"
or any other Website related consulting or development.
$1000.00 Monthly retainer (per physical machine) to admin your machines,
track logs + continually retune your machines + block any new
High Value Attacks as they arise in the wild. This includes
normal admin functions like keeping software updated.
In almost every case I use OVH Infrastructure Servers, because in most cases I can provision one of these in a few minutes. Provisioning is the time from purchase to when the server is online.
OVH is has a few features that make them shine.
- Servers usually become available in 5-10 minutes, where other companies may take
hours to days.
- Servers come with zero software installed, which means someone competent has
partition the disk + install the operating system + install software services,
like Apache + PHP + MariaDB (MySQL that works).
- Servers come with built in rescue mode. This is incredibly important for
production servers. If a problem occurs in the middle of the night or on a
holiday, the client or admin person, can boot the server into rescue mode
in 5 minutes + have complete control of server repair.
Since the entire initial installation is under client control, initial installations
are always correct. No mistakes are ever
made by OVH personnel, because they have zero
to do with initial install.
Software + Content Tuning Checklist
Here's my software setup checklist...
- Operating System - Ubuntu. Implementing these options takes less than
5 minutes + reboot time.
- Install Ubuntu -
Ubuntu is my OS of choice because major upgrades usually work. In other words, when
a major upgrade releases, only one simple command line incantation is required to
upgrade the entire operating system + all software packages. Other Linux Distros
have around a 50/50 chance of surviving a major upgrade, where survive means the
machine will actually boot after the upgrade.
- Install Latest Stable Kernel -
One annoying quirk of OVH is their default
installed kernel (OS gut) is usually 3-4 years old, which is unacceptable, so
my first step is to install the latest stable
Ubuntu Mainline PPA Kernel.
- Optimize vm.swappiness - kernel parameter which tells the system when to
start swapping memory to disk. Here's how this works. When your machine's memory fills
up, the kernel swaps memory to disk. Then next time the disk saved memory
is accessed, the kernel swaps disk to memory. If this situation persists, the system
is said to be *thrashing*, which drains performance into the ground. This process is
controlled by the vm.swappiness setting. The default for this setting is 60, which
means swapping will occur before all of memory is used. Servers are best set to a
value of 1, which means an attempt will be made to completely utilize all memory, before
any swapping occurs. Never, ever, ever set this to 0, as that means no swapping will
ever occur which can cause a machine to become unstable + possibly crash.
- TCP/IP tuning is rarely required for post kernel-3.17 versions.
The kernel-3.18 release rolled in some substantial TCP/IP performance enhancements,
along with new tuning defaults. Currently I'm using
kernel-3.18.x+ versions, so no tuning is required. For pre kernel-3.18.0
(3.17 + lower) versions TCP/IP tuning is complex.
You're better off just using a kernel-3.18.x+ kernel.
- Enable Preloading which allows the OS to learn what software is
accessed most frequently, then arranges for all parts of this software to always
be present in memory. So instead of loading all the code for Apache + PHP + MariaDB
off disk into memory for every Web visitor, all this code persists in memory,
so new visitors are served content at lightning speed.
- File Systems - which set the speed of your entire content serving process.
Implementing these options takes 30 seconds + reboot time.
- /tmp in Memory - one of the primary determining factors for the entire
speed of your server is the /tmp filesystem, which is where temporary files are
stored. Here's why...
- PHP Session Cookies - these files live on /tmp + provide access control
between pages. All users get a session cookie, whether they're logged in or
guests (for most modern CMS systems). When visitor clicks a link to move
from their current page to the next page, the session cookie must be accessed
+ checked to ensure the visitor has access to their target page
(new page they just clicked). If these session files live on a disk, newly
clicked pages are returned to visitors at disk speed, as session files have
to be pulled of disk first. If /tmp is in memory, page transitions run at
memory speed, rather than disk speed.
- MariaDB/MySQL Temp Files - When an SQL SELECT is done, records are
read out of the database into memory buffers. When selected records exceed
memory buffers, then temporary files are written into /tmp so this is similar
to PHP session data. If /tmp is on disk massive performance drain occurs,
so every database access can produce 1000s of disk accesses. Moving /tmp
to memory ensures your database subsystem runs a memory speed, rather
than disk speed.
- Main Filesystems Tuned - this is fairly simple + dramatically reduces
the number of overall file accesses required to retrieve a files from disk +
serve it through Apache to a visitor's browser. The primary options here to
set are Filesystem type of ext4 or btrfs with the options
which turn off rarely required filesystem accounting.
- Database Tuning
- Replace MySQL with MariaDB - which usually gives a 30%-50%+ increase in
Apache/PHP/Mariadb pipeline performance, especially
http://WebPageTest.org time to first byte
which is an essential indication of how well your content will convert.
MariaDB is simply a working version of MySQL, which means
years of bug fixes + performance enhancements for MySQL have
been rolled into MariaDB. Also MariaDB is a MySQL
drop in replacement.
- Initial MariaDB Tuning which means dropping in place a highly modified
MariaDB configuration file. Doing this before any data is imported
into the database subsystem is essential to a smooth running system. Deferring
this into the future may require hours of downtime, so best to do this at
initial OS installation.
- Realtime MariaDB Tuning which is fairly simple. This requires running
MySQLTuner ever 24 hours + retuning
MariaDB, based on MySQLTuner
- Webserver Tuning
- Webserver Selection which is likely best to be Apache. Here's why.
- Webserver Performance shows Apache to be faster than Nginx. I recently
did a test using Apache-2.4.10 + PHP-5.5.12 vs. Nginx-1.6.2 which showed
Apache (with default tuning/config files) to be around 10% faster than Nginx.
- Apache Tuning which requires zero time for Apache for most
sites. In rare cases, Apache requires tuning. For example, I have one
client serving 100K Unique Visitors/hour. This install required some
slight Apache + MariaDB tuning to ensure all visitors see content.
- PHP Setup Simplicity which is the time required to setup PHP. With
Apache, you just install one software package + PHP runs as a part of Apache.
Nginx requires using an external PHP manager like
PHP FPM, which is just one more thing to
keep up with, for no good reason.
- .htaccess management is completely missing from Nginx. This means
installing WordPress + Wordpress plugins requires an WordPress Wizard
be paid to convert any .htaccess rules into the Nginx equivalent
rules. This is great for consultants pitching you their services + only do this
if you've got money to burn.
- PHP Tuning
- PHP Config File dropping in a custom php.ini file is first thing I do,
to ensure PHP works under high traffic load + normal uploading of themes + plugins,
as the PHP defaults are far to low for most normal or high traffic sites.
- OpCache Installation is the replacement for the old APC subsystem. Most
Linux Distros fail to install this, so OpCache must be installed + configured
- OpCache Realtime Tuning is fairly simple. Using a tool like
OpCache Control Panel,
monitor OpCache statistics, tuning so both key lookup caching +
data object caching is running at no more than 50% utilization.
- XDebug Installation which is a tool used for all sorts of PHP debugging.
There is one config option that's is an absolute requirement to enable -
xdebug.scream=1 - which enables printing all PHP errors + warnings. More about
this in the WordPress Tuning (next) section.
- XHProf Installation which is a tool used to quickly profile (identify)
speed bottlenecks in your Apache/PHP/MariaDB/CMS pipeline. At a single glance,
you can tell exactly where your site is slowing down. Data is presented in
callgraph format. This is similar to a infographic, providing instant
identification of slow spots.
- WordPress Tuning
- Zero-Config Caching Plugin which I suggest to be
Quick Cache out of the
http://WordPress.org plugin repository or
Quick Cache Pro from the Developer's site. (NOTE: This plugin's name
will soon change to be ZenCache.) This plugin runs circles around
all the other caches including - W3TC + WP Super Cache. As a bonus
Quick Cache has one primary
tuning option Enable/Disable whereas you have to be very smart + willing to
invest hours into tuning W3TC + even after hours of tuning, you'll only get
a fraction of the speed
Quick Cache will give you.
Also Quick Cache works
correctly for Membership Sites with no additional config settings.
- Backups are additional to tuning + are required for production sites.
RULE: You only have a real backup, when you can migrate your site to another
machine + have it running in 5-10 minutes. Most people just make backups +
somehow expect they are correct. The most effective backup is to run LXC
on your machine + every night shutdown Apache + MaridDB on your production
LXC Instance + take a backup + restart your LXC Instance. Using this
approach you can move your LXC Snapshot + reboot your entire system anywhere
LXC is running. Alternatively you can backup your your databases + filesystem
files + rsync this data to a remote machine.
- Enable WordPress Debugging is best turned on + left on all the time.
Any errors or warnings, whether you have debug on or off, take exactly the same
time to process. If your site has errors or warnings, the entire PHP logic to
catch + report these is run, independent of you debug settings. After extensive
testing, there is no performance penalty for this, as disk writes are buffered
through memory, so logging this information takes very little system resources.
- Track WordPress Debug Data is best done daily. I've seen some sites
that generate 30Meg of debug data for every rendered page. This is crazy. If you'd
like a slow WordPress site, install a crappy theme + crappy plugins that
generate massive amounts of debug data for each page render.
- Change Default wp-debug.log Location which is necessary to avoid new
hacker exploit vectors. If you check your Apache logs, likely your see hacker
http://YourSite.com/wp-content/debug.log downloads occur at regular
intervals. What hackers are looking for are code signatures. What they do
is upload code to places like ThemeForest.net
(the worst offender) which contains exploit vectors - ways to hack your site.
What these hackers to is place specify types of errors, with unique error or warning
signatures in their code, so they can scrape wp-content/wp-debug.log via
their own spider or out of a
http://CommonCrawl.org/ data set + they
instantly know how to hack your site. If they use
http://CommonCrawl.org/ data, they're
completely cloaked. No way to tell where the attack is coming from.
- WordPress Code Vetting means only installing exploit vector free code -
themes + plugins. Here's how you do this...
- The Simple Way is only install code from
http://WordPress.org which has a brutal
code vetting process.
- Vetting Themes you can do one of two ways.
- Search http://ThemeCheck.org by
type this into google - site:ThemeCheck.org name-of-theme-to-check.
- Vet Themes by installing the
WordPress Theme Checker
then check any newly installed theme before activation.
What you're looking for are diagnostics related to
base64_decode() + eval() which are the real ugly ones, guaranteed
to give hackers access to your site. There are other exploit signatures + these
are the two big ones. Having a theme active with known exploit vectors
guarantees your site will eventually be hacked.
- Vet Plugins using WPscan which you
can either install or google for an online version. What you do is first
run WPscan for a baseline, then
install + activate your desired plugin + run
WPscan again. You're looking for diagnostics
related to exploit vectors. Having a plugin active with known exploit vectors
guarantees your site will eventually be hacked.
- Blocking Site Attacks which includes DDOS Attacks +
Brute Force Dictionary Login Attacks + High Value Attacks.
- Never Use WordPress Plugins for security. This includes plugins
like WordFence + Sucuri + BulletProof Security. These
plugins are best avoided because for them to work requires all attacks run
through the entire Apache/PHP/MariaDB/WordPress pipeline, which can drain
performance into the ground in a manner of minutes or seconds. Instead use
OS blocking as follows...
- Fail2Ban is one of the best tools for blocking all brute force
login attacks - ssh + ftp + WordPress. Since Fail2Ban
runs at the OS level, rather than WordPress level, the first few attack go through
the Apache/PHP/MariaDB/WordPress pipeline + once an attack is sensed
Apache's ports (80 + 443 or whatever ports are configured) are blocked at
the kernel, for some period of time. I usually setup this value as one hour.
- Close Wait Reaper is some simple custom code I wrote to fix the majority
of High Value Attacks which are attacks that are impossible to differentiate
from normal traffic. What occurs with these, is 1000s of browser simulations are
run from separate IPs against a site. Since these simulations appear exactly as
normal traffic they drain resources. A peculiarity of the normal TCP/IP stack is
sockets (connections) can be left in what's call the CLOSE_WAIT state for
many minutes. Currently there is no kernel tuning parameter to catch these + close
the quickly. As CLOSE_WAIT connections build up, memory is consumed causing
disk swapping. Once this situation occurs, it's only a matter of time till visitors
see errors instead of content + then the entire server (machine) crashes. My
custom code is simple. It scans for any connections left in a CLOSE_WAIT
state for some period of time (usually 10 seconds) + manually releases all
OS resources (memory) associated with these connections.
- SSL Site Wrapping is likely another way to block High Value Attacks
since many attack bots can only initiate normal (non-SSL) connections, likely this
is another great way to block attacks. I've only recently started SSL site wrapping +
initial testing appears to indicate most High Value Attacks stop instantly, once
a site is SSL wrapped. NOTE: SSL Wrapping is complex. Refer to next section for
more about this.
- Apache 304 Tracking which indicates a resource/object
(image or .js or .css or .pdf or .doc file) has been downloaded as unmodified.
Real Browsers (non-Bots) download a file + check the expires header tag. Then
Real Browsers only attempt to download a new copy, once this expires tag passes.
So if you set a future date of 10 years for a file, say jquery-2.1.3 + one IP
address downloads that file over + over, then likely your site is the target of a
High Value Attack. Enter Fail2Ban which requires only a simple one
line rule to track the combo of IP + file + 304 code, then if many of these occur
over a short period (5/minute is what I normally use) then block the Apache
ports for this IP address.
from bot traffic. Bots are usually coded very simplistically. They usually have
code that should fire within a few seconds of each visit. If this code never fires,
then likely you have a Bot attacking. This code is a bit complex to implement +
at this point is likely overkill. In the future as Bot intelligence evolves, this
type of code may be required. On thing, this code will also have to be written to
be smart enough to differentiate between normal traffic, which can be done + is
fairly complex, so is left as an exercise for cold + rainy nights.
- wp-content/wp-debug.log Tracking which is a great way to catch evil Bots.
What you do
is arrange for Google to index a bogus wp-content.log file
(by allowing GoogleBot IP addrs to access this file) + then any other IP that
attempts to download this file, use Fail2Ban to block the attack.
- SSL Site Wrapping which means wrapping an entire site, all content +
images + .js + .css files inside SSL. After running all sorts of tests, I
can say SSL sites run almost as fast as normal sites. Usually 95%+ the speed of
a normal site + block many High Value Attacks + provide
SSL Google Signal For SEO. An all around win + for this to work SSL
must be configured correctly, which is no small feat. SSL is in massive flux
right now as mod_spdy is being deprecated + replaced by the new
HTTP/2.0 spec. As such you'll have to refer to the current crop of
SSL tuning documentation for you SSL setup. You know your SSL is setup correctly
when you run your site through
https://www.ssllabs.com/ssltest + get
the following output...
- Overall A+ Score
- Detail Score of at least 100/90/90/100
- Downgrade Attack Protection - This server supports TLS_FALLBACK_SCSV to prevent protocol downgrade attacks.
- Heartbleed Protection must be on/yes.
- BEAST Protection no longer required as this fix has been rolled into all
- TLS Compression must be no/off.
- Next Protocol Negotiation (NPN) is best to be on + likely this will
be widely available sometime in mid to late 2015, as this is a function
of the OpenSSL library installed on servers + NPN has
only been implemented in very recent versions.
- Session Management both Resumption + Tickets should show
as on/yes. Both these on together give a huge performance increase.
- OCSP stapling must be on/yes, which gives a huge performance increase.
- Content Conversion Optimization
- Never Use a CDN as they only add complexity + slow down well tuned sites.
The only exception might be if you're streaming HD movies to clients
all over the world.
- Validate Your HTML by running your site pages through
http://validator.w3.org + verify you have...
- 0 HTML errors
- 1 HTML warning "Using experimental feature: HTML5 Conformance Checker"
to indicate your site is using modern HTML5, rather than old HTML versions.
This is required for fastest rendering in browsers.
- 1 HTML warning "No Character encoding declared at document level"
which is required for pages to render quickly in old Internet Explorer Browsers.
- Validate Your CSS by running your site pages through
http://jigsaw.w3.org/css-validator/ + verify you have...
- Preferably 0 errors
- Preferably 0 warnings
- Most Important is to avoid warnings related to foreground + background
colors being the same or similar. Google treats this type of error/warning as
a site that is likely to be keyword spamming + will penalize your site for this.
If you have 100s of these errors, you're site will likely be heavily penalized.
- Speed Test using
- Mobile Responsive Test using
Chris Pedderick's Browser Plugin or the http://Responsinator.com Site.
- Mobile Speed Test using
http://WebPageTest.org - same all "A Scores" using
the iOS test instance, to test iPhone site speed.