Tag Archive for WebPageTest

“Grunt” your way to frontend performance optimization

Performance optimization has been more than ever in the spotlight of web developers, especially for mobile web developers who have to understand and know by heart the challenges and constraints of mobile devices: these devices run off a battery that e.g. drains faster if performance is not taken seriously. The devices are powered by smaller CPUs than desktop devices. Unknown factors like latency and network connectivity challenge developers to build slim, light-weight and fast websites. Data plans still remain expensive and inconsiderate use of served data by web developers should not be ignored.

Clearly, performance is (and should) not (be) an after-thought anymore. When web developers create websites, performance can influence the success or failure of a web product. We’ve been hearing from leading performance advocates like Ilya Grigorik that speed is a feature and should not only be thought of just before a product hits production but rather as an essential part of the web product development cycle.

For example, instead of minifying and concatenating CSS and JavaScript files manually, tools and processes are out there that can help and put these performance tasks into an automated workflow, and more importantly right from the beginning of a product development cycle.

I’ve been using Maven to run most of the automated performance optimization at work, however I’ve been always interested in using Grunt for the same purpose. Grunt is a task runner, created for web products, based on JavaScript that can be leveraged to make performance part of the deployment process.

In today’s blog post I will be sharing some of the plugins for Grunt that can be used to speed up and automate performance optimization. At the end of the blog post, I will present performance results that will show that frontend optimization (FEO) can be fun and easily be automated to cut page load time.

Google’s “Make the Web Fast” team recommends frontend best practices as well as Steve Souder’s “High Performance Web Sites” outlines rules that can be applied for FEO. I decided to pick two of Steve’s rules “Make Fewer HTTP Requests” and “Minify JavaScript” (and CSS, HTML) by using Grunt plugins that can help automate those specific rules. So here it goes.

Note: The post assumes that you’ve worked with Grunt before and know how it’s been installed, and how to install plugins (I won’t go into details, however links at the bottom will help you)

“Let’s grunt it up”

(All plugin headings in this post are clickable links to their appropriate pages)

grunt-montage

Montage helps you sprite images to reduce HTTP requests. You will need ImageMagick to be installed. Alternatively, you can also try out grunt-spritefiles.

grunt-usemin

This plugin is useful when you want to develop and debug a version of your site that doesn’t use the minified and concatenated version of your JavaScript or CSS files. A comment blocks is wrapped around your JavaScript and CSS assets that will be concatenated to your destination after deployment.

<!-- build:js js/magic.min.js -->
<script src="js/1.js"></script>
<script src="js/2.js"></script>
<script src="js/3.js"></script>
<script src="js/4.js"></script>
<!-- endbuild -->

will become

<script src="js/magic.min.js"></script>

You can use grunt-processhtml instead.

grunt-closure-compiler

Alternatively you could use grunt-closure-compiler instead of combining concat and uglify and cssmin for JavaScript and CSS files.

grunt-contrib-uglify

Uglify and concat go almost hand in hand and should be used together, the concat plugin first makes sure to combine all defined JavaScript files. Uglify only works on JavaScript files. It minifies all code in a one line block of code. Use cssmin for CSS files.

grunt-contrib-cssmin

Same logic and idea than uglify, once your CSS files are all concatenated, use cssmin to shrink several lines of CSS code into one single one.

grunt-contrib-concat

Combine CSS and JavaScript files with this plugin, it allows you to reduce your HTTP requests for each and every file to just one combined file.

grunt-contrib-imagemin

imagemin minifies JPG and PNG images. It’s a handy Grunt plugin if you don’t know if the assets you got handed from your designer (or yourself) are optimized for web yet. By using this tool, you have the piece of mind that you use image files in an efficient way. Alternatively, you could use grunt-smushit, it’s based on Yahoo’s great smushit tool that is available in the YSlow plugin for several browsers.

grunt-image-embed

This plugin encodes images as base64 and leverages the technique of data URIs for images, something that can be used inline with CSS to reduce HTTP requests and hence reduce page load time. I’ve written a blog post where this is explained in more detail.

grunt-htmlcompressor

This plugin is using htmlcompressor tool to minify and compress HTML files. The options parameter is handy to tweak your compression, my example uses compressJS and preserveServerScript to also compress inline JavaScript and server script tags in case I wanted to include some SSI code.

spofcheck

Use grunt-exec to run the SPOFcheck. An excellent tool to identify bad 3rd party scripts includes, developed by the eBay team. I didn’t include the scripts asynchronously, hence SPOFcheck complaints to avoid SPOF.

“It’s Magic” Sample Page

I created a simple page themed “It’s magic” where I applied all mentioned plugins. You can find the files including Gruntfile.js here. Please note, I intentionally didn’t put a lot of effort into the styling (It is supposed to look as simple and cheesy as it feels to you)

In a nutshell, the page has a logo, uses JQuery from the Google CDN, includes a simple JQuery gallery with previous and next buttons. Simple JavaScript and CSS files are being used.

“without/magic.html”

The logo is a png logo, the images are not optimized. There are several CSS and JavaScript files that are all individual being included, not minified nor concatenated.

“with/magic.html”

This file is the one that Grunt will create for you. Visually, the file doesn’t look that different, besides the fact that the title has changed….see yourself

Below are screenshots of the two pages (and links) side by side, the one on the left before Grunt tasks were applied. The right one shows the page after Grunt tasks were applied. For the user they both look the same (except for the heading).

without/magic.html with/magic.html
No magic here!

Can you spot the differences?

  1. The logo was being transformed into a data URI
  2. The title has changed from It’s not magic to It’s magic
  3. The local CSS and JavaScript files were being minified and concatenated
  4. The HTML was being compressed, comments were taken out automatically
  5. The next and previous buttons were converted to a sprite file
  6. On build, SPOFcheck was applied and gave us the following warnings so we could address possible SPOF issues

Screen shot 2013-08-02 at 7.25.05 PM

Let’s take a look under the hood

Here are the waterfalls for both versions:

Without magic
notmagic-waterfall

With magicmagic

WebpageTest Results

I ran WebpageTest for both files with 9 runs for IE8 with a DSL connection to retrieve the median. Here are the performance results:

  1. Without magic results
  2. With magic results
  • HTTP requests dropped by ~48%
  • Page load time (PLT) dropped by ~10%
  • File sizes dropped by ~10%

Even if those numbers are not that high (mostly due to the simplicity of the experiment), it shows that Grunt can help you automatically optimize your deployment process.

As you can tell by the sample code, there are many mix and match options available, depending on the magnitude and granularity of your page structure. Nevertheless, this little sample shows how to use Grunt to optimize performance and to illustrate what is possible. Feel free to use the code as a starting point, and tweak or customize it to your likings.

General references and info to get you started with Grunt

Setup your own HTTP Archive to track and query your site trends

Be honest, ever wanted to play “Steve Souders” for a day and pull some cool stats or trends about some web sites of your choice? Well, how about setting up your own HTTP Archive then?

Httparchive.org is an excellent tool to track, monitor and review how the web is built. You can dig into trends around page size, page load time, CDN usage, distribution of different mimetypes and many other stats.

You can download an HTTP Archive MySQL dump and the source code from the download page and play around yourself with the current data. For example, do what Stoyan Stevanov did by asking yourself some questions: “Hm, I wonder what are common mime types these days”. Once you’ve setup the database, you can easily query anything you want.

However, what I personally find the most intriguing and fun is applying all of this to sites of of your choice. Alright, let’s break this down: if you’re famous and your site is listed under the Top * Alexa sites, then you can use the official dump, if your target site is a wee bit less famous and not part of any of the crawled sites, you might want to start using your own database and local instance of HTTP Archive. That way, you can run this handy tool on any of the web sites you want to test.

Things to consider before you get started

You need MySQL, PHP and your own webserver running. If you choose to run your own private instance of WebPagetest, you won’t have to request an API key. I decided to ask Patrick Meenan (pmeenanATwebpagetestDOTorg) for an API key with limited query access. That’suffcient for me for now, if I ever wanted to use more WebPagetest runs per day, I’d probably want to setup a private instance of WebPagetest. I’ve done this before but my computer had to be replaced, and I haven’t had the time after to set this up again.

Sample setup

bulktest: That’s the folder you really want to understand and work with when setting up your own little HTTP Archive baby.

  • bulktest/README.txt: This file gives you a general intro on how to use the folder, I recommend to read this.
  • bulktest/bootstrap.inc: In case you choose to us a private API key for WebPagetest, you will need to update this file with the provided key

To run a nice little batch, you want to execute the following scripts after each other via CLI (default setup for security reason)

  • bulktest/batch_start.php: This script takes pre-defined list of URLs (importurls.php) that you can specify or change. By default it’s downloading the latest  Alexa list (downloadAlexList()) and imports those into the urls table. I’ve changed this so it’s picking up my own csv file with the urls I want to crawl but you can also customize this the way you want it (default setup needs to run via CLI)
  • bulktest/batch_process.php: Run this as often after each other until you get confirmation that your runs were successfully recorded (default setup needs to run via CLI)

Batch summaryIt always gives you a nice batch summary at the end so you know where you are at (see screenshot)

  • bulktest/statscompute.php:  This is needed for the rendering of the stats under your local URL, e.g. http://localhost/httparchive (default setup needs to run via CLI)

More detailed steps on how to install HTTP Archive can be found under the blog post Setting up HttpArchive private instance. As suggested by the README.txt file and this blog post, it’s probably useful if you setup cronjobs in your environment to automate the batch steps.

Front-end piece: Visualizing your trends and stats by filling the charts

Congrats! Assuming you’ve successfully setup your own HTTP Archive instance  – wasn’t that fun and the bit of pain worthwhile? Now you can start viewing those charts and investigating trends and stats targeted to your defined URLs. The beautiful thing about having your own instance is that you can be your own master of data visualization: you can now create additional charts beside the ones that came out of the box provided by the default HTTP Archive setup.

And if you don’t like Google chart tools, you may even want to check out d3 or Highcharts to use instead.

From now on, the sky is the limit. Nobody can stop you now, my friend – you can even run some kick-ass raw database queries if you don’t really care much about the front-end visualization (I do ;))

Back-end piece: Querying the database directly

Sometimes, you want to get some questions answered without creating a pie or a chart. That’s when you can make use of the MySQL tables directly that have been setup for you (via schema sql file and filed by your batches).

Let’s run a simple query on the requests table.

For example, some of our sites use YUI, some use JQuery – but we would really like to avoid having pages serve both.

A simple sample query like the one below could help identify those sites:

select req_referer from requests where url like '%/i/l/yui%' or url like '%jquery-%.js' group by req_referer

Be prepared, some setup time is required

I’m not going to lie, it took me some after-work evenings, many debug statements via PHP to set everything up so I could run proper batch_start and a proper batch_process commands and fill in those pies and trends. Here are a few things to watch out for

  • I hadn’t installed pcntl with my PHP version, so I needed to set this up first. You will need this to run batch_process.php. Installing pcntl for php on osx Lion blog post helped a lot – Thank you Jacob!
  • You might have to adjust some of the tables values, I got lots of mysql insert/update errors regarding default values not set for certain fields
  • After I received my API key, I still had to change the WebPageTest URL after Patrick pointed me to the correct URL (Thanks again). The $gWPTUrl variable was set to http://httparchive.webpagetest.org as default instead of http://webpagetest.org in settings.inc
  • Some of the tables require you to have a temp/dev version of it as well, e.g. requests, urls, statsdev etc. Some of the scripts look for e.g. requestsdev. You might have to copy a few of the original tables during setup process. You can setup the naming convention in dbapi.inc

Next Steps

Where am I at? Well, I just finished a few successful batch_processes on a selected number of URLs. It’s fun to monitor trends based on the URLs I put in. I will be collecting more ideas and uses cases over the next few months, possibly also adding some more charts applicable to my needs.

If time permits, I hope to be sharing more of my customization and use cases of HTTP Archive at this year’s Velocity conference in San Jose. And if not there, I’ll try to update this blog post as best as possible. If you have any questions or suggestions, please leave a comment below.

I am always happy to receive feedback or happy help out with some roadblocks while setting this all up.

Thanks

to….