Last year I began work on a Windows based application for monitoring the requests that are made by a computer over HTTP and then apply a rules engine to modify those requests before being send to the server. It is also able to modify the responses before being passed on to the web browser. Based on FiddlerCore and WPF, the application sits in the system tray displaying information on the recent HTTP requests and any rules that have been applied. Rules can be enabled and disabled via the user interface and customised by the use of a simple XML file.

The application sets itself as the system proxy, so all browsers that are set to use the system proxy will start issuing requests via the application. A word of warning, if the process is forcefully closed it will not have an opportunity to clean up, this may result in all web connections being blocked, as the system proxy will not be set to an address that is listening for requests. A quick fix is to restart the application and close it cleanly via the menu.

Screenshots

Log of matched rules and URL that triggered the rule

A list of the requests that have been issued by the computer

A list of the rules and that status of each rule

Provided rules

• Block request – Each request can be blocked from being sent to the server.
• AdBlock Plus implementation – A cut down version of the AdBlock Plus for blocking advertising and tracking content. See below for more information.
• Https Everywhere implementation - Send requests over HTTPS instead of HTTP for popular sites.
• Python script – Run custom code for each request. This is provided to give extensible functionality.
• Modify header – Modify a header before it is sent to the server or returned to the browser, including removing it.
• Modify cookie – Similar to the modify header rule, with support for individual cookie values.
• Break action – Prevent any other rules from running for the request or response.
• Save file – Save request content to the disk if matching a pattern.
• Age filter – Implementation of my proposed header for restricting content that is inappropriate for minors.

AdBlock Plus rule

Included is a version of AdBlock Plus that can potentially give support to Internet Explorer, as well as other applications that issue HTTP requests via the system proxy. Since it does not run inside a browser only rules that are based on URL patterns are supported, all requests that work by hiding HTML elements will not run.

I’m not advocating the use of an ad blocker, as most websites are funded by advertising revenue, resulting in much of what we read on the internet being ‘free’. This was built mainly as a technical exercise for me to see any false positives that are preventing my pages from rendering properly when running the real AdBlock extension. It is extremely useful in that role, so I thought other web developers would benefit too.

I also built this as a way to write my own basic regular expression implementation, and as such it may not perform as well as a version based on the optimised Regex classes within the .NET framework. I was curious to see how hard it would be to write a simple regular expression parser and matcher, and the opportunity presented it quite nicely in the form of the AdBlock Plus rules engine.

The Firefox plugin, on which this is based, converts each rule to a regular expression, then runs it using the optimised engine within Firefox. I wanted to circumvent this step and see if I could directly parse and interpret each rule, as this would give me an insight into how a regular expression engine works. While I am very pleased with the results, it is still not up to the performance of the standard regular expression engine, and all the optimisations that have been added over the years. I may work on optimising my engine in the future, but only if time permits.

Performance

To put it bluntly, this will not speed up your system, quite the opposite in fact. On my Core i5 system the overhead is not great, however my Core 2 Duo laptop has a noticeable delay before requests are sent. Generally, the large sets of URL patterns by the AdBlock Plus rule are the culprit, so if performance is an issue try disabling that first. Additionally the Https Everywhere rule will cause a significant slowdown on sites that are forced to use HTTPS, due to the additional overhead of using a secure connection.

Download

I have just added the application to GitHub as an open source project. You can download the installer here.

Starting with this, I wrote an engine that finds related names, using common surnames as a base. Each name is calculated to find all the related names, then ranks them using 4 separate methods. The main problem that I faced what that the most popular names are the most related names for everyone – most people seem to have a Michael in their family. Each ranking method tries to find the names that are more common in the related name set compared to the global list of names, which are hence the most related.

I found that the results were initially a little bit inconsistent, for example the name Brad would be related to Brett, however the inverse wasn’t true. So counter this, I added a reverse lookup table and factored that into the ranking alorithm, which helped clean up the data and remove the odd combinations. This ensures that both names need to be related to each other, and removes names that are only related in one direction.

The results were better than I had hoped for, with over 25,000 of the most common names being computed and stored. I can see there are clear trends amongst regions, religions, similar sounding names, and even real world items, such as colours, fruits and emotions. I hope someone finds this useful, especially for trying to find odd or obscure baby names.

http://www.namedrop.it

Perhaps my favourite change is that now any of the free movies or TV icons have a ‘Free’ overlay applied to them automatically. This is great at calling out how much free content we have, which is actually quite a lot now (loving the snowboarding titles we have)! We also are using the same system for showing off trailers, in case you notice the similarity. Additionally for trailer titles we also show the date they are to be released in the cinemas, or if they are currently showing, underneath the player (for example Avatar).

Free titles shown with a blue overlay

For those wanting more than free content we have a number of improvements for purchasing our premium content. Firstly the browse catalogue now supports price filters, which is quite handy for finding cheap flicks. Secondly we now have box sets and bundles – so you can buy a whole TV season or a movie bundle (Godfather Trilogy perhaps?). And lastly we are now securely storing credit card details for repeat purchases, much like Amazon. Using this I think you can go from browsing to watching in under a minute if the title is available for streaming – which is much quicker than trekking to the local rental store.

We have also updated the Games area and added a new video jigsaw. This is good fun on the easy a medium skills, although the hard setting can take a little while – you’ve been warned. All the current competitions are now listed on a dedicated competition page for all the competition junkies.

Last but by no means least is the member referral system – pass on your invite code after a purchase and you’ll get £1, while your mates will get £5 credit each! That is more than enough for even the latest rentals. I think this is a great deal for everyone so get it while it’s hot.

• Ajax works, however it will only do a GET request – POST returns immediately with no data supplied.
• General browser model most closely follows Internet Explorer, however the accessors for the CSS opacity is like other browsers (Mozilla).
• Currently can’t read the opacity, gives a value of 1.0 always.
• Can’t fire custom events as the method for creating an event does not work (document.createEventObject).
• Since there are no custom events some functions do not work, such as Prototype’s document.observe('dom:loaded').

The event firing issue is a bit of a nasty one if you rely on custom event for updating page elements. To work around Prototype’s dom:loaded issue I have added a new method: Event.domLoaded(), that takes a function and stores it in an array. Once the DOM is ready the array is iterated through and each function is called. I have also enhanced the method to call the function parameter if the DOM has already been marked as ready, thereby preventing race conditions caused by a delayed function call. In other words it makes it safe to register a function to be called as soon as possible, regardless of the present state of the page.

I have uploaded an updated version of Prototype with the fixes applied – download here. Let me know if there are any issues.

The problem manifested itself mainly as a one second or more delay when hovering over images. Typically this would be a trivial operation for a browser, as it was simply applying the :hover classes for the element.

A bug was raised and I started the investigation by checking the usual culprits, such as incorrect binding of javascript events, HTML issues, CSS selectors that had run wild, and so on. A search on Google yielded no usable results that helped, either. Since none of these had improved the performance of the page, I started to get a bit more aggressive with my bug hunting.

I removed all the images from the page and this worked, although that wasn’t a solution that we could use. After carefully working through it, it turned out to be an issue with only the background images that we were using.

Now there was nothing really special about the background images, just simple PNGs, 1 pixel wide with repeat-x on them, which are then used to form a gradient. Standard stuff for a web page.

Notice the hover state on the Fast and Furious icon - that was where it was most noticable.

Technically, all of our gradients are bundled together into one image and then positioned using CSS, so I checked to see if that was the cause of the problem. One of the options of our image bundling system is the ability to have the images split up and served as separate files. After giving that a go it was apparent that it had no effect.

My next thought was that perhaps the number of repeating images were to blame. I made a quick check by disabling the repeat-x, and the site started running like normal. Realising that this was the issue I used the gradient generator to output wider images (20px), and voilà, the performance issues were sorted for good. So, if you are going to use a lot of gradients, and would like to keep your site running smoothly, then make sure they are more than 1px wide, as IE7 just can’t handle it.

*Of note blinkbox.com uses over 4ooo pixels of vertical gradients site wide, so this might be an extreme case