Brad Rees

Inspired by the power of Fiddler, I wanted to create a user friendly interface and rules engine to monitor and filter requests made by my computer when browsing the web. Fiddler itself is a great tool, however it is definitely aimed at technical people. Luckily Eric Lawrence, author of Fiddler, offers the core engine as a separate library, ready for other developers to customise and extend as they see fit.

Last year I began work on a Windows based application for monitoring the requests that are made by a computer over HTTP and then apply a rules engine to modify those requests before being send to the server. It is also able to modify the responses before being passed on to the web browser. Based on FiddlerCore and WPF, the application sits in the system tray displaying information on the recent HTTP requests and any rules that have been applied. Rules can be enabled and disabled via the user interface and customised by the use of a simple XML file.

The application sets itself as the system proxy, so all browsers that are set to use the system proxy will start issuing requests via the application. A word of warning, if the process is forcefully closed it will not have an opportunity to clean up, this may result in all web connections being blocked, as the system proxy will not be set to an address that is listening for requests. A quick fix is to restart the application and close it cleanly via the menu.

Screenshots

Log of matched rules and URL that triggered the rule

A list of the requests that have been issued by the computer

A list of the rules and that status of each rule

Provided rules

• Block request – Each request can be blocked from being sent to the server.
• AdBlock Plus implementation – A cut down version of the AdBlock Plus for blocking advertising and tracking content. See below for more information.
• Https Everywhere implementation - Send requests over HTTPS instead of HTTP for popular sites.
• Python script – Run custom code for each request. This is provided to give extensible functionality.
• Modify header – Modify a header before it is sent to the server or returned to the browser, including removing it.
• Modify cookie – Similar to the modify header rule, with support for individual cookie values.
• Break action – Prevent any other rules from running for the request or response.
• Save file – Save request content to the disk if matching a pattern.
• Age filter – Implementation of my proposed header for restricting content that is inappropriate for minors.

AdBlock Plus rule

Included is a version of AdBlock Plus that can potentially give support to Internet Explorer, as well as other applications that issue HTTP requests via the system proxy. Since it does not run inside a browser only rules that are based on URL patterns are supported, all requests that work by hiding HTML elements will not run.

I’m not advocating the use of an ad blocker, as most websites are funded by advertising revenue, resulting in much of what we read on the internet being ‘free’. This was built mainly as a technical exercise for me to see any false positives that are preventing my pages from rendering properly when running the real AdBlock extension. It is extremely useful in that role, so I thought other web developers would benefit too.

I also built this as a way to write my own basic regular expression implementation, and as such it may not perform as well as a version based on the optimised Regex classes within the .NET framework. I was curious to see how hard it would be to write a simple regular expression parser and matcher, and the opportunity presented it quite nicely in the form of the AdBlock Plus rules engine.

The Firefox plugin, on which this is based, converts each rule to a regular expression, then runs it using the optimised engine within Firefox. I wanted to circumvent this step and see if I could directly parse and interpret each rule, as this would give me an insight into how a regular expression engine works. While I am very pleased with the results, it is still not up to the performance of the standard regular expression engine, and all the optimisations that have been added over the years. I may work on optimising my engine in the future, but only if time permits.

Performance

To put it bluntly, this will not speed up your system, quite the opposite in fact. On my Core i5 system the overhead is not great, however my Core 2 Duo laptop has a noticeable delay before requests are sent. Generally, the large sets of URL patterns by the AdBlock Plus rule are the culprit, so if performance is an issue try disabling that first. Additionally the Https Everywhere rule will cause a significant slowdown on sites that are forced to use HTTPS, due to the additional overhead of using a secure connection.

Download

I have just added the application to GitHub as an open source project. You can download the installer here.

I received a call from my uncle Bruce 2 days ago – after saying hello I was greeted a friendly yet frantic “I’m in big trouble Brad, my computer has been hacked!”. After getting most of the details of what happened and realising that it was quite unlikely any bank details or other highly confidential details were compromised I arranged to leave work early and see what we could do.

While I spend a considerably amount of my time at work making sure our website is secure I had yet to personally have experienced the wrath of an online scammer. I was intrigued to say the least. Once I arrived we scoped out the problem – one of my uncle’s email accounts had been compromised when he inadvertently tricked into giving away his password. After reviewing the situation it was quite a simple mistake to make, as they had impersonated his email provider in a similar nature to previous contact he had regarding setting his BlackBerry.

It was quite interesting to see how the scammers work. They seemed to have a limited interest in trying to get money out of my uncle directly; they were more interested in trying to impersonate him. By using a line similar to: “I’m stuck in Africa at an art fair with no money, please can you send me £500 so I can get back” they actually managed to get about 3-4 responses from friends offering to help. I’m fairly sure they will also sell entire contact list to spammers for a reasonable fee.

After trying some common techniques for re-enabling access to his account we hit a wall. The scammers had changed all the details required to change an account, such as the alternate email address and secret question. Lucky for Bruce he had his email account linked to his BlackBerry. This they key we needed: by sending a password reset email back to the same address we were able to forward the email to another temporary email address and then follow the usual password reset steps. I say we used a temporary email as there was a chance that this wasn’t going to work, hence we didn’t want the scammer to be aware of another email address, should he check the sent items.

During the password reset process we were asked to provide country information. The scammer would have been asked the same question and it would appear that this is checked against a geo-ip database, hence they were forced to put in a reasonable location of Senegal. They may have been in a surrounding country and were exploiting the errors in the geo-ip database, however the general region seems correct based on the information provided when they were asking for money.

So this makes me ask the question – why can’t we lock our email accounts, such that it is only accessible from certain regions. Banks have this feature for credit cards, and even if they do get a bit over zealous sometimes and block my card while I’m on holiday, it is still a good system. It is my opinion that the leading webmail providers should include an option so that any access from unauthorised regions should be blocked – or at least prevented from making system changes.

For example in my case I would like to restrict access to Europe, North America, Australasia and possibly the Middle East (Another trip to Dubai would be nice!). Additionally I could state that changes to my account may only be done from the UK and Australia. Should I urgently need to make a change there are family and friends that I could call to make the change on my behalf, or simply use a VPN to gain access via a local ISP.

I would also like to be clear that this would be on a per account basis. I would not like to suggest that we block key internet services from developing countries, as that would surely result in even more hardship for the honest citizens of these regions. The system would not be perfect due to the limitations of GeoIP, however it would have most likely protected my uncle seeing as the hack seems to have originated from a place he has not visited.