Rules Engine for HTTP Requests
by Brad on Jul.10, 2011, under Security, Web Development
Inspired by the power of Fiddler, I wanted to create a user friendly interface and rules engine to monitor and filter requests made by my computer when browsing the web. Fiddler itself is a great tool, however it is definitely aimed at technical people. Luckily Eric Lawrence, author of Fiddler, offers the core engine as a separate library, ready for other developers to customise and extend as they see fit.
Last year I began work on a Windows based application for monitoring the requests that are made by a computer over HTTP and then apply a rules engine to modify those requests before being send to the server. It is also able to modify the responses before being passed on to the web browser. Based on FiddlerCore and WPF, the application sits in the system tray displaying information on the recent HTTP requests and any rules that have been applied. Rules can be enabled and disabled via the user interface and customised by the use of a simple XML file.
The application sets itself as the system proxy, so all browsers that are set to use the system proxy will start issuing requests via the application. A word of warning, if the process is forcefully closed it will not have an opportunity to clean up, this may result in all web connections being blocked, as the system proxy will not be set to an address that is listening for requests. A quick fix is to restart the application and close it cleanly via the menu.
Screenshots

Log of matched rules and URL that triggered the rule

A list of the requests that have been issued by the computer

A list of the rules and that status of each rule
Provided rules
- Block request – Each request can be blocked from being sent to the server.
- AdBlock Plus implementation – A cut down version of the AdBlock Plus for blocking advertising and tracking content. See below for more information.
- Https Everywhere implementation - Send requests over HTTPS instead of HTTP for popular sites.
- Python script – Run custom code for each request. This is provided to give extensible functionality.
- Modify header – Modify a header before it is sent to the server or returned to the browser, including removing it.
- Modify cookie – Similar to the modify header rule, with support for individual cookie values.
- Break action – Prevent any other rules from running for the request or response.
- Save file – Save request content to the disk if matching a pattern.
- Age filter – Implementation of my proposed header for restricting content that is inappropriate for minors.
AdBlock Plus rule
Included is a version of AdBlock Plus that can potentially give support to Internet Explorer, as well as other applications that issue HTTP requests via the system proxy. Since it does not run inside a browser only rules that are based on URL patterns are supported, all requests that work by hiding HTML elements will not run.
I’m not advocating the use of an ad blocker, as most websites are funded by advertising revenue, resulting in much of what we read on the internet being ‘free’. This was built mainly as a technical exercise for me to see any false positives that are preventing my pages from rendering properly when running the real AdBlock extension. It is extremely useful in that role, so I thought other web developers would benefit too.
I also built this as a way to write my own basic regular expression implementation, and as such it may not perform as well as a version based on the optimised Regex classes within the .NET framework. I was curious to see how hard it would be to write a simple regular expression parser and matcher, and the opportunity presented it quite nicely in the form of the AdBlock Plus rules engine.
The Firefox plugin, on which this is based, converts each rule to a regular expression, then runs it using the optimised engine within Firefox. I wanted to circumvent this step and see if I could directly parse and interpret each rule, as this would give me an insight into how a regular expression engine works. While I am very pleased with the results, it is still not up to the performance of the standard regular expression engine, and all the optimisations that have been added over the years. I may work on optimising my engine in the future, but only if time permits.
Performance
To put it bluntly, this will not speed up your system, quite the opposite in fact. On my Core i5 system the overhead is not great, however my Core 2 Duo laptop has a noticeable delay before requests are sent. Generally, the large sets of URL patterns by the AdBlock Plus rule are the culprit, so if performance is an issue try disabling that first. Additionally the Https Everywhere rule will cause a significant slowdown on sites that are forced to use HTTPS, due to the additional overhead of using a secure connection.
Download
I have just added the application to GitHub as an open source project. You can download the installer here.
Baby name recommendations using the names of 170 million facebook users
by Brad on Jul.08, 2011, under Web Development
I had a little spare time a few weekends ago, and had an idea to write a recommendation engine for finding names that are related to each other. I got off to a great start with a fantastic data source, the list of 170 million facebook users provided by a security researcher.
Starting with this, I wrote an engine that finds related names, using common surnames as a base. Each name is calculated to find all the related names, then ranks them using 4 separate methods. The main problem that I faced what that the most popular names are the most related names for everyone – most people seem to have a Michael in their family. Each ranking method tries to find the names that are more common in the related name set compared to the global list of names, which are hence the most related.
I found that the results were initially a little bit inconsistent, for example the name Brad would be related to Brett, however the inverse wasn’t true. So counter this, I added a reverse lookup table and factored that into the ranking alorithm, which helped clean up the data and remove the odd combinations. This ensures that both names need to be related to each other, and removes names that are only related in one direction.
The results were better than I had hoped for, with over 25,000 of the most common names being computed and stored. I can see there are clear trends amongst regions, religions, similar sounding names, and even real world items, such as colours, fruits and emotions. I hope someone finds this useful, especially for trying to find odd or obscure baby names.
iTunes deleted my Apps! Or did it?
by Brad on Apr.24, 2010, under iPhone
I recently upgraded my main PC, and as part of that I also had to transfer my music, photos and documents from the old machine to the new one. I used the trick here to rebuild my iTunes library on the new machine, as this allowed me to keep my ratings and playlists. This went off without a hitch. Great.
Then I plugged my iPhone in. Bad idea.
Without any warnings or confirmations it started to sync, which seemed fine as I now had a complete library of music, so I let it be. All my personal data, such as contacts and emails, lives in the cloud or on servers I have access to. Once the sync was finished I was shown a message that informed me that some of my apps were not authorised on the new machine. I fired up the iPhone, and to my dismay most of my apps had been deleted, with only a few exceptions.
The problem was that I had not yet signed into iTunes and authorised the new PC, which is quite an easy mistake to make. A warning before wiping everything would have been nice, thanks Apple. I had a look on the interwebs for a while, however this did not yield any great solutions for getting my precious apps back. The best solution was to manually go through my purchase history and re-purchase all the apps. Let me assure you this is boring, so I gave up after re-purchasing about 5 apps.
In the meantime, I decided to clean out my music folder. It was then I struck the goldmine of my old apps. iTunes saves all the downloaded apps in a directory with the extension .ipa. In my case this directory was D:\Users\Brad\Music\iTunes\Mobile Applications. For most people it will be in your My Music folder, then iTunes\Mobile Applications.
Simply click each .ipl and the app will be added back to iTunes. One sync and you should be golden.
Steve’s Wedding Chapel
by Brad on Mar.15, 2010, under blinkbox
Apparently our dev team gets angry if weddings are mentioned. Don’t believe me? Well let me explain.
In February and March 2008 I took some time off to visit Australia as I was going to attend a few weddings. On my return my desk had been covered in aluminium foil. Actually, to say my desk had been covered in al-foil is a massive understatement. Everything had been wrapped up – the Weetabix in my cereal box, all the loose change in my drawer, the drawer itself, the pens, pen lids and so on.
The pictures don’t fully do it justice, but you get the idea. It took me about a week to unwrap everything, and even now, over 2 years later, I still have my monitor stand and a bowl on my desk as a tribute to the 3 weeks of effort the dev team put in. It was truly epic. I was convinced that this would be the best office prank I would ever see…..
So recently one of our engineers, Steven, took 2 weeks of annual leave. When he was asked why the reply was “just for a holiday and a ski trip” or something similar. He forgot to mention it was for his wedding and honeymoon. Needless to say this riled up the dev team again, and a plan was formed. This one had to be be more epic than the al-foil prank and by a long way.
A group meeting was organised and the details fleshed out. This time construction would be required; we were going to build a wedding chapel on his desk.
Work got under way and even from the start this was going to be huge, so big in fact that wood, nails and tape would be required. We had the frame done by the end of the first week, which was a good start. I was then off to go snowboarding for a week, so that is where I left the project. I had no idea the result be be so grand.
On my return I was greeted with the same sight as Steven when he arrived on Monday morning. A huge 10 foot tall chapel on his desk complete with:
- bell tower
- fresco ceiling with flying spagetti monster artwork
- red carpet for his chair and keyboard
- wedding music from a gift card that plays when his mouse is moved
- illuminated signs for ‘Steve’s Wedding Chapel’
- internal fairy lights
- wedding car cans on his chair
- blinkbox-themed stained glass windows
- custom Slogfather artwork made from one of the wedding pictures (Steven’s last name is Sloggett)
and the list goes on. I’ll let the pictures to the rest of the talking, however Jodie has more info on the chapel construction here.
Testing the Martingale with Roulette
by Brad on Feb.15, 2010, under Simulation
I was at a loose end a few weekends ago and thought that I would build myself a little Roulette simulation and give one of the most well known betting strategies a go, the Martingale. The principle of the Martingale is that you double your bet if you lose, and then reset back to your starting bet if you win.
The theory is that you will always win back your losses once you have a win. For example; using a starting bet of 1 and having 3 losses before a win the bets would be: 1 (loss) + 2 (loss) + 4 (loss) results in a total loss of 7. The next bet should be 8 to recover the losses. As it turns out for every win you increase your pot by one, regardless of the number of losses. It sounds perfect on the surface, but can it really work?
It is commonly cited that the casino would have a limit on the table, resulting in a hypothetical punter eventually not being able to bet the required amount to get back all the losses after a large losing streak, even if they had the funds available. With a large number of gambling sites online I thought that there is a possibility that you may be able to find a venue willing to take even a large bet, hence I excluded this constraint from the testing. As it turns out it is very hard to make the system work even with some basic real world constraints, such as credit limits.
In fact if we were to change the multiplication from a factor of 2 to a factor of 3 (1,3,9,27, 81 etc), or a factor of 10 (1, 10, 100, 1000) the theory would still work and possibly the wins would be greater. A win after 3 losses would net 14 from a cost of 13 using a factor of 3, and 889 from a cost of 111 for a factor of 10. As such I though these higher factors should also be investigated as I had not read about people trying to use them.
Firstly, lets start with the ideal case. I have used a simulation of European Roulette, where the table only has one house spot (a single zero), as opposed to American Roulette, which has two house spots (zero and double zero). This would help tip it even further in favour of the punter. Each one of the graphs below represents a typical run, however, each simulation would vary quite a bit.
After playing 1000 games using no real world constraints we can see that in fact the punter has come out on top. The final amount won was in fact 471, which is quite close to what a statistical model would predict (one unit for each win, with an overall probability of around 0.47). The big issue is that in order to get to that level there were 3 occasions where very large losses had been incurred, putting the punter well in the red. In this specific simulation over 8000 was risked on one losing streak, while already 8000 in the red. This is only for an end payout of 471 – not a good risk in my books.
If we make it so the punter can only make single unit bets once they are in the red then we see how the picture changes – after only a few games the punter has blown their reserve cash trying to win it all back. From there it is a constant downhill spiral. This is not quite realistic, generally a player would go in with a larger reserve.
Updating the simulation to allow for a starting kitty of 100 we see that for a while the punter is doing well, however, as soon as there is a run of 6 losses the game is over. In fact, compared to no starting cash the player is actually worse off by about 90 units for this simulation.
At this point I thought it might be worth looking into using higher multiplication factors to see if the situation could be improved. My initial suspicions were that while the wins would be greater, so would the losses. I couldn’t quite get my head around the idea that perhaps if the multiplication was larger than Euler’s number (2.71828) then perhaps something magical would happen regarding exponential growth. It didn’t, so at least I proved something to myself with this experiment.
Tripling up showed some interesting results – in this case we have had to bet over 28000 for a final payout of around 19000. The numbers are larger, however, so is the risk. Some simulations have very large losing streaks resulting in any real player being long since wiped out. The situation is much the same when multiplying each loss by 10.
Wow, almost one billion made after only 500 games! Unfortunately, the bet was over one billion so the chances of any normal person getting a loan to cover a bet like that are, well, small.
I did not stop simulating there, trying all sorts of combinations using different numbers and different strategies. Nothing would work in the long run while some reasonable real life constraints were placed on the simulation. This is not a surprise result, as Einstein apparently once said: “No one can possibly win at roulette unless he steals money from the table while the croupier isn’t looking”. Nonetheless, it was still an interesting exercise.
















