Strange bandwidth issue

My sharpreader.net domain ran out of bandwidth earlier today. Checking the access log showed that this was due to a huge amount of downloads of SharpReader 0.9.4.0 - a version that's not even actively linked to from sharpreader.net.

All of these downloads have http://news.bbc.co.uk/rss/newsonline_world_edition/front_page/rss091.xml as the referrer, and all (except for a single MSIE request) have some version of Mozilla/5.0 (firefox/gecko) as the useragent. The BBC RSS feed does not contain any links to sharpreader0940.zip either.

I have no clue as to what is causing this. There are over 1000 unique IP's; some of which only request the file once, others multiple times in either random or fixed intervals (oftentimes as close as 2 minutes apart).

If anyone reading this has any idea on what may be going on here, please leave a comment below - thanks.

TrackBack URL for this entry: http://www.hutteman.com/scgi-bin/mt/mt-tb.cgi/191
Comments

What's eating Luke('s bandwidth)?
How could you get a thousand requests for an old version of SharpReader claiming to be from Firefox, and claiming to be referred from a BBC RSS feed, other than from someone with a bunch of zombies and a grudge?

Trackback from phil ringnalda dot com at June 23, 2005 11:48 PM

Is that a normal percentage of Win98 users for you? I didn't actually count the hits in your log, but in my current log 152 of 6004 Firefox hits were Win98, and only one of your mystery hits claimed to be Win98. A touch suspicious, methinks.

Posted by Phil Ringnalda at June 24, 2005 12:25 AM

Evidently, a proxy-using 'bot, with a fake Referer and fake User-Agent. You certainly could use mod-rewrite to block that Referer (perhaps in conjunction with that UA).

But if someone's out to get you, they can change the Referer, change the UA and continue sucking down your bandwidth.

Looking at the Log, it's not even the same UA string (claims to be various different versions of Firefox). So blocking this jackass may not be easy.

Posted by Jacques Distler at June 24, 2005 12:38 AM

I've removed that old version of SharpReader so it won't be anything but 404's on those requests, but I don't understand why someone would go through the trouble of using over 1000 proxies, a variety of firefox useragents, yet download the same file every time and leave the same referrer on every request...

is there a way to verify if those IP's are indeed proxies?

Posted by Luke Hutteman at June 24, 2005 1:18 AM

> Is that a normal percentage of Win98 users for you?

According to http://extremetracking.com/open;sys?login=shrprdr, 4.82% of the hits on the main sharpreader page come from win98. Not sure how relevant that is though as the current influx of hits bypasses the main page and downloads the zip directly. It is however interesting that there's just a single Win98, as well as a single MSIE in all these requests...

Posted by Luke Hutteman at June 24, 2005 1:35 AM

Far as I know, the only way to be sure whether a particular IP address has an open proxy server running is to try to connect to every single possible port, and when you get an answer try to GET a page from another host. However, for the quality of open proxies that are used to comment spam me, the simpler approach is just to google the IP address: most of them will wind up being listed in several dozen proxy lists. For the dozen or so I tried, none were, none were listed in opm.blitzed.org, and whois makes them look more like ISP customers than misconfigured servers. So, I'm thinking big network of zombies rather than just open proxies. Pissed off any big-time email spammers or virus writers lately?

Posted by Phil Ringnalda at June 24, 2005 2:03 AM

Given the huge range of ip addresses, the fact that it's all firefox 1.0.x versions requesting the file, and the fact that an attack in this way makes absolutely no sense... I suspect a non-malicious but just badly coded firefox extension, which under certain circumstances requests this file (hardcoded in somewhere). Problem is, there are at least 20 feed related firefox extensions, so getting any kind of confirmation of that theory won't be easy.
Might be worth a shot asking in the extension development forum at mozillazine if any extension authors recognizes himself in this behaviour.

Posted by Sander at June 24, 2005 3:26 AM

In fact, I just did this: http://forums.mozillazine.org/viewtopic.php?t=283799

Posted by Sander at June 24, 2005 3:34 AM

The problem might be related to prefetching. I don't remember exactly in which version of Firefox this functionlity was introduced, but several major sites didn't like it. For examlpe IMDB block prefetching, which can be easily seen by following results in Google searches to IMDB pages.

Maybe some search on BBC returns a link to your site and those poor soles prefetch it without even noticing...

Posted by Leonid Mamchenkov at June 24, 2005 5:44 AM

The referrer is a problem for both an extension and prefetching: the extension would need to not only be hardcoded to download a particular version of a Windows program, which would be odd, it would also need to explicitly set a fake referrer from the BBC's RSS feed. Prefetching would require either that someone at the BBC went mad, and included <link rel="prefetch" href="...0904.zip"> in the XSLT or a Link: ...0904.zip; rel=prefetch HTTP header for a while (or still does, but not for me), or that someone has a way of causing Firefox to send a fake referrer on a prefetch hint in an arbitrary page on another site.

Posted by Phil Ringnalda at June 24, 2005 11:06 AM

Strange thing I noticed: all of the most active clients use the User-Agent string "Mozilla/5.0". Given this, I doubt this is related to prefetching (which has been there since the beginings of Firefox) or any extension. This really looks like a DDoS attack but it doesn't make much sense, I agree.

Look at this: http://www.google.com/search?q=%2224.238.181.140%22
According to this, the IP address was frequently used by a guy with the nick ybtypical220, his profile is here: http://seadogs.bethsoft.com/forums/index.php?showuser=3904. He is probably the owner of one of the machines participating in the attack, you can try to send him an email. Maybe he can help you find out what is going on (if his computer in fact still has the same address).

Posted by Wladimir Palant at June 24, 2005 11:14 AM

65.57.245.11 is also interesting. According to Google this is a Linux machine with a static IP address running Webcollage (http://www.jwz.org/webcollage/webcollage). Highly unusual for a zombie - the IP address is probably only a corporate proxy. Here is an email address you can try: http://www.modpython.org/FAQ/faqw.py?req=revision&file=faq02.006.htp&rev=1.7

Posted by Wladimir Palant at June 24, 2005 12:00 PM

Prefetching seems unlikely, since Firefox will only prefetch a URL if it finds an <a> or <link> element with rel="prefetch" or rel="next", or the prefetch header Phil mentioned.

Posted by Kelson at June 24, 2005 12:21 PM

Possibly it is a batch downloader with problem, something like NetTransport, and it got stuck in a loop...

Posted by Blair at June 28, 2005 1:45 PM

i think i can help
i just installed sharp reader and the rss feed to your blog is installed by default. (the developer may be a fan?)
the internal reader would be mozilla based.

cheers.

Posted by paul at June 28, 2005 10:28 PM

Sounds like somebody is using a zombie and making ad-money from your download link.

Posted by NewMC at June 30, 2005 10:53 AM


Hi,
I just got here through what might have caused your traffic issue:
You are listed in 2nd place here:
http://www.repubblica.it/servizi/rss/index.html

You might have figured out by now

Posted by Toni at July 2, 2005 8:14 PM

I was doing some poking on this problem, just for fun, and came up with
this Google-cached page.

Posted by Cailean at July 11, 2005 11:05 AM

You might be interesed in knowing I have had to stop using sharpreader and switched to Newsgator, because SR was using over 150MB of Memory on my machine.

Posted by Charlie Barker at July 18, 2005 7:46 AM

Has it occurred to you that it might be a virus or Trojan in your system? There is a Trojan that Microsoft Antispyware picked up recently that is designed to invite other Trojans into your machine, while this one in particular looks for a less than strong administrator password to get in & generally play havoc with your machine

Posted by Max at July 19, 2005 3:19 PM

Could you please make Sharp Reader support ATOM 1.0? I already have two unreadable feeds and I'm sure this number will only grow...

Posted by Paul Goscicki at July 21, 2005 12:02 PM

The bandwidth and download issue is a result of the most recent Microsoft Newsletter on RSS which lists Sharpreader first in its list of RSS readers.

Posted by DAG at October 5, 2005 7:33 PM

Hey,
In regards to the download problem, why not make it only downloadable through a php script? Have the script look for the randomly generated key of the day (stored in a protected directory and only accessible by that script), and if that key is available and correct get the php script to send the file.

Posted by JW at February 22, 2006 2:05 PM
This discussion has been closed. If you wish to contact me about this post, you can do so by email.