Thursday, June 23, 2005 7:14 PM
Strange bandwidth issue
My sharpreader.net domain ran out of bandwidth earlier today. Checking the access log showed that this was due to a huge amount of downloads of SharpReader 0.9.4.0 - a version that's not even actively linked to from sharpreader.net.
All of these downloads have http://news.bbc.co.uk/rss/newsonline_world_edition/front_page/rss091.xml as the referrer, and all (except for a single MSIE request) have some version of Mozilla/5.0 (firefox/gecko) as the useragent. The BBC RSS feed does not contain any links to sharpreader0940.zip either.
I have no clue as to what is causing this. There are over 1000 unique IP's; some of which only request the file once, others multiple times in either random or fixed intervals (oftentimes as close as 2 minutes apart).
If anyone reading this has any idea on what may be going on here, please leave a comment below - thanks.
TrackBack URL for this entry: http://www.hutteman.com/scgi-bin/mt/mt-tb.cgi/191
What's eating Luke('s bandwidth)?
How could you get a thousand requests for an old version of SharpReader claiming to be from Firefox, and claiming to be referred from a BBC RSS feed, other than from someone with a bunch of zombies and a grudge?
Is that a normal percentage of Win98 users for you? I didn't actually count the hits in your log, but in my current log 152 of 6004 Firefox hits were Win98, and only one of your mystery hits claimed to be Win98. A touch suspicious, methinks.
Evidently, a proxy-using 'bot, with a fake Referer and fake User-Agent. You certainly could use mod-rewrite to block that Referer (perhaps in conjunction with that UA).
But if someone's out to get you, they can change the Referer, change the UA and continue sucking down your bandwidth.
Looking at the Log, it's not even the same UA string (claims to be various different versions of Firefox). So blocking this jackass may not be easy.
I've removed that old version of SharpReader so it won't be anything but 404's on those requests, but I don't understand why someone would go through the trouble of using over 1000 proxies, a variety of firefox useragents, yet download the same file every time and leave the same referrer on every request...
is there a way to verify if those IP's are indeed proxies?
> Is that a normal percentage of Win98 users for you?
According to http://extremetracking.com/open;sys?login=shrprdr, 4.82% of the hits on the main sharpreader page come from win98. Not sure how relevant that is though as the current influx of hits bypasses the main page and downloads the zip directly. It is however interesting that there's just a single Win98, as well as a single MSIE in all these requests...
Far as I know, the only way to be sure whether a particular IP address has an open proxy server running is to try to connect to every single possible port, and when you get an answer try to GET a page from another host. However, for the quality of open proxies that are used to comment spam me, the simpler approach is just to google the IP address: most of them will wind up being listed in several dozen proxy lists. For the dozen or so I tried, none were, none were listed in opm.blitzed.org, and whois makes them look more like ISP customers than misconfigured servers. So, I'm thinking big network of zombies rather than just open proxies. Pissed off any big-time email spammers or virus writers lately?
Given the huge range of ip addresses, the fact that it's all firefox 1.0.x versions requesting the file, and the fact that an attack in this way makes absolutely no sense... I suspect a non-malicious but just badly coded firefox extension, which under certain circumstances requests this file (hardcoded in somewhere). Problem is, there are at least 20 feed related firefox extensions, so getting any kind of confirmation of that theory won't be easy.
Might be worth a shot asking in the extension development forum at mozillazine if any extension authors recognizes himself in this behaviour.
The problem might be related to prefetching. I don't remember exactly in which version of Firefox this functionlity was introduced, but several major sites didn't like it. For examlpe IMDB block prefetching, which can be easily seen by following results in Google searches to IMDB pages.
Maybe some search on BBC returns a link to your site and those poor soles prefetch it without even noticing...
The referrer is a problem for both an extension and prefetching: the extension would need to not only be hardcoded to download a particular version of a Windows program, which would be odd, it would also need to explicitly set a fake referrer from the BBC's RSS feed. Prefetching would require either that someone at the BBC went mad, and included
<link rel="prefetch" href="...0904.zip"> in the XSLT or a
Link: ...0904.zip; rel=prefetch HTTP header for a while (or still does, but not for me), or that someone has a way of causing Firefox to send a fake referrer on a prefetch hint in an arbitrary page on another site.
Strange thing I noticed: all of the most active clients use the User-Agent string "Mozilla/5.0". Given this, I doubt this is related to prefetching (which has been there since the beginings of Firefox) or any extension. This really looks like a DDoS attack but it doesn't make much sense, I agree.
Look at this: http://www.google.com/search?q=%2220.127.116.11%22
According to this, the IP address was frequently used by a guy with the nick ybtypical220, his profile is here: http://seadogs.bethsoft.com/forums/index.php?showuser=3904. He is probably the owner of one of the machines participating in the attack, you can try to send him an email. Maybe he can help you find out what is going on (if his computer in fact still has the same address).
Prefetching seems unlikely, since Firefox will only prefetch a URL if it finds an
<link> element with
rel="next", or the prefetch header Phil mentioned.
Possibly it is a batch downloader with problem, something like NetTransport, and it got stuck in a loop...
i think i can help
i just installed sharp reader and the rss feed to your blog is installed by default. (the developer may be a fan?)
the internal reader would be mozilla based.
Sounds like somebody is using a zombie and making ad-money from your download link.
I was doing some poking on this problem, just for fun, and came up with
this Google-cached page.
You might be interesed in knowing I have had to stop using sharpreader and switched to Newsgator, because SR was using over 150MB of Memory on my machine.
Has it occurred to you that it might be a virus or Trojan in your system? There is a Trojan that Microsoft Antispyware picked up recently that is designed to invite other Trojans into your machine, while this one in particular looks for a less than strong administrator password to get in & generally play havoc with your machine
Could you please make Sharp Reader support ATOM 1.0? I already have two unreadable feeds and I'm sure this number will only grow...
The bandwidth and download issue is a result of the most recent Microsoft Newsletter on RSS which lists Sharpreader first in its list of RSS readers.
In regards to the download problem, why not make it only downloadable through a php script? Have the script look for the randomly generated key of the day (stored in a protected directory and only accessible by that script), and if that key is available and correct get the php script to send the file.