And we're back... again
dmorris68
23-04-2008 17:17:50
In case you hadn't already heard, FiPG, FC, A4F, and some other GPT sites were hit with a massive DDOS attack last Friday. The server was so hammered that I couldn't even serve a static web page informing people of the downtime, but info was posted on A4F so hopefully most of you regulars figured out what was happening.
A4F got back online pretty quickly with help from their hosting provider. Since we're self-hosted and Admin is out of town, things were a little slower for us. We're up now and the server is screaming right along. We've made some changes in the server configuration, and have other changes planned, to hopefully better mitigate any future attacks. We'll also most likely be involving the authorities, as we have some info on the attacker that may be useful in bringing him to justice. BTW as far as we know, he was not associated with FiPG, A4F, or any other related sites -- he was apparently just trolling for popular sites to extort money from.
Thanks for hanging in there, and sorry for the extended downtime, but these things happen and there's only so much I can do when I'm 3000 miles from the server. Hopefully nobody died from their lack of freebie fix. ;)
JOSHBOX
23-04-2008 17:27:19
that was wierd
samz465
23-04-2008 17:33:32
Hoorah FiPG is back!
Ps JoshBox I love your avatar. I haven't seen it around as much as I used to though.
TFOAF
23-04-2008 17:42:18
hey glad to be back guys how iz evry1????
hitnaui
23-04-2008 17:46:52
Hurray!
![]("cd1d32bb52)
http//forum.ogplanet.com/smileys/1.gif[" alt=""/imgcd1d32bb52]
sandra habina
23-04-2008 18:08:23
WOW - GREAT WORK everyone involved and especially dmorris
Thank you for all your work, it is greatly appreciated. )
TryinToGetPaid
23-04-2008 18:35:36
Great job D-Mo. I am going to PM you...
cubbieco
23-04-2008 18:48:49
[quote18a28be9f2="dmorris68"]
Thanks for hanging in there, and sorry for the extended downtime, but these things happen and there's only so much I can do when I'm 3000 miles from the server. Hopefully nobody died from their lack of freebie fix. ;)[/quote18a28be9f2]
I was close to death, but managed to survive with counseling.
ilanbg
23-04-2008 18:51:12
do you have any idea how many times i had to masturbate to pass the time
do not do that again
ricopet
23-04-2008 19:29:48
I went through some withdrawals, but all is well now. Thanks dmorris!!
manOFice
23-04-2008 19:52:31
amazing how much more work i got done at work while this site was down.... hrmmm
J4320
23-04-2008 20:08:33
[quoteecf39edba1="ilanbg"]do you have any idea how many times i had to masturbate to pass the time
do not do that again[/quoteecf39edba1]
Srsly. My sack was feeling worse than an eyeball dipped in battery acid.
terryishere
23-04-2008 20:11:56
you are awesome dmorris thanks for the hard work and dedication..so glad to have fipg back up and running!!! terry
Denise07
23-04-2008 20:13:30
Small withdrawals, but I am OK now. Thanks dmorris!!
YourGiftsFree
23-04-2008 20:16:06
Thanks David. You da man!
zr2152
23-04-2008 20:46:16
Yeah i had no clue what to do. I was so lost without FIPG!
theysayjump
23-04-2008 20:53:02
I died.
I got better!
ajasax
23-04-2008 21:25:52
Intarweb - FiPG = FAIL
kidd2108
23-04-2008 21:52:17
We're back!!
woot
puppeteer
23-04-2008 22:37:34
Oh yea! kma+ to dmo
nyne7lac
23-04-2008 23:05:47
Great that we're back
The down time messed up one of my trades because I couldn't communicate, but I'm glad things are back on track!
akalic
24-04-2008 01:04:33
WOO, man, after this incident i realize i check FIPG more frequently than i thought
CollidgeGraduit
24-04-2008 02:27:04
![]("b1f16b597a)
http//i153.photobucket.com/albums/s235/revmyspace2/graphics/greetings/welcome-back/welcomeback.gif[" alt=""/imgb1f16b597a]
[quote1ef7a54016="ilanbg"]do you have any idea how many times i had to masturbate to pass the time
do not do that again[/quote1ef7a54016]
Guaranteed not as many as me. I'm da champ.
sun-flower
24-04-2008 05:39:44
Great job and I'm glad you are back.
D
dmorris68
24-04-2008 06:33:30
Okay folks, to be honest, I'm not feeling comfortable accepting all the credit. Sure, I busted my tail to try to get us back online, but I wasn't the only one working on the problem. It was a combined effort, and we're not even going to pretend to be bullet-proof at this point, but at least we have a better idea of what to do next time.
I hope that the authorities take action.
TryinToGetPaid
24-04-2008 06:39:56
Everyone who worked their ass off, already knows my appreciation. I missed this place....
zdub08
24-04-2008 07:15:00
thanks for singlehandedly saving fipg dmorris
babetran
24-04-2008 07:23:17
I'm glad we're back up.
Iloveipods2
24-04-2008 09:26:27
hmmm flr was the only one not hit.........
I wanted to die.
And CG said I was banned. I H8 U BB BOY.
terryishere
24-04-2008 10:20:40
thanks and god bless all who helped fipg get back up and running you guys are awesome!!! you are all da man!!!!
mookieb2
24-04-2008 12:17:13
[quoteed64b883d5="zdub08"]thanks for singlehandedly saving fipg dmorris[/quoteed64b883d5]
This had me rolling, good stuff.
Thank goodness we're back! Excellent work everyone.
zdub08
24-04-2008 12:20:44
wat
mookieb2
24-04-2008 13:48:19
[quoteb9ebb1c3b3="zdub08"]wat[/quoteb9ebb1c3b3]
Oh, I thought you were being funny.
Dmorris posts a long post about he's not the only one that did the work to get us back up, and three posts later you give him all the credit.
It made me laugh, thought you were being cynical.
bballp6699
24-04-2008 13:51:23
I checked a4f, but I guess I didn't see the right thread. I did find out something pretty interesting...
Your current time Apr 24 2008, 0849 PM
Your last visit was on Oct 20 2006, 1258 AM
There have been 435543 new posts in 54222 topics
Total posts in the last 24 hours 979
[b5b05422a6d]You have 0 new PMs in your inbox[/b5b05422a6d]
Nobody loves me... cry
zdub08
24-04-2008 14:08:17
[quote2af845e60b="mookieb2"][quote2af845e60b="zdub08"]wat[/quote2af845e60b]
Oh, I thought you were being funny.
Dmorris posts a long post about he's not the only one that did the work to get us back up, and three posts later you give him all the credit.
It made me laugh, thought you were being cynical.[/quote2af845e60b]
o
ilanbg
24-04-2008 15:31:02
Let's be honest, though, guys... it [i4ffd666ad5]did[/i4ffd666ad5] take David and co. like over two (three?) days to get the forums back online. That seems sort of... lazy... to me. shrug
akalic
24-04-2008 17:24:20
insta-ban
dmorris68
26-04-2008 20:11:19
We were attacked yet again but it seems to have died down today, so here we are again.
EatChex89
26-04-2008 20:12:06
we seem to be popular.
hitnaui
26-04-2008 20:15:40
Was it the same person?
samz465
26-04-2008 22:33:14
I believe so.
Admin
27-04-2008 07:24:43
btw, ilanbg, the nature of a ddos makes it pretty difficult to even log in to a server remotely, let alone diagnose or fix it. it's an utter and complete saturation of the available network resources. dmo is geographically far from the server, and i'm further. we did what we could in the time that we could, but there was little 'lazy' about it.
potsey2007
27-04-2008 07:34:50
good work boys!
TravMan162
27-04-2008 08:52:16
If they find the guy that keeps doing this we should put him in a box full of knives and kick him down the stairs.
samz465
27-04-2008 11:16:49
[quote6f1f0fce7d="Admin"]btw, ilanbg, the nature of a ddos makes it pretty difficult to even log in to a server remotely, let alone diagnose or fix it. it's an utter and complete saturation of the available network resources. dmo is geographically far from the server, and i'm further. we did what we could in the time that we could, but there was little 'lazy' about it.[/quote6f1f0fce7d]
I thought ilan was just kidding shrug.
ilanbg
27-04-2008 12:17:47
[quote09df1c2216="samz465"][quote09df1c2216="Admin"]btw, ilanbg, the nature of a ddos makes it pretty difficult to even log in to a server remotely, let alone diagnose or fix it. it's an utter and complete saturation of the available network resources. dmo is geographically far from the server, and i'm further. we did what we could in the time that we could, but there was little 'lazy' about it.[/quote09df1c2216]
I thought ilan was just kidding shrug.[/quote09df1c2216]
Yeah, it was just a joke. I know you guys put a lot of effort in keeping this place as pimpin' as it is. Good job, y'all.
akalic
27-04-2008 12:38:30
[quote87972ac581="ilanbg"][quote87972ac581="samz465"][quote87972ac581="Admin"]btw, ilanbg, the nature of a ddos makes it pretty difficult to even log in to a server remotely, let alone diagnose or fix it. it's an utter and complete saturation of the available network resources. dmo is geographically far from the server, and i'm further. we did what we could in the time that we could, but there was little 'lazy' about it.[/quote87972ac581]
I thought ilan was just kidding shrug.[/quote87972ac581]
Yeah, it was just a joke. I know you guys put a lot of effort in keeping this place as pimpin' as it is. Good job, y'all.[/quote87972ac581]
oh you! playful
puppeteer
27-04-2008 15:20:59
hey.......... i was so drunk this morning... and the first thing i thought about was posting on a DRUNK POST thread lol
dmorris68
29-04-2008 06:54:41
From an update I posted at A4F just now
The attack stops and starts, but the good news is bos and I are making some headway. Most of the downtime over the last couple of days has been me taking the web server down to tweak & tune things. We've improved things to the point now that we can at least somewhat function through the attacks, although it's slow and sometimes your browser will timeout and need a refresh. At least the box is no longer being brought to its knees. Today I'm working on adding more functionality that should hopefully improve things further. I've also taken advantage of the downtime to do some server housecleaning and reconfiguration that I had put off due to the downtime requirements.
We've also initiated contact with the authorities -- you can believe we will be pursuing action on this.
Please bear with us while we work through this. The upshot is we'll have stronger, more response servers that are better able to withstand these attacks in the future. In the meantime, keep trying -- you'll find the forums are very responsive at times, slower at others, but for the most part functional when I don't have it down working on it.
gafdpc
29-04-2008 09:50:41
And TRAINN.
(
ajasax
29-04-2008 10:05:43
Trainn's working for me shrug
![]("cfe2c0caa5)
http//trainn.org/proof/SD530044.jpg[" alt=""/imgcfe2c0caa5]
J4320
29-04-2008 11:34:51
I decided not to swtich the stickies out until I think the current stickies have had their fair share of time/views. Just letting the mawdz and traders know.
this is pretty frustrating isn't it?
TryinToGetPaid
29-04-2008 12:51:35
GO DMO!
TravMan162
29-04-2008 13:43:45
Use the down time to do something productive.
Like wash your sheets.
I know half you freaks haven't done that since last July.
D D D ha just kidding.................... Sort of.
J4320
29-04-2008 14:25:33
lots of strange substances in them sheets
puppeteer
29-04-2008 22:09:01
![]("02d4823f34)
http//photos-h.ak.facebook.com/photos-ak-sctm/genericv2/892/117/01AwcAX2J8iZcAAAABAAAAAAAAAAA.jpg[" alt=""/img02d4823f34]
pqdrummer
30-04-2008 06:21:33
I'm still getting 403 errors all the time and it looks like the stylesheet isn't loading when the page itself does actually load
dmorris68
30-04-2008 07:24:50
[quote30bbcb7c9d="pqdrummer"]I'm still getting 403 errors all the time and it looks like the stylesheet isn't loading when the page itself does actually load[/quote30bbcb7c9d]
How hard are you hitting the site? As part of the DDOS botnet defense, 403 errors were being returned to clients who were attempting to hammer a page. Normal browser users should never get one, and with all the refreshing I've done, I've yet to see a 403. You sure your PC isn't infected with a bot? ;)
I'll check to see if your IP is found in the blacklist, but if it were there I don't think you'd get anything at all -- as of yesterday, those IP's are being blocked upstream of the web server, so they won't get any response at all now.
Regarding the stylesheet not loading, are you using Firefox? That is something I've noticed with FF on multiple PC's, and not just at FiPG. For some reason it just sometimes fails to load a stylesheet. Or it could be when the server is lagging and the browser is refusing to wait long enough to pull the stylesheet. I've not seen it at all since yesterday, however -- the forums are running better than ever for me, and I'm using FF.
[b30bbcb7c9d]EDIT[/b30bbcb7c9d] Yes, your IP was in the DDOS blacklist table, it was blacklisted at around 656am PST (server time) yesterday. Which is why you're getting frequent 403's. However it's not in the upstream drop list (obviously, or you wouldn't have posted from that IP) so it must have been listed before the latest server changes. I'm going to try to remove it manually, but it may require a server restart. Let me know if it keeps happening. Oh, and scan your system in you case you do have a bot infection.
TryinToGetPaid
30-04-2008 08:50:38
Dmo, again -- you and Tyler rock the fuckin' house.
J4320
30-04-2008 11:35:07
Yeah, nice job guys. )
TTGP has a major crush on Dwight by the way.
dmorris68
30-04-2008 11:39:51
pqdrummer, I saw where you were just blacklisted again.
[code10e0414e479]Apr 30 10:57:45 fipg mod_evasive[70845]: Blacklisting address xxx.xx.xx.xxx: possible DoS attack.[/code10e0414e479]
That was not quite an hour ago, after I removed you from the blacklist the first time. Something about your system or your access pattern is tripping the DoS filter. I don't see others having the issue so far, in fact there have been no further blacklists since yesterday, up until after I removed your IP from the blacklist. This time it also added you to the drop list, which would have kept you from coming back in at all. I've removed you again from the DoS blacklist and the drop list, but if you don't get it squared away this will keep happening. Hopefully you can read this message before getting blacklisted again...
manOFice
30-04-2008 11:46:02
pqdrummer so took down the site, ban him!!
dmorris68
30-04-2008 12:40:29
And it just happened again...
pqdrummer, I don't know if you'll see this soon or not, I may have to temporarily disable the blacklist logic, which should be okay for now since the attack seems to have stopped again. Do you have your browser configured differently than the default, with respect to opening a lot of simultaneous connections to a server? Because looking at the access logs, your browser is trying to pull a lot of resources (mostly images) simultaneously -- I mean, I noted about 10 server hits within one second, which is enough to trigger the DoS code to block you. Normally folks aren't pulling that much from the server with simultaneous connections, so I'm wondering if you've been tweaking your browser?
bballp6699
30-04-2008 12:45:58
One less user in the name of a safer forum. I'm all for it...
;)
J4320
30-04-2008 12:53:17
But he's my top post reporter. cry
TryinToGetPaid
30-04-2008 15:55:04
I do have a man crush on Dwight, he makes me fire rise.
hehehhehe
30-04-2008 17:21:38
[quotecca254f8be="bballp6699"]One less user in the name of a safer forum. I'm all for it...
;)[/quotecca254f8be]
True, we may have to say goodbye to him for the greater good.
Admin
01-05-2008 04:03:54
[quote16508fccde="dmorris68"]And it just happened again...
pqdrummer, I don't know if you'll see this soon or not, I may have to temporarily disable the blacklist logic, which should be okay for now since the attack seems to have stopped again. Do you have your browser configured differently than the default, with respect to opening a lot of simultaneous connections to a server? Because looking at the access logs, your browser is trying to pull a lot of resources (mostly images) simultaneously -- I mean, I noted about 10 server hits within one second, which is enough to trigger the DoS code to block you. Normally folks aren't pulling that much from the server with simultaneous connections, so I'm wondering if you've been tweaking your browser?[/quote16508fccde]he's probably running
this abusive POS[=https//addons.mozilla.org/firefox/addon/1269]this abusive POS
pqdrummer
01-05-2008 05:46:27
Well I don't know what's going on, but if it was in fact me causing the problems, I'm sorry ( I do have some addons (I'm using FF), not not FasterFox. I have contexthelper (honestly don't know what that is, don't remember adding it), Firebug (need that for work), Forecasefox, IE Tab, RealPlayer Browser Recoreder Plugin, Skype extension, StumbleUpon, Web Developer, and Winzy Toolbar. I'm not sure how any of those would be hitting you guys particularly hard, but I can remove a few of them if I need to.
And I run a full system virus scan every day.
dmorris68
01-05-2008 06:30:33
Don't worry, you weren't responsible alone for the server problems -- you were probably doing this all along and nobody noticed until I had to reconfigure the server after this attack -- it would be pretty hard for one PC to take down a site like this. But afterwards you fit the same profile as the DDOS bots, and thus were continuously blacklisted. So the problem is now affecting you, not us. ) Just be aware that if the attack starts back up, I'll probably re-enable the DDoS protection and you may get dropped again. But I'll see if I can whitelist your IP.
I don't think any of those extensions you indicated should effect how many open simultaneous connections your browser is making. Are you sure you haven't read one of those Firefox "optimization" tutorials that walk you through making tweaks in aboutconfig? Because it sure sounds like your browser has ratcheted up its simultaneous connection limit. Same thing the FasterFox extension that Admin mentioned does. Those kinds of setting can really oversaturate a server, especially when lots of people are using them. The minor perception of improved performance on the browser end can take a much larger toll on the servers you hit, particularly the small operations like ours.
EDIT Have you perhaps ever run Firetune? It makes similar tweaks to your Firefox config to improve browsing performance, but doesn't run as an extension.
EatChex89
01-05-2008 08:02:53
[quote9b35c84224="Admin"][quote9b35c84224="dmorris68"]And it just happened again...
pqdrummer, I don't know if you'll see this soon or not, I may have to temporarily disable the blacklist logic, which should be okay for now since the attack seems to have stopped again. Do you have your browser configured differently than the default, with respect to opening a lot of simultaneous connections to a server? Because looking at the access logs, your browser is trying to pull a lot of resources (mostly images) simultaneously -- I mean, I noted about 10 server hits within one second, which is enough to trigger the DoS code to block you. Normally folks aren't pulling that much from the server with simultaneous connections, so I'm wondering if you've been tweaking your browser?[/quote9b35c84224]he's probably running
this abusive POS[=https//addons.mozilla.org/firefox/addon/1269]this abusive POS[/quote9b35c84224]
i use that and my ip hasn't been blacklisted shrug
pqdrummer
01-05-2008 13:15:27
I'm not sure if it's related to, or the cause of, my problem, but it takes a god-awefully long time for my browser to load all of the damn images an icons in these threads and it almost looks like it's making a new connection for each one. I'm a coder, not a networker, so I don't know how all that stuff works, but maybe it's a lead for me to follow?
TravMan162
01-05-2008 13:25:06
[quotefb4607bd6d="J4320"]But he's my top post reporter. cry[/quotefb4607bd6d]
that used to be me cry
I've been replaced (
gafdpc
01-05-2008 13:50:15
[quote5a05570f33="pqdrummer"]I'm not sure if it's related to, or the cause of, my problem, but it takes a god-awefully long time for my browser to load all of the damn images an icons in these threads and it almost looks like it's making a new connection for each one. I'm a coder, not a networker, so I don't know how all that stuff works, but maybe it's a lead for me to follow?[/quote5a05570f33]
Go to "aboutconfig". In the filter type "network.http". Check your value for network.http.max-connections-per-server.
J4320
01-05-2008 14:56:04
[quote5f7fea3003="TravMan162"][quote5f7fea3003="J4320"]But he's my top post reporter. cry[/quote5f7fea3003]
that used to be me cry
I've been replaced ([/quote5f7fea3003]
Report moar and you will once again gain the throne of the reporter. twisted
J4320
01-05-2008 20:15:37
Are we pretty much in the clear now? Shall I commence the sticky action?
YourGiftsFree
01-05-2008 20:53:48
[quote5144dbb94a="J4320"]Are we pretty much in the clear now? Shall I commence the sticky action?[/quote5144dbb94a]
Look at this StickyMod impersonator. Only SitckyMod can do that... nub.
samz465
01-05-2008 21:13:09
[quote4a1d52d7ce="J4320"]Are we pretty much in the clear now? Shall I commence the sticky action?[/quote4a1d52d7ce]
Please commence sticky action asap.