is it just me, or ...

[Grateful Diver]

... has this site been loading really s ... l ... o ... w lately?

Sometimes it makes me want to get out and push ...

... Bob (Grateful Diver)

[Aquanautchuck]

I haven't noticed. But I did notice the nice sun glasses on your avatar.

[spatman]

i find that sometimes it takes forever, and then a few minutes later it's fine. totally unpredictable...

[defied]

i find that sometimes it takes forever, and then a few minutes later it's fine. totally unpredictable...

Yeah, that's how sporadic SQL injection attacks go sometimes.

D(B)

[Nwbrewer]

I've noticed it too.

[Jenbowes]

... has this site been loading really s ... l ... o ... w lately?

Sometimes it makes me want to get out and push ...

... Bob (Grateful Diver)

Yeah, it makes me feel like this:

[scottsax]

ROFL! Great pic, Jen!

I have the same issue. Sometimes quick as bunny, sometimes slow like molasses....

[Dashrynn]

perhaps it is a bandwidth issue? comcast somtimes blows lingcod. or the excessive image posting?

[defied]

If that were the case, it would probably affect the front page as well. I've noticed the frontpage (which is fairly static) loads just fine, but any thread calls (heavy DB usage) seems to lag the system.

So it's either an old a$$ server, or it's getting slammed with connection attempts.

Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.

D(B)

[Grateful Diver]

If that were the case, it would probably affect the front page as well. I've noticed the frontpage (which is fairly static) loads just fine, but any thread calls (heavy DB usage) seems to lag the system.

So it's either an old a$$ server, or it's getting slammed with connection attempts.

Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.

D(B)

Correct ... this morning I hit New Posts and went for coffee while it was loading ...

... Bob (Grateful Diver)

[lamont]

Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.

a lot of different things can be the bottle neck in a web system though.

the complexity of figuring out performance issues is half of what keeps me gainfully employed.

[defied]

Thanks Bob,

a lot of different things can be the bottle neck in a web system though.

the complexity of figuring out performance issues is half of what keeps me gainfully employed.

This is also true. So here are the datapoints:

Many people are experiencing. Not all are on comcast. Some are experiencing this behavior from work, or other locations that have low latency connections.

Web server queries respond within milliseconds/seconds which means the web server appears to be running appropriately.

DB queries seem to take some time to complete a query request.

What could this mean?
If the DB is exposed to the outside world, there are thousands of portscanners out there who could be launching distributed DB queries across the wire, and causing it to become latent . (This is an assumption based off of the past history of DB connection request exceeded errors that occur everytime this site gets super slow.

If the webserver is performing php sql queries on a backend, localnet, or a remote SQL server somewhere, latency could be occuring on that line. Now, by running a port scan, I do not see a common sql port open on the site, so this could be the case.

(The 1652 ports scanned but not shown below are in state: filtered)
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
25/tcp open smtp
80/tcp open http
113/tcp open auth
548/tcp open afpovertcp
587/tcp open submission
718/tcp open unknown

However, the SQL port can be changed wicked easy like, so that doesn't tell us much.

Another option is cpu resources on the SQL server itself could be having issues. (Hi load from other resources requesting transactional data, hardware/software issues, mem leaks, etc)

Now I did say that this was an assumption. I stick by it because I think it's more than likely.

Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.

Creating better performance for your networks in general is half of what keeps me gainfully employed. 0]
Beating Cisco in the market for the last two years running, has allowed me to continue to be employed.

D(B)

[spatman]

i like tacos.

beer, too.

let's ride bikes!

[Sockmonkey]

i like tacos.

beer, too.

let's ride bikes!

If you don't have a bicycle when you move to pdx do they allow you a grace period to purchase one before escorting you to the city limits?

-Eric

[spatman]

i like tacos.

beer, too.

let's ride bikes!

If you don't have a bicycle when you move to pdx do they allow you a grace period to purchase one before escorting you to the city limits?

-Eric

as long as you like beer, they let the bike thing slide. just don't get caught without a beer, though. it can get ugly.

[Dashrynn]

i like tacos.

beer, too.

let's ride bikes!

If you don't have a bicycle when you move to pdx do they allow you a grace period to purchase one before escorting you to the city limits?

-Eric

as long as you like beer, they let the bike thing slide. just don't get caught without a beer, though. it can get ugly.

i like potatos and gummy worms. Bicycles suck get a dual sport.

[kat]

been slow for me for about a week. although on my mom's dial up it seemed to load pretty fast. as far as 48k goes...

[dwashbur]

Mine's been slow for several days. I'm on Comcast, but considering how well most other sites load, I don't think they're the culprit.

[lamont]

Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.

yup.

one thing that can cause slowness is simply putting the webserver, appserver and database server on the same physical server and contention can start to run the server out of RAM, CPU or disk and it may be appropriate to add another server in order to increase the capacity.

if static images and light http content are being served quickly, it still doesn't stop the application itself from having issues (app tier locking issues, while static images are served quickly). one thing which masquerades as a database issue is having a database connection pool limit which is too small which causes DB queries to serialize on the app tier (which produces an idle database). if you don't have enough workers for application queries you can also serialize the thick application tier http requests while the light requests get served by the webserver directly. on larger SOA websites based on java where the SDEs are using the apache java httpclient libraries there's a bit of idiocy in there which the maxconnsperserver setting is set to 2 which means that any given tier1 appserver can only issue 2 simultaneous requests to the tier2 server behind it which causes a similar throttling issue. once i've found that memory, cpu, disk and network aren't being saturated during a performance issue, resource saturation issues with threads/workers/pools is a very common issue.

on the database tier, you may need more memory or cpu or disk. most of the time you should probably need more disk, in which case you can't beat modern RAID arrays with lots of battery cache (256MB-512MB) and 6x15k SAS drives in a RAID10 array -- doing something like software RAID or using some crappy SATA RAID controller will produce significantly slower performance even with the same drives. for about $4k you should be able to put together a system today with a disk config like that with 16GB of RAM and at least a single-quad processor which should make the database server scream.

you could conceivably need more networking bandwidth, but websites typically only use a few 10s of Mbits per server and GigE is orders of magnitudes faster than that (and if light transactions and images are quick like you said, that indicates that transit bandwidth is probably sufficient, which is where the pinch point should be in the network if it isn't a mess -- which it could be -- could always be a duplex problem or physical cabling issue between the app/web tier and the db server).

and a more common performance problem than scriptkiddie portscans and probes is webcrawlers walking through the site and doing deep queries on completely uncached rarely-accessed objects.

[Dashrynn]

Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.

yup.

one thing that can cause slowness is simply putting the webserver, appserver and database server on the same physical server and contention can start to run the server out of RAM, CPU or disk and it may be appropriate to add another server in order to increase the capacity.

if static images and light http content are being served quickly, it still doesn't stop the application itself from having issues (app tier locking issues, while static images are served quickly). one thing which masquerades as a database issue is having a database connection pool limit which is too small which causes DB queries to serialize on the app tier (which produces an idle database). if you don't have enough workers for application queries you can also serialize the thick application tier http requests while the light requests get served by the webserver directly. on larger SOA websites based on java where the SDEs are using the apache java httpclient libraries there's a bit of idiocy in there which the maxconnsperserver setting is set to 2 which means that any given tier1 appserver can only issue 2 simultaneous requests to the tier2 server behind it which causes a similar throttling issue. once i've found that memory, cpu, disk and network aren't being saturated during a performance issue, resource saturation issues with threads/workers/pools is a very common issue.

on the database tier, you may need more memory or cpu or disk. most of the time you should probably need more disk, in which case you can't beat modern RAID arrays with lots of battery cache (256MB-512MB) and 6x15k SAS drives in a RAID10 array -- doing something like software RAID or using some crappy SATA RAID controller will produce significantly slower performance even with the same drives. for about $4k you should be able to put together a system today with a disk config like that with 16GB of RAM and at least a single-quad processor which should make the database server scream.

you could conceivably need more networking bandwidth, but websites typically only use a few 10s of Mbits per server and GigE is orders of magnitudes faster than that (and if light transactions and images are quick like you said, that indicates that transit bandwidth is probably sufficient, which is where the pinch point should be in the network if it isn't a mess -- which it could be -- could always be a duplex problem or physical cabling issue between the app/web tier and the db server).

and a more common performance problem than scriptkiddie portscans and probes is webcrawlers walking through the site and doing deep queries on completely uncached rarely-accessed objects.

all i read was "im more nerdier than the rest of you and heres a long list to prove me right"

[defied]

one thing that can cause slowness is simply putting the webserver, appserver and database server on the same physical server and contention can start to run the server out of RAM, CPU or disk and it may be appropriate to add another server in order to increase the capacity.

Yes. That goes along with what was said about the DB being taxed on resources. Because this is a hosting service, I would hope that they are seperating their DB's from their webservers. For webhosts like this, it is usually SOP. This does not mean that the databases are kept clean, and up to date. Many times I've come across DB's with a ton of cruft from ages gone by.

if static images and light http content are being served quickly, it still doesn't stop the application itself from having issues (app tier locking issues, while static images are served quickly). one thing which masquerades as a database issue is having a database connection pool limit which is too small which causes DB queries to serialize on the app tier (which produces an idle database). if you don't have enough workers for application queries you can also serialize the thick application tier http requests while the light requests get served by the webserver directly. on larger SOA websites based on java where the SDEs are using the apache java httpclient libraries there's a bit of idiocy in there which the maxconnsperserver setting is set to 2 which means that any given tier1 appserver can only issue 2 simultaneous requests to the tier2 server behind it which causes a similar throttling issue. once i've found that memory, cpu, disk and network aren't being saturated during a performance issue, resource saturation issues with threads/workers/pools is a very common issue.

Network support!!!! Network Support!!!! Fix yo sh!t!!!!

on the database tier, you may need more memory or cpu or disk. most of the time you should probably need more disk, in which case you can't beat modern RAID arrays with lots of battery cache (256MB-512MB) and 6x15k SAS drives in a RAID10 array -- doing something like software RAID or using some crappy SATA RAID controller will produce significantly slower performance even with the same drives. for about $4k you should be able to put together a system today with a disk config like that with 16GB of RAM and at least a single-quad processor which should make the database server scream.

I would have said RAID5,0 personally, but I get your point. I'm not sure entirely why your quoting hardware types here. This has been discussed already. I could say this could all be resolved by incorporating an ARX to frontend old school backend storage, not to mention throwing in a couple of netapp boxes to handle your current content, and a data domain to keep you backups...
We could go ahead and multiply the servers over a farm, and front end it all with a BigIP 6900 performing load balancing and rate shaping to ensure traffic is delivered, and responded to within a decided upon amount of time. The configuration options are unlimited.
By using that configuration, you could build out 3000 386's running linux on a ramdisk, and deliver results faster than this site is doing right now. But that requires a boat load of physical space for such low singular processing power.
I'm sure we could google something better. 0]

you could conceivably need more networking bandwidth, but websites typically only use a few 10s of Mbits per server and GigE is orders of magnitudes faster than that (and if light transactions and images are quick like you said, that indicates that transit bandwidth is probably sufficient, which is where the pinch point should be in the network if it isn't a mess -- which it could be -- could always be a duplex problem or physical cabling issue between the app/web tier and the db server).

Websites that are not virtualised across one server, sure. You can, however, tax the network, and the cycles of a server that is hosting multiple virtuals, as most webhosting sites such as dreamhost commonly do.

and a more common performance problem than scriptkiddie portscans and probes is webcrawlers walking through the site and doing deep queries on completely uncached rarely-accessed objects.

[/quote]
Absolutely, however worms have been very prominent, and have continued to be invasive, even though we have been knocking them out of the trees when we find them. When a distributed worm decides it wants to work hard on a system, and calls all of it's cloned buddies, it will slow your sh!t faster than a legit webcrawler will, any day of the week.
And I wouldn't really consider these worms to be script kiddie any more. Those are days of old. There are corporations, and very lucrative money making "gangs" hiring professional developers for this kind of malware these days.

D(B)

[Gooch]

I have to admit- I wasn't going to complain but lately, this site does seem to be bogging down a lot. When I first hit the page and especially when you do the canned queries like "all new posts" or "show your posts" it goes into a granny low where you can go make an espresso and come back for the results. I blame it on all the fighting Josh started with that tec vs rec thread. Nice one....

[LCF]

Wow -- there's something seriously wrong with me. I read the back-and-forth between defied and lamont and find it irresistibly sexy.

I don't understand it, mind you . . . but there are some seriously smart people at work there.

[defied]

Wow -- there's something seriously wrong with me. I read the back-and-forth between defied and lamont and find it irresistibly sexy.

I don't understand it, mind you . . . but there are some seriously smart people at work there.

Nah, I was just guessing. >0]

D(B)

[lamont]

I would have said RAID5,0 personally, but I get your point. I'm not sure entirely why your quoting hardware types here. This has been discussed already. I could say this could all be resolved by incorporating an ARX to frontend old school backend storage, not to mention throwing in a couple of netapp boxes to handle your current content, and a data domain to keep you backups...

NetApps are awesome, but 'spensive.

One guy on IRC commented one time that EMC shops are full of people who are broke and are constantly responding to pagers, shops with NetApps are equally as broke, but they get to sleep all night long.

They're the only piece of equipment I think I've come across that I think is really four 9's, maybe five...

We could go ahead and multiply the servers over a farm, and front end it all with a BigIP 6900 performing load balancing and rate shaping to ensure traffic is delivered, and responded to within a decided upon amount of time. The configuration options are unlimited.

I'd say a Netscaler 17000 instead of BigIP.

Google runs on Netscaler and all the DoS problems we had at Amazon (>1 Gbps syn flood attacks) were absorbed by the Netscalers once we switched to them. I was actually in the process of writing a lightweight linux-based software proxy using epoll() to absorb the SYN flood attack with stacks of linux boxes in front of our Cisco load balancers right before Amazon pulled the trigger on Netscalers. I immediately stopped seeing all the DoS traffic that used to affect the webservers after the Netscalers were dropped in, and didn't see them again until I switched jobs and then had to deal with these effing layer-4 LVS servers that got DoS'd by garbage caused by 'normal' traffic from the Great Firewall of China (nobody who built that in China seems to care about RFC standards).

Websites that are not virtualised across one server, sure. You can, however, tax the network, and the cycles of a server that is hosting multiple virtuals, as most webhosting sites such as dreamhost commonly do.

I don't know what kind of compression they're running, but at work we only get about 10:1 compression, tops, which is still only about 100Mbps peak utilization of the network to any virt host.

We're thinking of going to 4900M's or equivalent switches from Juniper and running a complete 10Gig access layer now (which i think is back in the regime of massive overkill).

Anyway, if this is a hosted service, then it really should be changed to a different hosting service... I could probably run this better out of my server at home on a DSL line based on how well its been doing lately, and i figured this was just Josh's or someone's pet project -- but if this is a pay-for service, that is kind of bad...

Actually, thing to do would probably be to just run it in EC2 and rent space from Amazon... That's also virtualized, but the servers there are not as oversubscribed and you really get a guarantee of how much RAM and CPU horsepower you get...

Absolutely, however worms have been very prominent, and have continued to be invasive, even though we have been knocking them out of the trees when we find them. When a distributed worm decides it wants to work hard on a system, and calls all of it's cloned buddies, it will slow your sh!t faster than a legit webcrawler will, any day of the week.
And I wouldn't really consider these worms to be script kiddie any more. Those are days of old. There are corporations, and very lucrative money making "gangs" hiring professional developers for this kind of malware these days.

Yeah, clearly if you do get smacked by a worm, it isn't going to obey the kinds of courtesy in terms of hits per second that the Google webcrawlers at least attempt to...

And its Russian mafia and other criminal organizations operating outside of the reach of USA law enforcement which is responsible for a lot of the blackmailing operations and attacking people for profit...

all i read was "im more nerdier than the rest of you and heres a long list to prove me right"

I believe you read it correctly then... I must establish myself as the Dominant AlphaGeek when it comes to system engineering...