is it just me, or ...
- Grateful Diver
- I've Got Gills
- Posts: 5322
- Joined: Mon May 15, 2006 7:52 pm
is it just me, or ...
... has this site been loading really s ... l ... o ... w lately?
Sometimes it makes me want to get out and push ...
... Bob (Grateful Diver)
Sometimes it makes me want to get out and push ...
... Bob (Grateful Diver)
Threats and ultimatums are never the best answer. Public humiliation via Photoshop is always better - airsix
Come visit me at http://www.nwgratefuldiver.com/
Come visit me at http://www.nwgratefuldiver.com/
- Aquanautchuck
- Pelagic
- Posts: 919
- Joined: Thu Jun 01, 2006 4:33 pm
Re: is it just me, or ...
I haven't noticed. But I did notice the nice sun glasses on your avatar.
Last edited by Aquanautchuck on Thu Oct 29, 2009 12:07 pm, edited 1 time in total.
Charles
Re: is it just me, or ...
i find that sometimes it takes forever, and then a few minutes later it's fine. totally unpredictable...
Re: is it just me, or ...
Yeah, that's how sporadic SQL injection attacks go sometimes.spatman wrote:i find that sometimes it takes forever, and then a few minutes later it's fine. totally unpredictable...
D(B)
Re: is it just me, or ...
I've noticed it too.
"Screw "annual" service,... I get them serviced when they break." - CaptnJack (paraphrased)
"you do realize you're supposed to mix the with water and drink it, not snort the powder directly from the packet, right? " - Spatman
"you do realize you're supposed to mix the with water and drink it, not snort the powder directly from the packet, right? " - Spatman
Re: is it just me, or ...
Yeah, it makes me feel like this:Grateful Diver wrote:... has this site been loading really s ... l ... o ... w lately?
Sometimes it makes me want to get out and push ...
... Bob (Grateful Diver)
Re: is it just me, or ...
ROFL! Great pic, Jen!
I have the same issue. Sometimes quick as bunny, sometimes slow like molasses....
I have the same issue. Sometimes quick as bunny, sometimes slow like molasses....
I'm going to look like a moose on rollerskates. -airsix
... my Mom caught me fenestrating once. -lavachickie
And I get so tired of fainting and peeing all over myself when the hammer falls on an empty chamber! -Nailer
Want to know where I'm performing? Check out my Facebook fan page!
... my Mom caught me fenestrating once. -lavachickie
And I get so tired of fainting and peeing all over myself when the hammer falls on an empty chamber! -Nailer
Want to know where I'm performing? Check out my Facebook fan page!
Re: is it just me, or ...
perhaps it is a bandwidth issue? comcast somtimes blows lingcod. or the excessive image posting?
Re: is it just me, or ...
If that were the case, it would probably affect the front page as well. I've noticed the frontpage (which is fairly static) loads just fine, but any thread calls (heavy DB usage) seems to lag the system.
So it's either an old a$$ server, or it's getting slammed with connection attempts.
Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.
D(B)
So it's either an old a$$ server, or it's getting slammed with connection attempts.
Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.
D(B)
- Grateful Diver
- I've Got Gills
- Posts: 5322
- Joined: Mon May 15, 2006 7:52 pm
Re: is it just me, or ...
Correct ... this morning I hit New Posts and went for coffee while it was loading ...defied wrote:If that were the case, it would probably affect the front page as well. I've noticed the frontpage (which is fairly static) loads just fine, but any thread calls (heavy DB usage) seems to lag the system.
So it's either an old a$$ server, or it's getting slammed with connection attempts.
Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.
D(B)
... Bob (Grateful Diver)
Threats and ultimatums are never the best answer. Public humiliation via Photoshop is always better - airsix
Come visit me at http://www.nwgratefuldiver.com/
Come visit me at http://www.nwgratefuldiver.com/
Re: is it just me, or ...
a lot of different things can be the bottle neck in a web system though.defied wrote: Now, I've watched, and seen 15 users, and it's slow, and I've seen 3 users, and it's slow, so I'm only speculating here, but I would assume the DB is being queried a LOT.
the complexity of figuring out performance issues is half of what keeps me gainfully employed.
Re: is it just me, or ...
Thanks Bob,
Many people are experiencing. Not all are on comcast. Some are experiencing this behavior from work, or other locations that have low latency connections.
Web server queries respond within milliseconds/seconds which means the web server appears to be running appropriately.
DB queries seem to take some time to complete a query request.
What could this mean?
If the DB is exposed to the outside world, there are thousands of portscanners out there who could be launching distributed DB queries across the wire, and causing it to become latent . (This is an assumption based off of the past history of DB connection request exceeded errors that occur everytime this site gets super slow.
If the webserver is performing php sql queries on a backend, localnet, or a remote SQL server somewhere, latency could be occuring on that line. Now, by running a port scan, I do not see a common sql port open on the site, so this could be the case.
(The 1652 ports scanned but not shown below are in state: filtered)
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
25/tcp open smtp
80/tcp open http
113/tcp open auth
548/tcp open afpovertcp
587/tcp open submission
718/tcp open unknown
However, the SQL port can be changed wicked easy like, so that doesn't tell us much.
Another option is cpu resources on the SQL server itself could be having issues. (Hi load from other resources requesting transactional data, hardware/software issues, mem leaks, etc)
Now I did say that this was an assumption. I stick by it because I think it's more than likely.
Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.
Creating better performance for your networks in general is half of what keeps me gainfully employed. 0]
Beating Cisco in the market for the last two years running, has allowed me to continue to be employed.
D(B)
This is also true. So here are the datapoints:lamont wrote: a lot of different things can be the bottle neck in a web system though.
the complexity of figuring out performance issues is half of what keeps me gainfully employed.
Many people are experiencing. Not all are on comcast. Some are experiencing this behavior from work, or other locations that have low latency connections.
Web server queries respond within milliseconds/seconds which means the web server appears to be running appropriately.
DB queries seem to take some time to complete a query request.
What could this mean?
If the DB is exposed to the outside world, there are thousands of portscanners out there who could be launching distributed DB queries across the wire, and causing it to become latent . (This is an assumption based off of the past history of DB connection request exceeded errors that occur everytime this site gets super slow.
If the webserver is performing php sql queries on a backend, localnet, or a remote SQL server somewhere, latency could be occuring on that line. Now, by running a port scan, I do not see a common sql port open on the site, so this could be the case.
(The 1652 ports scanned but not shown below are in state: filtered)
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
25/tcp open smtp
80/tcp open http
113/tcp open auth
548/tcp open afpovertcp
587/tcp open submission
718/tcp open unknown
However, the SQL port can be changed wicked easy like, so that doesn't tell us much.
Another option is cpu resources on the SQL server itself could be having issues. (Hi load from other resources requesting transactional data, hardware/software issues, mem leaks, etc)
Now I did say that this was an assumption. I stick by it because I think it's more than likely.
Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.
Creating better performance for your networks in general is half of what keeps me gainfully employed. 0]
Beating Cisco in the market for the last two years running, has allowed me to continue to be employed.
D(B)
Last edited by defied on Thu Oct 29, 2009 4:37 pm, edited 1 time in total.
- Sockmonkey
- I've Got Gills
- Posts: 1467
- Joined: Sat Mar 15, 2008 9:43 am
Re: is it just me, or ...
If you don't have a bicycle when you move to pdx do they allow you a grace period to purchase one before escorting you to the city limits?spatman wrote:i like tacos.
beer, too.
let's ride bikes!
-Eric
Re: is it just me, or ...
as long as you like beer, they let the bike thing slide. just don't get caught without a beer, though. it can get ugly.Sockmonkey wrote:If you don't have a bicycle when you move to pdx do they allow you a grace period to purchase one before escorting you to the city limits?spatman wrote:i like tacos.
beer, too.
let's ride bikes!
-Eric
Re: is it just me, or ...
i like potatos and gummy worms. Bicycles suck get a dual sport.spatman wrote:as long as you like beer, they let the bike thing slide. just don't get caught without a beer, though. it can get ugly.Sockmonkey wrote:If you don't have a bicycle when you move to pdx do they allow you a grace period to purchase one before escorting you to the city limits?spatman wrote:i like tacos.
beer, too.
let's ride bikes!
-Eric
Re: is it just me, or ...
been slow for me for about a week. although on my mom's dial up it seemed to load pretty fast. as far as 48k goes...
"As long as you get everything tucked into your pants, you are good to go." -wasp7000
Re: is it just me, or ...
Mine's been slow for several days. I'm on Comcast, but considering how well most other sites load, I don't think they're the culprit.
Dave
"Clearly, you weren't listening to what I'm about to say."
--
Check out my Internet show:
http://www.irvingszoo.com
"Clearly, you weren't listening to what I'm about to say."
--
Check out my Internet show:
http://www.irvingszoo.com
Re: is it just me, or ...
yup.defied wrote: Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.
one thing that can cause slowness is simply putting the webserver, appserver and database server on the same physical server and contention can start to run the server out of RAM, CPU or disk and it may be appropriate to add another server in order to increase the capacity.
if static images and light http content are being served quickly, it still doesn't stop the application itself from having issues (app tier locking issues, while static images are served quickly). one thing which masquerades as a database issue is having a database connection pool limit which is too small which causes DB queries to serialize on the app tier (which produces an idle database). if you don't have enough workers for application queries you can also serialize the thick application tier http requests while the light requests get served by the webserver directly. on larger SOA websites based on java where the SDEs are using the apache java httpclient libraries there's a bit of idiocy in there which the maxconnsperserver setting is set to 2 which means that any given tier1 appserver can only issue 2 simultaneous requests to the tier2 server behind it which causes a similar throttling issue. once i've found that memory, cpu, disk and network aren't being saturated during a performance issue, resource saturation issues with threads/workers/pools is a very common issue.
on the database tier, you may need more memory or cpu or disk. most of the time you should probably need more disk, in which case you can't beat modern RAID arrays with lots of battery cache (256MB-512MB) and 6x15k SAS drives in a RAID10 array -- doing something like software RAID or using some crappy SATA RAID controller will produce significantly slower performance even with the same drives. for about $4k you should be able to put together a system today with a disk config like that with 16GB of RAM and at least a single-quad processor which should make the database server scream.
you could conceivably need more networking bandwidth, but websites typically only use a few 10s of Mbits per server and GigE is orders of magnitudes faster than that (and if light transactions and images are quick like you said, that indicates that transit bandwidth is probably sufficient, which is where the pinch point should be in the network if it isn't a mess -- which it could be -- could always be a duplex problem or physical cabling issue between the app/web tier and the db server).
and a more common performance problem than scriptkiddie portscans and probes is webcrawlers walking through the site and doing deep queries on completely uncached rarely-accessed objects.
Re: is it just me, or ...
all i read was "im more nerdier than the rest of you and heres a long list to prove me right"lamont wrote:yup.defied wrote: Without having access to the host to investigate what is occurring.... It's always just going to be an assumption.
one thing that can cause slowness is simply putting the webserver, appserver and database server on the same physical server and contention can start to run the server out of RAM, CPU or disk and it may be appropriate to add another server in order to increase the capacity.
if static images and light http content are being served quickly, it still doesn't stop the application itself from having issues (app tier locking issues, while static images are served quickly). one thing which masquerades as a database issue is having a database connection pool limit which is too small which causes DB queries to serialize on the app tier (which produces an idle database). if you don't have enough workers for application queries you can also serialize the thick application tier http requests while the light requests get served by the webserver directly. on larger SOA websites based on java where the SDEs are using the apache java httpclient libraries there's a bit of idiocy in there which the maxconnsperserver setting is set to 2 which means that any given tier1 appserver can only issue 2 simultaneous requests to the tier2 server behind it which causes a similar throttling issue. once i've found that memory, cpu, disk and network aren't being saturated during a performance issue, resource saturation issues with threads/workers/pools is a very common issue.
on the database tier, you may need more memory or cpu or disk. most of the time you should probably need more disk, in which case you can't beat modern RAID arrays with lots of battery cache (256MB-512MB) and 6x15k SAS drives in a RAID10 array -- doing something like software RAID or using some crappy SATA RAID controller will produce significantly slower performance even with the same drives. for about $4k you should be able to put together a system today with a disk config like that with 16GB of RAM and at least a single-quad processor which should make the database server scream.
you could conceivably need more networking bandwidth, but websites typically only use a few 10s of Mbits per server and GigE is orders of magnitudes faster than that (and if light transactions and images are quick like you said, that indicates that transit bandwidth is probably sufficient, which is where the pinch point should be in the network if it isn't a mess -- which it could be -- could always be a duplex problem or physical cabling issue between the app/web tier and the db server).
and a more common performance problem than scriptkiddie portscans and probes is webcrawlers walking through the site and doing deep queries on completely uncached rarely-accessed objects.
Re: is it just me, or ...
Yes. That goes along with what was said about the DB being taxed on resources. Because this is a hosting service, I would hope that they are seperating their DB's from their webservers. For webhosts like this, it is usually SOP. This does not mean that the databases are kept clean, and up to date. Many times I've come across DB's with a ton of cruft from ages gone by.lamont wrote: one thing that can cause slowness is simply putting the webserver, appserver and database server on the same physical server and contention can start to run the server out of RAM, CPU or disk and it may be appropriate to add another server in order to increase the capacity.
Network support!!!! Network Support!!!! Fix yo sh!t!!!!lamont wrote: if static images and light http content are being served quickly, it still doesn't stop the application itself from having issues (app tier locking issues, while static images are served quickly). one thing which masquerades as a database issue is having a database connection pool limit which is too small which causes DB queries to serialize on the app tier (which produces an idle database). if you don't have enough workers for application queries you can also serialize the thick application tier http requests while the light requests get served by the webserver directly. on larger SOA websites based on java where the SDEs are using the apache java httpclient libraries there's a bit of idiocy in there which the maxconnsperserver setting is set to 2 which means that any given tier1 appserver can only issue 2 simultaneous requests to the tier2 server behind it which causes a similar throttling issue. once i've found that memory, cpu, disk and network aren't being saturated during a performance issue, resource saturation issues with threads/workers/pools is a very common issue.
I would have said RAID5,0 personally, but I get your point. I'm not sure entirely why your quoting hardware types here. This has been discussed already. I could say this could all be resolved by incorporating an ARX to frontend old school backend storage, not to mention throwing in a couple of netapp boxes to handle your current content, and a data domain to keep you backups...lamont wrote: on the database tier, you may need more memory or cpu or disk. most of the time you should probably need more disk, in which case you can't beat modern RAID arrays with lots of battery cache (256MB-512MB) and 6x15k SAS drives in a RAID10 array -- doing something like software RAID or using some crappy SATA RAID controller will produce significantly slower performance even with the same drives. for about $4k you should be able to put together a system today with a disk config like that with 16GB of RAM and at least a single-quad processor which should make the database server scream.
We could go ahead and multiply the servers over a farm, and front end it all with a BigIP 6900 performing load balancing and rate shaping to ensure traffic is delivered, and responded to within a decided upon amount of time. The configuration options are unlimited.
By using that configuration, you could build out 3000 386's running linux on a ramdisk, and deliver results faster than this site is doing right now. But that requires a boat load of physical space for such low singular processing power.
I'm sure we could google something better. 0]
Websites that are not virtualised across one server, sure. You can, however, tax the network, and the cycles of a server that is hosting multiple virtuals, as most webhosting sites such as dreamhost commonly do.lamont wrote: you could conceivably need more networking bandwidth, but websites typically only use a few 10s of Mbits per server and GigE is orders of magnitudes faster than that (and if light transactions and images are quick like you said, that indicates that transit bandwidth is probably sufficient, which is where the pinch point should be in the network if it isn't a mess -- which it could be -- could always be a duplex problem or physical cabling issue between the app/web tier and the db server).
[/quote]lamont wrote: and a more common performance problem than scriptkiddie portscans and probes is webcrawlers walking through the site and doing deep queries on completely uncached rarely-accessed objects.
Absolutely, however worms have been very prominent, and have continued to be invasive, even though we have been knocking them out of the trees when we find them. When a distributed worm decides it wants to work hard on a system, and calls all of it's cloned buddies, it will slow your sh!t faster than a legit webcrawler will, any day of the week.
And I wouldn't really consider these worms to be script kiddie any more. Those are days of old. There are corporations, and very lucrative money making "gangs" hiring professional developers for this kind of malware these days.
D(B)
Re: is it just me, or ...
I have to admit- I wasn't going to complain but lately, this site does seem to be bogging down a lot. When I first hit the page and especially when you do the canned queries like "all new posts" or "show your posts" it goes into a granny low where you can go make an espresso and come back for the results. I blame it on all the fighting Josh started with that tec vs rec thread. Nice one....
http://nwdivers.me/blog/ Original articles and dive reports from local divers in the Vancouver, WA area. Suggestions for stories or your own reports are welcome!
Re: is it just me, or ...
Wow -- there's something seriously wrong with me. I read the back-and-forth between defied and lamont and find it irresistibly sexy.
I don't understand it, mind you . . . but there are some seriously smart people at work there.
I don't understand it, mind you . . . but there are some seriously smart people at work there.
"Sometimes, when your world is going sideways, the second best thing to everything working out right, is knowing you are loved..." ljjames
Re: is it just me, or ...
Nah, I was just guessing. >0]LCF wrote:Wow -- there's something seriously wrong with me. I read the back-and-forth between defied and lamont and find it irresistibly sexy.
I don't understand it, mind you . . . but there are some seriously smart people at work there.
D(B)
Re: is it just me, or ...
NetApps are awesome, but 'spensive.defied wrote: I would have said RAID5,0 personally, but I get your point. I'm not sure entirely why your quoting hardware types here. This has been discussed already. I could say this could all be resolved by incorporating an ARX to frontend old school backend storage, not to mention throwing in a couple of netapp boxes to handle your current content, and a data domain to keep you backups...
One guy on IRC commented one time that EMC shops are full of people who are broke and are constantly responding to pagers, shops with NetApps are equally as broke, but they get to sleep all night long.
They're the only piece of equipment I think I've come across that I think is really four 9's, maybe five...
I'd say a Netscaler 17000 instead of BigIP.We could go ahead and multiply the servers over a farm, and front end it all with a BigIP 6900 performing load balancing and rate shaping to ensure traffic is delivered, and responded to within a decided upon amount of time. The configuration options are unlimited.
Google runs on Netscaler and all the DoS problems we had at Amazon (>1 Gbps syn flood attacks) were absorbed by the Netscalers once we switched to them. I was actually in the process of writing a lightweight linux-based software proxy using epoll() to absorb the SYN flood attack with stacks of linux boxes in front of our Cisco load balancers right before Amazon pulled the trigger on Netscalers. I immediately stopped seeing all the DoS traffic that used to affect the webservers after the Netscalers were dropped in, and didn't see them again until I switched jobs and then had to deal with these effing layer-4 LVS servers that got DoS'd by garbage caused by 'normal' traffic from the Great Firewall of China (nobody who built that in China seems to care about RFC standards).
I don't know what kind of compression they're running, but at work we only get about 10:1 compression, tops, which is still only about 100Mbps peak utilization of the network to any virt host.Websites that are not virtualised across one server, sure. You can, however, tax the network, and the cycles of a server that is hosting multiple virtuals, as most webhosting sites such as dreamhost commonly do.
We're thinking of going to 4900M's or equivalent switches from Juniper and running a complete 10Gig access layer now (which i think is back in the regime of massive overkill).
Anyway, if this is a hosted service, then it really should be changed to a different hosting service... I could probably run this better out of my server at home on a DSL line based on how well its been doing lately, and i figured this was just Josh's or someone's pet project -- but if this is a pay-for service, that is kind of bad...
Actually, thing to do would probably be to just run it in EC2 and rent space from Amazon... That's also virtualized, but the servers there are not as oversubscribed and you really get a guarantee of how much RAM and CPU horsepower you get...
Yeah, clearly if you do get smacked by a worm, it isn't going to obey the kinds of courtesy in terms of hits per second that the Google webcrawlers at least attempt to...Absolutely, however worms have been very prominent, and have continued to be invasive, even though we have been knocking them out of the trees when we find them. When a distributed worm decides it wants to work hard on a system, and calls all of it's cloned buddies, it will slow your sh!t faster than a legit webcrawler will, any day of the week.
And I wouldn't really consider these worms to be script kiddie any more. Those are days of old. There are corporations, and very lucrative money making "gangs" hiring professional developers for this kind of malware these days.
And its Russian mafia and other criminal organizations operating outside of the reach of USA law enforcement which is responsible for a lot of the blackmailing operations and attacking people for profit...
I believe you read it correctly then... I must establish myself as the Dominant AlphaGeek when it comes to system engineering...Dashrynn wrote: all i read was "im more nerdier than the rest of you and heres a long list to prove me right"