Maxed server and resources, Sendy still slow

jereami

Hi Ben and gang,

After our last discussion here I went out and upgraded my server to a new one running these specs: Two - Xeon 2.2GHz, 12 cores, 32GB 1333MHz RAM, 3 - 600GB 15k RPM HHD. (I really needed to replace it anyway and this was as good an excuse as any).

Much to my surprise however, Sendy has not sped up one bit. It's still taking 4+hours to run through 113k sends. I've maxed memory allocation in PHP to 8GB but obviously I'm missing something. Is there an optimum configuration for PHP that I should be running?

Thanks in advance.

Ben

Hi @jereami,

Thanks for your feedback. There's no doubt the server you've upgraded to is pretty high powered. You've got that pretty much covered.

Content size

Can you provide me with the web version of the newsletter in question so I can calculate the size of your content? The contents of your email are base64 encoded as required by Amazon SES for reliable transport. Base64 encoded data takes up 33% more space than the original data.

The total size of data uploaded to Amazon SES taking into account Sendy sends multiple emails in parallel according to your send rate is size of 1 base64 encoded email x your_ses_send_rate.

Physical server location

The distance between the physical location of your sending server and Amazon SES's server affects latency too. Amazon SES's server is located in U.S. east while your server is located in U.S. west. While the distance is not that great (and I also don't think you can physically move your server), using a server located very near SES's server will be optimal, eg. EC2.

Lastly

Before each email is sent, Sendy does all the necessary and nothing more, eg. convert links, web version and unsubscribe tags, check for personalization tags and convert them etc. Then put them into a queue for sending.

I will continue my research and optimize wherever I can. The stage I realize takes the longest time is the parallel transfer of data to SES. Hence I'm requesting the web version of your newsletter so I can use it for analysis.

Thanks.

Ben

jereami

Thanks for the quick reply Ben. As you requested you can view the web version here.

Ben

Thanks for the web version, @jereami.

I did some calculations and found that your installation is trying to upload 1.86MB of data to Amazon SES for every iteration of transferring 70 emails in parallel.

It's beginning to look like something to do with the host's internet connection speed rather than your server's computing power. Another finding to add to the point is that the bottleneck is always when doing the transfer (not when adding emails to the queue, convert links and personalization tags etc).

Here's a breakdown:

• File size of one base_64 encoded email: 26.6kb
• File size of transfer per iteration: 26.6kb x 70 emails in parallel = 1.86MB
• Your host's internet connection's upload speed: 3GB / 4 hours = 208.73kb / second

As you can see your host's upload speed is 203.73kb / second, while, to reach the full speed of your SES send rate, you need the upload speed to be 1.86MB per second.

Your host's internet connection's upload speed is 88.79% short of what you need to reach your full speed (SES send rate).

Hope this helps to shed some light.

Thanks for all the information to make this analysis possible.

Ben

jereami

Wow, thanks for the tech spec's @Ben. I'll use this to beat my host into submission. I'll keep you in the loop, but your math makes sense to me.

jereami

@Ben, I'm being told by my Hosting provider that I have no bandwidth caps and a 10Gb connection through their data centers.

Ben

Hi @jereami,

Wow, thanks for the tech spec's @Ben. I'll use this to beat my host into submission. I'll keep you in the loop, but your math makes sense to me.

You're very welcome.

I'm being told by my Hosting provider that I have no bandwidth caps and a 10Gb connection through their data centers.

Bandwidth has nothing to do with the speed of transfer. 10GB connection means your server can upload 10GB per second? If so, why can't it upload 1.86MB per second (ruling out your server's hardware performance).

I agree that adding to queue, converting links and personalization tags takes time but they only take a fraction of a second. The rest is uploading to Amazon SES.

Ben

jereami

Thanks again @Ben. Well, I'm at a loss. They're blaming the application, you're blaming the hosting service. I'll see if I can get it escalated with their engineers and keep you posted.

jereami

After escalating this to the network engineers, this is what they had to say:

Hello,

Thank you for getting back to us.

Your current bandwidth usage is an insignificant fraction of what you're able to use totally on the server. We can see that your average usage is 199 B/s and that it has never peaked above 1500 B/s

verage: eth0 113.91 183.27 15.58 198.72 0.00 0.00 0.50

(According to sar's man page "B" indicates kilobytes. Your server never really hit anything more than about a Megabyte and a half)

At this rate you're not seeing any packetloss. If bandwidth is an issue it's not appearing locally.

The server has made this particularly hard to diagnose as well by disabling ping requests to check for latency. If amazon has any bandwidth checks they'll allow you to perform we can try them, but right now the remote server doesn't avail itself to the sort of diagnostic tools that can be used to test the connection more accurately.

At the end of the day bandwidth saturation isn't a bottleneck in this case, and troubleshooting should move to the next possibility.

Ben

Hi @jereami,

The network engineers are right. It's hard to diagnose SES's receiving speed.

Starting from 1.1.5 released recently, indexes will be added to your database if you upgrade to this version. This should speed up database reading speed. I'll see what else I can do.

For now I'll move this thread to http://sendy.co/discussion/702/look-into-database-usage-for-sending-optimization#Item_2

Thanks.

Ben